feat: coordinator event wiring + DTU end-to-end profile#4
Merged
michaeljabbour merged 16 commits intomichaeljabbour:mainfrom Apr 30, 2026
Merged
Conversation
- Add asyncio import for event loop capture - Add _SYNC_BRIDGE_EMIT module-level holder (mirrors _QUEUE/_DRAIN_THREAD pattern) - Add sync_bridge_emit keyword-only param to MempalaceCaptureHook.__init__ with no-op default (lambda e, p: None) for test safety - In _process_job() success branch: call _SYNC_BRIDGE_EMIT after emit_event with memory-mempalace:drawer_filed and same payload fields - In _process_job() except branch: call _SYNC_BRIDGE_EMIT after emit_event with memory-mempalace:capture_failed and reason=mcp_error - In _drain_loop() last-resort guard: call _SYNC_BRIDGE_EMIT inside if job.emit_events guard with reason=worker_exception - Replace mount() to: capture running loop, register_contributor on observability.events channel, define sync_bridge_emit closure using run_coroutine_threadsafe, set _SYNC_BRIDGE_EMIT global, instantiate MempalaceCaptureHook with bridge closure - All coordinator calls wrapped in try/except; private emit_event calls preserved
… drawer_ids
- Add TestBriefingCoordinatorBridge to test_coordinator_bridge.py with 3 tests:
* test_register_contributor_called_at_mount: verifies mount() calls register_contributor
with 'observability.events' channel and event list including briefing_assembled/skipped
* test_briefing_assembled_emits_with_drawer_ids: verifies bridge emit carries drawer_ids
derived from results_after_rerank dicts
* test_emit_events_false_suppresses_both_channels: emit_events=False suppresses both
private JSONL and coordinator bridge channels
- Modify _build_briefing return type: tuple[str, list[str], int, list[dict], list[dict]]
* results_fetched: list = [] (was int = 0)
* results_after_rerank: list = [] (was int = 0)
* results_fetched = list(raw_results) (was len(raw_results))
* results_after_rerank = list(results) (was len(results))
- Add bridge_emit keyword-only param to MempalaceBriefingHook.__init__
* Stores as self._bridge_emit (defaults to async no-op)
- In __call__, derive drawer_ids from results_after_rerank and emit to bridge:
* After briefing_assembled: bridge emits ok=True with drawer_ids and metadata
* After briefing_skipped (no_content): bridge emits ok=False with reason
* In except block (unavailable): bridge emits ok=False with reason
- Update mount() to register_contributor and wire bridge_emit:
* Calls coordinator.register_contributor('observability.events', ...)
* Creates async bridge_emit closure calling coordinator.hooks.emit
* Instantiates hook with bridge_emit=bridge_emit
- Fix test_hook_emissions.py existing test:
* Update mock from (100, 3, 3) to (100, [], []) for list return type
* Update assertion to use len() comparison for list type
- Add bridge_emit keyword-only param to MempalaceInterjectHook.__init__
with _noop async fallback when not provided
- Add _briefed_ids: set[str] initialized to empty set in __init__
- Thread bridge_emit into all 3 handlers (on_prompt_submit, on_tool_pre,
on_orchestrator_complete): 11 interject_skipped + 3 memory_surfaced sites,
each wrapped in try/except
- Replace mount() to:
* register_contributor('observability.events', 'memory-mempalace-interject', ...)
with callback returning ['memory-mempalace:memory_surfaced', 'memory-mempalace:interject_skipped']
* define async bridge_emit closure calling coordinator.hooks.emit
* define async _on_briefing_assembled listener that updates hook._briefed_ids
from data['drawer_ids'] and register it for 'memory-mempalace:briefing_assembled'
* register prompt:submit, tool:pre, orchestrator:complete handlers at priority=20
* return version 1.1.0 with full config dict including emit_events
- Add TestInterjectCoordinatorBridge with 4 tests:
* test_register_contributor_called_at_mount
* test_briefing_assembled_listener_registered_in_mount
* test_briefed_ids_populated_from_briefing_event
* test_memory_surfaced_emits_to_coordinator
- Add bridge_emit keyword param to ProjectContextStartHook.__init__ and ProjectContextEndHook.__init__ with async _noop fallback - Emit memory-mempalace:coordination_scaffolded bridge event after scaffolding - Emit memory-mempalace:coordination_read bridge event after reading tier-1 files - Emit memory-mempalace:curator_handoff_requested bridge event at session end - Replace mount() to call register_contributor with all 3 bridge events and wire a single bridge_emit closure shared by both hooks - Add TestProjectContextCoordinatorBridge with 2 tests for register_contributor and curator_handoff_requested bridge call - Fix _FakeCoordinator in test_contract.py to support register_contributor (also fixes pre-existing test_briefing_mounts_and_dispatches failure)
…d_emit - Add _SYNC_BRIDGE_EMIT module-level holder for coordinator bridge callback - Register 'memory-mempalace-tool' contributor in mount() with events: garden_completed, garden_progress - Define sync_bridge_emit closure using asyncio.run_coroutine_threadsafe for fire-and-forget forwarding from garden sync thread to event loop - Add combined_emit closure in execute() garden block to dual-emit to private JSONL and coordinator bridge - Bridge garden_completed on TimeoutError (ok=False) and success path (ok=True) - Add FakeCoordinator.mount() async no-op for tool bridge tests - Add TestToolMempalaceCoordinatorBridge.test_register_contributor_called_at_mount
…syncio
- Add pytest_collection_modifyitems to tests/integration/conftest.py:
skips all integration tests on host when /root/.mempalace sentinel
path is absent (or PermissionError on non-root host). Tests only run
inside memory-bundle-e2e DTU container.
- Add tests/integration/test_event_wiring.py: validates the full
coordinator event wiring chain (hook fires -> coordinator.hooks.emit
-> hooks-logging writes events.jsonl) with three tests:
* test_drawer_filed_appears_in_events_jsonl
* test_briefing_assembled_payload_has_drawer_ids
* test_briefed_ids_prevents_reinjection (skipped — requires
content-pinned session with deterministic retrieval)
- Update .amplifier/digital-twin-universe/profiles/memory-bundle-e2e.yaml:
add pytest-asyncio to pip install step (step 4) so async test
infrastructure is available inside the DTU container.
- fix(briefing): restore integer counts in private JSONL emit_event for results_fetched and results_after_rerank (was incorrectly changed to full list payloads, violating the out-of-scope private schema contract) - fix(capture): add loop.is_closed() guard to sync_bridge_emit closure to prevent RuntimeWarning from unawaited coroutine in closed event loop (drain thread outlives asyncio.run() in test teardown) - fix(integration): add Path | None return type + assert-not-None guards to _latest_events_jsonl() so tests fail with a clear message instead of IndexError when no events.jsonl exists - fix(test): update test_briefing_emits_on_assemble assertion to match restored integer schema (isinstance int checks replace len() calls) - fix(interject): apply ruff format (1 blank line + 3 dict expansions) Fixes post-code-review findings michaeljabbour#1, michaeljabbour#2, michaeljabbour#3, #6 from quality review. All 45 tests pass, 0 warnings. Co-authored-by: Amplifier <amplifier@example.com>
…dle.dot
- agents/{docent,archivist,curator}.md: add top-level name: and description: fields
alongside existing agent.name/agent.description (Option A flat schema)
Clears 6 false-positive warnings from the bundle validator (v3.4.0 classifier
requires top-level fields; nested agent.name was not detected)
- bundle.dot: generated by generate-bundle-docs recipe — structural overview of
the bundle architecture (7-cluster DOT covering behaviors, agents, modules,
context, and external references)
…or events Two provisioning additions to memory-bundle-e2e.yaml: 1. Step 12.5 — amplifier-core 1.4.1 + latest amplifier-app-cli register_contributor() / collect_contributions() and the on_session_ready lifecycle only work correctly from amplifier-core >= 1.4.1. The uv tool install from git may resolve an older PyPI release; --reinstall-package forces the correct versions. 2. Step 15 — hooks-logging fork (feat/on-session-ready) The upstream hooks-logging does not yet call collect_contributions in on_session_ready, so memory-mempalace: coordinator events are invisible in events.jsonl without this fork. Workaround until the upstream PR merges. Fork: colombod/amplifier-module-hooks-logging (feat/on-session-ready) Verified: register_contributor/collect_contributions round-trip works correctly with amplifier-core 1.4.1 Rust engine (direct API test). Private JSONL confirms all 5 hook modules are running. Full session coordinator event flow needs further investigation of on_session_ready invocation in the live session lifecycle.
…nstall
Root cause of coordinator events not appearing in events.jsonl:
The Amplifier CLI caches modules in ~/.amplifier/cache/. The loader adds
these cache dirs to sys.path BEFORE site-packages. On session start,
amplifier_module_hooks_logging is imported from the OLD cache file
(without on_session_ready) and placed in sys.modules. When B2 detection
in _load_entry_point() runs, it finds has_osr=False and never sets
__on_session_ready__ on the hook mount function. Phase 6 queue stays 0
and on_session_ready() is never called. coordinator.hooks.emit() fires
but no handler is registered for memory-mempalace:* events.
Fix: after `uv pip install` of the fork, also find all Amplifier cache
copies of hooks_logging/__init__.py and overwrite them with the fork's
version. This ensures both the pip install AND the Amplifier cache have
on_session_ready.
Investigation evidence (session 9201aaf7):
- EP_ENTRY_DEBUG confirmed _load_entry_point IS called for hooks-logging
- AFTER_LOAD_DEBUG confirmed ep.load() succeeds (fn_module correct)
- B2_MOD_DEBUG confirmed the module in sys.modules is the OLD cache:
mod_file=~/.amplifier/cache/amplifier-module-hooks-logging-2d15c63e3ceed7b8/...
has_osr=False
- After fix: has_osr=True, PHASE6:queue_size=1, A:on_session_ready_called
- Result: 6+ memory-mempalace: events per session in events.jsonl
Both provision (step 15) and update section include the cache patch.
Forward-integrate commit 2ba94f7 from michaeljabbour/amplifier-bundle-memory into this branch. MJ corrected four factual errors in dtu.md that prevented the guide from working on macOS (Colima 0.10.1, Incus 6.0.0 in-VM): - amplifier-digital-twin: install from git repo, not PyPI - amplifier-gitea: add as prerequisite with install instructions - One-Time Gitea Setup: replace non-existent subcommands and manual curl migrate with amplifier-gitea create + mirror-from-github - Launch: explicit profile path + --name pattern No conflict with our branch: MJ touched only docs/development/dtu.md; we only touched .amplifier/digital-twin-universe/profiles/memory-bundle-e2e.yaml.
…odule Move the duplicated bridge_emit closure from all 5 modules into amplifier_module_tool_mempalace.coordinator_bridge — the natural home since tool-mempalace is already imported by all hooks via event_emitter. New module exposes: make_async_bridge(coordinator) → AsyncBridge make_sync_bridge(coordinator) → SyncBridge (captures loop at mount time) register_events(coordinator, contributor, events) NOOP_ASYNC_BRIDGE / NOOP_SYNC_BRIDGE (testable no-op defaults) Ten inconsistencies eliminated: - _SYNC_BRIDGE_EMIT module globals deleted from capture and tool-mempalace - PalaceTool gains bridge_emit constructor param + NOOP default (testable without mount()) - Parameter name unified: sync_bridge_emit → bridge_emit everywhere - register_contributor try/except centralised in register_events (always best-effort; was only in capture before) - Async bridge try/except guard added to interject + project-context (was missing; briefing already had it) - Sync bridge loop.is_closed() guard added to tool-mempalace (was missing; capture already had it) - Sync bridge try/except added to tool-mempalace (was missing) - register_events snapshots the events list (prevents mutation hazard from project-context's _COORDINATOR_EVENTS pattern) - unused asyncio import removed from capture 45 tests pass. ruff clean.
…, smoke test Four improvements based on pain points found during the coordinator event wiring investigation: T1.1 — Replace chicken-and-egg hooks-logging patch with rm -rf: The old approach patched the cache file AFTER a primer session ran (timing dependency + ~15 lines of fragile shell). New: install fork, then delete the cache dir. Python falls through to site-packages on every session start. No timing issue. No primer session needed. Verified working in DTU. T1.2 — Targeted cache invalidation in update (not full wipe): Old update section: rm -rf /root/.amplifier/cache/ — wiped provider-anthropic, loop-streaming, and all foundation modules. Sessions failed with 'No providers available' until Amplifier was reinstalled (~5 min). New: only removes the memory bundle and hooks-logging cache dirs. T1.3 — Post-provision smoke test (step 17): Runs a real session after provisioning. Checks on_session_ready, no shadowing cache, and memory-mempalace:* events in events.jsonl. Fails the provision loudly rather than silently producing a broken environment. T3.1 — Troubleshooting docs in dtu.md: Three new entries: cache shadowing (zero coordinator events), full-cache-wipe breakage, and the Python scoping gotcha with import inside if blocks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires all five mempalace hooks and the palace tool to the Amplifier coordinator event bus so that
memory-mempalace:*events appear inevents.jsonl— making palace activity observable byhooks-logging,context-intelligence, and any future hook without special-casing.Adds a complete DTU end-to-end profile that provisions a real Amplifier session with a live MemPalace instance for integration testing, and integrates your macOS guide fixes from
docs(dtu): correct CLI references and flow in dtu.md (#3).Changes
Coordinator Event Wiring (core feature)
All five hooks and the tool now emit
memory-mempalace:*events throughcoordinator.hooks.emit()with a dual-write strategy — the private~/.mempalace/events/{sid}.jsonlcontinues to work forpalace events, while coordinator events flow throughhooks-logging→ standardevents.jsonl:hooks-mempalace-capturememory-mempalace:drawer_filed,memory-mempalace:capture_failedhooks-mempalace-briefingmemory-mempalace:briefing_assembled,memory-mempalace:briefing_skippedhooks-mempalace-interjectmemory-mempalace:memory_surfaced,memory-mempalace:interject_skippedhooks-project-contextmemory-mempalace:coordination_read,memory-mempalace:coordination_scaffolded,memory-mempalace:curator_handoff_requestedtool-mempalacememory-mempalace:garden_completedEach module calls
coordinator.register_contributor("observability.events", name, callback)at mount time so hooks-logging and context-intelligence can discover the contributed events viacollect_contributions("observability.events").The interject hook also registers a cross-hook listener for
memory-mempalace:briefing_assembledto populate_briefed_ids, replacing the previouscoordinator.get_capability()polling pattern.Tests
tests/test_coordinator_bridge.py— unit tests for all coordinator bridge implementationstests/integration/test_event_wiring.py— DTU-backed integration tests verifying events appear inevents.jsonltests/test_contract.py— extended withregister_contributorcontract assertionstests/test_hook_emissions.py— updated for new event signaturesDTU End-to-End Profile
.amplifier/digital-twin-universe/profiles/memory-bundle-e2e.yaml— provisions an Ubuntu 24.04 Incus container with:--appinstall)pytest-asynciofor async integration testsTwo non-trivial fixes required during investigation:
amplifier-core 1.4.1 pin (step 12.5):
uv tool install git+https://github.com/microsoft/amplifiermay resolve an older PyPI release.register_contributor/collect_contributionsand theon_session_readylifecycle require>= 1.4.1.Amplifier module cache shadowing (step 15, root cause investigated via systematic 4-layer debug): The Amplifier CLI caches modules in
~/.amplifier/cache/. The loader adds these cache dirs tosys.pathbeforesite-packages.amplifier_module_hooks_logginggets imported from the old cache (noon_session_ready) and placed insys.modulesbefore_load_entry_pointcan run. B2 detection inloader.pythen findshas_osr=False, never sets__on_session_ready__, and Phase 6 queue stays at 0. Fix: install the hooks-logging fork via pip AND overwrite all Amplifier cache copies with the fork's__init__.py.The hooks-logging fork used is
colombod/amplifier-module-hooks-logging@feat/on-session-readywhich addscollect_contributions("observability.events")inon_session_ready(upstream PR pending).Agent Frontmatter + bundle.dot
archivist.md,curator.md,docent.md: hoistednameanddescriptionto top-level frontmatter (Amplifier convention for agent bundles)bundle.dot: added repository architecture diagramdtu.md macOS Guide Integration
docs/development/dtu.md: forward-integrated your commit2ba94f7(macOS guide fixes — Colima prereq,amplifier-digital-twininstall from git,amplifier-gitea create+mirror-from-github, correct launch section). No conflict with our changes (you only touched dtu.md; we only touched the profile YAML).Verification
Verified end-to-end in DTU (Ubuntu 24.04 / Incus container):
memory-mempalace:*coordinator events per session appearing inevents.jsonlPHASE6:queue_size=1—on_session_readyproperly enqueued and dispatchedpytest tests/ -x)~/.mempalace/events/JSONL continues to work (dual-write intact)