diff --git a/docs/superpowers/plans/2026-06-15-hybrid-memory-gateway.md b/docs/superpowers/plans/2026-06-15-hybrid-memory-gateway.md new file mode 100644 index 0000000..59dd852 --- /dev/null +++ b/docs/superpowers/plans/2026-06-15-hybrid-memory-gateway.md @@ -0,0 +1,338 @@ +# Hybrid Memory Gateway Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Preserve Beaver curated memory while adding an isolated, best-effort Memory Gateway recall and per-turn persistence layer enabled by hybrid configuration. + +**Architecture:** Curated `MemoryService`, frozen snapshots, and the `memory` tool remain unconditional. A new optional `MemoryGatewayService` wraps a small async HTTP client and is attached by `EngineLoader` only when hybrid configuration is valid. `AgentLoop` conditionally adds Gateway recall before provider execution and add/flush after normal completion without copying data between the two stores. + +**Tech Stack:** Python 3.11, dataclasses, httpx, SQLite-backed session audit events, pytest/pytest-asyncio. + +--- + +### Task 1: Add typed hybrid memory configuration + +**Files:** +- Modify: `app-instance/backend/beaver/foundation/config/schema.py` +- Modify: `app-instance/backend/beaver/foundation/config/loader.py` +- Modify: `app-instance/backend/beaver/foundation/config/__init__.py` +- Modify: `app-instance/backend/tests/unit/test_config_loader.py` + +- [ ] **Step 1: Write failing configuration tests** + +Add tests covering implicit hybrid defaults, explicit curated, complete explicit hybrid, invalid modes/scopes/ranges, and explicit hybrid missing credentials. Assert secret values never appear in errors. + +```python +def test_missing_memory_config_defaults_to_implicit_hybrid(tmp_path): + config = load_config(config_path=tmp_path / "missing.json") + assert config.memory.mode == "hybrid" + assert config.memory.explicit is False + +def test_explicit_hybrid_requires_gateway_credentials(tmp_path): + path = tmp_path / "config.json" + path.write_text('{"memory":{"mode":"hybrid","gateway":{"userKey":"secret"}}}') + with pytest.raises(ValueError) as exc: + load_config(config_path=path) + assert "secret" not in str(exc.value) +``` + +- [ ] **Step 2: Run configuration tests and verify RED** + +Run: `uv run pytest -q tests/unit/test_config_loader.py` + +Expected: failures because `BeaverConfig.memory` and memory parsing do not exist. + +- [ ] **Step 3: Implement minimal typed configuration** + +Add `MemoryGatewayConfig` and `MemoryConfig` dataclasses. Mark `user_key` with `repr=False`. Parse camelCase/snake_case fields, preserve `explicit`, and validate the confirmed rules. + +```python +@dataclass(slots=True) +class MemoryGatewayConfig: + base_url: str = "" + user_id: str = "" + user_key: str = field(default="", repr=False) + app_id: str = "default" + project_id: str = "default" + scope: list[str] = field(default_factory=lambda: ["current_chat", "resources"]) + top_k: int = 8 + timeout_seconds: float = 10.0 + +@dataclass(slots=True) +class MemoryConfig: + mode: str = "hybrid" + explicit: bool = False + gateway: MemoryGatewayConfig = field(default_factory=MemoryGatewayConfig) +``` + +- [ ] **Step 4: Run configuration tests and verify GREEN** + +Run: `uv run pytest -q tests/unit/test_config_loader.py` + +Expected: all tests pass. + +- [ ] **Step 5: Commit configuration support** + +```bash +git add app-instance/backend/beaver/foundation/config app-instance/backend/tests/unit/test_config_loader.py +git commit -m "feat(memory): add hybrid gateway configuration" +``` + +### Task 2: Implement the Memory Gateway client and isolated service + +**Files:** +- Create: `app-instance/backend/beaver/integrations/memory_gateway/__init__.py` +- Create: `app-instance/backend/beaver/integrations/memory_gateway/client.py` +- Create: `app-instance/backend/beaver/services/memory_gateway_service.py` +- Modify: `app-instance/backend/beaver/services/__init__.py` +- Create: `app-instance/backend/tests/unit/test_memory_gateway_service.py` + +- [ ] **Step 1: Write failing client/service tests** + +Test exact search/add/flush paths and payloads, result sanitization, empty recall, add-failure skipping flush, flush failure reporting, and secret-free errors. Use a fake client for service tests and monkeypatch `httpx.AsyncClient` for transport tests. + +```python +@pytest.mark.asyncio +async def test_persist_after_run_adds_two_messages_then_flushes(): + client = FakeGatewayClient() + service = MemoryGatewayService(config, client=client) + outcome = await service.persist_after_run( + session_id="web:alpha", + user_text="hello", + assistant_text="hi", + user_timestamp_ms=1000, + assistant_timestamp_ms=1001, + ) + assert outcome.add_succeeded is True + assert outcome.flush_succeeded is True + assert [call[0] for call in client.calls] == ["add", "flush"] +``` + +- [ ] **Step 2: Run service tests and verify RED** + +Run: `uv run pytest -q tests/unit/test_memory_gateway_service.py` + +Expected: import failure because the integration and service do not exist. + +- [ ] **Step 3: Implement the minimal async client** + +Create `MemoryGatewayClient` with `search`, `add`, and `flush`. Raise `MemoryGatewayClientError(operation, category, status_code)` without embedding request bodies or credentials. + +```python +async def search(self, payload: dict[str, Any]) -> dict[str, Any]: + return await self._post("search", "/memories/search", payload) +``` + +- [ ] **Step 4: Implement the isolated Gateway service** + +Create typed recall/persist outcome dataclasses. The service builds configured payloads, strips result fields to the approved allowlist, renders one reference message, and never imports or calls `MemoryStore`. + +```python +@dataclass(slots=True) +class GatewayRecallOutcome: + reference_messages: list[dict[str, str]] = field(default_factory=list) + result_count: int = 0 + error: MemoryGatewayClientError | None = None +``` + +- [ ] **Step 5: Run service tests and verify GREEN** + +Run: `uv run pytest -q tests/unit/test_memory_gateway_service.py` + +Expected: all tests pass. + +- [ ] **Step 6: Commit client and service** + +```bash +git add app-instance/backend/beaver/integrations/memory_gateway app-instance/backend/beaver/services app-instance/backend/tests/unit/test_memory_gateway_service.py +git commit -m "feat(memory): add memory gateway client and service" +``` + +### Task 3: Extend context assembly for ephemeral Gateway recall + +**Files:** +- Modify: `app-instance/backend/beaver/engine/context/builder.py` +- Modify: `app-instance/backend/tests/unit/test_context_builder.py` + +- [ ] **Step 1: Write failing context ordering tests** + +Verify reference messages appear after activated skill messages and before persisted history/current user input, while recalled text is absent from the system prompt. + +```python +def test_context_builder_places_reference_messages_before_history(): + result = ContextBuilder().build_messages(ContextBuildInput( + reference_messages=[{"role": "user", "content": "[MEMORY REFERENCE] old fact"}], + history=[{"role": "assistant", "content": "prior reply"}], + current_user_input="new question", + )) + assert result.messages[-3:] == [ + {"role": "user", "content": "[MEMORY REFERENCE] old fact"}, + {"role": "assistant", "content": "prior reply"}, + {"role": "user", "content": "new question"}, + ] +``` + +- [ ] **Step 2: Run context tests and verify RED** + +Run: `uv run pytest -q tests/unit/test_context_builder.py` + +Expected: `ContextBuildInput` rejects `reference_messages`. + +- [ ] **Step 3: Implement reference message support** + +Add `reference_messages` to `ContextBuildInput` and append normalized non-system messages immediately after skill activation messages. + +- [ ] **Step 4: Run context tests and verify GREEN** + +Run: `uv run pytest -q tests/unit/test_context_builder.py` + +Expected: all tests pass. + +- [ ] **Step 5: Commit context support** + +```bash +git add app-instance/backend/beaver/engine/context/builder.py app-instance/backend/tests/unit/test_context_builder.py +git commit -m "feat(memory): support ephemeral gateway recall context" +``` + +### Task 4: Wire the optional Gateway service into EngineLoader + +**Files:** +- Modify: `app-instance/backend/beaver/engine/loader.py` +- Modify: `app-instance/backend/tests/unit/test_imports.py` +- Create: `app-instance/backend/tests/unit/test_memory_gateway_loader.py` + +- [ ] **Step 1: Write failing loader tests** + +Cover explicit curated, explicit valid hybrid, implicit hybrid degradation with a sanitized warning, and explicit invalid hybrid rejection. Assert curated store and `memory` tool are present in every successful mode. + +- [ ] **Step 2: Run loader tests and verify RED** + +Run: `uv run pytest -q tests/unit/test_imports.py tests/unit/test_memory_gateway_loader.py` + +Expected: failures because `EngineLoadResult.memory_gateway_service` does not exist. + +- [ ] **Step 3: Implement loader wiring** + +Add optional dependency injection and result fields for `MemoryGatewayService`. Always initialize curated memory and register `MemoryTool`; initialize Gateway only for valid hybrid configuration. Log one warning when implicit hybrid lacks credentials. + +```python +memory_gateway_service = self._memory_gateway_service +if memory_gateway_service is None and config.memory.mode == "hybrid": + if config.memory.gateway.is_configured: + memory_gateway_service = MemoryGatewayService(config.memory.gateway) + elif not config.memory.explicit: + logger.warning("Memory Gateway is not configured; continuing with curated memory only") +``` + +- [ ] **Step 4: Run loader tests and verify GREEN** + +Run: `uv run pytest -q tests/unit/test_imports.py tests/unit/test_memory_gateway_loader.py` + +Expected: all tests pass. + +- [ ] **Step 5: Commit loader wiring** + +```bash +git add app-instance/backend/beaver/engine/loader.py app-instance/backend/tests/unit/test_imports.py app-instance/backend/tests/unit/test_memory_gateway_loader.py +git commit -m "feat(memory): initialize optional gateway layer" +``` + +### Task 5: Integrate Gateway recall, persistence, and audit events into AgentLoop + +**Files:** +- Modify: `app-instance/backend/beaver/engine/loop.py` +- Create: `app-instance/backend/tests/unit/test_memory_gateway_agent_loop.py` + +- [ ] **Step 1: Write failing successful-flow AgentLoop test** + +Use a fake provider and injected fake Gateway service. Verify curated snapshot remains in the system prompt, Gateway recall is outside it and before the current user prompt, and add/flush persistence receives only the original user and final assistant text. + +- [ ] **Step 2: Run the successful-flow test and verify RED** + +Run: `uv run pytest -q tests/unit/test_memory_gateway_agent_loop.py::test_hybrid_run_keeps_curated_memory_and_persists_gateway_turn` + +Expected: failure because `AgentLoop` does not call the Gateway service. + +- [ ] **Step 3: Implement pre-run recall and success audit** + +When `loaded.memory_gateway_service` exists, call recall before context assembly, append hidden success/failure events, pass returned reference messages into `ContextBuildInput`, and add the stable untrusted-reference rule through `extra_sections`. + +- [ ] **Step 4: Implement post-run persistence and audit** + +Capture positive millisecond timestamps, call `persist_after_run` after final text is known and before returning, and append hidden add/flush success/failure events. Do not invoke persistence in the exception path. + +- [ ] **Step 5: Add failing failure-path tests** + +Cover recall failure, add failure, and flush failure. Assert the returned `AgentRunResult` is unchanged, curated snapshot remains present, add failure skips flush, and audit payloads contain no configured key. + +- [ ] **Step 6: Run AgentLoop tests and verify GREEN** + +Run: `uv run pytest -q tests/unit/test_memory_gateway_agent_loop.py tests/unit/test_agent_loop.py tests/unit/test_agent_team_v1.py` + +Expected: all tests pass. + +- [ ] **Step 7: Commit AgentLoop integration** + +```bash +git add app-instance/backend/beaver/engine/loop.py app-instance/backend/tests/unit/test_memory_gateway_agent_loop.py +git commit -m "feat(memory): add hybrid gateway runtime flow" +``` + +### Task 6: Document configuration and run full verification + +**Files:** +- Modify: `app-instance/backend/README.md` +- Modify: `app-instance/backend/env_template` if it contains runtime config guidance + +- [ ] **Step 1: Update backend documentation** + +Document implicit hybrid mode, explicit curated mode, full hybrid JSON configuration, degradation/validation behavior, restart requirement, and the secrecy of `userKey`. + +- [ ] **Step 2: Run targeted tests** + +Run: + +```bash +uv run pytest -q \ + tests/unit/test_config_loader.py \ + tests/unit/test_memory_gateway_service.py \ + tests/unit/test_context_builder.py \ + tests/unit/test_memory_gateway_loader.py \ + tests/unit/test_memory_gateway_agent_loop.py \ + tests/unit/test_imports.py \ + tests/unit/test_agent_loop.py +``` + +Expected: all targeted tests pass. + +- [ ] **Step 3: Run the backend unit suite** + +Run: `uv run pytest -q tests/unit` + +Expected: all unit tests pass. + +- [ ] **Step 4: Compile changed Python packages** + +Run: `uv run python -m compileall -q beaver tests/unit` + +Expected: exit code 0 with no output. + +- [ ] **Step 5: Review secret handling and diff** + +Run: + +```bash +git diff --check +rg -n "userKey|user_key" app-instance/backend/beaver app-instance/backend/tests/unit/test_memory_gateway* app-instance/backend/README.md +git status --short +``` + +Expected: credentials appear only as field names or test fixtures; no real key is logged or committed. + +- [ ] **Step 6: Commit documentation and verification adjustments** + +```bash +git add app-instance/backend/README.md app-instance/backend/env_template +git commit -m "docs(memory): document hybrid gateway configuration" +```