# Hybrid Memory Gateway Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Preserve Beaver curated memory while adding an isolated, best-effort Memory Gateway recall and per-turn persistence layer enabled by hybrid configuration.

**Architecture:** Curated `MemoryService`, frozen snapshots, and the `memory` tool remain unconditional. A new optional `MemoryGatewayService` wraps a small async HTTP client and is attached by `EngineLoader` only when hybrid configuration is valid. `AgentLoop` conditionally adds Gateway recall before provider execution and add/flush after normal completion without copying data between the two stores.

**Tech Stack:** Python 3.11, dataclasses, httpx, SQLite-backed session audit events, pytest/pytest-asyncio.

---

### Task 1: Add typed hybrid memory configuration

**Files:**
- Modify: `app-instance/backend/beaver/foundation/config/schema.py`
- Modify: `app-instance/backend/beaver/foundation/config/loader.py`
- Modify: `app-instance/backend/beaver/foundation/config/__init__.py`
- Modify: `app-instance/backend/tests/unit/test_config_loader.py`

- [ ] **Step 1: Write failing configuration tests**

Add tests covering implicit hybrid defaults, explicit curated, complete explicit hybrid, invalid modes/scopes/ranges, and explicit hybrid missing credentials. Assert secret values never appear in errors.

```python
def test_missing_memory_config_defaults_to_implicit_hybrid(tmp_path):
    config = load_config(config_path=tmp_path / "missing.json")
    assert config.memory.mode == "hybrid"
    assert config.memory.explicit is False

def test_explicit_hybrid_requires_gateway_credentials(tmp_path):
    path = tmp_path / "config.json"
    path.write_text('{"memory":{"mode":"hybrid","gateway":{"userKey":"secret"}}}')
    with pytest.raises(ValueError) as exc:
        load_config(config_path=path)
    assert "secret" not in str(exc.value)
```

- [ ] **Step 2: Run configuration tests and verify RED**

Run: `uv run pytest -q tests/unit/test_config_loader.py`

Expected: failures because `BeaverConfig.memory` and memory parsing do not exist.

- [ ] **Step 3: Implement minimal typed configuration**

Add `MemoryGatewayConfig` and `MemoryConfig` dataclasses. Mark `user_key` with `repr=False`. Parse camelCase/snake_case fields, preserve `explicit`, and validate the confirmed rules.

```python
@dataclass(slots=True)
class MemoryGatewayConfig:
    base_url: str = ""
    user_id: str = ""
    user_key: str = field(default="", repr=False)
    app_id: str = "default"
    project_id: str = "default"
    scope: list[str] = field(default_factory=lambda: ["current_chat", "resources"])
    top_k: int = 8
    timeout_seconds: float = 10.0

@dataclass(slots=True)
class MemoryConfig:
    mode: str = "hybrid"
    explicit: bool = False
    gateway: MemoryGatewayConfig = field(default_factory=MemoryGatewayConfig)
```

- [ ] **Step 4: Run configuration tests and verify GREEN**

Run: `uv run pytest -q tests/unit/test_config_loader.py`

Expected: all tests pass.

- [ ] **Step 5: Commit configuration support**

```bash
git add app-instance/backend/beaver/foundation/config app-instance/backend/tests/unit/test_config_loader.py
git commit -m "feat(memory): add hybrid gateway configuration"
```

### Task 2: Implement the Memory Gateway client and isolated service

**Files:**
- Create: `app-instance/backend/beaver/integrations/memory_gateway/__init__.py`
- Create: `app-instance/backend/beaver/integrations/memory_gateway/client.py`
- Create: `app-instance/backend/beaver/services/memory_gateway_service.py`
- Modify: `app-instance/backend/beaver/services/__init__.py`
- Create: `app-instance/backend/tests/unit/test_memory_gateway_service.py`

- [ ] **Step 1: Write failing client/service tests**

Test exact search/add/flush paths and payloads, result sanitization, empty recall, add-failure skipping flush, flush failure reporting, and secret-free errors. Use a fake client for service tests and monkeypatch `httpx.AsyncClient` for transport tests.

```python
@pytest.mark.asyncio
async def test_persist_after_run_adds_two_messages_then_flushes():
    client = FakeGatewayClient()
    service = MemoryGatewayService(config, client=client)
    outcome = await service.persist_after_run(
        session_id="web:alpha",
        user_text="hello",
        assistant_text="hi",
        user_timestamp_ms=1000,
        assistant_timestamp_ms=1001,
    )
    assert outcome.add_succeeded is True
    assert outcome.flush_succeeded is True
    assert [call[0] for call in client.calls] == ["add", "flush"]
```

- [ ] **Step 2: Run service tests and verify RED**

Run: `uv run pytest -q tests/unit/test_memory_gateway_service.py`

Expected: import failure because the integration and service do not exist.

- [ ] **Step 3: Implement the minimal async client**

Create `MemoryGatewayClient` with `search`, `add`, and `flush`. Raise `MemoryGatewayClientError(operation, category, status_code)` without embedding request bodies or credentials.

```python
async def search(self, payload: dict[str, Any]) -> dict[str, Any]:
    return await self._post("search", "/memories/search", payload)
```

- [ ] **Step 4: Implement the isolated Gateway service**

Create typed recall/persist outcome dataclasses. The service builds configured payloads, strips result fields to the approved allowlist, renders one reference message, and never imports or calls `MemoryStore`.

```python
@dataclass(slots=True)
class GatewayRecallOutcome:
    reference_messages: list[dict[str, str]] = field(default_factory=list)
    result_count: int = 0
    error: MemoryGatewayClientError | None = None
```

- [ ] **Step 5: Run service tests and verify GREEN**

Run: `uv run pytest -q tests/unit/test_memory_gateway_service.py`

Expected: all tests pass.

- [ ] **Step 6: Commit client and service**

```bash
git add app-instance/backend/beaver/integrations/memory_gateway app-instance/backend/beaver/services app-instance/backend/tests/unit/test_memory_gateway_service.py
git commit -m "feat(memory): add memory gateway client and service"
```

### Task 3: Extend context assembly for ephemeral Gateway recall

**Files:**
- Modify: `app-instance/backend/beaver/engine/context/builder.py`
- Modify: `app-instance/backend/tests/unit/test_context_builder.py`

- [ ] **Step 1: Write failing context ordering tests**

Verify reference messages appear after activated skill messages and before persisted history/current user input, while recalled text is absent from the system prompt.

```python
def test_context_builder_places_reference_messages_before_history():
    result = ContextBuilder().build_messages(ContextBuildInput(
        reference_messages=[{"role": "user", "content": "[MEMORY REFERENCE] old fact"}],
        history=[{"role": "assistant", "content": "prior reply"}],
        current_user_input="new question",
    ))
    assert result.messages[-3:] == [
        {"role": "user", "content": "[MEMORY REFERENCE] old fact"},
        {"role": "assistant", "content": "prior reply"},
        {"role": "user", "content": "new question"},
    ]
```

- [ ] **Step 2: Run context tests and verify RED**

Run: `uv run pytest -q tests/unit/test_context_builder.py`

Expected: `ContextBuildInput` rejects `reference_messages`.

- [ ] **Step 3: Implement reference message support**

Add `reference_messages` to `ContextBuildInput` and append normalized non-system messages immediately after skill activation messages.

- [ ] **Step 4: Run context tests and verify GREEN**

Run: `uv run pytest -q tests/unit/test_context_builder.py`

Expected: all tests pass.

- [ ] **Step 5: Commit context support**

```bash
git add app-instance/backend/beaver/engine/context/builder.py app-instance/backend/tests/unit/test_context_builder.py
git commit -m "feat(memory): support ephemeral gateway recall context"
```

### Task 4: Wire the optional Gateway service into EngineLoader

**Files:**
- Modify: `app-instance/backend/beaver/engine/loader.py`
- Modify: `app-instance/backend/tests/unit/test_imports.py`
- Create: `app-instance/backend/tests/unit/test_memory_gateway_loader.py`

- [ ] **Step 1: Write failing loader tests**

Cover explicit curated, explicit valid hybrid, implicit hybrid degradation with a sanitized warning, and explicit invalid hybrid rejection. Assert curated store and `memory` tool are present in every successful mode.

- [ ] **Step 2: Run loader tests and verify RED**

Run: `uv run pytest -q tests/unit/test_imports.py tests/unit/test_memory_gateway_loader.py`

Expected: failures because `EngineLoadResult.memory_gateway_service` does not exist.

- [ ] **Step 3: Implement loader wiring**

Add optional dependency injection and result fields for `MemoryGatewayService`. Always initialize curated memory and register `MemoryTool`; initialize Gateway only for valid hybrid configuration. Log one warning when implicit hybrid lacks credentials.

```python
memory_gateway_service = self._memory_gateway_service
if memory_gateway_service is None and config.memory.mode == "hybrid":
    if config.memory.gateway.is_configured:
        memory_gateway_service = MemoryGatewayService(config.memory.gateway)
    elif not config.memory.explicit:
        logger.warning("Memory Gateway is not configured; continuing with curated memory only")
```

- [ ] **Step 4: Run loader tests and verify GREEN**

Run: `uv run pytest -q tests/unit/test_imports.py tests/unit/test_memory_gateway_loader.py`

Expected: all tests pass.

- [ ] **Step 5: Commit loader wiring**

```bash
git add app-instance/backend/beaver/engine/loader.py app-instance/backend/tests/unit/test_imports.py app-instance/backend/tests/unit/test_memory_gateway_loader.py
git commit -m "feat(memory): initialize optional gateway layer"
```

### Task 5: Integrate Gateway recall, persistence, and audit events into AgentLoop

**Files:**
- Modify: `app-instance/backend/beaver/engine/loop.py`
- Create: `app-instance/backend/tests/unit/test_memory_gateway_agent_loop.py`

- [ ] **Step 1: Write failing successful-flow AgentLoop test**

Use a fake provider and injected fake Gateway service. Verify curated snapshot remains in the system prompt, Gateway recall is outside it and before the current user prompt, and add/flush persistence receives only the original user and final assistant text.

- [ ] **Step 2: Run the successful-flow test and verify RED**

Run: `uv run pytest -q tests/unit/test_memory_gateway_agent_loop.py::test_hybrid_run_keeps_curated_memory_and_persists_gateway_turn`

Expected: failure because `AgentLoop` does not call the Gateway service.

- [ ] **Step 3: Implement pre-run recall and success audit**

When `loaded.memory_gateway_service` exists, call recall before context assembly, append hidden success/failure events, pass returned reference messages into `ContextBuildInput`, and add the stable untrusted-reference rule through `extra_sections`.

- [ ] **Step 4: Implement post-run persistence and audit**

Capture positive millisecond timestamps, call `persist_after_run` after final text is known and before returning, and append hidden add/flush success/failure events. Do not invoke persistence in the exception path.

- [ ] **Step 5: Add failing failure-path tests**

Cover recall failure, add failure, and flush failure. Assert the returned `AgentRunResult` is unchanged, curated snapshot remains present, add failure skips flush, and audit payloads contain no configured key.

- [ ] **Step 6: Run AgentLoop tests and verify GREEN**

Run: `uv run pytest -q tests/unit/test_memory_gateway_agent_loop.py tests/unit/test_agent_loop.py tests/unit/test_agent_team_v1.py`

Expected: all tests pass.

- [ ] **Step 7: Commit AgentLoop integration**

```bash
git add app-instance/backend/beaver/engine/loop.py app-instance/backend/tests/unit/test_memory_gateway_agent_loop.py
git commit -m "feat(memory): add hybrid gateway runtime flow"
```

### Task 6: Document configuration and run full verification

**Files:**
- Modify: `app-instance/backend/README.md`
- Modify: `app-instance/backend/env_template` if it contains runtime config guidance

- [ ] **Step 1: Update backend documentation**

Document implicit hybrid mode, explicit curated mode, full hybrid JSON configuration, degradation/validation behavior, restart requirement, and the secrecy of `userKey`.

- [ ] **Step 2: Run targeted tests**

Run:

```bash
uv run pytest -q \
  tests/unit/test_config_loader.py \
  tests/unit/test_memory_gateway_service.py \
  tests/unit/test_context_builder.py \
  tests/unit/test_memory_gateway_loader.py \
  tests/unit/test_memory_gateway_agent_loop.py \
  tests/unit/test_imports.py \
  tests/unit/test_agent_loop.py
```

Expected: all targeted tests pass.

- [ ] **Step 3: Run the backend unit suite**

Run: `uv run pytest -q tests/unit`

Expected: all unit tests pass.

- [ ] **Step 4: Compile changed Python packages**

Run: `uv run python -m compileall -q beaver tests/unit`

Expected: exit code 0 with no output.

- [ ] **Step 5: Review secret handling and diff**

Run:

```bash
git diff --check
rg -n "userKey|user_key" app-instance/backend/beaver app-instance/backend/tests/unit/test_memory_gateway* app-instance/backend/README.md
git status --short
```

Expected: credentials appear only as field names or test fixtures; no real key is logged or committed.

- [ ] **Step 6: Commit documentation and verification adjustments**

```bash
git add app-instance/backend/README.md app-instance/backend/env_template
git commit -m "docs(memory): document hybrid gateway configuration"
```