feat(memory-gateway): merge memory mode with main
This commit is contained in:
338
docs/superpowers/plans/2026-06-15-hybrid-memory-gateway.md
Normal file
338
docs/superpowers/plans/2026-06-15-hybrid-memory-gateway.md
Normal file
@ -0,0 +1,338 @@
|
||||
# Hybrid Memory Gateway Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Preserve Beaver curated memory while adding an isolated, best-effort Memory Gateway recall and per-turn persistence layer enabled by hybrid configuration.
|
||||
|
||||
**Architecture:** Curated `MemoryService`, frozen snapshots, and the `memory` tool remain unconditional. A new optional `MemoryGatewayService` wraps a small async HTTP client and is attached by `EngineLoader` only when hybrid configuration is valid. `AgentLoop` conditionally adds Gateway recall before provider execution and add/flush after normal completion without copying data between the two stores.
|
||||
|
||||
**Tech Stack:** Python 3.11, dataclasses, httpx, SQLite-backed session audit events, pytest/pytest-asyncio.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add typed hybrid memory configuration
|
||||
|
||||
**Files:**
|
||||
- Modify: `app-instance/backend/beaver/foundation/config/schema.py`
|
||||
- Modify: `app-instance/backend/beaver/foundation/config/loader.py`
|
||||
- Modify: `app-instance/backend/beaver/foundation/config/__init__.py`
|
||||
- Modify: `app-instance/backend/tests/unit/test_config_loader.py`
|
||||
|
||||
- [ ] **Step 1: Write failing configuration tests**
|
||||
|
||||
Add tests covering implicit hybrid defaults, explicit curated, complete explicit hybrid, invalid modes/scopes/ranges, and explicit hybrid missing credentials. Assert secret values never appear in errors.
|
||||
|
||||
```python
|
||||
def test_missing_memory_config_defaults_to_implicit_hybrid(tmp_path):
|
||||
config = load_config(config_path=tmp_path / "missing.json")
|
||||
assert config.memory.mode == "hybrid"
|
||||
assert config.memory.explicit is False
|
||||
|
||||
def test_explicit_hybrid_requires_gateway_credentials(tmp_path):
|
||||
path = tmp_path / "config.json"
|
||||
path.write_text('{"memory":{"mode":"hybrid","gateway":{"userKey":"secret"}}}')
|
||||
with pytest.raises(ValueError) as exc:
|
||||
load_config(config_path=path)
|
||||
assert "secret" not in str(exc.value)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run configuration tests and verify RED**
|
||||
|
||||
Run: `uv run pytest -q tests/unit/test_config_loader.py`
|
||||
|
||||
Expected: failures because `BeaverConfig.memory` and memory parsing do not exist.
|
||||
|
||||
- [ ] **Step 3: Implement minimal typed configuration**
|
||||
|
||||
Add `MemoryGatewayConfig` and `MemoryConfig` dataclasses. Mark `user_key` with `repr=False`. Parse camelCase/snake_case fields, preserve `explicit`, and validate the confirmed rules.
|
||||
|
||||
```python
|
||||
@dataclass(slots=True)
|
||||
class MemoryGatewayConfig:
|
||||
base_url: str = ""
|
||||
user_id: str = ""
|
||||
user_key: str = field(default="", repr=False)
|
||||
app_id: str = "default"
|
||||
project_id: str = "default"
|
||||
scope: list[str] = field(default_factory=lambda: ["current_chat", "resources"])
|
||||
top_k: int = 8
|
||||
timeout_seconds: float = 10.0
|
||||
|
||||
@dataclass(slots=True)
|
||||
class MemoryConfig:
|
||||
mode: str = "hybrid"
|
||||
explicit: bool = False
|
||||
gateway: MemoryGatewayConfig = field(default_factory=MemoryGatewayConfig)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run configuration tests and verify GREEN**
|
||||
|
||||
Run: `uv run pytest -q tests/unit/test_config_loader.py`
|
||||
|
||||
Expected: all tests pass.
|
||||
|
||||
- [ ] **Step 5: Commit configuration support**
|
||||
|
||||
```bash
|
||||
git add app-instance/backend/beaver/foundation/config app-instance/backend/tests/unit/test_config_loader.py
|
||||
git commit -m "feat(memory): add hybrid gateway configuration"
|
||||
```
|
||||
|
||||
### Task 2: Implement the Memory Gateway client and isolated service
|
||||
|
||||
**Files:**
|
||||
- Create: `app-instance/backend/beaver/integrations/memory_gateway/__init__.py`
|
||||
- Create: `app-instance/backend/beaver/integrations/memory_gateway/client.py`
|
||||
- Create: `app-instance/backend/beaver/services/memory_gateway_service.py`
|
||||
- Modify: `app-instance/backend/beaver/services/__init__.py`
|
||||
- Create: `app-instance/backend/tests/unit/test_memory_gateway_service.py`
|
||||
|
||||
- [ ] **Step 1: Write failing client/service tests**
|
||||
|
||||
Test exact search/add/flush paths and payloads, result sanitization, empty recall, add-failure skipping flush, flush failure reporting, and secret-free errors. Use a fake client for service tests and monkeypatch `httpx.AsyncClient` for transport tests.
|
||||
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_persist_after_run_adds_two_messages_then_flushes():
|
||||
client = FakeGatewayClient()
|
||||
service = MemoryGatewayService(config, client=client)
|
||||
outcome = await service.persist_after_run(
|
||||
session_id="web:alpha",
|
||||
user_text="hello",
|
||||
assistant_text="hi",
|
||||
user_timestamp_ms=1000,
|
||||
assistant_timestamp_ms=1001,
|
||||
)
|
||||
assert outcome.add_succeeded is True
|
||||
assert outcome.flush_succeeded is True
|
||||
assert [call[0] for call in client.calls] == ["add", "flush"]
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run service tests and verify RED**
|
||||
|
||||
Run: `uv run pytest -q tests/unit/test_memory_gateway_service.py`
|
||||
|
||||
Expected: import failure because the integration and service do not exist.
|
||||
|
||||
- [ ] **Step 3: Implement the minimal async client**
|
||||
|
||||
Create `MemoryGatewayClient` with `search`, `add`, and `flush`. Raise `MemoryGatewayClientError(operation, category, status_code)` without embedding request bodies or credentials.
|
||||
|
||||
```python
|
||||
async def search(self, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
return await self._post("search", "/memories/search", payload)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Implement the isolated Gateway service**
|
||||
|
||||
Create typed recall/persist outcome dataclasses. The service builds configured payloads, strips result fields to the approved allowlist, renders one reference message, and never imports or calls `MemoryStore`.
|
||||
|
||||
```python
|
||||
@dataclass(slots=True)
|
||||
class GatewayRecallOutcome:
|
||||
reference_messages: list[dict[str, str]] = field(default_factory=list)
|
||||
result_count: int = 0
|
||||
error: MemoryGatewayClientError | None = None
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run service tests and verify GREEN**
|
||||
|
||||
Run: `uv run pytest -q tests/unit/test_memory_gateway_service.py`
|
||||
|
||||
Expected: all tests pass.
|
||||
|
||||
- [ ] **Step 6: Commit client and service**
|
||||
|
||||
```bash
|
||||
git add app-instance/backend/beaver/integrations/memory_gateway app-instance/backend/beaver/services app-instance/backend/tests/unit/test_memory_gateway_service.py
|
||||
git commit -m "feat(memory): add memory gateway client and service"
|
||||
```
|
||||
|
||||
### Task 3: Extend context assembly for ephemeral Gateway recall
|
||||
|
||||
**Files:**
|
||||
- Modify: `app-instance/backend/beaver/engine/context/builder.py`
|
||||
- Modify: `app-instance/backend/tests/unit/test_context_builder.py`
|
||||
|
||||
- [ ] **Step 1: Write failing context ordering tests**
|
||||
|
||||
Verify reference messages appear after activated skill messages and before persisted history/current user input, while recalled text is absent from the system prompt.
|
||||
|
||||
```python
|
||||
def test_context_builder_places_reference_messages_before_history():
|
||||
result = ContextBuilder().build_messages(ContextBuildInput(
|
||||
reference_messages=[{"role": "user", "content": "[MEMORY REFERENCE] old fact"}],
|
||||
history=[{"role": "assistant", "content": "prior reply"}],
|
||||
current_user_input="new question",
|
||||
))
|
||||
assert result.messages[-3:] == [
|
||||
{"role": "user", "content": "[MEMORY REFERENCE] old fact"},
|
||||
{"role": "assistant", "content": "prior reply"},
|
||||
{"role": "user", "content": "new question"},
|
||||
]
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run context tests and verify RED**
|
||||
|
||||
Run: `uv run pytest -q tests/unit/test_context_builder.py`
|
||||
|
||||
Expected: `ContextBuildInput` rejects `reference_messages`.
|
||||
|
||||
- [ ] **Step 3: Implement reference message support**
|
||||
|
||||
Add `reference_messages` to `ContextBuildInput` and append normalized non-system messages immediately after skill activation messages.
|
||||
|
||||
- [ ] **Step 4: Run context tests and verify GREEN**
|
||||
|
||||
Run: `uv run pytest -q tests/unit/test_context_builder.py`
|
||||
|
||||
Expected: all tests pass.
|
||||
|
||||
- [ ] **Step 5: Commit context support**
|
||||
|
||||
```bash
|
||||
git add app-instance/backend/beaver/engine/context/builder.py app-instance/backend/tests/unit/test_context_builder.py
|
||||
git commit -m "feat(memory): support ephemeral gateway recall context"
|
||||
```
|
||||
|
||||
### Task 4: Wire the optional Gateway service into EngineLoader
|
||||
|
||||
**Files:**
|
||||
- Modify: `app-instance/backend/beaver/engine/loader.py`
|
||||
- Modify: `app-instance/backend/tests/unit/test_imports.py`
|
||||
- Create: `app-instance/backend/tests/unit/test_memory_gateway_loader.py`
|
||||
|
||||
- [ ] **Step 1: Write failing loader tests**
|
||||
|
||||
Cover explicit curated, explicit valid hybrid, implicit hybrid degradation with a sanitized warning, and explicit invalid hybrid rejection. Assert curated store and `memory` tool are present in every successful mode.
|
||||
|
||||
- [ ] **Step 2: Run loader tests and verify RED**
|
||||
|
||||
Run: `uv run pytest -q tests/unit/test_imports.py tests/unit/test_memory_gateway_loader.py`
|
||||
|
||||
Expected: failures because `EngineLoadResult.memory_gateway_service` does not exist.
|
||||
|
||||
- [ ] **Step 3: Implement loader wiring**
|
||||
|
||||
Add optional dependency injection and result fields for `MemoryGatewayService`. Always initialize curated memory and register `MemoryTool`; initialize Gateway only for valid hybrid configuration. Log one warning when implicit hybrid lacks credentials.
|
||||
|
||||
```python
|
||||
memory_gateway_service = self._memory_gateway_service
|
||||
if memory_gateway_service is None and config.memory.mode == "hybrid":
|
||||
if config.memory.gateway.is_configured:
|
||||
memory_gateway_service = MemoryGatewayService(config.memory.gateway)
|
||||
elif not config.memory.explicit:
|
||||
logger.warning("Memory Gateway is not configured; continuing with curated memory only")
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run loader tests and verify GREEN**
|
||||
|
||||
Run: `uv run pytest -q tests/unit/test_imports.py tests/unit/test_memory_gateway_loader.py`
|
||||
|
||||
Expected: all tests pass.
|
||||
|
||||
- [ ] **Step 5: Commit loader wiring**
|
||||
|
||||
```bash
|
||||
git add app-instance/backend/beaver/engine/loader.py app-instance/backend/tests/unit/test_imports.py app-instance/backend/tests/unit/test_memory_gateway_loader.py
|
||||
git commit -m "feat(memory): initialize optional gateway layer"
|
||||
```
|
||||
|
||||
### Task 5: Integrate Gateway recall, persistence, and audit events into AgentLoop
|
||||
|
||||
**Files:**
|
||||
- Modify: `app-instance/backend/beaver/engine/loop.py`
|
||||
- Create: `app-instance/backend/tests/unit/test_memory_gateway_agent_loop.py`
|
||||
|
||||
- [ ] **Step 1: Write failing successful-flow AgentLoop test**
|
||||
|
||||
Use a fake provider and injected fake Gateway service. Verify curated snapshot remains in the system prompt, Gateway recall is outside it and before the current user prompt, and add/flush persistence receives only the original user and final assistant text.
|
||||
|
||||
- [ ] **Step 2: Run the successful-flow test and verify RED**
|
||||
|
||||
Run: `uv run pytest -q tests/unit/test_memory_gateway_agent_loop.py::test_hybrid_run_keeps_curated_memory_and_persists_gateway_turn`
|
||||
|
||||
Expected: failure because `AgentLoop` does not call the Gateway service.
|
||||
|
||||
- [ ] **Step 3: Implement pre-run recall and success audit**
|
||||
|
||||
When `loaded.memory_gateway_service` exists, call recall before context assembly, append hidden success/failure events, pass returned reference messages into `ContextBuildInput`, and add the stable untrusted-reference rule through `extra_sections`.
|
||||
|
||||
- [ ] **Step 4: Implement post-run persistence and audit**
|
||||
|
||||
Capture positive millisecond timestamps, call `persist_after_run` after final text is known and before returning, and append hidden add/flush success/failure events. Do not invoke persistence in the exception path.
|
||||
|
||||
- [ ] **Step 5: Add failing failure-path tests**
|
||||
|
||||
Cover recall failure, add failure, and flush failure. Assert the returned `AgentRunResult` is unchanged, curated snapshot remains present, add failure skips flush, and audit payloads contain no configured key.
|
||||
|
||||
- [ ] **Step 6: Run AgentLoop tests and verify GREEN**
|
||||
|
||||
Run: `uv run pytest -q tests/unit/test_memory_gateway_agent_loop.py tests/unit/test_agent_loop.py tests/unit/test_agent_team_v1.py`
|
||||
|
||||
Expected: all tests pass.
|
||||
|
||||
- [ ] **Step 7: Commit AgentLoop integration**
|
||||
|
||||
```bash
|
||||
git add app-instance/backend/beaver/engine/loop.py app-instance/backend/tests/unit/test_memory_gateway_agent_loop.py
|
||||
git commit -m "feat(memory): add hybrid gateway runtime flow"
|
||||
```
|
||||
|
||||
### Task 6: Document configuration and run full verification
|
||||
|
||||
**Files:**
|
||||
- Modify: `app-instance/backend/README.md`
|
||||
- Modify: `app-instance/backend/env_template` if it contains runtime config guidance
|
||||
|
||||
- [ ] **Step 1: Update backend documentation**
|
||||
|
||||
Document implicit hybrid mode, explicit curated mode, full hybrid JSON configuration, degradation/validation behavior, restart requirement, and the secrecy of `userKey`.
|
||||
|
||||
- [ ] **Step 2: Run targeted tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
uv run pytest -q \
|
||||
tests/unit/test_config_loader.py \
|
||||
tests/unit/test_memory_gateway_service.py \
|
||||
tests/unit/test_context_builder.py \
|
||||
tests/unit/test_memory_gateway_loader.py \
|
||||
tests/unit/test_memory_gateway_agent_loop.py \
|
||||
tests/unit/test_imports.py \
|
||||
tests/unit/test_agent_loop.py
|
||||
```
|
||||
|
||||
Expected: all targeted tests pass.
|
||||
|
||||
- [ ] **Step 3: Run the backend unit suite**
|
||||
|
||||
Run: `uv run pytest -q tests/unit`
|
||||
|
||||
Expected: all unit tests pass.
|
||||
|
||||
- [ ] **Step 4: Compile changed Python packages**
|
||||
|
||||
Run: `uv run python -m compileall -q beaver tests/unit`
|
||||
|
||||
Expected: exit code 0 with no output.
|
||||
|
||||
- [ ] **Step 5: Review secret handling and diff**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
git diff --check
|
||||
rg -n "userKey|user_key" app-instance/backend/beaver app-instance/backend/tests/unit/test_memory_gateway* app-instance/backend/README.md
|
||||
git status --short
|
||||
```
|
||||
|
||||
Expected: credentials appear only as field names or test fixtures; no real key is logged or committed.
|
||||
|
||||
- [ ] **Step 6: Commit documentation and verification adjustments**
|
||||
|
||||
```bash
|
||||
git add app-instance/backend/README.md app-instance/backend/env_template
|
||||
git commit -m "docs(memory): document hybrid gateway configuration"
|
||||
```
|
||||
@ -0,0 +1,265 @@
|
||||
# Memory Gateway User Provisioning Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Move Beaver's Gateway code into `beaver.memory.gateway`, load one shared non-secret Gateway configuration, provision Gateway users during Beaver registration, and resolve per-user credentials for each authenticated chat run.
|
||||
|
||||
**Architecture:** `EngineLoader` loads curated memory, a shared Gateway config, and an instance-local credential store. Registration calls Gateway `/users` and atomically stores credentials by Beaver username. REST/WebSocket chat derive a trusted username from the access token and `AgentLoop` creates a run-local Gateway service for that user, leaving unauthenticated or unprovisioned runs on curated memory only.
|
||||
|
||||
**Tech Stack:** Python 3.14, dataclasses, FastAPI, httpx, pytest, shell-based Docker instance creation.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Move Gateway source and load shared configuration
|
||||
|
||||
**Files:**
|
||||
- Create: `app-instance/backend/beaver/memory/gateway/__init__.py`
|
||||
- Create: `app-instance/backend/beaver/memory/gateway/config.py`
|
||||
- Create: `app-instance/backend/beaver/memory/gateway/client.py`
|
||||
- Create: `app-instance/backend/beaver/memory/gateway/service.py`
|
||||
- Create: `app-instance/backend/memory/config.json`
|
||||
- Modify: `app-instance/backend/beaver/foundation/config/schema.py`
|
||||
- Modify: `app-instance/backend/beaver/foundation/config/loader.py`
|
||||
- Modify: `app-instance/backend/beaver/foundation/config/__init__.py`
|
||||
- Delete: `app-instance/backend/beaver/integrations/memory_gateway/`
|
||||
- Delete: `app-instance/backend/beaver/services/memory_gateway_service.py`
|
||||
- Modify: `app-instance/backend/beaver/services/__init__.py`
|
||||
- Test: `app-instance/backend/tests/unit/test_config_loader.py`
|
||||
- Test: `app-instance/backend/tests/unit/test_memory_gateway_service.py`
|
||||
|
||||
- [ ] **Step 1: Write failing shared-config and import tests**
|
||||
|
||||
Set `BEAVER_MEMORY_CONFIG_PATH` to a temporary JSON file and assert `load_config()` obtains `memory.mode`, URL, and all three scopes from that file. Change all Gateway tests to import from `beaver.memory.gateway`.
|
||||
|
||||
- [ ] **Step 2: Run tests and verify RED**
|
||||
|
||||
```bash
|
||||
cd app-instance/backend
|
||||
.venv/bin/pytest -q tests/unit/test_config_loader.py tests/unit/test_memory_gateway_service.py
|
||||
```
|
||||
|
||||
Expected: failures because the new package and shared config loading do not exist.
|
||||
|
||||
- [ ] **Step 3: Implement package migration and shared config parsing**
|
||||
|
||||
Move existing client/service behavior without changing payloads. Define `MemoryConfig` and `MemoryGatewayConfig` in `beaver.memory.gateway.config`, without `userId/userKey`. Add `default_memory_config_path()` using `BEAVER_MEMORY_CONFIG_PATH` then `<backend-root>/memory/config.json`. Instance config parsing remains responsible for non-memory settings; shared config supplies `BeaverConfig.memory`.
|
||||
|
||||
Create tracked `memory/config.json` with `http://172.19.207.37:8010`, scopes `current_chat`, `resources`, `all_user_memory`, top K 8, and timeout 10.
|
||||
|
||||
- [ ] **Step 4: Run targeted tests and verify GREEN**
|
||||
|
||||
Run the command from Step 2. Expected: selected tests pass.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add app-instance/backend/beaver/memory/gateway app-instance/backend/memory/config.json app-instance/backend/beaver/foundation/config app-instance/backend/beaver/services app-instance/backend/tests/unit/test_config_loader.py app-instance/backend/tests/unit/test_memory_gateway_service.py
|
||||
git commit -m "refactor(memory): move gateway into memory domain"
|
||||
```
|
||||
|
||||
### Task 2: Add per-instance Gateway credential storage
|
||||
|
||||
**Files:**
|
||||
- Create: `app-instance/backend/beaver/memory/gateway/credentials.py`
|
||||
- Modify: `app-instance/backend/beaver/memory/gateway/__init__.py`
|
||||
- Create: `app-instance/backend/tests/unit/test_memory_gateway_credentials.py`
|
||||
- Modify: `app-instance/create-instance.sh`
|
||||
- Modify: `app-instance/entrypoint.sh`
|
||||
- Modify: `app-instance/README.md`
|
||||
|
||||
- [ ] **Step 1: Write failing credential-store tests**
|
||||
|
||||
Cover missing files, multi-user round trips, updates preserving other users, secret-free repr, atomic replace, and mode `0600`.
|
||||
|
||||
- [ ] **Step 2: Run test and verify RED**
|
||||
|
||||
```bash
|
||||
cd app-instance/backend
|
||||
.venv/bin/pytest -q tests/unit/test_memory_gateway_credentials.py
|
||||
```
|
||||
|
||||
Expected: import failure because the credential store does not exist.
|
||||
|
||||
- [ ] **Step 3: Implement atomic credential persistence**
|
||||
|
||||
Implement `MemoryGatewayUserCredential(user_id, user_key)` and `MemoryGatewayCredentialStore.get/save`. Use JSON shape `{"users": {username: {"userId": ..., "userKey": ...}}}`, sibling temporary file, `os.replace`, and `chmod(0o600)`.
|
||||
|
||||
Update `create-instance.sh` to create `$BEAVER_HOME/memory_gateway_users.json` as `{"users": {}}`, chmod it `0600`, and pass `BEAVER_MEMORY_GATEWAY_USERS_PATH=/root/.beaver/memory_gateway_users.json`. `entrypoint.sh` exports the same default.
|
||||
|
||||
- [ ] **Step 4: Run credential and shell syntax tests**
|
||||
|
||||
```bash
|
||||
cd app-instance/backend
|
||||
.venv/bin/pytest -q tests/unit/test_memory_gateway_credentials.py
|
||||
cd ../..
|
||||
bash -n app-instance/create-instance.sh app-instance/entrypoint.sh
|
||||
```
|
||||
|
||||
Expected: tests pass and shell syntax exits zero.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add app-instance/backend/beaver/memory/gateway app-instance/backend/tests/unit/test_memory_gateway_credentials.py app-instance/create-instance.sh app-instance/entrypoint.sh app-instance/README.md
|
||||
git commit -m "feat(memory): persist gateway user credentials"
|
||||
```
|
||||
|
||||
### Task 3: Provision Gateway identities during frontend registration
|
||||
|
||||
**Files:**
|
||||
- Modify: `app-instance/backend/beaver/memory/gateway/client.py`
|
||||
- Modify: `app-instance/backend/beaver/interfaces/web/app.py`
|
||||
- Create: `app-instance/backend/tests/unit/test_memory_gateway_registration.py`
|
||||
|
||||
- [ ] **Step 1: Write failing registration tests**
|
||||
|
||||
Use a temporary auth file and fake Gateway client. Assert registration sends `{"user_id": "tom"}`, stores the returned key, never returns the key to the browser, and still registers the Beaver user without a partial credential when Gateway provisioning fails.
|
||||
|
||||
- [ ] **Step 2: Run tests and verify RED**
|
||||
|
||||
```bash
|
||||
cd app-instance/backend
|
||||
.venv/bin/pytest -q tests/unit/test_memory_gateway_registration.py
|
||||
```
|
||||
|
||||
Expected: failures because `/users` provisioning is not connected.
|
||||
|
||||
- [ ] **Step 3: Implement provisioning**
|
||||
|
||||
Add `MemoryGatewayClient.create_user(user_id)`, validating non-empty response `user_id/user_key`. During `/api/auth/register`, after local/AuthZ registration succeeds, call it with the Beaver username and save through the credential store. Catch sanitized Gateway failures without retrying or rolling back Beaver registration. Never include the Gateway credential in the response.
|
||||
|
||||
- [ ] **Step 4: Run registration tests and verify GREEN**
|
||||
|
||||
Run the command from Step 2. Expected: all registration tests pass.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add app-instance/backend/beaver/memory/gateway/client.py app-instance/backend/beaver/interfaces/web/app.py app-instance/backend/tests/unit/test_memory_gateway_registration.py
|
||||
git commit -m "feat(memory): provision gateway users on registration"
|
||||
```
|
||||
|
||||
### Task 4: Pass trusted authenticated identity into chat runs
|
||||
|
||||
**Files:**
|
||||
- Modify: `app-instance/backend/beaver/interfaces/web/app.py`
|
||||
- Modify: `app-instance/backend/beaver/engine/loop.py`
|
||||
- Modify: `app-instance/backend/beaver/services/agent_service.py`
|
||||
- Modify: `app-instance/backend/tests/unit/test_websocket_chat.py`
|
||||
|
||||
- [ ] **Step 1: Write failing REST/WebSocket identity tests**
|
||||
|
||||
Issue a web token for `tom`. Assert REST and WebSocket calls pass `gateway_user_id="tom"`. Send a conflicting client `user_id="other"` and assert the trusted identity remains `tom`. Unauthenticated calls pass `gateway_user_id=None`.
|
||||
|
||||
- [ ] **Step 2: Run tests and verify RED**
|
||||
|
||||
```bash
|
||||
cd app-instance/backend
|
||||
.venv/bin/pytest -q tests/unit/test_websocket_chat.py
|
||||
```
|
||||
|
||||
Expected: identity assertions fail because chat does not pass a trusted Gateway principal.
|
||||
|
||||
- [ ] **Step 3: Implement optional trusted identity resolution**
|
||||
|
||||
Add `gateway_user_id: str | None` to AgentLoop direct-run kwargs. REST reads the optional bearer token from `Authorization`; WebSocket reads the existing `?token=` parameter. Both resolve only through `app.state.auth_tokens`. Request `user_id` remains session metadata and never selects Gateway credentials.
|
||||
|
||||
- [ ] **Step 4: Run identity tests and verify GREEN**
|
||||
|
||||
Run the command from Step 2. Expected: tests pass.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add app-instance/backend/beaver/interfaces/web/app.py app-instance/backend/beaver/engine/loop.py app-instance/backend/beaver/services/agent_service.py app-instance/backend/tests/unit/test_websocket_chat.py
|
||||
git commit -m "feat(memory): bind gateway runs to authenticated users"
|
||||
```
|
||||
|
||||
### Task 5: Resolve a run-local Gateway service per user
|
||||
|
||||
**Files:**
|
||||
- Modify: `app-instance/backend/beaver/engine/loader.py`
|
||||
- Modify: `app-instance/backend/beaver/engine/loop.py`
|
||||
- Modify: `app-instance/backend/beaver/memory/gateway/service.py`
|
||||
- Modify: `app-instance/backend/tests/unit/test_memory_gateway_loader.py`
|
||||
- Modify: `app-instance/backend/tests/unit/test_memory_gateway_agent_loop.py`
|
||||
|
||||
- [ ] **Step 1: Write failing loader and AgentLoop tests**
|
||||
|
||||
Assert loader exposes shared config, credential store, and service factory instead of a fixed-user service. Add two users with different keys and verify each run constructs a service from only the selected credential. Missing identity or credential performs no Gateway calls while curated memory remains present.
|
||||
|
||||
- [ ] **Step 2: Run tests and verify RED**
|
||||
|
||||
```bash
|
||||
cd app-instance/backend
|
||||
.venv/bin/pytest -q tests/unit/test_memory_gateway_loader.py tests/unit/test_memory_gateway_agent_loop.py
|
||||
```
|
||||
|
||||
Expected: failures because loader still creates one fixed-user service.
|
||||
|
||||
- [ ] **Step 3: Implement per-run service resolution**
|
||||
|
||||
Expose `memory_gateway_config`, `memory_gateway_credentials`, and a service factory on `EngineLoadResult`. At run start, resolve the credential by `gateway_user_id`; construct a fresh service only in hybrid mode when a credential exists. Pass shared config and credential separately to the service and preserve current recall/add/flush/audit behavior.
|
||||
|
||||
- [ ] **Step 4: Run Gateway runtime tests and verify GREEN**
|
||||
|
||||
```bash
|
||||
cd app-instance/backend
|
||||
.venv/bin/pytest -q tests/unit/test_memory_gateway_loader.py tests/unit/test_memory_gateway_agent_loop.py tests/unit/test_memory_gateway_service.py tests/unit/test_context_builder.py
|
||||
```
|
||||
|
||||
Expected: all selected tests pass.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add app-instance/backend/beaver/engine app-instance/backend/beaver/memory/gateway app-instance/backend/tests/unit/test_memory_gateway_loader.py app-instance/backend/tests/unit/test_memory_gateway_agent_loop.py
|
||||
git commit -m "feat(memory): resolve gateway service per user"
|
||||
```
|
||||
|
||||
### Task 6: Update documentation and perform final verification
|
||||
|
||||
**Files:**
|
||||
- Modify: `app-instance/backend/README.md`
|
||||
- Modify: `app-instance/README.md`
|
||||
- Modify: `docs/superpowers/plans/2026-06-15-hybrid-memory-gateway.md`
|
||||
|
||||
- [ ] **Step 1: Update operational documentation**
|
||||
|
||||
Document the shared config path, instance credential path, registration provisioning, token-based identity, secret handling, and rebuild/restart requirements. Remove examples that place `userId/userKey` in instance `config.json`.
|
||||
|
||||
- [ ] **Step 2: Verify removed source imports**
|
||||
|
||||
```bash
|
||||
rg -n "beaver\.integrations\.memory_gateway|beaver\.services\.memory_gateway_service" app-instance/backend/beaver app-instance/backend/tests
|
||||
```
|
||||
|
||||
Expected: no matches.
|
||||
|
||||
- [ ] **Step 3: Run full verification**
|
||||
|
||||
```bash
|
||||
cd app-instance/backend
|
||||
.venv/bin/python -m compileall -q beaver
|
||||
.venv/bin/pytest -q
|
||||
cd ../..
|
||||
bash -n app-instance/create-instance.sh app-instance/entrypoint.sh
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: compile and shell checks exit zero, all tests pass, and diff check is clean.
|
||||
|
||||
- [ ] **Step 4: Scan tracked content for credentials**
|
||||
|
||||
```bash
|
||||
git grep -nE 'uk_[A-Za-z0-9]{8,}' -- ':!docs/superpowers/specs/*' ':!docs/superpowers/plans/*'
|
||||
```
|
||||
|
||||
Expected: no real Gateway key in tracked source or runtime files; obvious test placeholders are reviewed manually.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add app-instance/backend/README.md app-instance/README.md docs/superpowers/plans/2026-06-15-hybrid-memory-gateway.md
|
||||
git commit -m "docs(memory): document gateway user provisioning"
|
||||
```
|
||||
@ -0,0 +1,351 @@
|
||||
# Hybrid Memory Gateway Integration Design
|
||||
|
||||
## Goal
|
||||
|
||||
Keep Beaver's existing curated memory as the permanent baseline and optionally
|
||||
add Memory Gateway as an independent second memory layer.
|
||||
|
||||
- Curated memory continues to load `MEMORY.md` and `USER.md` into a frozen
|
||||
per-run snapshot and continues to expose the existing `memory` tool.
|
||||
- Memory Gateway independently recalls conversation/resource memory through
|
||||
`POST /memories/search` and persists each completed conversation turn through
|
||||
one `POST /memories/add` followed by one `POST /memories/flush`.
|
||||
- The two layers do not synchronize, overwrite, merge, deduplicate, or resolve
|
||||
conflicts with each other.
|
||||
|
||||
Memory Gateway is best-effort. Gateway failures must be auditable without
|
||||
affecting curated memory or turning an otherwise successful chat run into a
|
||||
failure.
|
||||
|
||||
## Scope
|
||||
|
||||
This change includes:
|
||||
|
||||
- Runtime configuration for `curated` and `hybrid` modes.
|
||||
- Fixed Memory Gateway credentials and search scopes in instance config.
|
||||
- An asynchronous Memory Gateway HTTP client.
|
||||
- An optional `MemoryGatewayService` alongside the existing `MemoryService`.
|
||||
- Gateway recall before each provider run in hybrid mode.
|
||||
- Gateway add and flush after each normally completed run in hybrid mode.
|
||||
- Hidden session audit events for Gateway outcomes.
|
||||
- Unit and integration-style tests using fake transports and providers.
|
||||
|
||||
This change does not include:
|
||||
|
||||
- Replacing or disabling curated memory.
|
||||
- Synchronizing curated `memory` tool writes to Memory Gateway.
|
||||
- Writing Gateway conversation turns into `MEMORY.md` or `USER.md`.
|
||||
- Conflict resolution or automatic deduplication across the two layers.
|
||||
- Automatic `POST /users` calls or credential provisioning.
|
||||
- A memory settings UI or memory administration UI.
|
||||
- Resource upload support from Beaver.
|
||||
- Gateway override or deletion APIs.
|
||||
- Persisting tool calls, tool results, system events, reasoning, recalled
|
||||
memory, or skill activation messages to Gateway.
|
||||
|
||||
## Configuration
|
||||
|
||||
Beaver adds a top-level `memory` section:
|
||||
|
||||
```json
|
||||
{
|
||||
"memory": {
|
||||
"mode": "hybrid",
|
||||
"gateway": {
|
||||
"baseUrl": "http://127.0.0.1:8010",
|
||||
"userId": "gateway_test_user",
|
||||
"userKey": "uk_xxx",
|
||||
"appId": "default",
|
||||
"projectId": "default",
|
||||
"scope": ["current_chat", "resources"],
|
||||
"topK": 8,
|
||||
"timeoutSeconds": 10
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Configuration rules:
|
||||
|
||||
- Valid modes are `curated` and `hybrid`.
|
||||
- Curated memory is initialized and enabled in both modes.
|
||||
- If the entire `memory` section is absent, the effective mode is implicitly
|
||||
`hybrid`. Missing Gateway credentials in this implicit-default case produce
|
||||
a startup warning and degrade only the Gateway layer; Beaver continues with
|
||||
curated memory.
|
||||
- If `mode: "hybrid"` is explicitly present, non-empty `baseUrl`, `userId`, and
|
||||
`userKey` are required. Missing required values fail runtime loading.
|
||||
- `mode: "curated"` disables Gateway initialization and ignores an optional
|
||||
Gateway block.
|
||||
- `appId` and `projectId` default to `default`.
|
||||
- `scope` must be a non-empty subset of `current_chat`, `resources`, and
|
||||
`all_user_memory`. The initial integration uses `current_chat` and
|
||||
`resources`.
|
||||
- `topK` defaults to 8 and must be between 1 and 100.
|
||||
- `timeoutSeconds` defaults to 10 and must be positive.
|
||||
- `userKey` must never appear in status payloads, warnings, logs produced by
|
||||
this integration, session events, or raised configuration/client errors.
|
||||
|
||||
The parsed configuration must retain whether hybrid mode was explicit or
|
||||
implicit so runtime loading can apply the different validation behavior.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing curated memory remains unchanged
|
||||
|
||||
`MemoryStore`, `MemorySnapshot`, `MemoryService`, and `MemoryTool` retain their
|
||||
current responsibilities:
|
||||
|
||||
- `EngineLoader` always initializes `MemoryService`.
|
||||
- `AgentLoop` always captures a per-run frozen curated snapshot.
|
||||
- `ContextBuilder` always receives that snapshot for system-prompt injection.
|
||||
- The original `memory` tool remains registered and always operates only on
|
||||
`MEMORY.md` and `USER.md`.
|
||||
- Gateway availability and Gateway failures do not change curated behavior.
|
||||
|
||||
### Optional Gateway service
|
||||
|
||||
Add a separate `MemoryGatewayService` rather than a mutually exclusive backend
|
||||
strategy. It is present only when hybrid mode has a valid Gateway configuration.
|
||||
|
||||
The service exposes two runtime operations:
|
||||
|
||||
1. `recall_before_run`: search Gateway using the current Beaver session and
|
||||
user prompt, then return sanitized reference messages plus audit metadata.
|
||||
2. `persist_after_run`: add the current user message and final assistant answer,
|
||||
then flush the Gateway chat session.
|
||||
|
||||
`EngineLoadResult` exposes `memory_gateway_service: MemoryGatewayService | None`.
|
||||
`AgentLoop` uses it conditionally while continuing its existing curated path
|
||||
unconditionally.
|
||||
|
||||
`session_search` remains independent and available in both modes.
|
||||
|
||||
### Memory Gateway HTTP client
|
||||
|
||||
The HTTP client owns transport and response validation for:
|
||||
|
||||
- `POST {baseUrl}/memories/search`
|
||||
- `POST {baseUrl}/memories/add`
|
||||
- `POST {baseUrl}/memories/flush`
|
||||
|
||||
It uses an asynchronous HTTP client, the configured timeout, JSON request
|
||||
bodies, and sanitized typed exceptions containing operation/path/status
|
||||
metadata without credentials or complete request bodies.
|
||||
|
||||
Beaver adds no automatic retries in this first integration. Gateway already
|
||||
retries upstream ingestion, and retrying add from Beaver could duplicate a
|
||||
turn when the first request succeeded but its response was lost.
|
||||
|
||||
## Recall Data Flow
|
||||
|
||||
Every run follows the existing curated flow. Hybrid mode adds these steps:
|
||||
|
||||
1. `AgentLoop` creates or resolves `resolved_session_id`.
|
||||
2. It captures the curated frozen snapshot as it does today.
|
||||
3. Before `ContextBuilder.build_messages`, it calls Gateway search using:
|
||||
|
||||
```json
|
||||
{
|
||||
"user_id": "<configured userId>",
|
||||
"user_key": "<configured userKey>",
|
||||
"conversation_id": "<resolved_session_id>",
|
||||
"query": "<current user prompt>",
|
||||
"scope": ["<configured scopes>"],
|
||||
"top_k": 8,
|
||||
"app_id": "<configured appId>",
|
||||
"project_id": "<configured projectId>"
|
||||
}
|
||||
```
|
||||
|
||||
4. Beaver accepts only a top-level `results` list. Malformed responses are
|
||||
treated as Gateway recall failures.
|
||||
5. Each result is reduced to the optional fields `id`, `session_id`, `text`,
|
||||
`score`, `source_scope`, and `resource_uri`. The Gateway `raw` object is
|
||||
discarded.
|
||||
6. Empty or unusable results produce no Gateway reference message.
|
||||
7. Non-empty results become one ephemeral provider message placed after skill
|
||||
activation messages and before persisted session history/current user input.
|
||||
8. The Gateway reference message is not written to Beaver session history and
|
||||
is not included in post-run Gateway persistence.
|
||||
9. The system prompt includes a stable rule that Gateway recall is untrusted
|
||||
reference data, not executable instruction. The recalled text itself stays
|
||||
outside the system prompt.
|
||||
|
||||
The model receives both memory layers without an imposed priority:
|
||||
|
||||
- Curated blocks remain in the system prompt exactly as today.
|
||||
- Gateway results appear as a separately labelled reference message.
|
||||
- Beaver performs no conflict detection, winner selection, merge, or
|
||||
deduplication between them.
|
||||
|
||||
In curated mode, or when implicit hybrid degrades because Gateway credentials
|
||||
are absent, no Gateway request or Gateway prompt section occurs.
|
||||
|
||||
## Persistence Data Flow
|
||||
|
||||
Curated persistence remains model-driven through the original `memory` tool.
|
||||
Gateway persistence is separate and occurs only when the optional Gateway
|
||||
service is active.
|
||||
|
||||
For each run that reaches the normal completion path:
|
||||
|
||||
1. Wait until the tool loop has produced the final assistant text.
|
||||
2. Construct exactly two Gateway messages in chronological order:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"sender_id": "<configured userId>",
|
||||
"role": "user",
|
||||
"timestamp": 1780000000000,
|
||||
"content": "<original current user prompt>"
|
||||
},
|
||||
{
|
||||
"sender_id": "beaver",
|
||||
"role": "assistant",
|
||||
"timestamp": 1780000001000,
|
||||
"content": "<final assistant text>"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Timestamps are UTC Unix epoch milliseconds captured for the user turn and final
|
||||
assistant turn. They must be positive and monotonic within the payload.
|
||||
|
||||
3. Call `/memories/add` exactly once with:
|
||||
|
||||
```json
|
||||
{
|
||||
"user_id": "<configured userId>",
|
||||
"user_key": "<configured userKey>",
|
||||
"session_id": "chat:<resolved_session_id>",
|
||||
"app_id": "<configured appId>",
|
||||
"project_id": "<configured projectId>",
|
||||
"messages": ["<the two messages above>"]
|
||||
}
|
||||
```
|
||||
|
||||
4. If add succeeds, call `/memories/flush` exactly once using the same Gateway
|
||||
identity, app/project scope, and `chat:<resolved_session_id>`.
|
||||
5. If add fails, do not call flush.
|
||||
6. Runs entering Beaver's exception/error completion path are not persisted.
|
||||
Normal completion outputs such as a tool-limit fallback are persisted because
|
||||
they are returned to the user.
|
||||
7. Tool calls, tool results, hidden events, system prompts, curated snapshot
|
||||
text, Gateway recalled text, reasoning, and activated skill text are never
|
||||
included in the Gateway add payload.
|
||||
8. Gateway persistence never modifies `MEMORY.md` or `USER.md`.
|
||||
9. Curated `memory` tool add/replace/remove operations never call Gateway.
|
||||
|
||||
## Session Audit Events
|
||||
|
||||
When the Gateway service is active, Beaver writes hidden
|
||||
(`context_visible=false`) session events without credentials or full response
|
||||
bodies:
|
||||
|
||||
- `memory_gateway_recall_succeeded`: configured scopes and result count.
|
||||
- `memory_gateway_recall_failed`: operation, sanitized error category, and
|
||||
optional HTTP status.
|
||||
- `memory_gateway_add_succeeded`: Gateway chat session and message count.
|
||||
- `memory_gateway_add_failed`: sanitized failure metadata.
|
||||
- `memory_gateway_flush_succeeded`: Gateway chat session.
|
||||
- `memory_gateway_flush_failed`: sanitized failure metadata and indication that
|
||||
add already succeeded.
|
||||
|
||||
For implicit hybrid degradation at runtime boot, use a normal application
|
||||
warning rather than a session event because no session exists yet. The warning
|
||||
must not contain credential values.
|
||||
|
||||
## Failure Semantics
|
||||
|
||||
- Curated initialization or writes retain their existing behavior and are not
|
||||
caught or changed by Gateway code.
|
||||
- Missing Gateway credentials in implicit-default hybrid mode: warn, leave the
|
||||
Gateway service unset, and continue with curated memory.
|
||||
- Missing/invalid Gateway configuration in explicit hybrid mode: fail runtime
|
||||
loading with a sanitized configuration error.
|
||||
- Search timeout, connection failure, 401, other HTTP error, or malformed JSON:
|
||||
record recall failure and continue with curated memory and normal context.
|
||||
- Add failure: record add failure, skip flush, and return the normal assistant
|
||||
result.
|
||||
- Flush failure: record flush failure and return the normal assistant result.
|
||||
- Gateway failures do not disable, roll back, or mutate curated memory.
|
||||
- Gateway failures are not surfaced as user-facing chat errors in this phase.
|
||||
|
||||
## Security and Privacy
|
||||
|
||||
- Fixed Gateway credentials come only from Beaver instance configuration.
|
||||
- `userKey` is passed only in Gateway request bodies and retained in memory by
|
||||
the typed config/client objects.
|
||||
- Client exceptions, startup warnings, and audit payloads never serialize
|
||||
request bodies or credentials.
|
||||
- Gateway conversation/resource text is treated as untrusted data.
|
||||
- Gateway `raw` fields are discarded before prompt construction.
|
||||
- Curated and Gateway stores remain isolated. No content is copied between
|
||||
them: curated receives only explicit `memory` tool mutations, while Gateway
|
||||
receives only the configured per-run conversation payload.
|
||||
|
||||
## Testing
|
||||
|
||||
### Configuration tests
|
||||
|
||||
- Missing memory configuration produces implicit hybrid mode.
|
||||
- Implicit hybrid without credentials leaves Gateway disabled and curated
|
||||
enabled, with one sanitized warning.
|
||||
- Explicit curated mode does not require or initialize Gateway.
|
||||
- Complete explicit hybrid config parses camelCase fields and initializes both
|
||||
memory layers.
|
||||
- Explicit hybrid with missing credentials fails loading.
|
||||
- Invalid mode, empty/unknown scope, invalid `topK`, and non-positive timeout
|
||||
fail with explicit sanitized errors.
|
||||
- No warning or exception text contains `userKey`.
|
||||
|
||||
### HTTP client tests
|
||||
|
||||
- Search, add, and flush use the exact paths and payload shapes above.
|
||||
- Configured timeout is applied.
|
||||
- Non-2xx, network, invalid JSON, and invalid response shapes produce sanitized
|
||||
client exceptions.
|
||||
- Exception strings never contain the configured key.
|
||||
|
||||
### Gateway service tests
|
||||
|
||||
- Search uses configured scopes and strips `raw` fields.
|
||||
- Empty search results produce no reference message.
|
||||
- Persistence sends exactly the original user prompt and final assistant
|
||||
response, then flushes once.
|
||||
- Add failure skips flush; flush failure preserves the successful add outcome.
|
||||
- Service methods never read or write curated files or call `MemoryStore`.
|
||||
|
||||
### Agent loop and loader tests
|
||||
|
||||
- Curated snapshot injection and `memory` tool availability remain present in
|
||||
both curated and hybrid modes.
|
||||
- Hybrid search occurs before the provider call while the curated snapshot is
|
||||
still present in the system prompt.
|
||||
- Gateway recall appears before the current user prompt and outside the system
|
||||
prompt body.
|
||||
- The system prompt contains the untrusted-reference rule only when Gateway is
|
||||
active.
|
||||
- Add and flush happen after the final assistant response and exactly once each.
|
||||
- Tool/system/reasoning/curated/Gateway-recall content is absent from the add
|
||||
payload.
|
||||
- Recall/add/flush failures do not change the returned `AgentRunResult` or the
|
||||
curated snapshot/tool behavior.
|
||||
- Hidden success/failure audit events contain no credentials.
|
||||
- Curated `memory` tool operations produce no Gateway calls.
|
||||
- Gateway persistence produces no changes to `MEMORY.md` or `USER.md`.
|
||||
- Curated mode and degraded implicit hybrid perform no Gateway HTTP calls.
|
||||
|
||||
## Documentation
|
||||
|
||||
Update the backend README/config example with:
|
||||
|
||||
- `hybrid` as the implicit default.
|
||||
- Explicit `curated` mode for disabling Gateway.
|
||||
- A complete explicit hybrid example.
|
||||
- The implicit-default degradation rule and explicit-hybrid validation rule.
|
||||
- A warning that `userKey` is a secret.
|
||||
- A note that changing memory mode/config requires runtime reload or restart
|
||||
because `EngineLoader` constructs the optional Gateway service during boot.
|
||||
@ -0,0 +1,282 @@
|
||||
# Memory Gateway Package and User Provisioning Design
|
||||
|
||||
## Goal
|
||||
|
||||
Reorganize Beaver's Memory Gateway code under the `beaver.memory` domain and
|
||||
replace the single fixed Gateway identity with per-Beaver-user credentials.
|
||||
|
||||
The final model has two independent configuration layers:
|
||||
|
||||
- One shared, non-secret Memory Gateway configuration used by every Beaver
|
||||
instance.
|
||||
- One per-instance credential file containing the Gateway identities created
|
||||
for Beaver frontend users.
|
||||
|
||||
Curated memory remains enabled and isolated. Gateway failures or missing user
|
||||
credentials must not modify `MEMORY.md`, `USER.md`, or the `memory` tool.
|
||||
|
||||
## Source Package
|
||||
|
||||
All Beaver-side Gateway source moves to:
|
||||
|
||||
```text
|
||||
app-instance/backend/beaver/memory/gateway/
|
||||
├── __init__.py
|
||||
├── config.py
|
||||
├── client.py
|
||||
├── credentials.py
|
||||
└── service.py
|
||||
```
|
||||
|
||||
- `config.py` owns the shared typed Gateway configuration.
|
||||
- `client.py` owns `MemoryGatewayClient` and sanitized client exceptions.
|
||||
- `credentials.py` owns typed user credentials and atomic credential-file
|
||||
persistence.
|
||||
- `service.py` owns search/add/flush orchestration and result types.
|
||||
- `__init__.py` exposes the supported public Gateway API.
|
||||
|
||||
Remove the old source locations:
|
||||
|
||||
- `beaver/integrations/memory_gateway/`
|
||||
- `beaver/services/memory_gateway_service.py`
|
||||
- Gateway configuration dataclasses in `beaver.foundation.config.schema`
|
||||
- The lazy `MemoryGatewayService` export from `beaver.services`
|
||||
|
||||
No compatibility forwarding modules are retained. After migration,
|
||||
`beaver.memory.gateway` is the only supported source entry point.
|
||||
|
||||
## Shared Configuration
|
||||
|
||||
All Beaver instances read the same public Gateway configuration from:
|
||||
|
||||
```text
|
||||
/home/tom/beaver_project/app-instance/backend/memory/config.json
|
||||
```
|
||||
|
||||
Inside the app-instance image this is available as:
|
||||
|
||||
```text
|
||||
/opt/app/backend/memory/config.json
|
||||
```
|
||||
|
||||
The file contains no user credentials:
|
||||
|
||||
```json
|
||||
{
|
||||
"memory": {
|
||||
"mode": "hybrid",
|
||||
"gateway": {
|
||||
"baseUrl": "http://172.19.207.37:8010",
|
||||
"appId": "default",
|
||||
"projectId": "default",
|
||||
"scope": ["current_chat", "resources", "all_user_memory"],
|
||||
"topK": 8,
|
||||
"timeoutSeconds": 10
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Valid modes remain `curated` and `hybrid`.
|
||||
- Curated memory is always initialized.
|
||||
- `hybrid` enables Gateway only for runs with a resolved user credential.
|
||||
- `baseUrl` is fixed to `http://172.19.207.37:8010` in the initial shared
|
||||
configuration.
|
||||
- Scope includes `current_chat`, `resources`, and `all_user_memory`.
|
||||
- The shared file is the authoritative Memory Gateway configuration. Instance
|
||||
`config.json` files continue to own providers, tools, channels, AuthZ, and
|
||||
backend identity, but no longer carry Gateway user credentials.
|
||||
- An optional `BEAVER_MEMORY_CONFIG_PATH` may override the shared file path for
|
||||
tests or non-image development runs.
|
||||
|
||||
## Per-Instance User Credentials
|
||||
|
||||
Each Beaver instance stores Gateway user credentials alongside its existing
|
||||
`config.json`, `runtime.env`, and `web_auth_users.json`:
|
||||
|
||||
```text
|
||||
app-instance/runtime/instances/<instance-slug>/beaver-home/
|
||||
├── config.json
|
||||
├── runtime.env
|
||||
├── web_auth_users.json
|
||||
└── memory_gateway_users.json
|
||||
```
|
||||
|
||||
The existing `beaver-home` mount exposes the file inside the container as:
|
||||
|
||||
```text
|
||||
/root/.beaver/memory_gateway_users.json
|
||||
```
|
||||
|
||||
The JSON format is:
|
||||
|
||||
```json
|
||||
{
|
||||
"users": {
|
||||
"tom": {
|
||||
"userId": "tom",
|
||||
"userKey": "uk_xxx"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- The map key is the authenticated Beaver login username.
|
||||
- Gateway `userId` is exactly the Beaver login username, with no prefix.
|
||||
- `userKey` is secret and must never appear in API responses, logs, audit
|
||||
events, exceptions, or tracked configuration.
|
||||
- Writes use a sibling temporary file followed by atomic replace.
|
||||
- The credential file is created with mode `0600`.
|
||||
- `BEAVER_MEMORY_GATEWAY_USERS_PATH` may override the default path for tests.
|
||||
|
||||
## Frontend User Provisioning
|
||||
|
||||
The frontend continues to call Beaver's existing `POST /api/auth/register`
|
||||
endpoint. The browser never calls Memory Gateway directly and never receives
|
||||
the Gateway `userKey`.
|
||||
|
||||
For a registration request with username `tom`, Beaver performs:
|
||||
|
||||
```http
|
||||
POST http://172.19.207.37:8010/users
|
||||
Content-Type: application/json
|
||||
|
||||
{"user_id":"tom"}
|
||||
```
|
||||
|
||||
Beaver validates that the response contains non-empty `user_id` and
|
||||
`user_key`, requires the returned `user_id` to equal `tom`, and stores the
|
||||
credential under the `tom` entry in `memory_gateway_users.json`.
|
||||
|
||||
The Gateway `/users` API is treated as idempotent. Registering an existing
|
||||
Beaver username may refresh the same credential entry without creating a
|
||||
second local identity.
|
||||
|
||||
For this first version:
|
||||
|
||||
- Gateway provisioning has no Beaver-side retries.
|
||||
- A Gateway provisioning failure does not roll back an otherwise valid Beaver
|
||||
registration.
|
||||
- A user without stored Gateway credentials continues with curated memory only.
|
||||
- No separate repair UI or background credential provisioning job is added.
|
||||
|
||||
## Authenticated Chat Identity
|
||||
|
||||
Gateway credential selection must use a trusted server-side principal.
|
||||
|
||||
- REST and WebSocket frontend chat paths resolve the Beaver username from the
|
||||
issued access token.
|
||||
- The resolved username is passed separately into the agent runtime as the
|
||||
Gateway identity key.
|
||||
- Client-provided `user_id` fields do not select Gateway credentials and cannot
|
||||
impersonate another Gateway user.
|
||||
- Runs without an authenticated frontend username, including channel or
|
||||
scheduled runs without a trusted mapped identity, continue with curated
|
||||
memory only.
|
||||
|
||||
This identity key is runtime-only. It is not included in provider prompts or
|
||||
Gateway persisted message content.
|
||||
|
||||
## Runtime Architecture
|
||||
|
||||
`EngineLoader` loads:
|
||||
|
||||
1. Curated `MemoryService`, unconditionally.
|
||||
2. Shared `MemoryGatewayConfig` from `memory/config.json`.
|
||||
3. A `MemoryGatewayCredentialStore` for the instance credential file.
|
||||
|
||||
It does not construct one fixed-user `MemoryGatewayService` at startup.
|
||||
|
||||
For each authenticated run in hybrid mode:
|
||||
|
||||
1. `AgentLoop` receives the trusted Beaver username.
|
||||
2. It reads that username's credential from the credential store.
|
||||
3. If a credential exists, it constructs a run-local Gateway service/client
|
||||
from the shared config and that credential.
|
||||
4. It performs Gateway recall before context construction.
|
||||
5. It performs Gateway add and flush after normal completion.
|
||||
|
||||
The run-local service has no shared mutable credential state, so concurrent
|
||||
runs for different users cannot exchange identities. No service cache is added
|
||||
in this version.
|
||||
|
||||
## Recall and Persistence
|
||||
|
||||
The existing hybrid behavior remains unchanged once a user credential has
|
||||
been resolved:
|
||||
|
||||
- Search uses the current Beaver session id, current prompt, configured top K,
|
||||
and all three configured scopes.
|
||||
- Sanitized Gateway results are injected as one ephemeral untrusted-reference
|
||||
message outside the system prompt.
|
||||
- Normal completion persists exactly the original current user prompt and final
|
||||
assistant text.
|
||||
- Add is called once, followed by flush once only after add succeeds.
|
||||
- Tool calls, tool results, system prompts, curated memory, recalled Gateway
|
||||
text, reasoning, and skills are not persisted to Gateway.
|
||||
- Gateway and curated memory remain isolated and do not synchronize, merge,
|
||||
overwrite, or deduplicate each other.
|
||||
|
||||
## Security
|
||||
|
||||
- The shared configuration is safe to track because it contains no `userKey`.
|
||||
- Per-user credentials live only under ignored instance runtime data.
|
||||
- Credential-file permissions are `0600`.
|
||||
- Credential objects suppress secrets from `repr`.
|
||||
- Gateway client exceptions contain only operation, category, path, and status
|
||||
metadata.
|
||||
- Registration responses expose Beaver authentication data only; Gateway
|
||||
credentials remain server-side.
|
||||
- Hidden Gateway audit events may include the Beaver/Gateway user id but never
|
||||
the user key or complete request/response body.
|
||||
|
||||
## Testing
|
||||
|
||||
### Package migration
|
||||
|
||||
- All imports use `beaver.memory.gateway`.
|
||||
- No references remain to the removed integration/service modules.
|
||||
- Gateway config, client, service, and credential-store tests remain isolated
|
||||
from curated memory.
|
||||
|
||||
### Shared configuration
|
||||
|
||||
- The shared file parses the fixed URL and three scopes.
|
||||
- Invalid mode, URL, scope, top K, or timeout fails with sanitized errors.
|
||||
- Instance config loading remains unchanged for non-memory settings.
|
||||
- Test overrides can select a temporary shared config file.
|
||||
|
||||
### Credential persistence
|
||||
|
||||
- Missing files produce an empty credential map.
|
||||
- Credentials round-trip by Beaver username.
|
||||
- Updating one user preserves all other users.
|
||||
- Files are atomically replaced and have mode `0600`.
|
||||
- No exception or representation contains `userKey`.
|
||||
|
||||
### Registration
|
||||
|
||||
- New frontend registration calls `/users` with the Beaver username.
|
||||
- Valid Gateway responses are stored without returning the key to the browser.
|
||||
- Existing usernames refresh the same credential entry.
|
||||
- Provisioning failure does not roll back Beaver registration and stores no
|
||||
partial credential.
|
||||
|
||||
### Agent runtime
|
||||
|
||||
- Authenticated username selects only its own Gateway credential.
|
||||
- Client-provided `user_id` cannot select another user's credential.
|
||||
- Concurrent users construct independent run-local Gateway services.
|
||||
- Missing credentials perform no Gateway calls and preserve curated behavior.
|
||||
- Existing recall/add/flush ordering, payload, audit, and failure tests remain
|
||||
valid.
|
||||
|
||||
### Verification
|
||||
|
||||
- Run targeted Gateway/config/auth/chat tests.
|
||||
- Run Python compile checks and the complete backend test suite.
|
||||
- Scan tracked files and diffs for real `userKey` values.
|
||||
Reference in New Issue
Block a user