From 8b57159d46760505238703f4b0373eba9b4e7b63 Mon Sep 17 00:00:00 2001 From: tomtan Date: Mon, 15 Jun 2026 18:02:22 +0800 Subject: [PATCH] docs: define shared gateway config and user provisioning --- ...memory-gateway-package-migration-design.md | 294 ++++++++++++++---- 1 file changed, 236 insertions(+), 58 deletions(-) diff --git a/docs/superpowers/specs/2026-06-15-memory-gateway-package-migration-design.md b/docs/superpowers/specs/2026-06-15-memory-gateway-package-migration-design.md index f413ec4..cac7a24 100644 --- a/docs/superpowers/specs/2026-06-15-memory-gateway-package-migration-design.md +++ b/docs/superpowers/specs/2026-06-15-memory-gateway-package-migration-design.md @@ -1,83 +1,75 @@ -# Memory Gateway Package Migration Design +# Memory Gateway Package and User Provisioning Design ## Goal -Move Beaver's Memory Gateway source code and typed configuration into the -existing `beaver.memory` domain package without changing runtime behavior. +Reorganize Beaver's Memory Gateway code under the `beaver.memory` domain and +replace the single fixed Gateway identity with per-Beaver-user credentials. -This migration affects Python source organization only. It does not move or -modify runtime data under `app-instance/backend/memory/`, the standalone -`/home/tom/memory-gateway` service, or Beaver instance configuration files. +The final model has two independent configuration layers: -## Target Structure +- One shared, non-secret Memory Gateway configuration used by every Beaver + instance. +- One per-instance credential file containing the Gateway identities created + for Beaver frontend users. + +Curated memory remains enabled and isolated. Gateway failures or missing user +credentials must not modify `MEMORY.md`, `USER.md`, or the `memory` tool. + +## Source Package + +All Beaver-side Gateway source moves to: ```text app-instance/backend/beaver/memory/gateway/ ├── __init__.py ├── config.py ├── client.py +├── credentials.py └── service.py ``` -- `config.py` owns `MemoryConfig` and `MemoryGatewayConfig`. -- `client.py` owns `MemoryGatewayClient` and - `MemoryGatewayClientError`. -- `service.py` owns `MemoryGatewayService`, `GatewayRecallOutcome`, and - `GatewayPersistOutcome`. -- `__init__.py` exposes the public Gateway API used by the loader and tests. +- `config.py` owns the shared typed Gateway configuration. +- `client.py` owns `MemoryGatewayClient` and sanitized client exceptions. +- `credentials.py` owns typed user credentials and atomic credential-file + persistence. +- `service.py` owns search/add/flush orchestration and result types. +- `__init__.py` exposes the supported public Gateway API. -## Source Changes +Remove the old source locations: -- Remove `beaver/integrations/memory_gateway/`. -- Remove `beaver/services/memory_gateway_service.py`. -- Remove the lazy `MemoryGatewayService` export from `beaver.services`. -- Move the Gateway configuration dataclasses out of - `beaver.foundation.config.schema`. -- Keep `beaver.foundation.config.loader` responsible for parsing the top-level - `memory` JSON section, importing the typed models from - `beaver.memory.gateway`. -- Update `EngineLoader`, tests, README references, and implementation-plan - source paths to use the new package. -- Do not retain compatibility import modules. After migration, - `beaver.memory.gateway` is the only supported source entry point. +- `beaver/integrations/memory_gateway/` +- `beaver/services/memory_gateway_service.py` +- Gateway configuration dataclasses in `beaver.foundation.config.schema` +- The lazy `MemoryGatewayService` export from `beaver.services` -## Configuration Location +No compatibility forwarding modules are retained. After migration, +`beaver.memory.gateway` is the only supported source entry point. -Memory remains configured in each Beaver instance's `config.json`, not inside -the Python package or runtime memory data directory. +## Shared Configuration -Configuration lookup order remains: - -1. `BEAVER_CONFIG_PATH` -2. `$BEAVER_HOME/config.json` -3. `/.beaver/config.json` -4. `./.beaver/config.json` - -For the Docker app instance, the container reads: +All Beaver instances read the same public Gateway configuration from: ```text -/root/.beaver/config.json +/home/tom/beaver_project/app-instance/backend/memory/config.json ``` -The corresponding host file is: +Inside the app-instance image this is available as: ```text -app-instance/runtime/instances//beaver-home/config.json +/opt/app/backend/memory/config.json ``` -The `memory` section remains unchanged: +The file contains no user credentials: ```json { "memory": { "mode": "hybrid", "gateway": { - "baseUrl": "http://172.19.0.1:8010", - "userId": "gateway_test_user", - "userKey": "uk_xxx", + "baseUrl": "http://172.19.207.37:8010", "appId": "default", "projectId": "default", - "scope": ["current_chat", "resources"], + "scope": ["current_chat", "resources", "all_user_memory"], "topK": 8, "timeoutSeconds": 10 } @@ -85,20 +77,206 @@ The `memory` section remains unchanged: } ``` -`userKey` remains a secret and must not be committed or logged. +Rules: -## Behavioral Guarantees +- Valid modes remain `curated` and `hybrid`. +- Curated memory is always initialized. +- `hybrid` enables Gateway only for runs with a resolved user credential. +- `baseUrl` is fixed to `http://172.19.207.37:8010` in the initial shared + configuration. +- Scope includes `current_chat`, `resources`, and `all_user_memory`. +- The shared file is the authoritative Memory Gateway configuration. Instance + `config.json` files continue to own providers, tools, channels, AuthZ, and + backend identity, but no longer carry Gateway user credentials. +- An optional `BEAVER_MEMORY_CONFIG_PATH` may override the shared file path for + tests or non-image development runs. -- Curated memory behavior is unchanged. -- Hybrid search/add/flush request payloads and ordering are unchanged. -- Gateway audit events and best-effort failure semantics are unchanged. -- The standalone Memory Gateway deployment remains outside Beaver. -- `app-instance/backend/memory/` continues to contain runtime memory data only. +## Per-Instance User Credentials -## Verification +Each Beaver instance stores Gateway user credentials alongside its existing +`config.json`, `runtime.env`, and `web_auth_users.json`: -- Update all imports and assert no references remain to the removed modules. -- Run Gateway configuration, client/service, loader, context, and AgentLoop - tests. -- Run the complete backend test suite and Python compile check. -- Scan tracked diffs for credential values. +```text +app-instance/runtime/instances//beaver-home/ +├── config.json +├── runtime.env +├── web_auth_users.json +└── memory_gateway_users.json +``` + +The existing `beaver-home` mount exposes the file inside the container as: + +```text +/root/.beaver/memory_gateway_users.json +``` + +The JSON format is: + +```json +{ + "users": { + "tom": { + "userId": "tom", + "userKey": "uk_xxx" + } + } +} +``` + +Rules: + +- The map key is the authenticated Beaver login username. +- Gateway `userId` is exactly the Beaver login username, with no prefix. +- `userKey` is secret and must never appear in API responses, logs, audit + events, exceptions, or tracked configuration. +- Writes use a sibling temporary file followed by atomic replace. +- The credential file is created with mode `0600`. +- `BEAVER_MEMORY_GATEWAY_USERS_PATH` may override the default path for tests. + +## Frontend User Provisioning + +The frontend continues to call Beaver's existing `POST /api/auth/register` +endpoint. The browser never calls Memory Gateway directly and never receives +the Gateway `userKey`. + +For a registration request with username `tom`, Beaver performs: + +```http +POST http://172.19.207.37:8010/users +Content-Type: application/json + +{"user_id":"tom"} +``` + +Beaver validates that the response contains non-empty `user_id` and +`user_key`, requires the returned `user_id` to equal `tom`, and stores the +credential under the `tom` entry in `memory_gateway_users.json`. + +The Gateway `/users` API is treated as idempotent. Registering an existing +Beaver username may refresh the same credential entry without creating a +second local identity. + +For this first version: + +- Gateway provisioning has no Beaver-side retries. +- A Gateway provisioning failure does not roll back an otherwise valid Beaver + registration. +- A user without stored Gateway credentials continues with curated memory only. +- No separate repair UI or background credential provisioning job is added. + +## Authenticated Chat Identity + +Gateway credential selection must use a trusted server-side principal. + +- REST and WebSocket frontend chat paths resolve the Beaver username from the + issued access token. +- The resolved username is passed separately into the agent runtime as the + Gateway identity key. +- Client-provided `user_id` fields do not select Gateway credentials and cannot + impersonate another Gateway user. +- Runs without an authenticated frontend username, including channel or + scheduled runs without a trusted mapped identity, continue with curated + memory only. + +This identity key is runtime-only. It is not included in provider prompts or +Gateway persisted message content. + +## Runtime Architecture + +`EngineLoader` loads: + +1. Curated `MemoryService`, unconditionally. +2. Shared `MemoryGatewayConfig` from `memory/config.json`. +3. A `MemoryGatewayCredentialStore` for the instance credential file. + +It does not construct one fixed-user `MemoryGatewayService` at startup. + +For each authenticated run in hybrid mode: + +1. `AgentLoop` receives the trusted Beaver username. +2. It reads that username's credential from the credential store. +3. If a credential exists, it constructs a run-local Gateway service/client + from the shared config and that credential. +4. It performs Gateway recall before context construction. +5. It performs Gateway add and flush after normal completion. + +The run-local service has no shared mutable credential state, so concurrent +runs for different users cannot exchange identities. No service cache is added +in this version. + +## Recall and Persistence + +The existing hybrid behavior remains unchanged once a user credential has +been resolved: + +- Search uses the current Beaver session id, current prompt, configured top K, + and all three configured scopes. +- Sanitized Gateway results are injected as one ephemeral untrusted-reference + message outside the system prompt. +- Normal completion persists exactly the original current user prompt and final + assistant text. +- Add is called once, followed by flush once only after add succeeds. +- Tool calls, tool results, system prompts, curated memory, recalled Gateway + text, reasoning, and skills are not persisted to Gateway. +- Gateway and curated memory remain isolated and do not synchronize, merge, + overwrite, or deduplicate each other. + +## Security + +- The shared configuration is safe to track because it contains no `userKey`. +- Per-user credentials live only under ignored instance runtime data. +- Credential-file permissions are `0600`. +- Credential objects suppress secrets from `repr`. +- Gateway client exceptions contain only operation, category, path, and status + metadata. +- Registration responses expose Beaver authentication data only; Gateway + credentials remain server-side. +- Hidden Gateway audit events may include the Beaver/Gateway user id but never + the user key or complete request/response body. + +## Testing + +### Package migration + +- All imports use `beaver.memory.gateway`. +- No references remain to the removed integration/service modules. +- Gateway config, client, service, and credential-store tests remain isolated + from curated memory. + +### Shared configuration + +- The shared file parses the fixed URL and three scopes. +- Invalid mode, URL, scope, top K, or timeout fails with sanitized errors. +- Instance config loading remains unchanged for non-memory settings. +- Test overrides can select a temporary shared config file. + +### Credential persistence + +- Missing files produce an empty credential map. +- Credentials round-trip by Beaver username. +- Updating one user preserves all other users. +- Files are atomically replaced and have mode `0600`. +- No exception or representation contains `userKey`. + +### Registration + +- New frontend registration calls `/users` with the Beaver username. +- Valid Gateway responses are stored without returning the key to the browser. +- Existing usernames refresh the same credential entry. +- Provisioning failure does not roll back Beaver registration and stores no + partial credential. + +### Agent runtime + +- Authenticated username selects only its own Gateway credential. +- Client-provided `user_id` cannot select another user's credential. +- Concurrent users construct independent run-local Gateway services. +- Missing credentials perform no Gateway calls and preserve curated behavior. +- Existing recall/add/flush ordering, payload, audit, and failure tests remain + valid. + +### Verification + +- Run targeted Gateway/config/auth/chat tests. +- Run Python compile checks and the complete backend test suite. +- Scan tracked files and diffs for real `userKey` values.