beaver_project/docs/superpowers/specs/2026-06-15-memory-gateway-package-migration-design.md

Memory Gateway Package and User Provisioning Design

Goal

Reorganize Beaver's Memory Gateway code under the beaver.memory domain and replace the single fixed Gateway identity with per-Beaver-user credentials.

The final model has two independent configuration layers:

• One shared, non-secret Memory Gateway configuration used by every Beaver instance.
• One per-instance credential file containing the Gateway identities created for Beaver frontend users.

Curated memory remains enabled and isolated. Gateway failures or missing user credentials must not modify MEMORY.md, USER.md, or the memory tool.

Source Package

All Beaver-side Gateway source moves to:

app-instance/backend/beaver/memory/gateway/
├── __init__.py
├── config.py
├── client.py
├── credentials.py
└── service.py

• config.py owns the shared typed Gateway configuration.
• client.py owns MemoryGatewayClient and sanitized client exceptions.
• credentials.py owns typed user credentials and atomic credential-file persistence.
• service.py owns search/add/flush orchestration and result types.
• __init__.py exposes the supported public Gateway API.

Remove the old source locations:

• beaver/integrations/memory_gateway/
• beaver/services/memory_gateway_service.py
• Gateway configuration dataclasses in beaver.foundation.config.schema
• The lazy MemoryGatewayService export from beaver.services

No compatibility forwarding modules are retained. After migration, beaver.memory.gateway is the only supported source entry point.

Shared Configuration

All Beaver instances read the same public Gateway configuration from:

/home/tom/beaver_project/app-instance/backend/memory/config.json

Inside the app-instance image this is available as:

/opt/app/backend/memory/config.json

The file contains no user credentials:

{
  "memory": {
    "mode": "hybrid",
    "gateway": {
      "baseUrl": "http://172.19.207.37:8010",
      "appId": "default",
      "projectId": "default",
      "scope": ["current_chat", "resources", "all_user_memory"],
      "topK": 8,
      "timeoutSeconds": 10
    }
  }
}

Rules:

• Valid modes remain curated and hybrid.
• Curated memory is always initialized.
• hybrid enables Gateway only for runs with a resolved user credential.
• baseUrl is fixed to http://172.19.207.37:8010 in the initial shared configuration.
• Scope includes current_chat, resources, and all_user_memory.
• The shared file is the authoritative Memory Gateway configuration. Instance config.json files continue to own providers, tools, channels, AuthZ, and backend identity, but no longer carry Gateway user credentials.
• An optional BEAVER_MEMORY_CONFIG_PATH may override the shared file path for tests or non-image development runs.

Per-Instance User Credentials

Each Beaver instance stores Gateway user credentials alongside its existing config.json, runtime.env, and web_auth_users.json:

app-instance/runtime/instances/<instance-slug>/beaver-home/
├── config.json
├── runtime.env
├── web_auth_users.json
└── memory_gateway_users.json

The existing beaver-home mount exposes the file inside the container as:

/root/.beaver/memory_gateway_users.json

The JSON format is:

{
  "users": {
    "tom": {
      "userId": "tom",
      "userKey": "uk_xxx"
    }
  }
}

Rules:

• The map key is the authenticated Beaver login username.
• Gateway userId is exactly the Beaver login username, with no prefix.
• userKey is secret and must never appear in API responses, logs, audit events, exceptions, or tracked configuration.
• Writes use a sibling temporary file followed by atomic replace.
• The credential file is created with mode 0600.
• BEAVER_MEMORY_GATEWAY_USERS_PATH may override the default path for tests.

Frontend User Provisioning

The frontend continues to call Beaver's existing POST /api/auth/register endpoint. The browser never calls Memory Gateway directly and never receives the Gateway userKey.

For a registration request with username tom, Beaver performs:

POST http://172.19.207.37:8010/users
Content-Type: application/json

{"user_id":"tom"}

Beaver validates that the response contains non-empty user_id and user_key, requires the returned user_id to equal tom, and stores the credential under the tom entry in memory_gateway_users.json.

The Gateway /users API is treated as idempotent. Registering an existing Beaver username may refresh the same credential entry without creating a second local identity.

For this first version:

• Gateway provisioning has no Beaver-side retries.
• A Gateway provisioning failure does not roll back an otherwise valid Beaver registration.
• A user without stored Gateway credentials continues with curated memory only.
• No separate repair UI or background credential provisioning job is added.

Authenticated Chat Identity

Gateway credential selection must use a trusted server-side principal.

• REST and WebSocket frontend chat paths resolve the Beaver username from the issued access token.
• The resolved username is passed separately into the agent runtime as the Gateway identity key.
• Client-provided user_id fields do not select Gateway credentials and cannot impersonate another Gateway user.
• Runs without an authenticated frontend username, including channel or scheduled runs without a trusted mapped identity, continue with curated memory only.

This identity key is runtime-only. It is not included in provider prompts or Gateway persisted message content.

Runtime Architecture

EngineLoader loads:

1. Curated MemoryService, unconditionally.
2. Shared MemoryGatewayConfig from memory/config.json.
3. A MemoryGatewayCredentialStore for the instance credential file.

It does not construct one fixed-user MemoryGatewayService at startup.

For each authenticated run in hybrid mode:

1. AgentLoop receives the trusted Beaver username.
2. It reads that username's credential from the credential store.
3. If a credential exists, it constructs a run-local Gateway service/client from the shared config and that credential.
4. It performs Gateway recall before context construction.
5. It performs Gateway add and flush after normal completion.

The run-local service has no shared mutable credential state, so concurrent runs for different users cannot exchange identities. No service cache is added in this version.

Recall and Persistence

The existing hybrid behavior remains unchanged once a user credential has been resolved:

• Search uses the current Beaver session id, current prompt, configured top K, and all three configured scopes.
• Sanitized Gateway results are injected as one ephemeral untrusted-reference message outside the system prompt.
• Normal completion persists exactly the original current user prompt and final assistant text.
• Add is called once, followed by flush once only after add succeeds.
• Tool calls, tool results, system prompts, curated memory, recalled Gateway text, reasoning, and skills are not persisted to Gateway.
• Gateway and curated memory remain isolated and do not synchronize, merge, overwrite, or deduplicate each other.

Security

• The shared configuration is safe to track because it contains no userKey.
• Per-user credentials live only under ignored instance runtime data.
• Credential-file permissions are 0600.
• Credential objects suppress secrets from repr.
• Gateway client exceptions contain only operation, category, path, and status metadata.
• Registration responses expose Beaver authentication data only; Gateway credentials remain server-side.
• Hidden Gateway audit events may include the Beaver/Gateway user id but never the user key or complete request/response body.

Testing

Package migration

• All imports use beaver.memory.gateway.
• No references remain to the removed integration/service modules.
• Gateway config, client, service, and credential-store tests remain isolated from curated memory.

Shared configuration

• The shared file parses the fixed URL and three scopes.
• Invalid mode, URL, scope, top K, or timeout fails with sanitized errors.
• Instance config loading remains unchanged for non-memory settings.
• Test overrides can select a temporary shared config file.

Credential persistence

• Missing files produce an empty credential map.
• Credentials round-trip by Beaver username.
• Updating one user preserves all other users.
• Files are atomically replaced and have mode 0600.
• No exception or representation contains userKey.

Registration

• New frontend registration calls /users with the Beaver username.
• Valid Gateway responses are stored without returning the key to the browser.
• Existing usernames refresh the same credential entry.
• Provisioning failure does not roll back Beaver registration and stores no partial credential.

Agent runtime

• Authenticated username selects only its own Gateway credential.
• Client-provided user_id cannot select another user's credential.
• Concurrent users construct independent run-local Gateway services.
• Missing credentials perform no Gateway calls and preserve curated behavior.
• Existing recall/add/flush ordering, payload, audit, and failure tests remain valid.

Verification

• Run targeted Gateway/config/auth/chat tests.
• Run Python compile checks and the complete backend test suite.
• Scan tracked files and diffs for real userKey values.