beaver_project/docs/superpowers/specs/2026-06-15-memory-gateway-package-migration-design.md

# Memory Gateway Package and User Provisioning Design

## Goal

Reorganize Beaver's Memory Gateway code under the `beaver.memory` domain and
replace the single fixed Gateway identity with per-Beaver-user credentials.

The final model has two independent configuration layers:

- One shared, non-secret Memory Gateway configuration used by every Beaver
  instance.
- One per-instance credential file containing the Gateway identities created
  for Beaver frontend users.

Curated memory remains enabled and isolated. Gateway failures or missing user
credentials must not modify `MEMORY.md`, `USER.md`, or the `memory` tool.

## Source Package

All Beaver-side Gateway source moves to:

```text
app-instance/backend/beaver/memory/gateway/
├── __init__.py
├── config.py
├── client.py
├── credentials.py
└── service.py
```

- `config.py` owns the shared typed Gateway configuration.
- `client.py` owns `MemoryGatewayClient` and sanitized client exceptions.
- `credentials.py` owns typed user credentials and atomic credential-file
  persistence.
- `service.py` owns search/add/flush orchestration and result types.
- `__init__.py` exposes the supported public Gateway API.

Remove the old source locations:

- `beaver/integrations/memory_gateway/`
- `beaver/services/memory_gateway_service.py`
- Gateway configuration dataclasses in `beaver.foundation.config.schema`
- The lazy `MemoryGatewayService` export from `beaver.services`

No compatibility forwarding modules are retained. After migration,
`beaver.memory.gateway` is the only supported source entry point.

## Shared Configuration

All Beaver instances read the same public Gateway configuration from:

```text
/home/tom/beaver_project/app-instance/backend/memory/config.json
```

Inside the app-instance image this is available as:

```text
/opt/app/backend/memory/config.json
```

The file contains no user credentials:

```json
{
  "memory": {
    "mode": "hybrid",
    "gateway": {
      "baseUrl": "http://172.19.207.37:8010",
      "appId": "default",
      "projectId": "default",
      "scope": ["current_chat", "resources", "all_user_memory"],
      "topK": 8,
      "timeoutSeconds": 10
    }
  }
}
```

Rules:

- Valid modes remain `curated` and `hybrid`.
- Curated memory is always initialized.
- `hybrid` enables Gateway only for runs with a resolved user credential.
- `baseUrl` is fixed to `http://172.19.207.37:8010` in the initial shared
  configuration.
- Scope includes `current_chat`, `resources`, and `all_user_memory`.
- The shared file is the authoritative Memory Gateway configuration. Instance
  `config.json` files continue to own providers, tools, channels, AuthZ, and
  backend identity, but no longer carry Gateway user credentials.
- An optional `BEAVER_MEMORY_CONFIG_PATH` may override the shared file path for
  tests or non-image development runs.

## Per-Instance User Credentials

Each Beaver instance stores Gateway user credentials alongside its existing
`config.json`, `runtime.env`, and `web_auth_users.json`:

```text
app-instance/runtime/instances/<instance-slug>/beaver-home/
├── config.json
├── runtime.env
├── web_auth_users.json
└── memory_gateway_users.json
```

The existing `beaver-home` mount exposes the file inside the container as:

```text
/root/.beaver/memory_gateway_users.json
```

The JSON format is:

```json
{
  "users": {
    "tom": {
      "userId": "tom",
      "userKey": "uk_xxx"
    }
  }
}
```

Rules:

- The map key is the authenticated Beaver login username.
- Gateway `userId` is exactly the Beaver login username, with no prefix.
- `userKey` is secret and must never appear in API responses, logs, audit
  events, exceptions, or tracked configuration.
- Writes use a sibling temporary file followed by atomic replace.
- The credential file is created with mode `0600`.
- `BEAVER_MEMORY_GATEWAY_USERS_PATH` may override the default path for tests.

## Frontend User Provisioning

The frontend continues to call Beaver's existing `POST /api/auth/register`
endpoint. The browser never calls Memory Gateway directly and never receives
the Gateway `userKey`.

For a registration request with username `tom`, Beaver performs:

```http
POST http://172.19.207.37:8010/users
Content-Type: application/json

{"user_id":"tom"}
```

Beaver validates that the response contains non-empty `user_id` and
`user_key`, requires the returned `user_id` to equal `tom`, and stores the
credential under the `tom` entry in `memory_gateway_users.json`.

The Gateway `/users` API is treated as idempotent. Registering an existing
Beaver username may refresh the same credential entry without creating a
second local identity.

For this first version:

- Gateway provisioning has no Beaver-side retries.
- A Gateway provisioning failure does not roll back an otherwise valid Beaver
  registration.
- A user without stored Gateway credentials continues with curated memory only.
- No separate repair UI or background credential provisioning job is added.

## Authenticated Chat Identity

Gateway credential selection must use a trusted server-side principal.

- REST and WebSocket frontend chat paths resolve the Beaver username from the
  issued access token.
- The resolved username is passed separately into the agent runtime as the
  Gateway identity key.
- Client-provided `user_id` fields do not select Gateway credentials and cannot
  impersonate another Gateway user.
- Runs without an authenticated frontend username, including channel or
  scheduled runs without a trusted mapped identity, continue with curated
  memory only.

This identity key is runtime-only. It is not included in provider prompts or
Gateway persisted message content.

## Runtime Architecture

`EngineLoader` loads:

1. Curated `MemoryService`, unconditionally.
2. Shared `MemoryGatewayConfig` from `memory/config.json`.
3. A `MemoryGatewayCredentialStore` for the instance credential file.

It does not construct one fixed-user `MemoryGatewayService` at startup.

For each authenticated run in hybrid mode:

1. `AgentLoop` receives the trusted Beaver username.
2. It reads that username's credential from the credential store.
3. If a credential exists, it constructs a run-local Gateway service/client
   from the shared config and that credential.
4. It performs Gateway recall before context construction.
5. It performs Gateway add and flush after normal completion.

The run-local service has no shared mutable credential state, so concurrent
runs for different users cannot exchange identities. No service cache is added
in this version.

## Recall and Persistence

The existing hybrid behavior remains unchanged once a user credential has
been resolved:

- Search uses the current Beaver session id, current prompt, configured top K,
  and all three configured scopes.
- Sanitized Gateway results are injected as one ephemeral untrusted-reference
  message outside the system prompt.
- Normal completion persists exactly the original current user prompt and final
  assistant text.
- Add is called once, followed by flush once only after add succeeds.
- Tool calls, tool results, system prompts, curated memory, recalled Gateway
  text, reasoning, and skills are not persisted to Gateway.
- Gateway and curated memory remain isolated and do not synchronize, merge,
  overwrite, or deduplicate each other.

## Security

- The shared configuration is safe to track because it contains no `userKey`.
- Per-user credentials live only under ignored instance runtime data.
- Credential-file permissions are `0600`.
- Credential objects suppress secrets from `repr`.
- Gateway client exceptions contain only operation, category, path, and status
  metadata.
- Registration responses expose Beaver authentication data only; Gateway
  credentials remain server-side.
- Hidden Gateway audit events may include the Beaver/Gateway user id but never
  the user key or complete request/response body.

## Testing

### Package migration

- All imports use `beaver.memory.gateway`.
- No references remain to the removed integration/service modules.
- Gateway config, client, service, and credential-store tests remain isolated
  from curated memory.

### Shared configuration

- The shared file parses the fixed URL and three scopes.
- Invalid mode, URL, scope, top K, or timeout fails with sanitized errors.
- Instance config loading remains unchanged for non-memory settings.
- Test overrides can select a temporary shared config file.

### Credential persistence

- Missing files produce an empty credential map.
- Credentials round-trip by Beaver username.
- Updating one user preserves all other users.
- Files are atomically replaced and have mode `0600`.
- No exception or representation contains `userKey`.

### Registration

- New frontend registration calls `/users` with the Beaver username.
- Valid Gateway responses are stored without returning the key to the browser.
- Existing usernames refresh the same credential entry.
- Provisioning failure does not roll back Beaver registration and stores no
  partial credential.

### Agent runtime

- Authenticated username selects only its own Gateway credential.
- Client-provided `user_id` cannot select another user's credential.
- Concurrent users construct independent run-local Gateway services.
- Missing credentials perform no Gateway calls and preserve curated behavior.
- Existing recall/add/flush ordering, payload, audit, and failure tests remain
  valid.

### Verification

- Run targeted Gateway/config/auth/chat tests.
- Run Python compile checks and the complete backend test suite.
- Scan tracked files and diffs for real `userKey` values.