[codex] feat(memory-gateway): merge memory mode with main #2
@ -1,83 +1,75 @@
|
||||
# Memory Gateway Package Migration Design
|
||||
# Memory Gateway Package and User Provisioning Design
|
||||
|
||||
## Goal
|
||||
|
||||
Move Beaver's Memory Gateway source code and typed configuration into the
|
||||
existing `beaver.memory` domain package without changing runtime behavior.
|
||||
Reorganize Beaver's Memory Gateway code under the `beaver.memory` domain and
|
||||
replace the single fixed Gateway identity with per-Beaver-user credentials.
|
||||
|
||||
This migration affects Python source organization only. It does not move or
|
||||
modify runtime data under `app-instance/backend/memory/`, the standalone
|
||||
`/home/tom/memory-gateway` service, or Beaver instance configuration files.
|
||||
The final model has two independent configuration layers:
|
||||
|
||||
## Target Structure
|
||||
- One shared, non-secret Memory Gateway configuration used by every Beaver
|
||||
instance.
|
||||
- One per-instance credential file containing the Gateway identities created
|
||||
for Beaver frontend users.
|
||||
|
||||
Curated memory remains enabled and isolated. Gateway failures or missing user
|
||||
credentials must not modify `MEMORY.md`, `USER.md`, or the `memory` tool.
|
||||
|
||||
## Source Package
|
||||
|
||||
All Beaver-side Gateway source moves to:
|
||||
|
||||
```text
|
||||
app-instance/backend/beaver/memory/gateway/
|
||||
├── __init__.py
|
||||
├── config.py
|
||||
├── client.py
|
||||
├── credentials.py
|
||||
└── service.py
|
||||
```
|
||||
|
||||
- `config.py` owns `MemoryConfig` and `MemoryGatewayConfig`.
|
||||
- `client.py` owns `MemoryGatewayClient` and
|
||||
`MemoryGatewayClientError`.
|
||||
- `service.py` owns `MemoryGatewayService`, `GatewayRecallOutcome`, and
|
||||
`GatewayPersistOutcome`.
|
||||
- `__init__.py` exposes the public Gateway API used by the loader and tests.
|
||||
- `config.py` owns the shared typed Gateway configuration.
|
||||
- `client.py` owns `MemoryGatewayClient` and sanitized client exceptions.
|
||||
- `credentials.py` owns typed user credentials and atomic credential-file
|
||||
persistence.
|
||||
- `service.py` owns search/add/flush orchestration and result types.
|
||||
- `__init__.py` exposes the supported public Gateway API.
|
||||
|
||||
## Source Changes
|
||||
Remove the old source locations:
|
||||
|
||||
- Remove `beaver/integrations/memory_gateway/`.
|
||||
- Remove `beaver/services/memory_gateway_service.py`.
|
||||
- Remove the lazy `MemoryGatewayService` export from `beaver.services`.
|
||||
- Move the Gateway configuration dataclasses out of
|
||||
`beaver.foundation.config.schema`.
|
||||
- Keep `beaver.foundation.config.loader` responsible for parsing the top-level
|
||||
`memory` JSON section, importing the typed models from
|
||||
`beaver.memory.gateway`.
|
||||
- Update `EngineLoader`, tests, README references, and implementation-plan
|
||||
source paths to use the new package.
|
||||
- Do not retain compatibility import modules. After migration,
|
||||
`beaver.memory.gateway` is the only supported source entry point.
|
||||
- `beaver/integrations/memory_gateway/`
|
||||
- `beaver/services/memory_gateway_service.py`
|
||||
- Gateway configuration dataclasses in `beaver.foundation.config.schema`
|
||||
- The lazy `MemoryGatewayService` export from `beaver.services`
|
||||
|
||||
## Configuration Location
|
||||
No compatibility forwarding modules are retained. After migration,
|
||||
`beaver.memory.gateway` is the only supported source entry point.
|
||||
|
||||
Memory remains configured in each Beaver instance's `config.json`, not inside
|
||||
the Python package or runtime memory data directory.
|
||||
## Shared Configuration
|
||||
|
||||
Configuration lookup order remains:
|
||||
|
||||
1. `BEAVER_CONFIG_PATH`
|
||||
2. `$BEAVER_HOME/config.json`
|
||||
3. `<workspace>/.beaver/config.json`
|
||||
4. `./.beaver/config.json`
|
||||
|
||||
For the Docker app instance, the container reads:
|
||||
All Beaver instances read the same public Gateway configuration from:
|
||||
|
||||
```text
|
||||
/root/.beaver/config.json
|
||||
/home/tom/beaver_project/app-instance/backend/memory/config.json
|
||||
```
|
||||
|
||||
The corresponding host file is:
|
||||
Inside the app-instance image this is available as:
|
||||
|
||||
```text
|
||||
app-instance/runtime/instances/<instance-slug>/beaver-home/config.json
|
||||
/opt/app/backend/memory/config.json
|
||||
```
|
||||
|
||||
The `memory` section remains unchanged:
|
||||
The file contains no user credentials:
|
||||
|
||||
```json
|
||||
{
|
||||
"memory": {
|
||||
"mode": "hybrid",
|
||||
"gateway": {
|
||||
"baseUrl": "http://172.19.0.1:8010",
|
||||
"userId": "gateway_test_user",
|
||||
"userKey": "uk_xxx",
|
||||
"baseUrl": "http://172.19.207.37:8010",
|
||||
"appId": "default",
|
||||
"projectId": "default",
|
||||
"scope": ["current_chat", "resources"],
|
||||
"scope": ["current_chat", "resources", "all_user_memory"],
|
||||
"topK": 8,
|
||||
"timeoutSeconds": 10
|
||||
}
|
||||
@ -85,20 +77,206 @@ The `memory` section remains unchanged:
|
||||
}
|
||||
```
|
||||
|
||||
`userKey` remains a secret and must not be committed or logged.
|
||||
Rules:
|
||||
|
||||
## Behavioral Guarantees
|
||||
- Valid modes remain `curated` and `hybrid`.
|
||||
- Curated memory is always initialized.
|
||||
- `hybrid` enables Gateway only for runs with a resolved user credential.
|
||||
- `baseUrl` is fixed to `http://172.19.207.37:8010` in the initial shared
|
||||
configuration.
|
||||
- Scope includes `current_chat`, `resources`, and `all_user_memory`.
|
||||
- The shared file is the authoritative Memory Gateway configuration. Instance
|
||||
`config.json` files continue to own providers, tools, channels, AuthZ, and
|
||||
backend identity, but no longer carry Gateway user credentials.
|
||||
- An optional `BEAVER_MEMORY_CONFIG_PATH` may override the shared file path for
|
||||
tests or non-image development runs.
|
||||
|
||||
- Curated memory behavior is unchanged.
|
||||
- Hybrid search/add/flush request payloads and ordering are unchanged.
|
||||
- Gateway audit events and best-effort failure semantics are unchanged.
|
||||
- The standalone Memory Gateway deployment remains outside Beaver.
|
||||
- `app-instance/backend/memory/` continues to contain runtime memory data only.
|
||||
## Per-Instance User Credentials
|
||||
|
||||
## Verification
|
||||
Each Beaver instance stores Gateway user credentials alongside its existing
|
||||
`config.json`, `runtime.env`, and `web_auth_users.json`:
|
||||
|
||||
- Update all imports and assert no references remain to the removed modules.
|
||||
- Run Gateway configuration, client/service, loader, context, and AgentLoop
|
||||
tests.
|
||||
- Run the complete backend test suite and Python compile check.
|
||||
- Scan tracked diffs for credential values.
|
||||
```text
|
||||
app-instance/runtime/instances/<instance-slug>/beaver-home/
|
||||
├── config.json
|
||||
├── runtime.env
|
||||
├── web_auth_users.json
|
||||
└── memory_gateway_users.json
|
||||
```
|
||||
|
||||
The existing `beaver-home` mount exposes the file inside the container as:
|
||||
|
||||
```text
|
||||
/root/.beaver/memory_gateway_users.json
|
||||
```
|
||||
|
||||
The JSON format is:
|
||||
|
||||
```json
|
||||
{
|
||||
"users": {
|
||||
"tom": {
|
||||
"userId": "tom",
|
||||
"userKey": "uk_xxx"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- The map key is the authenticated Beaver login username.
|
||||
- Gateway `userId` is exactly the Beaver login username, with no prefix.
|
||||
- `userKey` is secret and must never appear in API responses, logs, audit
|
||||
events, exceptions, or tracked configuration.
|
||||
- Writes use a sibling temporary file followed by atomic replace.
|
||||
- The credential file is created with mode `0600`.
|
||||
- `BEAVER_MEMORY_GATEWAY_USERS_PATH` may override the default path for tests.
|
||||
|
||||
## Frontend User Provisioning
|
||||
|
||||
The frontend continues to call Beaver's existing `POST /api/auth/register`
|
||||
endpoint. The browser never calls Memory Gateway directly and never receives
|
||||
the Gateway `userKey`.
|
||||
|
||||
For a registration request with username `tom`, Beaver performs:
|
||||
|
||||
```http
|
||||
POST http://172.19.207.37:8010/users
|
||||
Content-Type: application/json
|
||||
|
||||
{"user_id":"tom"}
|
||||
```
|
||||
|
||||
Beaver validates that the response contains non-empty `user_id` and
|
||||
`user_key`, requires the returned `user_id` to equal `tom`, and stores the
|
||||
credential under the `tom` entry in `memory_gateway_users.json`.
|
||||
|
||||
The Gateway `/users` API is treated as idempotent. Registering an existing
|
||||
Beaver username may refresh the same credential entry without creating a
|
||||
second local identity.
|
||||
|
||||
For this first version:
|
||||
|
||||
- Gateway provisioning has no Beaver-side retries.
|
||||
- A Gateway provisioning failure does not roll back an otherwise valid Beaver
|
||||
registration.
|
||||
- A user without stored Gateway credentials continues with curated memory only.
|
||||
- No separate repair UI or background credential provisioning job is added.
|
||||
|
||||
## Authenticated Chat Identity
|
||||
|
||||
Gateway credential selection must use a trusted server-side principal.
|
||||
|
||||
- REST and WebSocket frontend chat paths resolve the Beaver username from the
|
||||
issued access token.
|
||||
- The resolved username is passed separately into the agent runtime as the
|
||||
Gateway identity key.
|
||||
- Client-provided `user_id` fields do not select Gateway credentials and cannot
|
||||
impersonate another Gateway user.
|
||||
- Runs without an authenticated frontend username, including channel or
|
||||
scheduled runs without a trusted mapped identity, continue with curated
|
||||
memory only.
|
||||
|
||||
This identity key is runtime-only. It is not included in provider prompts or
|
||||
Gateway persisted message content.
|
||||
|
||||
## Runtime Architecture
|
||||
|
||||
`EngineLoader` loads:
|
||||
|
||||
1. Curated `MemoryService`, unconditionally.
|
||||
2. Shared `MemoryGatewayConfig` from `memory/config.json`.
|
||||
3. A `MemoryGatewayCredentialStore` for the instance credential file.
|
||||
|
||||
It does not construct one fixed-user `MemoryGatewayService` at startup.
|
||||
|
||||
For each authenticated run in hybrid mode:
|
||||
|
||||
1. `AgentLoop` receives the trusted Beaver username.
|
||||
2. It reads that username's credential from the credential store.
|
||||
3. If a credential exists, it constructs a run-local Gateway service/client
|
||||
from the shared config and that credential.
|
||||
4. It performs Gateway recall before context construction.
|
||||
5. It performs Gateway add and flush after normal completion.
|
||||
|
||||
The run-local service has no shared mutable credential state, so concurrent
|
||||
runs for different users cannot exchange identities. No service cache is added
|
||||
in this version.
|
||||
|
||||
## Recall and Persistence
|
||||
|
||||
The existing hybrid behavior remains unchanged once a user credential has
|
||||
been resolved:
|
||||
|
||||
- Search uses the current Beaver session id, current prompt, configured top K,
|
||||
and all three configured scopes.
|
||||
- Sanitized Gateway results are injected as one ephemeral untrusted-reference
|
||||
message outside the system prompt.
|
||||
- Normal completion persists exactly the original current user prompt and final
|
||||
assistant text.
|
||||
- Add is called once, followed by flush once only after add succeeds.
|
||||
- Tool calls, tool results, system prompts, curated memory, recalled Gateway
|
||||
text, reasoning, and skills are not persisted to Gateway.
|
||||
- Gateway and curated memory remain isolated and do not synchronize, merge,
|
||||
overwrite, or deduplicate each other.
|
||||
|
||||
## Security
|
||||
|
||||
- The shared configuration is safe to track because it contains no `userKey`.
|
||||
- Per-user credentials live only under ignored instance runtime data.
|
||||
- Credential-file permissions are `0600`.
|
||||
- Credential objects suppress secrets from `repr`.
|
||||
- Gateway client exceptions contain only operation, category, path, and status
|
||||
metadata.
|
||||
- Registration responses expose Beaver authentication data only; Gateway
|
||||
credentials remain server-side.
|
||||
- Hidden Gateway audit events may include the Beaver/Gateway user id but never
|
||||
the user key or complete request/response body.
|
||||
|
||||
## Testing
|
||||
|
||||
### Package migration
|
||||
|
||||
- All imports use `beaver.memory.gateway`.
|
||||
- No references remain to the removed integration/service modules.
|
||||
- Gateway config, client, service, and credential-store tests remain isolated
|
||||
from curated memory.
|
||||
|
||||
### Shared configuration
|
||||
|
||||
- The shared file parses the fixed URL and three scopes.
|
||||
- Invalid mode, URL, scope, top K, or timeout fails with sanitized errors.
|
||||
- Instance config loading remains unchanged for non-memory settings.
|
||||
- Test overrides can select a temporary shared config file.
|
||||
|
||||
### Credential persistence
|
||||
|
||||
- Missing files produce an empty credential map.
|
||||
- Credentials round-trip by Beaver username.
|
||||
- Updating one user preserves all other users.
|
||||
- Files are atomically replaced and have mode `0600`.
|
||||
- No exception or representation contains `userKey`.
|
||||
|
||||
### Registration
|
||||
|
||||
- New frontend registration calls `/users` with the Beaver username.
|
||||
- Valid Gateway responses are stored without returning the key to the browser.
|
||||
- Existing usernames refresh the same credential entry.
|
||||
- Provisioning failure does not roll back Beaver registration and stores no
|
||||
partial credential.
|
||||
|
||||
### Agent runtime
|
||||
|
||||
- Authenticated username selects only its own Gateway credential.
|
||||
- Client-provided `user_id` cannot select another user's credential.
|
||||
- Concurrent users construct independent run-local Gateway services.
|
||||
- Missing credentials perform no Gateway calls and preserve curated behavior.
|
||||
- Existing recall/add/flush ordering, payload, audit, and failure tests remain
|
||||
valid.
|
||||
|
||||
### Verification
|
||||
|
||||
- Run targeted Gateway/config/auth/chat tests.
|
||||
- Run Python compile checks and the complete backend test suite.
|
||||
- Scan tracked files and diffs for real `userKey` values.
|
||||
|
||||
Reference in New Issue
Block a user