docs: define shared gateway config and user provisioning
This commit is contained in:
@ -1,83 +1,75 @@
|
|||||||
# Memory Gateway Package Migration Design
|
# Memory Gateway Package and User Provisioning Design
|
||||||
|
|
||||||
## Goal
|
## Goal
|
||||||
|
|
||||||
Move Beaver's Memory Gateway source code and typed configuration into the
|
Reorganize Beaver's Memory Gateway code under the `beaver.memory` domain and
|
||||||
existing `beaver.memory` domain package without changing runtime behavior.
|
replace the single fixed Gateway identity with per-Beaver-user credentials.
|
||||||
|
|
||||||
This migration affects Python source organization only. It does not move or
|
The final model has two independent configuration layers:
|
||||||
modify runtime data under `app-instance/backend/memory/`, the standalone
|
|
||||||
`/home/tom/memory-gateway` service, or Beaver instance configuration files.
|
|
||||||
|
|
||||||
## Target Structure
|
- One shared, non-secret Memory Gateway configuration used by every Beaver
|
||||||
|
instance.
|
||||||
|
- One per-instance credential file containing the Gateway identities created
|
||||||
|
for Beaver frontend users.
|
||||||
|
|
||||||
|
Curated memory remains enabled and isolated. Gateway failures or missing user
|
||||||
|
credentials must not modify `MEMORY.md`, `USER.md`, or the `memory` tool.
|
||||||
|
|
||||||
|
## Source Package
|
||||||
|
|
||||||
|
All Beaver-side Gateway source moves to:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
app-instance/backend/beaver/memory/gateway/
|
app-instance/backend/beaver/memory/gateway/
|
||||||
├── __init__.py
|
├── __init__.py
|
||||||
├── config.py
|
├── config.py
|
||||||
├── client.py
|
├── client.py
|
||||||
|
├── credentials.py
|
||||||
└── service.py
|
└── service.py
|
||||||
```
|
```
|
||||||
|
|
||||||
- `config.py` owns `MemoryConfig` and `MemoryGatewayConfig`.
|
- `config.py` owns the shared typed Gateway configuration.
|
||||||
- `client.py` owns `MemoryGatewayClient` and
|
- `client.py` owns `MemoryGatewayClient` and sanitized client exceptions.
|
||||||
`MemoryGatewayClientError`.
|
- `credentials.py` owns typed user credentials and atomic credential-file
|
||||||
- `service.py` owns `MemoryGatewayService`, `GatewayRecallOutcome`, and
|
persistence.
|
||||||
`GatewayPersistOutcome`.
|
- `service.py` owns search/add/flush orchestration and result types.
|
||||||
- `__init__.py` exposes the public Gateway API used by the loader and tests.
|
- `__init__.py` exposes the supported public Gateway API.
|
||||||
|
|
||||||
## Source Changes
|
Remove the old source locations:
|
||||||
|
|
||||||
- Remove `beaver/integrations/memory_gateway/`.
|
- `beaver/integrations/memory_gateway/`
|
||||||
- Remove `beaver/services/memory_gateway_service.py`.
|
- `beaver/services/memory_gateway_service.py`
|
||||||
- Remove the lazy `MemoryGatewayService` export from `beaver.services`.
|
- Gateway configuration dataclasses in `beaver.foundation.config.schema`
|
||||||
- Move the Gateway configuration dataclasses out of
|
- The lazy `MemoryGatewayService` export from `beaver.services`
|
||||||
`beaver.foundation.config.schema`.
|
|
||||||
- Keep `beaver.foundation.config.loader` responsible for parsing the top-level
|
|
||||||
`memory` JSON section, importing the typed models from
|
|
||||||
`beaver.memory.gateway`.
|
|
||||||
- Update `EngineLoader`, tests, README references, and implementation-plan
|
|
||||||
source paths to use the new package.
|
|
||||||
- Do not retain compatibility import modules. After migration,
|
|
||||||
`beaver.memory.gateway` is the only supported source entry point.
|
|
||||||
|
|
||||||
## Configuration Location
|
No compatibility forwarding modules are retained. After migration,
|
||||||
|
`beaver.memory.gateway` is the only supported source entry point.
|
||||||
|
|
||||||
Memory remains configured in each Beaver instance's `config.json`, not inside
|
## Shared Configuration
|
||||||
the Python package or runtime memory data directory.
|
|
||||||
|
|
||||||
Configuration lookup order remains:
|
All Beaver instances read the same public Gateway configuration from:
|
||||||
|
|
||||||
1. `BEAVER_CONFIG_PATH`
|
|
||||||
2. `$BEAVER_HOME/config.json`
|
|
||||||
3. `<workspace>/.beaver/config.json`
|
|
||||||
4. `./.beaver/config.json`
|
|
||||||
|
|
||||||
For the Docker app instance, the container reads:
|
|
||||||
|
|
||||||
```text
|
```text
|
||||||
/root/.beaver/config.json
|
/home/tom/beaver_project/app-instance/backend/memory/config.json
|
||||||
```
|
```
|
||||||
|
|
||||||
The corresponding host file is:
|
Inside the app-instance image this is available as:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
app-instance/runtime/instances/<instance-slug>/beaver-home/config.json
|
/opt/app/backend/memory/config.json
|
||||||
```
|
```
|
||||||
|
|
||||||
The `memory` section remains unchanged:
|
The file contains no user credentials:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"memory": {
|
"memory": {
|
||||||
"mode": "hybrid",
|
"mode": "hybrid",
|
||||||
"gateway": {
|
"gateway": {
|
||||||
"baseUrl": "http://172.19.0.1:8010",
|
"baseUrl": "http://172.19.207.37:8010",
|
||||||
"userId": "gateway_test_user",
|
|
||||||
"userKey": "uk_xxx",
|
|
||||||
"appId": "default",
|
"appId": "default",
|
||||||
"projectId": "default",
|
"projectId": "default",
|
||||||
"scope": ["current_chat", "resources"],
|
"scope": ["current_chat", "resources", "all_user_memory"],
|
||||||
"topK": 8,
|
"topK": 8,
|
||||||
"timeoutSeconds": 10
|
"timeoutSeconds": 10
|
||||||
}
|
}
|
||||||
@ -85,20 +77,206 @@ The `memory` section remains unchanged:
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
`userKey` remains a secret and must not be committed or logged.
|
Rules:
|
||||||
|
|
||||||
## Behavioral Guarantees
|
- Valid modes remain `curated` and `hybrid`.
|
||||||
|
- Curated memory is always initialized.
|
||||||
|
- `hybrid` enables Gateway only for runs with a resolved user credential.
|
||||||
|
- `baseUrl` is fixed to `http://172.19.207.37:8010` in the initial shared
|
||||||
|
configuration.
|
||||||
|
- Scope includes `current_chat`, `resources`, and `all_user_memory`.
|
||||||
|
- The shared file is the authoritative Memory Gateway configuration. Instance
|
||||||
|
`config.json` files continue to own providers, tools, channels, AuthZ, and
|
||||||
|
backend identity, but no longer carry Gateway user credentials.
|
||||||
|
- An optional `BEAVER_MEMORY_CONFIG_PATH` may override the shared file path for
|
||||||
|
tests or non-image development runs.
|
||||||
|
|
||||||
- Curated memory behavior is unchanged.
|
## Per-Instance User Credentials
|
||||||
- Hybrid search/add/flush request payloads and ordering are unchanged.
|
|
||||||
- Gateway audit events and best-effort failure semantics are unchanged.
|
|
||||||
- The standalone Memory Gateway deployment remains outside Beaver.
|
|
||||||
- `app-instance/backend/memory/` continues to contain runtime memory data only.
|
|
||||||
|
|
||||||
## Verification
|
Each Beaver instance stores Gateway user credentials alongside its existing
|
||||||
|
`config.json`, `runtime.env`, and `web_auth_users.json`:
|
||||||
|
|
||||||
- Update all imports and assert no references remain to the removed modules.
|
```text
|
||||||
- Run Gateway configuration, client/service, loader, context, and AgentLoop
|
app-instance/runtime/instances/<instance-slug>/beaver-home/
|
||||||
tests.
|
├── config.json
|
||||||
- Run the complete backend test suite and Python compile check.
|
├── runtime.env
|
||||||
- Scan tracked diffs for credential values.
|
├── web_auth_users.json
|
||||||
|
└── memory_gateway_users.json
|
||||||
|
```
|
||||||
|
|
||||||
|
The existing `beaver-home` mount exposes the file inside the container as:
|
||||||
|
|
||||||
|
```text
|
||||||
|
/root/.beaver/memory_gateway_users.json
|
||||||
|
```
|
||||||
|
|
||||||
|
The JSON format is:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"users": {
|
||||||
|
"tom": {
|
||||||
|
"userId": "tom",
|
||||||
|
"userKey": "uk_xxx"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- The map key is the authenticated Beaver login username.
|
||||||
|
- Gateway `userId` is exactly the Beaver login username, with no prefix.
|
||||||
|
- `userKey` is secret and must never appear in API responses, logs, audit
|
||||||
|
events, exceptions, or tracked configuration.
|
||||||
|
- Writes use a sibling temporary file followed by atomic replace.
|
||||||
|
- The credential file is created with mode `0600`.
|
||||||
|
- `BEAVER_MEMORY_GATEWAY_USERS_PATH` may override the default path for tests.
|
||||||
|
|
||||||
|
## Frontend User Provisioning
|
||||||
|
|
||||||
|
The frontend continues to call Beaver's existing `POST /api/auth/register`
|
||||||
|
endpoint. The browser never calls Memory Gateway directly and never receives
|
||||||
|
the Gateway `userKey`.
|
||||||
|
|
||||||
|
For a registration request with username `tom`, Beaver performs:
|
||||||
|
|
||||||
|
```http
|
||||||
|
POST http://172.19.207.37:8010/users
|
||||||
|
Content-Type: application/json
|
||||||
|
|
||||||
|
{"user_id":"tom"}
|
||||||
|
```
|
||||||
|
|
||||||
|
Beaver validates that the response contains non-empty `user_id` and
|
||||||
|
`user_key`, requires the returned `user_id` to equal `tom`, and stores the
|
||||||
|
credential under the `tom` entry in `memory_gateway_users.json`.
|
||||||
|
|
||||||
|
The Gateway `/users` API is treated as idempotent. Registering an existing
|
||||||
|
Beaver username may refresh the same credential entry without creating a
|
||||||
|
second local identity.
|
||||||
|
|
||||||
|
For this first version:
|
||||||
|
|
||||||
|
- Gateway provisioning has no Beaver-side retries.
|
||||||
|
- A Gateway provisioning failure does not roll back an otherwise valid Beaver
|
||||||
|
registration.
|
||||||
|
- A user without stored Gateway credentials continues with curated memory only.
|
||||||
|
- No separate repair UI or background credential provisioning job is added.
|
||||||
|
|
||||||
|
## Authenticated Chat Identity
|
||||||
|
|
||||||
|
Gateway credential selection must use a trusted server-side principal.
|
||||||
|
|
||||||
|
- REST and WebSocket frontend chat paths resolve the Beaver username from the
|
||||||
|
issued access token.
|
||||||
|
- The resolved username is passed separately into the agent runtime as the
|
||||||
|
Gateway identity key.
|
||||||
|
- Client-provided `user_id` fields do not select Gateway credentials and cannot
|
||||||
|
impersonate another Gateway user.
|
||||||
|
- Runs without an authenticated frontend username, including channel or
|
||||||
|
scheduled runs without a trusted mapped identity, continue with curated
|
||||||
|
memory only.
|
||||||
|
|
||||||
|
This identity key is runtime-only. It is not included in provider prompts or
|
||||||
|
Gateway persisted message content.
|
||||||
|
|
||||||
|
## Runtime Architecture
|
||||||
|
|
||||||
|
`EngineLoader` loads:
|
||||||
|
|
||||||
|
1. Curated `MemoryService`, unconditionally.
|
||||||
|
2. Shared `MemoryGatewayConfig` from `memory/config.json`.
|
||||||
|
3. A `MemoryGatewayCredentialStore` for the instance credential file.
|
||||||
|
|
||||||
|
It does not construct one fixed-user `MemoryGatewayService` at startup.
|
||||||
|
|
||||||
|
For each authenticated run in hybrid mode:
|
||||||
|
|
||||||
|
1. `AgentLoop` receives the trusted Beaver username.
|
||||||
|
2. It reads that username's credential from the credential store.
|
||||||
|
3. If a credential exists, it constructs a run-local Gateway service/client
|
||||||
|
from the shared config and that credential.
|
||||||
|
4. It performs Gateway recall before context construction.
|
||||||
|
5. It performs Gateway add and flush after normal completion.
|
||||||
|
|
||||||
|
The run-local service has no shared mutable credential state, so concurrent
|
||||||
|
runs for different users cannot exchange identities. No service cache is added
|
||||||
|
in this version.
|
||||||
|
|
||||||
|
## Recall and Persistence
|
||||||
|
|
||||||
|
The existing hybrid behavior remains unchanged once a user credential has
|
||||||
|
been resolved:
|
||||||
|
|
||||||
|
- Search uses the current Beaver session id, current prompt, configured top K,
|
||||||
|
and all three configured scopes.
|
||||||
|
- Sanitized Gateway results are injected as one ephemeral untrusted-reference
|
||||||
|
message outside the system prompt.
|
||||||
|
- Normal completion persists exactly the original current user prompt and final
|
||||||
|
assistant text.
|
||||||
|
- Add is called once, followed by flush once only after add succeeds.
|
||||||
|
- Tool calls, tool results, system prompts, curated memory, recalled Gateway
|
||||||
|
text, reasoning, and skills are not persisted to Gateway.
|
||||||
|
- Gateway and curated memory remain isolated and do not synchronize, merge,
|
||||||
|
overwrite, or deduplicate each other.
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
- The shared configuration is safe to track because it contains no `userKey`.
|
||||||
|
- Per-user credentials live only under ignored instance runtime data.
|
||||||
|
- Credential-file permissions are `0600`.
|
||||||
|
- Credential objects suppress secrets from `repr`.
|
||||||
|
- Gateway client exceptions contain only operation, category, path, and status
|
||||||
|
metadata.
|
||||||
|
- Registration responses expose Beaver authentication data only; Gateway
|
||||||
|
credentials remain server-side.
|
||||||
|
- Hidden Gateway audit events may include the Beaver/Gateway user id but never
|
||||||
|
the user key or complete request/response body.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
### Package migration
|
||||||
|
|
||||||
|
- All imports use `beaver.memory.gateway`.
|
||||||
|
- No references remain to the removed integration/service modules.
|
||||||
|
- Gateway config, client, service, and credential-store tests remain isolated
|
||||||
|
from curated memory.
|
||||||
|
|
||||||
|
### Shared configuration
|
||||||
|
|
||||||
|
- The shared file parses the fixed URL and three scopes.
|
||||||
|
- Invalid mode, URL, scope, top K, or timeout fails with sanitized errors.
|
||||||
|
- Instance config loading remains unchanged for non-memory settings.
|
||||||
|
- Test overrides can select a temporary shared config file.
|
||||||
|
|
||||||
|
### Credential persistence
|
||||||
|
|
||||||
|
- Missing files produce an empty credential map.
|
||||||
|
- Credentials round-trip by Beaver username.
|
||||||
|
- Updating one user preserves all other users.
|
||||||
|
- Files are atomically replaced and have mode `0600`.
|
||||||
|
- No exception or representation contains `userKey`.
|
||||||
|
|
||||||
|
### Registration
|
||||||
|
|
||||||
|
- New frontend registration calls `/users` with the Beaver username.
|
||||||
|
- Valid Gateway responses are stored without returning the key to the browser.
|
||||||
|
- Existing usernames refresh the same credential entry.
|
||||||
|
- Provisioning failure does not roll back Beaver registration and stores no
|
||||||
|
partial credential.
|
||||||
|
|
||||||
|
### Agent runtime
|
||||||
|
|
||||||
|
- Authenticated username selects only its own Gateway credential.
|
||||||
|
- Client-provided `user_id` cannot select another user's credential.
|
||||||
|
- Concurrent users construct independent run-local Gateway services.
|
||||||
|
- Missing credentials perform no Gateway calls and preserve curated behavior.
|
||||||
|
- Existing recall/add/flush ordering, payload, audit, and failure tests remain
|
||||||
|
valid.
|
||||||
|
|
||||||
|
### Verification
|
||||||
|
|
||||||
|
- Run targeted Gateway/config/auth/chat tests.
|
||||||
|
- Run Python compile checks and the complete backend test suite.
|
||||||
|
- Scan tracked files and diffs for real `userKey` values.
|
||||||
|
|||||||
Reference in New Issue
Block a user