Compare commits
5 Commits
a29009dc07
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| e5cd87789f | |||
| 15462a95cb | |||
| 8fcf0941c0 | |||
| 42de7f9da0 | |||
| 126ae4eafa |
18
.env.example
18
.env.example
@ -1,13 +1,13 @@
|
||||
# EverOS HTTP server used by the gateway client.
|
||||
EVEROS_BASE_URL=http://127.0.0.1:1995
|
||||
# Upstream memory service used by the gateway client.
|
||||
MEMORY_GATEWAY_BACKEND_BASE_URL=http://127.0.0.1:1995
|
||||
|
||||
# Gateway-owned SQLite database. This does not point at EverOS internal storage.
|
||||
# Gateway-owned SQLite database. This does not point at upstream internal storage.
|
||||
MEMORY_GATEWAY_DB_PATH=./data/memory_gateway.sqlite3
|
||||
|
||||
# Raw uploaded files are stored here before being passed to EverOS by file URI.
|
||||
# Raw uploaded files are stored here for gateway-managed ingestion.
|
||||
MEMORY_GATEWAY_STORAGE_DIR=./data/storage
|
||||
|
||||
# Number of resource session IDs sent per EverOS search request.
|
||||
# Number of resource session IDs sent per upstream search request.
|
||||
MEMORY_GATEWAY_RESOURCE_SEARCH_BATCH_SIZE=50
|
||||
|
||||
# Max upload size in bytes. Default here is 25 MiB.
|
||||
@ -16,10 +16,10 @@ MEMORY_GATEWAY_MAX_UPLOAD_BYTES=26214400
|
||||
# Comma-separated MIME allowlist. Prefix wildcards such as image/* are supported.
|
||||
MEMORY_GATEWAY_ALLOWED_MIME_TYPES=image/*,audio/*,application/pdf,text/html,application/xhtml+xml,text/plain,text/markdown,text/csv,application/json,application/msword,application/vnd.openxmlformats-officedocument.wordprocessingml.document,application/vnd.ms-excel,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,application/vnd.ms-powerpoint,application/vnd.openxmlformats-officedocument.presentationml.presentation
|
||||
|
||||
# EverOS add/flush retry policy during resource ingestion.
|
||||
MEMORY_GATEWAY_EVEROS_INGEST_ATTEMPTS=3
|
||||
MEMORY_GATEWAY_EVEROS_RETRY_DELAY_SECONDS=0.25
|
||||
MEMORY_GATEWAY_EVEROS_TIMEOUT_SECONDS=120
|
||||
# Upstream add/flush retry policy during resource ingestion.
|
||||
MEMORY_GATEWAY_BACKEND_INGEST_ATTEMPTS=3
|
||||
MEMORY_GATEWAY_BACKEND_RETRY_DELAY_SECONDS=0.25
|
||||
MEMORY_GATEWAY_BACKEND_TIMEOUT_SECONDS=120
|
||||
|
||||
# API server settings used by python main.py.
|
||||
MEMORY_GATEWAY_HOST=0.0.0.0
|
||||
|
||||
2
.gitignore
vendored
2
.gitignore
vendored
@ -1,6 +1,6 @@
|
||||
# Local environment files
|
||||
.env
|
||||
everos.env
|
||||
backend.env
|
||||
*.env.local
|
||||
|
||||
# Gateway runtime data
|
||||
|
||||
168
README.md
168
README.md
@ -1,18 +1,18 @@
|
||||
# Memory Gateway
|
||||
|
||||
Memory Gateway 是一个轻量级 FastAPI 服务，用于在 EverOS 现有
|
||||
Memory Gateway 是一个轻量级 FastAPI 服务，用于在上游记忆服务现有
|
||||
`/api/v1/memory/add`、`/api/v1/memory/flush`、`/api/v1/memory/search`
|
||||
能力之上构建用户资源记忆层。
|
||||
|
||||
它只维护 Gateway 自己的 SQLite 元数据表、软删除记录和手动覆盖记录，
|
||||
不会直接修改 EverOS 的 Markdown、SQLite 或 LanceDB 内部文件。
|
||||
不会直接修改上游记忆服务的 Markdown、SQLite 或 LanceDB 内部文件。
|
||||
|
||||
## 功能范围
|
||||
|
||||
- 上传用户资源：文件、图片、音频、PDF、HTML、普通文档、纯文本。
|
||||
- 保存资源元数据到 SQLite。
|
||||
- 为每个资源生成独立 EverOS `session_id`。
|
||||
- 调用 EverOS `add` 和 `flush` 完成资源记忆摄入，并对临时失败做轻量重试。
|
||||
- 为每个资源生成独立的上游记忆服务 `session_id`。
|
||||
- 调用上游记忆服务的 `add` 和 `flush` 完成资源记忆摄入，并对临时失败做轻量重试。
|
||||
- 提供资源列表、详情、软删除。
|
||||
- 支持上传大小限制、MIME 白名单、同用户同 app/project 下按 sha256 幂等复用资源。
|
||||
- 编排记忆搜索，支持当前聊天、资源记忆、全部用户记忆。
|
||||
@ -28,7 +28,7 @@ Memory Gateway 是一个轻量级 FastAPI 服务，用于在 EverOS 现有
|
||||
│ ├── api.py # FastAPI 路由
|
||||
│ ├── config.py # 环境变量配置
|
||||
│ ├── db.py # SQLite schema 初始化
|
||||
│ ├── everos_client.py # EverOS HTTP client
|
||||
│ ├── backend_client.py # 上游记忆服务 HTTP client
|
||||
│ ├── repository.py # SQLite 读写
|
||||
│ └── service.py # 业务编排
|
||||
├── main.py # Python 启动入口
|
||||
@ -50,21 +50,21 @@ cp .env.example .env
|
||||
|
||||
| 变量 | 默认值 | 说明 |
|
||||
|---|---|---|
|
||||
| `EVEROS_BASE_URL` | `http://127.0.0.1:1995` | EverOS API 服务地址；EverOS 可监听 `0.0.0.0:1995`，本机客户端通常连接 `127.0.0.1:1995` |
|
||||
| `MEMORY_GATEWAY_BACKEND_BASE_URL` | `http://127.0.0.1:1995` | 上游记忆服务 API 地址；服务端可监听 `0.0.0.0:1995`，本机客户端通常连接 `127.0.0.1:1995` |
|
||||
| `MEMORY_GATEWAY_DB_PATH` | `./data/memory_gateway.sqlite3` | Gateway 自己的 SQLite 数据库路径 |
|
||||
| `MEMORY_GATEWAY_STORAGE_DIR` | `./data/storage` | 用户上传原始文件保存路径 |
|
||||
| `MEMORY_GATEWAY_RESOURCE_SEARCH_BATCH_SIZE` | `50` | resources scope 搜索时每批 session_id 数量 |
|
||||
| `MEMORY_GATEWAY_MAX_UPLOAD_BYTES` | `26214400` | 单个上传文件最大字节数，默认 25MB |
|
||||
| `MEMORY_GATEWAY_ALLOWED_MIME_TYPES` | 常见图片、音频、PDF、HTML、文本和 Office 文档 | 逗号分隔的上传 MIME 白名单，支持 `image/*` 这类前缀匹配 |
|
||||
| `MEMORY_GATEWAY_EVEROS_INGEST_ATTEMPTS` | `3` | EverOS `add` 和 `flush` 各自最多重试次数 |
|
||||
| `MEMORY_GATEWAY_EVEROS_RETRY_DELAY_SECONDS` | `0.25` | EverOS 摄入重试间隔秒数 |
|
||||
| `MEMORY_GATEWAY_EVEROS_TIMEOUT_SECONDS` | `120` | 单次 EverOS HTTP 请求超时秒数 |
|
||||
| `MEMORY_GATEWAY_BACKEND_INGEST_ATTEMPTS` | `3` | 上游记忆服务 `add` 和 `flush` 各自最多重试次数 |
|
||||
| `MEMORY_GATEWAY_BACKEND_RETRY_DELAY_SECONDS` | `0.25` | 上游记忆服务摄入重试间隔秒数 |
|
||||
| `MEMORY_GATEWAY_BACKEND_TIMEOUT_SECONDS` | `120` | 单次上游记忆服务 HTTP 请求超时秒数 |
|
||||
| `MEMORY_GATEWAY_HOST` | `127.0.0.1` | Gateway API 监听地址 |
|
||||
| `MEMORY_GATEWAY_PORT` | `8010` | Gateway API 监听端口 |
|
||||
| `MEMORY_GATEWAY_RELOAD` | `false` | 是否启用 uvicorn reload，开发时可设为 `true` |
|
||||
|
||||
注意：`MEMORY_GATEWAY_DB_PATH` 和 `MEMORY_GATEWAY_STORAGE_DIR` 是 Gateway
|
||||
自己的存储位置，不要配置成 EverOS 的内部存储目录。
|
||||
自己的存储位置，不要配置成上游记忆服务的内部存储目录。
|
||||
|
||||
## 安装依赖
|
||||
|
||||
@ -133,7 +133,7 @@ Gateway 会通过 `memory_gateway.api` logger 为每个 API 请求输出一条 J
|
||||
GET /health
|
||||
```
|
||||
|
||||
该接口不需要 `user_id` 或 `user_key`，用于确认 Gateway API 是否可响应，以及上游 EverOS 是否可访问。
|
||||
该接口不需要 `user_id` 或 `user_key`，用于确认 Gateway API 是否可响应，以及上游记忆服务是否可访问。
|
||||
|
||||
请求示例：
|
||||
|
||||
@ -141,7 +141,7 @@ GET /health
|
||||
curl http://127.0.0.1:8010/health
|
||||
```
|
||||
|
||||
EverOS 正常时响应示例：
|
||||
上游记忆服务正常时响应示例：
|
||||
|
||||
```json
|
||||
{
|
||||
@ -149,7 +149,7 @@ EverOS 正常时响应示例：
|
||||
"api": {
|
||||
"status": "ok"
|
||||
},
|
||||
"everos": {
|
||||
"backend": {
|
||||
"status": "ok",
|
||||
"base_url": "http://127.0.0.1:1995",
|
||||
"data": {
|
||||
@ -159,7 +159,7 @@ EverOS 正常时响应示例：
|
||||
}
|
||||
```
|
||||
|
||||
EverOS 不可访问时仍返回 HTTP 200，但 `status` 会变成 `degraded`，便于区分“Gateway API 活着”和“上游 EverOS 故障”：
|
||||
上游记忆服务不可访问时仍返回 HTTP 200，但 `status` 会变成 `degraded`，便于区分“Gateway API 活着”和“上游记忆服务故障”：
|
||||
|
||||
```json
|
||||
{
|
||||
@ -167,7 +167,7 @@ EverOS 不可访问时仍返回 HTTP 200，但 `status` 会变成 `degraded`，
|
||||
"api": {
|
||||
"status": "ok"
|
||||
},
|
||||
"everos": {
|
||||
"backend": {
|
||||
"status": "unavailable",
|
||||
"base_url": "http://127.0.0.1:1995",
|
||||
"error": "Connection refused"
|
||||
@ -221,8 +221,8 @@ Content-Type: multipart/form-data
|
||||
|---|---|---|---|---|
|
||||
| `user_id` | string | 是 | 无 | 用户 ID |
|
||||
| `user_key` | string | 是 | 无 | 用户 key |
|
||||
| `app_id` | string | 否 | `default` | EverOS app scope |
|
||||
| `project_id` | string | 否 | `default` | EverOS project scope |
|
||||
| `app_id` | string | 否 | `default` | 上游记忆服务 app scope |
|
||||
| `project_id` | string | 否 | `default` | 上游记忆服务 project scope |
|
||||
| `file` | file | 是 | 无 | 上传资源文件 |
|
||||
| `title` | string | 否 | `null` | 资源标题 |
|
||||
| `description` | string | 否 | `null` | 资源描述 |
|
||||
@ -234,22 +234,22 @@ Content-Type: multipart/form-data
|
||||
3. 生成 `resource_id`。
|
||||
4. 生成 `session_id = resource:{user_id}:{resource_id}`。
|
||||
5. 写入 `user_resources`，状态为 `ingesting`。
|
||||
6. 根据 MIME 类型映射 EverOS content type。
|
||||
7. 构造 EverOS content item：文本类上传以内联 `text` 发送，非文本上传以内联 `base64` 发送，不要求 EverOS 访问 Gateway 本地 `file://` 路径。
|
||||
8. 调用 EverOS `/api/v1/memory/add`。
|
||||
9. 调用 EverOS `/api/v1/memory/flush`。
|
||||
6. 根据 MIME 类型映射上游记忆服务 content type。
|
||||
7. 构造上游记忆服务 content item：文本类上传以内联 `text` 发送，非文本上传以内联 `base64` 发送，不要求上游记忆服务访问 Gateway 本地 `file://` 路径。
|
||||
8. 调用上游记忆服务的 `/api/v1/memory/add`。
|
||||
9. 调用上游记忆服务的 `/api/v1/memory/flush`。
|
||||
10. 成功后状态改为 `extracted`，失败后状态改为 `failed`。
|
||||
|
||||
上传策略：
|
||||
|
||||
- 文件会按流式方式写入磁盘，超过 `MEMORY_GATEWAY_MAX_UPLOAD_BYTES` 会返回 `413`，不会写入资源记录。
|
||||
- MIME 类型不在 `MEMORY_GATEWAY_ALLOWED_MIME_TYPES` 白名单内会返回 `415`。
|
||||
- 同一用户在同一 `app_id`、`project_id` 下重复上传相同 sha256 的活跃资源，会直接返回已有资源，避免重复调用 EverOS 摄入。
|
||||
- EverOS `add` 和 `flush` 临时失败时会分别按配置重试；单次请求受 `MEMORY_GATEWAY_EVEROS_TIMEOUT_SECONDS` 控制；全部失败后资源状态为 `failed`，并记录 `error_message`。
|
||||
- 同一用户在同一 `app_id`、`project_id` 下重复上传相同 sha256 的活跃资源，会直接返回已有资源，避免重复调用上游记忆服务摄入。
|
||||
- 上游记忆服务的 `add` 和 `flush` 临时失败时会分别按配置重试；单次请求受 `MEMORY_GATEWAY_BACKEND_TIMEOUT_SECONDS` 控制；全部失败后资源状态为 `failed`，并记录 `error_message`。
|
||||
|
||||
content type 映射：
|
||||
|
||||
| 文件类型 | EverOS content type |
|
||||
| 文件类型 | 上游记忆服务 content type |
|
||||
|---|---|
|
||||
| `image/*` | `image` |
|
||||
| `audio/*` | `audio` |
|
||||
@ -282,7 +282,8 @@ curl -X POST http://127.0.0.1:8010/resources \
|
||||
}
|
||||
```
|
||||
|
||||
对外返回的 `uri` 永远是 `resource://{user_id}/{resource_id}`，不会泄露内部 `file://` 路径。
|
||||
资源上传接口返回的 `uri` 始终是 `resource://{user_id}/{resource_id}`。按文件名
|
||||
命中的记忆搜索结果会另外通过 `attachments[].internal_uri` 返回真实 URI。
|
||||
|
||||
### 4. 查询资源列表
|
||||
|
||||
@ -406,7 +407,7 @@ DELETE /resources/{resource_id}?user_id={user_id}&user_key={user_key}
|
||||
- 设置 `status = deleted`。
|
||||
- 后续 `resources` scope 搜索会排除该资源的 `session_id`。
|
||||
- 清理 Gateway 自己在 `MEMORY_GATEWAY_STORAGE_DIR` 下保存的原始上传文件。
|
||||
- 不物理删除 EverOS 内部记忆或索引。
|
||||
- 不物理删除上游记忆服务内部记忆或索引。
|
||||
|
||||
请求示例：
|
||||
|
||||
@ -438,12 +439,18 @@ Content-Type: application/json
|
||||
|---|---|---|---|---|
|
||||
| `user_id` | string | 是 | 无 | 用户 ID |
|
||||
| `user_key` | string | 是 | 无 | 用户 key |
|
||||
| `agent_id` | string | 否 | `null` | 设置后查询该 agent 的记忆；`user_id` 仍用于 Gateway 鉴权和本地数据隔离 |
|
||||
| `query` | string | 是 | 无 | 搜索问题 |
|
||||
| `conversation_id` | string | 否 | `null` | `scope` 包含 `current_chat` 时使用 |
|
||||
| `scope` | string[] | 否 | `["current_chat", "resources"]` | 搜索范围 |
|
||||
| `top_k` | integer | 否 | `8` | 每次 EverOS 搜索返回数量，范围 `1..100` |
|
||||
| `app_id` | string | 否 | `default` | EverOS app scope |
|
||||
| `project_id` | string | 否 | `default` | EverOS project scope |
|
||||
| `method` | string | 否 | `hybrid` | 搜索方法：`keyword`、`vector`、`hybrid` 或 `agentic` |
|
||||
| `top_k` | integer | 否 | `8` | 每次上游搜索返回数量，支持 `-1` 或 `1..100`；`-1` 表示使用上游默认数量 |
|
||||
| `radius` | number | 否 | `null` | 相似度半径，范围 `0..1`；未提供时不发送给上游 |
|
||||
| `include_profile` | boolean | 否 | `true` | 是否同时获取用户 profile；agent 查询由上游决定是否忽略该参数 |
|
||||
| `enable_llm_rerank` | boolean | 否 | `true` | 是否启用上游 LLM rerank；具体生效范围由搜索方法和记忆类型决定 |
|
||||
| `filters` | object | 否 | `null` | 上游过滤 DSL，支持字段条件以及嵌套 `AND`、`OR` |
|
||||
| `app_id` | string | 否 | `default` | 上游记忆服务 app scope |
|
||||
| `project_id` | string | 否 | `default` | 上游记忆服务 project scope |
|
||||
|
||||
`scope` 支持：
|
||||
|
||||
@ -464,7 +471,13 @@ curl -X POST http://127.0.0.1:8010/memories/search \
|
||||
"conversation_id": "c_456",
|
||||
"query": "合同里的付款条款是什么？",
|
||||
"scope": ["current_chat", "resources"],
|
||||
"method": "hybrid",
|
||||
"top_k": 8,
|
||||
"include_profile": true,
|
||||
"enable_llm_rerank": true,
|
||||
"filters": {
|
||||
"type": "Conversation"
|
||||
},
|
||||
"app_id": "default",
|
||||
"project_id": "default"
|
||||
}'
|
||||
@ -472,12 +485,42 @@ curl -X POST http://127.0.0.1:8010/memories/search \
|
||||
|
||||
搜索编排逻辑：
|
||||
|
||||
1. `current_chat`：调用 EverOS search，过滤 `filters.session_id = chat:{conversation_id}`。
|
||||
2. `resources`：先查当前用户的 `user_resources`，只取 `status = extracted` 且未删除资源；再按批次调用 EverOS search，过滤这些资源的 `session_id`。
|
||||
3. `all_user_memory`：调用 EverOS search，不加 `session_id` 过滤。
|
||||
4. 合并结果。
|
||||
5. 过滤 `memory_tombstones` 命中的 `memory_id` 或 `session_id`。
|
||||
6. 应用 active `memory_overrides`，把 `text` 替换为 `override_text`。
|
||||
1. `current_chat`：调用上游记忆服务 search，过滤 `filters.session_id = chat:{conversation_id}`。
|
||||
2. `resources`：先查当前用户的 `user_resources`，只取 `status = extracted` 且未删除资源；再按批次调用上游记忆服务 search，过滤这些资源的 `session_id`。
|
||||
3. `all_user_memory`：调用上游记忆服务 search，不加 `session_id` 过滤。
|
||||
4. 同时存在请求 `filters` 和 scope 生成的 session 条件时，使用 `{"AND": [filters, scope_filters]}` 合并，避免调用方过滤条件覆盖资源或聊天隔离条件。
|
||||
5. 设置 `agent_id` 时，上游请求只发送 `agent_id`；否则发送已鉴权的 `user_id`。
|
||||
6. 合并结果。
|
||||
7. 过滤 `memory_tombstones` 命中的 `memory_id` 或 `session_id`。
|
||||
8. 应用 active `memory_overrides`，把 `text` 替换为 `override_text`。
|
||||
|
||||
响应中的 `memory_type` 对应上游结果类型：
|
||||
|
||||
| `memory_type` | 说明 |
|
||||
|---|---|
|
||||
| `episode` | 用户 episode |
|
||||
| `profile` | 用户 profile |
|
||||
| `agent_case` | agent case |
|
||||
| `agent_skill` | agent skill |
|
||||
| `unprocessed_message` | 尚未完成边界提取的原始消息 |
|
||||
|
||||
附件路径映射规则：
|
||||
|
||||
1. `/resources` 上传成功后，将资源真实 URI 与资源 session 写入
|
||||
`memory_attachments`。数据库初始化会自动回填已有 `user_resources`。
|
||||
2. `/memories/add` 中含 `uri` 的 content item 会直接登记 URI。
|
||||
3. `/memories/add` 中只有 `base64` 的 content item 会保存到
|
||||
`MEMORY_GATEWAY_STORAGE_DIR/{user_id}/memory_attachments/{sha256}/`，再登记
|
||||
生成的 `file://` URI。相同用户、session、文件名和内容的重试会复用路径。
|
||||
4. 搜索时根据当前用户和结果 `session_id` 查询附件，递归检查 `raw` 中的字符串
|
||||
值。只有完整文件名出现时才返回对应附件；匹配不区分大小写。
|
||||
5. `raw` 中键名为 `base64` 的内容不会参与匹配。未匹配时返回
|
||||
`"attachments": []`。
|
||||
6. 历史 `/memories/add` 请求未保存在 Gateway 数据库中，无法自动补录映射；新
|
||||
版本上线后的请求会建立映射。
|
||||
|
||||
`attachments[].internal_uri` 会按配置和调用方输入直接暴露服务器真实 URI，调用
|
||||
该接口的客户端必须被视为可信客户端。
|
||||
|
||||
响应示例：
|
||||
|
||||
@ -486,16 +529,24 @@ curl -X POST http://127.0.0.1:8010/memories/search \
|
||||
"results": [
|
||||
{
|
||||
"id": "mem_abc",
|
||||
"memory_type": "episode",
|
||||
"session_id": "resource:u_123:r_xxx",
|
||||
"text": "付款期限为收到发票后 30 天内。",
|
||||
"score": 0.82,
|
||||
"source_scope": "resources",
|
||||
"resource_id": "r_xxx",
|
||||
"resource_uri": "resource://u_123/r_xxx",
|
||||
"attachments": [
|
||||
{
|
||||
"type": "pdf",
|
||||
"name": "contract.pdf",
|
||||
"internal_uri": "file:///srv/memory-gateway/u_123/r_xxx/contract.pdf"
|
||||
}
|
||||
],
|
||||
"raw": {
|
||||
"id": "mem_abc",
|
||||
"session_id": "resource:u_123:r_xxx",
|
||||
"episode": "原始 EverOS 返回内容"
|
||||
"episode": "原始上游记忆服务返回内容"
|
||||
}
|
||||
}
|
||||
]
|
||||
@ -518,7 +569,7 @@ Content-Type: application/json
|
||||
| `session_id` | string | 是 | memory 所属 session，必须属于当前用户 |
|
||||
| `override_text` | string | 是 | 修正后的记忆文本 |
|
||||
|
||||
该接口只写入或更新 `memory_overrides`，不会修改 EverOS 原始文件。写入前会校验 `session_id` 属于当前用户：当前版本支持当前用户的 `resource:{user_id}:{resource_id}` 和 `memory_edit:{user_id}`。后续搜索结果命中该 `memory_id` 时，返回的 `text` 会替换为 `override_text`，但保留原始 memory id。
|
||||
该接口只写入或更新 `memory_overrides`，不会修改上游记忆服务原始文件。写入前会校验 `session_id` 属于当前用户：当前版本支持当前用户的 `resource:{user_id}:{resource_id}` 和 `memory_edit:{user_id}`。后续搜索结果命中该 `memory_id` 时，返回的 `text` 会替换为 `override_text`，但保留原始 memory id。
|
||||
|
||||
请求示例：
|
||||
|
||||
@ -559,7 +610,7 @@ Content-Type: application/json
|
||||
| `session_id` | string | 是 | memory 所属 session，必须属于当前用户 |
|
||||
| `reason` | string | 否 | 删除原因 |
|
||||
|
||||
该接口只写入 `memory_tombstones`，不会修改 EverOS 原始文件。写入前会校验 `session_id` 属于当前用户：当前版本支持当前用户的 `resource:{user_id}:{resource_id}` 和 `memory_edit:{user_id}`。后续搜索结果如果命中 tombstone 的 `memory_id` 或 `session_id`，会被过滤。
|
||||
该接口只写入 `memory_tombstones`，不会修改上游记忆服务原始文件。写入前会校验 `session_id` 属于当前用户：当前版本支持当前用户的 `resource:{user_id}:{resource_id}` 和 `memory_edit:{user_id}`。后续搜索结果如果命中 tombstone 的 `memory_id` 或 `session_id`，会被过滤。
|
||||
|
||||
请求示例：
|
||||
|
||||
@ -584,15 +635,34 @@ curl -X DELETE http://127.0.0.1:8010/memories/mem_abc \
|
||||
}
|
||||
```
|
||||
|
||||
## EverOS client 封装
|
||||
## 上游记忆服务客户端封装
|
||||
|
||||
Gateway 内部通过 `core/everos_client.py` 调用 EverOS：
|
||||
Gateway 内部通过 `core/backend_client.py` 调用上游记忆服务：
|
||||
|
||||
- `add_memory(payload)` -> `POST /api/v1/memory/add`
|
||||
- `flush_memory(session_id, app_id, project_id)` -> `POST /api/v1/memory/flush`
|
||||
- `search_memory(payload)` -> `POST /api/v1/memory/search`
|
||||
- `health_check()` -> `GET /health`
|
||||
|
||||
## AI Agent Skill
|
||||
|
||||
项目提供可供 AI Agent 使用的 Skill：
|
||||
|
||||
```text
|
||||
skill/memory-gateway-agent
|
||||
```
|
||||
|
||||
其中 `SKILL.md` 定义 Agent 工作流，`scripts/memory_gateway.py` 提供无额外依赖的命令行客户端，`references/api.md` 提供完整参数说明。使用前设置：
|
||||
|
||||
```bash
|
||||
export MEMORY_GATEWAY_BASE_URL=http://127.0.0.1:8010
|
||||
export MEMORY_GATEWAY_USER_ID=u_123
|
||||
export MEMORY_GATEWAY_USER_KEY=uk_xxx
|
||||
|
||||
python skill/memory-gateway-agent/scripts/memory_gateway.py health
|
||||
python skill/memory-gateway-agent/scripts/memory_gateway.py list-resources
|
||||
```
|
||||
|
||||
## 运行测试
|
||||
|
||||
```bash
|
||||
@ -600,21 +670,21 @@ cd /home/tom/memory-gateway
|
||||
.venv/bin/python -B -m pytest -q -p no:cacheprovider
|
||||
```
|
||||
|
||||
默认测试不会访问真实 EverOS。若要对已部署的 EverOS 做 health 集成验证，先确认 EverOS 正在监听 `0.0.0.0:1995`，然后从 Gateway 所在机器用客户端可访问地址访问：
|
||||
默认测试不会访问真实上游记忆服务。若要对已部署的上游记忆服务做 health 集成验证，先确认上游记忆服务正在监听 `0.0.0.0:1995`，然后从 Gateway 所在机器用客户端可访问地址访问：
|
||||
|
||||
```bash
|
||||
cd /home/tom/memory-gateway
|
||||
RUN_EVEROS_INTEGRATION=1 \
|
||||
EVEROS_BASE_URL=http://10.6.80.123:1995 \
|
||||
.venv/bin/python -B -m pytest -q tests/test_everos_integration.py -p no:cacheprovider
|
||||
RUN_BACKEND_INTEGRATION=1 \
|
||||
MEMORY_GATEWAY_BACKEND_BASE_URL=http://10.6.80.123:1995 \
|
||||
.venv/bin/python -B -m pytest -q tests/test_backend_integration.py -p no:cacheprovider
|
||||
```
|
||||
|
||||
真实 add/flush 上传会写入 EverOS，且可能受上游解析、LLM、embedding 服务耗时影响。需要验证完整摄入链路时再打开第二层开关：
|
||||
真实 add/flush 上传会写入上游记忆服务，且可能受上游解析、LLM、embedding 服务耗时影响。需要验证完整摄入链路时再打开第二层开关：
|
||||
|
||||
```bash
|
||||
cd /home/tom/memory-gateway
|
||||
RUN_EVEROS_INTEGRATION=1 \
|
||||
RUN_EVEROS_INGEST_INTEGRATION=1 \
|
||||
EVEROS_BASE_URL=http://10.6.80.123:1995 \
|
||||
.venv/bin/python -B -m pytest -q tests/test_everos_integration.py -p no:cacheprovider
|
||||
RUN_BACKEND_INTEGRATION=1 \
|
||||
RUN_BACKEND_INGEST_INTEGRATION=1 \
|
||||
MEMORY_GATEWAY_BACKEND_BASE_URL=http://10.6.80.123:1995 \
|
||||
.venv/bin/python -B -m pytest -q tests/test_backend_integration.py -p no:cacheprovider
|
||||
```
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
"""Lightweight Memory Gateway for EverOS."""
|
||||
"""Lightweight user resource memory gateway."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
72
core/api.py
72
core/api.py
@ -8,14 +8,19 @@ from typing import Any, Literal
|
||||
from urllib.parse import parse_qsl, quote, urlsplit, urlunsplit
|
||||
|
||||
from fastapi import APIRouter, FastAPI, File, Form, HTTPException, Request, UploadFile
|
||||
from pydantic import BaseModel, Field
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
from starlette.responses import Response
|
||||
|
||||
from .config import GatewayConfig
|
||||
from .db import init_db
|
||||
from .everos_client import EverOSClient
|
||||
from .backend_client import BackendClient
|
||||
from .repository import MemoryRepository
|
||||
from .service import MemoryGatewayService, UnsupportedContentType, UploadTooLarge
|
||||
from .service import (
|
||||
InvalidAttachment,
|
||||
MemoryGatewayService,
|
||||
UnsupportedContentType,
|
||||
UploadTooLarge,
|
||||
)
|
||||
|
||||
|
||||
API_LOGGER = logging.getLogger("memory_gateway.api")
|
||||
@ -34,15 +39,28 @@ SENSITIVE_FIELD_NAMES = {
|
||||
class SearchMemoriesRequest(BaseModel):
|
||||
user_id: str = Field(min_length=1)
|
||||
user_key: str = Field(min_length=1)
|
||||
agent_id: str | None = Field(default=None, min_length=1)
|
||||
conversation_id: str | None = None
|
||||
query: str = Field(min_length=1)
|
||||
scope: list[Literal["current_chat", "resources", "all_user_memory"]] = Field(
|
||||
default_factory=lambda: ["current_chat", "resources"]
|
||||
)
|
||||
top_k: int = Field(default=8, ge=1, le=100)
|
||||
method: Literal["keyword", "vector", "hybrid", "agentic"] = "hybrid"
|
||||
top_k: int = 8
|
||||
radius: float | None = Field(default=None, ge=0, le=1)
|
||||
include_profile: bool = True
|
||||
enable_llm_rerank: bool = True
|
||||
filters: dict[str, Any] | None = None
|
||||
app_id: str = "default"
|
||||
project_id: str = "default"
|
||||
|
||||
@field_validator("top_k")
|
||||
@classmethod
|
||||
def validate_top_k(cls, value: int) -> int:
|
||||
if value != -1 and not 1 <= value <= 100:
|
||||
raise ValueError("top_k must be -1 or in 1..100")
|
||||
return value
|
||||
|
||||
|
||||
class AddMemoryMessage(BaseModel):
|
||||
sender_id: str = Field(min_length=1)
|
||||
@ -181,21 +199,21 @@ def _body_for_log(body: bytes, content_type: str | None) -> Any:
|
||||
def create_app(
|
||||
*,
|
||||
config: GatewayConfig | None = None,
|
||||
everos_client: Any | None = None,
|
||||
backend_client: Any | None = None,
|
||||
) -> FastAPI:
|
||||
cfg = config or GatewayConfig.from_env()
|
||||
init_db(cfg.database_path)
|
||||
repository = MemoryRepository(cfg.database_path)
|
||||
client = everos_client or EverOSClient(
|
||||
cfg.everos_base_url,
|
||||
timeout=cfg.everos_timeout_seconds,
|
||||
client = backend_client or BackendClient(
|
||||
cfg.backend_base_url,
|
||||
timeout=cfg.backend_timeout_seconds,
|
||||
)
|
||||
service = MemoryGatewayService(cfg, repository, client)
|
||||
|
||||
app = FastAPI(title="memory-gateway", version="0.1.0")
|
||||
app.state.config = cfg
|
||||
app.state.repository = repository
|
||||
app.state.everos_client = client
|
||||
app.state.backend_client = client
|
||||
app.state.gateway_service = service
|
||||
|
||||
router = APIRouter()
|
||||
@ -278,24 +296,24 @@ def create_app(
|
||||
@router.get("/health")
|
||||
async def health() -> dict[str, Any]:
|
||||
try:
|
||||
everos_health = await client.health_check()
|
||||
backend_health = await client.health_check()
|
||||
except Exception as exc:
|
||||
return {
|
||||
"status": "degraded",
|
||||
"api": {"status": "ok"},
|
||||
"everos": {
|
||||
"backend": {
|
||||
"status": "unavailable",
|
||||
"base_url": cfg.everos_base_url,
|
||||
"base_url": cfg.backend_base_url,
|
||||
"error": str(exc),
|
||||
},
|
||||
}
|
||||
return {
|
||||
"status": "ok",
|
||||
"api": {"status": "ok"},
|
||||
"everos": {
|
||||
"backend": {
|
||||
"status": "ok",
|
||||
"base_url": cfg.everos_base_url,
|
||||
"data": everos_health,
|
||||
"base_url": cfg.backend_base_url,
|
||||
"data": backend_health,
|
||||
},
|
||||
}
|
||||
|
||||
@ -367,10 +385,16 @@ def create_app(
|
||||
require_user(request.user_id, request.user_key)
|
||||
return await service.search_memories(
|
||||
user_id=request.user_id,
|
||||
agent_id=request.agent_id,
|
||||
query=request.query,
|
||||
conversation_id=request.conversation_id,
|
||||
scope=request.scope,
|
||||
method=request.method,
|
||||
top_k=request.top_k,
|
||||
radius=request.radius,
|
||||
include_profile=request.include_profile,
|
||||
enable_llm_rerank=request.enable_llm_rerank,
|
||||
filters=request.filters,
|
||||
app_id=request.app_id,
|
||||
project_id=request.project_id,
|
||||
)
|
||||
@ -380,12 +404,18 @@ def create_app(
|
||||
request: AddMemoryRequest,
|
||||
) -> dict[str, Any]:
|
||||
require_user(request.user_id, request.user_key)
|
||||
return await service.add_memory(
|
||||
session_id=request.session_id,
|
||||
app_id=request.app_id,
|
||||
project_id=request.project_id,
|
||||
messages=[message.model_dump() for message in request.messages],
|
||||
)
|
||||
try:
|
||||
return await service.add_memory(
|
||||
user_id=request.user_id,
|
||||
session_id=request.session_id,
|
||||
app_id=request.app_id,
|
||||
project_id=request.project_id,
|
||||
messages=[message.model_dump() for message in request.messages],
|
||||
)
|
||||
except UploadTooLarge as exc:
|
||||
raise HTTPException(status_code=413, detail=str(exc)) from exc
|
||||
except InvalidAttachment as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
|
||||
@router.post("/memories/flush")
|
||||
async def flush_memory(
|
||||
|
||||
@ -5,7 +5,7 @@ from typing import Any
|
||||
import httpx
|
||||
|
||||
|
||||
class EverOSClient:
|
||||
class BackendClient:
|
||||
def __init__(self, base_url: str, timeout: float = 120.0) -> None:
|
||||
self.base_url = base_url.rstrip("/")
|
||||
self.timeout = timeout
|
||||
@ -27,15 +27,15 @@ _DEFAULT_ALLOWED_MIME_TYPES = (
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GatewayConfig:
|
||||
everos_base_url: str = "http://127.0.0.1:1995"
|
||||
backend_base_url: str = "http://127.0.0.1:1995"
|
||||
database_path: Path = _PROJECT_ROOT / "data" / "memory_gateway.sqlite3"
|
||||
storage_dir: Path = _PROJECT_ROOT / "data" / "storage"
|
||||
resource_search_batch_size: int = 50
|
||||
max_upload_bytes: int = 25 * 1024 * 1024
|
||||
allowed_mime_types: tuple[str, ...] = _DEFAULT_ALLOWED_MIME_TYPES
|
||||
everos_ingest_attempts: int = 3
|
||||
everos_retry_delay_seconds: float = 0.25
|
||||
everos_timeout_seconds: float = 120.0
|
||||
backend_ingest_attempts: int = 3
|
||||
backend_retry_delay_seconds: float = 0.25
|
||||
backend_timeout_seconds: float = 120.0
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> GatewayConfig:
|
||||
@ -48,8 +48,8 @@ class GatewayConfig:
|
||||
if item.strip()
|
||||
)
|
||||
return cls(
|
||||
everos_base_url=os.environ.get(
|
||||
"EVEROS_BASE_URL",
|
||||
backend_base_url=os.environ.get(
|
||||
"MEMORY_GATEWAY_BACKEND_BASE_URL",
|
||||
"http://127.0.0.1:1995",
|
||||
).rstrip("/"),
|
||||
database_path=Path(
|
||||
@ -71,13 +71,13 @@ class GatewayConfig:
|
||||
os.environ.get("MEMORY_GATEWAY_MAX_UPLOAD_BYTES", str(25 * 1024 * 1024))
|
||||
),
|
||||
allowed_mime_types=allowed_mime_types,
|
||||
everos_ingest_attempts=int(
|
||||
os.environ.get("MEMORY_GATEWAY_EVEROS_INGEST_ATTEMPTS", "3")
|
||||
backend_ingest_attempts=int(
|
||||
os.environ.get("MEMORY_GATEWAY_BACKEND_INGEST_ATTEMPTS", "3")
|
||||
),
|
||||
everos_retry_delay_seconds=float(
|
||||
os.environ.get("MEMORY_GATEWAY_EVEROS_RETRY_DELAY_SECONDS", "0.25")
|
||||
backend_retry_delay_seconds=float(
|
||||
os.environ.get("MEMORY_GATEWAY_BACKEND_RETRY_DELAY_SECONDS", "0.25")
|
||||
),
|
||||
everos_timeout_seconds=float(
|
||||
os.environ.get("MEMORY_GATEWAY_EVEROS_TIMEOUT_SECONDS", "120")
|
||||
backend_timeout_seconds=float(
|
||||
os.environ.get("MEMORY_GATEWAY_BACKEND_TIMEOUT_SECONDS", "120")
|
||||
),
|
||||
)
|
||||
|
||||
56
core/db.py
56
core/db.py
@ -43,6 +43,62 @@ ON user_resources (session_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_user_resources_user_id
|
||||
ON user_resources (user_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS memory_attachments (
|
||||
id TEXT PRIMARY KEY,
|
||||
user_id TEXT NOT NULL,
|
||||
app_id TEXT NOT NULL DEFAULT 'default',
|
||||
project_id TEXT NOT NULL DEFAULT 'default',
|
||||
session_id TEXT NOT NULL,
|
||||
resource_id TEXT,
|
||||
content_type TEXT NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
internal_uri TEXT NOT NULL,
|
||||
source TEXT NOT NULL,
|
||||
sha256 TEXT,
|
||||
created_at TIMESTAMP NOT NULL,
|
||||
deleted_at TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_memory_attachments_unique_uri
|
||||
ON memory_attachments (user_id, session_id, internal_uri);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_memory_attachments_user_session
|
||||
ON memory_attachments (user_id, session_id, deleted_at);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_memory_attachments_resource
|
||||
ON memory_attachments (resource_id, deleted_at);
|
||||
|
||||
INSERT OR IGNORE INTO memory_attachments (
|
||||
id,
|
||||
user_id,
|
||||
app_id,
|
||||
project_id,
|
||||
session_id,
|
||||
resource_id,
|
||||
content_type,
|
||||
name,
|
||||
internal_uri,
|
||||
source,
|
||||
sha256,
|
||||
created_at,
|
||||
deleted_at
|
||||
)
|
||||
SELECT
|
||||
'a_resource_' || id,
|
||||
user_id,
|
||||
app_id,
|
||||
project_id,
|
||||
session_id,
|
||||
id,
|
||||
content_type,
|
||||
COALESCE(original_filename, id),
|
||||
uri,
|
||||
'resource_upload',
|
||||
sha256,
|
||||
created_at,
|
||||
deleted_at
|
||||
FROM user_resources;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS memory_tombstones (
|
||||
id TEXT PRIMARY KEY,
|
||||
user_id TEXT NOT NULL,
|
||||
|
||||
@ -96,9 +96,13 @@ class MemoryRepository:
|
||||
now = utc_now()
|
||||
where = "id = ? AND deleted_at IS NULL"
|
||||
params: tuple[Any, ...] = (now, now, resource_id)
|
||||
attachment_where = "resource_id = ? AND deleted_at IS NULL"
|
||||
attachment_params: tuple[Any, ...] = (now, resource_id)
|
||||
if user_id is not None:
|
||||
where += " AND user_id = ?"
|
||||
params = (now, now, resource_id, user_id)
|
||||
attachment_where += " AND user_id = ?"
|
||||
attachment_params = (now, resource_id, user_id)
|
||||
with connect(self.db_path) as conn:
|
||||
conn.execute(
|
||||
f"""
|
||||
@ -108,6 +112,14 @@ class MemoryRepository:
|
||||
""",
|
||||
params,
|
||||
)
|
||||
conn.execute(
|
||||
f"""
|
||||
UPDATE memory_attachments
|
||||
SET deleted_at = ?
|
||||
WHERE {attachment_where}
|
||||
""",
|
||||
attachment_params,
|
||||
)
|
||||
conn.commit()
|
||||
return self.get_resource(resource_id)
|
||||
|
||||
@ -215,6 +227,62 @@ class MemoryRepository:
|
||||
).fetchall()
|
||||
return [dict(row) for row in rows]
|
||||
|
||||
def create_attachment(self, **values: Any) -> dict[str, Any]:
|
||||
attachment_id = str(values.get("id") or f"a_{uuid.uuid4().hex}")
|
||||
payload = {
|
||||
"id": attachment_id,
|
||||
"created_at": utc_now(),
|
||||
"deleted_at": None,
|
||||
**values,
|
||||
}
|
||||
with connect(self.db_path) as conn:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT OR IGNORE INTO memory_attachments (
|
||||
id, user_id, app_id, project_id, session_id, resource_id,
|
||||
content_type, name, internal_uri, source, sha256,
|
||||
created_at, deleted_at
|
||||
) VALUES (
|
||||
:id, :user_id, :app_id, :project_id, :session_id, :resource_id,
|
||||
:content_type, :name, :internal_uri, :source, :sha256,
|
||||
:created_at, :deleted_at
|
||||
)
|
||||
""",
|
||||
payload,
|
||||
)
|
||||
row = conn.execute(
|
||||
"""
|
||||
SELECT * FROM memory_attachments
|
||||
WHERE user_id = ? AND session_id = ? AND internal_uri = ?
|
||||
""",
|
||||
(
|
||||
payload["user_id"],
|
||||
payload["session_id"],
|
||||
payload["internal_uri"],
|
||||
),
|
||||
).fetchone()
|
||||
conn.commit()
|
||||
attachment = _row_to_dict(row)
|
||||
if attachment is None:
|
||||
raise RuntimeError("created attachment could not be read back")
|
||||
return attachment
|
||||
|
||||
def list_attachments_for_session(
|
||||
self,
|
||||
user_id: str,
|
||||
session_id: str,
|
||||
) -> list[dict[str, Any]]:
|
||||
with connect(self.db_path) as conn:
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT * FROM memory_attachments
|
||||
WHERE user_id = ? AND session_id = ? AND deleted_at IS NULL
|
||||
ORDER BY created_at ASC, id ASC
|
||||
""",
|
||||
(user_id, session_id),
|
||||
).fetchall()
|
||||
return [dict(row) for row in rows]
|
||||
|
||||
def add_tombstone(
|
||||
self,
|
||||
user_id: str,
|
||||
|
||||
296
core/service.py
296
core/service.py
@ -2,6 +2,7 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import binascii
|
||||
import hashlib
|
||||
import mimetypes
|
||||
import secrets
|
||||
@ -63,6 +64,10 @@ class UnsupportedContentType(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class InvalidAttachment(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
def _copy_upload(
|
||||
file: UploadFile,
|
||||
destination: Path,
|
||||
@ -127,11 +132,11 @@ class MemoryGatewayService:
|
||||
self,
|
||||
config: GatewayConfig,
|
||||
repository: MemoryRepository,
|
||||
everos_client: Any,
|
||||
backend_client: Any,
|
||||
) -> None:
|
||||
self.config = config
|
||||
self.repository = repository
|
||||
self.everos_client = everos_client
|
||||
self.backend_client = backend_client
|
||||
|
||||
def create_user(self, user_id: str) -> dict[str, Any]:
|
||||
user_key = f"uk_{secrets.token_urlsafe(32)}"
|
||||
@ -180,6 +185,7 @@ class MemoryGatewayService:
|
||||
)
|
||||
if existing is not None:
|
||||
shutil.rmtree(stored_path.parent, ignore_errors=True)
|
||||
self._register_resource_attachment(existing)
|
||||
return self._resource_summary(existing)
|
||||
|
||||
internal_uri = stored_path.resolve().as_uri()
|
||||
@ -202,10 +208,11 @@ class MemoryGatewayService:
|
||||
status="ingesting",
|
||||
error_message=None,
|
||||
)
|
||||
self._register_resource_attachment(resource)
|
||||
|
||||
try:
|
||||
await self._retry_everos_call(
|
||||
lambda: self.everos_client.add_memory(
|
||||
await self._retry_backend_call(
|
||||
lambda: self.backend_client.add_memory(
|
||||
self._build_add_payload(
|
||||
resource=resource,
|
||||
user_id=user_id,
|
||||
@ -215,8 +222,8 @@ class MemoryGatewayService:
|
||||
)
|
||||
)
|
||||
)
|
||||
await self._retry_everos_call(
|
||||
lambda: self.everos_client.flush_memory(session_id, app_id, project_id)
|
||||
await self._retry_backend_call(
|
||||
lambda: self.backend_client.flush_memory(session_id, app_id, project_id)
|
||||
)
|
||||
except Exception as exc:
|
||||
failed = self.repository.update_resource_status(
|
||||
@ -229,8 +236,8 @@ class MemoryGatewayService:
|
||||
extracted = self.repository.update_resource_status(resource_id, "extracted")
|
||||
return self._resource_summary(extracted or resource)
|
||||
|
||||
async def _retry_everos_call(self, operation: Any) -> Any:
|
||||
attempts = max(1, self.config.everos_ingest_attempts)
|
||||
async def _retry_backend_call(self, operation: Any) -> Any:
|
||||
attempts = max(1, self.config.backend_ingest_attempts)
|
||||
last_error: Exception | None = None
|
||||
for attempt in range(attempts):
|
||||
try:
|
||||
@ -239,11 +246,11 @@ class MemoryGatewayService:
|
||||
last_error = exc
|
||||
if attempt == attempts - 1:
|
||||
break
|
||||
delay = self.config.everos_retry_delay_seconds
|
||||
delay = self.config.backend_retry_delay_seconds
|
||||
if delay > 0:
|
||||
await asyncio.sleep(delay)
|
||||
if last_error is None:
|
||||
raise RuntimeError("EverOS operation failed")
|
||||
raise RuntimeError("upstream memory service operation failed")
|
||||
raise last_error
|
||||
|
||||
def _build_add_payload(
|
||||
@ -346,10 +353,16 @@ class MemoryGatewayService:
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str | None,
|
||||
query: str,
|
||||
conversation_id: str | None,
|
||||
scope: list[str],
|
||||
method: str,
|
||||
top_k: int,
|
||||
radius: float | None,
|
||||
include_profile: bool,
|
||||
enable_llm_rerank: bool,
|
||||
filters: dict[str, Any] | None,
|
||||
app_id: str,
|
||||
project_id: str,
|
||||
) -> dict[str, Any]:
|
||||
@ -359,15 +372,23 @@ class MemoryGatewayService:
|
||||
if "current_chat" in scope and conversation_id:
|
||||
payload = self._search_payload(
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
query=query,
|
||||
method=method,
|
||||
top_k=top_k,
|
||||
radius=radius,
|
||||
include_profile=include_profile,
|
||||
enable_llm_rerank=enable_llm_rerank,
|
||||
app_id=app_id,
|
||||
project_id=project_id,
|
||||
filters={"session_id": f"chat:{conversation_id}"},
|
||||
filters=_combine_filters(
|
||||
filters,
|
||||
{"session_id": f"chat:{conversation_id}"},
|
||||
),
|
||||
)
|
||||
results.extend(
|
||||
self._extract_results(
|
||||
await self.everos_client.search_memory(payload),
|
||||
await self.backend_client.search_memory(payload),
|
||||
source_scope="current_chat",
|
||||
session_resource_map=session_resource_map,
|
||||
user_id=user_id,
|
||||
@ -385,15 +406,23 @@ class MemoryGatewayService:
|
||||
for batch in _chunks(session_ids, self.config.resource_search_batch_size):
|
||||
payload = self._search_payload(
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
query=query,
|
||||
method=method,
|
||||
top_k=top_k,
|
||||
radius=radius,
|
||||
include_profile=include_profile,
|
||||
enable_llm_rerank=enable_llm_rerank,
|
||||
app_id=app_id,
|
||||
project_id=project_id,
|
||||
filters={"session_id": {"in": batch}},
|
||||
filters=_combine_filters(
|
||||
filters,
|
||||
{"session_id": {"in": batch}},
|
||||
),
|
||||
)
|
||||
results.extend(
|
||||
self._extract_results(
|
||||
await self.everos_client.search_memory(payload),
|
||||
await self.backend_client.search_memory(payload),
|
||||
source_scope="resources",
|
||||
session_resource_map=session_resource_map,
|
||||
user_id=user_id,
|
||||
@ -403,15 +432,20 @@ class MemoryGatewayService:
|
||||
if "all_user_memory" in scope:
|
||||
payload = self._search_payload(
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
query=query,
|
||||
method=method,
|
||||
top_k=top_k,
|
||||
radius=radius,
|
||||
include_profile=include_profile,
|
||||
enable_llm_rerank=enable_llm_rerank,
|
||||
app_id=app_id,
|
||||
project_id=project_id,
|
||||
filters=None,
|
||||
filters=filters,
|
||||
)
|
||||
results.extend(
|
||||
self._extract_results(
|
||||
await self.everos_client.search_memory(payload),
|
||||
await self.backend_client.search_memory(payload),
|
||||
source_scope="all_user_memory",
|
||||
session_resource_map=session_resource_map,
|
||||
user_id=user_id,
|
||||
@ -425,21 +459,126 @@ class MemoryGatewayService:
|
||||
async def add_memory(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
session_id: str,
|
||||
app_id: str,
|
||||
project_id: str,
|
||||
messages: list[dict[str, Any]],
|
||||
) -> dict[str, Any]:
|
||||
attachments, generated_paths = self._prepare_memory_attachments(
|
||||
user_id=user_id,
|
||||
session_id=session_id,
|
||||
app_id=app_id,
|
||||
project_id=project_id,
|
||||
messages=messages,
|
||||
)
|
||||
payload = {
|
||||
"session_id": session_id,
|
||||
"app_id": app_id,
|
||||
"project_id": project_id,
|
||||
"messages": messages,
|
||||
}
|
||||
return {
|
||||
"session_id": session_id,
|
||||
"everos": await self.everos_client.add_memory(payload),
|
||||
}
|
||||
try:
|
||||
backend = await self.backend_client.add_memory(payload)
|
||||
for attachment in attachments:
|
||||
self.repository.create_attachment(**attachment)
|
||||
except Exception:
|
||||
for path in generated_paths:
|
||||
path.unlink(missing_ok=True)
|
||||
_remove_empty_parents(path.parent, stop_at=self.config.storage_dir)
|
||||
raise
|
||||
return {"session_id": session_id, "backend": backend}
|
||||
|
||||
def _register_resource_attachment(self, resource: dict[str, Any]) -> None:
|
||||
self.repository.create_attachment(
|
||||
user_id=resource["user_id"],
|
||||
app_id=resource["app_id"],
|
||||
project_id=resource["project_id"],
|
||||
session_id=resource["session_id"],
|
||||
resource_id=resource["id"],
|
||||
content_type=resource["content_type"],
|
||||
name=resource["original_filename"] or resource["id"],
|
||||
internal_uri=resource["uri"],
|
||||
source="resource_upload",
|
||||
sha256=resource["sha256"],
|
||||
)
|
||||
|
||||
def _prepare_memory_attachments(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
session_id: str,
|
||||
app_id: str,
|
||||
project_id: str,
|
||||
messages: list[dict[str, Any]],
|
||||
) -> tuple[list[dict[str, Any]], list[Path]]:
|
||||
attachments: list[dict[str, Any]] = []
|
||||
generated_paths: list[Path] = []
|
||||
try:
|
||||
for message in messages:
|
||||
content = message.get("content")
|
||||
if not isinstance(content, list):
|
||||
continue
|
||||
for item in content:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
uri = item.get("uri")
|
||||
encoded = item.get("base64")
|
||||
if not uri and not encoded:
|
||||
continue
|
||||
attachment_id = f"a_{uuid.uuid4().hex}"
|
||||
name = _attachment_name(item, str(uri) if uri else None)
|
||||
sha256: str | None = None
|
||||
if uri:
|
||||
internal_uri = str(uri)
|
||||
source = "memory_add_uri"
|
||||
else:
|
||||
try:
|
||||
data = base64.b64decode(str(encoded), validate=True)
|
||||
except (binascii.Error, ValueError) as exc:
|
||||
raise InvalidAttachment(
|
||||
f"invalid base64 attachment: {name}"
|
||||
) from exc
|
||||
if len(data) > self.config.max_upload_bytes:
|
||||
raise UploadTooLarge(
|
||||
f"attachment exceeds max size of "
|
||||
f"{self.config.max_upload_bytes} bytes"
|
||||
)
|
||||
sha256 = hashlib.sha256(data).hexdigest()
|
||||
path = (
|
||||
self.config.storage_dir
|
||||
/ user_id
|
||||
/ "memory_attachments"
|
||||
/ sha256
|
||||
/ name
|
||||
)
|
||||
if not path.exists():
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_bytes(data)
|
||||
generated_paths.append(path)
|
||||
internal_uri = path.resolve().as_uri()
|
||||
source = "memory_add_base64"
|
||||
attachments.append(
|
||||
{
|
||||
"id": attachment_id,
|
||||
"user_id": user_id,
|
||||
"app_id": app_id,
|
||||
"project_id": project_id,
|
||||
"session_id": session_id,
|
||||
"resource_id": None,
|
||||
"content_type": str(item.get("type") or "doc"),
|
||||
"name": name,
|
||||
"internal_uri": internal_uri,
|
||||
"source": source,
|
||||
"sha256": sha256,
|
||||
}
|
||||
)
|
||||
except Exception:
|
||||
for path in generated_paths:
|
||||
path.unlink(missing_ok=True)
|
||||
_remove_empty_parents(path.parent, stop_at=self.config.storage_dir)
|
||||
raise
|
||||
return attachments, generated_paths
|
||||
|
||||
async def flush_memory(
|
||||
self,
|
||||
@ -450,7 +589,7 @@ class MemoryGatewayService:
|
||||
) -> dict[str, Any]:
|
||||
return {
|
||||
"session_id": session_id,
|
||||
"everos": await self.everos_client.flush_memory(
|
||||
"backend": await self.backend_client.flush_memory(
|
||||
session_id,
|
||||
app_id,
|
||||
project_id,
|
||||
@ -461,19 +600,29 @@ class MemoryGatewayService:
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str | None,
|
||||
query: str,
|
||||
method: str,
|
||||
top_k: int,
|
||||
radius: float | None,
|
||||
include_profile: bool,
|
||||
enable_llm_rerank: bool,
|
||||
app_id: str,
|
||||
project_id: str,
|
||||
filters: dict[str, Any] | None,
|
||||
) -> dict[str, Any]:
|
||||
payload: dict[str, Any] = {
|
||||
"user_id": user_id,
|
||||
"query": query,
|
||||
"method": method,
|
||||
"top_k": top_k,
|
||||
"include_profile": include_profile,
|
||||
"enable_llm_rerank": enable_llm_rerank,
|
||||
"app_id": app_id,
|
||||
"project_id": project_id,
|
||||
}
|
||||
payload["agent_id" if agent_id else "user_id"] = agent_id or user_id
|
||||
if radius is not None:
|
||||
payload["radius"] = radius
|
||||
if filters is not None:
|
||||
payload["filters"] = filters
|
||||
return payload
|
||||
@ -487,18 +636,22 @@ class MemoryGatewayService:
|
||||
user_id: str,
|
||||
) -> list[dict[str, Any]]:
|
||||
data = response.get("data", {})
|
||||
raw_items: list[dict[str, Any]] = []
|
||||
for key in (
|
||||
"episodes",
|
||||
"profiles",
|
||||
"agent_cases",
|
||||
"agent_skills",
|
||||
"unprocessed_messages",
|
||||
):
|
||||
raw_items.extend(data.get(key, []) or [])
|
||||
raw_items: list[tuple[str, dict[str, Any]]] = []
|
||||
memory_types = {
|
||||
"episodes": "episode",
|
||||
"profiles": "profile",
|
||||
"agent_cases": "agent_case",
|
||||
"agent_skills": "agent_skill",
|
||||
"unprocessed_messages": "unprocessed_message",
|
||||
}
|
||||
for key, memory_type in memory_types.items():
|
||||
raw_items.extend(
|
||||
(memory_type, item) for item in (data.get(key, []) or [])
|
||||
)
|
||||
|
||||
normalized = []
|
||||
for raw in raw_items:
|
||||
attachment_cache: dict[str, list[dict[str, Any]]] = {}
|
||||
for memory_type, raw in raw_items:
|
||||
session_id = raw.get("session_id")
|
||||
resource = session_resource_map.get(session_id)
|
||||
if resource is None and isinstance(session_id, str):
|
||||
@ -506,9 +659,21 @@ class MemoryGatewayService:
|
||||
session_id,
|
||||
user_id,
|
||||
)
|
||||
attachments: list[dict[str, Any]] = []
|
||||
if isinstance(session_id, str):
|
||||
if session_id not in attachment_cache:
|
||||
attachment_cache[session_id] = (
|
||||
self.repository.list_attachments_for_session(
|
||||
user_id,
|
||||
session_id,
|
||||
)
|
||||
)
|
||||
session_attachments = attachment_cache[session_id]
|
||||
attachments = _matching_attachments(raw, session_attachments)
|
||||
normalized.append(
|
||||
{
|
||||
"id": raw.get("id"),
|
||||
"memory_type": memory_type,
|
||||
"session_id": session_id,
|
||||
"text": _display_text(raw),
|
||||
"score": raw.get("score"),
|
||||
@ -517,6 +682,7 @@ class MemoryGatewayService:
|
||||
"resource_uri": (
|
||||
public_resource_uri(user_id, resource["id"]) if resource else None
|
||||
),
|
||||
"attachments": attachments,
|
||||
"raw": raw,
|
||||
}
|
||||
)
|
||||
@ -623,6 +789,72 @@ class MemoryGatewayService:
|
||||
}
|
||||
|
||||
|
||||
def _combine_filters(
|
||||
custom_filters: dict[str, Any] | None,
|
||||
scope_filters: dict[str, Any] | None,
|
||||
) -> dict[str, Any] | None:
|
||||
if custom_filters is None:
|
||||
return scope_filters
|
||||
if scope_filters is None:
|
||||
return custom_filters
|
||||
return {"AND": [custom_filters, scope_filters]}
|
||||
|
||||
|
||||
def _attachment_name(item: dict[str, Any], uri: str | None) -> str:
|
||||
if item.get("name"):
|
||||
return _safe_filename(str(item["name"]))
|
||||
if uri:
|
||||
parsed = urlparse(uri)
|
||||
uri_name = Path(unquote(parsed.path)).name
|
||||
if uri_name:
|
||||
return _safe_filename(uri_name)
|
||||
extension = str(item.get("ext") or "bin").lstrip(".") or "bin"
|
||||
return f"attachment.{extension}"
|
||||
|
||||
|
||||
def _matching_attachments(
|
||||
raw: dict[str, Any],
|
||||
attachments: list[dict[str, Any]],
|
||||
) -> list[dict[str, Any]]:
|
||||
strings = [value.casefold() for value in _raw_string_values(raw)]
|
||||
matched: list[dict[str, Any]] = []
|
||||
seen_uris: set[str] = set()
|
||||
for attachment in attachments:
|
||||
name = str(attachment["name"])
|
||||
internal_uri = str(attachment["internal_uri"])
|
||||
if internal_uri in seen_uris:
|
||||
continue
|
||||
if not any(name.casefold() in value for value in strings):
|
||||
continue
|
||||
seen_uris.add(internal_uri)
|
||||
matched.append(
|
||||
{
|
||||
"type": attachment["content_type"],
|
||||
"name": name,
|
||||
"internal_uri": internal_uri,
|
||||
}
|
||||
)
|
||||
return matched
|
||||
|
||||
|
||||
def _raw_string_values(value: Any, key: str | None = None) -> list[str]:
|
||||
if key is not None and key.casefold() == "base64":
|
||||
return []
|
||||
if isinstance(value, str):
|
||||
return [value]
|
||||
if isinstance(value, dict):
|
||||
strings: list[str] = []
|
||||
for item_key, item_value in value.items():
|
||||
strings.extend(_raw_string_values(item_value, str(item_key)))
|
||||
return strings
|
||||
if isinstance(value, list):
|
||||
strings = []
|
||||
for item in value:
|
||||
strings.extend(_raw_string_values(item))
|
||||
return strings
|
||||
return []
|
||||
|
||||
|
||||
def _chunks(items: list[str], size: int) -> list[list[str]]:
|
||||
if not items:
|
||||
return []
|
||||
|
||||
@ -0,0 +1,45 @@
|
||||
# Memory Gateway Agent Skill Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Create a reusable AI-agent skill that safely operates the Memory Gateway API through a deterministic Python CLI.
|
||||
|
||||
**Architecture:** Keep procedural guidance in `SKILL.md`, detailed endpoint schemas in `references/api.md`, and all HTTP/multipart behavior in one standard-library CLI. Read credentials from environment variables or explicit flags and never persist secrets in the skill.
|
||||
|
||||
**Tech Stack:** Agent Skills format, Python 3 standard library, pytest, Memory Gateway HTTP API.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Scaffold the skill
|
||||
|
||||
**Files:**
|
||||
- Create: `skill/memory-gateway-agent/SKILL.md`
|
||||
- Create: `skill/memory-gateway-agent/agents/openai.yaml`
|
||||
- Create: `skill/memory-gateway-agent/scripts/memory_gateway.py`
|
||||
- Create: `skill/memory-gateway-agent/references/api.md`
|
||||
|
||||
- [x] Initialize the standard skill structure with `init_skill.py`.
|
||||
- [x] Remove generated placeholders and keep only required resources.
|
||||
|
||||
### Task 2: Implement and test the CLI
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/test_memory_gateway_skill.py`
|
||||
- Modify: `skill/memory-gateway-agent/scripts/memory_gateway.py`
|
||||
|
||||
- [x] Write failing tests for environment credentials, JSON requests, multipart uploads, and HTTP errors.
|
||||
- [x] Run the focused tests and confirm they fail for missing implementation.
|
||||
- [x] Implement the standard-library CLI with commands for health, users, resources, search, add/flush, override, and delete.
|
||||
- [x] Run the focused tests and confirm they pass.
|
||||
|
||||
### Task 3: Author and validate the skill
|
||||
|
||||
**Files:**
|
||||
- Modify: `skill/memory-gateway-agent/SKILL.md`
|
||||
- Modify: `skill/memory-gateway-agent/references/api.md`
|
||||
- Modify: `skill/memory-gateway-agent/agents/openai.yaml`
|
||||
|
||||
- [x] Document the agent workflow, authentication rules, ownership checks, and safe handling of secrets.
|
||||
- [x] Document endpoint parameters and CLI examples in the API reference.
|
||||
- [x] Generate UI metadata with the official skill-creator script.
|
||||
- [x] Run `quick_validate.py`, CLI `--help`, focused tests, and the full project test suite.
|
||||
@ -0,0 +1,62 @@
|
||||
# Upstream Brand Neutralization Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Remove the upstream product identity from the current Memory Gateway files without changing the upstream memory HTTP protocol.
|
||||
|
||||
**Architecture:** Replace product-specific names with `backend` terminology at configuration, client, service, API, test, and documentation boundaries. Enforce the result with a repository-level regression test that scans both file names and text content.
|
||||
|
||||
**Tech Stack:** Python 3, FastAPI, pytest, Markdown, environment configuration.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add the neutral-brand regression test
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/test_branding.py`
|
||||
|
||||
- [x] Add a test that constructs the forbidden token from separate string fragments.
|
||||
- [x] Scan non-generated project file names and UTF-8 text contents.
|
||||
- [x] Run the test and verify it fails against the existing product-specific names.
|
||||
|
||||
### Task 2: Rename runtime boundaries
|
||||
|
||||
**Files:**
|
||||
- Rename: `core/backend_client.py` to `core/backend_client.py`
|
||||
- Modify: `core/api.py`
|
||||
- Modify: `core/config.py`
|
||||
- Modify: `core/service.py`
|
||||
- Modify: `core/__init__.py`
|
||||
- Modify: `.env.example`
|
||||
- Modify: `.gitignore`
|
||||
- Delete: `backend.env.example`
|
||||
|
||||
- [x] Rename the client class, dependency attributes, retry helpers, and configuration fields to `backend` terminology.
|
||||
- [x] Rename environment variables to `MEMORY_GATEWAY_BACKEND_*`.
|
||||
- [x] Rename health and direct add/flush response fields to `backend`.
|
||||
- [x] Preserve all `/api/v1/memory/*` paths.
|
||||
|
||||
### Task 3: Rename tests and public documentation
|
||||
|
||||
**Files:**
|
||||
- Rename: `tests/test_backend_integration.py` to `tests/test_backend_integration.py`
|
||||
- Modify: `tests/test_gateway.py`
|
||||
- Modify: `tests/test_command.md`
|
||||
- Modify: `README.md`
|
||||
- Modify: `pyproject.toml`
|
||||
- Modify: `skill/memory-gateway-agent/SKILL.md`
|
||||
- Modify: `skill/memory-gateway-agent/references/api.md`
|
||||
|
||||
- [x] Rename fixtures, tests, environment flags, examples, and expected JSON fields.
|
||||
- [x] Describe the dependency only as an upstream memory service.
|
||||
- [x] Update integration commands and package metadata.
|
||||
|
||||
### Task 4: Verify the current working tree
|
||||
|
||||
**Files:**
|
||||
- Modify: `docs/superpowers/plans/2026-06-12-upstream-brand-neutralization.md`
|
||||
|
||||
- [x] Remove generated bytecode caches.
|
||||
- [x] Run the branding regression test.
|
||||
- [x] Run the full test suite and Python compilation.
|
||||
- [x] Run a final case-insensitive content and file-name scan, excluding `.git` history.
|
||||
@ -0,0 +1,57 @@
|
||||
# Memory Attachment Path Mapping Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Persist attachment-to-session mappings for resource and direct memory ingestion, then return filename-matched real URIs from memory search results.
|
||||
|
||||
**Architecture:** Add one SQLite attachment table and repository methods. Register resource files directly, materialize base64 memory attachments under Gateway storage, and enrich normalized search results by matching attachment names against recursive raw string values.
|
||||
|
||||
**Tech Stack:** Python 3.10+, FastAPI, SQLite, Pydantic, pytest, httpx.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Attachment persistence
|
||||
|
||||
**Files:**
|
||||
- Modify: `core/db.py`
|
||||
- Modify: `core/repository.py`
|
||||
- Modify: `tests/test_gateway.py`
|
||||
|
||||
- [x] Write failing tests proving attachment records can be created, listed by user/session, deduplicated, and soft-deleted with resources.
|
||||
- [x] Run focused tests and verify failure because the table and methods do not exist.
|
||||
- [x] Add `memory_attachments`, indexes, resource backfill SQL, and focused repository methods.
|
||||
- [x] Run focused tests and verify they pass.
|
||||
|
||||
### Task 2: Register attachments during ingestion
|
||||
|
||||
**Files:**
|
||||
- Modify: `core/api.py`
|
||||
- Modify: `core/service.py`
|
||||
- Modify: `tests/test_gateway.py`
|
||||
|
||||
- [x] Write failing tests for `/resources`, `/memories/add` URI items, and `/memories/add` base64 items.
|
||||
- [x] Run focused tests and verify missing mappings and files.
|
||||
- [x] Register resource mappings, pass authenticated `user_id` into add service, materialize base64 files, and persist successful add mappings.
|
||||
- [x] Run focused tests and verify they pass.
|
||||
|
||||
### Task 3: Enrich search results
|
||||
|
||||
**Files:**
|
||||
- Modify: `core/service.py`
|
||||
- Modify: `tests/test_gateway.py`
|
||||
|
||||
- [x] Write failing tests for filename match, no match, base64-key exclusion, and cross-user isolation.
|
||||
- [x] Run focused tests and verify `attachments` is absent.
|
||||
- [x] Recursively collect raw strings excluding base64 and return deduplicated matching attachments.
|
||||
- [x] Run focused tests and verify they pass.
|
||||
|
||||
### Task 4: Documentation and regression
|
||||
|
||||
**Files:**
|
||||
- Modify: `README.md`
|
||||
- Modify: `tests/test_command.md`
|
||||
|
||||
- [x] Document attachment persistence, historical backfill limits, matching behavior, and response shape.
|
||||
- [x] Update the search response example with `attachments`.
|
||||
- [x] Run `git diff --check`, compile checks, and the complete pytest suite.
|
||||
- [x] Review the final diff for user isolation and unintended URI exposure outside search.
|
||||
@ -0,0 +1,118 @@
|
||||
# Memory Search Upstream Options Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Extend `POST /memories/search` with all upstream search options while preserving Gateway authentication, scopes, resource isolation, tombstones, and overrides.
|
||||
|
||||
**Architecture:** Extend the existing Pydantic request model and pass the validated values through `MemoryGatewayService`. Keep scope orchestration intact, combine caller filters with scope-generated session filters using `AND`, and tag normalized results according to their upstream response array.
|
||||
|
||||
**Tech Stack:** Python 3.10+, FastAPI, Pydantic v2, pytest, pytest-asyncio, httpx ASGI transport.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Search request options and defaults
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/test_gateway.py`
|
||||
- Modify: `core/api.py`
|
||||
- Modify: `core/service.py`
|
||||
|
||||
- [x] **Step 1: Write failing tests for defaults, custom options, and validation**
|
||||
|
||||
Add API tests that assert a default search sends `method="hybrid"`, `include_profile=true`, and `enable_llm_rerank=true`; a custom request forwards `agent_id`, `keyword`, `radius`, `top_k=-1`, and both false flags; and invalid `method`, `radius`, and `top_k=0` return HTTP 422.
|
||||
|
||||
- [x] **Step 2: Run tests and verify expected failures**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
uv run pytest tests/test_gateway.py -k 'search_forwards_default_upstream_options or search_forwards_all_upstream_options or search_rejects_invalid_upstream_options' -q
|
||||
```
|
||||
|
||||
Expected: assertions fail because the request model and service do not yet accept or forward the new fields.
|
||||
|
||||
- [x] **Step 3: Implement request fields and payload forwarding**
|
||||
|
||||
Extend `SearchMemoriesRequest` with:
|
||||
|
||||
```python
|
||||
agent_id: str | None = Field(default=None, min_length=1)
|
||||
method: Literal["keyword", "vector", "hybrid", "agentic"] = "hybrid"
|
||||
radius: float | None = Field(default=None, ge=0, le=1)
|
||||
include_profile: bool = True
|
||||
enable_llm_rerank: bool = True
|
||||
filters: dict[str, Any] | None = None
|
||||
```
|
||||
|
||||
Validate `top_k` as `-1` or `1..100`, pass all values to the service, and make `_search_payload` select exactly one upstream owner key (`agent_id` when present, otherwise `user_id`).
|
||||
|
||||
- [x] **Step 4: Run focused tests and verify they pass**
|
||||
|
||||
Run the command from Step 2. Expected: all selected tests pass.
|
||||
|
||||
### Task 2: Filter composition and result memory types
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/test_gateway.py`
|
||||
- Modify: `core/service.py`
|
||||
|
||||
- [x] **Step 1: Write failing tests for filter composition and result types**
|
||||
|
||||
Add a resource-scope test asserting caller filters and `session_id in [...]` are combined as:
|
||||
|
||||
```python
|
||||
{"AND": [caller_filters, {"session_id": {"in": [session_id]}}]}
|
||||
```
|
||||
|
||||
Extend the fake backend to return all response arrays and assert normalized results have `memory_type` values `episode`, `profile`, `agent_case`, `agent_skill`, and `unprocessed_message`.
|
||||
|
||||
- [x] **Step 2: Run tests and verify expected failures**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
uv run pytest tests/test_gateway.py -k 'search_combines_custom_and_scope_filters or search_labels_all_memory_types' -q
|
||||
```
|
||||
|
||||
Expected: failures because caller filters are not composed and normalized results have no `memory_type`.
|
||||
|
||||
- [x] **Step 3: Implement composition and typed normalization**
|
||||
|
||||
Add a small `_combine_filters` helper that returns either condition directly, returns `None` when both are absent, or returns `{"AND": [custom, scope]}` when both exist. Iterate an explicit mapping from response array name to memory type in `_extract_results` and include the mapped value in every normalized result.
|
||||
|
||||
- [x] **Step 4: Run focused tests and verify they pass**
|
||||
|
||||
Run the command from Step 2. Expected: both tests pass.
|
||||
|
||||
### Task 3: Documentation and regression verification
|
||||
|
||||
**Files:**
|
||||
- Modify: `README.md`
|
||||
- Verify: `tests/test_gateway.py`
|
||||
- Verify: `tests/test_memory_gateway_skill.py`
|
||||
|
||||
- [x] **Step 1: Update the Chinese API documentation**
|
||||
|
||||
Document `agent_id`, `method`, `radius`, `include_profile`, `enable_llm_rerank`, `filters`, the `top_k=-1` rule, filter composition, and the `memory_type` response field. Update the curl and JSON examples with the new defaults.
|
||||
|
||||
- [x] **Step 2: Run formatting and full tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
git diff --check
|
||||
uv run pytest -q
|
||||
```
|
||||
|
||||
Expected: no whitespace errors and all tests pass.
|
||||
|
||||
- [x] **Step 3: Review the final diff**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
git diff --stat
|
||||
git diff -- core/api.py core/service.py tests/test_gateway.py README.md
|
||||
```
|
||||
|
||||
Expected: changes are limited to the approved search compatibility scope and documentation.
|
||||
@ -0,0 +1,23 @@
|
||||
# Upstream Brand Neutralization Design
|
||||
|
||||
## Goal
|
||||
|
||||
Remove the upstream product name from the current Memory Gateway working tree while preserving the upstream HTTP protocol and application behavior.
|
||||
|
||||
## Scope
|
||||
|
||||
- Rename the upstream client module, class, configuration fields, environment variables, state attributes, response fields, tests, and integration test file to neutral `backend` terminology.
|
||||
- Rewrite README, Skill documentation, examples, package metadata, and test records to describe an "upstream memory service".
|
||||
- Remove the upstream-specific environment example because its variable names identify the product.
|
||||
- Preserve `/api/v1/memory/add`, `/api/v1/memory/flush`, and `/api/v1/memory/search` paths.
|
||||
- Do not rewrite Git history.
|
||||
|
||||
## Compatibility
|
||||
|
||||
This is an intentional configuration and response-schema rename. Deployments must move to `MEMORY_GATEWAY_BACKEND_*` variables, and health/add/flush consumers must read the `backend` field. No legacy aliases are retained because they would defeat the neutralization requirement.
|
||||
|
||||
## Verification
|
||||
|
||||
- Add an automated repository scan that rejects the forbidden upstream token in current files and file names.
|
||||
- Run the full unit suite and compilation checks.
|
||||
- Run a final case-insensitive repository scan excluding `.git`, virtual environments, runtime data, and generated bytecode.
|
||||
@ -0,0 +1,101 @@
|
||||
# Memory 附件真实路径映射设计
|
||||
|
||||
## 目标
|
||||
|
||||
让 `/resources` 和 `/memories/add` 两种摄入方式都保存附件与 session 的映射。
|
||||
`/memories/search` 返回结果时，根据结果 `session_id` 查询当前用户附件，并且只有
|
||||
当附件完整文件名出现在结果 `raw` 的字符串字段中时，才返回该附件真实 URI。
|
||||
|
||||
## 数据模型
|
||||
|
||||
新增 SQLite 表 `memory_attachments`：
|
||||
|
||||
- `id TEXT PRIMARY KEY`
|
||||
- `user_id TEXT NOT NULL`
|
||||
- `app_id TEXT NOT NULL DEFAULT 'default'`
|
||||
- `project_id TEXT NOT NULL DEFAULT 'default'`
|
||||
- `session_id TEXT NOT NULL`
|
||||
- `resource_id TEXT`
|
||||
- `content_type TEXT NOT NULL`
|
||||
- `name TEXT NOT NULL`
|
||||
- `internal_uri TEXT NOT NULL`
|
||||
- `source TEXT NOT NULL`
|
||||
- `sha256 TEXT`
|
||||
- `created_at TIMESTAMP NOT NULL`
|
||||
- `deleted_at TIMESTAMP`
|
||||
|
||||
以 `(user_id, session_id, internal_uri)` 建立唯一索引，避免幂等上传产生重复映射；
|
||||
以 `(user_id, session_id, deleted_at)` 建立查询索引。
|
||||
|
||||
数据库初始化时，将现有未删除 `user_resources` 回填为附件映射。历史
|
||||
`/memories/add` 请求没有保存在 Gateway 数据库中，因此无法自动回填。
|
||||
|
||||
## 摄入规则
|
||||
|
||||
### `/resources`
|
||||
|
||||
资源记录创建后，为保存的真实 `file://` URI 创建附件映射：
|
||||
|
||||
- `session_id` 使用 `resource:{user_id}:{resource_id}`；
|
||||
- `resource_id` 指向资源；
|
||||
- `source` 为 `resource_upload`；
|
||||
- `content_type`、文件名、SHA256 复用资源元数据。
|
||||
|
||||
重复资源上传时确保已有资源对应的附件映射存在。
|
||||
|
||||
### `/memories/add`
|
||||
|
||||
API 将已鉴权的 `user_id` 一并传给 service。逐条检查 message 的 content item：
|
||||
|
||||
- 只有字符串 content 或纯文本 item 时不创建附件；
|
||||
- 有 `uri` 时记录该 URI，`source=memory_add_uri`；
|
||||
- 没有 `uri` 但有 `base64` 时，解码并保存到
|
||||
`storage/{user_id}/memory_attachments/{attachment_id}/{safe_name}`，记录生成的
|
||||
`file://` URI，`source=memory_add_base64`；
|
||||
- 同时存在 `uri` 和 `base64` 时优先使用 `uri`，不重复落盘；
|
||||
- 文件名优先使用 `name`，否则从 URI 路径或 `ext` 生成安全名称。
|
||||
|
||||
上游 add 调用失败时，删除本次 base64 生成的文件，不写入映射。调用成功后写入
|
||||
附件映射。上游请求体保持原样，不修改现有 add 行为。
|
||||
|
||||
## 搜索匹配规则
|
||||
|
||||
对每条标准化搜索结果：
|
||||
|
||||
1. 根据已鉴权 `user_id` 和结果 `session_id` 查询未删除附件；
|
||||
2. 递归遍历 `raw` 中 dict、list 的字符串值；
|
||||
3. 跳过键名为 `base64` 的值，避免扫描大块编码数据；
|
||||
4. 使用附件完整文件名做不区分大小写的子串匹配；
|
||||
5. 仅命中的附件进入 `attachments`，按 `internal_uri` 去重；
|
||||
6. 没有 session 或没有命中时返回 `attachments: []`。
|
||||
|
||||
响应附件格式：
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "image",
|
||||
"name": "simple-multimodal-image.png",
|
||||
"internal_uri": "file:///home/tom/memory-gateway/tests/simple-multimodal-image.png"
|
||||
}
|
||||
```
|
||||
|
||||
episode 是 session 级记忆，因此只能在同一 session 的附件中按文件名匹配，不能
|
||||
证明具体附件是向量召回的直接来源。
|
||||
|
||||
## 删除与隔离
|
||||
|
||||
- 所有附件查询必须同时匹配 `user_id` 和 `session_id`；
|
||||
- 删除 `/resources` 时，对应附件映射设置 `deleted_at`；
|
||||
- 真实路径按用户明确要求直接出现在搜索结果中；
|
||||
- 不改变资源列表和详情现有的 `resource://` 对外 URI。
|
||||
|
||||
## 测试
|
||||
|
||||
- 资源上传创建附件映射；
|
||||
- 资源搜索仅在 raw 出现文件名时返回真实 URI；
|
||||
- raw 不含文件名时返回空附件数组；
|
||||
- `/memories/add` 的 URI content 创建映射；
|
||||
- `/memories/add` 的 base64 content 落盘并创建映射；
|
||||
- 不扫描 raw 中的 base64 字段；
|
||||
- 不返回其他用户同 session 的附件；
|
||||
- 现有测试继续通过。
|
||||
@ -0,0 +1,145 @@
|
||||
# Memory Search 上游参数增量兼容设计
|
||||
|
||||
## 目标
|
||||
|
||||
扩展 Memory Gateway 的 `POST /memories/search`，在保留现有用户鉴权、
|
||||
`scope` 搜索编排、资源隔离、软删除和覆盖修改能力的前提下，支持上游
|
||||
搜索接口的全部请求选项。
|
||||
|
||||
本次只修改 `/memories/search`，不新增 `/memories/get`，也不提供上游路径的
|
||||
线协议兼容接口。
|
||||
|
||||
## 请求模型
|
||||
|
||||
保留现有字段：
|
||||
|
||||
- `user_id`：必填，始终用于 Gateway 用户鉴权和本地数据隔离。
|
||||
- `user_key`：必填，用于 Gateway 用户鉴权。
|
||||
- `conversation_id`：可选，供 `current_chat` scope 生成 session 过滤条件。
|
||||
- `query`：必填。
|
||||
- `scope`：保留 `current_chat`、`resources`、`all_user_memory`。
|
||||
- `app_id`、`project_id`：默认 `default`。
|
||||
|
||||
新增或扩展字段：
|
||||
|
||||
- `agent_id`：可选。存在时，上游搜索使用 `agent_id`；不存在时使用
|
||||
Gateway 鉴权用户的 `user_id`。请求中不会同时向上游发送两种 owner ID。
|
||||
- `method`：支持 `keyword`、`vector`、`hybrid`、`agentic`，默认 `hybrid`。
|
||||
- `top_k`：支持 `-1` 或 `1..100`，保留 Gateway 默认值 `8`。
|
||||
- `radius`：可选，范围 `0..1`。
|
||||
- `include_profile`：布尔值，默认 `true`。
|
||||
- `enable_llm_rerank`：布尔值，默认 `true`。
|
||||
- `filters`：可选对象，支持上游开放字段过滤 DSL，包括嵌套 `AND`、`OR`。
|
||||
|
||||
`agent_id` 只改变上游记忆 owner，不替代 Gateway 的 `user_id/user_key` 鉴权。
|
||||
这可以防止调用者绕过 Gateway 用户体系，同时允许同一已认证用户查询被授权
|
||||
使用的 agent memory。当前版本不新增 agent 权限表，因此仅校验 Gateway 用户
|
||||
凭据，不声明 agent 的独立所有权关系。
|
||||
|
||||
## 搜索编排与过滤器合并
|
||||
|
||||
每一次上游 search 调用都必须透传：
|
||||
|
||||
- owner：`agent_id` 或 `user_id`，二选一；
|
||||
- `query`；
|
||||
- `method`；
|
||||
- `top_k`；
|
||||
- `radius`，仅在请求提供时发送；
|
||||
- `include_profile`；
|
||||
- `enable_llm_rerank`；
|
||||
- `app_id`；
|
||||
- `project_id`；
|
||||
- 合并后的 `filters`。
|
||||
|
||||
现有 scope 继续生成内部 session 条件：
|
||||
|
||||
- `current_chat`：`session_id = chat:{conversation_id}`；
|
||||
- `resources`：按批次生成 `session_id in [...]`；
|
||||
- `all_user_memory`：不生成 session 条件。
|
||||
|
||||
当请求同时提供自定义 `filters` 和 scope session 条件时，使用以下结构合并：
|
||||
|
||||
```json
|
||||
{
|
||||
"AND": [
|
||||
{"自定义过滤条件": "..."},
|
||||
{"session_id": "scope 生成的条件"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
仅存在其中一个条件时直接使用该条件；两者都不存在时不发送 `filters`。
|
||||
Gateway 不解析或重写自定义过滤 DSL 的内部字段，由上游执行完整校验。
|
||||
|
||||
`agent_id` 与所有 scope 均可组合。对于没有对应数据的组合，上游自然返回空数组；
|
||||
Gateway 不额外禁止这些组合，以保持接口简单并完整透传搜索能力。
|
||||
|
||||
## 响应标准化
|
||||
|
||||
继续返回 Gateway 的统一结构：
|
||||
|
||||
```json
|
||||
{
|
||||
"results": []
|
||||
}
|
||||
```
|
||||
|
||||
每个结果新增 `memory_type`：
|
||||
|
||||
| 上游数组 | `memory_type` |
|
||||
|---|---|
|
||||
| `episodes` | `episode` |
|
||||
| `profiles` | `profile` |
|
||||
| `agent_cases` | `agent_case` |
|
||||
| `agent_skills` | `agent_skill` |
|
||||
| `unprocessed_messages` | `unprocessed_message` |
|
||||
|
||||
其余字段保持现状：`id`、`session_id`、`text`、`score`、`source_scope`、
|
||||
`resource_id`、`resource_uri` 和 `raw`。
|
||||
|
||||
profile 和 agent skill 等没有 `session_id` 的结果允许返回 `null`。资源映射只对
|
||||
能匹配当前用户资源 session 的结果生效，不泄露其他用户的内部资源 URI。
|
||||
|
||||
所有类型的结果继续按现有顺序执行：
|
||||
|
||||
1. 合并各 scope 的结果；
|
||||
2. 应用当前用户的 memory tombstone；
|
||||
3. 按 memory ID 应用当前用户的 active override；
|
||||
4. 返回统一结果。
|
||||
|
||||
## 错误处理
|
||||
|
||||
- 不合法的 `method`、`top_k` 或 `radius` 由 Gateway 请求模型返回 HTTP 422。
|
||||
- 上游过滤 DSL 错误和其他 HTTP 错误继续由现有 client 行为向外传播。
|
||||
- 不改变当前 `current_chat` 缺少 `conversation_id` 时跳过该 scope 的行为。
|
||||
- 不为 `agent_id` 引入新的数据库表或权限模型。
|
||||
|
||||
## 代码改动
|
||||
|
||||
- `core/api.py`
|
||||
- 扩展 `SearchMemoriesRequest`。
|
||||
- 将新增参数传给 service。
|
||||
- `core/service.py`
|
||||
- 扩展 `search_memories` 和 `_search_payload`。
|
||||
- 合并自定义 filters 与 scope filters。
|
||||
- 标准化结果时增加 `memory_type`。
|
||||
- `tests/test_gateway.py`
|
||||
- 验证默认参数透传。
|
||||
- 验证全部自定义搜索选项透传。
|
||||
- 验证 agent owner 与用户鉴权身份分离。
|
||||
- 验证 filters 与 scope 条件使用 `AND` 合并。
|
||||
- 验证五类结果的 `memory_type`。
|
||||
- `README.md`
|
||||
- 更新 `/memories/search` 参数和响应说明。
|
||||
|
||||
## 验收标准
|
||||
|
||||
1. 未提供新字段时，上游收到 `method=hybrid`、`include_profile=true`、
|
||||
`enable_llm_rerank=true`。
|
||||
2. 所有上游搜索选项均能通过 Gateway 请求并原样传递。
|
||||
3. `top_k=-1` 被接受，`top_k=0` 和范围外值被拒绝。
|
||||
4. 自定义 filters 不会覆盖 scope 的资源或聊天 session 隔离条件。
|
||||
5. 设置 `agent_id` 后，上游只收到 `agent_id`，Gateway 仍使用
|
||||
`user_id/user_key` 完成鉴权。
|
||||
6. 每个搜索结果包含准确的 `memory_type`。
|
||||
7. 现有 tombstone、override、资源 URI 隔离测试继续通过。
|
||||
@ -1,36 +0,0 @@
|
||||
# EverOS server settings used by the upstream EverOS process.
|
||||
# Copy this file to everos.env or to the EverOS project .env, then fill secrets.
|
||||
|
||||
# API listener. The Memory Gateway should point EVEROS_BASE_URL at this host/port.
|
||||
EVEROS_API__HOST=127.0.0.1
|
||||
EVEROS_API__PORT=8000
|
||||
|
||||
# Logging
|
||||
EVEROS_LOG_LEVEL=INFO
|
||||
EVEROS_LOG_FORMAT=console
|
||||
TZ=Asia/Shanghai
|
||||
|
||||
# LLM provider
|
||||
EVEROS_LLM__BASE_URL=https://api.openai.com/v1
|
||||
EVEROS_LLM__API_KEY=replace-with-llm-api-key
|
||||
EVEROS_LLM__MODEL=gpt-4o-mini
|
||||
EVEROS_LLM__TIMEOUT_SECONDS=120
|
||||
|
||||
# Embedding provider
|
||||
EVEROS_EMBEDDING__BASE_URL=https://api.openai.com/v1
|
||||
EVEROS_EMBEDDING__API_KEY=replace-with-embedding-api-key
|
||||
EVEROS_EMBEDDING__MODEL=text-embedding-3-small
|
||||
EVEROS_EMBEDDING__TIMEOUT_SECONDS=120
|
||||
|
||||
# Rerank provider
|
||||
EVEROS_RERANK__BASE_URL=https://api.example.com/v1
|
||||
EVEROS_RERANK__API_KEY=replace-with-rerank-api-key
|
||||
EVEROS_RERANK__MODEL=replace-with-rerank-model
|
||||
EVEROS_RERANK__TIMEOUT_SECONDS=120
|
||||
|
||||
# Multimodal parsing provider
|
||||
EVEROS_MULTIMODAL__BASE_URL=https://api.openai.com/v1
|
||||
EVEROS_MULTIMODAL__API_KEY=replace-with-multimodal-api-key
|
||||
EVEROS_MULTIMODAL__MODEL=gpt-4o-mini
|
||||
EVEROS_MULTIMODAL__TIMEOUT_SECONDS=120
|
||||
EVEROS_MULTIMODAL__RESIZE_IMAGES_FOR_VLM=true
|
||||
@ -1,7 +1,7 @@
|
||||
[project]
|
||||
name = "memory-gateway"
|
||||
version = "0.1.0"
|
||||
description = "Lightweight Memory Gateway for EverOS user resources"
|
||||
description = "Lightweight user resource memory gateway"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
"fastapi>=0.104.0",
|
||||
@ -31,5 +31,5 @@ testpaths = ["tests"]
|
||||
python_files = ["test_*.py"]
|
||||
asyncio_mode = "auto"
|
||||
markers = [
|
||||
"integration: tests that call a real EverOS service",
|
||||
"integration: tests that call a real upstream memory service",
|
||||
]
|
||||
|
||||
64
skill/memory-gateway-agent/SKILL.md
Normal file
64
skill/memory-gateway-agent/SKILL.md
Normal file
@ -0,0 +1,64 @@
|
||||
---
|
||||
name: memory-gateway-agent
|
||||
description: Use when an AI agent needs to create or authenticate a Memory Gateway user, upload and manage user resources, add or flush chat memories, search scoped memory, or apply user-approved memory corrections and deletions through the Memory Gateway HTTP API.
|
||||
---
|
||||
|
||||
# Memory Gateway Agent
|
||||
|
||||
Use the bundled CLI instead of constructing HTTP requests manually. It produces JSON output and uses only the Python standard library.
|
||||
|
||||
## Configure
|
||||
|
||||
Set these variables before authenticated operations:
|
||||
|
||||
```bash
|
||||
export MEMORY_GATEWAY_BASE_URL="http://127.0.0.1:8010"
|
||||
export MEMORY_GATEWAY_USER_ID="u_123"
|
||||
export MEMORY_GATEWAY_USER_KEY="uk_xxx"
|
||||
SKILL_DIR="/path/to/memory-gateway-agent"
|
||||
CLI="python $SKILL_DIR/scripts/memory_gateway.py"
|
||||
```
|
||||
|
||||
Do not write a real `user_key` into source files, prompts, logs, or committed documentation. Command-line flags may override environment variables, but environment variables are preferred because process arguments may be observable.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Run `$CLI health` when connectivity or upstream memory service availability is uncertain.
|
||||
2. Use an existing `user_id` and `user_key`. Run `create-user` only when the user explicitly needs a new Gateway identity.
|
||||
3. Choose the operation:
|
||||
- Upload durable user files with `upload-resource`.
|
||||
- Add conversational or multimodal messages with `add-memory`, then call `flush-memory`.
|
||||
- Search with the narrowest useful scope.
|
||||
- List or inspect resources before deleting them.
|
||||
4. Treat JSON output as the source of truth. Preserve returned `resource_id`, memory `id`, and `session_id` exactly.
|
||||
5. For override or deletion, use the memory `id` and `session_id` returned by search. Never invent IDs or apply changes across users.
|
||||
|
||||
## Common Commands
|
||||
|
||||
```bash
|
||||
$CLI health
|
||||
$CLI list-resources
|
||||
$CLI upload-resource ./contract.pdf --title "Contract"
|
||||
$CLI search "What are the payment terms?" --scope resources --top-k 8
|
||||
$CLI search "What did we discuss?" --scope current_chat --conversation-id c_456
|
||||
$CLI override-memory mem_abc --session-id resource:u_123:r_xxx --text "Corrected text"
|
||||
$CLI delete-memory mem_abc --session-id resource:u_123:r_xxx --reason "User requested deletion"
|
||||
```
|
||||
|
||||
For chat ingestion, put the Gateway `messages` array in a JSON file:
|
||||
|
||||
```bash
|
||||
$CLI add-memory --session-id chat:c_456 --messages /tmp/messages.json
|
||||
$CLI flush-memory --session-id chat:c_456
|
||||
```
|
||||
|
||||
Read [references/api.md](references/api.md) when choosing scopes, constructing multimodal messages, interpreting errors, or using less common commands.
|
||||
|
||||
## Safety Rules
|
||||
|
||||
- Do not expose internal file paths. Return the Gateway's `resource://{user_id}/{resource_id}` URI to users.
|
||||
- Do not claim ingestion succeeded unless the upload status is `extracted` or flush reports success.
|
||||
- Treat `health.status = degraded` as Gateway available but upstream memory service unavailable.
|
||||
- Resource deletion is soft deletion in Gateway search scope and removes the Gateway upload copy; it does not delete upstream memory service internal indexes.
|
||||
- Memory override and deletion require an owned `resource:{user_id}:{resource_id}` or `memory_edit:{user_id}` session.
|
||||
- Ask for confirmation before destructive deletion unless the user's current request explicitly instructs deletion.
|
||||
4
skill/memory-gateway-agent/agents/openai.yaml
Normal file
4
skill/memory-gateway-agent/agents/openai.yaml
Normal file
@ -0,0 +1,4 @@
|
||||
interface:
|
||||
display_name: "Memory Gateway Agent"
|
||||
short_description: "Operate Memory Gateway resources and user memories"
|
||||
default_prompt: "Use $memory-gateway-agent to store, search, edit, or delete user memory through Memory Gateway."
|
||||
221
skill/memory-gateway-agent/references/api.md
Normal file
221
skill/memory-gateway-agent/references/api.md
Normal file
@ -0,0 +1,221 @@
|
||||
# Memory Gateway API Reference
|
||||
|
||||
## Contents
|
||||
|
||||
- Configuration
|
||||
- Session IDs
|
||||
- CLI commands
|
||||
- Message format
|
||||
- Search scopes
|
||||
- Errors and result handling
|
||||
|
||||
## Configuration
|
||||
|
||||
The CLI reads:
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `MEMORY_GATEWAY_BASE_URL` | `http://127.0.0.1:8010` | Gateway URL |
|
||||
| `MEMORY_GATEWAY_USER_ID` | none | Authenticated user ID |
|
||||
| `MEMORY_GATEWAY_USER_KEY` | none | Authenticated user key |
|
||||
| `MEMORY_GATEWAY_TIMEOUT_SECONDS` | `120` | HTTP timeout |
|
||||
|
||||
Global CLI flags `--base-url`, `--user-id`, `--user-key`, and `--timeout` override these values. Put global flags before the subcommand.
|
||||
|
||||
## Session IDs
|
||||
|
||||
| Memory source | Format |
|
||||
|---|---|
|
||||
| Chat | `chat:{conversation_id}` |
|
||||
| Uploaded resource | `resource:{user_id}:{resource_id}` |
|
||||
| Manual correction | `memory_edit:{user_id}` |
|
||||
|
||||
Use the exact `session_id` returned by the API whenever possible.
|
||||
|
||||
## CLI Commands
|
||||
|
||||
Assume:
|
||||
|
||||
```bash
|
||||
CLI="python skill/memory-gateway-agent/scripts/memory_gateway.py"
|
||||
```
|
||||
|
||||
### Health
|
||||
|
||||
```bash
|
||||
$CLI health
|
||||
```
|
||||
|
||||
No credentials required. HTTP 200 may contain `"status": "degraded"` when upstream memory service is unavailable.
|
||||
|
||||
### Create User
|
||||
|
||||
```bash
|
||||
$CLI create-user u_123
|
||||
```
|
||||
|
||||
Returns a randomly generated `user_key`. Store it securely. Repeating the same `user_id` returns the existing key.
|
||||
|
||||
### Upload Resource
|
||||
|
||||
```bash
|
||||
$CLI upload-resource ./document.pdf \
|
||||
--app-id default \
|
||||
--project-id default \
|
||||
--title "Document title" \
|
||||
--description "Optional description"
|
||||
```
|
||||
|
||||
Requires credentials. Supported resources depend on the server MIME allowlist. Success returns:
|
||||
|
||||
```json
|
||||
{
|
||||
"resource_id": "r_xxx",
|
||||
"session_id": "resource:u_123:r_xxx",
|
||||
"uri": "resource://u_123/r_xxx",
|
||||
"status": "extracted"
|
||||
}
|
||||
```
|
||||
|
||||
`failed` means the record exists but upstream memory service ingestion failed. Identical active content for the same user/app/project may return the existing resource.
|
||||
|
||||
### List, Get, and Delete Resources
|
||||
|
||||
```bash
|
||||
$CLI list-resources
|
||||
$CLI get-resource r_xxx
|
||||
$CLI delete-resource r_xxx
|
||||
```
|
||||
|
||||
Missing or foreign resource details return `{"resources": []}`. Delete excludes the resource from future `resources` searches.
|
||||
|
||||
### Search
|
||||
|
||||
```bash
|
||||
$CLI search "payment terms" --scope resources --top-k 8
|
||||
$CLI search "previous discussion" \
|
||||
--scope current_chat \
|
||||
--conversation-id c_456
|
||||
$CLI search "known preferences" --scope all_user_memory
|
||||
```
|
||||
|
||||
Repeat `--scope` to combine scopes:
|
||||
|
||||
```bash
|
||||
$CLI search "query" --scope current_chat --scope resources \
|
||||
--conversation-id c_456
|
||||
```
|
||||
|
||||
When no scope is provided, the CLI searches `resources` only unless a conversation ID is supplied; with a conversation ID it searches `current_chat` and `resources`. Explicit `current_chat` scope requires `--conversation-id`.
|
||||
|
||||
Each result includes normalized `id`, `session_id`, `text`, `source_scope`, and optional resource metadata. The `raw` field preserves the upstream memory service response.
|
||||
|
||||
### Add and Flush Memory
|
||||
|
||||
Create `/tmp/messages.json` containing an array:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"sender_id": "u_123",
|
||||
"role": "user",
|
||||
"timestamp": 1781172177000,
|
||||
"content": [
|
||||
{"type": "text", "text": "Remember this note"}
|
||||
]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Then run:
|
||||
|
||||
```bash
|
||||
$CLI add-memory --session-id chat:c_456 --messages /tmp/messages.json
|
||||
$CLI flush-memory --session-id chat:c_456
|
||||
```
|
||||
|
||||
`--messages` accepts either a JSON array string or a path to a JSON file. Always flush after all messages for the session have been added.
|
||||
|
||||
### Override and Delete Memory
|
||||
|
||||
Use IDs from a search result:
|
||||
|
||||
```bash
|
||||
$CLI override-memory mem_abc \
|
||||
--session-id resource:u_123:r_xxx \
|
||||
--text "Corrected memory text"
|
||||
|
||||
$CLI delete-memory mem_abc \
|
||||
--session-id resource:u_123:r_xxx \
|
||||
--reason "User requested deletion"
|
||||
```
|
||||
|
||||
These operations write Gateway overrides or tombstones. They do not modify upstream memory service files directly. The server rejects sessions not owned by the authenticated user.
|
||||
|
||||
## Message Format
|
||||
|
||||
Each message requires:
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| `sender_id` | string | Usually the current user ID |
|
||||
| `role` | string | `user`, `assistant`, or `tool` |
|
||||
| `timestamp` | integer | Unix milliseconds, greater than zero |
|
||||
| `content` | string or array | Text or upstream memory service content items |
|
||||
|
||||
Common content items:
|
||||
|
||||
```json
|
||||
{"type": "text", "text": "Plain text"}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "image",
|
||||
"base64": "BASE64_DATA",
|
||||
"ext": "png",
|
||||
"name": "image.png"
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "audio",
|
||||
"base64": "BASE64_DATA",
|
||||
"ext": "wav",
|
||||
"name": "audio.wav"
|
||||
}
|
||||
```
|
||||
|
||||
Prefer base64 for local binary files. A `file://` URI is only usable when upstream memory service can access the same filesystem path.
|
||||
|
||||
## Search Scopes
|
||||
|
||||
| Scope | Behavior |
|
||||
|---|---|
|
||||
| `current_chat` | Searches `chat:{conversation_id}`; provide `--conversation-id` |
|
||||
| `resources` | Searches extracted, non-deleted resources belonging to the user/app/project |
|
||||
| `all_user_memory` | Searches user memory without a session filter |
|
||||
|
||||
Use the narrowest scope that answers the request. This reduces unrelated results and prevents accidental cross-context use.
|
||||
|
||||
## Errors and Result Handling
|
||||
|
||||
The CLI exits nonzero and writes JSON to stderr:
|
||||
|
||||
```json
|
||||
{"error": "Memory Gateway returned HTTP 401: invalid user credentials"}
|
||||
```
|
||||
|
||||
Common statuses:
|
||||
|
||||
| Status | Meaning |
|
||||
|---|---|
|
||||
| `401` | Invalid `user_id` or `user_key` |
|
||||
| `403` | Memory session is not owned by the user |
|
||||
| `404` | Resource not found for deletion |
|
||||
| `413` | Upload exceeds server size limit |
|
||||
| `415` | MIME type is not allowed |
|
||||
| `422` | Request fields are invalid or missing |
|
||||
|
||||
Do not retry `401`, `403`, `413`, `415`, or `422` without changing the request. Retry connectivity or transient server failures only when appropriate for the caller's workflow.
|
||||
466
skill/memory-gateway-agent/scripts/memory_gateway.py
Executable file
466
skill/memory-gateway-agent/scripts/memory_gateway.py
Executable file
@ -0,0 +1,466 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import mimetypes
|
||||
import os
|
||||
import sys
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.error import HTTPError, URLError
|
||||
from urllib.parse import urlencode
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
|
||||
class GatewayError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class Settings:
|
||||
def __init__(
|
||||
self,
|
||||
base_url: str,
|
||||
user_id: str | None,
|
||||
user_key: str | None,
|
||||
timeout: float = 120.0,
|
||||
) -> None:
|
||||
self.base_url = base_url.rstrip("/")
|
||||
self.user_id = user_id
|
||||
self.user_key = user_key
|
||||
self.timeout = timeout
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> Settings:
|
||||
return cls(
|
||||
base_url=os.environ.get(
|
||||
"MEMORY_GATEWAY_BASE_URL",
|
||||
"http://127.0.0.1:8010",
|
||||
),
|
||||
user_id=os.environ.get("MEMORY_GATEWAY_USER_ID"),
|
||||
user_key=os.environ.get("MEMORY_GATEWAY_USER_KEY"),
|
||||
timeout=float(os.environ.get("MEMORY_GATEWAY_TIMEOUT_SECONDS", "120")),
|
||||
)
|
||||
|
||||
|
||||
class MemoryGatewayClient:
|
||||
def __init__(
|
||||
self,
|
||||
base_url: str,
|
||||
*,
|
||||
user_id: str | None = None,
|
||||
user_key: str | None = None,
|
||||
timeout: float = 120.0,
|
||||
) -> None:
|
||||
self.base_url = base_url.rstrip("/")
|
||||
self.user_id = user_id
|
||||
self.user_key = user_key
|
||||
self.timeout = timeout
|
||||
|
||||
def _credentials(self) -> dict[str, str]:
|
||||
if not self.user_id or not self.user_key:
|
||||
raise GatewayError(
|
||||
"user credentials are required; set MEMORY_GATEWAY_USER_ID and "
|
||||
"MEMORY_GATEWAY_USER_KEY or pass --user-id and --user-key"
|
||||
)
|
||||
return {"user_id": self.user_id, "user_key": self.user_key}
|
||||
|
||||
def _request(
|
||||
self,
|
||||
method: str,
|
||||
path: str,
|
||||
*,
|
||||
query: dict[str, Any] | None = None,
|
||||
json_body: dict[str, Any] | None = None,
|
||||
body: bytes | None = None,
|
||||
headers: dict[str, str] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
url = f"{self.base_url}{path}"
|
||||
if query:
|
||||
url = f"{url}?{urlencode(query, doseq=True)}"
|
||||
request_headers = dict(headers or {})
|
||||
request_body = body
|
||||
if json_body is not None:
|
||||
request_body = json.dumps(json_body, ensure_ascii=False).encode("utf-8")
|
||||
request_headers["Content-Type"] = "application/json"
|
||||
request = Request(
|
||||
url,
|
||||
data=request_body,
|
||||
headers=request_headers,
|
||||
method=method,
|
||||
)
|
||||
try:
|
||||
with urlopen(request, timeout=self.timeout) as response:
|
||||
raw = response.read()
|
||||
except HTTPError as exc:
|
||||
raw = exc.read()
|
||||
detail = _error_detail(raw, exc.reason)
|
||||
raise GatewayError(f"Memory Gateway returned HTTP {exc.code}: {detail}") from exc
|
||||
except URLError as exc:
|
||||
raise GatewayError(f"cannot connect to Memory Gateway: {exc.reason}") from exc
|
||||
if not raw:
|
||||
return {}
|
||||
try:
|
||||
value = json.loads(raw.decode("utf-8"))
|
||||
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||
raise GatewayError("Memory Gateway returned a non-JSON response") from exc
|
||||
if not isinstance(value, dict):
|
||||
raise GatewayError("Memory Gateway returned an unexpected JSON response")
|
||||
return value
|
||||
|
||||
def health(self) -> dict[str, Any]:
|
||||
return self._request("GET", "/health")
|
||||
|
||||
def create_user(self, user_id: str) -> dict[str, Any]:
|
||||
return self._request("POST", "/users", json_body={"user_id": user_id})
|
||||
|
||||
def upload_resource(
|
||||
self,
|
||||
file_path: Path,
|
||||
*,
|
||||
app_id: str = "default",
|
||||
project_id: str = "default",
|
||||
title: str | None = None,
|
||||
description: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
if not file_path.is_file():
|
||||
raise GatewayError(f"upload file does not exist: {file_path}")
|
||||
fields: dict[str, str] = {
|
||||
**self._credentials(),
|
||||
"app_id": app_id,
|
||||
"project_id": project_id,
|
||||
}
|
||||
if title is not None:
|
||||
fields["title"] = title
|
||||
if description is not None:
|
||||
fields["description"] = description
|
||||
boundary = f"memory-gateway-{uuid.uuid4().hex}"
|
||||
mime_type = mimetypes.guess_type(file_path.name)[0] or "application/octet-stream"
|
||||
body = _multipart_body(
|
||||
boundary,
|
||||
fields,
|
||||
field_name="file",
|
||||
file_path=file_path,
|
||||
mime_type=mime_type,
|
||||
)
|
||||
return self._request(
|
||||
"POST",
|
||||
"/resources",
|
||||
body=body,
|
||||
headers={"Content-Type": f"multipart/form-data; boundary={boundary}"},
|
||||
)
|
||||
|
||||
def list_resources(self) -> dict[str, Any]:
|
||||
return self._request("GET", "/resources", query=self._credentials())
|
||||
|
||||
def get_resource(self, resource_id: str) -> dict[str, Any]:
|
||||
return self._request(
|
||||
"GET",
|
||||
f"/resources/{resource_id}",
|
||||
query=self._credentials(),
|
||||
)
|
||||
|
||||
def delete_resource(self, resource_id: str) -> dict[str, Any]:
|
||||
return self._request(
|
||||
"DELETE",
|
||||
f"/resources/{resource_id}",
|
||||
query=self._credentials(),
|
||||
)
|
||||
|
||||
def search(
|
||||
self,
|
||||
query: str,
|
||||
*,
|
||||
conversation_id: str | None = None,
|
||||
scopes: list[str] | None = None,
|
||||
top_k: int = 8,
|
||||
app_id: str = "default",
|
||||
project_id: str = "default",
|
||||
) -> dict[str, Any]:
|
||||
selected_scopes = scopes or (
|
||||
["current_chat", "resources"] if conversation_id else ["resources"]
|
||||
)
|
||||
if "current_chat" in selected_scopes and not conversation_id:
|
||||
raise GatewayError(
|
||||
"conversation_id is required when search scope includes current_chat"
|
||||
)
|
||||
payload: dict[str, Any] = {
|
||||
**self._credentials(),
|
||||
"query": query,
|
||||
"scope": selected_scopes,
|
||||
"top_k": top_k,
|
||||
"app_id": app_id,
|
||||
"project_id": project_id,
|
||||
}
|
||||
if conversation_id is not None:
|
||||
payload["conversation_id"] = conversation_id
|
||||
return self._request("POST", "/memories/search", json_body=payload)
|
||||
|
||||
def add_memory(
|
||||
self,
|
||||
session_id: str,
|
||||
messages: list[dict[str, Any]],
|
||||
*,
|
||||
app_id: str = "default",
|
||||
project_id: str = "default",
|
||||
) -> dict[str, Any]:
|
||||
return self._request(
|
||||
"POST",
|
||||
"/memories/add",
|
||||
json_body={
|
||||
**self._credentials(),
|
||||
"session_id": session_id,
|
||||
"messages": messages,
|
||||
"app_id": app_id,
|
||||
"project_id": project_id,
|
||||
},
|
||||
)
|
||||
|
||||
def flush_memory(
|
||||
self,
|
||||
session_id: str,
|
||||
*,
|
||||
app_id: str = "default",
|
||||
project_id: str = "default",
|
||||
) -> dict[str, Any]:
|
||||
return self._request(
|
||||
"POST",
|
||||
"/memories/flush",
|
||||
json_body={
|
||||
**self._credentials(),
|
||||
"session_id": session_id,
|
||||
"app_id": app_id,
|
||||
"project_id": project_id,
|
||||
},
|
||||
)
|
||||
|
||||
def override_memory(
|
||||
self,
|
||||
memory_id: str,
|
||||
session_id: str,
|
||||
override_text: str,
|
||||
) -> dict[str, Any]:
|
||||
return self._request(
|
||||
"PATCH",
|
||||
f"/memories/{memory_id}",
|
||||
json_body={
|
||||
**self._credentials(),
|
||||
"session_id": session_id,
|
||||
"override_text": override_text,
|
||||
},
|
||||
)
|
||||
|
||||
def delete_memory(
|
||||
self,
|
||||
memory_id: str,
|
||||
session_id: str,
|
||||
*,
|
||||
reason: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
payload: dict[str, Any] = {
|
||||
**self._credentials(),
|
||||
"session_id": session_id,
|
||||
}
|
||||
if reason is not None:
|
||||
payload["reason"] = reason
|
||||
return self._request(
|
||||
"DELETE",
|
||||
f"/memories/{memory_id}",
|
||||
json_body=payload,
|
||||
)
|
||||
|
||||
|
||||
def _error_detail(raw: bytes, fallback: Any) -> str:
|
||||
try:
|
||||
body = json.loads(raw.decode("utf-8"))
|
||||
except (UnicodeDecodeError, json.JSONDecodeError):
|
||||
return str(fallback)
|
||||
if isinstance(body, dict) and body.get("detail"):
|
||||
return str(body["detail"])
|
||||
return str(fallback)
|
||||
|
||||
|
||||
def _multipart_body(
|
||||
boundary: str,
|
||||
fields: dict[str, str],
|
||||
*,
|
||||
field_name: str,
|
||||
file_path: Path,
|
||||
mime_type: str,
|
||||
) -> bytes:
|
||||
marker = boundary.encode("ascii")
|
||||
chunks: list[bytes] = []
|
||||
for name, value in fields.items():
|
||||
chunks.extend(
|
||||
[
|
||||
b"--" + marker + b"\r\n",
|
||||
f'Content-Disposition: form-data; name="{name}"\r\n\r\n'.encode(),
|
||||
value.encode("utf-8"),
|
||||
b"\r\n",
|
||||
]
|
||||
)
|
||||
chunks.extend(
|
||||
[
|
||||
b"--" + marker + b"\r\n",
|
||||
(
|
||||
f'Content-Disposition: form-data; name="{field_name}"; '
|
||||
f'filename="{file_path.name}"\r\n'
|
||||
).encode(),
|
||||
f"Content-Type: {mime_type}\r\n\r\n".encode(),
|
||||
file_path.read_bytes(),
|
||||
b"\r\n--" + marker + b"--\r\n",
|
||||
]
|
||||
)
|
||||
return b"".join(chunks)
|
||||
|
||||
|
||||
def _load_json_array(value: str) -> list[dict[str, Any]]:
|
||||
source = Path(value)
|
||||
text = source.read_text(encoding="utf-8") if source.is_file() else value
|
||||
try:
|
||||
parsed = json.loads(text)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise GatewayError(f"invalid messages JSON: {exc}") from exc
|
||||
if not isinstance(parsed, list) or not all(isinstance(item, dict) for item in parsed):
|
||||
raise GatewayError("messages JSON must be an array of objects")
|
||||
return parsed
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description="Memory Gateway agent CLI")
|
||||
parser.add_argument("--base-url")
|
||||
parser.add_argument("--user-id")
|
||||
parser.add_argument("--user-key")
|
||||
parser.add_argument("--timeout", type=float)
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
subparsers.add_parser("health")
|
||||
create_user = subparsers.add_parser("create-user")
|
||||
create_user.add_argument("user_id")
|
||||
|
||||
upload = subparsers.add_parser("upload-resource")
|
||||
upload.add_argument("file", type=Path)
|
||||
_add_scope_arguments(upload)
|
||||
upload.add_argument("--title")
|
||||
upload.add_argument("--description")
|
||||
|
||||
subparsers.add_parser("list-resources")
|
||||
get_resource = subparsers.add_parser("get-resource")
|
||||
get_resource.add_argument("resource_id")
|
||||
delete_resource = subparsers.add_parser("delete-resource")
|
||||
delete_resource.add_argument("resource_id")
|
||||
|
||||
search = subparsers.add_parser("search")
|
||||
search.add_argument("query")
|
||||
search.add_argument("--conversation-id")
|
||||
search.add_argument(
|
||||
"--scope",
|
||||
action="append",
|
||||
choices=["current_chat", "resources", "all_user_memory"],
|
||||
)
|
||||
search.add_argument("--top-k", type=int, default=8)
|
||||
_add_scope_arguments(search)
|
||||
|
||||
add = subparsers.add_parser("add-memory")
|
||||
add.add_argument("--session-id", required=True)
|
||||
add.add_argument(
|
||||
"--messages",
|
||||
required=True,
|
||||
help="JSON array or path to a JSON file containing messages",
|
||||
)
|
||||
_add_scope_arguments(add)
|
||||
|
||||
flush = subparsers.add_parser("flush-memory")
|
||||
flush.add_argument("--session-id", required=True)
|
||||
_add_scope_arguments(flush)
|
||||
|
||||
override = subparsers.add_parser("override-memory")
|
||||
override.add_argument("memory_id")
|
||||
override.add_argument("--session-id", required=True)
|
||||
override.add_argument("--text", required=True)
|
||||
|
||||
delete_memory = subparsers.add_parser("delete-memory")
|
||||
delete_memory.add_argument("memory_id")
|
||||
delete_memory.add_argument("--session-id", required=True)
|
||||
delete_memory.add_argument("--reason")
|
||||
return parser
|
||||
|
||||
|
||||
def _add_scope_arguments(parser: argparse.ArgumentParser) -> None:
|
||||
parser.add_argument("--app-id", default="default")
|
||||
parser.add_argument("--project-id", default="default")
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
settings = Settings.from_env()
|
||||
args = build_parser().parse_args(argv)
|
||||
client = MemoryGatewayClient(
|
||||
args.base_url or settings.base_url,
|
||||
user_id=args.user_id or settings.user_id,
|
||||
user_key=args.user_key or settings.user_key,
|
||||
timeout=args.timeout or settings.timeout,
|
||||
)
|
||||
try:
|
||||
result = _run_command(client, args)
|
||||
except GatewayError as exc:
|
||||
print(json.dumps({"error": str(exc)}, ensure_ascii=False), file=sys.stderr)
|
||||
return 1
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
def _run_command(client: MemoryGatewayClient, args: argparse.Namespace) -> dict[str, Any]:
|
||||
if args.command == "health":
|
||||
return client.health()
|
||||
if args.command == "create-user":
|
||||
return client.create_user(args.user_id)
|
||||
if args.command == "upload-resource":
|
||||
return client.upload_resource(
|
||||
args.file,
|
||||
app_id=args.app_id,
|
||||
project_id=args.project_id,
|
||||
title=args.title,
|
||||
description=args.description,
|
||||
)
|
||||
if args.command == "list-resources":
|
||||
return client.list_resources()
|
||||
if args.command == "get-resource":
|
||||
return client.get_resource(args.resource_id)
|
||||
if args.command == "delete-resource":
|
||||
return client.delete_resource(args.resource_id)
|
||||
if args.command == "search":
|
||||
return client.search(
|
||||
args.query,
|
||||
conversation_id=args.conversation_id,
|
||||
scopes=args.scope,
|
||||
top_k=args.top_k,
|
||||
app_id=args.app_id,
|
||||
project_id=args.project_id,
|
||||
)
|
||||
if args.command == "add-memory":
|
||||
return client.add_memory(
|
||||
args.session_id,
|
||||
_load_json_array(args.messages),
|
||||
app_id=args.app_id,
|
||||
project_id=args.project_id,
|
||||
)
|
||||
if args.command == "flush-memory":
|
||||
return client.flush_memory(
|
||||
args.session_id,
|
||||
app_id=args.app_id,
|
||||
project_id=args.project_id,
|
||||
)
|
||||
if args.command == "override-memory":
|
||||
return client.override_memory(args.memory_id, args.session_id, args.text)
|
||||
if args.command == "delete-memory":
|
||||
return client.delete_memory(
|
||||
args.memory_id,
|
||||
args.session_id,
|
||||
reason=args.reason,
|
||||
)
|
||||
raise GatewayError(f"unsupported command: {args.command}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@ -9,31 +9,31 @@ import pytest
|
||||
|
||||
from core.api import create_app
|
||||
from core.config import GatewayConfig
|
||||
from core.everos_client import EverOSClient
|
||||
from core.backend_client import BackendClient
|
||||
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
def _integration_enabled() -> bool:
|
||||
return os.environ.get("RUN_EVEROS_INTEGRATION") == "1"
|
||||
return os.environ.get("RUN_BACKEND_INTEGRATION") == "1"
|
||||
|
||||
|
||||
def _ingest_integration_enabled() -> bool:
|
||||
return os.environ.get("RUN_EVEROS_INGEST_INTEGRATION") == "1"
|
||||
return os.environ.get("RUN_BACKEND_INGEST_INTEGRATION") == "1"
|
||||
|
||||
|
||||
def _everos_base_url() -> str:
|
||||
return os.environ.get("EVEROS_BASE_URL", "http://127.0.0.1:1995")
|
||||
def _backend_base_url() -> str:
|
||||
return os.environ.get("MEMORY_GATEWAY_BACKEND_BASE_URL", "http://127.0.0.1:1995")
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
not _integration_enabled(),
|
||||
reason="set RUN_EVEROS_INTEGRATION=1 to run against a real EverOS service",
|
||||
reason="set RUN_BACKEND_INTEGRATION=1 to run against an upstream memory service",
|
||||
)
|
||||
@pytest.mark.asyncio
|
||||
async def test_real_everos_health_check() -> None:
|
||||
client = EverOSClient(_everos_base_url(), timeout=10)
|
||||
async def test_real_backend_health_check() -> None:
|
||||
client = BackendClient(_backend_base_url(), timeout=10)
|
||||
|
||||
health = await client.health_check()
|
||||
|
||||
@ -43,17 +43,17 @@ async def test_real_everos_health_check() -> None:
|
||||
@pytest.mark.skipif(
|
||||
not _ingest_integration_enabled(),
|
||||
reason=(
|
||||
"set RUN_EVEROS_INGEST_INTEGRATION=1 to run real EverOS add/flush ingestion"
|
||||
"set RUN_BACKEND_INGEST_INTEGRATION=1 to run upstream add/flush ingestion"
|
||||
),
|
||||
)
|
||||
@pytest.mark.asyncio
|
||||
async def test_gateway_uploads_text_resource_to_real_everos(tmp_path: Path) -> None:
|
||||
async def test_gateway_uploads_text_resource_to_real_backend(tmp_path: Path) -> None:
|
||||
config = GatewayConfig(
|
||||
everos_base_url=_everos_base_url(),
|
||||
backend_base_url=_backend_base_url(),
|
||||
database_path=tmp_path / "gateway.sqlite3",
|
||||
storage_dir=tmp_path / "storage",
|
||||
everos_ingest_attempts=1,
|
||||
everos_timeout_seconds=30,
|
||||
backend_ingest_attempts=1,
|
||||
backend_timeout_seconds=30,
|
||||
)
|
||||
app = create_app(config=config)
|
||||
transport = httpx.ASGITransport(app=app)
|
||||
@ -69,8 +69,8 @@ async def test_gateway_uploads_text_resource_to_real_everos(tmp_path: Path) -> N
|
||||
data={"user_id": user_id, "user_key": user_key},
|
||||
files={
|
||||
"file": (
|
||||
"real-everos.txt",
|
||||
b"real everos integration",
|
||||
"integration.txt",
|
||||
b"upstream memory service integration",
|
||||
"text/plain",
|
||||
)
|
||||
},
|
||||
35
tests/test_branding.py
Normal file
35
tests/test_branding.py
Normal file
@ -0,0 +1,35 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
FORBIDDEN_TOKEN = "ever" + "os"
|
||||
SKIPPED_PARTS = {
|
||||
".git",
|
||||
".pytest_cache",
|
||||
".venv",
|
||||
"__pycache__",
|
||||
"data",
|
||||
}
|
||||
|
||||
|
||||
def test_current_project_does_not_expose_upstream_product_name() -> None:
|
||||
matches: list[str] = []
|
||||
for path in PROJECT_ROOT.rglob("*"):
|
||||
relative = path.relative_to(PROJECT_ROOT)
|
||||
if any(part in SKIPPED_PARTS for part in relative.parts):
|
||||
continue
|
||||
if FORBIDDEN_TOKEN in path.name.lower():
|
||||
matches.append(f"filename: {relative}")
|
||||
if not path.is_file():
|
||||
continue
|
||||
try:
|
||||
text = path.read_text(encoding="utf-8")
|
||||
except UnicodeDecodeError:
|
||||
continue
|
||||
for line_number, line in enumerate(text.splitlines(), start=1):
|
||||
if FORBIDDEN_TOKEN in line.lower():
|
||||
matches.append(f"content: {relative}:{line_number}")
|
||||
|
||||
assert matches == []
|
||||
@ -1,6 +1,6 @@
|
||||
# Memory Gateway multimodal API test
|
||||
|
||||
This file records a real end-to-end test through **Memory Gateway**, not direct EverOS calls.
|
||||
This file records a real end-to-end test through **Memory Gateway**, not direct upstream memory service calls.
|
||||
|
||||
Gateway URL used by curl:
|
||||
|
||||
@ -8,7 +8,7 @@ Gateway URL used by curl:
|
||||
http://127.0.0.1:8010
|
||||
```
|
||||
|
||||
Gateway upstream EverOS:
|
||||
Gateway upstream memory service:
|
||||
|
||||
```text
|
||||
http://10.6.80.123:1995
|
||||
@ -35,7 +35,7 @@ Command:
|
||||
```bash
|
||||
cd /home/tom/memory-gateway
|
||||
|
||||
EVEROS_BASE_URL=http://10.6.80.123:1995 \
|
||||
MEMORY_GATEWAY_BACKEND_BASE_URL=http://10.6.80.123:1995 \
|
||||
MEMORY_GATEWAY_DB_PATH=/tmp/memory_gateway_curl.sqlite3 \
|
||||
MEMORY_GATEWAY_STORAGE_DIR=/tmp/memory_gateway_curl_storage \
|
||||
MEMORY_GATEWAY_HOST=127.0.0.1 \
|
||||
@ -57,11 +57,13 @@ INFO: Uvicorn running on http://127.0.0.1:8010 (Press CTRL+C to quit)
|
||||
Request:
|
||||
|
||||
```bash
|
||||
USER_ID="gateway_user_20260611180257"
|
||||
|
||||
curl -sS --location 'http://127.0.0.1:8010/users' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{
|
||||
"user_id": "gateway_user_20260611180257"
|
||||
}'
|
||||
--data "{
|
||||
\"user_id\": \"${USER_ID}\"
|
||||
}"
|
||||
```
|
||||
|
||||
Response:
|
||||
@ -88,7 +90,6 @@ Request:
|
||||
```bash
|
||||
cd /home/tom/memory-gateway
|
||||
|
||||
USER_ID="gateway_user_20260611180257"
|
||||
USER_KEY="uk_REDACTED"
|
||||
CONVERSATION_ID="gateway-multimodal-20260611180257"
|
||||
SESSION_ID="chat:${CONVERSATION_ID}"
|
||||
@ -136,7 +137,7 @@ Response:
|
||||
```json
|
||||
{
|
||||
"session_id": "chat:gateway-multimodal-20260611180257",
|
||||
"everos": {
|
||||
"backend": {
|
||||
"request_id": "c9e24b8d27ee4ad08a8df70273336637",
|
||||
"data": {
|
||||
"message_count": 1,
|
||||
@ -160,13 +161,13 @@ Request:
|
||||
```bash
|
||||
curl -sS --location 'http://127.0.0.1:8010/memories/flush' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{
|
||||
"user_id": "gateway_user_20260611180257",
|
||||
"user_key": "uk_REDACTED",
|
||||
"session_id": "chat:gateway-multimodal-20260611180257",
|
||||
"app_id": "default",
|
||||
"project_id": "default"
|
||||
}'
|
||||
--data "{
|
||||
\"user_id\": \"${USER_ID}\",
|
||||
\"user_key\": \"${USER_KEY}\",
|
||||
\"session_id\": \"${SESSION_ID}\",
|
||||
\"app_id\": \"default\",
|
||||
\"project_id\": \"default\"
|
||||
}"
|
||||
```
|
||||
|
||||
Response:
|
||||
@ -174,7 +175,7 @@ Response:
|
||||
```json
|
||||
{
|
||||
"session_id": "chat:gateway-multimodal-20260611180257",
|
||||
"everos": {
|
||||
"backend": {
|
||||
"request_id": "8eb7d5db2d3b43f4999f445aabb813b1",
|
||||
"data": {
|
||||
"status": "extracted"
|
||||
@ -192,7 +193,7 @@ TOTAL_TIME:2.135721
|
||||
|
||||
## 4. Search through Gateway
|
||||
|
||||
EverOS indexing can lag briefly after `flush`, so this test waited about 2 seconds before searching.
|
||||
upstream memory service indexing can lag briefly after `flush`, so this test waited about 2 seconds before searching.
|
||||
|
||||
Request:
|
||||
|
||||
@ -201,16 +202,16 @@ sleep 2
|
||||
|
||||
curl -sS --location 'http://127.0.0.1:8010/memories/search' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{
|
||||
"user_id": "gateway_user_20260611180257",
|
||||
"user_key": "uk_REDACTED",
|
||||
"conversation_id": "gateway-multimodal-20260611180257",
|
||||
"query": "图片里的蓝色圆形在哪里？音频是什么？",
|
||||
"scope": ["current_chat"],
|
||||
"top_k": 5,
|
||||
"app_id": "default",
|
||||
"project_id": "default"
|
||||
}'
|
||||
--data "{
|
||||
\"user_id\": \"${USER_ID}\",
|
||||
\"user_key\": \"${USER_KEY}\",
|
||||
\"conversation_id\": \"${CONVERSATION_ID}\",
|
||||
\"query\": \"图片里的蓝色圆形在哪里？音频是什么？\",
|
||||
\"scope\": [\"current_chat\"],
|
||||
\"top_k\": 5,
|
||||
\"app_id\": \"default\",
|
||||
\"project_id\": \"default\"
|
||||
}"
|
||||
```
|
||||
|
||||
Response:
|
||||
@ -279,7 +280,7 @@ Response:
|
||||
{
|
||||
"status": "ok",
|
||||
"api": {"status": "ok"},
|
||||
"everos": {
|
||||
"backend": {
|
||||
"status": "ok",
|
||||
"base_url": "http://10.6.80.123:1995",
|
||||
"data": {"status": "ok"}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
204
tests/test_memory_gateway_skill.py
Normal file
204
tests/test_memory_gateway_skill.py
Normal file
@ -0,0 +1,204 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
from urllib.error import HTTPError
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
SCRIPT_PATH = (
|
||||
Path(__file__).parents[1]
|
||||
/ "skill"
|
||||
/ "memory-gateway-agent"
|
||||
/ "scripts"
|
||||
/ "memory_gateway.py"
|
||||
)
|
||||
|
||||
|
||||
def load_cli():
|
||||
spec = importlib.util.spec_from_file_location("memory_gateway_skill_cli", SCRIPT_PATH)
|
||||
assert spec is not None and spec.loader is not None
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
class FakeResponse:
|
||||
def __init__(self, body: dict[str, object], status: int = 200) -> None:
|
||||
self.body = json.dumps(body).encode()
|
||||
self.status = status
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *args: object) -> None:
|
||||
return None
|
||||
|
||||
def read(self) -> bytes:
|
||||
return self.body
|
||||
|
||||
def close(self) -> None:
|
||||
return None
|
||||
|
||||
|
||||
def test_settings_read_gateway_credentials_from_environment(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
cli = load_cli()
|
||||
monkeypatch.setenv("MEMORY_GATEWAY_BASE_URL", "http://gateway.test/")
|
||||
monkeypatch.setenv("MEMORY_GATEWAY_USER_ID", "u_agent")
|
||||
monkeypatch.setenv("MEMORY_GATEWAY_USER_KEY", "uk_secret")
|
||||
|
||||
settings = cli.Settings.from_env()
|
||||
|
||||
assert settings.base_url == "http://gateway.test"
|
||||
assert settings.user_id == "u_agent"
|
||||
assert settings.user_key == "uk_secret"
|
||||
|
||||
|
||||
def test_json_request_sends_authenticated_payload(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
cli = load_cli()
|
||||
captured: dict[str, object] = {}
|
||||
|
||||
def fake_urlopen(request, timeout):
|
||||
captured["url"] = request.full_url
|
||||
captured["method"] = request.method
|
||||
captured["body"] = json.loads(request.data)
|
||||
captured["content_type"] = request.headers["Content-type"]
|
||||
captured["timeout"] = timeout
|
||||
return FakeResponse({"results": []})
|
||||
|
||||
monkeypatch.setattr(cli, "urlopen", fake_urlopen)
|
||||
client = cli.MemoryGatewayClient(
|
||||
"http://gateway.test",
|
||||
user_id="u_agent",
|
||||
user_key="uk_secret",
|
||||
timeout=9,
|
||||
)
|
||||
|
||||
result = client.search("contract", scopes=["resources"], top_k=5)
|
||||
|
||||
assert result == {"results": []}
|
||||
assert captured == {
|
||||
"url": "http://gateway.test/memories/search",
|
||||
"method": "POST",
|
||||
"body": {
|
||||
"user_id": "u_agent",
|
||||
"user_key": "uk_secret",
|
||||
"query": "contract",
|
||||
"scope": ["resources"],
|
||||
"top_k": 5,
|
||||
"app_id": "default",
|
||||
"project_id": "default",
|
||||
},
|
||||
"content_type": "application/json",
|
||||
"timeout": 9,
|
||||
}
|
||||
|
||||
|
||||
def test_upload_builds_multipart_request_without_exposing_file_uri(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
cli = load_cli()
|
||||
upload = tmp_path / "note.txt"
|
||||
upload.write_text("remember this", encoding="utf-8")
|
||||
captured: dict[str, object] = {}
|
||||
|
||||
def fake_urlopen(request, timeout):
|
||||
captured["url"] = request.full_url
|
||||
captured["method"] = request.method
|
||||
captured["body"] = request.data
|
||||
captured["content_type"] = request.headers["Content-type"]
|
||||
return FakeResponse(
|
||||
{
|
||||
"resource_id": "r_1",
|
||||
"uri": "resource://u_agent/r_1",
|
||||
"status": "extracted",
|
||||
}
|
||||
)
|
||||
|
||||
monkeypatch.setattr(cli, "urlopen", fake_urlopen)
|
||||
client = cli.MemoryGatewayClient(
|
||||
"http://gateway.test",
|
||||
user_id="u_agent",
|
||||
user_key="uk_secret",
|
||||
)
|
||||
|
||||
result = client.upload_resource(upload, title="Agent note")
|
||||
|
||||
body = captured["body"]
|
||||
assert isinstance(body, bytes)
|
||||
assert captured["url"] == "http://gateway.test/resources"
|
||||
assert captured["method"] == "POST"
|
||||
assert str(captured["content_type"]).startswith("multipart/form-data; boundary=")
|
||||
assert b'name="user_id"' in body
|
||||
assert b"u_agent" in body
|
||||
assert b'name="file"; filename="note.txt"' in body
|
||||
assert b"remember this" in body
|
||||
assert b"file://" not in body
|
||||
assert result["uri"] == "resource://u_agent/r_1"
|
||||
|
||||
|
||||
def test_http_error_raises_gateway_error_without_leaking_user_key(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
cli = load_cli()
|
||||
|
||||
def fake_urlopen(request, timeout):
|
||||
raise HTTPError(
|
||||
request.full_url,
|
||||
401,
|
||||
"Unauthorized",
|
||||
hdrs=None,
|
||||
fp=FakeResponse({"detail": "invalid user credentials"}),
|
||||
)
|
||||
|
||||
monkeypatch.setattr(cli, "urlopen", fake_urlopen)
|
||||
client = cli.MemoryGatewayClient(
|
||||
"http://gateway.test",
|
||||
user_id="u_agent",
|
||||
user_key="uk_super_secret",
|
||||
)
|
||||
|
||||
with pytest.raises(cli.GatewayError) as exc_info:
|
||||
client.list_resources()
|
||||
|
||||
message = str(exc_info.value)
|
||||
assert "401" in message
|
||||
assert "invalid user credentials" in message
|
||||
assert "uk_super_secret" not in message
|
||||
|
||||
|
||||
def test_load_messages_accepts_large_inline_json() -> None:
|
||||
cli = load_cli()
|
||||
value = json.dumps(
|
||||
[
|
||||
{
|
||||
"sender_id": "u_agent",
|
||||
"role": "user",
|
||||
"timestamp": 1234567890123,
|
||||
"content": "x" * 5000,
|
||||
}
|
||||
]
|
||||
)
|
||||
|
||||
messages = cli._load_json_array(value)
|
||||
|
||||
assert messages[0]["content"] == "x" * 5000
|
||||
|
||||
|
||||
def test_search_requires_conversation_id_for_current_chat_scope() -> None:
|
||||
cli = load_cli()
|
||||
client = cli.MemoryGatewayClient(
|
||||
"http://gateway.test",
|
||||
user_id="u_agent",
|
||||
user_key="uk_secret",
|
||||
)
|
||||
|
||||
with pytest.raises(cli.GatewayError, match="conversation_id"):
|
||||
client.search("what did we discuss", scopes=["current_chat"])
|
||||
Reference in New Issue
Block a user