diff --git a/README.md b/README.md index 5228c08..b0f81e1 100644 --- a/README.md +++ b/README.md @@ -1,196 +1,167 @@ # Memory Gateway -Memory Gateway 是一个轻量级 FastAPI 服务，用于在上游记忆服务现有 -`/api/v1/memory/add`、`/api/v1/memory/flush`、`/api/v1/memory/search` -能力之上构建用户资源记忆层。 +Memory Gateway 是一个轻量级 FastAPI 服务，位于调用方和上游 memory service +之间。它负责用户鉴权、文件接收、资源元数据管理、 +附件映射、软删除、手动覆盖，以及按范围编排记忆搜索。 -它只维护 Gateway 自己的 SQLite 元数据表、软删除记录和手动覆盖记录， -不会直接修改上游记忆服务的 Markdown、SQLite 或 LanceDB 内部文件。 +Gateway 不直接修改上游记忆服务的 Markdown、SQLite 或向量索引文件。它通过 +上游现有 API 完成记忆写入、flush 和搜索： -## 功能范围 +- `POST /api/v1/memory/add` +- `POST /api/v1/memory/flush` +- `POST /api/v1/memory/search` -- 上传用户资源：文件、图片、音频、PDF、HTML、普通文档、纯文本。 -- 登记外部资源：文件真实保存在 Beaver user_files/MinIO 等外部存储， - Gateway 只保存元数据和 URI 映射，不保存原始文件副本。 -- 保存资源元数据到 SQLite。 -- 为每个资源生成独立的上游记忆服务 `session_id`。 -- 调用上游记忆服务的 `add` 和 `flush` 完成资源记忆摄入，并对临时失败做轻量重试。 -- 提供资源列表、详情、软删除。 -- 支持上传大小限制、MIME 白名单、同用户同 app/project 下按 sha256 幂等复用资源。 -- 编排记忆搜索，支持当前聊天、资源记忆、全部用户记忆。 -- 支持记忆 tombstone 软删除。 -- 支持记忆手动 override。 -- 搜索结果返回前自动过滤 tombstone 并应用 override。 +## 适用场景 -## 目录结构 +- 客户端要把聊天消息写入记忆。 +- 客户端要在同一个聊天 session 中上传图片、音频或文件，但不能自行转 base64。 +- 用户要上传独立资源，如图片、音频、PDF、HTML、文档、文本文件。 +- 文件已经在外部存储中，需要登记长期 URI 和本次摄入 URI。 +- 搜索时需要同时覆盖当前聊天、资源记忆、全部用户记忆。 +- 需要在 Gateway 层实现用户隔离、资源软删除、记忆 tombstone 和人工覆盖。 + +## 关键原则 + +`file://` 不是上传协议。它只表示“某台机器上的本地路径”。如果客户端和 Gateway +不在同一台机器、同一个容器、或同一个挂载路径下，直接把客户端本机 `file://` +传给 Gateway 或上游服务会失败。 + +正确选择： + +| 需求 | 推荐接口 | +|---|---| +| 聊天消息里带本地文件，且客户端不能转 base64 | `POST /memories/add/multipart` | +| 上传一个独立、长期可检索资源 | `POST /resources` | +| 文件已经在 MinIO/S3/Beaver user_files 等外部存储 | `POST /resources/external` | +| 调用方已经有 base64 或上游可访问 URI | `POST /memories/add` | + +## 项目结构 ```text -/home/tom/memory-gateway -├── core/ # Gateway 核心代码 -│ ├── api.py # FastAPI 路由 -│ ├── config.py # 环境变量配置 -│ ├── db.py # SQLite schema 初始化 -│ ├── backend_client.py # 上游记忆服务 HTTP client -│ ├── repository.py # SQLite 读写 -│ └── service.py # 业务编排 -├── main.py # Python 启动入口 -├── tests/ # 测试 -├── .env.example # 环境变量示例 -└── pyproject.toml +memory-gateway/ +├── core/ +│ ├── api.py # FastAPI 路由、请求校验、HTTP 错误映射 +│ ├── backend_client.py # 上游 memory service HTTP client +│ ├── config.py # 环境变量配置 +│ ├── db.py # SQLite schema 初始化 +│ ├── repository.py # SQLite 读写 +│ └── service.py # 业务编排、上传、附件、搜索、软删除 +├── main.py # uvicorn 启动入口 +├── skill/ # Memory Gateway agent skill 和 CLI +├── tests/ # 单元、集成、实测命令记录 +├── pyproject.toml +└── uv.lock ``` -## 环境配置 +## 安装和启动 -复制示例配置： - -```bash -cd /home/tom/memory-gateway -cp .env.example .env -``` - -配置项说明： - -| 变量 | 默认值 | 说明 | -|---|---|---| -| `MEMORY_GATEWAY_BACKEND_BASE_URL` | `http://127.0.0.1:1995` | 上游记忆服务 API 地址；服务端可监听 `0.0.0.0:1995`，本机客户端通常连接 `127.0.0.1:1995` | -| `MEMORY_GATEWAY_DB_PATH` | `./data/memory_gateway.sqlite3` | Gateway 自己的 SQLite 数据库路径 | -| `MEMORY_GATEWAY_STORAGE_DIR` | `./data/storage` | 用户上传原始文件保存路径 | -| `MEMORY_GATEWAY_RESOURCE_SEARCH_BATCH_SIZE` | `50` | resources scope 搜索时每批 session_id 数量 | -| `MEMORY_GATEWAY_MAX_UPLOAD_BYTES` | `26214400` | 单个上传文件最大字节数，默认 25MB | -| `MEMORY_GATEWAY_ALLOWED_MIME_TYPES` | 常见图片、音频、PDF、HTML、文本和 Office 文档 | 逗号分隔的上传 MIME 白名单，支持 `image/*` 这类前缀匹配 | -| `MEMORY_GATEWAY_BACKEND_INGEST_ATTEMPTS` | `3` | 上游记忆服务 `add` 和 `flush` 各自最多重试次数 | -| `MEMORY_GATEWAY_BACKEND_RETRY_DELAY_SECONDS` | `0.25` | 上游记忆服务摄入重试间隔秒数 | -| `MEMORY_GATEWAY_BACKEND_TIMEOUT_SECONDS` | `120` | 单次上游记忆服务 HTTP 请求超时秒数 | -| `MEMORY_GATEWAY_HOST` | `127.0.0.1` | Gateway API 监听地址 | -| `MEMORY_GATEWAY_PORT` | `8010` | Gateway API 监听端口 | -| `MEMORY_GATEWAY_RELOAD` | `false` | 是否启用 uvicorn reload，开发时可设为 `true` | - -注意：`MEMORY_GATEWAY_DB_PATH` 和 `MEMORY_GATEWAY_STORAGE_DIR` 是 Gateway -自己的存储位置，不要配置成上游记忆服务的内部存储目录。 - -## 安装依赖 +安装依赖： ```bash cd /home/tom/memory-gateway uv pip install -e . ``` -## 启动 API - -使用 Python 启动： +启动 API： ```bash -cd /home/tom/memory-gateway python main.py ``` -默认监听： +默认服务地址： ```text http://127.0.0.1:8010 ``` -## session_id 规范 - -Gateway 遵循以下 `session_id` 规范： - -| 场景 | 格式 | -|---|---| -| 普通聊天 | `chat:{conversation_id}` | -| 用户上传资源 | `resource:{user_id}:{resource_id}` | -| 用户手动修正 | `memory_edit:{user_id}` | - -当前实现中，资源上传会自动生成： - -```text -resource:{user_id}:{resource_id} -``` - -## 数据库表 - -启动时会自动创建以下 SQLite 表： - -- `user_resources`：资源元数据、内部 URI、状态、软删除时间。 -- `users`：用户 ID 和 Gateway 生成的 `user_key`。 -- `memory_tombstones`：用户删除的 memory id 或 session_id。 -- `memory_overrides`：用户手动修正后的 memory 文本。 - -## API 日志 - -Gateway 会通过 `memory_gateway.api` logger 为每个 API 请求输出一条 JSON 日志，字段包括： - -- `request_time`：请求进入 Gateway 的 UTC 时间。 -- `duration_ms`：接口处理耗时。 -- `method`、`path`、`url`、`client`：请求方法、地址和客户端地址。 -- `input`：query 参数和请求体。`user_key`、token、password、secret、API key 等敏感字段会记录为 `[REDACTED]`；multipart 上传只记录 content type 和大小，不记录文件内容。 -- `output`：HTTP 状态码和响应体；敏感字段同样会遮蔽。 - -## API 使用说明 - -除 `POST /users` 外，所有业务 API 都需要携带 `user_id` 和 `user_key`。认证失败返回 `401`。 - -### 1. 健康检查 - -```http -GET /health -``` - -该接口不需要 `user_id` 或 `user_key`，用于确认 Gateway API 是否可响应，以及上游记忆服务是否可访问。 - -请求示例： +健康检查： ```bash curl http://127.0.0.1:8010/health ``` -上游记忆服务正常时响应示例： +健康响应中： -```json -{ - "status": "ok", - "api": { - "status": "ok" - }, - "backend": { - "status": "ok", - "base_url": "http://127.0.0.1:1995", - "data": { - "status": "ok" - } - } -} +- `status: ok` 表示 Gateway 和上游服务都可访问。 +- `status: degraded` 表示 Gateway 可访问，但上游服务不可访问。 + +## 配置 + +配置来自环境变量。常用 `.env` 示例： + +```bash +MEMORY_GATEWAY_BACKEND_BASE_URL=http://127.0.0.1:1995 +MEMORY_GATEWAY_DB_PATH=./data/memory_gateway.sqlite3 +MEMORY_GATEWAY_STORAGE_DIR=./data/storage +MEMORY_GATEWAY_RESOURCE_SEARCH_BATCH_SIZE=50 +MEMORY_GATEWAY_MAX_UPLOAD_BYTES=26214400 +MEMORY_GATEWAY_ALLOWED_MIME_TYPES=image/*,audio/*,application/pdf,text/html,application/xhtml+xml,text/plain,text/markdown,text/csv,application/json,application/msword,application/vnd.openxmlformats-officedocument.wordprocessingml.document,application/vnd.ms-excel,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,application/vnd.ms-powerpoint,application/vnd.openxmlformats-officedocument.presentationml.presentation +MEMORY_GATEWAY_BACKEND_INGEST_ATTEMPTS=3 +MEMORY_GATEWAY_BACKEND_RETRY_DELAY_SECONDS=0.25 +MEMORY_GATEWAY_BACKEND_TIMEOUT_SECONDS=120 +MEMORY_GATEWAY_HOST=127.0.0.1 +MEMORY_GATEWAY_PORT=8010 +MEMORY_GATEWAY_RELOAD=false ``` -上游记忆服务不可访问时仍返回 HTTP 200，但 `status` 会变成 `degraded`，便于区分“Gateway API 活着”和“上游记忆服务故障”： +配置说明： -```json -{ - "status": "degraded", - "api": { - "status": "ok" - }, - "backend": { - "status": "unavailable", - "base_url": "http://127.0.0.1:1995", - "error": "Connection refused" - } -} -``` +| 变量 | 默认值 | 说明 | +|---|---|---| +| `MEMORY_GATEWAY_BACKEND_BASE_URL` | `http://127.0.0.1:1995` | 上游 memory service 地址 | +| `MEMORY_GATEWAY_DB_PATH` | `./data/memory_gateway.sqlite3` | Gateway SQLite 数据库 | +| `MEMORY_GATEWAY_STORAGE_DIR` | `./data/storage` | Gateway 保存上传文件的位置 | +| `MEMORY_GATEWAY_RESOURCE_SEARCH_BATCH_SIZE` | `50` | resources 搜索每批 session 数量 | +| `MEMORY_GATEWAY_MAX_UPLOAD_BYTES` | `26214400` | 单文件上传大小限制，默认 25 MiB | +| `MEMORY_GATEWAY_ALLOWED_MIME_TYPES` | 常见图片、音频、PDF、HTML、文本、Office 文档 | MIME 白名单，支持 `image/*` | +| `MEMORY_GATEWAY_BACKEND_INGEST_ATTEMPTS` | `3` | 上游 add/flush 重试次数 | +| `MEMORY_GATEWAY_BACKEND_RETRY_DELAY_SECONDS` | `0.25` | 上游重试间隔 | +| `MEMORY_GATEWAY_BACKEND_TIMEOUT_SECONDS` | `120` | 上游请求超时 | +| `MEMORY_GATEWAY_HOST` | `127.0.0.1` | Gateway 监听地址 | +| `MEMORY_GATEWAY_PORT` | `8010` | Gateway 监听端口 | +| `MEMORY_GATEWAY_RELOAD` | `false` | 是否启用 uvicorn reload | -### 2. 创建用户 +不要把 Gateway 的数据库或 storage 配到上游服务内部目录。Gateway 只管理自己的状态。 -```http -POST /users -Content-Type: application/json -``` +## 数据模型 -请求参数： +Gateway 启动时自动创建 SQLite 表： -| 参数 | 类型 | 必填 | 说明 | -|---|---|---|---| -| `user_id` | string | 是 | 用户 ID | +| 表 | 用途 | +|---|---| +| `users` | 用户 ID 和 Gateway 生成的 `user_key` | +| `user_resources` | 独立资源元数据、状态、内部 URI、软删除时间 | +| `memory_attachments` | 聊天/资源 session 到真实附件 URI 的映射 | +| `memory_tombstones` | 用户删除的 memory id 或 session_id | +| `memory_overrides` | 用户人工修正后的 memory 文本 | -请求示例： +附件来源 `source`： + +| source | 来源 | +|---|---| +| `resource_upload` | `/resources` 上传 | +| `external_resource` | `/resources/external` 登记 | +| `memory_add_uri` | `/memories/add` 中的 URI item | +| `memory_add_base64` | `/memories/add` 中的 base64 item | +| `memory_add_upload` | `/memories/add/multipart` 中的上传文件 | + +## Session 约定 + +| 场景 | session_id | +|---|---| +| 普通聊天 | `chat:{conversation_id}` | +| 独立上传资源 | `resource:{user_id}:{resource_id}` | +| 手动修正 | `memory_edit:{user_id}` | + +`/resources` 会自动生成资源 session。`/memories/add` 和 +`/memories/add/multipart` 使用调用方传入的 session。 + +## 鉴权 + +除 `POST /users` 和 `GET /health` 外，所有业务接口都需要 `user_id` 和 +`user_key`。认证失败返回 `401`。 + +创建用户： ```bash curl -X POST http://127.0.0.1:8010/users \ @@ -198,378 +169,132 @@ curl -X POST http://127.0.0.1:8010/users \ -d '{"user_id":"u_123"}' ``` -响应示例： +响应： ```json { "user_id": "u_123", "user_key": "uk_xxx", - "created_at": "2026-06-10T10:00:00+00:00" + "created_at": "2026-06-22T06:54:35.823262+00:00" } ``` -`user_key` 需要由调用方保存，后续上传、查询、搜索、修改和删除都要传入。如果同一个 `user_id` 已存在，接口会返回已有 `user_key`。 +同一个 `user_id` 重复创建会返回已有用户记录。 -### 3. 上传资源 +## API 概览 + +| 方法 | 路径 | 说明 | +|---|---|---| +| `GET` | `/health` | Gateway 和上游健康检查 | +| `POST` | `/users` | 创建或读取用户 key | +| `POST` | `/resources` | multipart 上传独立资源 | +| `POST` | `/resources/external` | 登记外部资源 | +| `GET` | `/resources` | 列出用户资源 | +| `GET` | `/resources/{resource_id}` | 读取资源详情 | +| `DELETE` | `/resources/{resource_id}` | 软删除资源 | +| `POST` | `/memories/add` | JSON 追加记忆消息 | +| `POST` | `/memories/add/multipart` | multipart 追加消息并上传附件 | +| `POST` | `/memories/flush` | flush 指定 session | +| `POST` | `/memories/search` | 编排搜索 | +| `PATCH` | `/memories/{memory_id}` | 人工覆盖 memory 文本 | +| `DELETE` | `/memories/{memory_id}` | tombstone 删除 memory | + +## 上传聊天附件：`/memories/add/multipart` + +用于“同一个聊天 session 中发送消息并上传附件”。客户端不需要转 base64，也不需要 +传 `file://`。 + +请求类型： ```http -POST /resources +POST /memories/add/multipart Content-Type: multipart/form-data ``` -表单参数： +表单字段： -| 参数 | 类型 | 必填 | 默认值 | 说明 | -|---|---|---|---|---| -| `user_id` | string | 是 | 无 | 用户 ID | -| `user_key` | string | 是 | 无 | 用户 key | -| `app_id` | string | 否 | `default` | 上游记忆服务 app scope | -| `project_id` | string | 否 | `default` | 上游记忆服务 project scope | -| `file` | file | 是 | 无 | 上传资源文件 | -| `title` | string | 否 | `null` | 资源标题 | -| `description` | string | 否 | `null` | 资源描述 | +| 字段 | 类型 | 必填 | 说明 | +|---|---|---|---| +| `user_id` | string | 是 | 用户 ID | +| `user_key` | string | 是 | 用户 key | +| `session_id` | string | 是 | 通常是 `chat:{conversation_id}` | +| `app_id` | string | 否 | 默认 `default` | +| `project_id` | string | 否 | 默认 `default` | +| `messages` | string 或 JSON file | 是 | JSON array，结构同 `/memories/add` | +| 动态文件字段 | file | 条件必填 | 如果 content item 包含 `upload_id`，必须上传同名文件字段 | -处理流程： +`upload_id` 由调用方自定义。Gateway 不会随机生成，也不要求固定格式 +（例如 `user_id_filetype_number`）。它只需要在同一次请求里非空、唯一，并且和 +multipart 文件字段名一致。推荐使用简单可读的值，如 `image_1`、`audio_1`、 +`doc_1`。 -1. 保存原始文件到 `MEMORY_GATEWAY_STORAGE_DIR`。 -2. 计算 `sha256` 和 `size_bytes`。 -3. 生成 `resource_id`。 -4. 生成 `session_id = resource:{user_id}:{resource_id}`。 -5. 写入 `user_resources`，状态为 `ingesting`。 -6. 根据 MIME 类型映射上游记忆服务 content type。 -7. 构造上游记忆服务 content item：文本类上传以内联 `text` 发送，非文本上传以内联 `base64` 发送，不要求上游记忆服务访问 Gateway 本地 `file://` 路径。 -8. 调用上游记忆服务的 `/api/v1/memory/add`。 -9. 调用上游记忆服务的 `/api/v1/memory/flush`。 -10. 成功后状态改为 `extracted`，失败后状态改为 `failed`。 +`messages` 示例： -上传策略： +```json +[ + { + "sender_id": "u_123", + "role": "user", + "timestamp": 1782111275810, + "content": [ + {"type": "text", "text": "请记住这张图片和音频"}, + { + "type": "image", + "upload_id": "image_1", + "name": "simple-multimodal-image.png", + "ext": "png" + }, + { + "type": "audio", + "upload_id": "audio_1", + "name": "simple-tone.wav", + "ext": "wav" + } + ] + } +] +``` -- 文件会按流式方式写入磁盘，超过 `MEMORY_GATEWAY_MAX_UPLOAD_BYTES` 会返回 `413`，不会写入资源记录。 -- MIME 类型不在 `MEMORY_GATEWAY_ALLOWED_MIME_TYPES` 白名单内会返回 `415`。 -- 同一用户在同一 `app_id`、`project_id` 下重复上传相同 sha256 的活跃资源，会直接返回已有资源，避免重复调用上游记忆服务摄入。 -- 上游记忆服务的 `add` 和 `flush` 临时失败时会分别按配置重试；单次请求受 `MEMORY_GATEWAY_BACKEND_TIMEOUT_SECONDS` 控制；全部失败后资源状态为 `failed`，并记录 `error_message`。 - -content type 映射： - -| 文件类型 | 上游记忆服务 content type | -|---|---| -| `image/*` | `image` | -| `audio/*` | `audio` | -| PDF | `pdf` | -| HTML | `html` | -| 纯文本、Markdown、CSV、日志 | `text` | -| 其他文档 | `doc` | - -请求示例： +curl 示例： ```bash -curl -X POST http://127.0.0.1:8010/resources \ +curl -X POST http://127.0.0.1:8010/memories/add/multipart \ -F user_id=u_123 \ -F user_key=uk_xxx \ + -F session_id=chat:c_456 \ -F app_id=default \ -F project_id=default \ - -F title="合同文件" \ - -F description="2026 年付款合同" \ - -F file=@./contract.pdf + -F 'messages=@messages.json;type=application/json' \ + -F 'image_1=@tests/simple-multimodal-image.png;type=image/png' \ + -F 'audio_1=@tests/simple-tone.wav;type=audio/wav' ``` -响应示例： - -```json -{ - "resource_id": "r_xxx", - "session_id": "resource:u_123:r_xxx", - "uri": "resource://u_123/r_xxx", - "status": "extracted" -} -``` - -资源上传接口返回的 `uri` 始终是 `resource://{user_id}/{resource_id}`。按文件名 -命中的记忆搜索结果会另外通过 `attachments[].internal_uri` 返回真实 URI。 - -### 3b. 登记外部资源 - -```http -POST /resources/external -Content-Type: application/json -``` - -当文件已经保存在 Beaver user_files/MinIO 等外部存储时，调用该接口登记资源。 -Gateway 不会保存原始文件副本，也不会下载 `source_uri` 或 `ingest_uri`；它只写入 -SQLite 元数据、建立附件映射，并把 `ingest_uri` 作为 content item URI 传给上游 -记忆服务完成解析和索引。 - -请求参数： - -| 参数 | 类型 | 必填 | 默认值 | 说明 | -|---|---|---|---|---| -| `user_id` | string | 是 | 无 | 用户 ID | -| `user_key` | string | 是 | 无 | 用户 key | -| `app_id` | string | 否 | `default` | 上游记忆服务 app scope | -| `project_id` | string | 否 | `default` | 上游记忆服务 project scope | -| `filename` | string | 是 | 无 | 文件名，用于检索附件匹配和上游解析 | -| `mime_type` | string | 否 | `null` | MIME 类型，如 `image/png`、`application/pdf` | -| `content_type` | string | 否 | 自动推断 | 上游 content type：`image`、`audio`、`pdf`、`html`、`text` 或 `doc`。如果传入 MIME 类型，Gateway 会按 MIME/扩展名重新推断 | -| `size_bytes` | integer | 否 | `null` | 文件大小 | -| `sha256` | string | 否 | `null` | 文件内容 sha256；同用户同 app/project 下相同 sha256 的活跃资源会复用 | -| `source_uri` | string | 是 | 无 | 长期真实映射 URI，如 `minio://bucket/users/u_123/outputs/chart.png` 或本地开发的 `file://...` | -| `ingest_uri` | string | 否 | `source_uri` | 上游记忆服务当次摄入可读取的 URI。MinIO 场景推荐传短期 presigned GET URL | -| `title` | string | 否 | `null` | 资源标题 | -| `description` | string | 否 | `null` | 资源描述 | - 处理流程： -1. 生成 `resource_id`。 -2. 生成 `session_id = resource:{user_id}:{resource_id}`。 -3. 写入 `user_resources`，`uri = source_uri`，状态为 `ingesting`。 -4. 写入 `memory_attachments`，`internal_uri = source_uri`，`source = external_resource`。 -5. 构造上游 content item：`{"type": "...", "name": filename, "uri": ingest_uri, "extras": {"resource_id": "...", "source": "external_resource"}}`。 -6. 调用上游记忆服务 `/api/v1/memory/add`。 -7. 调用上游记忆服务 `/api/v1/memory/flush`。 -8. 成功后状态改为 `extracted`，失败后状态改为 `failed` 并记录错误。 +1. Gateway 校验用户。 +2. 解析 `messages`。 +3. 找到 content item 中的 `upload_id`。 +4. 从 multipart 中读取同名文件字段。 +5. 校验 MIME 和大小。 +6. 保存文件到 `MEMORY_GATEWAY_STORAGE_DIR/{user_id}/memory_attachments/{sha256}/`。 +7. 将 item 转成上游可消费的 `text` 或 `base64`，不透传客户端本机路径。 +8. 调用上游 memory add。 +9. 上游 add 成功后写入 `memory_attachments`，`source = memory_add_upload`。 -请求示例： +该接口只追加消息，不自动 flush。需要抽取和索引时继续调用 `/memories/flush`。 -```bash -curl -X POST http://127.0.0.1:8010/resources/external \ - -H 'Content-Type: application/json' \ - -d '{ - "user_id": "u_123", - "user_key": "uk_xxx", - "app_id": "default", - "project_id": "default", - "filename": "chart.png", - "mime_type": "image/png", - "size_bytes": 12345, - "sha256": "abc123...", - "source_uri": "minio://beaver-user-files/users/u_123/outputs/chart.png", - "ingest_uri": "https://minio.example/presigned/chart.png", - "title": "chart.png", - "description": "Beaver user file outputs/chart.png" - }' -``` +## 追加 JSON 记忆：`/memories/add` -响应示例： +用于调用方已经有纯文本、base64，或明确有上游可读取 URI 的场景。 -```json -{ - "resource_id": "r_xxx", - "session_id": "resource:u_123:r_xxx", - "uri": "resource://u_123/r_xxx", - "status": "extracted" -} -``` - -`source_uri` 是 Gateway 长期保存和搜索结果附件映射使用的真实 URI； -`ingest_uri` 只用于本次上游解析。不要把短期 presigned URL 作为 `source_uri` -长期入库。 - -### 4. 查询资源列表 - -```http -GET /resources?user_id={user_id}&user_key={user_key} -``` - -参数： - -| 参数 | 类型 | 必填 | 说明 | -|---|---|---|---| -| `user_id` | string | 是 | 用户 ID | -| `user_key` | string | 是 | 用户 key | - -只返回当前用户 `deleted_at IS NULL` 的资源。不同用户的资源彼此隔离。 - -请求示例： - -```bash -curl "http://127.0.0.1:8010/resources?user_id=u_123&user_key=uk_xxx" -``` - -响应示例： - -```json -{ - "resources": [ - { - "resource_id": "r_xxx", - "user_id": "u_123", - "filename": "contract.pdf", - "content_type": "pdf", - "mime_type": "application/pdf", - "uri": "resource://u_123/r_xxx", - "session_id": "resource:u_123:r_xxx", - "status": "extracted", - "title": "合同文件", - "description": "2026 年付款合同", - "created_at": "2026-06-10T10:00:00+00:00", - "updated_at": "2026-06-10T10:00:05+00:00" - } - ] -} -``` - -如果用户没有上传过资源，返回： - -```json -{ - "resources": [] -} -``` - -### 5. 查询资源详情 - -```http -GET /resources/{resource_id}?user_id={user_id}&user_key={user_key} -``` - -路径参数： - -| 参数 | 类型 | 必填 | 说明 | -|---|---|---|---| -| `resource_id` | string | 是 | 资源 ID | - -Query 参数： - -| 参数 | 类型 | 必填 | 说明 | -|---|---|---|---| -| `user_id` | string | 是 | 用户 ID | -| `user_key` | string | 是 | 用户 key | - -请求示例： - -```bash -curl "http://127.0.0.1:8010/resources/r_xxx?user_id=u_123&user_key=uk_xxx" -``` - -响应示例： - -```json -{ - "resources": [ - { - "resource_id": "r_xxx", - "user_id": "u_123", - "filename": "contract.pdf", - "content_type": "pdf", - "mime_type": "application/pdf", - "uri": "resource://u_123/r_xxx", - "session_id": "resource:u_123:r_xxx", - "status": "extracted", - "title": "合同文件", - "description": "2026 年付款合同", - "created_at": "2026-06-10T10:00:00+00:00", - "updated_at": "2026-06-10T10:00:05+00:00" - } - ] -} -``` - -如果当前用户没有上传过 resources、资源不存在、或资源属于其他用户，返回： - -```json -{ - "resources": [] -} -``` - -这种设计避免通过资源 ID 探测其他用户的数据。`uri` 同样只返回公开 `resource://{user_id}/{resource_id}`，不会泄露内部 URI。 - -### 6. 删除资源 - -```http -DELETE /resources/{resource_id}?user_id={user_id}&user_key={user_key} -``` - -第一版只做软删除： - -- 设置 `deleted_at = now()`。 -- 设置 `status = deleted`。 -- 后续 `resources` scope 搜索会排除该资源的 `session_id`。 -- 清理 Gateway 自己在 `MEMORY_GATEWAY_STORAGE_DIR` 下保存的原始上传文件。 -- 不物理删除上游记忆服务内部记忆或索引。 - -请求示例： - -```bash -curl -X DELETE "http://127.0.0.1:8010/resources/r_xxx?user_id=u_123&user_key=uk_xxx" -``` - -响应示例： - -```json -{ - "resource_id": "r_xxx", - "session_id": "resource:u_123:r_xxx", - "uri": "resource://u_123/r_xxx", - "status": "deleted" -} -``` - -### 7. 添加记忆 +请求： ```http POST /memories/add Content-Type: application/json ``` -该接口把一批消息追加到指定 `session_id`，并调用上游记忆服务 -`/api/v1/memory/add`。如果消息 content item 中包含 `uri` 或 `base64` 附件， -Gateway 会额外登记附件映射，供后续 `/memories/search` 返回 -`attachments[].internal_uri`。 - -请求参数： - -| 参数 | 类型 | 必填 | 默认值 | 说明 | -|---|---|---|---|---| -| `user_id` | string | 是 | 无 | 用户 ID | -| `user_key` | string | 是 | 无 | 用户 key | -| `session_id` | string | 是 | 无 | 记忆 session，如 `chat:{conversation_id}`、`resource:{user_id}:{resource_id}` 或 `memory_edit:{user_id}` | -| `messages` | object[] | 是 | 无 | 追加到该 session 的消息数组，至少 1 条 | -| `app_id` | string | 否 | `default` | 上游记忆服务 app scope | -| `project_id` | string | 否 | `default` | 上游记忆服务 project scope | - -`messages[]` 字段： - -| 字段 | 类型 | 必填 | 说明 | -|---|---|---|---| -| `sender_id` | string | 是 | 发送方 ID，通常为当前用户 ID | -| `role` | string | 是 | `user`、`assistant` 或 `tool` | -| `timestamp` | integer | 是 | Unix 毫秒时间戳，必须大于 0 | -| `content` | string 或 object[] | 是 | 纯文本，或上游记忆服务 content item 数组 | - -常见 content item： - -```json -{"type": "text", "text": "用户喜欢简洁的中文回答"} -``` - -```json -{ - "type": "image", - "uri": "file:///home/tom/memory-gateway/tests/simple-multimodal-image.png", - "ext": "png", - "name": "simple-multimodal-image.png" -} -``` - -```json -{ - "type": "audio", - "base64": "BASE64_DATA", - "ext": "wav", - "name": "tone.wav" -} -``` - -处理流程： - -1. 校验 `user_id` 和 `user_key`。 -2. 检查 `messages` 基本结构。 -3. 遍历 object[] content item：含 `uri` 时登记该 URI；只有 `base64` 时先校验并保存到 `MEMORY_GATEWAY_STORAGE_DIR/{user_id}/memory_attachments/{sha256}/`，再登记生成的 `file://` URI。 -4. 将原始 `session_id`、`app_id`、`project_id`、`messages` 原样转发给上游记忆服务 `/api/v1/memory/add`。 -5. 上游 add 成功后写入附件映射；失败时不会保留本次新生成的 base64 附件文件。 - -请求示例： +示例： ```bash curl -X POST http://127.0.0.1:8010/memories/add \ @@ -584,17 +309,14 @@ curl -X POST http://127.0.0.1:8010/memories/add \ { "sender_id": "u_123", "role": "user", - "timestamp": 1781172177000, + "timestamp": 1782111275810, "content": [ + {"type": "text", "text": "用户喜欢简洁的中文回答"}, { - "type": "text", - "text": "记住：我偏好 concise 的中文说明" - }, - { - "type": "image", - "uri": "file:///home/tom/memory-gateway/tests/simple-multimodal-image.png", - "ext": "png", - "name": "simple-multimodal-image.png" + "type": "audio", + "base64": "BASE64_DATA", + "ext": "wav", + "name": "tone.wav" } ] } @@ -602,58 +324,161 @@ curl -X POST http://127.0.0.1:8010/memories/add \ }' ``` +`/memories/add` 中： + +- `uri` item 会原样转发给上游，并登记附件映射。 +- `base64` item 会保存到 Gateway storage，并登记生成的内部 `file://`。 +- 如果 URI 是客户端本机路径，且上游读不到该路径，请改用 `/memories/add/multipart`。 + +## Flush：`/memories/flush` + +```bash +curl -X POST http://127.0.0.1:8010/memories/flush \ + -H 'Content-Type: application/json' \ + -d '{ + "user_id": "u_123", + "user_key": "uk_xxx", + "session_id": "chat:c_456", + "app_id": "default", + "project_id": "default" + }' +``` + 响应示例： ```json { "session_id": "chat:c_456", "backend": { - "request_id": "add", + "request_id": "request_id", "data": { - "status": "accumulated" + "status": "extracted" } } } ``` -`/memories/add` 只负责追加消息。需要让上游记忆服务对该 session 完成抽取和索引时， -继续调用 `/memories/flush`。 +## 上传独立资源：`/resources` -### 8. 搜索记忆 +用于把文件作为长期资源纳入 `resources` 搜索范围。 + +请求类型： ```http -POST /memories/search -Content-Type: application/json +POST /resources +Content-Type: multipart/form-data ``` -请求参数： +表单字段： -| 参数 | 类型 | 必填 | 默认值 | 说明 | -|---|---|---|---|---| -| `user_id` | string | 是 | 无 | 用户 ID | -| `user_key` | string | 是 | 无 | 用户 key | -| `agent_id` | string | 否 | `null` | 设置后查询该 agent 的记忆；`user_id` 仍用于 Gateway 鉴权和本地数据隔离 | -| `query` | string | 是 | 无 | 搜索问题 | -| `conversation_id` | string | 否 | `null` | `scope` 包含 `current_chat` 时使用 | -| `scope` | string[] | 否 | `["current_chat", "resources"]` | 搜索范围 | -| `method` | string | 否 | `hybrid` | 搜索方法：`keyword`、`vector`、`hybrid` 或 `agentic` | -| `top_k` | integer | 否 | `8` | 每次上游搜索返回数量，支持 `-1` 或 `1..100`；`-1` 表示使用上游默认数量 | -| `radius` | number | 否 | `null` | 相似度半径，范围 `0..1`；未提供时不发送给上游 | -| `include_profile` | boolean | 否 | `true` | 是否同时获取用户 profile；agent 查询由上游决定是否忽略该参数 | -| `enable_llm_rerank` | boolean | 否 | `true` | 是否启用上游 LLM rerank；具体生效范围由搜索方法和记忆类型决定 | -| `filters` | object | 否 | `null` | 上游过滤 DSL，支持字段条件以及嵌套 `AND`、`OR` | -| `app_id` | string | 否 | `default` | 上游记忆服务 app scope | -| `project_id` | string | 否 | `default` | 上游记忆服务 project scope | +| 字段 | 类型 | 必填 | 说明 | +|---|---|---|---| +| `user_id` | string | 是 | 用户 ID | +| `user_key` | string | 是 | 用户 key | +| `app_id` | string | 否 | 默认 `default` | +| `project_id` | string | 否 | 默认 `default` | +| `title` | string | 否 | 资源标题 | +| `description` | string | 否 | 资源描述 | +| `file` | file | 是 | 上传文件 | -`scope` 支持： +示例： -| scope | 说明 | -|---|---| -| `current_chat` | 只搜索 `chat:{conversation_id}` | -| `resources` | 搜索当前用户未删除且已提取完成的上传资源 | -| `all_user_memory` | 搜索用户全部记忆，不加 `session_id` 过滤 | +```bash +curl -X POST http://127.0.0.1:8010/resources \ + -F user_id=u_123 \ + -F user_key=uk_xxx \ + -F app_id=default \ + -F project_id=default \ + -F title="Gateway image resource" \ + -F description="Example image resource" \ + -F 'file=@tests/simple-multimodal-image.png;type=image/png' +``` -请求示例： +响应： + +```json +{ + "resource_id": "r_xxx", + "session_id": "resource:u_123:r_xxx", + "uri": "resource://u_123/r_xxx", + "status": "extracted" +} +``` + +`/resources` 会： + +1. 保存文件到 Gateway storage。 +2. 写入 `user_resources`。 +3. 写入 `memory_attachments`。 +4. 构造上游 content item；文本用 `text`，二进制用 `base64`。 +5. 自动调用上游 add 和 flush。 +6. 成功后状态为 `extracted`。 + +同一用户、同一 app/project 下相同 sha256 的活跃资源会复用已有资源。 + +## 登记外部资源：`/resources/external` + +用于文件已经在外部存储中，Gateway 只保存元数据和 URI 映射。 + +示例： + +```bash +curl -X POST http://127.0.0.1:8010/resources/external \ + -H 'Content-Type: application/json' \ + -d '{ + "user_id": "u_123", + "user_key": "uk_xxx", + "app_id": "default", + "project_id": "default", + "filename": "chart.png", + "mime_type": "image/png", + "size_bytes": 12345, + "sha256": "abc123", + "source_uri": "minio://bucket/users/u_123/chart.png", + "ingest_uri": "https://minio.example/presigned/chart.png", + "title": "chart.png", + "description": "External image" + }' +``` + +字段含义： + +- `source_uri`：长期保存的真实映射 URI。 +- `ingest_uri`：上游本次摄入可读取的 URI，通常是短期 presigned URL。 + +Gateway 不会下载 `source_uri` 或 `ingest_uri`。 + +## 资源读取和删除 + +列出资源： + +```bash +curl "http://127.0.0.1:8010/resources?user_id=u_123&user_key=uk_xxx" +``` + +读取资源详情： + +```bash +curl "http://127.0.0.1:8010/resources/r_xxx?user_id=u_123&user_key=uk_xxx" +``` + +软删除资源： + +```bash +curl -X DELETE "http://127.0.0.1:8010/resources/r_xxx?user_id=u_123&user_key=uk_xxx" +``` + +删除资源会： + +- 设置资源 `status = deleted`。 +- 设置资源 `deleted_at`。 +- 软删除相关附件映射。 +- 清理 Gateway storage 中属于该资源的本地文件。 +- 不直接删除上游记忆服务内部文件或索引。 + +## 搜索：`/memories/search` + +请求： ```bash curl -X POST http://127.0.0.1:8010/memories/search \ @@ -662,111 +487,61 @@ curl -X POST http://127.0.0.1:8010/memories/search \ "user_id": "u_123", "user_key": "uk_xxx", "conversation_id": "c_456", - "query": "合同里的付款条款是什么？", + "query": "图片里的蓝色圆形在哪里？", "scope": ["current_chat", "resources"], "method": "hybrid", "top_k": 8, "include_profile": true, "enable_llm_rerank": true, - "filters": { - "type": "Conversation" - }, "app_id": "default", "project_id": "default" }' ``` -搜索编排逻辑： +搜索 scope： -1. `current_chat`：调用上游记忆服务 search，过滤 `filters.session_id = chat:{conversation_id}`。 -2. `resources`：先查当前用户的 `user_resources`，只取 `status = extracted` 且未删除资源；再按批次调用上游记忆服务 search，过滤这些资源的 `session_id`。 -3. `all_user_memory`：调用上游记忆服务 search，不加 `session_id` 过滤。 -4. 同时存在请求 `filters` 和 scope 生成的 session 条件时，使用 `{"AND": [filters, scope_filters]}` 合并，避免调用方过滤条件覆盖资源或聊天隔离条件。 -5. 设置 `agent_id` 时，上游请求只发送 `agent_id`；否则发送已鉴权的 `user_id`。 -6. 合并结果。 -7. 过滤 `memory_tombstones` 命中的 `memory_id` 或 `session_id`。 -8. 应用 active `memory_overrides`，把 `text` 替换为 `override_text`。 - -响应中的 `memory_type` 对应上游结果类型： - -| `memory_type` | 说明 | +| scope | 行为 | |---|---| -| `episode` | 用户 episode | -| `profile` | 用户 profile | -| `agent_case` | agent case | -| `agent_skill` | agent skill | -| `unprocessed_message` | 尚未完成边界提取的原始消息 | +| `current_chat` | 搜索 `chat:{conversation_id}`，需要传 `conversation_id` | +| `resources` | 搜索当前用户已提取且未删除的资源 session | +| `all_user_memory` | 搜索用户全部记忆，不加 session 过滤 | -附件路径映射规则： - -1. `/resources` 上传成功后，将 Gateway 本地保存的真实 URI 与资源 session 写入 - `memory_attachments`。数据库初始化会自动回填已有 `user_resources`。 -2. `/resources/external` 登记成功后，将外部存储的 `source_uri` 与资源 session - 写入 `memory_attachments`，但 Gateway 不保存原始文件副本。 -3. `/memories/add` 中含 `uri` 的 content item 会直接登记 URI。 -4. `/memories/add` 中只有 `base64` 的 content item 会保存到 - `MEMORY_GATEWAY_STORAGE_DIR/{user_id}/memory_attachments/{sha256}/`，再登记 - 生成的 `file://` URI。相同用户、session、文件名和内容的重试会复用路径。 -5. 搜索时根据当前用户和结果 `session_id` 查询附件，递归检查 `raw` 中的字符串 - 值。只有完整文件名出现时才返回对应附件；匹配不区分大小写。 -6. `raw` 中键名为 `base64` 的内容不会参与匹配。未匹配时返回 - `"attachments": []`。 -7. 历史 `/memories/add` 请求未保存在 Gateway 数据库中，无法自动补录映射；新 - 版本上线后的请求会建立映射。 - -`attachments[].internal_uri` 会按配置和调用方输入直接暴露服务器真实 URI，调用 -该接口的客户端必须被视为可信客户端。 - -响应示例： +返回结果会标准化为： ```json { "results": [ { - "id": "mem_abc", - "memory_type": "episode", - "session_id": "resource:u_123:r_xxx", - "text": "付款期限为收到发票后 30 天内。", - "score": 0.82, - "source_scope": "resources", - "resource_id": "r_xxx", - "resource_uri": "resource://u_123/r_xxx", + "id": "memory_id", + "session_id": "chat:c_456", + "text": "memory text", + "score": 0.61, + "source_scope": "current_chat", + "resource_id": null, + "resource_uri": null, "attachments": [ { - "type": "pdf", - "name": "contract.pdf", - "internal_uri": "file:///srv/memory-gateway/u_123/r_xxx/contract.pdf" + "type": "image", + "name": "image.png", + "internal_uri": "file:///..." } ], - "raw": { - "id": "mem_abc", - "session_id": "resource:u_123:r_xxx", - "episode": "原始上游记忆服务返回内容" - } + "raw": {} } ] } ``` -### 9. 修改记忆 +附件匹配规则： -```http -PATCH /memories/{memory_id} -Content-Type: application/json -``` +- Gateway 只返回当前用户、当前 session 的附件映射。 +- Gateway 根据搜索结果 raw/text 中出现的文件名匹配附件。 +- `base64` 字段内容不会参与匹配。 +- 未匹配到附件时不返回 `attachments`。 -请求参数： +## 手动覆盖和删除记忆 -| 参数 | 类型 | 必填 | 说明 | -|---|---|---|---| -| `user_id` | string | 是 | 用户 ID | -| `user_key` | string | 是 | 用户 key | -| `session_id` | string | 是 | memory 所属 session，必须属于当前用户 | -| `override_text` | string | 是 | 修正后的记忆文本 | - -该接口只写入或更新 `memory_overrides`，不会修改上游记忆服务原始文件。写入前会校验 `session_id` 属于当前用户：当前版本支持当前用户的 `resource:{user_id}:{resource_id}` 和 `memory_edit:{user_id}`。后续搜索结果命中该 `memory_id` 时，返回的 `text` 会替换为 `override_text`，但保留原始 memory id。 - -请求示例： +覆盖 memory 文本： ```bash curl -X PATCH http://127.0.0.1:8010/memories/mem_abc \ @@ -774,40 +549,12 @@ curl -X PATCH http://127.0.0.1:8010/memories/mem_abc \ -d '{ "user_id": "u_123", "user_key": "uk_xxx", - "session_id": "resource:u_123:r_xxx", - "override_text": "修正后的记忆内容" + "session_id": "chat:c_456", + "override_text": "修正后的记忆文本" }' ``` -响应示例： - -```json -{ - "memory_id": "mem_abc", - "override_id": "o_xxx", - "status": "active" -} -``` - -### 10. 删除记忆 - -```http -DELETE /memories/{memory_id} -Content-Type: application/json -``` - -请求参数： - -| 参数 | 类型 | 必填 | 说明 | -|---|---|---|---| -| `user_id` | string | 是 | 用户 ID | -| `user_key` | string | 是 | 用户 key | -| `session_id` | string | 是 | memory 所属 session，必须属于当前用户 | -| `reason` | string | 否 | 删除原因 | - -该接口只写入 `memory_tombstones`，不会修改上游记忆服务原始文件。写入前会校验 `session_id` 属于当前用户：当前版本支持当前用户的 `resource:{user_id}:{resource_id}` 和 `memory_edit:{user_id}`。后续搜索结果如果命中 tombstone 的 `memory_id` 或 `session_id`，会被过滤。 - -请求示例： +删除 memory： ```bash curl -X DELETE http://127.0.0.1:8010/memories/mem_abc \ @@ -815,71 +562,126 @@ curl -X DELETE http://127.0.0.1:8010/memories/mem_abc \ -d '{ "user_id": "u_123", "user_key": "uk_xxx", - "session_id": "resource:u_123:r_xxx", - "reason": "用户手动删除" + "session_id": "chat:c_456", + "reason": "用户要求删除" }' ``` -响应示例： +Gateway 会校验 session 归属，防止用户覆盖或删除其他用户资源 session 中的记忆。 -```json -{ - "memory_id": "mem_abc", - "tombstone_id": "t_xxx", - "status": "deleted" -} +## `/memories/add/multipart` 与 `/resources` 的区别 + +| 对比项 | `/memories/add/multipart` | `/resources` | +|---|---|---| +| 目标 | 聊天 session 中的消息附件 | 独立长期资源 | +| session | 调用方传入，如 `chat:c_456` | Gateway 生成 `resource:{user_id}:{resource_id}` | +| 文件字段 | 动态字段名匹配 `upload_id` | 固定字段 `file` | +| 元数据表 | 不写 `user_resources` | 写 `user_resources` | +| 附件 source | `memory_add_upload` | `resource_upload` | +| 上游调用 | 只 add，调用方按需 flush | Gateway 自动 add + flush | +| 常用搜索 | `current_chat` 或 `all_user_memory` | `resources` | + +## 错误码 + +| 状态码 | 常见原因 | +|---|---| +| `400` | `messages` 不是合法 JSON array | +| `401` | `user_id` 或 `user_key` 无效 | +| `403` | 操作的 session 不属于当前用户 | +| `404` | 删除资源时资源不存在 | +| `413` | 上传文件超过 `MEMORY_GATEWAY_MAX_UPLOAD_BYTES` | +| `415` | MIME 类型不在白名单 | +| `422` | 请求字段缺失、`upload_id` 缺文件、重复文件字段、无效 base64 | + +## API 日志 + +Gateway 使用 `memory_gateway.api` logger 输出 JSON 日志： + +- 请求时间、耗时、方法、路径、URL、客户端。 +- Query 参数和小型请求体。 +- `user_key`、token、password、secret、API key 等敏感字段会遮蔽。 +- multipart 请求只记录 content type 和大小，不记录文件内容。 +- 响应体会按同样规则遮蔽敏感字段。 + +## Agent CLI + +项目包含 Memory Gateway agent skill CLI： + +```bash +python skill/memory-gateway-agent/scripts/memory_gateway.py --help ``` -## 上游记忆服务客户端封装 - -Gateway 内部通过 `core/backend_client.py` 调用上游记忆服务： - -- `add_memory(payload)` -> `POST /api/v1/memory/add` -- `flush_memory(session_id, app_id, project_id)` -> `POST /api/v1/memory/flush` -- `search_memory(payload)` -> `POST /api/v1/memory/search` -- `health_check()` -> `GET /health` - -## AI Agent Skill - -项目提供可供 AI Agent 使用的 Skill： - -```text -skill/memory-gateway-agent -``` - -其中 `SKILL.md` 定义 Agent 工作流，`scripts/memory_gateway.py` 提供无额外依赖的命令行客户端，`references/api.md` 提供完整参数说明。使用前设置： +常用环境变量： ```bash export MEMORY_GATEWAY_BASE_URL=http://127.0.0.1:8010 export MEMORY_GATEWAY_USER_ID=u_123 export MEMORY_GATEWAY_USER_KEY=uk_xxx - -python skill/memory-gateway-agent/scripts/memory_gateway.py health -python skill/memory-gateway-agent/scripts/memory_gateway.py list-resources ``` -## 运行测试 +示例： ```bash -cd /home/tom/memory-gateway -.venv/bin/python -B -m pytest -q -p no:cacheprovider +CLI="python skill/memory-gateway-agent/scripts/memory_gateway.py" + +$CLI health +$CLI create-user u_123 +$CLI upload-resource ./document.pdf --title "Document" +$CLI search "payment terms" --scope resources +$CLI add-memory --session-id chat:c_456 --messages ./messages.json +$CLI flush-memory --session-id chat:c_456 ``` -默认测试不会访问真实上游记忆服务。若要对已部署的上游记忆服务做 health 集成验证，先确认上游记忆服务正在监听 `0.0.0.0:1995`，然后从 Gateway 所在机器用客户端可访问地址访问： +CLI 的 `upload-resource` 使用 multipart，不会暴露本地 `file://` 路径。 + +## 测试 + +运行自动化测试： ```bash -cd /home/tom/memory-gateway -RUN_BACKEND_INTEGRATION=1 \ -MEMORY_GATEWAY_BACKEND_BASE_URL=http://10.6.80.123:1995 \ -.venv/bin/python -B -m pytest -q tests/test_backend_integration.py -p no:cacheprovider +uv run pytest -q ``` -真实 add/flush 上传会写入上游记忆服务，且可能受上游解析、LLM、embedding 服务耗时影响。需要验证完整摄入链路时再打开第二层开关： +当前验证结果： + +```text +56 passed, 2 skipped +``` + +运行聚焦测试： ```bash -cd /home/tom/memory-gateway -RUN_BACKEND_INTEGRATION=1 \ -RUN_BACKEND_INGEST_INTEGRATION=1 \ -MEMORY_GATEWAY_BACKEND_BASE_URL=http://10.6.80.123:1995 \ -.venv/bin/python -B -m pytest -q tests/test_backend_integration.py -p no:cacheprovider +uv run pytest tests/test_gateway.py -k "add_memory or upload_resource or attachment" -q ``` + +当前验证结果： + +```text +16 passed, 33 deselected +``` + +真实部署命令记录： + +```text +tests/test_command.md +``` + +该文件记录了 2026-06-22 对已部署 Gateway 和上游 memory service 执行的真实 curl +命令，包括： + +- `GET /health` +- `POST /memories/add/multipart` +- `POST /memories/flush` +- `POST /memories/search` +- `POST /resources` +- `GET /resources` +- resources scope 搜索 + +## 开发注意事项 + +- 保持 Gateway 与上游服务职责分离；不要直接写上游内部文件。 +- 新上传入口应优先使用 multipart 或外部存储 URI，避免要求客户端构造 base64。 +- 不要在响应中泄露内部 `file://`，除搜索附件映射 `attachments[].internal_uri` 外。 +- 资源详情和列表返回公开 `resource://{user_id}/{resource_id}`。 +- 删除资源只做 Gateway 层软删除和本地文件清理，不直接删除上游记忆。 +- 搜索必须保持用户隔离，尤其是附件映射和资源 session。 diff --git a/core/api.py b/core/api.py index b99fbcf..baec889 100644 --- a/core/api.py +++ b/core/api.py @@ -9,7 +9,9 @@ from urllib.parse import parse_qsl, quote, urlsplit, urlunsplit import httpx from fastapi import APIRouter, FastAPI, File, Form, HTTPException, Request, UploadFile +from pydantic import ValidationError from pydantic import BaseModel, Field, field_validator +from starlette.datastructures import UploadFile as StarletteUploadFile from starlette.responses import Response from .config import GatewayConfig @@ -220,6 +222,60 @@ def _backend_http_error_detail(exc: httpx.HTTPStatusError) -> Any: return exc.response.text +def _form_text(form: Any, field: str, default: str | None = None) -> str: + value = form.get(field) + if value is None: + if default is not None: + return default + raise HTTPException(status_code=422, detail=f"missing form field: {field}") + if isinstance(value, StarletteUploadFile): + raise HTTPException(status_code=422, detail=f"form field must be text: {field}") + return str(value) + + +async def _form_json_text(form: Any, field: str) -> str: + value = form.get(field) + if value is None: + raise HTTPException(status_code=422, detail=f"missing form field: {field}") + if isinstance(value, StarletteUploadFile): + raw = await value.read() + return raw.decode("utf-8") + return str(value) + + +def _upload_files_from_form(form: Any) -> dict[str, UploadFile]: + files: dict[str, UploadFile] = {} + for key, value in form.multi_items(): + if not isinstance(value, StarletteUploadFile): + continue + if key == "messages": + continue + if key in files: + raise HTTPException( + status_code=422, + detail=f"duplicate upload file field: {key}", + ) + files[key] = value + return files + + +async def _multipart_messages(form: Any) -> list[dict[str, Any]]: + raw = await _form_json_text(form, "messages") + try: + parsed = json.loads(raw) + except json.JSONDecodeError as exc: + raise HTTPException( + status_code=400, + detail=f"invalid messages JSON: {exc.msg}", + ) from exc + if not isinstance(parsed, list): + raise HTTPException(status_code=400, detail="messages must be a JSON array") + try: + return [AddMemoryMessage.model_validate(item).model_dump() for item in parsed] + except ValidationError as exc: + raise HTTPException(status_code=422, detail=exc.errors()) from exc + + def create_app( *, config: GatewayConfig | None = None, @@ -466,6 +522,33 @@ def create_app( except InvalidAttachment as exc: raise HTTPException(status_code=422, detail=str(exc)) from exc + @router.post("/memories/add/multipart") + async def add_memory_multipart(request: Request) -> dict[str, Any]: + form = await request.form() + user_id = _form_text(form, "user_id") + user_key = _form_text(form, "user_key") + require_user(user_id, user_key) + try: + return await service.add_memory_with_uploads( + user_id=user_id, + session_id=_form_text(form, "session_id"), + app_id=_form_text(form, "app_id", "default"), + project_id=_form_text(form, "project_id", "default"), + messages=await _multipart_messages(form), + upload_files=_upload_files_from_form(form), + ) + except httpx.HTTPStatusError as exc: + raise HTTPException( + status_code=exc.response.status_code, + detail=_backend_http_error_detail(exc), + ) from exc + except UploadTooLarge as exc: + raise HTTPException(status_code=413, detail=str(exc)) from exc + except UnsupportedContentType as exc: + raise HTTPException(status_code=415, detail=str(exc)) from exc + except InvalidAttachment as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + @router.post("/memories/flush") async def flush_memory( request: FlushMemoryRequest, diff --git a/core/service.py b/core/service.py index 9e62a82..cebbce8 100644 --- a/core/service.py +++ b/core/service.py @@ -138,6 +138,25 @@ def _remove_empty_parents(path: Path, stop_at: Path | None = None) -> None: current = parent +def _read_upload_bytes( + file: UploadFile, + max_upload_bytes: int, +) -> tuple[bytes, str, int]: + sha256 = hashlib.sha256() + size = 0 + chunks: list[bytes] = [] + while True: + chunk = file.file.read(1024 * 1024) + if not chunk: + break + size += len(chunk) + if size > max_upload_bytes: + raise UploadTooLarge(f"upload exceeds max size of {max_upload_bytes} bytes") + sha256.update(chunk) + chunks.append(chunk) + return b"".join(chunks), sha256.hexdigest(), size + + class MemoryGatewayService: def __init__( self, @@ -617,6 +636,41 @@ class MemoryGatewayService: raise return {"session_id": session_id, "backend": backend} + async def add_memory_with_uploads( + self, + *, + user_id: str, + session_id: str, + app_id: str, + project_id: str, + messages: list[dict[str, Any]], + upload_files: dict[str, UploadFile], + ) -> dict[str, Any]: + messages, attachments, generated_paths = self._prepare_uploaded_memory_files( + user_id=user_id, + session_id=session_id, + app_id=app_id, + project_id=project_id, + messages=messages, + upload_files=upload_files, + ) + payload = { + "session_id": session_id, + "app_id": app_id, + "project_id": project_id, + "messages": messages, + } + try: + backend = await self.backend_client.add_memory(payload) + for attachment in attachments: + self.repository.create_attachment(**attachment) + except Exception: + for path in generated_paths: + path.unlink(missing_ok=True) + _remove_empty_parents(path.parent, stop_at=self.config.storage_dir) + raise + return {"session_id": session_id, "backend": backend} + def _register_resource_attachment( self, resource: dict[str, Any], @@ -713,6 +767,119 @@ class MemoryGatewayService: raise return attachments, generated_paths + def _prepare_uploaded_memory_files( + self, + *, + user_id: str, + session_id: str, + app_id: str, + project_id: str, + messages: list[dict[str, Any]], + upload_files: dict[str, UploadFile], + ) -> tuple[list[dict[str, Any]], list[dict[str, Any]], list[Path]]: + attachments: list[dict[str, Any]] = [] + generated_paths: list[Path] = [] + used_upload_ids: set[str] = set() + try: + for message in messages: + content = message.get("content") + if not isinstance(content, list): + continue + for index, item in enumerate(content): + if not isinstance(item, dict) or "upload_id" not in item: + continue + upload_id = str(item.get("upload_id") or "").strip() + if not upload_id: + raise InvalidAttachment("upload_id must not be empty") + if upload_id in used_upload_ids: + raise InvalidAttachment(f"duplicate upload_id: {upload_id}") + file = upload_files.get(upload_id) + if file is None: + raise InvalidAttachment( + f"missing upload file for upload_id: {upload_id}" + ) + used_upload_ids.add(upload_id) + content[index] = self._materialize_uploaded_content_item( + user_id=user_id, + session_id=session_id, + app_id=app_id, + project_id=project_id, + item=item, + file=file, + attachments=attachments, + generated_paths=generated_paths, + ) + unused_upload_ids = sorted(set(upload_files) - used_upload_ids) + if unused_upload_ids: + raise InvalidAttachment( + f"unused upload file field: {unused_upload_ids[0]}" + ) + except Exception: + for path in generated_paths: + path.unlink(missing_ok=True) + _remove_empty_parents(path.parent, stop_at=self.config.storage_dir) + raise + return messages, attachments, generated_paths + + def _materialize_uploaded_content_item( + self, + *, + user_id: str, + session_id: str, + app_id: str, + project_id: str, + item: dict[str, Any], + file: UploadFile, + attachments: list[dict[str, Any]], + generated_paths: list[Path], + ) -> dict[str, Any]: + name = _safe_filename(str(item.get("name") or file.filename or "upload.bin")) + mime_type = file.content_type or mimetypes.guess_type(name)[0] + if not _mime_allowed(mime_type, self.config.allowed_mime_types): + raise UnsupportedContentType(f"unsupported content type: {mime_type}") + content_type = normalize_content_type( + name, + mime_type, + str(item.get("type") or ""), + ) + data, sha256, _size_bytes = _read_upload_bytes( + file, + self.config.max_upload_bytes, + ) + path = self.config.storage_dir / user_id / "memory_attachments" / sha256 / name + if not path.exists(): + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(data) + generated_paths.append(path) + content_item = { + key: value for key, value in item.items() if key not in {"upload_id", "uri"} + } + content_item["type"] = content_type + content_item["name"] = name + content_item["ext"] = Path(name).suffix.lstrip(".") or content_item.get("ext") + if content_type == "text": + content_item.pop("base64", None) + content_item["text"] = data.decode("utf-8", errors="replace") + else: + content_item.pop("text", None) + content_item["base64"] = base64.b64encode(data).decode("ascii") + attachments.append( + { + "id": f"a_{uuid.uuid4().hex}", + "user_id": user_id, + "app_id": app_id, + "project_id": project_id, + "session_id": session_id, + "resource_id": None, + "content_type": content_type, + "name": name, + "internal_uri": path.resolve().as_uri(), + "source": "memory_add_upload", + "sha256": sha256, + } + ) + return content_item + async def flush_memory( self, *, diff --git a/skill/memory-gateway-agent/references/api.md b/skill/memory-gateway-agent/references/api.md index 32fc935..7399f56 100644 --- a/skill/memory-gateway-agent/references/api.md +++ b/skill/memory-gateway-agent/references/api.md @@ -136,6 +136,43 @@ $CLI flush-memory --session-id chat:c_456 `--messages` accepts either a JSON array string or a path to a JSON file. Always flush after all messages for the session have been added. +For local binary files that cannot be converted to base64 by the caller, use the +multipart API directly. Put an `upload_id` in the content item and send a file +field with the same name: + +```bash +curl -X POST "$MEMORY_GATEWAY_BASE_URL/memories/add/multipart" \ + -F user_id="$MEMORY_GATEWAY_USER_ID" \ + -F user_key="$MEMORY_GATEWAY_USER_KEY" \ + -F session_id=chat:c_456 \ + -F app_id=default \ + -F project_id=default \ + -F 'messages=[ + { + "sender_id": "u_123", + "role": "user", + "timestamp": 1781172177000, + "content": [ + {"type": "text", "text": "Remember this image"}, + { + "type": "image", + "upload_id": "image_1", + "name": "image.png", + "ext": "png" + } + ] + } + ]' \ + -F 'image_1=@./image.png;type=image/png' +``` + +The multipart endpoint appends messages to the provided chat session. It stores +the uploaded file under Gateway storage, forwards text/base64 content to the +upstream memory service, and records an attachment mapping. Call `flush-memory` +afterward when the session should be extracted and indexed. This differs from +`upload-resource`, which creates an independent `resource:{user_id}:{resource_id}` +session and automatically performs add plus flush for resource searches. + ### Override and Delete Memory Use IDs from a search result: @@ -188,6 +225,8 @@ Common content items: ``` Prefer base64 for local binary files. A `file://` URI is only usable when upstream memory service can access the same filesystem path. +If that shared path guarantee is not true, use `/memories/add/multipart`, +`upload-resource`, or `/resources/external`. ## Search Scopes diff --git a/tests/test_command.md b/tests/test_command.md index 8647d84..6a0b0fd 100644 --- a/tests/test_command.md +++ b/tests/test_command.md @@ -1,144 +1,126 @@ -# Memory Gateway multimodal API test +# Memory Gateway API curl examples -This file records a real end-to-end test through **Memory Gateway**, not direct upstream memory service calls. +This file keeps only the concrete API curl shapes and short notes. Replace +`` with the key returned by `POST /users`. -Gateway URL used by curl: +Base URL used in the live deployment test: ```text http://127.0.0.1:8010 ``` -Gateway upstream memory service: +Test files: ```text -http://10.6.80.123:1995 +tests/simple-multimodal-image.png +tests/simple-tone.wav ``` -Test assets: - -```text -/home/tom/memory-gateway/tests/simple-multimodal-image.png -/home/tom/memory-gateway/tests/simple-tone.wav -``` - -Asset check: - -```text -tests/simple-multimodal-image.png: PNG image data, 96 x 64, 8-bit/color RGB, non-interlaced -tests/simple-tone.wav: RIFF (little-endian) data, WAVE audio, Microsoft PCM, 16 bit, mono 8000 Hz -``` - -## Start Gateway - -Command: +## 1. Health ```bash -cd /home/tom/memory-gateway - -MEMORY_GATEWAY_BACKEND_BASE_URL=http://10.6.80.123:1995 \ -MEMORY_GATEWAY_DB_PATH=/tmp/memory_gateway_curl.sqlite3 \ -MEMORY_GATEWAY_STORAGE_DIR=/tmp/memory_gateway_curl_storage \ -MEMORY_GATEWAY_HOST=127.0.0.1 \ -MEMORY_GATEWAY_PORT=8010 \ -.venv/bin/python main.py +curl -sS http://127.0.0.1:8010/health ``` -Observed startup: - -```text -INFO: Started server process [771099] -INFO: Waiting for application startup. -INFO: Application startup complete. -INFO: Uvicorn running on http://127.0.0.1:8010 (Press CTRL+C to quit) -``` - -## 1. Create Gateway user - -Request: - -```bash -USER_ID="gateway_user_20260611180257" - -curl -sS --location 'http://127.0.0.1:8010/users' \ - --header 'Content-Type: application/json' \ - --data "{ - \"user_id\": \"${USER_ID}\" - }" -``` - -Response: +Expected shape: ```json { - "user_id": "gateway_user_20260611180257", - "user_key": "uk_REDACTED", - "created_at": "2026-06-11T10:02:57.435437+00:00" + "status": "ok", + "api": {"status": "ok"}, + "backend": { + "status": "ok", + "base_url": "http://0.0.0.0:1995", + "data": {"status": "ok"} + } } ``` -HTTP metadata: +## 2. Create user -```text -HTTP_STATUS:200 -TOTAL_TIME:0.022431 +```bash +curl -sS -X POST http://127.0.0.1:8010/users \ + -H 'Content-Type: application/json' \ + -d '{"user_id":"gateway_demo_user"}' ``` -## 2. Add text + audio(base64) + image(file) through Gateway +Expected shape: + +```json +{ + "user_id": "gateway_demo_user", + "user_key": "uk_REDACTED", + "created_at": "2026-06-22T06:54:35.823262+00:00" +} +``` + +Use the returned `user_key` in later requests. + +## 3. Add chat memory with multipart files + +Use this when files belong to a chat/session message and the client should not +or cannot convert the files to base64. + +`upload_id` rules: + +- `upload_id` is defined by the caller. +- Gateway does not generate it. +- Gateway does not require a format such as `user_id_filetype_number`. +- It only needs to be non-empty, unique inside the request, and equal to the + multipart file field name. +- Good simple values are `image_1`, `image_2`, `audio_1`, `doc_1`. + +In the `messages` JSON, `upload_id: "image_1"` points to this file field: + +```bash +-F 'image_1=@tests/simple-multimodal-image.png;type=image/png' +``` Request: ```bash -cd /home/tom/memory-gateway - -USER_KEY="uk_REDACTED" -CONVERSATION_ID="gateway-multimodal-20260611180257" -SESSION_ID="chat:${CONVERSATION_ID}" -TIMESTAMP_MS="1781172177000" -AUDIO_BASE64="$(base64 -w0 tests/simple-tone.wav)" - -curl -sS --location 'http://127.0.0.1:8010/memories/add' \ - --header 'Content-Type: application/json' \ - --data "{ - \"user_id\": \"${USER_ID}\", - \"user_key\": \"${USER_KEY}\", - \"session_id\": \"${SESSION_ID}\", - \"app_id\": \"default\", - \"project_id\": \"default\", - \"messages\": [ - { - \"sender_id\": \"${USER_ID}\", - \"role\": \"user\", - \"timestamp\": ${TIMESTAMP_MS}, - \"content\": [ - { - \"type\": \"text\", - \"text\": \"请通过 Memory Gateway 同时记住这段文字、音频和图片：图片里有左侧红色方块、右侧蓝色圆形、底部绿色横条；音频是一段短促的测试音。以后可能会问图片中各个物体的位置和颜色。\" - }, - { - \"type\": \"audio\", - \"base64\": \"${AUDIO_BASE64}\", - \"ext\": \"wav\", - \"name\": \"simple-tone.wav\" - }, - { - \"type\": \"image\", - \"uri\": \"file:///home/tom/memory-gateway/tests/simple-multimodal-image.png\", - \"ext\": \"png\", - \"name\": \"simple-multimodal-image.png\" - } - ] - } - ] - }" +curl -sS -X POST http://127.0.0.1:8010/memories/add/multipart \ + -F 'user_id=gateway_demo_user' \ + -F 'user_key=' \ + -F 'session_id=chat:gateway_demo_conversation' \ + -F 'app_id=default' \ + -F 'project_id=default' \ + -F 'messages=[ + { + "sender_id": "gateway_demo_user", + "role": "user", + "timestamp": 1782111275810, + "content": [ + { + "type": "text", + "text": "请记住这次上传：图片里有左上红色方块、右上蓝色圆形、底部绿色横条；音频是一段短促测试音。" + }, + { + "type": "image", + "upload_id": "image_1", + "name": "simple-multimodal-image.png", + "ext": "png" + }, + { + "type": "audio", + "upload_id": "audio_1", + "name": "simple-tone.wav", + "ext": "wav" + } + ] + } + ]' \ + -F 'image_1=@tests/simple-multimodal-image.png;type=image/png' \ + -F 'audio_1=@tests/simple-tone.wav;type=audio/wav' ``` -Response: +Expected shape: ```json { - "session_id": "chat:gateway-multimodal-20260611180257", + "session_id": "chat:gateway_demo_conversation", "backend": { - "request_id": "c9e24b8d27ee4ad08a8df70273336637", + "request_id": "0d6451f4077040e4af207cc6b034ea34", "data": { "message_count": 1, "status": "accumulated" @@ -147,308 +129,153 @@ Response: } ``` -HTTP metadata: +Gateway stores the uploaded files and forwards upstream-compatible `base64` or +`text` content. The client does not send `file://` and does not send base64. -```text -HTTP_STATUS:200 -TOTAL_TIME:1.552665 -``` +Common errors: -## 3. Flush through Gateway +- Missing file field for an `upload_id`: `422` +- Duplicate `upload_id`: `422` +- Extra uploaded file field not referenced by `messages`: `422` +- Unsupported MIME type: `415` +- File too large: `413` -Request: +## 4. Flush chat session + +`/memories/add/multipart` only appends messages. Call flush when the session +should be extracted and indexed. ```bash -curl -sS --location 'http://127.0.0.1:8010/memories/flush' \ - --header 'Content-Type: application/json' \ - --data "{ - \"user_id\": \"${USER_ID}\", - \"user_key\": \"${USER_KEY}\", - \"session_id\": \"${SESSION_ID}\", - \"app_id\": \"default\", - \"project_id\": \"default\" - }" +curl -sS -X POST http://127.0.0.1:8010/memories/flush \ + -H 'Content-Type: application/json' \ + -d '{ + "user_id": "gateway_demo_user", + "user_key": "", + "session_id": "chat:gateway_demo_conversation", + "app_id": "default", + "project_id": "default" + }' ``` -Response: +Expected shape: ```json { - "session_id": "chat:gateway-multimodal-20260611180257", + "session_id": "chat:gateway_demo_conversation", "backend": { - "request_id": "8eb7d5db2d3b43f4999f445aabb813b1", - "data": { - "status": "extracted" - } + "request_id": "4df5415115a34f109c564abd2f9012c6", + "data": {"status": "extracted"} } } ``` -HTTP metadata: - -```text -HTTP_STATUS:200 -TOTAL_TIME:2.135721 -``` - -## 4. Search through Gateway - -upstream memory service indexing can lag briefly after `flush`, so this test waited about 2 seconds before searching. - -Request: +## 5. Search chat session ```bash -sleep 2 - -curl -sS --location 'http://127.0.0.1:8010/memories/search' \ - --header 'Content-Type: application/json' \ - --data "{ - \"user_id\": \"${USER_ID}\", - \"user_key\": \"${USER_KEY}\", - \"conversation_id\": \"${CONVERSATION_ID}\", - \"query\": \"图片里的蓝色圆形在哪里？音频是什么？\", - \"scope\": [\"current_chat\"], - \"top_k\": 5, - \"app_id\": \"default\", - \"project_id\": \"default\" - }" +curl -sS -X POST http://127.0.0.1:8010/memories/search \ + -H 'Content-Type: application/json' \ + -d '{ + "user_id": "gateway_demo_user", + "user_key": "", + "conversation_id": "gateway_demo_conversation", + "query": "图片里的蓝色圆形在哪里？底部是什么颜色的横条？", + "scope": ["current_chat"], + "top_k": 5, + "app_id": "default", + "project_id": "default" + }' ``` -Response: +Expected result excerpt: ```json { "results": [ { - "id": "gateway_user_20260611180257_ep_20260611_00000001", - "session_id": "chat:gateway-multimodal-20260611180257", - "text": "On June 11, 2026 at 10:02 AM UTC, user gateway_user_20260611180257 uploaded a multimodal memory package via Memory Gateway. The package included an image file named simple-multimodal-image.png and a short test audio clip. The image displayed three geometric shapes on a light gray background: a solid red square in the upper-left, a solid blue circle in the upper-right (horizontally aligned with the square), and a long, thin green horizontal rectangle spanning the bottom below both shapes. The user instructed the system to retain these details, anticipating future queries regarding the objects' positions and colors.", - "score": 0.6069304347038269, + "session_id": "chat:gateway_demo_conversation", "source_scope": "current_chat", - "resource_id": null, - "resource_uri": null, - "raw": { - "id": "gateway_user_20260611180257_ep_20260611_00000001", - "user_id": "gateway_user_20260611180257", - "app_id": "default", - "project_id": "default", - "session_id": "chat:gateway-multimodal-20260611180257", - "timestamp": "2026-06-11T10:02:57Z", - "sender_ids": [ - "gateway_user_20260611180257" - ], - "summary": "On June 11, 2026 at 10:02 AM UTC, user gateway_user_20260611180257 uploaded a multimodal memory package via Memory Gateway. The package included an image file named simple-multimodal-image.png and a s", - "subject": "gateway_user_20260611180257 Multimodal Memory Upload June 11, 2026", - "episode": "On June 11, 2026 at 10:02 AM UTC, user gateway_user_20260611180257 uploaded a multimodal memory package via Memory Gateway. The package included an image file named simple-multimodal-image.png and a short test audio clip. The image displayed three geometric shapes on a light gray background: a solid red square in the upper-left, a solid blue circle in the upper-right (horizontally aligned with the square), and a long, thin green horizontal rectangle spanning the bottom below both shapes. The user instructed the system to retain these details, anticipating future queries regarding the objects' positions and colors.", - "type": "Conversation", - "score": 0.6069304347038269, - "atomic_facts": [ - { - "id": "gateway_user_20260611180257_af_20260611_00000004", - "content": "gateway_user_20260611180257 stated that questions about the positions and colors of the objects in the image might be asked in the future.", - "score": 0.6069304347038269 - } - ] - } + "text": "The image contained a red square, a blue circle, and a green horizontal rectangle.", + "attachments": [ + { + "type": "image", + "name": "simple-multimodal-image.png", + "internal_uri": "file:///home/tom/memory-gateway/data/storage/..." + }, + { + "type": "audio", + "name": "simple-tone.wav", + "internal_uri": "file:///home/tom/memory-gateway/data/storage/..." + } + ] } ] } ``` -HTTP metadata: +## 6. Upload an independent resource -```text -HTTP_STATUS:200 -TOTAL_TIME:0.064128 -``` - -# Other Memory Gateway API tests - -The following calls used a temporary Gateway database and storage directory. All requests target Memory Gateway at `http://127.0.0.1:8010`. - -## 5. Health - -Request: +Use `/resources` when the file is an independent resource, not just an +attachment inside one chat message. ```bash -curl -sS --location 'http://127.0.0.1:8010/health' +curl -sS -X POST http://127.0.0.1:8010/resources \ + -F 'user_id=gateway_demo_user' \ + -F 'user_key=' \ + -F 'app_id=default' \ + -F 'project_id=default' \ + -F 'title=Gateway demo image resource' \ + -F 'description=Demo upload for simple multimodal image' \ + -F 'file=@tests/simple-multimodal-image.png;type=image/png' ``` -Response: +Expected shape: ```json { - "status": "ok", - "api": {"status": "ok"}, - "backend": { - "status": "ok", - "base_url": "http://10.6.80.123:1995", - "data": {"status": "ok"} - } -} -``` - -```text -HTTP_STATUS:200 -TOTAL_TIME:0.034914 -``` - -## 6. Invalid credentials - -Request: - -```bash -curl -sS --location \ - 'http://127.0.0.1:8010/resources?user_id=other_api_20260612095541&user_key=wrong-key' -``` - -Response: - -```json -{"detail":"invalid user credentials"} -``` - -```text -HTTP_STATUS:401 -TOTAL_TIME:0.001447 -``` - -## 7. Upload resource - -The temporary test user was created with: - -```bash -curl -sS --location 'http://127.0.0.1:8010/users' \ - --header 'Content-Type: application/json' \ - --data '{"user_id":"other_api_20260612095541"}' -``` - -User response: - -```json -{ - "user_id": "other_api_20260612095541", - "user_key": "uk_REDACTED", - "created_at": "2026-06-12T01:55:41.448076+00:00" -} -``` - -Upload request: - -```bash -cd /home/tom/memory-gateway - -curl -sS --location 'http://127.0.0.1:8010/resources' \ - --form 'user_id=other_api_20260612095541' \ - --form 'user_key=uk_REDACTED' \ - --form 'app_id=default' \ - --form 'project_id=default' \ - --form 'title=Gateway API image resource' \ - --form 'description=Resource lifecycle test through Memory Gateway' \ - --form 'file=@tests/simple-multimodal-image.png;type=image/png' -``` - -Response: - -```json -{ - "resource_id": "r_2700e435f72a49e6a7f736d17f8c7ac7", - "session_id": "resource:other_api_20260612095541:r_2700e435f72a49e6a7f736d17f8c7ac7", - "uri": "resource://other_api_20260612095541/r_2700e435f72a49e6a7f736d17f8c7ac7", + "resource_id": "r_1678eacf3e8c49f9a8863454c5b35e68", + "session_id": "resource:gateway_demo_user:r_1678eacf3e8c49f9a8863454c5b35e68", + "uri": "resource://gateway_demo_user/r_1678eacf3e8c49f9a8863454c5b35e68", "status": "extracted" } ``` -```text -HTTP_STATUS:200 -TOTAL_TIME:4.700296 -``` +Unlike `/memories/add/multipart`, `/resources` automatically calls upstream add +and flush. -## 8. List resources - -Request: +## 7. List resources ```bash -curl -sS --location \ - 'http://127.0.0.1:8010/resources?user_id=other_api_20260612095541&user_key=uk_REDACTED' +curl -sS \ + 'http://127.0.0.1:8010/resources?user_id=gateway_demo_user&user_key=' ``` -Response: +Expected shape: ```json { "resources": [ { - "resource_id": "r_2700e435f72a49e6a7f736d17f8c7ac7", - "user_id": "other_api_20260612095541", + "resource_id": "r_1678eacf3e8c49f9a8863454c5b35e68", "filename": "simple-multimodal-image.png", "content_type": "image", "mime_type": "image/png", - "uri": "resource://other_api_20260612095541/r_2700e435f72a49e6a7f736d17f8c7ac7", - "session_id": "resource:other_api_20260612095541:r_2700e435f72a49e6a7f736d17f8c7ac7", - "status": "extracted", - "title": "Gateway API image resource", - "description": "Resource lifecycle test through Memory Gateway", - "created_at": "2026-06-12T01:55:41.527716+00:00", - "updated_at": "2026-06-12T01:55:46.204082+00:00" + "uri": "resource://gateway_demo_user/r_1678eacf3e8c49f9a8863454c5b35e68", + "session_id": "resource:gateway_demo_user:r_1678eacf3e8c49f9a8863454c5b35e68", + "status": "extracted" } ] } ``` -```text -HTTP_STATUS:200 -TOTAL_TIME:0.001785 -``` - -## 9. Resource detail - -Request: +## 8. Search resources ```bash -curl -sS --location \ - 'http://127.0.0.1:8010/resources/r_2700e435f72a49e6a7f736d17f8c7ac7?user_id=other_api_20260612095541&user_key=uk_REDACTED' -``` - -Response: - -```json -{ - "resources": [ - { - "resource_id": "r_2700e435f72a49e6a7f736d17f8c7ac7", - "user_id": "other_api_20260612095541", - "filename": "simple-multimodal-image.png", - "content_type": "image", - "mime_type": "image/png", - "uri": "resource://other_api_20260612095541/r_2700e435f72a49e6a7f736d17f8c7ac7", - "session_id": "resource:other_api_20260612095541:r_2700e435f72a49e6a7f736d17f8c7ac7", - "status": "extracted", - "title": "Gateway API image resource", - "description": "Resource lifecycle test through Memory Gateway", - "created_at": "2026-06-12T01:55:41.527716+00:00", - "updated_at": "2026-06-12T01:55:46.204082+00:00" - } - ] -} -``` - -```text -HTTP_STATUS:200 -TOTAL_TIME:0.001634 -``` - -## 10. Search resource memory - -Request: - -```bash -curl -sS --location 'http://127.0.0.1:8010/memories/search' \ - --header 'Content-Type: application/json' \ - --data '{ - "user_id": "other_api_20260612095541", - "user_key": "uk_REDACTED", - "query": "图片中有哪些颜色和形状？", +curl -sS -X POST http://127.0.0.1:8010/memories/search \ + -H 'Content-Type: application/json' \ + -d '{ + "user_id": "gateway_demo_user", + "user_key": "", + "query": "这张资源图片里有哪些几何图形和颜色？", "scope": ["resources"], "top_k": 5, "app_id": "default", @@ -456,251 +283,43 @@ curl -sS --location 'http://127.0.0.1:8010/memories/search' \ }' ``` -Response: +Expected result excerpt: ```json { "results": [ { - "id": "other_api_20260612095541_ep_20260612_00000001", - "session_id": "resource:other_api_20260612095541:r_2700e435f72a49e6a7f736d17f8c7ac7", - "text": "On June 12, 2026 at 01:55 AM UTC, the user other_api_20260612095541 uploaded an image titled 'simple-multimodal-image.png' for visual analysis. The image displayed three distinct geometric shapes on a plain, light gray background. The composition included a solid red square in the upper-left portion, a solid blue circle in the upper-right portion, and a long, thin, horizontal green rectangle situated below both shapes. The red square and blue circle were roughly aligned horizontally, while the green rectangle spanned a width greater than either of the upper shapes.", - "score": 0.6418947577476501, "source_scope": "resources", - "resource_id": "r_2700e435f72a49e6a7f736d17f8c7ac7", - "resource_uri": "resource://other_api_20260612095541/r_2700e435f72a49e6a7f736d17f8c7ac7", - "raw": { - "id": "other_api_20260612095541_ep_20260612_00000001", - "user_id": "other_api_20260612095541", - "app_id": "default", - "project_id": "default", - "session_id": "resource:other_api_20260612095541:r_2700e435f72a49e6a7f736d17f8c7ac7", - "timestamp": "2026-06-12T01:55:41.541000Z", - "sender_ids": ["other_api_20260612095541"], - "summary": "On June 12, 2026 at 01:55 AM UTC, the user other_api_20260612095541 uploaded an image titled 'simple-multimodal-image.png' for visual analysis. The image displayed three distinct geometric shapes on a", - "subject": "Visual Analysis of Geometric Shapes Uploaded by other_api_20260612095541 on June 12, 2026", - "episode": "On June 12, 2026 at 01:55 AM UTC, the user other_api_20260612095541 uploaded an image titled 'simple-multimodal-image.png' for visual analysis. The image displayed three distinct geometric shapes on a plain, light gray background. The composition included a solid red square in the upper-left portion, a solid blue circle in the upper-right portion, and a long, thin, horizontal green rectangle situated below both shapes. The red square and blue circle were roughly aligned horizontally, while the green rectangle spanned a width greater than either of the upper shapes.", - "type": "Conversation", - "score": 0.6418947577476501, - "atomic_facts": [ - { - "id": "other_api_20260612095541_af_20260612_00000001", - "content": "The image displays three distinct geometric shapes on a plain, light gray background.", - "score": 0.6418947577476501 - } - ] - } + "resource_id": "r_1678eacf3e8c49f9a8863454c5b35e68", + "resource_uri": "resource://gateway_demo_user/r_1678eacf3e8c49f9a8863454c5b35e68", + "text": "The image displayed a red square, a blue circle, and a green rectangle.", + "attachments": [ + { + "type": "image", + "name": "simple-multimodal-image.png", + "internal_uri": "file:///home/tom/memory-gateway/data/storage/..." + } + ] } ] } ``` -```text -HTTP_STATUS:200 -TOTAL_TIME:0.176981 -``` +## Multipart vs resources -## 11. Override memory +Use `/memories/add/multipart` when the upload belongs to a chat/session message: -Request: +- caller supplies `session_id`, usually `chat:{conversation_id}`; +- caller defines `upload_id` values in `messages`; +- caller uploads files as form fields with names matching `upload_id`; +- Gateway only calls upstream add; +- caller should call `/memories/flush`; +- search normally uses `current_chat` or `all_user_memory`. -```bash -curl -sS --location --request PATCH \ - 'http://127.0.0.1:8010/memories/other_api_20260612095541_ep_20260612_00000001' \ - --header 'Content-Type: application/json' \ - --data '{ - "user_id": "other_api_20260612095541", - "user_key": "uk_REDACTED", - "session_id": "resource:other_api_20260612095541:r_2700e435f72a49e6a7f736d17f8c7ac7", - "override_text": "OVERRIDE: 图片左侧是红色方块，右侧是蓝色圆形，底部是绿色横条。" - }' -``` +Use `/resources` when the upload is an independent resource: -Response: - -```json -{ - "memory_id": "other_api_20260612095541_ep_20260612_00000001", - "override_id": "o_328f03b40b164c4896640fd2567042cb", - "status": "active" -} -``` - -```text -HTTP_STATUS:200 -TOTAL_TIME:0.007037 -``` - -The next search returned the overridden text: - -Request: - -```bash -curl -sS --location 'http://127.0.0.1:8010/memories/search' \ - --header 'Content-Type: application/json' \ - --data '{ - "user_id": "other_api_20260612095541", - "user_key": "uk_REDACTED", - "query": "图片中有哪些颜色和形状？", - "scope": ["resources"], - "top_k": 5, - "app_id": "default", - "project_id": "default" - }' -``` - -```json -{ - "results": [ - { - "id": "other_api_20260612095541_ep_20260612_00000001", - "session_id": "resource:other_api_20260612095541:r_2700e435f72a49e6a7f736d17f8c7ac7", - "text": "OVERRIDE: 图片左侧是红色方块，右侧是蓝色圆形，底部是绿色横条。", - "score": 0.6418947577476501, - "source_scope": "resources", - "resource_id": "r_2700e435f72a49e6a7f736d17f8c7ac7", - "resource_uri": "resource://other_api_20260612095541/r_2700e435f72a49e6a7f736d17f8c7ac7", - "raw": { - "id": "other_api_20260612095541_ep_20260612_00000001", - "user_id": "other_api_20260612095541", - "app_id": "default", - "project_id": "default", - "session_id": "resource:other_api_20260612095541:r_2700e435f72a49e6a7f736d17f8c7ac7", - "timestamp": "2026-06-12T01:55:41.541000Z", - "sender_ids": ["other_api_20260612095541"], - "summary": "On June 12, 2026 at 01:55 AM UTC, the user other_api_20260612095541 uploaded an image titled 'simple-multimodal-image.png' for visual analysis. The image displayed three distinct geometric shapes on a", - "subject": "Visual Analysis of Geometric Shapes Uploaded by other_api_20260612095541 on June 12, 2026", - "episode": "On June 12, 2026 at 01:55 AM UTC, the user other_api_20260612095541 uploaded an image titled 'simple-multimodal-image.png' for visual analysis. The image displayed three distinct geometric shapes on a plain, light gray background. The composition included a solid red square in the upper-left portion, a solid blue circle in the upper-right portion, and a long, thin, horizontal green rectangle situated below both shapes. The red square and blue circle were roughly aligned horizontally, while the green rectangle spanned a width greater than either of the upper shapes.", - "type": "Conversation", - "score": 0.6418947577476501, - "atomic_facts": [ - { - "id": "other_api_20260612095541_af_20260612_00000001", - "content": "The image displays three distinct geometric shapes on a plain, light gray background.", - "score": 0.6418947577476501 - } - ] - }, - "override_id": "o_328f03b40b164c4896640fd2567042cb" - } - ] -} -``` - -```text -HTTP_STATUS:200 -TOTAL_TIME:0.055485 -``` - -## 12. Delete memory with tombstone - -Request: - -```bash -curl -sS --location --request DELETE \ - 'http://127.0.0.1:8010/memories/other_api_20260612095541_ep_20260612_00000001' \ - --header 'Content-Type: application/json' \ - --data '{ - "user_id": "other_api_20260612095541", - "user_key": "uk_REDACTED", - "session_id": "resource:other_api_20260612095541:r_2700e435f72a49e6a7f736d17f8c7ac7", - "reason": "Gateway API tombstone test" - }' -``` - -Response: - -```json -{ - "memory_id": "other_api_20260612095541_ep_20260612_00000001", - "tombstone_id": "t_2cba49bf3b6641ea96865612deebc036", - "status": "deleted" -} -``` - -```text -HTTP_STATUS:200 -TOTAL_TIME:0.006502 -``` - -Repeating the resource search after creating the tombstone: - -```bash -curl -sS --location 'http://127.0.0.1:8010/memories/search' \ - --header 'Content-Type: application/json' \ - --data '{ - "user_id": "other_api_20260612095541", - "user_key": "uk_REDACTED", - "query": "图片中有哪些颜色和形状？", - "scope": ["resources"], - "top_k": 5, - "app_id": "default", - "project_id": "default" - }' -``` - -```json -{"results":[]} -``` - -```text -HTTP_STATUS:200 -TOTAL_TIME:0.067841 -``` - -## 13. Delete resource - -Request: - -```bash -curl -sS --location --request DELETE \ - 'http://127.0.0.1:8010/resources/r_2700e435f72a49e6a7f736d17f8c7ac7?user_id=other_api_20260612095541&user_key=uk_REDACTED' -``` - -Response: - -```json -{ - "resource_id": "r_2700e435f72a49e6a7f736d17f8c7ac7", - "session_id": "resource:other_api_20260612095541:r_2700e435f72a49e6a7f736d17f8c7ac7", - "uri": "resource://other_api_20260612095541/r_2700e435f72a49e6a7f736d17f8c7ac7", - "status": "deleted" -} -``` - -```text -HTTP_STATUS:200 -TOTAL_TIME:0.014089 -``` - -List after deletion: - -```bash -curl -sS --location \ - 'http://127.0.0.1:8010/resources?user_id=other_api_20260612095541&user_key=uk_REDACTED' -``` - -```json -{"resources":[]} -``` - -```text -HTTP_STATUS:200 -TOTAL_TIME:0.001226 -``` - -Detail after deletion: - -```bash -curl -sS --location \ - 'http://127.0.0.1:8010/resources/r_2700e435f72a49e6a7f736d17f8c7ac7?user_id=other_api_20260612095541&user_key=uk_REDACTED' -``` - -```json -{"resources":[]} -``` - -```text -HTTP_STATUS:200 -TOTAL_TIME:0.001223 -``` +- Gateway creates `resource_id`; +- Gateway creates `session_id = resource:{user_id}:{resource_id}`; +- Gateway writes `user_resources`; +- Gateway automatically calls upstream add and flush; +- search normally uses `resources`. diff --git a/tests/test_gateway.py b/tests/test_gateway.py index 92e7799..47d7b54 100644 --- a/tests/test_gateway.py +++ b/tests/test_gateway.py @@ -897,6 +897,117 @@ async def test_add_memory_materializes_base64_attachment( assert backend.add_calls[0]["messages"][0]["content"][0]["base64"] == encoded +@pytest.mark.asyncio +async def test_add_memory_multipart_uploads_files_to_chat_session( + config: GatewayConfig, + repo: MemoryRepository, +) -> None: + backend = FakeBackendClient() + messages = [ + { + "sender_id": "u_123", + "role": "user", + "timestamp": 1234567890123, + "content": [ + {"type": "text", "text": "remember these attachments"}, + { + "type": "image", + "upload_id": "image_1", + "name": "picture.png", + "ext": "png", + }, + { + "type": "audio", + "upload_id": "audio_1", + "name": "tone.wav", + "ext": "wav", + }, + ], + } + ] + + async with app_client(config, backend) as client: + user_key = await create_user(client) + response = await client.post( + "/memories/add/multipart", + data={ + "user_id": "u_123", + "user_key": user_key, + "session_id": "chat:c_uploads", + "app_id": "default", + "project_id": "default", + "messages": json.dumps(messages), + }, + files=[ + ("image_1", ("picture.png", b"png bytes", "image/png")), + ("audio_1", ("tone.wav", b"wav bytes", "audio/wav")), + ], + ) + + assert response.status_code == 200, response.text + assert response.json() == { + "session_id": "chat:c_uploads", + "backend": {"request_id": "add", "data": {"status": "accumulated"}}, + } + content = backend.add_calls[0]["messages"][0]["content"] + assert content == [ + {"type": "text", "text": "remember these attachments"}, + { + "type": "image", + "name": "picture.png", + "ext": "png", + "base64": base64.b64encode(b"png bytes").decode("ascii"), + }, + { + "type": "audio", + "name": "tone.wav", + "ext": "wav", + "base64": base64.b64encode(b"wav bytes").decode("ascii"), + }, + ] + attachments = repo.list_attachments_for_session("u_123", "chat:c_uploads") + assert [(item["name"], item["source"]) for item in attachments] == [ + ("picture.png", "memory_add_upload"), + ("tone.wav", "memory_add_upload"), + ] + for attachment in attachments: + path = Path(attachment["internal_uri"].removeprefix("file://")) + assert path.exists() + + +@pytest.mark.asyncio +async def test_add_memory_multipart_rejects_missing_upload_file( + config: GatewayConfig, +) -> None: + backend = FakeBackendClient() + messages = [ + { + "sender_id": "u_123", + "role": "user", + "timestamp": 1234567890123, + "content": [ + {"type": "image", "upload_id": "image_1", "name": "picture.png"} + ], + } + ] + + async with app_client(config, backend) as client: + user_key = await create_user(client) + response = await client.post( + "/memories/add/multipart", + data={ + "user_id": "u_123", + "user_key": user_key, + "session_id": "chat:c_missing_upload", + "messages": json.dumps(messages), + }, + ) + + assert response.status_code == 422 + assert "missing upload file for upload_id: image_1" in response.text + assert backend.add_calls == [] + + @pytest.mark.asyncio async def test_add_memory_deduplicates_retried_base64_attachment( config: GatewayConfig,