tomtan/memory-gateway
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Simplify to memory system api

Browse Source
This commit is contained in:
tomtan
2026-05-18 09:54:26 +08:00
parent b226749c61
commit e689b13e4a
134 changed files with 982 additions and 14575 deletions
Download Patch File Download Diff File Expand all files Collapse all files

375
README.md
View File

@ -1,129 +1,50 @@
# Memory Gateway
# Memory System API
Memory Gateway 是一个本地 memory/context gateway，用统一的 HTTP、MCP 和 Hermes skill 入口，把上层 agent 的记忆写入、上下文检索、会话提交和知识沉淀路由到 OpenViking、EverOS、SQLite metadata store 和可选 Obsidian vault。
Memory System API is a lightweight HTTP facade over two memory backends:
当前项目的主线是 **OpenViking + EverOS 双后端**：
- OpenViking stores session conversation memory.
- EverOS stores user profile and episodic memory.
- OpenViking 运行在 `127.0.0.1:1933`，负责 session/resource/context 层能力。
- EverOS/EverCore 运行在 `127.0.0.1:1995`，负责长期记忆、profile 和检索。
- Memory Gateway 默认运行在 `127.0.0.1:1934`，提供统一 API、认证、metadata、outbox 和 adapter 编排。
The caller only sends `user_id`, `session_id`, and optional `user_message` / `assistant_message`.
The API creates or reuses the OpenViking user key, writes messages to both backends, and exposes simple endpoints for commit, immediate extraction, search, and profile reads.
## 核心能力
## Endpoints
- `/v2/conversations/ingest`：把一轮对话写入 OpenViking 和 EverOS，并在本地保存 `memory_refs` 控制面引用。
- `/v2/context/retrieve`：实际调用 OpenViking / EverOS 的 retrieve 接口，把两个后端返回的上下文合并到 `items`。
- `/v2/conversations/{session_id}/commit`：创建 commit job 和 outbox events，用于异步生成长期 ref。
- `/v2/admin/outbox/process`：处理 pending outbox，生成 OpenViking session ref、EverOS profile ref、EverOS long-term ref。
- `/v2/memory/refs`：查询本地保存的后端引用元数据。
- `/v1/*`：保留基础用户、memory、episode、session commit、audit 和 EverOS health 能力。
- `/api/*`：保留旧版搜索、写 memory/resource、LLM summary、文档上传到 Obsidian/OpenViking 的兼容接口。
- `/mcp/rpc` 与 `/mcp/sse`：提供 MCP 调用入口。
## 架构
```mermaid
flowchart LR
Agent[Agent / Hermes / MCP Client] --> Gateway[Memory Gateway]
Gateway --> SQLite[(SQLite metadata)]
Gateway --> OpenViking[OpenViking API]
Gateway --> EverOS[EverOS / EverCore API]
Gateway --> Obsidian[Obsidian Markdown Vault]
Gateway --> LLM[OpenAI-compatible LLM]
SQLite --> Refs[memory_refs / outbox_events / commit_jobs]
OpenViking --> OVStore[(session / resource / context index)]
EverOS --> EverStore[(memory / profile / long-term store)]
```
Memory Gateway 不把完整对话正文长期保存到 SQLite。SQLite 主要保存控制面 metadata：用户、episode、audit、`memory_refs`、`commit_jobs`、`outbox_events`。真正的记忆正文和检索上下文在 OpenViking / EverOS 后端中。
## 项目结构
Base URL:
```text
memory_gateway/
api_v1.py # v1 REST API
api_v2.py # v2 workflow API
server.py # FastAPI + MCP + legacy /api entrypoint
services.py # v1 service
services_v2.py # v2 ingest/retrieve/commit/outbox orchestration
openviking_client.py # OpenViking adapter
everos_client.py # EverOS adapter
repositories.py # in-memory / SQLite metadata repository
schemas.py # v1 schemas
schemas_v2.py # v2 schemas
backend_contracts.py # backend adapter result contracts
backend_normalization.py # backend response normalization
backend_ref_mapping.py # native ref type -> MemoryRefType mapping
obsidian_review.py # Obsidian review draft support
integrations/hermes/memory-gateway/
SKILL.md
scripts/
plugins/memory-gateway-agent/
tests/
config.example.yaml
pyproject.toml
http://127.0.0.1:1934
```
## 安装
Routes:
要求 Python 3.10+。
- `GET /memory-system/health`
- `POST /memory-system/messages`
- `POST /memory-system/sessions/{session_id}/commit`
- `POST /memory-system/sessions/{session_id}/extract`
- `GET /memory-system/openviking/tasks/{task_id}?user_id=...`
- `POST /memory-system/search`
- `GET /memory-system/users/{user_id}/profile`
## Install
```bash
cd /home/tom/memory-gateway
python3 -m venv .venv
python -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -e ".[dev]"
```
如果使用 `uv`：
## Configure
Copy the example config and edit backend URLs or keys as needed:
```bash
cd /home/tom/memory-gateway
uv sync --extra dev
```
## 依赖服务
### OpenViking
参考 `/home/tom/OpenViking/CONTRIBUTING.md`。当前约定启动方式：
```bash
openviking-server --host 127.0.0.1 --port 1933
```
配置文件：
```text
/home/tom/.openviking/ov.conf
```
### EverOS / EverCore
参考 `/home/tom/EverOS/methods/EverCore/docs/installation/SETUP.md`。当前约定启动方式：
```bash
cd /home/tom/EverOS/methods/EverCore
uv run python src/run.py --port 1995
```
配置文件：
```text
/home/tom/EverOS/methods/EverCore/.env
```
## 配置
复制示例配置：
```bash
cd /home/tom/memory-gateway
cp config.example.yaml config.yaml
```
核心配置示例：
Important fields:
```yaml
server:
@ -133,242 +54,88 @@ server:
openviking:
url: "http://127.0.0.1:1933"
api_key: ""
timeout: 30
api_key: "your-secret-root-key"
everos:
enabled: true
mode: "real"
url: "http://127.0.0.1:1995"
api_key: ""
timeout: 30
health_path: "/health"
ingest_path: "/api/v1/memories"
search_path: "/api/v1/memories/search"
flush_path: "/api/v1/memories/flush"
retrieve_method: "keyword"
storage:
backend: "sqlite"
sqlite_path: "/home/tom/memory-gateway/memory_gateway.sqlite3"
```
也可以用环境变量覆盖后端配置，例如：
If `server.api_key` is set, clients must send `X-API-Key`.
## Start
Start OpenViking and EverOS first, then run:
```bash
export OPENVIKING_URL=http://127.0.0.1:1933
export EVEROS_URL=http://127.0.0.1:1995
export EVEROS_MODE=real
python -m memory_system_api.server --config config.yaml --host 127.0.0.1 --port 1934
```
## 启动
## Real Test Flow
Health:
```bash
cd /home/tom/memory-gateway
source .venv/bin/activate
python -m memory_gateway.server --config config.yaml
curl -s http://127.0.0.1:1934/memory-system/health
```
也可以显式指定 host/port：
Write user and assistant messages:
```bash
python -m memory_gateway.server --config config.yaml --host 127.0.0.1 --port 1934
```
健康检查：
```bash
curl http://127.0.0.1:1934/health
curl http://127.0.0.1:1934/v1/everos/health
```
如果设置了 `server.api_key`，请求需要带：
```bash
-H "X-API-Key: <your-api-key>"
```
## v2 工作流
### 1. Ingest 一轮对话
```bash
curl -s http://127.0.0.1:1934/v2/conversations/ingest \
-H 'Content-Type: application/json' \
curl -s -X POST http://127.0.0.1:1934/memory-system/messages \
-H "Content-Type: application/json" \
-d '{
"workspace_id": "ws_1",
"user_id": "user_a",
"agent_id": "agent_cli",
"session_id": "sess_1",
"turn_id": "turn_1",
"namespace": "workspace/ws_1/user/user_a",
"role": "user",
"content": "Remember that the demo environment uses EverOS and OpenViking.",
"metadata": {"channel": "manual-test"}
"user_id": "real_user_001",
"session_id": "real_sess_001",
"user_message": "我喜欢喝拿铁，不喜欢美式。",
"assistant_message": "好的，我会记住你的咖啡偏好。"
}'
```
结果中的 `refs` 是本地 `memory_refs` 控制面引用，通常包括：
- OpenViking `session_archive` ref
- EverOS `message_memory` ref
这些 refs 保存的是 native id/uri、状态、hash、trace 等 metadata，不是完整记忆正文。
### 2. Retrieve 上下文
Commit OpenViking and flush EverOS:
```bash
curl -s http://127.0.0.1:1934/v2/context/retrieve \
-H 'Content-Type: application/json' \
curl -s -X POST http://127.0.0.1:1934/memory-system/sessions/real_sess_001/commit \
-H "Content-Type: application/json" \
-d '{"user_id": "real_user_001"}'
```
Search without LLM planning:
```bash
curl -s -X POST http://127.0.0.1:1934/memory-system/search \
-H "Content-Type: application/json" \
-d '{
"workspace_id": "ws_1",
"user_id": "user_a",
"agent_id": "agent_cli",
"session_id": "sess_1",
"namespace": "workspace/ws_1/user/user_a",
"query": "EverOS OpenViking demo environment",
"limit": 5,
"metadata": {"trace_id": "trace_manual_1"}
"user_id": "real_user_001",
"session_id": "real_sess_001",
"query": "我喜欢喝什么咖啡？",
"use_llm": false,
"limit": 10
}'
```
返回结构重点：
- `items`：真实上下文，由 OpenViking / EverOS retrieve 返回后合并，包含 `text`、`source_backend`、`ref_id`、`score`、`memory_type`。
- `refs`：本地已有的 `memory_refs` 视图，用于追踪哪些后端引用已保存。
- `metadata.backend_results`：每个后端 retrieve 的状态、返回数量和错误信息。
### 3. Commit 一个 session
Search with LLM planning:
```bash
curl -s http://127.0.0.1:1934/v2/conversations/sess_1/commit \
-H 'Content-Type: application/json' \
curl -s -X POST http://127.0.0.1:1934/memory-system/search \
-H "Content-Type: application/json" \
-d '{
"workspace_id": "ws_1",
"user_id": "user_a",
"agent_id": "agent_cli",
"namespace": "workspace/ws_1/user/user_a"
"user_id": "real_user_001",
"session_id": "real_sess_001",
"query": "我的偏好是什么？",
"use_llm": true,
"limit": 10
}'
```
该接口只创建 commit job 和 outbox events，不直接执行长期记忆生成。返回中会有 `job_id` 和 `metadata.gateway_id`。
### 4. Process outbox
Read EverOS profile:
```bash
curl -s -X POST 'http://127.0.0.1:1934/v2/admin/outbox/process?limit=20'
curl -s http://127.0.0.1:1934/memory-system/users/real_user_001/profile
```
处理成功后会生成长期 refs：
- OpenViking `session_archive` ref：session archive / summary 的 native 引用。
- EverOS `profile` ref：用户 profile 的 native 引用。
- EverOS `long_term_memory` ref：session 提炼出的长期记忆 native 引用。
这些 ref 保存在 SQLite 的 `memory_refs` 表中。
### 5. 查看 refs 和 job
## Development Checks
```bash
curl -s 'http://127.0.0.1:1934/v2/memory/refs?workspace_id=ws_1&user_id=user_a&session_id=sess_1&limit=20'
curl -s http://127.0.0.1:1934/v2/jobs/<job_id>
python -m pytest -q
python -m compileall -q memory_system_api tests
```
SQLite 默认路径取决于配置，例如：
```text
/home/tom/memory-gateway/memory_gateway.sqlite3
```
主要表：
- `memory_refs`
- `outbox_events`
- `commit_jobs`
- `audit_logs`
- `users` / `memories` / `episodes` / `profiles`
## v1 和 legacy API
v1 保留用户隔离、namespace、visibility/ACL、episode、session commit、audit 等基础能力：
```text
POST /v1/users
GET /v1/users/{user_id}
POST /v1/memory/search
POST /v1/memory
GET /v1/memory/{memory_id}
PATCH /v1/memory/{memory_id}
DELETE /v1/memory/{memory_id}
POST /v1/episodes
POST /v1/sessions/{session_id}/commit
GET /v1/users/{user_id}/profile
POST /v1/memory/{memory_id}/feedback
GET /v1/namespaces
GET /v1/audit
GET /v1/everos/health
```
旧 `/api/*` 接口仍保留：
```text
POST /api/search
POST /api/memory
POST /api/resource
POST /api/summary
POST /api/knowledge/upload
```
## MCP / Hermes
MCP endpoints：
```text
POST /mcp/rpc
GET /mcp/sse
```
Hermes skill 位于：
```text
integrations/hermes/memory-gateway/
```
常用脚本示例：
```bash
python integrations/hermes/memory-gateway/scripts/everos_health.py
python integrations/hermes/memory-gateway/scripts/memory_commit_session.py --help
```
## 开发与验证
运行测试：
```bash
cd /home/tom/memory-gateway
PYTHONPATH=/home/tom/memory-gateway pytest -q
```
编译检查：
```bash
python -m compileall -q memory_gateway tests integrations/hermes/memory-gateway plugins/memory-gateway-agent
```
Ruff 已在 `pyproject.toml` 中配置。如果本地环境安装了 ruff：
```bash
python -m ruff check .
```
当前仓库不要求真实 OpenViking / EverOS 服务才能跑单元测试；真实服务流程需要先启动 `127.0.0.1:1933` 和 `127.0.0.1:1995`。
## 设计约束
- SQLite 保存控制面 metadata，不作为长期记忆正文数据库。
- `refs` 是后端 native 对象引用，不等于上下文正文。
- `retrieve.items` 才是运行时上下文内容。
- `commit` 只创建 job/outbox；长期 refs 由 outbox process 生成。
- 默认只绑定本机地址；远程暴露时必须设置 API key、TLS 和网络访问控制。