375 lines
9.8 KiB
Markdown
375 lines
9.8 KiB
Markdown
# Memory Gateway
|
||
|
||
Memory Gateway 是一个本地 memory/context gateway，用统一的 HTTP、MCP 和 Hermes skill 入口，把上层 agent 的记忆写入、上下文检索、会话提交和知识沉淀路由到 OpenViking、EverOS、SQLite metadata store 和可选 Obsidian vault。
|
||
|
||
当前项目的主线是 **OpenViking + EverOS 双后端**：
|
||
|
||
- OpenViking 运行在 `127.0.0.1:1933`，负责 session/resource/context 层能力。
|
||
- EverOS/EverCore 运行在 `127.0.0.1:1995`，负责长期记忆、profile 和检索。
|
||
- Memory Gateway 默认运行在 `127.0.0.1:1934`，提供统一 API、认证、metadata、outbox 和 adapter 编排。
|
||
|
||
## 核心能力
|
||
|
||
- `/v2/conversations/ingest`：把一轮对话写入 OpenViking 和 EverOS，并在本地保存 `memory_refs` 控制面引用。
|
||
- `/v2/context/retrieve`：实际调用 OpenViking / EverOS 的 retrieve 接口，把两个后端返回的上下文合并到 `items`。
|
||
- `/v2/conversations/{session_id}/commit`：创建 commit job 和 outbox events，用于异步生成长期 ref。
|
||
- `/v2/admin/outbox/process`：处理 pending outbox，生成 OpenViking session ref、EverOS profile ref、EverOS long-term ref。
|
||
- `/v2/memory/refs`：查询本地保存的后端引用元数据。
|
||
- `/v1/*`：保留基础用户、memory、episode、session commit、audit 和 EverOS health 能力。
|
||
- `/api/*`：保留旧版搜索、写 memory/resource、LLM summary、文档上传到 Obsidian/OpenViking 的兼容接口。
|
||
- `/mcp/rpc` 与 `/mcp/sse`：提供 MCP 调用入口。
|
||
|
||
## 架构
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
Agent[Agent / Hermes / MCP Client] --> Gateway[Memory Gateway]
|
||
Gateway --> SQLite[(SQLite metadata)]
|
||
Gateway --> OpenViking[OpenViking API]
|
||
Gateway --> EverOS[EverOS / EverCore API]
|
||
Gateway --> Obsidian[Obsidian Markdown Vault]
|
||
Gateway --> LLM[OpenAI-compatible LLM]
|
||
|
||
SQLite --> Refs[memory_refs / outbox_events / commit_jobs]
|
||
OpenViking --> OVStore[(session / resource / context index)]
|
||
EverOS --> EverStore[(memory / profile / long-term store)]
|
||
```
|
||
|
||
Memory Gateway 不把完整对话正文长期保存到 SQLite。SQLite 主要保存控制面 metadata：用户、episode、audit、`memory_refs`、`commit_jobs`、`outbox_events`。真正的记忆正文和检索上下文在 OpenViking / EverOS 后端中。
|
||
|
||
## 项目结构
|
||
|
||
```text
|
||
memory_gateway/
|
||
api_v1.py # v1 REST API
|
||
api_v2.py # v2 workflow API
|
||
server.py # FastAPI + MCP + legacy /api entrypoint
|
||
services.py # v1 service
|
||
services_v2.py # v2 ingest/retrieve/commit/outbox orchestration
|
||
openviking_client.py # OpenViking adapter
|
||
everos_client.py # EverOS adapter
|
||
repositories.py # in-memory / SQLite metadata repository
|
||
schemas.py # v1 schemas
|
||
schemas_v2.py # v2 schemas
|
||
backend_contracts.py # backend adapter result contracts
|
||
backend_normalization.py # backend response normalization
|
||
backend_ref_mapping.py # native ref type -> MemoryRefType mapping
|
||
obsidian_review.py # Obsidian review draft support
|
||
integrations/hermes/memory-gateway/
|
||
SKILL.md
|
||
scripts/
|
||
plugins/memory-gateway-agent/
|
||
tests/
|
||
config.example.yaml
|
||
pyproject.toml
|
||
```
|
||
|
||
## 安装
|
||
|
||
要求 Python 3.10+。
|
||
|
||
```bash
|
||
cd /home/tom/memory-gateway
|
||
python3 -m venv .venv
|
||
source .venv/bin/activate
|
||
pip install -U pip
|
||
pip install -e ".[dev]"
|
||
```
|
||
|
||
如果使用 `uv`：
|
||
|
||
```bash
|
||
cd /home/tom/memory-gateway
|
||
uv sync --extra dev
|
||
```
|
||
|
||
## 依赖服务
|
||
|
||
### OpenViking
|
||
|
||
参考 `/home/tom/OpenViking/CONTRIBUTING.md`。当前约定启动方式：
|
||
|
||
```bash
|
||
openviking-server --host 127.0.0.1 --port 1933
|
||
```
|
||
|
||
配置文件：
|
||
|
||
```text
|
||
/home/tom/.openviking/ov.conf
|
||
```
|
||
|
||
### EverOS / EverCore
|
||
|
||
参考 `/home/tom/EverOS/methods/EverCore/docs/installation/SETUP.md`。当前约定启动方式：
|
||
|
||
```bash
|
||
cd /home/tom/EverOS/methods/EverCore
|
||
uv run python src/run.py --port 1995
|
||
```
|
||
|
||
配置文件：
|
||
|
||
```text
|
||
/home/tom/EverOS/methods/EverCore/.env
|
||
```
|
||
|
||
## 配置
|
||
|
||
复制示例配置：
|
||
|
||
```bash
|
||
cd /home/tom/memory-gateway
|
||
cp config.example.yaml config.yaml
|
||
```
|
||
|
||
核心配置示例：
|
||
|
||
```yaml
|
||
server:
|
||
host: "127.0.0.1"
|
||
port: 1934
|
||
api_key: ""
|
||
|
||
openviking:
|
||
url: "http://127.0.0.1:1933"
|
||
api_key: ""
|
||
timeout: 30
|
||
|
||
everos:
|
||
enabled: true
|
||
mode: "real"
|
||
url: "http://127.0.0.1:1995"
|
||
api_key: ""
|
||
timeout: 30
|
||
health_path: "/health"
|
||
ingest_path: "/api/v1/memories"
|
||
search_path: "/api/v1/memories/search"
|
||
flush_path: "/api/v1/memories/flush"
|
||
retrieve_method: "keyword"
|
||
|
||
storage:
|
||
backend: "sqlite"
|
||
sqlite_path: "/home/tom/memory-gateway/memory_gateway.sqlite3"
|
||
```
|
||
|
||
也可以用环境变量覆盖后端配置，例如：
|
||
|
||
```bash
|
||
export OPENVIKING_URL=http://127.0.0.1:1933
|
||
export EVEROS_URL=http://127.0.0.1:1995
|
||
export EVEROS_MODE=real
|
||
```
|
||
|
||
## 启动
|
||
|
||
```bash
|
||
cd /home/tom/memory-gateway
|
||
source .venv/bin/activate
|
||
python -m memory_gateway.server --config config.yaml
|
||
```
|
||
|
||
也可以显式指定 host/port：
|
||
|
||
```bash
|
||
python -m memory_gateway.server --config config.yaml --host 127.0.0.1 --port 1934
|
||
```
|
||
|
||
健康检查：
|
||
|
||
```bash
|
||
curl http://127.0.0.1:1934/health
|
||
curl http://127.0.0.1:1934/v1/everos/health
|
||
```
|
||
|
||
如果设置了 `server.api_key`，请求需要带：
|
||
|
||
```bash
|
||
-H "X-API-Key: <your-api-key>"
|
||
```
|
||
|
||
## v2 工作流
|
||
|
||
### 1. Ingest 一轮对话
|
||
|
||
```bash
|
||
curl -s http://127.0.0.1:1934/v2/conversations/ingest \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{
|
||
"workspace_id": "ws_1",
|
||
"user_id": "user_a",
|
||
"agent_id": "agent_cli",
|
||
"session_id": "sess_1",
|
||
"turn_id": "turn_1",
|
||
"namespace": "workspace/ws_1/user/user_a",
|
||
"role": "user",
|
||
"content": "Remember that the demo environment uses EverOS and OpenViking.",
|
||
"metadata": {"channel": "manual-test"}
|
||
}'
|
||
```
|
||
|
||
结果中的 `refs` 是本地 `memory_refs` 控制面引用，通常包括：
|
||
|
||
- OpenViking `session_archive` ref
|
||
- EverOS `message_memory` ref
|
||
|
||
这些 refs 保存的是 native id/uri、状态、hash、trace 等 metadata，不是完整记忆正文。
|
||
|
||
### 2. Retrieve 上下文
|
||
|
||
```bash
|
||
curl -s http://127.0.0.1:1934/v2/context/retrieve \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{
|
||
"workspace_id": "ws_1",
|
||
"user_id": "user_a",
|
||
"agent_id": "agent_cli",
|
||
"session_id": "sess_1",
|
||
"namespace": "workspace/ws_1/user/user_a",
|
||
"query": "EverOS OpenViking demo environment",
|
||
"limit": 5,
|
||
"metadata": {"trace_id": "trace_manual_1"}
|
||
}'
|
||
```
|
||
|
||
返回结构重点：
|
||
|
||
- `items`：真实上下文，由 OpenViking / EverOS retrieve 返回后合并，包含 `text`、`source_backend`、`ref_id`、`score`、`memory_type`。
|
||
- `refs`：本地已有的 `memory_refs` 视图，用于追踪哪些后端引用已保存。
|
||
- `metadata.backend_results`：每个后端 retrieve 的状态、返回数量和错误信息。
|
||
|
||
### 3. Commit 一个 session
|
||
|
||
```bash
|
||
curl -s http://127.0.0.1:1934/v2/conversations/sess_1/commit \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{
|
||
"workspace_id": "ws_1",
|
||
"user_id": "user_a",
|
||
"agent_id": "agent_cli",
|
||
"namespace": "workspace/ws_1/user/user_a"
|
||
}'
|
||
```
|
||
|
||
该接口只创建 commit job 和 outbox events，不直接执行长期记忆生成。返回中会有 `job_id` 和 `metadata.gateway_id`。
|
||
|
||
### 4. Process outbox
|
||
|
||
```bash
|
||
curl -s -X POST 'http://127.0.0.1:1934/v2/admin/outbox/process?limit=20'
|
||
```
|
||
|
||
处理成功后会生成长期 refs：
|
||
|
||
- OpenViking `session_archive` ref：session archive / summary 的 native 引用。
|
||
- EverOS `profile` ref：用户 profile 的 native 引用。
|
||
- EverOS `long_term_memory` ref：session 提炼出的长期记忆 native 引用。
|
||
|
||
这些 ref 保存在 SQLite 的 `memory_refs` 表中。
|
||
|
||
### 5. 查看 refs 和 job
|
||
|
||
```bash
|
||
curl -s 'http://127.0.0.1:1934/v2/memory/refs?workspace_id=ws_1&user_id=user_a&session_id=sess_1&limit=20'
|
||
|
||
curl -s http://127.0.0.1:1934/v2/jobs/<job_id>
|
||
```
|
||
|
||
SQLite 默认路径取决于配置，例如：
|
||
|
||
```text
|
||
/home/tom/memory-gateway/memory_gateway.sqlite3
|
||
```
|
||
|
||
主要表：
|
||
|
||
- `memory_refs`
|
||
- `outbox_events`
|
||
- `commit_jobs`
|
||
- `audit_logs`
|
||
- `users` / `memories` / `episodes` / `profiles`
|
||
|
||
## v1 和 legacy API
|
||
|
||
v1 保留用户隔离、namespace、visibility/ACL、episode、session commit、audit 等基础能力：
|
||
|
||
```text
|
||
POST /v1/users
|
||
GET /v1/users/{user_id}
|
||
POST /v1/memory/search
|
||
POST /v1/memory
|
||
GET /v1/memory/{memory_id}
|
||
PATCH /v1/memory/{memory_id}
|
||
DELETE /v1/memory/{memory_id}
|
||
POST /v1/episodes
|
||
POST /v1/sessions/{session_id}/commit
|
||
GET /v1/users/{user_id}/profile
|
||
POST /v1/memory/{memory_id}/feedback
|
||
GET /v1/namespaces
|
||
GET /v1/audit
|
||
GET /v1/everos/health
|
||
```
|
||
|
||
旧 `/api/*` 接口仍保留：
|
||
|
||
```text
|
||
POST /api/search
|
||
POST /api/memory
|
||
POST /api/resource
|
||
POST /api/summary
|
||
POST /api/knowledge/upload
|
||
```
|
||
|
||
## MCP / Hermes
|
||
|
||
MCP endpoints：
|
||
|
||
```text
|
||
POST /mcp/rpc
|
||
GET /mcp/sse
|
||
```
|
||
|
||
Hermes skill 位于：
|
||
|
||
```text
|
||
integrations/hermes/memory-gateway/
|
||
```
|
||
|
||
常用脚本示例：
|
||
|
||
```bash
|
||
python integrations/hermes/memory-gateway/scripts/everos_health.py
|
||
python integrations/hermes/memory-gateway/scripts/memory_commit_session.py --help
|
||
```
|
||
|
||
## 开发与验证
|
||
|
||
运行测试：
|
||
|
||
```bash
|
||
cd /home/tom/memory-gateway
|
||
PYTHONPATH=/home/tom/memory-gateway pytest -q
|
||
```
|
||
|
||
编译检查：
|
||
|
||
```bash
|
||
python -m compileall -q memory_gateway tests integrations/hermes/memory-gateway plugins/memory-gateway-agent
|
||
```
|
||
|
||
Ruff 已在 `pyproject.toml` 中配置。如果本地环境安装了 ruff：
|
||
|
||
```bash
|
||
python -m ruff check .
|
||
```
|
||
|
||
当前仓库不要求真实 OpenViking / EverOS 服务才能跑单元测试；真实服务流程需要先启动 `127.0.0.1:1933` 和 `127.0.0.1:1995`。
|
||
|
||
## 设计约束
|
||
|
||
- SQLite 保存控制面 metadata，不作为长期记忆正文数据库。
|
||
- `refs` 是后端 native 对象引用，不等于上下文正文。
|
||
- `retrieve.items` 才是运行时上下文内容。
|
||
- `commit` 只创建 job/outbox；长期 refs 由 outbox process 生成。
|
||
- 默认只绑定本机地址；远程暴露时必须设置 API key、TLS 和网络访问控制。
|