Simplify to memory system api
This commit is contained in:
375
README.md
375
README.md
@ -1,129 +1,50 @@
|
|||||||
# Memory Gateway
|
# Memory System API
|
||||||
|
|
||||||
Memory Gateway 是一个本地 memory/context gateway，用统一的 HTTP、MCP 和 Hermes skill 入口，把上层 agent 的记忆写入、上下文检索、会话提交和知识沉淀路由到 OpenViking、EverOS、SQLite metadata store 和可选 Obsidian vault。
|
Memory System API is a lightweight HTTP facade over two memory backends:
|
||||||
|
|
||||||
当前项目的主线是 **OpenViking + EverOS 双后端**：
|
- OpenViking stores session conversation memory.
|
||||||
|
- EverOS stores user profile and episodic memory.
|
||||||
|
|
||||||
- OpenViking 运行在 `127.0.0.1:1933`，负责 session/resource/context 层能力。
|
The caller only sends `user_id`, `session_id`, and optional `user_message` / `assistant_message`.
|
||||||
- EverOS/EverCore 运行在 `127.0.0.1:1995`，负责长期记忆、profile 和检索。
|
The API creates or reuses the OpenViking user key, writes messages to both backends, and exposes simple endpoints for commit, immediate extraction, search, and profile reads.
|
||||||
- Memory Gateway 默认运行在 `127.0.0.1:1934`，提供统一 API、认证、metadata、outbox 和 adapter 编排。
|
|
||||||
|
|
||||||
## 核心能力
|
## Endpoints
|
||||||
|
|
||||||
- `/v2/conversations/ingest`：把一轮对话写入 OpenViking 和 EverOS，并在本地保存 `memory_refs` 控制面引用。
|
Base URL:
|
||||||
- `/v2/context/retrieve`：实际调用 OpenViking / EverOS 的 retrieve 接口，把两个后端返回的上下文合并到 `items`。
|
|
||||||
- `/v2/conversations/{session_id}/commit`：创建 commit job 和 outbox events，用于异步生成长期 ref。
|
|
||||||
- `/v2/admin/outbox/process`：处理 pending outbox，生成 OpenViking session ref、EverOS profile ref、EverOS long-term ref。
|
|
||||||
- `/v2/memory/refs`：查询本地保存的后端引用元数据。
|
|
||||||
- `/v1/*`：保留基础用户、memory、episode、session commit、audit 和 EverOS health 能力。
|
|
||||||
- `/api/*`：保留旧版搜索、写 memory/resource、LLM summary、文档上传到 Obsidian/OpenViking 的兼容接口。
|
|
||||||
- `/mcp/rpc` 与 `/mcp/sse`：提供 MCP 调用入口。
|
|
||||||
|
|
||||||
## 架构
|
|
||||||
|
|
||||||
```mermaid
|
|
||||||
flowchart LR
|
|
||||||
Agent[Agent / Hermes / MCP Client] --> Gateway[Memory Gateway]
|
|
||||||
Gateway --> SQLite[(SQLite metadata)]
|
|
||||||
Gateway --> OpenViking[OpenViking API]
|
|
||||||
Gateway --> EverOS[EverOS / EverCore API]
|
|
||||||
Gateway --> Obsidian[Obsidian Markdown Vault]
|
|
||||||
Gateway --> LLM[OpenAI-compatible LLM]
|
|
||||||
|
|
||||||
SQLite --> Refs[memory_refs / outbox_events / commit_jobs]
|
|
||||||
OpenViking --> OVStore[(session / resource / context index)]
|
|
||||||
EverOS --> EverStore[(memory / profile / long-term store)]
|
|
||||||
```
|
|
||||||
|
|
||||||
Memory Gateway 不把完整对话正文长期保存到 SQLite。SQLite 主要保存控制面 metadata：用户、episode、audit、`memory_refs`、`commit_jobs`、`outbox_events`。真正的记忆正文和检索上下文在 OpenViking / EverOS 后端中。
|
|
||||||
|
|
||||||
## 项目结构
|
|
||||||
|
|
||||||
```text
|
```text
|
||||||
memory_gateway/
|
http://127.0.0.1:1934
|
||||||
api_v1.py # v1 REST API
|
|
||||||
api_v2.py # v2 workflow API
|
|
||||||
server.py # FastAPI + MCP + legacy /api entrypoint
|
|
||||||
services.py # v1 service
|
|
||||||
services_v2.py # v2 ingest/retrieve/commit/outbox orchestration
|
|
||||||
openviking_client.py # OpenViking adapter
|
|
||||||
everos_client.py # EverOS adapter
|
|
||||||
repositories.py # in-memory / SQLite metadata repository
|
|
||||||
schemas.py # v1 schemas
|
|
||||||
schemas_v2.py # v2 schemas
|
|
||||||
backend_contracts.py # backend adapter result contracts
|
|
||||||
backend_normalization.py # backend response normalization
|
|
||||||
backend_ref_mapping.py # native ref type -> MemoryRefType mapping
|
|
||||||
obsidian_review.py # Obsidian review draft support
|
|
||||||
integrations/hermes/memory-gateway/
|
|
||||||
SKILL.md
|
|
||||||
scripts/
|
|
||||||
plugins/memory-gateway-agent/
|
|
||||||
tests/
|
|
||||||
config.example.yaml
|
|
||||||
pyproject.toml
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## 安装
|
Routes:
|
||||||
|
|
||||||
要求 Python 3.10+。
|
- `GET /memory-system/health`
|
||||||
|
- `POST /memory-system/messages`
|
||||||
|
- `POST /memory-system/sessions/{session_id}/commit`
|
||||||
|
- `POST /memory-system/sessions/{session_id}/extract`
|
||||||
|
- `GET /memory-system/openviking/tasks/{task_id}?user_id=...`
|
||||||
|
- `POST /memory-system/search`
|
||||||
|
- `GET /memory-system/users/{user_id}/profile`
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd /home/tom/memory-gateway
|
cd /home/tom/memory-gateway
|
||||||
python3 -m venv .venv
|
python -m venv .venv
|
||||||
source .venv/bin/activate
|
source .venv/bin/activate
|
||||||
pip install -U pip
|
pip install -U pip
|
||||||
pip install -e ".[dev]"
|
pip install -e ".[dev]"
|
||||||
```
|
```
|
||||||
|
|
||||||
如果使用 `uv`：
|
## Configure
|
||||||
|
|
||||||
|
Copy the example config and edit backend URLs or keys as needed:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd /home/tom/memory-gateway
|
|
||||||
uv sync --extra dev
|
|
||||||
```
|
|
||||||
|
|
||||||
## 依赖服务
|
|
||||||
|
|
||||||
### OpenViking
|
|
||||||
|
|
||||||
参考 `/home/tom/OpenViking/CONTRIBUTING.md`。当前约定启动方式：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
openviking-server --host 127.0.0.1 --port 1933
|
|
||||||
```
|
|
||||||
|
|
||||||
配置文件：
|
|
||||||
|
|
||||||
```text
|
|
||||||
/home/tom/.openviking/ov.conf
|
|
||||||
```
|
|
||||||
|
|
||||||
### EverOS / EverCore
|
|
||||||
|
|
||||||
参考 `/home/tom/EverOS/methods/EverCore/docs/installation/SETUP.md`。当前约定启动方式：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd /home/tom/EverOS/methods/EverCore
|
|
||||||
uv run python src/run.py --port 1995
|
|
||||||
```
|
|
||||||
|
|
||||||
配置文件：
|
|
||||||
|
|
||||||
```text
|
|
||||||
/home/tom/EverOS/methods/EverCore/.env
|
|
||||||
```
|
|
||||||
|
|
||||||
## 配置
|
|
||||||
|
|
||||||
复制示例配置：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd /home/tom/memory-gateway
|
|
||||||
cp config.example.yaml config.yaml
|
cp config.example.yaml config.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
核心配置示例：
|
Important fields:
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
server:
|
server:
|
||||||
@ -133,242 +54,88 @@ server:
|
|||||||
|
|
||||||
openviking:
|
openviking:
|
||||||
url: "http://127.0.0.1:1933"
|
url: "http://127.0.0.1:1933"
|
||||||
api_key: ""
|
api_key: "your-secret-root-key"
|
||||||
timeout: 30
|
|
||||||
|
|
||||||
everos:
|
everos:
|
||||||
enabled: true
|
|
||||||
mode: "real"
|
|
||||||
url: "http://127.0.0.1:1995"
|
url: "http://127.0.0.1:1995"
|
||||||
api_key: ""
|
|
||||||
timeout: 30
|
|
||||||
health_path: "/health"
|
|
||||||
ingest_path: "/api/v1/memories"
|
|
||||||
search_path: "/api/v1/memories/search"
|
|
||||||
flush_path: "/api/v1/memories/flush"
|
|
||||||
retrieve_method: "keyword"
|
|
||||||
|
|
||||||
storage:
|
|
||||||
backend: "sqlite"
|
|
||||||
sqlite_path: "/home/tom/memory-gateway/memory_gateway.sqlite3"
|
|
||||||
```
|
```
|
||||||
|
|
||||||
也可以用环境变量覆盖后端配置，例如：
|
If `server.api_key` is set, clients must send `X-API-Key`.
|
||||||
|
|
||||||
|
## Start
|
||||||
|
|
||||||
|
Start OpenViking and EverOS first, then run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
export OPENVIKING_URL=http://127.0.0.1:1933
|
python -m memory_system_api.server --config config.yaml --host 127.0.0.1 --port 1934
|
||||||
export EVEROS_URL=http://127.0.0.1:1995
|
|
||||||
export EVEROS_MODE=real
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## 启动
|
## Real Test Flow
|
||||||
|
|
||||||
|
Health:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd /home/tom/memory-gateway
|
curl -s http://127.0.0.1:1934/memory-system/health
|
||||||
source .venv/bin/activate
|
|
||||||
python -m memory_gateway.server --config config.yaml
|
|
||||||
```
|
```
|
||||||
|
|
||||||
也可以显式指定 host/port：
|
Write user and assistant messages:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python -m memory_gateway.server --config config.yaml --host 127.0.0.1 --port 1934
|
curl -s -X POST http://127.0.0.1:1934/memory-system/messages \
|
||||||
```
|
-H "Content-Type: application/json" \
|
||||||
|
|
||||||
健康检查：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl http://127.0.0.1:1934/health
|
|
||||||
curl http://127.0.0.1:1934/v1/everos/health
|
|
||||||
```
|
|
||||||
|
|
||||||
如果设置了 `server.api_key`，请求需要带：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
-H "X-API-Key: <your-api-key>"
|
|
||||||
```
|
|
||||||
|
|
||||||
## v2 工作流
|
|
||||||
|
|
||||||
### 1. Ingest 一轮对话
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl -s http://127.0.0.1:1934/v2/conversations/ingest \
|
|
||||||
-H 'Content-Type: application/json' \
|
|
||||||
-d '{
|
-d '{
|
||||||
"workspace_id": "ws_1",
|
"user_id": "real_user_001",
|
||||||
"user_id": "user_a",
|
"session_id": "real_sess_001",
|
||||||
"agent_id": "agent_cli",
|
"user_message": "我喜欢喝拿铁，不喜欢美式。",
|
||||||
"session_id": "sess_1",
|
"assistant_message": "好的，我会记住你的咖啡偏好。"
|
||||||
"turn_id": "turn_1",
|
|
||||||
"namespace": "workspace/ws_1/user/user_a",
|
|
||||||
"role": "user",
|
|
||||||
"content": "Remember that the demo environment uses EverOS and OpenViking.",
|
|
||||||
"metadata": {"channel": "manual-test"}
|
|
||||||
}'
|
}'
|
||||||
```
|
```
|
||||||
|
|
||||||
结果中的 `refs` 是本地 `memory_refs` 控制面引用，通常包括：
|
Commit OpenViking and flush EverOS:
|
||||||
|
|
||||||
- OpenViking `session_archive` ref
|
|
||||||
- EverOS `message_memory` ref
|
|
||||||
|
|
||||||
这些 refs 保存的是 native id/uri、状态、hash、trace 等 metadata，不是完整记忆正文。
|
|
||||||
|
|
||||||
### 2. Retrieve 上下文
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -s http://127.0.0.1:1934/v2/context/retrieve \
|
curl -s -X POST http://127.0.0.1:1934/memory-system/sessions/real_sess_001/commit \
|
||||||
-H 'Content-Type: application/json' \
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"user_id": "real_user_001"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
Search without LLM planning:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -X POST http://127.0.0.1:1934/memory-system/search \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
-d '{
|
-d '{
|
||||||
"workspace_id": "ws_1",
|
"user_id": "real_user_001",
|
||||||
"user_id": "user_a",
|
"session_id": "real_sess_001",
|
||||||
"agent_id": "agent_cli",
|
"query": "我喜欢喝什么咖啡？",
|
||||||
"session_id": "sess_1",
|
"use_llm": false,
|
||||||
"namespace": "workspace/ws_1/user/user_a",
|
"limit": 10
|
||||||
"query": "EverOS OpenViking demo environment",
|
|
||||||
"limit": 5,
|
|
||||||
"metadata": {"trace_id": "trace_manual_1"}
|
|
||||||
}'
|
}'
|
||||||
```
|
```
|
||||||
|
|
||||||
返回结构重点：
|
Search with LLM planning:
|
||||||
|
|
||||||
- `items`：真实上下文，由 OpenViking / EverOS retrieve 返回后合并，包含 `text`、`source_backend`、`ref_id`、`score`、`memory_type`。
|
|
||||||
- `refs`：本地已有的 `memory_refs` 视图，用于追踪哪些后端引用已保存。
|
|
||||||
- `metadata.backend_results`：每个后端 retrieve 的状态、返回数量和错误信息。
|
|
||||||
|
|
||||||
### 3. Commit 一个 session
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -s http://127.0.0.1:1934/v2/conversations/sess_1/commit \
|
curl -s -X POST http://127.0.0.1:1934/memory-system/search \
|
||||||
-H 'Content-Type: application/json' \
|
-H "Content-Type: application/json" \
|
||||||
-d '{
|
-d '{
|
||||||
"workspace_id": "ws_1",
|
"user_id": "real_user_001",
|
||||||
"user_id": "user_a",
|
"session_id": "real_sess_001",
|
||||||
"agent_id": "agent_cli",
|
"query": "我的偏好是什么？",
|
||||||
"namespace": "workspace/ws_1/user/user_a"
|
"use_llm": true,
|
||||||
|
"limit": 10
|
||||||
}'
|
}'
|
||||||
```
|
```
|
||||||
|
|
||||||
该接口只创建 commit job 和 outbox events，不直接执行长期记忆生成。返回中会有 `job_id` 和 `metadata.gateway_id`。
|
Read EverOS profile:
|
||||||
|
|
||||||
### 4. Process outbox
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -s -X POST 'http://127.0.0.1:1934/v2/admin/outbox/process?limit=20'
|
curl -s http://127.0.0.1:1934/memory-system/users/real_user_001/profile
|
||||||
```
|
```
|
||||||
|
|
||||||
处理成功后会生成长期 refs：
|
## Development Checks
|
||||||
|
|
||||||
- OpenViking `session_archive` ref：session archive / summary 的 native 引用。
|
|
||||||
- EverOS `profile` ref：用户 profile 的 native 引用。
|
|
||||||
- EverOS `long_term_memory` ref：session 提炼出的长期记忆 native 引用。
|
|
||||||
|
|
||||||
这些 ref 保存在 SQLite 的 `memory_refs` 表中。
|
|
||||||
|
|
||||||
### 5. 查看 refs 和 job
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -s 'http://127.0.0.1:1934/v2/memory/refs?workspace_id=ws_1&user_id=user_a&session_id=sess_1&limit=20'
|
python -m pytest -q
|
||||||
|
python -m compileall -q memory_system_api tests
|
||||||
curl -s http://127.0.0.1:1934/v2/jobs/<job_id>
|
|
||||||
```
|
```
|
||||||
|
|
||||||
SQLite 默认路径取决于配置，例如：
|
|
||||||
|
|
||||||
```text
|
|
||||||
/home/tom/memory-gateway/memory_gateway.sqlite3
|
|
||||||
```
|
|
||||||
|
|
||||||
主要表：
|
|
||||||
|
|
||||||
- `memory_refs`
|
|
||||||
- `outbox_events`
|
|
||||||
- `commit_jobs`
|
|
||||||
- `audit_logs`
|
|
||||||
- `users` / `memories` / `episodes` / `profiles`
|
|
||||||
|
|
||||||
## v1 和 legacy API
|
|
||||||
|
|
||||||
v1 保留用户隔离、namespace、visibility/ACL、episode、session commit、audit 等基础能力：
|
|
||||||
|
|
||||||
```text
|
|
||||||
POST /v1/users
|
|
||||||
GET /v1/users/{user_id}
|
|
||||||
POST /v1/memory/search
|
|
||||||
POST /v1/memory
|
|
||||||
GET /v1/memory/{memory_id}
|
|
||||||
PATCH /v1/memory/{memory_id}
|
|
||||||
DELETE /v1/memory/{memory_id}
|
|
||||||
POST /v1/episodes
|
|
||||||
POST /v1/sessions/{session_id}/commit
|
|
||||||
GET /v1/users/{user_id}/profile
|
|
||||||
POST /v1/memory/{memory_id}/feedback
|
|
||||||
GET /v1/namespaces
|
|
||||||
GET /v1/audit
|
|
||||||
GET /v1/everos/health
|
|
||||||
```
|
|
||||||
|
|
||||||
旧 `/api/*` 接口仍保留：
|
|
||||||
|
|
||||||
```text
|
|
||||||
POST /api/search
|
|
||||||
POST /api/memory
|
|
||||||
POST /api/resource
|
|
||||||
POST /api/summary
|
|
||||||
POST /api/knowledge/upload
|
|
||||||
```
|
|
||||||
|
|
||||||
## MCP / Hermes
|
|
||||||
|
|
||||||
MCP endpoints：
|
|
||||||
|
|
||||||
```text
|
|
||||||
POST /mcp/rpc
|
|
||||||
GET /mcp/sse
|
|
||||||
```
|
|
||||||
|
|
||||||
Hermes skill 位于：
|
|
||||||
|
|
||||||
```text
|
|
||||||
integrations/hermes/memory-gateway/
|
|
||||||
```
|
|
||||||
|
|
||||||
常用脚本示例：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python integrations/hermes/memory-gateway/scripts/everos_health.py
|
|
||||||
python integrations/hermes/memory-gateway/scripts/memory_commit_session.py --help
|
|
||||||
```
|
|
||||||
|
|
||||||
## 开发与验证
|
|
||||||
|
|
||||||
运行测试：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd /home/tom/memory-gateway
|
|
||||||
PYTHONPATH=/home/tom/memory-gateway pytest -q
|
|
||||||
```
|
|
||||||
|
|
||||||
编译检查：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m compileall -q memory_gateway tests integrations/hermes/memory-gateway plugins/memory-gateway-agent
|
|
||||||
```
|
|
||||||
|
|
||||||
Ruff 已在 `pyproject.toml` 中配置。如果本地环境安装了 ruff：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m ruff check .
|
|
||||||
```
|
|
||||||
|
|
||||||
当前仓库不要求真实 OpenViking / EverOS 服务才能跑单元测试；真实服务流程需要先启动 `127.0.0.1:1933` 和 `127.0.0.1:1995`。
|
|
||||||
|
|
||||||
## 设计约束
|
|
||||||
|
|
||||||
- SQLite 保存控制面 metadata，不作为长期记忆正文数据库。
|
|
||||||
- `refs` 是后端 native 对象引用，不等于上下文正文。
|
|
||||||
- `retrieve.items` 才是运行时上下文内容。
|
|
||||||
- `commit` 只创建 job/outbox；长期 refs 由 outbox process 生成。
|
|
||||||
- 默认只绑定本机地址；远程暴露时必须设置 API key、TLS 和网络访问控制。
|
|
||||||
|
|||||||
@ -1,68 +1,23 @@
|
|||||||
# Memory Gateway 配置示例。
|
|
||||||
# 复制为 config.yaml 并根据实际服务器路径、端口和密钥修改。
|
|
||||||
# 不要提交 config.yaml；它应包含本机/服务器密钥。
|
|
||||||
|
|
||||||
# Memory Gateway 服务配置
|
|
||||||
server:
|
server:
|
||||||
# 本机测试可用 127.0.0.1；需要远程调用时使用 0.0.0.0 并配置防火墙/反向代理。
|
|
||||||
host: "127.0.0.1"
|
host: "127.0.0.1"
|
||||||
# REST API、MCP RPC 和 SSE 共用端口。
|
|
||||||
port: 1934
|
port: 1934
|
||||||
# 强烈建议生产/远程调用时设置；客户端通过 X-API-Key 传入。
|
|
||||||
api_key: ""
|
api_key: ""
|
||||||
|
|
||||||
# OpenViking 后端配置
|
|
||||||
openviking:
|
openviking:
|
||||||
# OpenViking 服务器地址。Memory Gateway 通过它检索 context/resource/memory。
|
|
||||||
url: "http://127.0.0.1:1933"
|
url: "http://127.0.0.1:1933"
|
||||||
# OpenViking API Key。按 OpenViking 实际配置填写。
|
api_key: "your-secret-root-key"
|
||||||
api_key: ""
|
|
||||||
timeout: 30
|
timeout: 30
|
||||||
|
verify_ssl: true
|
||||||
|
|
||||||
# EverOS / EverCore 后台长期记忆整理服务
|
|
||||||
everos:
|
everos:
|
||||||
enabled: true
|
|
||||||
mode: "real"
|
|
||||||
# 指向 /home/tom/EverOS/methods/EverCore 启动的 API。
|
|
||||||
url: "http://127.0.0.1:1995"
|
url: "http://127.0.0.1:1995"
|
||||||
api_key: ""
|
api_key: ""
|
||||||
timeout: 30
|
timeout: 30
|
||||||
|
verify_ssl: true
|
||||||
health_path: "/health"
|
health_path: "/health"
|
||||||
ingest_path: "/api/v1/memories"
|
|
||||||
search_path: "/api/v1/memories/search"
|
|
||||||
flush_path: "/api/v1/memories/flush"
|
|
||||||
retrieve_method: "keyword"
|
|
||||||
|
|
||||||
# 记忆配置
|
storage:
|
||||||
memory:
|
sqlite_path: "/home/tom/memory-gateway/memory_system_api.sqlite3"
|
||||||
# 旧 /api/* 接口使用的默认命名空间。v1 API 会按 user/agent/workspace/session 自动展开 namespace。
|
|
||||||
default_namespace: "memory-gateway"
|
|
||||||
search_limit: 10
|
|
||||||
|
|
||||||
# 日志配置
|
|
||||||
logging:
|
logging:
|
||||||
level: "INFO"
|
level: "INFO"
|
||||||
format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
|
||||||
|
|
||||||
# LLM 配置：用于 /api/summary 和 /api/knowledge/upload
|
|
||||||
# 兼容 OpenAI Chat Completions API；也可指向本地 vLLM / Ollama OpenAI-compatible endpoint。
|
|
||||||
llm:
|
|
||||||
base_url: "https://api.openai.com/v1"
|
|
||||||
api_key: ""
|
|
||||||
model: ""
|
|
||||||
timeout: 60
|
|
||||||
max_input_chars: 24000
|
|
||||||
|
|
||||||
# Obsidian Vault 配置。
|
|
||||||
# 服务端不要求安装 Obsidian 桌面应用；这里本质上是一个 Markdown vault 目录。
|
|
||||||
obsidian:
|
|
||||||
vault_path: "/opt/memory-gateway/obsidian-vault"
|
|
||||||
knowledge_dir: "01_Knowledge/Uploaded"
|
|
||||||
review_dir: "Reviews/Queue"
|
|
||||||
|
|
||||||
# v1 metadata storage。
|
|
||||||
# SQLite 保存 users、memories、episodes、profiles、audit，是用户隔离和 ACL 判断的主要 metadata store。
|
|
||||||
# Use "memory" only for isolated unit tests.
|
|
||||||
storage:
|
|
||||||
backend: "sqlite"
|
|
||||||
sqlite_path: "/opt/memory-gateway/data/memory_gateway.sqlite3"
|
|
||||||
|
|||||||
@ -1,767 +0,0 @@
|
|||||||
# 通用 Memory Gateway 方案与 POC 骨架
|
|
||||||
|
|
||||||
本文基于当前仓库的轻量 FastAPI + MCP + OpenViking + Obsidian 能力扩展，不把系统设计成重平台。第一阶段目标是先跑通多用户隔离、namespace routing、记忆检索、写入、session commit 和人工 review 草稿，后续再替换持久化、向量索引和 EverOS worker。
|
|
||||||
|
|
||||||
## A. 总体架构图
|
|
||||||
|
|
||||||
```mermaid
|
|
||||||
flowchart TB
|
|
||||||
subgraph Agents["Agent Frameworks"]
|
|
||||||
Nanobot[Nanobot]
|
|
||||||
Hermes[Hermes Agent]
|
|
||||||
OpenClaw[OpenClaw]
|
|
||||||
Other[Other Agents]
|
|
||||||
end
|
|
||||||
|
|
||||||
subgraph Gateway["Memory Gateway"]
|
|
||||||
HTTP[HTTP API /v1]
|
|
||||||
MCP[MCP tools]
|
|
||||||
Auth[Auth / API Key / Future Login]
|
|
||||||
ACL[ACL & Visibility Policy]
|
|
||||||
Router[Namespace Router]
|
|
||||||
Audit[Audit Log]
|
|
||||||
Retrieval[Retrieval Orchestrator]
|
|
||||||
Writeback[Writeback Orchestrator]
|
|
||||||
end
|
|
||||||
|
|
||||||
subgraph Skills["Skills Layer"]
|
|
||||||
Ingest[ingest]
|
|
||||||
Extract[extract]
|
|
||||||
Classify[classify]
|
|
||||||
Retrieve[retrieve]
|
|
||||||
Commit[commit]
|
|
||||||
Merge[merge]
|
|
||||||
Prune[prune]
|
|
||||||
Summarize[summarize]
|
|
||||||
end
|
|
||||||
|
|
||||||
subgraph OpenViking["OpenViking"]
|
|
||||||
OVFS[context filesystem]
|
|
||||||
OVMem[memory]
|
|
||||||
OVRes[resources]
|
|
||||||
OVSkills[skills]
|
|
||||||
OVWorkspace[workspace]
|
|
||||||
end
|
|
||||||
|
|
||||||
subgraph EverOS["EverOS"]
|
|
||||||
LTE[long-term extraction]
|
|
||||||
Consolidation[consolidation]
|
|
||||||
Decay[decay]
|
|
||||||
Dedup[dedup]
|
|
||||||
Profile[profile evolution]
|
|
||||||
end
|
|
||||||
|
|
||||||
subgraph Obsidian["Obsidian"]
|
|
||||||
Vault[human editable memory vault]
|
|
||||||
Reviews[review queue]
|
|
||||||
Profiles[profiles]
|
|
||||||
LongTerm[long-term notes]
|
|
||||||
end
|
|
||||||
|
|
||||||
subgraph Storage["Storage"]
|
|
||||||
DB[(metadata DB)]
|
|
||||||
Vector[(vector index)]
|
|
||||||
Files[(object / file storage)]
|
|
||||||
end
|
|
||||||
|
|
||||||
Nanobot --> HTTP
|
|
||||||
Hermes --> MCP
|
|
||||||
OpenClaw --> HTTP
|
|
||||||
Other --> HTTP
|
|
||||||
Other --> MCP
|
|
||||||
|
|
||||||
HTTP --> Auth --> ACL --> Router
|
|
||||||
MCP --> Auth
|
|
||||||
Router --> Retrieval
|
|
||||||
Router --> Writeback
|
|
||||||
ACL --> Audit
|
|
||||||
|
|
||||||
Retrieval --> Skills
|
|
||||||
Writeback --> Skills
|
|
||||||
Skills --> OpenViking
|
|
||||||
Skills --> EverOS
|
|
||||||
Skills --> Obsidian
|
|
||||||
|
|
||||||
Gateway --> DB
|
|
||||||
Gateway --> Vector
|
|
||||||
Gateway --> Files
|
|
||||||
OpenViking --> DB
|
|
||||||
OpenViking --> Vector
|
|
||||||
Obsidian --> Files
|
|
||||||
EverOS --> DB
|
|
||||||
EverOS --> Vector
|
|
||||||
```
|
|
||||||
|
|
||||||
## B. 核心数据模型
|
|
||||||
|
|
||||||
代码骨架见 `memory_gateway/schemas.py`。核心模型如下。
|
|
||||||
|
|
||||||
### User
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"id": "user_tom",
|
|
||||||
"display_name": "Tom",
|
|
||||||
"status": "active",
|
|
||||||
"profile_namespace": "user/user_tom/profile",
|
|
||||||
"preferences": {"language": "zh-CN"},
|
|
||||||
"created_at": "2026-04-30T10:00:00Z",
|
|
||||||
"updated_at": "2026-04-30T10:00:00Z"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Agent
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"id": "agent_hermes_default",
|
|
||||||
"name": "Hermes Default Agent",
|
|
||||||
"framework": "hermes",
|
|
||||||
"owner_user_id": "user_tom",
|
|
||||||
"created_at": "2026-04-30T10:00:00Z"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Workspace
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"id": "ws_memory_gateway",
|
|
||||||
"name": "Memory Gateway POC",
|
|
||||||
"owner_user_id": "user_tom",
|
|
||||||
"member_user_ids": ["user_tom"],
|
|
||||||
"allowed_agent_ids": ["agent_hermes_default"]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Session
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"id": "sess_20260430_001",
|
|
||||||
"user_id": "user_tom",
|
|
||||||
"agent_id": "agent_hermes_default",
|
|
||||||
"workspace_id": "ws_memory_gateway",
|
|
||||||
"status": "open",
|
|
||||||
"expires_at": "2026-05-07T10:00:00Z"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### MemoryRecord
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"id": "mem_abc123",
|
|
||||||
"user_id": "user_tom",
|
|
||||||
"agent_id": "agent_hermes_default",
|
|
||||||
"workspace_id": "ws_memory_gateway",
|
|
||||||
"session_id": "sess_20260430_001",
|
|
||||||
"namespace": "user/user_tom/long_term",
|
|
||||||
"memory_type": "preference",
|
|
||||||
"content": "用户偏好中文输出，结构化但不要过度平台化。",
|
|
||||||
"summary": "中文、结构化、轻量 POC 优先。",
|
|
||||||
"tags": ["preference", "style"],
|
|
||||||
"importance": 0.8,
|
|
||||||
"confidence": 0.9,
|
|
||||||
"visibility": "private",
|
|
||||||
"source": "conversation",
|
|
||||||
"created_at": "2026-04-30T10:00:00Z",
|
|
||||||
"updated_at": "2026-04-30T10:00:00Z",
|
|
||||||
"expires_at": null,
|
|
||||||
"version": 1
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### EpisodeRecord
|
|
||||||
|
|
||||||
短期过程记录，默认不进入 Obsidian，不自动成为长期记忆。
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"id": "epi_abc123",
|
|
||||||
"user_id": "user_tom",
|
|
||||||
"agent_id": "agent_hermes_default",
|
|
||||||
"workspace_id": "ws_memory_gateway",
|
|
||||||
"session_id": "sess_20260430_001",
|
|
||||||
"namespace": "session/sess_20260430_001/episodic",
|
|
||||||
"content": "本轮讨论了 Memory Gateway POC 范围。",
|
|
||||||
"summary": "确认 POC 优先做隔离、检索、写入和整理。",
|
|
||||||
"events": [],
|
|
||||||
"tags": ["design"]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### ProfileRecord
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"id": "profile_user_tom",
|
|
||||||
"user_id": "user_tom",
|
|
||||||
"namespace": "user/user_tom/profile",
|
|
||||||
"display_name": "Tom",
|
|
||||||
"stable_facts": ["正在设计通用 Memory Gateway"],
|
|
||||||
"preferences": {"language": "Chinese"},
|
|
||||||
"working_style": ["偏好可落地 POC"],
|
|
||||||
"updated_from_memory_ids": ["mem_abc123"],
|
|
||||||
"version": 3
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### ACL / Visibility
|
|
||||||
|
|
||||||
`visibility` 四档：
|
|
||||||
|
|
||||||
- `private`：仅 `user_id` 相同可读写。
|
|
||||||
- `agent-only`：同一 `user_id` 且同一 `agent_id` 可读写。
|
|
||||||
- `workspace-shared`：在同一 `workspace_id` 且通过 workspace membership 授权后可读。
|
|
||||||
- `global`：可公开检索，只能由受信任 actor 写入。
|
|
||||||
|
|
||||||
### AuditLog
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"id": "audit_abc123",
|
|
||||||
"actor_user_id": "user_tom",
|
|
||||||
"actor_agent_id": "agent_hermes_default",
|
|
||||||
"action": "memory_search",
|
|
||||||
"target_type": "memory",
|
|
||||||
"target_id": "mem_abc123",
|
|
||||||
"namespace": "user/user_tom/long_term",
|
|
||||||
"decision": "allow",
|
|
||||||
"reason": "private owner",
|
|
||||||
"created_at": "2026-04-30T10:00:00Z"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## C. Namespace 与隔离设计
|
|
||||||
|
|
||||||
推荐 namespace：
|
|
||||||
|
|
||||||
```text
|
|
||||||
user/{user_id}/profile
|
|
||||||
user/{user_id}/preferences
|
|
||||||
user/{user_id}/long_term
|
|
||||||
agent/{agent_id}/memory
|
|
||||||
workspace/{workspace_id}/shared
|
|
||||||
session/{session_id}/episodic
|
|
||||||
global/public
|
|
||||||
```
|
|
||||||
|
|
||||||
隔离规则：
|
|
||||||
|
|
||||||
- 用户隔离：所有 `user/{user_id}/...` 默认只允许同一 `user_id` 访问。Gateway 先校验 actor，再把 namespace 映射到 OpenViking URI。
|
|
||||||
- Agent 隔离：`agent/{agent_id}/memory` 用于某个 agent 的工具经验、失败教训、prompt working notes。默认 `agent-only`。
|
|
||||||
- Workspace 共享：`workspace/{workspace_id}/shared` 必须检查用户是否属于 workspace，agent 是否在 `allowed_agent_ids` 内。
|
|
||||||
- Session 过期：`session/{session_id}/episodic` 必须有 TTL。过期后不可检索；只保留必要 audit。
|
|
||||||
- 可跨 agent 共享：用户显式确认的 profile、preferences、user long_term、workspace shared、global public。
|
|
||||||
- 不可跨 agent 共享：agent-only memory、未 commit 的 session episodic、低置信度候选记忆、含敏感凭据或临时日志的内容。
|
|
||||||
|
|
||||||
OpenViking URI 映射：
|
|
||||||
|
|
||||||
```text
|
|
||||||
viking://memory/user/{user_id}/long_term/{memory_id}.json
|
|
||||||
viking://resources/workspace/{workspace_id}/shared/{slug}.md
|
|
||||||
viking://skills/memory-gateway/{skill_name}
|
|
||||||
```
|
|
||||||
|
|
||||||
## D. API 设计
|
|
||||||
|
|
||||||
第一阶段代码已挂载 `/v1` router，见 `memory_gateway/api_v1.py`。
|
|
||||||
|
|
||||||
### POST /v1/users
|
|
||||||
|
|
||||||
Request:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"user_id": "user_tom", "display_name": "Tom", "preferences": {"language": "zh-CN"}}
|
|
||||||
```
|
|
||||||
|
|
||||||
Response:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"id": "user_tom", "display_name": "Tom", "profile_namespace": "user/user_tom/profile", "status": "active"}
|
|
||||||
```
|
|
||||||
|
|
||||||
### GET /v1/users/{user_id}
|
|
||||||
|
|
||||||
Response:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"id": "user_tom", "display_name": "Tom", "status": "active"}
|
|
||||||
```
|
|
||||||
|
|
||||||
### POST /v1/memory/search
|
|
||||||
|
|
||||||
Request:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"user_id": "user_tom",
|
|
||||||
"agent_id": "agent_hermes_default",
|
|
||||||
"workspace_id": "ws_memory_gateway",
|
|
||||||
"query": "中文输出偏好",
|
|
||||||
"namespaces": ["user/user_tom/long_term"],
|
|
||||||
"limit": 5
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Response:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"results": [
|
|
||||||
{
|
|
||||||
"memory": {
|
|
||||||
"id": "mem_abc123",
|
|
||||||
"namespace": "user/user_tom/long_term",
|
|
||||||
"summary": "中文、结构化、轻量 POC 优先。"
|
|
||||||
},
|
|
||||||
"score": 2.7
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"total": 1
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### POST /v1/memory
|
|
||||||
|
|
||||||
Request:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"user_id": "user_tom",
|
|
||||||
"agent_id": "agent_hermes_default",
|
|
||||||
"workspace_id": "ws_memory_gateway",
|
|
||||||
"memory_type": "preference",
|
|
||||||
"content": "用户偏好中文输出。",
|
|
||||||
"summary": "中文输出偏好",
|
|
||||||
"tags": ["preference"],
|
|
||||||
"importance": 0.8,
|
|
||||||
"confidence": 0.9,
|
|
||||||
"visibility": "private",
|
|
||||||
"source": "manual"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Response:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"id": "mem_abc123", "namespace": "user/user_tom/long_term", "version": 1}
|
|
||||||
```
|
|
||||||
|
|
||||||
### GET /v1/memory/{memory_id}
|
|
||||||
|
|
||||||
Request query:
|
|
||||||
|
|
||||||
```text
|
|
||||||
?user_id=user_tom&agent_id=agent_hermes_default&workspace_id=ws_memory_gateway
|
|
||||||
```
|
|
||||||
|
|
||||||
Response:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"id": "mem_abc123", "content": "用户偏好中文输出。", "visibility": "private"}
|
|
||||||
```
|
|
||||||
|
|
||||||
### PATCH /v1/memory/{memory_id}
|
|
||||||
|
|
||||||
Request:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"summary": "用户偏好中文、结构化、少废话。", "importance": 0.9}
|
|
||||||
```
|
|
||||||
|
|
||||||
Response:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"id": "mem_abc123", "version": 2, "importance": 0.9}
|
|
||||||
```
|
|
||||||
|
|
||||||
### DELETE /v1/memory/{memory_id}
|
|
||||||
|
|
||||||
Response:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"deleted": true, "id": "mem_abc123"}
|
|
||||||
```
|
|
||||||
|
|
||||||
### POST /v1/episodes
|
|
||||||
|
|
||||||
Request:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"user_id": "user_tom",
|
|
||||||
"agent_id": "agent_hermes_default",
|
|
||||||
"workspace_id": "ws_memory_gateway",
|
|
||||||
"session_id": "sess_001",
|
|
||||||
"content": "本轮完成了 namespace 和 ACL 设计。",
|
|
||||||
"tags": ["design"]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Response:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"id": "epi_abc123", "namespace": "session/sess_001/episodic"}
|
|
||||||
```
|
|
||||||
|
|
||||||
### POST /v1/sessions/{session_id}/commit
|
|
||||||
|
|
||||||
Request:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"user_id": "user_tom",
|
|
||||||
"agent_id": "agent_hermes_default",
|
|
||||||
"workspace_id": "ws_memory_gateway",
|
|
||||||
"promote": true,
|
|
||||||
"min_importance": 0.6,
|
|
||||||
"target_namespace": "user/user_tom/long_term"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Response:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"session_id": "sess_001", "episodes": 3, "promoted": [{"id": "mem_def456"}]}
|
|
||||||
```
|
|
||||||
|
|
||||||
### GET /v1/users/{user_id}/profile
|
|
||||||
|
|
||||||
Response:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"user_id": "user_tom", "namespace": "user/user_tom/profile", "preferences": {"language": "zh-CN"}}
|
|
||||||
```
|
|
||||||
|
|
||||||
### POST /v1/memory/{memory_id}/feedback
|
|
||||||
|
|
||||||
Request:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"user_id": "user_tom", "feedback": "incorrect", "comment": "这是一次临时偏好，不应长期保留。"}
|
|
||||||
```
|
|
||||||
|
|
||||||
Response:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"status": "ok", "memory_id": "mem_abc123", "feedback": "incorrect"}
|
|
||||||
```
|
|
||||||
|
|
||||||
### GET /v1/namespaces
|
|
||||||
|
|
||||||
Request query:
|
|
||||||
|
|
||||||
```text
|
|
||||||
?user_id=user_tom&agent_id=agent_hermes_default&workspace_id=ws_memory_gateway&session_id=sess_001
|
|
||||||
```
|
|
||||||
|
|
||||||
Response:
|
|
||||||
|
|
||||||
```json
|
|
||||||
[
|
|
||||||
{"namespace": "user/user_tom/profile", "visibility": "private"},
|
|
||||||
{"namespace": "agent/agent_hermes_default/memory", "visibility": "agent-only"},
|
|
||||||
{"namespace": "workspace/ws_memory_gateway/shared", "visibility": "workspace-shared"}
|
|
||||||
]
|
|
||||||
```
|
|
||||||
|
|
||||||
### GET /v1/audit
|
|
||||||
|
|
||||||
Response:
|
|
||||||
|
|
||||||
```json
|
|
||||||
[{"action": "upsert_memory", "target_type": "memory", "decision": "allow"}]
|
|
||||||
```
|
|
||||||
|
|
||||||
### MCP tools
|
|
||||||
|
|
||||||
目标 v1 tools 见 `memory_gateway/mcp_tools_v1.py`：
|
|
||||||
|
|
||||||
- `memory_search`
|
|
||||||
- `memory_upsert`
|
|
||||||
- `memory_append_episode`
|
|
||||||
- `memory_commit_session`
|
|
||||||
- `memory_get_profile`
|
|
||||||
- `memory_list_namespaces`
|
|
||||||
- `memory_delete`
|
|
||||||
- `memory_feedback`
|
|
||||||
|
|
||||||
示例 MCP call:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"name": "memory_search",
|
|
||||||
"arguments": {
|
|
||||||
"user_id": "user_tom",
|
|
||||||
"agent_id": "agent_hermes_default",
|
|
||||||
"workspace_id": "ws_memory_gateway",
|
|
||||||
"query": "项目 POC 决策",
|
|
||||||
"limit": 5
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## E. Skills 设计
|
|
||||||
|
|
||||||
代码骨架位于 `memory_gateway/skills/`。
|
|
||||||
|
|
||||||
| Skill | 功能 | 输入 | 输出 | 触发时机 | 组件 | 写长期记忆 |
|
|
||||||
|---|---|---|---|---|---|---|
|
|
||||||
| `ingest_skill` | 标准化对话、文件、任务事件 | raw text/file/events | normalized payload | agent 写入 episode 前 | Gateway, file storage | 否 |
|
|
||||||
| `extract_memory_skill` | 从 episode/session 抽取候选记忆 | episode/session content | memory candidates | session commit / worker 定时 | LLM, EverOS | 否 |
|
|
||||||
| `classify_memory_skill` | 判断 memory_type、visibility、namespace | candidate memory | classification | 写入前 | ACL, namespace router | 否 |
|
|
||||||
| `retrieve_context_skill` | 聚合用户、agent、workspace 上下文 | query + context ids | ranked contexts | agent 调用前 | OpenViking, vector index | 否 |
|
|
||||||
| `commit_memory_skill` | 写入长期记忆 | MemoryRecord | stored record | 人工确认或 commit 通过 | DB, OpenViking | 是 |
|
|
||||||
| `summarize_episode_skill` | 压缩 episode | episode content | summary | session commit | LLM | 否 |
|
|
||||||
| `merge_memory_skill` | 合并重复或相近记忆 | memory ids | merged memory | EverOS 整理 | DB, vector index | 是 |
|
|
||||||
| `prune_memory_skill` | 衰减、归档、删除低质记忆 | policy + memory ids | archived/deleted list | 定时 worker | EverOS | 是 |
|
|
||||||
| `export_to_obsidian_skill` | 生成 Obsidian review draft | high-value memory | markdown draft | 高价值或需人工确认 | Obsidian | 否 |
|
|
||||||
| `import_from_obsidian_skill` | 从人工维护笔记导入记忆 | markdown path | MemoryRecord | vault sync | Obsidian, OpenViking | 是 |
|
|
||||||
|
|
||||||
## F. Obsidian Vault 设计
|
|
||||||
|
|
||||||
推荐目录：
|
|
||||||
|
|
||||||
```text
|
|
||||||
obsidian-vault/
|
|
||||||
├── Users/
|
|
||||||
│ └── {user_id}/
|
|
||||||
│ ├── Profile.md
|
|
||||||
│ ├── Preferences.md
|
|
||||||
│ └── LongTerm/
|
|
||||||
├── Agents/
|
|
||||||
│ └── {agent_id}/Experience.md
|
|
||||||
├── Workspaces/
|
|
||||||
│ └── {workspace_id}/Shared.md
|
|
||||||
├── Memories/
|
|
||||||
│ ├── LongTerm/
|
|
||||||
│ └── Archived/
|
|
||||||
├── Profiles/
|
|
||||||
├── Reviews/
|
|
||||||
│ ├── Queue/
|
|
||||||
│ ├── Accepted/
|
|
||||||
│ └── Rejected/
|
|
||||||
├── Exports/
|
|
||||||
└── Templates/
|
|
||||||
```
|
|
||||||
|
|
||||||
进入 Obsidian 的内容：
|
|
||||||
|
|
||||||
- 人工可维护 profile、preferences、长期总结。
|
|
||||||
- 高价值 workspace 知识、项目决策、复用经验。
|
|
||||||
- EverOS 标记为 `needs_review` 的长期记忆草稿。
|
|
||||||
|
|
||||||
不进入 Obsidian 的内容：
|
|
||||||
|
|
||||||
- 全量原始对话。
|
|
||||||
- 高频工具日志、临时 session trace。
|
|
||||||
- 低置信度候选记忆。
|
|
||||||
- 敏感凭据、token、临时错误栈。
|
|
||||||
|
|
||||||
标签体系：
|
|
||||||
|
|
||||||
```text
|
|
||||||
#memory/profile
|
|
||||||
#memory/preference
|
|
||||||
#memory/long-term
|
|
||||||
#memory/workspace
|
|
||||||
#memory/agent-experience
|
|
||||||
#memory/review
|
|
||||||
#memory/conflict
|
|
||||||
#memory/deprecated
|
|
||||||
#source/everos
|
|
||||||
#source/manual
|
|
||||||
#visibility/private
|
|
||||||
#visibility/workspace-shared
|
|
||||||
```
|
|
||||||
|
|
||||||
模板文件已加入 `obsidian-vault/05_Templates/`。
|
|
||||||
|
|
||||||
## G. OpenViking 设计
|
|
||||||
|
|
||||||
OpenViking 作为统一 context 层，Gateway 不要求 agent 直接理解 OpenViking 内部结构。
|
|
||||||
|
|
||||||
组织方式：
|
|
||||||
|
|
||||||
```text
|
|
||||||
viking://memory/user/{user_id}/profile
|
|
||||||
viking://memory/user/{user_id}/preferences
|
|
||||||
viking://memory/user/{user_id}/long_term
|
|
||||||
viking://memory/agent/{agent_id}/memory
|
|
||||||
viking://memory/workspace/{workspace_id}/shared
|
|
||||||
viking://resources/user/{user_id}/obsidian/{note_id}.md
|
|
||||||
viking://skills/memory-gateway/{skill_name}
|
|
||||||
```
|
|
||||||
|
|
||||||
检索路径：
|
|
||||||
|
|
||||||
1. Agent 调用 Gateway `/v1/memory/search` 或 MCP `memory_search`。
|
|
||||||
2. Gateway 执行 Auth、ACL、namespace expansion。
|
|
||||||
3. Gateway 查询 metadata DB 和 vector index，必要时调用 OpenViking search。
|
|
||||||
4. 返回统一 `MemoryRecord` 或 context chunk，不暴露底层差异。
|
|
||||||
|
|
||||||
同步：
|
|
||||||
|
|
||||||
- Obsidian accepted note 通过 `import_from_obsidian_skill` 写回 Gateway，再同步 OpenViking resource。
|
|
||||||
- EverOS consolidation 后写入 `user/{user_id}/long_term` 或 `workspace/{workspace_id}/shared`。
|
|
||||||
- Gateway 保存 `source_ref`，避免 OpenViking 与 Obsidian 互相重复导入。
|
|
||||||
|
|
||||||
## H. EverOS 设计
|
|
||||||
|
|
||||||
输入来源：
|
|
||||||
|
|
||||||
- `EpisodeRecord`：对话片段、任务执行摘要、agent 过程事件。
|
|
||||||
- `SessionRecord`：session commit 包。
|
|
||||||
- `MemoryFeedback`：incorrect、duplicate、outdated 等反馈。
|
|
||||||
- Obsidian review 结果：accepted/rejected/edited。
|
|
||||||
|
|
||||||
整理流程：
|
|
||||||
|
|
||||||
1. 抽取：从 episode 中提炼候选事实、偏好、决策、经验。
|
|
||||||
2. 打分：根据重要性、稳定性、重复出现次数、来源可信度打分。
|
|
||||||
3. 去重：按 semantic hash + embedding 相似度查找近似 MemoryRecord。
|
|
||||||
4. 合并：相同事实合并 evidence；更高置信度覆盖低置信度。
|
|
||||||
5. 冲突检测：同一 subject 的相反陈述标记 `needs_review`，不自动覆盖。
|
|
||||||
6. 衰减：长时间未命中且低反馈的记忆降低 importance。
|
|
||||||
7. 归档：过期、错误、低置信度、被人工拒绝的记忆转 archived。
|
|
||||||
8. profile evolution：只有稳定、重复、高置信偏好进入 ProfileRecord。
|
|
||||||
|
|
||||||
污染控制：
|
|
||||||
|
|
||||||
- session 临时内容不直接提升为长期记忆。
|
|
||||||
- LLM 抽取结果默认是 candidate，需阈值或人工确认。
|
|
||||||
- 每条长期记忆保留 source、confidence、version、feedback。
|
|
||||||
- 对 profile 更新采用 evidence count，禁止一次对话永久改写强偏好。
|
|
||||||
|
|
||||||
## I. 工程目录结构
|
|
||||||
|
|
||||||
当前仓库保留 `memory_gateway/` 包名，目标结构如下：
|
|
||||||
|
|
||||||
```text
|
|
||||||
memory-gateway/
|
|
||||||
├── memory_gateway/
|
|
||||||
│ ├── api_v1.py # v1 HTTP API
|
|
||||||
│ ├── mcp_tools_v1.py # v1 MCP tool contract
|
|
||||||
│ ├── schemas.py # User/Memory/Episode/Profile/ACL/Audit
|
|
||||||
│ ├── namespace.py # namespace builder + ACL helpers
|
|
||||||
│ ├── services.py # orchestration service
|
|
||||||
│ ├── repositories.py # POC in-memory repo; later DB repo
|
|
||||||
│ ├── security/ # future auth, RBAC, audit policy
|
|
||||||
│ ├── skills/
|
|
||||||
│ │ ├── ingest_skill.py
|
|
||||||
│ │ ├── extract_memory_skill.py
|
|
||||||
│ │ ├── classify_memory_skill.py
|
|
||||||
│ │ ├── retrieve_context_skill.py
|
|
||||||
│ │ ├── commit_memory_skill.py
|
|
||||||
│ │ ├── summarize_episode_skill.py
|
|
||||||
│ │ ├── merge_memory_skill.py
|
|
||||||
│ │ ├── prune_memory_skill.py
|
|
||||||
│ │ ├── export_to_obsidian_skill.py
|
|
||||||
│ │ └── import_from_obsidian_skill.py
|
|
||||||
│ ├── adapters/
|
|
||||||
│ │ ├── openviking.py
|
|
||||||
│ │ ├── everos.py
|
|
||||||
│ │ └── obsidian.py
|
|
||||||
│ └── workers/
|
|
||||||
│ └── everos_worker.py
|
|
||||||
├── obsidian-vault/
|
|
||||||
├── integrations/
|
|
||||||
│ ├── nanobot/
|
|
||||||
│ ├── hermes/
|
|
||||||
│ └── openclaw/
|
|
||||||
└── tests/
|
|
||||||
```
|
|
||||||
|
|
||||||
如果未来迁移到更标准的 `app/`，可把 `memory_gateway/api_v1.py` 对应到 `app/api`，`schemas.py` 对应到 `app/schemas`，`services.py` 对应到 `app/services`。
|
|
||||||
|
|
||||||
## J. 2 到 4 周 POC 实施计划
|
|
||||||
|
|
||||||
第一周：
|
|
||||||
|
|
||||||
- 完成 `/v1/users`、`/v1/memory`、`/v1/memory/search`、`/v1/episodes`。
|
|
||||||
- 实现 namespace router、visibility、基础 audit。
|
|
||||||
- 存储先用 SQLite 或当前内存 repo，搜索先用 lexical，OpenViking 作为可选后端。
|
|
||||||
|
|
||||||
第二周：
|
|
||||||
|
|
||||||
- 接入 OpenViking URI 写入和检索。
|
|
||||||
- 实现 `retrieve_context_skill`、`commit_memory_skill`、`summarize_episode_skill`。
|
|
||||||
- 给 Hermes/Nanobot/OpenClaw 提供最小 client 示例。
|
|
||||||
|
|
||||||
第三周：
|
|
||||||
|
|
||||||
- 加 EverOS worker 原型：session commit、candidate extraction、dedup、merge。
|
|
||||||
- 增加 feedback 流程：incorrect、duplicate、outdated 影响 prune/merge。
|
|
||||||
- 生成 Obsidian review draft，而不是直接写入最终知识库。
|
|
||||||
|
|
||||||
第四周：
|
|
||||||
|
|
||||||
- Obsidian import/export 双向同步。
|
|
||||||
- 增加 profile evolution 的阈值和 evidence 机制。
|
|
||||||
- 补充权限测试、污染测试、重复记忆测试、跨 agent 检索测试。
|
|
||||||
|
|
||||||
先做：
|
|
||||||
|
|
||||||
- 用户隔离、namespace、memory CRUD、episode append、session commit、basic search、audit。
|
|
||||||
|
|
||||||
暂不做：
|
|
||||||
|
|
||||||
- 完整登录系统、复杂 RBAC、多租户计费、实时同步、复杂 UI、全量向量数据库治理。
|
|
||||||
|
|
||||||
POC 成功指标：
|
|
||||||
|
|
||||||
- 不同 `user_id` 之间无法互相读写 private memory。
|
|
||||||
- 同一 workspace 的共享记忆可被授权 agent 检索。
|
|
||||||
- session 记忆不会自动污染长期记忆。
|
|
||||||
- 10 条重复候选能合并到 1 到 2 条长期记忆。
|
|
||||||
- 错误反馈后，该记忆不再进入默认 retrieval。
|
|
||||||
- Hermes/Nanobot/OpenClaw 至少两个框架能通过统一 API 调用。
|
|
||||||
|
|
||||||
## K. 推荐默认方案
|
|
||||||
|
|
||||||
第一阶段最合理默认方案：
|
|
||||||
|
|
||||||
- FastAPI 提供 `/v1` 统一 HTTP API。
|
|
||||||
- MCP 先保留现有 `/mcp/rpc`，新增 `memory_gateway/mcp_tools_v1.py` 作为目标 contract。
|
|
||||||
- 存储使用 SQLite metadata + 本地文件存 object；当前代码先用 in-memory repo 验证接口。
|
|
||||||
- 搜索先用 OpenViking search + 简单 lexical fallback；向量索引第二阶段引入。
|
|
||||||
- Obsidian 只保存人工可读的高价值长期记忆和 review draft。
|
|
||||||
- EverOS 第一阶段不做独立大系统，只做 worker 模块：extract、dedup、merge、prune、profile update。
|
|
||||||
|
|
||||||
第一阶段实现 API：
|
|
||||||
|
|
||||||
- `POST /v1/users`
|
|
||||||
- `GET /v1/users/{user_id}`
|
|
||||||
- `POST /v1/memory/search`
|
|
||||||
- `POST /v1/memory`
|
|
||||||
- `GET /v1/memory/{memory_id}`
|
|
||||||
- `POST /v1/episodes`
|
|
||||||
- `POST /v1/sessions/{session_id}/commit`
|
|
||||||
- `GET /v1/users/{user_id}/profile`
|
|
||||||
- `GET /v1/namespaces`
|
|
||||||
|
|
||||||
第一阶段实现 skills：
|
|
||||||
|
|
||||||
- `ingest_skill`
|
|
||||||
- `summarize_episode_skill`
|
|
||||||
- `retrieve_context_skill`
|
|
||||||
- `commit_memory_skill`
|
|
||||||
- `export_to_obsidian_skill`
|
|
||||||
|
|
||||||
第二阶段再补：
|
|
||||||
|
|
||||||
- `extract_memory_skill`
|
|
||||||
- `classify_memory_skill`
|
|
||||||
- `merge_memory_skill`
|
|
||||||
- `prune_memory_skill`
|
|
||||||
- `import_from_obsidian_skill`
|
|
||||||
- 更完整的 EverOS consolidation 和 profile evolution。
|
|
||||||
|
|
||||||
角色分工：
|
|
||||||
|
|
||||||
- Obsidian 第一阶段：review draft、人类确认 profile/长期知识。第二阶段：双向同步。
|
|
||||||
- OpenViking 第一阶段：统一 context/resource 检索入口。第二阶段：承载多 namespace context filesystem 和 skill registry。
|
|
||||||
- EverOS 第一阶段：session commit worker。第二阶段：长期记忆治理、衰减、冲突检测、profile evolution。
|
|
||||||
|
|
||||||
@ -1,93 +0,0 @@
|
|||||||
# OpenViking Adapter Config
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
Memory Gateway v2 keeps the OpenViking ingest adapter in `offline` / `skeleton`
|
|
||||||
mode by default. In the default configuration it does not send any HTTP
|
|
||||||
requests.
|
|
||||||
|
|
||||||
## Modes
|
|
||||||
|
|
||||||
### Offline
|
|
||||||
|
|
||||||
`mode: offline`
|
|
||||||
|
|
||||||
The adapter must not touch the network. It returns fixture-backed normalized
|
|
||||||
results through the existing skeleton path.
|
|
||||||
|
|
||||||
### Skeleton
|
|
||||||
|
|
||||||
`mode: skeleton`
|
|
||||||
|
|
||||||
This behaves like `offline` for now. It keeps the same normalized result path
|
|
||||||
without sending HTTP requests.
|
|
||||||
|
|
||||||
### Real
|
|
||||||
|
|
||||||
Real mode is enabled only when:
|
|
||||||
|
|
||||||
- `mode: real`
|
|
||||||
|
|
||||||
When real mode is active, the adapter may send an HTTP request for OpenViking
|
|
||||||
ingest only. Commit and retrieve remain offline/skeleton in the current phase.
|
|
||||||
The legacy `enabled` field is retained for config compatibility, but it does
|
|
||||||
not open the network path by itself.
|
|
||||||
|
|
||||||
## Config Fields
|
|
||||||
|
|
||||||
- `base_url`
|
|
||||||
The OpenViking API base URL.
|
|
||||||
- `api_key`
|
|
||||||
Token used only for request headers.
|
|
||||||
- `timeout`
|
|
||||||
Request timeout in seconds.
|
|
||||||
- `verify_ssl`
|
|
||||||
TLS verification toggle for the real HTTP path.
|
|
||||||
- `ingest_path`
|
|
||||||
Configurable ingest endpoint path template. The current placeholder is
|
|
||||||
`/api/v1/sessions/{session_id}/messages`.
|
|
||||||
|
|
||||||
## Example Config
|
|
||||||
|
|
||||||
### Offline Example
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
openviking:
|
|
||||||
enabled: false
|
|
||||||
mode: offline
|
|
||||||
url: http://localhost:1933
|
|
||||||
timeout: 30
|
|
||||||
verify_ssl: true
|
|
||||||
```
|
|
||||||
|
|
||||||
### Real Example
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
openviking:
|
|
||||||
enabled: false
|
|
||||||
mode: real
|
|
||||||
url: https://openviking.example.internal
|
|
||||||
api_key: YOUR_OPENVIKING_TOKEN
|
|
||||||
timeout: 30
|
|
||||||
verify_ssl: true
|
|
||||||
ingest_path: /api/v1/sessions/{session_id}/messages
|
|
||||||
```
|
|
||||||
|
|
||||||
## Security
|
|
||||||
|
|
||||||
Runtime ingest requests may temporarily include `content` while the current
|
|
||||||
request is in flight. Memory Gateway does not persist `content`,
|
|
||||||
`raw_request`, `messages`, or `transcript` into SQLite metadata, outbox
|
|
||||||
payloads, or audit summaries.
|
|
||||||
|
|
||||||
`api_key` / tokens are used only in request headers. They do not belong in:
|
|
||||||
|
|
||||||
- adapter result metadata
|
|
||||||
- audit summaries
|
|
||||||
- persisted MemoryRef metadata
|
|
||||||
- error messages
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
The current ingest endpoint path is still a configurable placeholder. It should
|
|
||||||
be calibrated once the real OpenViking API contract is stable.
|
|
||||||
@ -1,290 +0,0 @@
|
|||||||
---
|
|
||||||
name: memory-gateway
|
|
||||||
description: Use this skill when Hermes needs shared long-term memory, user-scoped preferences/profile, workspace memory, session episode capture, Memory Gateway retrieval, OpenViking context search, Obsidian document upload/review, or session commit through the standalone EverOS service. This skill is domain-neutral.
|
|
||||||
version: 3.1.0
|
|
||||||
metadata:
|
|
||||||
hermes:
|
|
||||||
tags: [memory, memory-gateway, openviking, obsidian, everos, long-term-memory, retrieval, agent-context]
|
|
||||||
---
|
|
||||||
|
|
||||||
# Memory Gateway
|
|
||||||
|
|
||||||
Use this skill as Hermes' generic memory layer. It connects Hermes to the local Memory Gateway at `http://127.0.0.1:1934`.
|
|
||||||
|
|
||||||
The gateway provides:
|
|
||||||
|
|
||||||
- v1 user/agent/workspace/session aware memory APIs backed by SQLite metadata.
|
|
||||||
- ACL and namespace routing before retrieval.
|
|
||||||
- OpenViking fan-out search for visible namespaces.
|
|
||||||
- Session episode capture and commit through the standalone EverOS HTTP service, with Gateway local fallback only when configured.
|
|
||||||
- Obsidian review drafts for high-value or conflicting long-term memory candidates.
|
|
||||||
- Legacy summary/document upload endpoints for LLM summarization and Obsidian knowledge ingestion.
|
|
||||||
|
|
||||||
## Environment
|
|
||||||
|
|
||||||
Defaults:
|
|
||||||
|
|
||||||
- Memory Gateway URL: `http://127.0.0.1:1934`
|
|
||||||
- EverOS URL through Gateway config: `http://127.0.0.1:1995`
|
|
||||||
- Obsidian vault: `/home/tom/memory-gateway/obsidian-vault`
|
|
||||||
- Default review queue: `/home/tom/memory-gateway/obsidian-vault/Reviews/Queue`
|
|
||||||
|
|
||||||
Optional env vars:
|
|
||||||
|
|
||||||
- `MEMORY_GATEWAY_URL`
|
|
||||||
- `MEMORY_GATEWAY_API_KEY`
|
|
||||||
- `MEMORY_GATEWAY_OBSIDIAN_VAULT`
|
|
||||||
|
|
||||||
## Recommended Hermes Workflow
|
|
||||||
|
|
||||||
For normal agent work:
|
|
||||||
|
|
||||||
1. Search memory before answering if prior context may matter.
|
|
||||||
2. Append important session episodes while working.
|
|
||||||
3. Commit the session at the end so EverOS can promote stable memories.
|
|
||||||
4. Use feedback to mark incorrect, duplicate, outdated, or useful memories.
|
|
||||||
5. Upload documents only when they are reusable knowledge, not raw noisy logs.
|
|
||||||
|
|
||||||
Do not write full transcripts to long-term memory. Use episodes for temporary process capture and commit only stable conclusions.
|
|
||||||
|
|
||||||
## v1 Memory Commands
|
|
||||||
|
|
||||||
### Check EverOS
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python /home/tom/.hermes/skills/memory-gateway/scripts/everos_health.py
|
|
||||||
```
|
|
||||||
|
|
||||||
Expected healthy response includes `status: ok` and `response.service: everos-local`.
|
|
||||||
|
|
||||||
### Create User
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python /home/tom/.hermes/skills/memory-gateway/scripts/memory_create_user.py \
|
|
||||||
--user-id user_tom \
|
|
||||||
--display-name "Tom" \
|
|
||||||
--preference language=zh-CN
|
|
||||||
```
|
|
||||||
|
|
||||||
### Search ACL-Aware Memory
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python /home/tom/.hermes/skills/memory-gateway/scripts/memory_search.py \
|
|
||||||
--user-id user_tom \
|
|
||||||
--agent-id agent_hermes \
|
|
||||||
--workspace-id ws_memory_gateway \
|
|
||||||
--query "namespace ACL decision" \
|
|
||||||
--limit 5
|
|
||||||
```
|
|
||||||
|
|
||||||
Equivalent backward-compatible command:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python /home/tom/.hermes/skills/memory-gateway/scripts/retrieve_memory.py \
|
|
||||||
--user-id user_tom \
|
|
||||||
--agent-id agent_hermes \
|
|
||||||
--workspace-id ws_memory_gateway \
|
|
||||||
--query "namespace ACL decision" \
|
|
||||||
--limit 5
|
|
||||||
```
|
|
||||||
|
|
||||||
If `retrieve_memory.py` is called without `--user-id`, it falls back to the legacy `/api/search` endpoint.
|
|
||||||
|
|
||||||
### Upsert Long-Term Memory
|
|
||||||
|
|
||||||
Use this only for stable, concise, reusable memory.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python /home/tom/.hermes/skills/memory-gateway/scripts/memory_upsert.py \
|
|
||||||
--user-id user_tom \
|
|
||||||
--agent-id agent_hermes \
|
|
||||||
--workspace-id ws_memory_gateway \
|
|
||||||
--memory-type preference \
|
|
||||||
--visibility private \
|
|
||||||
--importance 0.8 \
|
|
||||||
--confidence 0.9 \
|
|
||||||
--tag preference \
|
|
||||||
--summary "中文、结构化、轻量 POC 优先" \
|
|
||||||
--text "用户偏好中文输出，结构化但不要过度工程化。"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Append Session Episode
|
|
||||||
|
|
||||||
Use this during a task to record useful process notes without immediately polluting long-term memory.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python /home/tom/.hermes/skills/memory-gateway/scripts/memory_append_episode.py \
|
|
||||||
--user-id user_tom \
|
|
||||||
--agent-id agent_hermes \
|
|
||||||
--workspace-id ws_memory_gateway \
|
|
||||||
--session-id sess_demo \
|
|
||||||
--tag decision \
|
|
||||||
--text "结论：这个项目必须保留用户隔离和 namespace ACL。"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Commit Session Through EverOS
|
|
||||||
|
|
||||||
This asks Memory Gateway to call the standalone EverOS service configured in `config.yaml`.
|
|
||||||
For local POC the default service is `http://127.0.0.1:1995`. If `everos.fallback_to_local` is true and the service is unavailable, Gateway returns `everos_backend: local-fallback`.
|
|
||||||
|
|
||||||
- extracts candidate memories from session episodes
|
|
||||||
- deduplicates exact repeated candidates
|
|
||||||
- detects simple conflicts
|
|
||||||
- promotes normal stable memories into SQLite long-term memory
|
|
||||||
- sends high-value or conflicting candidates to Obsidian review drafts
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python /home/tom/.hermes/skills/memory-gateway/scripts/memory_commit_session.py \
|
|
||||||
--user-id user_tom \
|
|
||||||
--agent-id agent_hermes \
|
|
||||||
--workspace-id ws_memory_gateway \
|
|
||||||
--session-id sess_demo \
|
|
||||||
--min-importance 0.6
|
|
||||||
```
|
|
||||||
|
|
||||||
Review drafts are written under:
|
|
||||||
|
|
||||||
```text
|
|
||||||
/home/tom/memory-gateway/obsidian-vault/Reviews/Queue/
|
|
||||||
```
|
|
||||||
|
|
||||||
### Get Profile
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python /home/tom/.hermes/skills/memory-gateway/scripts/memory_get_profile.py \
|
|
||||||
--user-id user_tom
|
|
||||||
```
|
|
||||||
|
|
||||||
### List Visible Namespaces
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python /home/tom/.hermes/skills/memory-gateway/scripts/memory_list_namespaces.py \
|
|
||||||
--user-id user_tom \
|
|
||||||
--agent-id agent_hermes \
|
|
||||||
--workspace-id ws_memory_gateway \
|
|
||||||
--session-id sess_demo
|
|
||||||
```
|
|
||||||
|
|
||||||
### Patch Memory
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python /home/tom/.hermes/skills/memory-gateway/scripts/memory_patch.py \
|
|
||||||
--memory-id mem_xxx \
|
|
||||||
--user-id user_tom \
|
|
||||||
--agent-id agent_hermes \
|
|
||||||
--workspace-id ws_memory_gateway \
|
|
||||||
--summary "用户偏好中文、结构化、少废话。" \
|
|
||||||
--importance 0.9 \
|
|
||||||
--tag preference \
|
|
||||||
--tag confirmed
|
|
||||||
```
|
|
||||||
|
|
||||||
### Feedback
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python /home/tom/.hermes/skills/memory-gateway/scripts/memory_feedback.py \
|
|
||||||
--memory-id mem_xxx \
|
|
||||||
--user-id user_tom \
|
|
||||||
--agent-id agent_hermes \
|
|
||||||
--workspace-id ws_memory_gateway \
|
|
||||||
--feedback incorrect \
|
|
||||||
--comment "这是临时偏好，不应长期保留。"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Delete Memory
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python /home/tom/.hermes/skills/memory-gateway/scripts/memory_delete.py \
|
|
||||||
--memory-id mem_xxx \
|
|
||||||
--user-id user_tom \
|
|
||||||
--agent-id agent_hermes \
|
|
||||||
--workspace-id ws_memory_gateway
|
|
||||||
```
|
|
||||||
|
|
||||||
## Knowledge And Obsidian Commands
|
|
||||||
|
|
||||||
### Summarize And Commit Via Legacy LLM Endpoint
|
|
||||||
|
|
||||||
Use this for high-value text that should become an OpenViking resource or summarized memory.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python /home/tom/.hermes/skills/memory-gateway/scripts/commit_summary.py \
|
|
||||||
--title "Project decision summary" \
|
|
||||||
--namespace memory-gateway \
|
|
||||||
--memory-type decision \
|
|
||||||
--tag project --tag decision \
|
|
||||||
--persist-as resource \
|
|
||||||
--text "<final conclusion or reusable knowledge>"
|
|
||||||
```
|
|
||||||
|
|
||||||
This calls `POST /api/summary`.
|
|
||||||
|
|
||||||
### Upload Document As Knowledge
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python /home/tom/.hermes/skills/memory-gateway/scripts/upload_knowledge.py \
|
|
||||||
--file /path/to/document.pdf \
|
|
||||||
--title "Design Notes" \
|
|
||||||
--namespace memory-gateway \
|
|
||||||
--knowledge-type design_doc \
|
|
||||||
--tags project,design,reference \
|
|
||||||
--persist-as resource
|
|
||||||
```
|
|
||||||
|
|
||||||
This calls `POST /api/knowledge/upload`: document -> MarkItDown Markdown -> Obsidian note -> LLM summary -> OpenViking resource.
|
|
||||||
|
|
||||||
### Search Local Obsidian Notes
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python /home/tom/.hermes/skills/memory-gateway/scripts/search_obsidian.py \
|
|
||||||
--query "design notes memory gateway" \
|
|
||||||
--limit 5
|
|
||||||
```
|
|
||||||
|
|
||||||
## MCP Tool Names
|
|
||||||
|
|
||||||
The gateway also exposes these v1 tools through `/mcp/rpc`:
|
|
||||||
|
|
||||||
- `memory_search`
|
|
||||||
- `memory_upsert`
|
|
||||||
- `memory_append_episode`
|
|
||||||
- `memory_commit_session`
|
|
||||||
- `memory_get_profile`
|
|
||||||
- `memory_list_namespaces`
|
|
||||||
- `memory_delete`
|
|
||||||
- `memory_feedback`
|
|
||||||
|
|
||||||
Use MCP tools when Hermes has an MCP bridge available. Use the scripts above when Hermes runs skills as shell commands.
|
|
||||||
|
|
||||||
## Output Template
|
|
||||||
|
|
||||||
When using this skill, answer with:
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
## Answer
|
|
||||||
<direct answer or synthesis>
|
|
||||||
|
|
||||||
## Memory References
|
|
||||||
- `<memory_id or URI>` — `<namespace>` — why it matters
|
|
||||||
|
|
||||||
## Obsidian Review
|
|
||||||
- `<draft path>` — why it needs review
|
|
||||||
|
|
||||||
## Memory Action
|
|
||||||
- searched: yes/no
|
|
||||||
- appended_episode: yes/no
|
|
||||||
- committed_session: yes/no
|
|
||||||
- promoted_memory_count:
|
|
||||||
- review_draft_count:
|
|
||||||
```
|
|
||||||
|
|
||||||
## Guardrails
|
|
||||||
|
|
||||||
- Do not store raw noisy data as long-term memory.
|
|
||||||
- Use `memory_append_episode.py` for temporary process notes.
|
|
||||||
- Use `memory_commit_session.py` at task end to let EverOS decide what should persist.
|
|
||||||
- Use `memory_upsert.py` directly only for stable, concise, user-approved memory.
|
|
||||||
- Do not commit secrets, credentials, tokens, private keys, or unnecessary personal data.
|
|
||||||
- If content is sensitive, summarize and redact before committing.
|
|
||||||
- High-value or conflicting candidates should go to Obsidian review drafts before becoming durable memory.
|
|
||||||
- Always report whether retrieval, episode append, session commit, or upload actually succeeded.
|
|
||||||
@ -1,52 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import urllib.parse
|
|
||||||
import urllib.request
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
DEFAULT_GATEWAY_URL = os.environ.get("MEMORY_GATEWAY_URL", "http://127.0.0.1:1934")
|
|
||||||
DEFAULT_GATEWAY_API_KEY = os.environ.get("MEMORY_GATEWAY_API_KEY", "")
|
|
||||||
|
|
||||||
|
|
||||||
def post_json(path: str, payload: dict[str, Any], gateway_url: str = DEFAULT_GATEWAY_URL, api_key: str = DEFAULT_GATEWAY_API_KEY, timeout: int = 120) -> dict[str, Any]:
|
|
||||||
data = json.dumps(payload, ensure_ascii=False).encode("utf-8")
|
|
||||||
req = urllib.request.Request(gateway_url.rstrip("/") + path, data=data, method="POST")
|
|
||||||
req.add_header("Content-Type", "application/json")
|
|
||||||
if api_key:
|
|
||||||
req.add_header("X-API-Key", api_key)
|
|
||||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
|
||||||
return json.loads(resp.read().decode("utf-8"))
|
|
||||||
|
|
||||||
|
|
||||||
def get_json(path: str, params: dict[str, Any] | None = None, gateway_url: str = DEFAULT_GATEWAY_URL, api_key: str = DEFAULT_GATEWAY_API_KEY, timeout: int = 120) -> dict[str, Any] | list[Any]:
|
|
||||||
query = urllib.parse.urlencode({k: v for k, v in (params or {}).items() if v not in (None, "")})
|
|
||||||
url = gateway_url.rstrip("/") + path + (f"?{query}" if query else "")
|
|
||||||
req = urllib.request.Request(url, method="GET")
|
|
||||||
if api_key:
|
|
||||||
req.add_header("X-API-Key", api_key)
|
|
||||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
|
||||||
return json.loads(resp.read().decode("utf-8"))
|
|
||||||
|
|
||||||
|
|
||||||
def patch_json(path: str, payload: dict[str, Any], params: dict[str, Any] | None = None, gateway_url: str = DEFAULT_GATEWAY_URL, api_key: str = DEFAULT_GATEWAY_API_KEY, timeout: int = 120) -> dict[str, Any]:
|
|
||||||
query = urllib.parse.urlencode({k: v for k, v in (params or {}).items() if v not in (None, "")})
|
|
||||||
url = gateway_url.rstrip("/") + path + (f"?{query}" if query else "")
|
|
||||||
data = json.dumps(payload, ensure_ascii=False).encode("utf-8")
|
|
||||||
req = urllib.request.Request(url, data=data, method="PATCH")
|
|
||||||
req.add_header("Content-Type", "application/json")
|
|
||||||
if api_key:
|
|
||||||
req.add_header("X-API-Key", api_key)
|
|
||||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
|
||||||
return json.loads(resp.read().decode("utf-8"))
|
|
||||||
|
|
||||||
|
|
||||||
def delete_json(path: str, params: dict[str, Any] | None = None, gateway_url: str = DEFAULT_GATEWAY_URL, api_key: str = DEFAULT_GATEWAY_API_KEY, timeout: int = 120) -> dict[str, Any]:
|
|
||||||
query = urllib.parse.urlencode({k: v for k, v in (params or {}).items() if v not in (None, "")})
|
|
||||||
url = gateway_url.rstrip("/") + path + (f"?{query}" if query else "")
|
|
||||||
req = urllib.request.Request(url, method="DELETE")
|
|
||||||
if api_key:
|
|
||||||
req.add_header("X-API-Key", api_key)
|
|
||||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
|
||||||
return json.loads(resp.read().decode("utf-8"))
|
|
||||||
@ -1,53 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, post_json
|
|
||||||
|
|
||||||
|
|
||||||
def load_text(args: argparse.Namespace) -> str:
|
|
||||||
if args.file:
|
|
||||||
return Path(args.file).read_text(encoding="utf-8")
|
|
||||||
if args.text:
|
|
||||||
return args.text
|
|
||||||
return sys.stdin.read().strip()
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(description="Summarize arbitrary content with the Gateway LLM and commit it as memory/resource.")
|
|
||||||
parser.add_argument("--text", help="Text to summarize; stdin is used if omitted")
|
|
||||||
parser.add_argument("--file", help="File containing text to summarize")
|
|
||||||
parser.add_argument("--title", default="")
|
|
||||||
parser.add_argument("--summary", default="", help="Optional summary hint")
|
|
||||||
parser.add_argument("--namespace", default="memory-gateway")
|
|
||||||
parser.add_argument("--memory-type", default="summary")
|
|
||||||
parser.add_argument("--tag", action="append", default=[])
|
|
||||||
parser.add_argument("--source", default="hermes:memory-gateway")
|
|
||||||
parser.add_argument("--resource-uri", default="")
|
|
||||||
parser.add_argument("--persist-as", choices=["memory", "resource", "both", "none"], default="resource")
|
|
||||||
parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL)
|
|
||||||
parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY)
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
content = load_text(args)
|
|
||||||
if not content:
|
|
||||||
parser.error("No content provided via --text, --file, or stdin")
|
|
||||||
payload = {
|
|
||||||
"content": content,
|
|
||||||
"title": args.title or None,
|
|
||||||
"summary": args.summary or None,
|
|
||||||
"namespace": args.namespace,
|
|
||||||
"memory_type": args.memory_type,
|
|
||||||
"tags": args.tag,
|
|
||||||
"source": args.source,
|
|
||||||
"resource_uri": args.resource_uri or None,
|
|
||||||
"persist_as": args.persist_as,
|
|
||||||
}
|
|
||||||
print(json.dumps(post_json("/api/summary", payload, args.gateway_url, args.api_key), ensure_ascii=False, indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@ -1,21 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Check EverOS health through Memory Gateway."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
|
|
||||||
from _client import get_json
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(description="Check EverOS health through Memory Gateway.")
|
|
||||||
parser.add_argument("--gateway-url", default="http://127.0.0.1:1934")
|
|
||||||
parser.add_argument("--api-key", default=None)
|
|
||||||
args = parser.parse_args()
|
|
||||||
print(json.dumps(get_json("/v1/everos/health", gateway_url=args.gateway_url, api_key=args.api_key), ensure_ascii=False, indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@ -1,54 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, post_json
|
|
||||||
|
|
||||||
|
|
||||||
def load_content(args: argparse.Namespace) -> str:
|
|
||||||
if args.file:
|
|
||||||
return Path(args.file).read_text(encoding="utf-8")
|
|
||||||
if args.text:
|
|
||||||
return args.text
|
|
||||||
return sys.stdin.read().strip()
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(description="Append session episode memory without directly promoting it.")
|
|
||||||
parser.add_argument("--user-id", required=True)
|
|
||||||
parser.add_argument("--session-id", required=True)
|
|
||||||
parser.add_argument("--agent-id", default="")
|
|
||||||
parser.add_argument("--workspace-id", default="")
|
|
||||||
parser.add_argument("--namespace", default="")
|
|
||||||
parser.add_argument("--text", default="")
|
|
||||||
parser.add_argument("--file", default="")
|
|
||||||
parser.add_argument("--tag", action="append", default=[])
|
|
||||||
parser.add_argument("--source", default="conversation")
|
|
||||||
parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL)
|
|
||||||
parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY)
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
content = load_content(args)
|
|
||||||
if not content:
|
|
||||||
parser.error("No episode content provided via --text, --file, or stdin")
|
|
||||||
|
|
||||||
payload = {
|
|
||||||
"user_id": args.user_id,
|
|
||||||
"agent_id": args.agent_id or None,
|
|
||||||
"workspace_id": args.workspace_id or None,
|
|
||||||
"session_id": args.session_id,
|
|
||||||
"namespace": args.namespace or None,
|
|
||||||
"content": content,
|
|
||||||
"tags": args.tag,
|
|
||||||
"source": args.source,
|
|
||||||
}
|
|
||||||
print(json.dumps(post_json("/v1/episodes", payload, args.gateway_url, args.api_key), ensure_ascii=False, indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
|
|
||||||
@ -1,37 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
|
|
||||||
from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, post_json
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(description="Commit a session through the minimal EverOS consolidation worker.")
|
|
||||||
parser.add_argument("--user-id", required=True)
|
|
||||||
parser.add_argument("--session-id", required=True)
|
|
||||||
parser.add_argument("--agent-id", default="")
|
|
||||||
parser.add_argument("--workspace-id", default="")
|
|
||||||
parser.add_argument("--target-namespace", default="")
|
|
||||||
parser.add_argument("--min-importance", type=float, default=0.6)
|
|
||||||
parser.add_argument("--no-promote", action="store_true")
|
|
||||||
parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL)
|
|
||||||
parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY)
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
payload = {
|
|
||||||
"user_id": args.user_id,
|
|
||||||
"agent_id": args.agent_id or None,
|
|
||||||
"workspace_id": args.workspace_id or None,
|
|
||||||
"session_id": args.session_id,
|
|
||||||
"promote": not args.no_promote,
|
|
||||||
"min_importance": args.min_importance,
|
|
||||||
"target_namespace": args.target_namespace or None,
|
|
||||||
}
|
|
||||||
print(json.dumps(post_json(f"/v1/sessions/{args.session_id}/commit", payload, args.gateway_url, args.api_key), ensure_ascii=False, indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
|
|
||||||
@ -1,36 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
|
|
||||||
from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, post_json
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(description="Create or replace a Memory Gateway v1 user.")
|
|
||||||
parser.add_argument("--user-id", required=True)
|
|
||||||
parser.add_argument("--display-name", required=True)
|
|
||||||
parser.add_argument("--preference", action="append", default=[], help="Preference as key=value; repeatable")
|
|
||||||
parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL)
|
|
||||||
parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY)
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
preferences = {}
|
|
||||||
for item in args.preference:
|
|
||||||
if "=" not in item:
|
|
||||||
parser.error(f"Invalid --preference {item!r}; expected key=value")
|
|
||||||
key, value = item.split("=", 1)
|
|
||||||
preferences[key.strip()] = value.strip()
|
|
||||||
|
|
||||||
payload = {
|
|
||||||
"user_id": args.user_id,
|
|
||||||
"display_name": args.display_name,
|
|
||||||
"preferences": preferences,
|
|
||||||
}
|
|
||||||
print(json.dumps(post_json("/v1/users", payload, args.gateway_url, args.api_key), ensure_ascii=False, indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
|
|
||||||
@ -1,31 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
|
|
||||||
from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, delete_json
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(description="Delete a MemoryRecord if the caller has access.")
|
|
||||||
parser.add_argument("--memory-id", required=True)
|
|
||||||
parser.add_argument("--user-id", required=True)
|
|
||||||
parser.add_argument("--agent-id", default="")
|
|
||||||
parser.add_argument("--workspace-id", default="")
|
|
||||||
parser.add_argument("--session-id", default="")
|
|
||||||
parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL)
|
|
||||||
parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY)
|
|
||||||
args = parser.parse_args()
|
|
||||||
params = {
|
|
||||||
"user_id": args.user_id,
|
|
||||||
"agent_id": args.agent_id,
|
|
||||||
"workspace_id": args.workspace_id,
|
|
||||||
"session_id": args.session_id,
|
|
||||||
}
|
|
||||||
print(json.dumps(delete_json(f"/v1/memory/{args.memory_id}", params=params, gateway_url=args.gateway_url, api_key=args.api_key), ensure_ascii=False, indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
|
|
||||||
@ -1,35 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
|
|
||||||
from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, post_json
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(description="Attach quality feedback to a MemoryRecord.")
|
|
||||||
parser.add_argument("--memory-id", required=True)
|
|
||||||
parser.add_argument("--user-id", required=True)
|
|
||||||
parser.add_argument("--agent-id", default="")
|
|
||||||
parser.add_argument("--workspace-id", default="")
|
|
||||||
parser.add_argument("--session-id", default="")
|
|
||||||
parser.add_argument("--feedback", required=True, choices=["useful", "not_useful", "incorrect", "duplicate", "outdated"])
|
|
||||||
parser.add_argument("--comment", default="")
|
|
||||||
parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL)
|
|
||||||
parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY)
|
|
||||||
args = parser.parse_args()
|
|
||||||
payload = {
|
|
||||||
"user_id": args.user_id,
|
|
||||||
"agent_id": args.agent_id or None,
|
|
||||||
"workspace_id": args.workspace_id or None,
|
|
||||||
"session_id": args.session_id or None,
|
|
||||||
"feedback": args.feedback,
|
|
||||||
"comment": args.comment or None,
|
|
||||||
}
|
|
||||||
print(json.dumps(post_json(f"/v1/memory/{args.memory_id}/feedback", payload, args.gateway_url, args.api_key), ensure_ascii=False, indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
|
|
||||||
@ -1,21 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
|
|
||||||
from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, get_json
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(description="Get a user's Memory Gateway profile.")
|
|
||||||
parser.add_argument("--user-id", required=True)
|
|
||||||
parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL)
|
|
||||||
parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY)
|
|
||||||
args = parser.parse_args()
|
|
||||||
print(json.dumps(get_json(f"/v1/users/{args.user_id}/profile", gateway_url=args.gateway_url, api_key=args.api_key), ensure_ascii=False, indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
|
|
||||||
@ -1,30 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
|
|
||||||
from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, get_json
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(description="List namespaces visible to a user/agent/workspace/session context.")
|
|
||||||
parser.add_argument("--user-id", required=True)
|
|
||||||
parser.add_argument("--agent-id", default="")
|
|
||||||
parser.add_argument("--workspace-id", default="")
|
|
||||||
parser.add_argument("--session-id", default="")
|
|
||||||
parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL)
|
|
||||||
parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY)
|
|
||||||
args = parser.parse_args()
|
|
||||||
params = {
|
|
||||||
"user_id": args.user_id,
|
|
||||||
"agent_id": args.agent_id,
|
|
||||||
"workspace_id": args.workspace_id,
|
|
||||||
"session_id": args.session_id,
|
|
||||||
}
|
|
||||||
print(json.dumps(get_json("/v1/namespaces", params=params, gateway_url=args.gateway_url, api_key=args.api_key), ensure_ascii=False, indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
|
|
||||||
@ -1,54 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
|
|
||||||
from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, patch_json
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(description="Patch a MemoryRecord.")
|
|
||||||
parser.add_argument("--memory-id", required=True)
|
|
||||||
parser.add_argument("--user-id", required=True)
|
|
||||||
parser.add_argument("--agent-id", default="")
|
|
||||||
parser.add_argument("--workspace-id", default="")
|
|
||||||
parser.add_argument("--session-id", default="")
|
|
||||||
parser.add_argument("--content", default="")
|
|
||||||
parser.add_argument("--summary", default="")
|
|
||||||
parser.add_argument("--tag", action="append", default=None)
|
|
||||||
parser.add_argument("--importance", type=float, default=None)
|
|
||||||
parser.add_argument("--confidence", type=float, default=None)
|
|
||||||
parser.add_argument("--visibility", choices=["private", "agent-only", "workspace-shared", "global"], default=None)
|
|
||||||
parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL)
|
|
||||||
parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY)
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
payload = {}
|
|
||||||
if args.content:
|
|
||||||
payload["content"] = args.content
|
|
||||||
if args.summary:
|
|
||||||
payload["summary"] = args.summary
|
|
||||||
if args.tag is not None:
|
|
||||||
payload["tags"] = args.tag
|
|
||||||
if args.importance is not None:
|
|
||||||
payload["importance"] = args.importance
|
|
||||||
if args.confidence is not None:
|
|
||||||
payload["confidence"] = args.confidence
|
|
||||||
if args.visibility:
|
|
||||||
payload["visibility"] = args.visibility
|
|
||||||
if not payload:
|
|
||||||
parser.error("No patch fields provided")
|
|
||||||
|
|
||||||
params = {
|
|
||||||
"user_id": args.user_id,
|
|
||||||
"agent_id": args.agent_id,
|
|
||||||
"workspace_id": args.workspace_id,
|
|
||||||
"session_id": args.session_id,
|
|
||||||
}
|
|
||||||
print(json.dumps(patch_json(f"/v1/memory/{args.memory_id}", payload, params=params, gateway_url=args.gateway_url, api_key=args.api_key), ensure_ascii=False, indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
|
|
||||||
@ -1,41 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
|
|
||||||
from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, post_json
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(description="Search v1 Memory Gateway with user/agent/workspace/session ACL.")
|
|
||||||
parser.add_argument("--query", required=True)
|
|
||||||
parser.add_argument("--user-id", required=True)
|
|
||||||
parser.add_argument("--agent-id", default="")
|
|
||||||
parser.add_argument("--workspace-id", default="")
|
|
||||||
parser.add_argument("--session-id", default="")
|
|
||||||
parser.add_argument("--namespace", action="append", default=[], help="Allowed namespace to search; repeatable")
|
|
||||||
parser.add_argument("--memory-type", action="append", default=[], help="Memory type filter; repeatable")
|
|
||||||
parser.add_argument("--tag", action="append", default=[], help="Tag filter; repeatable")
|
|
||||||
parser.add_argument("--limit", type=int, default=5)
|
|
||||||
parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL)
|
|
||||||
parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY)
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
payload = {
|
|
||||||
"user_id": args.user_id,
|
|
||||||
"agent_id": args.agent_id or None,
|
|
||||||
"workspace_id": args.workspace_id or None,
|
|
||||||
"session_id": args.session_id or None,
|
|
||||||
"query": args.query,
|
|
||||||
"namespaces": args.namespace,
|
|
||||||
"memory_types": args.memory_type,
|
|
||||||
"tags": args.tag,
|
|
||||||
"limit": args.limit,
|
|
||||||
}
|
|
||||||
print(json.dumps(post_json("/v1/memory/search", payload, args.gateway_url, args.api_key), ensure_ascii=False, indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
|
|
||||||
@ -1,64 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, post_json
|
|
||||||
|
|
||||||
|
|
||||||
def load_content(args: argparse.Namespace) -> str:
|
|
||||||
if args.file:
|
|
||||||
return Path(args.file).read_text(encoding="utf-8")
|
|
||||||
if args.text:
|
|
||||||
return args.text
|
|
||||||
return sys.stdin.read().strip()
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(description="Create a v1 MemoryRecord through Memory Gateway.")
|
|
||||||
parser.add_argument("--user-id", required=True)
|
|
||||||
parser.add_argument("--agent-id", default="")
|
|
||||||
parser.add_argument("--workspace-id", default="")
|
|
||||||
parser.add_argument("--session-id", default="")
|
|
||||||
parser.add_argument("--namespace", default="")
|
|
||||||
parser.add_argument("--memory-type", default="fact")
|
|
||||||
parser.add_argument("--text", default="")
|
|
||||||
parser.add_argument("--file", default="")
|
|
||||||
parser.add_argument("--summary", default="")
|
|
||||||
parser.add_argument("--tag", action="append", default=[])
|
|
||||||
parser.add_argument("--importance", type=float, default=0.5)
|
|
||||||
parser.add_argument("--confidence", type=float, default=0.8)
|
|
||||||
parser.add_argument("--visibility", choices=["private", "agent-only", "workspace-shared", "global"], default="private")
|
|
||||||
parser.add_argument("--source", default="manual")
|
|
||||||
parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL)
|
|
||||||
parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY)
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
content = load_content(args)
|
|
||||||
if not content:
|
|
||||||
parser.error("No memory content provided via --text, --file, or stdin")
|
|
||||||
|
|
||||||
payload = {
|
|
||||||
"user_id": args.user_id,
|
|
||||||
"agent_id": args.agent_id or None,
|
|
||||||
"workspace_id": args.workspace_id or None,
|
|
||||||
"session_id": args.session_id or None,
|
|
||||||
"namespace": args.namespace or None,
|
|
||||||
"memory_type": args.memory_type,
|
|
||||||
"content": content,
|
|
||||||
"summary": args.summary or None,
|
|
||||||
"tags": args.tag,
|
|
||||||
"importance": args.importance,
|
|
||||||
"confidence": args.confidence,
|
|
||||||
"visibility": args.visibility,
|
|
||||||
"source": args.source,
|
|
||||||
}
|
|
||||||
print(json.dumps(post_json("/v1/memory", payload, args.gateway_url, args.api_key), ensure_ascii=False, indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
|
|
||||||
@ -1,45 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, post_json
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(description="Retrieve memory/resources from Memory Gateway. Defaults to v1 ACL-aware search when --user-id is provided.")
|
|
||||||
parser.add_argument("--query", required=True, help="Search query")
|
|
||||||
parser.add_argument("--uri", default="", help="Optional OpenViking URI scope, e.g. viking://resources/project")
|
|
||||||
parser.add_argument("--namespace", default="", help="Optional namespace if URI is not provided")
|
|
||||||
parser.add_argument("--user-id", default="", help="Use v1 ACL-aware search when provided")
|
|
||||||
parser.add_argument("--agent-id", default="")
|
|
||||||
parser.add_argument("--workspace-id", default="")
|
|
||||||
parser.add_argument("--session-id", default="")
|
|
||||||
parser.add_argument("--limit", type=int, default=5)
|
|
||||||
parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL)
|
|
||||||
parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY)
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
if args.user_id:
|
|
||||||
payload = {
|
|
||||||
"user_id": args.user_id,
|
|
||||||
"agent_id": args.agent_id or None,
|
|
||||||
"workspace_id": args.workspace_id or None,
|
|
||||||
"session_id": args.session_id or None,
|
|
||||||
"query": args.query,
|
|
||||||
"namespaces": [args.namespace] if args.namespace else [],
|
|
||||||
"limit": args.limit,
|
|
||||||
}
|
|
||||||
result = post_json("/v1/memory/search", payload, args.gateway_url, args.api_key)
|
|
||||||
else:
|
|
||||||
payload = {"query": args.query, "limit": args.limit}
|
|
||||||
if args.uri:
|
|
||||||
payload["uri"] = args.uri
|
|
||||||
if args.namespace:
|
|
||||||
payload["namespace"] = args.namespace
|
|
||||||
result = post_json("/api/search", payload, args.gateway_url, args.api_key)
|
|
||||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@ -1,55 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
DEFAULT_VAULT = os.environ.get("MEMORY_GATEWAY_OBSIDIAN_VAULT", "/home/tom/memory-gateway/obsidian-vault")
|
|
||||||
|
|
||||||
|
|
||||||
def tokenize(query: str) -> list[str]:
|
|
||||||
return [t.lower() for t in re.split(r"[^\w\u4e00-\u9fff.-]+", query) if len(t.strip()) > 1]
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(description="Search local Obsidian Markdown notes by keyword.")
|
|
||||||
parser.add_argument("--query", required=True)
|
|
||||||
parser.add_argument("--vault-root", default=DEFAULT_VAULT)
|
|
||||||
parser.add_argument("--limit", type=int, default=5)
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
root = Path(args.vault_root)
|
|
||||||
tokens = tokenize(args.query)
|
|
||||||
results = []
|
|
||||||
for file in root.rglob("*.md"):
|
|
||||||
try:
|
|
||||||
text = file.read_text(encoding="utf-8")
|
|
||||||
except UnicodeDecodeError:
|
|
||||||
continue
|
|
||||||
haystack = (file.name + "\n" + text).lower()
|
|
||||||
matched = [token for token in tokens if token in haystack]
|
|
||||||
if not matched:
|
|
||||||
continue
|
|
||||||
summary = ""
|
|
||||||
for line in text.splitlines():
|
|
||||||
line = line.strip("# -\t")
|
|
||||||
if len(line) > 30:
|
|
||||||
summary = line[:240]
|
|
||||||
break
|
|
||||||
results.append({
|
|
||||||
"score": len(matched) * 10 + min(len(matched), 10),
|
|
||||||
"file_name": file.name,
|
|
||||||
"relative_path": str(file.relative_to(root)),
|
|
||||||
"absolute_path": str(file),
|
|
||||||
"matched_terms": matched,
|
|
||||||
"summary": summary,
|
|
||||||
})
|
|
||||||
results.sort(key=lambda item: item["score"], reverse=True)
|
|
||||||
print(json.dumps({"query": args.query, "vault_root": str(root), "matched_docs": results[:args.limit]}, ensure_ascii=False, indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@ -1,65 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
import mimetypes
|
|
||||||
import urllib.request
|
|
||||||
from pathlib import Path
|
|
||||||
from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL
|
|
||||||
|
|
||||||
|
|
||||||
def multipart_upload(url: str, fields: dict[str, str], file_path: Path, api_key: str = "") -> dict:
|
|
||||||
boundary = "----memorygatewayboundary"
|
|
||||||
body = bytearray()
|
|
||||||
for name, value in fields.items():
|
|
||||||
if value == "":
|
|
||||||
continue
|
|
||||||
body.extend(f"--{boundary}\r\n".encode())
|
|
||||||
body.extend(f'Content-Disposition: form-data; name="{name}"\r\n\r\n{value}\r\n'.encode())
|
|
||||||
body.extend(f"--{boundary}\r\n".encode())
|
|
||||||
mime = mimetypes.guess_type(file_path.name)[0] or "application/octet-stream"
|
|
||||||
body.extend(f'Content-Disposition: form-data; name="file"; filename="{file_path.name}"\r\n'.encode())
|
|
||||||
body.extend(f"Content-Type: {mime}\r\n\r\n".encode())
|
|
||||||
body.extend(file_path.read_bytes())
|
|
||||||
body.extend(b"\r\n")
|
|
||||||
body.extend(f"--{boundary}--\r\n".encode())
|
|
||||||
req = urllib.request.Request(url, data=bytes(body), method="POST")
|
|
||||||
req.add_header("Content-Type", f"multipart/form-data; boundary={boundary}")
|
|
||||||
if api_key:
|
|
||||||
req.add_header("X-API-Key", api_key)
|
|
||||||
with urllib.request.urlopen(req, timeout=180) as resp:
|
|
||||||
return json.loads(resp.read().decode("utf-8"))
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(description="Upload a document, convert to Markdown, save to Obsidian, summarize with LLM, and commit to OpenViking.")
|
|
||||||
parser.add_argument("--file", required=True)
|
|
||||||
parser.add_argument("--title", default="")
|
|
||||||
parser.add_argument("--namespace", default="memory-gateway")
|
|
||||||
parser.add_argument("--knowledge-type", default="knowledge")
|
|
||||||
parser.add_argument("--tags", default="")
|
|
||||||
parser.add_argument("--source", default="")
|
|
||||||
parser.add_argument("--obsidian-dir", default="")
|
|
||||||
parser.add_argument("--resource-uri", default="")
|
|
||||||
parser.add_argument("--persist-as", choices=["memory", "resource", "both", "none"], default="resource")
|
|
||||||
parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL)
|
|
||||||
parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY)
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
fields = {
|
|
||||||
"title": args.title,
|
|
||||||
"namespace": args.namespace,
|
|
||||||
"knowledge_type": args.knowledge_type,
|
|
||||||
"tags": args.tags,
|
|
||||||
"source": args.source,
|
|
||||||
"obsidian_dir": args.obsidian_dir,
|
|
||||||
"resource_uri": args.resource_uri,
|
|
||||||
"persist_as": args.persist_as,
|
|
||||||
}
|
|
||||||
result = multipart_upload(args.gateway_url.rstrip("/") + "/api/knowledge/upload", fields, Path(args.file), args.api_key)
|
|
||||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@ -1 +0,0 @@
|
|||||||
"""Memory Gateway 核心模块"""
|
|
||||||
@ -1,115 +0,0 @@
|
|||||||
"""Generic Memory Gateway v1 HTTP API."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import Optional
|
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, Query
|
|
||||||
|
|
||||||
from .schemas import (
|
|
||||||
AccessContext,
|
|
||||||
CommitSessionRequest,
|
|
||||||
CreateUserRequest,
|
|
||||||
EpisodeAppendRequest,
|
|
||||||
MemoryFeedbackRequest,
|
|
||||||
MemoryPatchRequest,
|
|
||||||
MemorySearchRequest,
|
|
||||||
MemoryUpsertRequest,
|
|
||||||
)
|
|
||||||
from .server_auth import verify_api_key_compat
|
|
||||||
from .services import service
|
|
||||||
|
|
||||||
router = APIRouter(prefix="/v1", tags=["memory-v1"], dependencies=[Depends(verify_api_key_compat)])
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/users")
|
|
||||||
async def create_user(request: CreateUserRequest):
|
|
||||||
return service.create_user(request)
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/users/{user_id}")
|
|
||||||
async def get_user(user_id: str):
|
|
||||||
return service.get_user(user_id)
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/memory/search")
|
|
||||||
async def search_memory(request: MemorySearchRequest):
|
|
||||||
return await service.search_memory_with_openviking(request)
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/memory")
|
|
||||||
async def upsert_memory(request: MemoryUpsertRequest):
|
|
||||||
return service.upsert_memory(request)
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/memory/{memory_id}")
|
|
||||||
async def get_memory(
|
|
||||||
memory_id: str,
|
|
||||||
user_id: str = Query(...),
|
|
||||||
agent_id: Optional[str] = Query(default=None),
|
|
||||||
workspace_id: Optional[str] = Query(default=None),
|
|
||||||
session_id: Optional[str] = Query(default=None),
|
|
||||||
):
|
|
||||||
return service.get_memory(memory_id, AccessContext(user_id=user_id, agent_id=agent_id, workspace_id=workspace_id, session_id=session_id))
|
|
||||||
|
|
||||||
|
|
||||||
@router.patch("/memory/{memory_id}")
|
|
||||||
async def patch_memory(
|
|
||||||
memory_id: str,
|
|
||||||
patch: MemoryPatchRequest,
|
|
||||||
user_id: str = Query(...),
|
|
||||||
agent_id: Optional[str] = Query(default=None),
|
|
||||||
workspace_id: Optional[str] = Query(default=None),
|
|
||||||
session_id: Optional[str] = Query(default=None),
|
|
||||||
):
|
|
||||||
return service.patch_memory(memory_id, AccessContext(user_id=user_id, agent_id=agent_id, workspace_id=workspace_id, session_id=session_id), patch)
|
|
||||||
|
|
||||||
|
|
||||||
@router.delete("/memory/{memory_id}")
|
|
||||||
async def delete_memory(
|
|
||||||
memory_id: str,
|
|
||||||
user_id: str = Query(...),
|
|
||||||
agent_id: Optional[str] = Query(default=None),
|
|
||||||
workspace_id: Optional[str] = Query(default=None),
|
|
||||||
session_id: Optional[str] = Query(default=None),
|
|
||||||
):
|
|
||||||
return service.delete_memory(memory_id, AccessContext(user_id=user_id, agent_id=agent_id, workspace_id=workspace_id, session_id=session_id))
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/episodes")
|
|
||||||
async def append_episode(request: EpisodeAppendRequest):
|
|
||||||
return service.append_episode(request)
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/sessions/{session_id}/commit")
|
|
||||||
async def commit_session(session_id: str, request: CommitSessionRequest):
|
|
||||||
return service.commit_session(session_id, request)
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/users/{user_id}/profile")
|
|
||||||
async def get_profile(user_id: str):
|
|
||||||
return service.get_profile(user_id)
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/memory/{memory_id}/feedback")
|
|
||||||
async def memory_feedback(memory_id: str, request: MemoryFeedbackRequest):
|
|
||||||
return service.add_feedback(memory_id, request)
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/namespaces")
|
|
||||||
async def list_namespaces(
|
|
||||||
user_id: str = Query(...),
|
|
||||||
agent_id: Optional[str] = Query(default=None),
|
|
||||||
workspace_id: Optional[str] = Query(default=None),
|
|
||||||
session_id: Optional[str] = Query(default=None),
|
|
||||||
):
|
|
||||||
return service.list_namespaces(AccessContext(user_id=user_id, agent_id=agent_id, workspace_id=workspace_id, session_id=session_id))
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/audit")
|
|
||||||
async def list_audit(limit: int = Query(default=100, ge=1, le=1000)):
|
|
||||||
return service.list_audit(limit)
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/everos/health")
|
|
||||||
async def everos_health():
|
|
||||||
return service.everos_health()
|
|
||||||
@ -1,90 +0,0 @@
|
|||||||
"""Memory Gateway v2 workflow API."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import Optional
|
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, Query
|
|
||||||
|
|
||||||
from .schemas_v2 import (
|
|
||||||
BackendRefStatus,
|
|
||||||
BackendType,
|
|
||||||
CommitJobView,
|
|
||||||
CommitRequest,
|
|
||||||
CommitResponse,
|
|
||||||
FeedbackRequest,
|
|
||||||
FeedbackResponse,
|
|
||||||
IngestRequest,
|
|
||||||
IngestResponse,
|
|
||||||
MemoryRefType,
|
|
||||||
MemoryRefView,
|
|
||||||
OutboxProcessResponse,
|
|
||||||
RetrieveRequest,
|
|
||||||
RetrieveResponse,
|
|
||||||
)
|
|
||||||
from .server_auth import verify_api_key_compat
|
|
||||||
from .services_v2 import v2_service
|
|
||||||
|
|
||||||
router = APIRouter(prefix="/v2", tags=["memory-v2"], dependencies=[Depends(verify_api_key_compat)])
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/conversations/ingest", response_model=IngestResponse)
|
|
||||||
async def ingest_conversation(request: IngestRequest):
|
|
||||||
return await v2_service.ingest_conversation_turn(request)
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/conversations/{session_id}/commit", response_model=CommitResponse)
|
|
||||||
async def commit_conversation(session_id: str, request: CommitRequest):
|
|
||||||
return await v2_service.commit_session(session_id, request)
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/jobs/{job_id}", response_model=CommitJobView)
|
|
||||||
async def get_commit_job(job_id: str):
|
|
||||||
return v2_service.get_commit_job_view(job_id)
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/context/retrieve", response_model=RetrieveResponse)
|
|
||||||
async def retrieve_context(request: RetrieveRequest):
|
|
||||||
return await v2_service.retrieve_context(request)
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/memory/refs", response_model=list[MemoryRefView])
|
|
||||||
async def list_memory_refs(
|
|
||||||
workspace_id: Optional[str] = Query(default=None),
|
|
||||||
user_id: Optional[str] = Query(default=None),
|
|
||||||
agent_id: Optional[str] = Query(default=None),
|
|
||||||
session_id: Optional[str] = Query(default=None),
|
|
||||||
namespace: Optional[str] = Query(default=None),
|
|
||||||
backend_type: Optional[BackendType] = Query(default=None),
|
|
||||||
ref_type: Optional[MemoryRefType] = Query(default=None),
|
|
||||||
status: Optional[BackendRefStatus] = Query(default=None),
|
|
||||||
limit: int = Query(default=100, ge=1, le=1000),
|
|
||||||
):
|
|
||||||
return v2_service.list_memory_refs(
|
|
||||||
workspace_id=workspace_id,
|
|
||||||
user_id=user_id,
|
|
||||||
agent_id=agent_id,
|
|
||||||
session_id=session_id,
|
|
||||||
namespace=namespace,
|
|
||||||
backend_type=backend_type,
|
|
||||||
ref_type=ref_type,
|
|
||||||
status=status,
|
|
||||||
limit=limit,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/memory/feedback", response_model=FeedbackResponse)
|
|
||||||
async def memory_feedback(request: FeedbackRequest):
|
|
||||||
return await v2_service.record_memory_feedback(request)
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/admin/outbox/process", response_model=OutboxProcessResponse, tags=["memory-v2-admin"])
|
|
||||||
async def process_outbox(
|
|
||||||
limit: int = Query(default=100, ge=1, le=1000),
|
|
||||||
worker_id: Optional[str] = Query(default=None),
|
|
||||||
lease_seconds: int = Query(default=300, ge=1, le=3600),
|
|
||||||
):
|
|
||||||
return await v2_service.process_pending_outbox_events_summary(
|
|
||||||
limit=limit,
|
|
||||||
worker_id=worker_id,
|
|
||||||
lease_seconds=lease_seconds,
|
|
||||||
)
|
|
||||||
@ -1,129 +0,0 @@
|
|||||||
"""Contract-first mapping spec for future v2 backend adapters.
|
|
||||||
|
|
||||||
This module intentionally does not call OpenViking, EverOS, or Obsidian.
|
|
||||||
It documents the stable Gateway control-plane fields that may be persisted in
|
|
||||||
outbox payload refs, SQLite metadata_json, audit summaries, and related control
|
|
||||||
records. It is not a validator for transient runtime adapter request objects:
|
|
||||||
the Gateway may pass conversation content to a backend during the current
|
|
||||||
request lifecycle, but must not persist that content in SQLite/outbox/audit
|
|
||||||
control-plane records.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import Final, NamedTuple
|
|
||||||
|
|
||||||
from .backend_contracts import BackendCommitResult, BackendOperation, BackendRetrieveResult, BackendWriteResult
|
|
||||||
from .schemas_v2 import BackendType
|
|
||||||
|
|
||||||
|
|
||||||
CONTROL_PLANE_PERSISTED_PAYLOAD_FIELDS: Final[frozenset[str]] = frozenset(
|
|
||||||
{
|
|
||||||
"event_id",
|
|
||||||
"gateway_id",
|
|
||||||
"workspace_id",
|
|
||||||
"user_id",
|
|
||||||
"agent_id",
|
|
||||||
"session_id",
|
|
||||||
"turn_id",
|
|
||||||
"namespace",
|
|
||||||
"source_type",
|
|
||||||
"source_event_id",
|
|
||||||
"backend_type",
|
|
||||||
"operation",
|
|
||||||
"payload_ref",
|
|
||||||
"metadata",
|
|
||||||
"trace",
|
|
||||||
"role",
|
|
||||||
}
|
|
||||||
)
|
|
||||||
CONTROL_PLANE_PAYLOAD_FIELDS: Final[frozenset[str]] = CONTROL_PLANE_PERSISTED_PAYLOAD_FIELDS
|
|
||||||
|
|
||||||
DISALLOWED_PAYLOAD_FIELDS: Final[frozenset[str]] = frozenset(
|
|
||||||
{
|
|
||||||
"content",
|
|
||||||
"raw_request",
|
|
||||||
"messages",
|
|
||||||
"conversation",
|
|
||||||
"transcript",
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class AdapterMappingSpec(NamedTuple):
|
|
||||||
backend_type: BackendType
|
|
||||||
operation: BackendOperation
|
|
||||||
adapter_method: str
|
|
||||||
backend_capability: str
|
|
||||||
result_model: type[BackendWriteResult] | type[BackendCommitResult] | type[BackendRetrieveResult]
|
|
||||||
allowed_payload_fields: frozenset[str] = CONTROL_PLANE_PERSISTED_PAYLOAD_FIELDS
|
|
||||||
|
|
||||||
|
|
||||||
ADAPTER_MAPPING_SPECS: Final[tuple[AdapterMappingSpec, ...]] = (
|
|
||||||
AdapterMappingSpec(
|
|
||||||
backend_type=BackendType.OPENVIKING,
|
|
||||||
operation=BackendOperation.INGEST_TURN,
|
|
||||||
adapter_method="ingest_conversation_turn",
|
|
||||||
backend_capability="session archive append / resource context organization",
|
|
||||||
result_model=BackendWriteResult,
|
|
||||||
),
|
|
||||||
AdapterMappingSpec(
|
|
||||||
backend_type=BackendType.OPENVIKING,
|
|
||||||
operation=BackendOperation.COMMIT_SESSION,
|
|
||||||
adapter_method="commit_session_v2",
|
|
||||||
backend_capability="session commit and session archive ref creation",
|
|
||||||
result_model=BackendCommitResult,
|
|
||||||
),
|
|
||||||
AdapterMappingSpec(
|
|
||||||
backend_type=BackendType.OPENVIKING,
|
|
||||||
operation=BackendOperation.RETRIEVE_CONTEXT,
|
|
||||||
adapter_method="retrieve_context_v2",
|
|
||||||
backend_capability="runtime session/resource context retrieval",
|
|
||||||
result_model=BackendRetrieveResult,
|
|
||||||
),
|
|
||||||
AdapterMappingSpec(
|
|
||||||
backend_type=BackendType.EVEROS,
|
|
||||||
operation=BackendOperation.INGEST_TURN,
|
|
||||||
adapter_method="ingest_message",
|
|
||||||
backend_capability="message-level memory ingestion",
|
|
||||||
result_model=BackendWriteResult,
|
|
||||||
),
|
|
||||||
AdapterMappingSpec(
|
|
||||||
backend_type=BackendType.EVEROS,
|
|
||||||
operation=BackendOperation.COMMIT_SESSION,
|
|
||||||
adapter_method="extract_profile_long_term_v2",
|
|
||||||
backend_capability="episodic/profile/long-term extraction",
|
|
||||||
result_model=BackendCommitResult,
|
|
||||||
),
|
|
||||||
AdapterMappingSpec(
|
|
||||||
backend_type=BackendType.EVEROS,
|
|
||||||
operation=BackendOperation.RETRIEVE_CONTEXT,
|
|
||||||
adapter_method="retrieve_context_v2",
|
|
||||||
backend_capability="episodic/profile/long-term memory retrieval",
|
|
||||||
result_model=BackendRetrieveResult,
|
|
||||||
),
|
|
||||||
AdapterMappingSpec(
|
|
||||||
backend_type=BackendType.OBSIDIAN,
|
|
||||||
operation=BackendOperation.CREATE_REVIEW_DRAFT,
|
|
||||||
adapter_method="create_review_draft_v2",
|
|
||||||
backend_capability="human review draft creation for high-risk/high-conflict candidates",
|
|
||||||
result_model=BackendWriteResult,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def get_adapter_mapping_spec(backend_type: BackendType, operation: BackendOperation) -> AdapterMappingSpec:
|
|
||||||
for spec in ADAPTER_MAPPING_SPECS:
|
|
||||||
if spec.backend_type == backend_type and spec.operation == operation:
|
|
||||||
return spec
|
|
||||||
raise KeyError(f"No v2 adapter mapping for {backend_type.value}:{operation.value}")
|
|
||||||
|
|
||||||
|
|
||||||
def validate_control_plane_payload(payload: dict[str, object]) -> None:
|
|
||||||
"""Validate only persisted control-plane payloads, not runtime adapter requests."""
|
|
||||||
blocked = sorted(DISALLOWED_PAYLOAD_FIELDS.intersection(payload))
|
|
||||||
if blocked:
|
|
||||||
raise ValueError(f"Control-plane persisted payload includes disallowed content fields: {', '.join(blocked)}")
|
|
||||||
|
|
||||||
|
|
||||||
def validate_control_plane_persisted_payload(payload: dict[str, object]) -> None:
|
|
||||||
validate_control_plane_payload(payload)
|
|
||||||
@ -1,134 +0,0 @@
|
|||||||
"""Backend adapter contracts for Memory Gateway v2."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from datetime import datetime
|
|
||||||
from enum import Enum
|
|
||||||
from typing import Any, Optional
|
|
||||||
from uuid import uuid4
|
|
||||||
|
|
||||||
from pydantic import BaseModel, Field
|
|
||||||
|
|
||||||
from .schemas import utc_now
|
|
||||||
from .schemas_v2 import BackendType, MemoryRefType, OperationStatus
|
|
||||||
|
|
||||||
|
|
||||||
class BackendOperation(str, Enum):
|
|
||||||
INGEST_TURN = "ingest_turn"
|
|
||||||
COMMIT_SESSION = "commit_session"
|
|
||||||
RETRIEVE_CONTEXT = "retrieve_context"
|
|
||||||
CREATE_REVIEW_DRAFT = "create_review_draft"
|
|
||||||
|
|
||||||
|
|
||||||
class BackendResultStatus(str, Enum):
|
|
||||||
SUCCESS = "success"
|
|
||||||
FAILED = "failed"
|
|
||||||
SKIPPED = "skipped"
|
|
||||||
PENDING = "pending"
|
|
||||||
|
|
||||||
|
|
||||||
class OutboxEventStatus(str, Enum):
|
|
||||||
PENDING = "pending"
|
|
||||||
PROCESSING = "processing"
|
|
||||||
SUCCESS = "success"
|
|
||||||
SKIPPED = "skipped"
|
|
||||||
FAILED = "failed"
|
|
||||||
DEAD_LETTER = "dead_letter"
|
|
||||||
|
|
||||||
|
|
||||||
class BackendWriteResult(BaseModel):
|
|
||||||
backend_type: BackendType
|
|
||||||
operation: BackendOperation
|
|
||||||
status: BackendResultStatus
|
|
||||||
native_id: Optional[str] = None
|
|
||||||
native_uri: Optional[str] = None
|
|
||||||
retryable: bool = False
|
|
||||||
error_code: Optional[str] = None
|
|
||||||
error_message: Optional[str] = None
|
|
||||||
latency_ms: Optional[float] = None
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class BackendProducedRef(BaseModel):
|
|
||||||
ref_type: MemoryRefType
|
|
||||||
native_id: Optional[str] = None
|
|
||||||
native_uri: Optional[str] = None
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class BackendCommitResult(BaseModel):
|
|
||||||
backend_type: BackendType
|
|
||||||
operation: BackendOperation = BackendOperation.COMMIT_SESSION
|
|
||||||
status: BackendResultStatus
|
|
||||||
native_id: Optional[str] = None
|
|
||||||
native_uri: Optional[str] = None
|
|
||||||
retryable: bool = False
|
|
||||||
error_code: Optional[str] = None
|
|
||||||
error_message: Optional[str] = None
|
|
||||||
latency_ms: Optional[float] = None
|
|
||||||
created_refs: list[str] = Field(default_factory=list)
|
|
||||||
refs: list[BackendProducedRef] = Field(default_factory=list)
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class BackendRetrieveItem(BaseModel):
|
|
||||||
text: Optional[str] = None
|
|
||||||
source_backend: BackendType
|
|
||||||
ref_id: Optional[str] = None
|
|
||||||
score: float = 0.0
|
|
||||||
memory_type: Optional[str] = None
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class BackendRetrieveResult(BaseModel):
|
|
||||||
backend_type: BackendType
|
|
||||||
operation: BackendOperation = BackendOperation.RETRIEVE_CONTEXT
|
|
||||||
status: BackendResultStatus
|
|
||||||
native_id: Optional[str] = None
|
|
||||||
native_uri: Optional[str] = None
|
|
||||||
retryable: bool = False
|
|
||||||
error_code: Optional[str] = None
|
|
||||||
error_message: Optional[str] = None
|
|
||||||
latency_ms: Optional[float] = None
|
|
||||||
items: list[BackendRetrieveItem] = Field(default_factory=list)
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class OutboxEvent(BaseModel):
|
|
||||||
id: str = Field(default_factory=lambda: f"outbox_{uuid4().hex[:16]}")
|
|
||||||
event_type: str
|
|
||||||
gateway_id: str
|
|
||||||
workspace_id: str
|
|
||||||
user_id: str
|
|
||||||
agent_id: Optional[str] = None
|
|
||||||
session_id: Optional[str] = None
|
|
||||||
backend_type: BackendType
|
|
||||||
operation: BackendOperation
|
|
||||||
payload_ref: Optional[str] = None
|
|
||||||
status: OutboxEventStatus = OutboxEventStatus.PENDING
|
|
||||||
attempt_count: int = 0
|
|
||||||
max_attempts: int = 3
|
|
||||||
next_retry_at: Optional[datetime] = None
|
|
||||||
last_error: Optional[str] = None
|
|
||||||
locked_by: Optional[str] = None
|
|
||||||
locked_at: Optional[datetime] = None
|
|
||||||
lease_expires_at: Optional[datetime] = None
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
created_at: datetime = Field(default_factory=utc_now)
|
|
||||||
updated_at: datetime = Field(default_factory=utc_now)
|
|
||||||
|
|
||||||
|
|
||||||
class CommitJob(BaseModel):
|
|
||||||
job_id: str = Field(default_factory=lambda: f"job_{uuid4().hex[:16]}")
|
|
||||||
workspace_id: str
|
|
||||||
user_id: str
|
|
||||||
agent_id: Optional[str] = None
|
|
||||||
session_id: str
|
|
||||||
namespace: Optional[str] = None
|
|
||||||
status: OperationStatus = OperationStatus.ACCEPTED
|
|
||||||
requested_by: Optional[str] = None
|
|
||||||
created_refs_count: int = 0
|
|
||||||
error_message: Optional[str] = None
|
|
||||||
created_at: datetime = Field(default_factory=utc_now)
|
|
||||||
updated_at: datetime = Field(default_factory=utc_now)
|
|
||||||
started_at: Optional[datetime] = None
|
|
||||||
finished_at: Optional[datetime] = None
|
|
||||||
@ -1,261 +0,0 @@
|
|||||||
"""Offline response normalization helpers for future v2 backend adapters."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
from .backend_contracts import (
|
|
||||||
BackendCommitResult,
|
|
||||||
BackendOperation,
|
|
||||||
BackendProducedRef,
|
|
||||||
BackendResultStatus,
|
|
||||||
BackendRetrieveItem,
|
|
||||||
BackendRetrieveResult,
|
|
||||||
BackendWriteResult,
|
|
||||||
)
|
|
||||||
from .backend_ref_mapping import map_backend_ref_type
|
|
||||||
from .schemas_v2 import BackendType
|
|
||||||
|
|
||||||
|
|
||||||
SAFE_METADATA_KEYS = {
|
|
||||||
"backend_request_id",
|
|
||||||
"request_id",
|
|
||||||
"trace_id",
|
|
||||||
"latency_ms",
|
|
||||||
"schema_version",
|
|
||||||
"source_channel",
|
|
||||||
"reason",
|
|
||||||
"original_ref_type",
|
|
||||||
"confidence",
|
|
||||||
"score",
|
|
||||||
"version",
|
|
||||||
"created_at",
|
|
||||||
}
|
|
||||||
BLOCKED_METADATA_KEYS = {"content", "raw_request", "messages", "conversation", "transcript"}
|
|
||||||
|
|
||||||
|
|
||||||
def safe_backend_metadata(metadata: dict[str, Any] | None) -> dict[str, Any]:
|
|
||||||
if not metadata:
|
|
||||||
return {}
|
|
||||||
safe: dict[str, Any] = {}
|
|
||||||
for key, value in metadata.items():
|
|
||||||
if key in BLOCKED_METADATA_KEYS or key not in SAFE_METADATA_KEYS:
|
|
||||||
continue
|
|
||||||
if isinstance(value, (str, int, float, bool)) or value is None:
|
|
||||||
safe[key] = value
|
|
||||||
return safe
|
|
||||||
|
|
||||||
|
|
||||||
def normalize_openviking_commit_response(raw: dict[str, Any]) -> BackendCommitResult:
|
|
||||||
status = _result_status(raw)
|
|
||||||
refs = [_produced_ref(BackendType.OPENVIKING, item) for item in _extract_ref_items(raw)]
|
|
||||||
return BackendCommitResult(
|
|
||||||
backend_type=BackendType.OPENVIKING,
|
|
||||||
operation=BackendOperation.COMMIT_SESSION,
|
|
||||||
status=status,
|
|
||||||
native_id=raw.get("native_id") or raw.get("session_id"),
|
|
||||||
native_uri=raw.get("native_uri") or raw.get("uri"),
|
|
||||||
retryable=_retryable_from_raw(BackendType.OPENVIKING, raw),
|
|
||||||
error_code=raw.get("error_code"),
|
|
||||||
error_message=raw.get("error") or raw.get("error_message"),
|
|
||||||
latency_ms=raw.get("latency_ms"),
|
|
||||||
refs=refs,
|
|
||||||
metadata=safe_backend_metadata(raw.get("metadata") or raw),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def normalize_everos_commit_response(raw: dict[str, Any]) -> BackendCommitResult:
|
|
||||||
status = _result_status(raw)
|
|
||||||
refs = [_produced_ref(BackendType.EVEROS, item) for item in _extract_ref_items(raw)]
|
|
||||||
return BackendCommitResult(
|
|
||||||
backend_type=BackendType.EVEROS,
|
|
||||||
operation=BackendOperation.COMMIT_SESSION,
|
|
||||||
status=status,
|
|
||||||
native_id=raw.get("native_id") or raw.get("session_id"),
|
|
||||||
native_uri=raw.get("native_uri") or raw.get("uri"),
|
|
||||||
retryable=_retryable_from_raw(BackendType.EVEROS, raw),
|
|
||||||
error_code=raw.get("error_code"),
|
|
||||||
error_message=raw.get("error") or raw.get("error_message"),
|
|
||||||
latency_ms=raw.get("latency_ms"),
|
|
||||||
refs=refs,
|
|
||||||
metadata=safe_backend_metadata(raw.get("metadata") or raw),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def normalize_openviking_ingest_response(raw: dict[str, Any]) -> BackendWriteResult:
|
|
||||||
return _write_result(BackendType.OPENVIKING, raw)
|
|
||||||
|
|
||||||
|
|
||||||
def normalize_everos_ingest_response(raw: dict[str, Any]) -> BackendWriteResult:
|
|
||||||
return _write_result(BackendType.EVEROS, raw)
|
|
||||||
|
|
||||||
|
|
||||||
def normalize_openviking_retrieve_response(raw: dict[str, Any]) -> BackendRetrieveResult:
|
|
||||||
return _retrieve_result(BackendType.OPENVIKING, raw)
|
|
||||||
|
|
||||||
|
|
||||||
def normalize_everos_retrieve_response(raw: dict[str, Any]) -> BackendRetrieveResult:
|
|
||||||
return _retrieve_result(BackendType.EVEROS, raw)
|
|
||||||
|
|
||||||
|
|
||||||
def map_backend_error_to_retryable(
|
|
||||||
backend_type: BackendType,
|
|
||||||
status_code: int | None = None,
|
|
||||||
error_code: str | None = None,
|
|
||||||
error_message: str | None = None,
|
|
||||||
) -> bool:
|
|
||||||
"""Map backend errors into retryable/non-retryable categories.
|
|
||||||
|
|
||||||
Unknown errors default to retryable because adapter contracts are still
|
|
||||||
unstable and transient backend/API rollout failures are more likely during
|
|
||||||
integration.
|
|
||||||
"""
|
|
||||||
if status_code in {429, 500, 502, 503, 504}:
|
|
||||||
return True
|
|
||||||
if status_code in {400, 401, 403, 404, 422}:
|
|
||||||
return False
|
|
||||||
text = f"{error_code or ''} {error_message or ''}".lower()
|
|
||||||
if "timeout" in text or "network_error" in text or "connection" in text:
|
|
||||||
return True
|
|
||||||
if "validation" in text or "unauthorized" in text or "forbidden" in text or "not_found" in text:
|
|
||||||
return False
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
def _write_result(backend_type: BackendType, raw: dict[str, Any]) -> BackendWriteResult:
|
|
||||||
data = raw.get("result") or raw.get("data") or {}
|
|
||||||
if not isinstance(data, dict):
|
|
||||||
data = {}
|
|
||||||
native_id = (
|
|
||||||
raw.get("native_id")
|
|
||||||
or raw.get("id")
|
|
||||||
or raw.get("memory_id")
|
|
||||||
or raw.get("request_id")
|
|
||||||
or raw.get("session_id")
|
|
||||||
or data.get("native_id")
|
|
||||||
or data.get("id")
|
|
||||||
or data.get("memory_id")
|
|
||||||
or data.get("request_id")
|
|
||||||
or data.get("session_id")
|
|
||||||
)
|
|
||||||
native_uri = (
|
|
||||||
raw.get("native_uri")
|
|
||||||
or raw.get("uri")
|
|
||||||
or raw.get("url")
|
|
||||||
or data.get("native_uri")
|
|
||||||
or data.get("uri")
|
|
||||||
or data.get("url")
|
|
||||||
)
|
|
||||||
if not native_uri and backend_type == BackendType.OPENVIKING and native_id:
|
|
||||||
native_uri = f"viking://sessions/{native_id}"
|
|
||||||
return BackendWriteResult(
|
|
||||||
backend_type=backend_type,
|
|
||||||
operation=BackendOperation.INGEST_TURN,
|
|
||||||
status=_result_status(raw),
|
|
||||||
native_id=native_id,
|
|
||||||
native_uri=native_uri,
|
|
||||||
retryable=_retryable_from_raw(backend_type, raw),
|
|
||||||
error_code=raw.get("error_code") or raw.get("code"),
|
|
||||||
error_message=raw.get("error") or raw.get("error_message") or raw.get("message"),
|
|
||||||
latency_ms=raw.get("latency_ms"),
|
|
||||||
metadata=safe_backend_metadata(raw.get("metadata") or raw),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _retrieve_result(backend_type: BackendType, raw: dict[str, Any]) -> BackendRetrieveResult:
|
|
||||||
if not isinstance(raw, dict) or not raw:
|
|
||||||
return BackendRetrieveResult(
|
|
||||||
backend_type=backend_type,
|
|
||||||
operation=BackendOperation.RETRIEVE_CONTEXT,
|
|
||||||
status=BackendResultStatus.SKIPPED,
|
|
||||||
metadata={"reason": "malformed_or_empty_response"},
|
|
||||||
)
|
|
||||||
return BackendRetrieveResult(
|
|
||||||
backend_type=backend_type,
|
|
||||||
operation=BackendOperation.RETRIEVE_CONTEXT,
|
|
||||||
status=_result_status(raw),
|
|
||||||
native_id=raw.get("native_id") or raw.get("session_id"),
|
|
||||||
native_uri=raw.get("native_uri") or raw.get("uri"),
|
|
||||||
retryable=_retryable_from_raw(backend_type, raw),
|
|
||||||
error_code=raw.get("error_code") or raw.get("code"),
|
|
||||||
error_message=raw.get("error") or raw.get("error_message") or raw.get("message"),
|
|
||||||
latency_ms=raw.get("latency_ms"),
|
|
||||||
items=[_retrieve_item(backend_type, item) for item in _extract_retrieve_items(raw)],
|
|
||||||
metadata=safe_backend_metadata(raw.get("metadata") or raw),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _retrieve_item(backend_type: BackendType, item: dict[str, Any]) -> BackendRetrieveItem:
|
|
||||||
metadata = safe_backend_metadata(item.get("metadata") if isinstance(item.get("metadata"), dict) else item)
|
|
||||||
return BackendRetrieveItem(
|
|
||||||
text=item.get("text") or item.get("summary") or item.get("abstract"),
|
|
||||||
source_backend=backend_type,
|
|
||||||
ref_id=item.get("ref_id") or item.get("id") or item.get("memory_id") or item.get("profile_id") or item.get("session_id") or item.get("uri"),
|
|
||||||
score=float(item.get("score") or 0.0),
|
|
||||||
memory_type=item.get("memory_type") or item.get("ref_type") or item.get("type") or item.get("kind"),
|
|
||||||
metadata=metadata,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _produced_ref(backend_type: BackendType, item: dict[str, Any]) -> BackendProducedRef:
|
|
||||||
ref_type, mapping_metadata = map_backend_ref_type(backend_type, item.get("ref_type") or item.get("type") or item.get("kind"))
|
|
||||||
metadata = {
|
|
||||||
**safe_backend_metadata(item.get("metadata") if isinstance(item.get("metadata"), dict) else item),
|
|
||||||
**mapping_metadata,
|
|
||||||
}
|
|
||||||
return BackendProducedRef(
|
|
||||||
ref_type=ref_type,
|
|
||||||
native_id=item.get("native_id") or item.get("id") or item.get("memory_id") or item.get("profile_id") or item.get("session_id"),
|
|
||||||
native_uri=item.get("native_uri") or item.get("uri") or item.get("url"),
|
|
||||||
metadata=metadata,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _extract_ref_items(raw: dict[str, Any]) -> list[dict[str, Any]]:
|
|
||||||
data = raw.get("result") or raw.get("data") or raw
|
|
||||||
candidates = (
|
|
||||||
data.get("refs")
|
|
||||||
or data.get("produced_refs")
|
|
||||||
or data.get("created_refs")
|
|
||||||
or data.get("memories")
|
|
||||||
or data.get("items")
|
|
||||||
or []
|
|
||||||
)
|
|
||||||
return [item for item in candidates if isinstance(item, dict)]
|
|
||||||
|
|
||||||
|
|
||||||
def _extract_retrieve_items(raw: dict[str, Any]) -> list[dict[str, Any]]:
|
|
||||||
data = raw.get("result") or raw.get("data") or raw
|
|
||||||
if not isinstance(data, dict):
|
|
||||||
return []
|
|
||||||
candidates = (
|
|
||||||
data.get("items")
|
|
||||||
or data.get("results")
|
|
||||||
or data.get("memories")
|
|
||||||
or data.get("resources")
|
|
||||||
or data.get("contexts")
|
|
||||||
or []
|
|
||||||
)
|
|
||||||
return [item for item in candidates if isinstance(item, dict)]
|
|
||||||
|
|
||||||
|
|
||||||
def _result_status(raw: dict[str, Any]) -> BackendResultStatus:
|
|
||||||
status = str(raw.get("status") or "success").lower()
|
|
||||||
if status in {"ok", "created", "accepted"}:
|
|
||||||
return BackendResultStatus.SUCCESS
|
|
||||||
try:
|
|
||||||
return BackendResultStatus(status)
|
|
||||||
except ValueError:
|
|
||||||
return BackendResultStatus.SUCCESS if not raw.get("error") and not raw.get("error_message") else BackendResultStatus.FAILED
|
|
||||||
|
|
||||||
|
|
||||||
def _retryable_from_raw(backend_type: BackendType, raw: dict[str, Any]) -> bool:
|
|
||||||
if "retryable" in raw:
|
|
||||||
return bool(raw["retryable"])
|
|
||||||
if raw.get("error") or raw.get("error_message") or raw.get("error_code") or raw.get("status_code"):
|
|
||||||
return map_backend_error_to_retryable(
|
|
||||||
backend_type,
|
|
||||||
status_code=raw.get("status_code"),
|
|
||||||
error_code=raw.get("error_code"),
|
|
||||||
error_message=raw.get("error") or raw.get("error_message"),
|
|
||||||
)
|
|
||||||
return False
|
|
||||||
@ -1,66 +0,0 @@
|
|||||||
"""Backend-specific ref type mapping for Memory Gateway v2."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from .schemas_v2 import BackendType, MemoryRefType
|
|
||||||
|
|
||||||
|
|
||||||
OPENVIKING_REF_TYPE_MAP = {
|
|
||||||
"session_archive": MemoryRefType.SESSION_ARCHIVE,
|
|
||||||
"context_resource": MemoryRefType.CONTEXT_RESOURCE,
|
|
||||||
"resource": MemoryRefType.CONTEXT_RESOURCE,
|
|
||||||
"session_summary": MemoryRefType.SESSION_ARCHIVE,
|
|
||||||
}
|
|
||||||
|
|
||||||
EVEROS_REF_TYPE_MAP = {
|
|
||||||
"message_memory": MemoryRefType.MESSAGE_MEMORY,
|
|
||||||
"episodic_memory": MemoryRefType.EPISODIC_MEMORY,
|
|
||||||
"episode": MemoryRefType.EPISODIC_MEMORY,
|
|
||||||
"profile": MemoryRefType.PROFILE,
|
|
||||||
"long_term_memory": MemoryRefType.LONG_TERM_MEMORY,
|
|
||||||
"memory": MemoryRefType.LONG_TERM_MEMORY,
|
|
||||||
"preference": MemoryRefType.PROFILE,
|
|
||||||
}
|
|
||||||
|
|
||||||
OBSIDIAN_REF_TYPE_MAP = {
|
|
||||||
"review_draft": MemoryRefType.DRAFT_REVIEW,
|
|
||||||
"draft_review": MemoryRefType.DRAFT_REVIEW,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def map_backend_ref_type(
|
|
||||||
backend_type: BackendType,
|
|
||||||
backend_ref_type: str | None,
|
|
||||||
) -> tuple[MemoryRefType, dict[str, str]]:
|
|
||||||
"""Map backend-native ref type to a Gateway MemoryRefType.
|
|
||||||
|
|
||||||
Unknown values fall back to a backend-appropriate default and preserve the
|
|
||||||
original value in returned metadata for later inspection.
|
|
||||||
"""
|
|
||||||
raw_type = (backend_ref_type or "").strip()
|
|
||||||
normalized = raw_type.lower()
|
|
||||||
|
|
||||||
if backend_type == BackendType.OPENVIKING:
|
|
||||||
mapped = OPENVIKING_REF_TYPE_MAP.get(normalized, MemoryRefType.SESSION_ARCHIVE)
|
|
||||||
elif backend_type == BackendType.EVEROS:
|
|
||||||
mapped = EVEROS_REF_TYPE_MAP.get(normalized, MemoryRefType.LONG_TERM_MEMORY)
|
|
||||||
elif backend_type == BackendType.OBSIDIAN:
|
|
||||||
mapped = OBSIDIAN_REF_TYPE_MAP.get(normalized, MemoryRefType.DRAFT_REVIEW)
|
|
||||||
else:
|
|
||||||
mapped = MemoryRefType.LONG_TERM_MEMORY
|
|
||||||
|
|
||||||
metadata: dict[str, str] = {}
|
|
||||||
if raw_type and raw_type not in {mapped.value, normalized}:
|
|
||||||
metadata["original_ref_type"] = raw_type
|
|
||||||
elif raw_type and normalized not in _known_backend_ref_types(backend_type):
|
|
||||||
metadata["original_ref_type"] = raw_type
|
|
||||||
return mapped, metadata
|
|
||||||
|
|
||||||
|
|
||||||
def _known_backend_ref_types(backend_type: BackendType) -> set[str]:
|
|
||||||
if backend_type == BackendType.OPENVIKING:
|
|
||||||
return set(OPENVIKING_REF_TYPE_MAP)
|
|
||||||
if backend_type == BackendType.EVEROS:
|
|
||||||
return set(EVEROS_REF_TYPE_MAP)
|
|
||||||
if backend_type == BackendType.OBSIDIAN:
|
|
||||||
return set(OBSIDIAN_REF_TYPE_MAP)
|
|
||||||
return set()
|
|
||||||
@ -1,102 +0,0 @@
|
|||||||
"""配置加载模块"""
|
|
||||||
import os
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Optional
|
|
||||||
|
|
||||||
import yaml
|
|
||||||
from pydantic import ValidationError
|
|
||||||
|
|
||||||
from .types import Config, ServerConfig, OpenVikingConfig, EverOSConfig, MemoryConfig, LoggingConfig, LLMConfig, ObsidianConfig, StorageConfig
|
|
||||||
|
|
||||||
|
|
||||||
def load_config(config_path: Optional[str] = None) -> Config:
|
|
||||||
"""加载配置文件"""
|
|
||||||
if config_path is None:
|
|
||||||
config_path = os.environ.get("MEMORY_GATEWAY_CONFIG", "config.yaml")
|
|
||||||
|
|
||||||
config_file = Path(config_path)
|
|
||||||
|
|
||||||
if not config_file.exists():
|
|
||||||
# 返回默认配置
|
|
||||||
return _apply_env_overrides(Config())
|
|
||||||
|
|
||||||
try:
|
|
||||||
with open(config_file, "r", encoding="utf-8") as f:
|
|
||||||
data = yaml.safe_load(f)
|
|
||||||
|
|
||||||
if data is None:
|
|
||||||
return _apply_env_overrides(Config())
|
|
||||||
|
|
||||||
config = Config(
|
|
||||||
server=ServerConfig(**data.get("server", {})),
|
|
||||||
openviking=OpenVikingConfig(**data.get("openviking", {})),
|
|
||||||
everos=EverOSConfig(**data.get("everos", {})),
|
|
||||||
memory=MemoryConfig(**data.get("memory", {})),
|
|
||||||
logging=LoggingConfig(**data.get("logging", {})),
|
|
||||||
llm=LLMConfig(**data.get("llm", {})),
|
|
||||||
obsidian=ObsidianConfig(**data.get("obsidian", {})),
|
|
||||||
storage=StorageConfig(**data.get("storage", {})),
|
|
||||||
)
|
|
||||||
return _apply_env_overrides(config)
|
|
||||||
except (ValidationError, yaml.YAMLError) as e:
|
|
||||||
print(f"配置文件解析错误: {e}")
|
|
||||||
return _apply_env_overrides(Config())
|
|
||||||
|
|
||||||
|
|
||||||
def get_config() -> Config:
|
|
||||||
"""获取全局配置（单例）"""
|
|
||||||
global _config
|
|
||||||
if _config is None:
|
|
||||||
_config = load_config()
|
|
||||||
return _config
|
|
||||||
|
|
||||||
|
|
||||||
def set_config(config: Config) -> None:
|
|
||||||
"""设置全局配置"""
|
|
||||||
global _config
|
|
||||||
_config = config
|
|
||||||
|
|
||||||
|
|
||||||
_config: Optional[Config] = None
|
|
||||||
|
|
||||||
|
|
||||||
def _apply_env_overrides(config: Config) -> Config:
|
|
||||||
openviking_updates = _backend_env_updates("OPENVIKING")
|
|
||||||
everos_updates = _backend_env_updates("EVEROS")
|
|
||||||
if openviking_updates:
|
|
||||||
config.openviking = config.openviking.model_copy(update=openviking_updates)
|
|
||||||
if everos_updates:
|
|
||||||
config.everos = config.everos.model_copy(update=everos_updates)
|
|
||||||
return config
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_env_updates(prefix: str) -> dict:
|
|
||||||
updates = {}
|
|
||||||
env_map = {
|
|
||||||
"ENABLED": "enabled",
|
|
||||||
"MODE": "mode",
|
|
||||||
"BASE_URL": "url",
|
|
||||||
"URL": "url",
|
|
||||||
"API_KEY": "api_key",
|
|
||||||
"TOKEN": "api_key",
|
|
||||||
"TIMEOUT": "timeout",
|
|
||||||
"TIMEOUT_SECONDS": "timeout",
|
|
||||||
"VERIFY_SSL": "verify_ssl",
|
|
||||||
"INGEST_PATH": "ingest_path",
|
|
||||||
"SEARCH_PATH": "search_path",
|
|
||||||
"FLUSH_PATH": "flush_path",
|
|
||||||
"RETRIEVE_METHOD": "retrieve_method",
|
|
||||||
}
|
|
||||||
for env_name, field_name in env_map.items():
|
|
||||||
value = os.environ.get(f"{prefix}_{env_name}")
|
|
||||||
if value is None:
|
|
||||||
continue
|
|
||||||
if field_name == "enabled":
|
|
||||||
updates[field_name] = value.lower() in {"1", "true", "yes", "on"}
|
|
||||||
elif field_name == "timeout":
|
|
||||||
updates[field_name] = int(value)
|
|
||||||
elif field_name == "verify_ssl":
|
|
||||||
updates[field_name] = value.lower() not in {"0", "false", "no", "off"}
|
|
||||||
else:
|
|
||||||
updates[field_name] = value
|
|
||||||
return updates
|
|
||||||
@ -1,87 +0,0 @@
|
|||||||
"""Document ingestion helpers for Memory Gateway."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import re
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
|
|
||||||
def slugify(value: str, fallback: str = "document") -> str:
|
|
||||||
slug = re.sub(r"[^a-zA-Z0-9\u4e00-\u9fff_-]+", "-", value.lower()).strip("-")
|
|
||||||
slug = re.sub(r"-+", "-", slug)[:100].strip("-")
|
|
||||||
return slug or fallback
|
|
||||||
|
|
||||||
|
|
||||||
def convert_file_to_markdown(file_path: str | Path) -> str:
|
|
||||||
"""Convert a local document to Markdown using Microsoft MarkItDown."""
|
|
||||||
try:
|
|
||||||
from markitdown import MarkItDown
|
|
||||||
except ModuleNotFoundError as exc:
|
|
||||||
raise RuntimeError("markitdown is not installed. Install with: pip install 'markitdown[all]'") from exc
|
|
||||||
|
|
||||||
file_path = Path(file_path)
|
|
||||||
converter = MarkItDown(enable_plugins=False)
|
|
||||||
if hasattr(converter, "convert_local"):
|
|
||||||
result = converter.convert_local(str(file_path))
|
|
||||||
else:
|
|
||||||
result = converter.convert(str(file_path))
|
|
||||||
markdown = getattr(result, "text_content", "") or ""
|
|
||||||
if not markdown.strip():
|
|
||||||
raise RuntimeError("Document conversion produced empty Markdown")
|
|
||||||
return markdown
|
|
||||||
|
|
||||||
|
|
||||||
def build_markdown_note(
|
|
||||||
*,
|
|
||||||
title: str,
|
|
||||||
markdown: str,
|
|
||||||
source_filename: str,
|
|
||||||
tags: list[str],
|
|
||||||
knowledge_type: str,
|
|
||||||
summary: str | None = None,
|
|
||||||
) -> str:
|
|
||||||
tag_text = ", ".join(tags)
|
|
||||||
frontmatter = [
|
|
||||||
"---",
|
|
||||||
f"title: {title}",
|
|
||||||
f"knowledge_type: {knowledge_type}",
|
|
||||||
f"source_filename: {source_filename}",
|
|
||||||
f"created_at: {datetime.now(timezone.utc).isoformat()}",
|
|
||||||
f"tags: [{tag_text}]" if tag_text else "tags: []",
|
|
||||||
]
|
|
||||||
if summary:
|
|
||||||
escaped = summary.replace('"', '\\"')
|
|
||||||
frontmatter.append(f'summary: "{escaped}"')
|
|
||||||
frontmatter.extend(["---", "", f"# {title}", "", markdown.strip(), ""])
|
|
||||||
return "\n".join(frontmatter)
|
|
||||||
|
|
||||||
|
|
||||||
def save_markdown_to_obsidian(
|
|
||||||
*,
|
|
||||||
vault_path: str | Path,
|
|
||||||
relative_dir: str,
|
|
||||||
title: str,
|
|
||||||
markdown: str,
|
|
||||||
source_filename: str,
|
|
||||||
tags: list[str],
|
|
||||||
knowledge_type: str,
|
|
||||||
summary: str | None = None,
|
|
||||||
) -> Path:
|
|
||||||
vault = Path(vault_path)
|
|
||||||
target_dir = vault / relative_dir.strip("/")
|
|
||||||
target_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
digest = slugify(source_filename.rsplit(".", 1)[0] or title)
|
|
||||||
note_name = f"{slugify(title, digest)}.md"
|
|
||||||
target = target_dir / note_name
|
|
||||||
target.write_text(
|
|
||||||
build_markdown_note(
|
|
||||||
title=title,
|
|
||||||
markdown=markdown,
|
|
||||||
source_filename=source_filename,
|
|
||||||
tags=tags,
|
|
||||||
knowledge_type=knowledge_type,
|
|
||||||
summary=summary,
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
return target
|
|
||||||
@ -1,496 +0,0 @@
|
|||||||
"""Client for the external EverOS memory service."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from json import JSONDecodeError
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
import httpx
|
|
||||||
|
|
||||||
from .backend_contracts import BackendCommitResult, BackendOperation, BackendResultStatus, BackendRetrieveResult, BackendWriteResult
|
|
||||||
from .backend_normalization import (
|
|
||||||
map_backend_error_to_retryable,
|
|
||||||
normalize_everos_commit_response,
|
|
||||||
normalize_everos_ingest_response,
|
|
||||||
normalize_everos_retrieve_response,
|
|
||||||
)
|
|
||||||
from .config import get_config
|
|
||||||
from .schemas import AccessContext, EpisodeRecord, MemoryRecord
|
|
||||||
from .schemas_v2 import BackendType
|
|
||||||
|
|
||||||
|
|
||||||
class EverOSError(RuntimeError):
|
|
||||||
"""Raised when the external EverOS service cannot process a request."""
|
|
||||||
|
|
||||||
|
|
||||||
class EverOSClient:
|
|
||||||
"""Small HTTP client with a tolerant response normalizer.
|
|
||||||
|
|
||||||
The deployed EverOS API may evolve independently from Memory Gateway.
|
|
||||||
Gateway sends a stable payload and accepts several common response shapes:
|
|
||||||
`result`, `data`, or the raw top-level object with `candidates/promoted`.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
base_url: str | None = None,
|
|
||||||
api_key: str | None = None,
|
|
||||||
timeout: int | None = None,
|
|
||||||
enabled: bool | None = None,
|
|
||||||
mode: str | None = None,
|
|
||||||
verify_ssl: bool | None = None,
|
|
||||||
health_path: str | None = None,
|
|
||||||
ingest_path: str | None = None,
|
|
||||||
search_path: str | None = None,
|
|
||||||
flush_path: str | None = None,
|
|
||||||
retrieve_method: str | None = None,
|
|
||||||
transport: httpx.BaseTransport | None = None,
|
|
||||||
) -> None:
|
|
||||||
config = get_config().everos
|
|
||||||
self.base_url = (base_url if base_url is not None else config.url).rstrip("/")
|
|
||||||
self.api_key = api_key if api_key is not None else config.api_key
|
|
||||||
self.timeout = timeout or config.timeout
|
|
||||||
self.enabled = config.enabled if enabled is None else enabled
|
|
||||||
self.mode = mode or config.mode
|
|
||||||
self.verify_ssl = config.verify_ssl if verify_ssl is None else verify_ssl
|
|
||||||
self.health_path = health_path or config.health_path
|
|
||||||
self.ingest_path = ingest_path or config.ingest_path
|
|
||||||
self.search_path = search_path or config.search_path
|
|
||||||
self.flush_path = flush_path or config.flush_path
|
|
||||||
self.retrieve_method = retrieve_method or config.retrieve_method
|
|
||||||
self.transport = transport
|
|
||||||
|
|
||||||
def _headers(self) -> dict[str, str]:
|
|
||||||
headers = {"Content-Type": "application/json"}
|
|
||||||
if self.api_key:
|
|
||||||
headers["X-API-Key"] = self.api_key
|
|
||||||
headers["Authorization"] = f"Bearer {self.api_key}"
|
|
||||||
return headers
|
|
||||||
|
|
||||||
def health(self) -> dict[str, Any]:
|
|
||||||
url = self.base_url + self.health_path
|
|
||||||
try:
|
|
||||||
health_timeout = httpx.Timeout(min(self.timeout, 2.0), connect=min(self.timeout, 0.5))
|
|
||||||
with httpx.Client(timeout=health_timeout, headers=self._headers()) as client:
|
|
||||||
response = client.get(url)
|
|
||||||
response.raise_for_status()
|
|
||||||
return {"status": "ok", "url": self.base_url, "response": response.json()}
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
return {"status": "error", "url": self.base_url, "error": str(exc)}
|
|
||||||
|
|
||||||
def ingest_message(self, payload: dict[str, Any]) -> BackendWriteResult:
|
|
||||||
"""Write one Gateway turn to EverOS."""
|
|
||||||
runtime_payload = self._build_ingest_payload(payload)
|
|
||||||
if self._use_real_api:
|
|
||||||
return self._ingest_message_real(runtime_payload)
|
|
||||||
raw = {
|
|
||||||
"status": "skipped",
|
|
||||||
"memory_id": (runtime_payload.get("messages") or [{}])[0].get("message_id"),
|
|
||||||
"metadata": {
|
|
||||||
"reason": "everos_v2_ingest_adapter_not_configured",
|
|
||||||
"schema_version": "everos.fixture.ingest.v2",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
return self._normalize_ingest_response(raw)
|
|
||||||
|
|
||||||
@property
|
|
||||||
def _use_real_api(self) -> bool:
|
|
||||||
# Real ingest is strictly gated by mode=real. The legacy `enabled`
|
|
||||||
# field is retained for config compatibility, but must not trigger
|
|
||||||
# network traffic by itself.
|
|
||||||
return self.mode == "real"
|
|
||||||
|
|
||||||
def _ingest_message_real(self, runtime_payload: dict[str, Any]) -> BackendWriteResult:
|
|
||||||
if not self.base_url:
|
|
||||||
return self._failed_ingest_result(
|
|
||||||
error_code="config_error",
|
|
||||||
error_message="EverOS real ingest is enabled but base_url is missing",
|
|
||||||
retryable=False,
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
with httpx.Client(
|
|
||||||
base_url=self.base_url,
|
|
||||||
headers=self._headers(),
|
|
||||||
timeout=self.timeout,
|
|
||||||
verify=self.verify_ssl,
|
|
||||||
transport=self.transport,
|
|
||||||
) as client:
|
|
||||||
response = client.post(self.ingest_path, json=runtime_payload)
|
|
||||||
if response.status_code >= 400:
|
|
||||||
return self._failed_ingest_result(
|
|
||||||
error_code=f"http_{response.status_code}",
|
|
||||||
error_message=f"EverOS ingest failed with HTTP {response.status_code}",
|
|
||||||
retryable=self._map_error(response),
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
raw = response.json()
|
|
||||||
except (JSONDecodeError, ValueError):
|
|
||||||
return self._failed_ingest_result(
|
|
||||||
error_code="invalid_json",
|
|
||||||
error_message="EverOS ingest returned invalid JSON",
|
|
||||||
retryable=True,
|
|
||||||
)
|
|
||||||
if not isinstance(raw, dict):
|
|
||||||
return self._failed_ingest_result(
|
|
||||||
error_code="unexpected_response",
|
|
||||||
error_message="EverOS ingest returned an unexpected response shape",
|
|
||||||
retryable=True,
|
|
||||||
)
|
|
||||||
return self._normalize_ingest_response(raw)
|
|
||||||
except httpx.TimeoutException as exc:
|
|
||||||
return self._failed_ingest_result("timeout", self._safe_error_message(exc), retryable=self._map_error(exc))
|
|
||||||
except httpx.RequestError as exc:
|
|
||||||
return self._failed_ingest_result("network_error", self._safe_error_message(exc), retryable=self._map_error(exc))
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
return self._failed_ingest_result("unexpected_error", self._safe_error_message(exc), retryable=self._map_error(exc))
|
|
||||||
|
|
||||||
def extract_profile_long_term_v2(self, payload: dict[str, Any]) -> BackendCommitResult:
|
|
||||||
"""v2 adapter placeholder for profile / long-term extraction.
|
|
||||||
|
|
||||||
Mapping spec: commit_session returns BackendCommitResult and should
|
|
||||||
produce native episodic/profile/long-term refs once the real API is stable.
|
|
||||||
"""
|
|
||||||
runtime_payload = self._build_commit_payload(payload)
|
|
||||||
raw = {
|
|
||||||
"status": "success",
|
|
||||||
"session_id": runtime_payload.get("session_id"),
|
|
||||||
"metadata": {
|
|
||||||
"reason": "everos_v2_commit_fixture",
|
|
||||||
"schema_version": "everos.fixture.commit.v2",
|
|
||||||
},
|
|
||||||
"data": {
|
|
||||||
"produced_refs": [
|
|
||||||
{
|
|
||||||
"ref_type": "profile",
|
|
||||||
"profile_id": f"everos_profile:{runtime_payload.get('user_id') or 'unknown'}",
|
|
||||||
"metadata": {"schema_version": "everos.fixture.profile.v2"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"ref_type": "long_term_memory",
|
|
||||||
"memory_id": f"everos_long_term:{runtime_payload.get('session_id')}",
|
|
||||||
"metadata": {"schema_version": "everos.fixture.long_term.v2"},
|
|
||||||
},
|
|
||||||
]
|
|
||||||
},
|
|
||||||
}
|
|
||||||
return self._normalize_commit_response(raw)
|
|
||||||
|
|
||||||
def retrieve_context_v2(self, payload: dict[str, Any]) -> BackendRetrieveResult:
|
|
||||||
"""
|
|
||||||
Calls EverOS native API to retrieve memories.
|
|
||||||
"""
|
|
||||||
if not self._use_real_api:
|
|
||||||
return BackendRetrieveResult(
|
|
||||||
backend_type=BackendType.EVEROS,
|
|
||||||
operation=BackendOperation.RETRIEVE_CONTEXT,
|
|
||||||
status=BackendResultStatus.SKIPPED,
|
|
||||||
items=[],
|
|
||||||
metadata={"reason": "everos_retrieve_requires_real_mode"},
|
|
||||||
)
|
|
||||||
|
|
||||||
query = payload.get("query", "")
|
|
||||||
user_id = payload.get("user_id", "")
|
|
||||||
try:
|
|
||||||
with httpx.Client(
|
|
||||||
base_url=self.base_url,
|
|
||||||
headers=self._headers(),
|
|
||||||
timeout=self.timeout,
|
|
||||||
verify=self.verify_ssl,
|
|
||||||
transport=self.transport,
|
|
||||||
) as client:
|
|
||||||
resp = client.post(
|
|
||||||
self.search_path,
|
|
||||||
json={
|
|
||||||
"query": query,
|
|
||||||
"method": self.retrieve_method,
|
|
||||||
"memory_types": ["episodic_memory", "profile", "raw_message"],
|
|
||||||
"top_k": payload.get("limit", 10),
|
|
||||||
"filters": self._search_filters(user_id=user_id, session_id=payload.get("session_id")),
|
|
||||||
},
|
|
||||||
)
|
|
||||||
if resp.status_code >= 400:
|
|
||||||
return BackendRetrieveResult(
|
|
||||||
backend_type=BackendType.EVEROS,
|
|
||||||
operation=BackendOperation.RETRIEVE_CONTEXT,
|
|
||||||
status=BackendResultStatus.FAILED,
|
|
||||||
items=[],
|
|
||||||
error_code=f"http_{resp.status_code}",
|
|
||||||
error_message=f"EverOS retrieve failed: {resp.text}",
|
|
||||||
retryable=False
|
|
||||||
)
|
|
||||||
|
|
||||||
items = self._items_from_search_response(resp.json())
|
|
||||||
|
|
||||||
raw = {
|
|
||||||
"status": "success",
|
|
||||||
"data": {
|
|
||||||
"items": items
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return self._normalize_retrieve_response(raw)
|
|
||||||
except Exception as exc:
|
|
||||||
return BackendRetrieveResult(
|
|
||||||
backend_type=BackendType.EVEROS,
|
|
||||||
operation=BackendOperation.RETRIEVE_CONTEXT,
|
|
||||||
status=BackendResultStatus.FAILED,
|
|
||||||
items=[],
|
|
||||||
error_code="request_error",
|
|
||||||
error_message=str(exc),
|
|
||||||
retryable=True
|
|
||||||
)
|
|
||||||
|
|
||||||
def _build_ingest_payload(self, payload: dict[str, Any]) -> dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Builds the payload according to EverOS native message schema.
|
|
||||||
"""
|
|
||||||
return {
|
|
||||||
"user_id": payload.get("user_id") or "gateway_user",
|
|
||||||
"session_id": payload.get("session_id"),
|
|
||||||
"messages": [
|
|
||||||
{
|
|
||||||
"message_id": payload.get("turn_id") or f"msg_{int(datetime.now(timezone.utc).timestamp() * 1000)}",
|
|
||||||
"sender_id": payload.get("user_id") or "gateway_user",
|
|
||||||
"sender_name": payload.get("user_id") or "gateway_user",
|
|
||||||
"role": self._everos_role(payload.get("role", "user")),
|
|
||||||
"timestamp": self._timestamp_ms(payload),
|
|
||||||
"content": payload.get("content", ""),
|
|
||||||
}
|
|
||||||
],
|
|
||||||
}
|
|
||||||
|
|
||||||
def _build_commit_payload(self, payload: dict[str, Any]) -> dict[str, Any]:
|
|
||||||
return dict(payload)
|
|
||||||
|
|
||||||
def _normalize_ingest_response(self, raw: dict[str, Any]) -> BackendWriteResult:
|
|
||||||
return normalize_everos_ingest_response(raw)
|
|
||||||
|
|
||||||
def _normalize_commit_response(self, raw: dict[str, Any]) -> BackendCommitResult:
|
|
||||||
return normalize_everos_commit_response(raw)
|
|
||||||
|
|
||||||
def _normalize_retrieve_response(self, raw: dict[str, Any]) -> BackendRetrieveResult:
|
|
||||||
return normalize_everos_retrieve_response(raw)
|
|
||||||
|
|
||||||
def _map_error(self, exc_or_response: Any) -> bool:
|
|
||||||
status_code = getattr(exc_or_response, "status_code", None)
|
|
||||||
error_code = getattr(exc_or_response, "error_code", None)
|
|
||||||
error_message = str(exc_or_response) if exc_or_response is not None else None
|
|
||||||
return map_backend_error_to_retryable(
|
|
||||||
BackendType.EVEROS,
|
|
||||||
status_code=status_code,
|
|
||||||
error_code=error_code,
|
|
||||||
error_message=error_message,
|
|
||||||
)
|
|
||||||
|
|
||||||
def _failed_ingest_result(self, error_code: str, error_message: str, retryable: bool) -> BackendWriteResult:
|
|
||||||
return BackendWriteResult(
|
|
||||||
backend_type=BackendType.EVEROS,
|
|
||||||
operation=BackendOperation.INGEST_TURN,
|
|
||||||
status=BackendResultStatus.FAILED,
|
|
||||||
retryable=retryable,
|
|
||||||
error_code=error_code,
|
|
||||||
error_message=error_message,
|
|
||||||
metadata={"error_code": error_code},
|
|
||||||
)
|
|
||||||
|
|
||||||
def _safe_error_message(self, exc: Exception) -> str:
|
|
||||||
return exc.__class__.__name__
|
|
||||||
|
|
||||||
def consolidate_session(
|
|
||||||
self,
|
|
||||||
session_id: str,
|
|
||||||
ctx: AccessContext,
|
|
||||||
episodes: list[EpisodeRecord],
|
|
||||||
existing_memories: list[MemoryRecord],
|
|
||||||
min_importance: float,
|
|
||||||
target_namespace: str | None,
|
|
||||||
) -> dict[str, Any]:
|
|
||||||
if not self.base_url:
|
|
||||||
raise EverOSError("EverOS real mode requires base_url")
|
|
||||||
|
|
||||||
user_id = ctx.user_id or "gateway_user"
|
|
||||||
agent_id = ctx.agent_id or "gateway_agent"
|
|
||||||
|
|
||||||
with httpx.Client(
|
|
||||||
base_url=self.base_url,
|
|
||||||
timeout=self.timeout,
|
|
||||||
headers=self._headers(),
|
|
||||||
verify=self.verify_ssl,
|
|
||||||
transport=self.transport,
|
|
||||||
) as client:
|
|
||||||
for episode in episodes:
|
|
||||||
self._memorize_episode(client, episode=episode, session_id=session_id, user_id=user_id, agent_id=agent_id)
|
|
||||||
self._flush_session(client, session_id=session_id, user_id=user_id)
|
|
||||||
promoted = self._fetch_session_memories(client, session_id=session_id, user_id=user_id, target_namespace=target_namespace)
|
|
||||||
|
|
||||||
return {
|
|
||||||
"backend": "external",
|
|
||||||
"service_url": self.base_url,
|
|
||||||
"endpoint": self.ingest_path,
|
|
||||||
"raw": {"result": {"memories": promoted}},
|
|
||||||
"session_id": session_id,
|
|
||||||
"episodes": len(episodes),
|
|
||||||
"candidates": promoted,
|
|
||||||
"promoted": promoted,
|
|
||||||
"duplicates": [],
|
|
||||||
"conflicts": [],
|
|
||||||
"review_drafts": [],
|
|
||||||
}
|
|
||||||
|
|
||||||
def _memorize_episode(
|
|
||||||
self,
|
|
||||||
client: httpx.Client,
|
|
||||||
*,
|
|
||||||
episode: dict[str, Any],
|
|
||||||
session_id: str,
|
|
||||||
user_id: str,
|
|
||||||
agent_id: str,
|
|
||||||
) -> None:
|
|
||||||
episode_data = episode.model_dump(mode="json") if hasattr(episode, "model_dump") else dict(episode)
|
|
||||||
episode_id = str(episode_data.get("id") or f"epi_{int(datetime.now(timezone.utc).timestamp())}")
|
|
||||||
sender = agent_id if episode_data.get("source") == "agent" else user_id
|
|
||||||
role = "assistant" if sender == agent_id else "user"
|
|
||||||
created_at = episode_data.get("created_at") or datetime.now(timezone.utc).isoformat()
|
|
||||||
payload = {
|
|
||||||
"user_id": user_id,
|
|
||||||
"session_id": session_id,
|
|
||||||
"messages": [
|
|
||||||
{
|
|
||||||
"message_id": episode_id,
|
|
||||||
"sender_id": sender,
|
|
||||||
"sender_name": sender,
|
|
||||||
"role": role,
|
|
||||||
"timestamp": self._datetime_to_ms(created_at),
|
|
||||||
"content": episode_data.get("content") or "",
|
|
||||||
}
|
|
||||||
],
|
|
||||||
}
|
|
||||||
response = client.post(self.ingest_path, json=payload)
|
|
||||||
response.raise_for_status()
|
|
||||||
|
|
||||||
def _flush_session(self, client: httpx.Client, *, session_id: str, user_id: str) -> None:
|
|
||||||
response = client.post(self.flush_path, json={"user_id": user_id, "session_id": session_id})
|
|
||||||
response.raise_for_status()
|
|
||||||
|
|
||||||
def _fetch_session_memories(
|
|
||||||
self,
|
|
||||||
client: httpx.Client,
|
|
||||||
*,
|
|
||||||
session_id: str,
|
|
||||||
user_id: str,
|
|
||||||
target_namespace: str | None,
|
|
||||||
) -> list[dict[str, Any]]:
|
|
||||||
response = client.post(
|
|
||||||
self.search_path,
|
|
||||||
json={
|
|
||||||
"query": "memory",
|
|
||||||
"method": self.retrieve_method,
|
|
||||||
"memory_types": ["episodic_memory"],
|
|
||||||
"top_k": 20,
|
|
||||||
"filters": self._search_filters(user_id=user_id, session_id=session_id),
|
|
||||||
},
|
|
||||||
)
|
|
||||||
response.raise_for_status()
|
|
||||||
memories = self._items_from_search_response(response.json())
|
|
||||||
normalized: list[dict[str, Any]] = []
|
|
||||||
for index, memory in enumerate(memories, start=1):
|
|
||||||
content = memory.get("text") or memory.get("content") or memory.get("summary") or ""
|
|
||||||
if not content:
|
|
||||||
continue
|
|
||||||
normalized.append(
|
|
||||||
{
|
|
||||||
"id": memory.get("memory_id") or memory.get("id") or f"everos_{session_id}_{index}",
|
|
||||||
"namespace": target_namespace or f"user/{user_id}/long_term",
|
|
||||||
"memory_type": memory.get("memory_type") or "episodic_memory",
|
|
||||||
"content": content,
|
|
||||||
"summary": memory.get("summary") or content[:180],
|
|
||||||
"tags": ["everos-real", "memory-gateway"],
|
|
||||||
"importance": 0.7,
|
|
||||||
"confidence": 0.7,
|
|
||||||
"source": "everos",
|
|
||||||
"source_ref": memory.get("memory_id") or memory.get("id"),
|
|
||||||
}
|
|
||||||
)
|
|
||||||
return normalized
|
|
||||||
|
|
||||||
def _items_from_search_response(self, payload: dict[str, Any]) -> list[dict[str, Any]]:
|
|
||||||
data = payload.get("data") if isinstance(payload.get("data"), dict) else payload
|
|
||||||
items: list[dict[str, Any]] = []
|
|
||||||
for memory_type, key in (
|
|
||||||
("episodic_memory", "episodes"),
|
|
||||||
("profile", "profiles"),
|
|
||||||
("raw_message", "raw_messages"),
|
|
||||||
):
|
|
||||||
for item in data.get(key, []) or []:
|
|
||||||
if isinstance(item, dict):
|
|
||||||
items.append({**item, "memory_type": item.get("memory_type") or memory_type, "text": self._memory_text(item)})
|
|
||||||
agent_memory = data.get("agent_memory") or {}
|
|
||||||
if isinstance(agent_memory, dict):
|
|
||||||
for item in agent_memory.get("cases", []) or []:
|
|
||||||
if isinstance(item, dict):
|
|
||||||
items.append({**item, "memory_type": "agent_case", "text": self._memory_text(item)})
|
|
||||||
for item in agent_memory.get("skills", []) or []:
|
|
||||||
if isinstance(item, dict):
|
|
||||||
items.append({**item, "memory_type": "agent_skill", "text": self._memory_text(item)})
|
|
||||||
return items
|
|
||||||
|
|
||||||
def _memory_text(self, item: dict[str, Any]) -> str:
|
|
||||||
content_items = item.get("content_items")
|
|
||||||
if isinstance(content_items, list):
|
|
||||||
content_text = "\n".join(
|
|
||||||
str(content.get("text") or content.get("content") or "")
|
|
||||||
for content in content_items
|
|
||||||
if isinstance(content, dict)
|
|
||||||
).strip()
|
|
||||||
else:
|
|
||||||
content_text = ""
|
|
||||||
profile_data = item.get("profile_data")
|
|
||||||
if isinstance(profile_data, dict):
|
|
||||||
profile_text = str(profile_data)
|
|
||||||
else:
|
|
||||||
profile_text = ""
|
|
||||||
return (
|
|
||||||
item.get("episode")
|
|
||||||
or item.get("summary")
|
|
||||||
or item.get("subject")
|
|
||||||
or item.get("atomic_fact")
|
|
||||||
or item.get("task_intent")
|
|
||||||
or item.get("approach")
|
|
||||||
or item.get("content")
|
|
||||||
or content_text
|
|
||||||
or item.get("description")
|
|
||||||
or profile_text
|
|
||||||
or ""
|
|
||||||
)
|
|
||||||
|
|
||||||
def _search_filters(self, *, user_id: str | None, session_id: str | None = None) -> dict[str, Any]:
|
|
||||||
filters: dict[str, Any] = {"user_id": user_id or "gateway_user"}
|
|
||||||
if session_id:
|
|
||||||
filters["session_id"] = session_id
|
|
||||||
return filters
|
|
||||||
|
|
||||||
def _timestamp_ms(self, payload: dict[str, Any]) -> int:
|
|
||||||
trace = payload.get("trace") if isinstance(payload.get("trace"), dict) else {}
|
|
||||||
timestamp = trace.get("timestamp") or payload.get("created_at")
|
|
||||||
if timestamp:
|
|
||||||
return self._datetime_to_ms(timestamp)
|
|
||||||
return int(datetime.now(timezone.utc).timestamp() * 1000)
|
|
||||||
|
|
||||||
def _datetime_to_ms(self, value: Any) -> int:
|
|
||||||
if isinstance(value, (int, float)):
|
|
||||||
return int(value if value > 1_000_000_000_000 else value * 1000)
|
|
||||||
if isinstance(value, str):
|
|
||||||
text = value.replace("Z", "+00:00")
|
|
||||||
try:
|
|
||||||
return int(datetime.fromisoformat(text).timestamp() * 1000)
|
|
||||||
except ValueError:
|
|
||||||
return int(datetime.now(timezone.utc).timestamp() * 1000)
|
|
||||||
if isinstance(value, datetime):
|
|
||||||
return int(value.timestamp() * 1000)
|
|
||||||
return int(datetime.now(timezone.utc).timestamp() * 1000)
|
|
||||||
|
|
||||||
def _everos_role(self, role: str) -> str:
|
|
||||||
if role in {"assistant", "agent"}:
|
|
||||||
return "assistant"
|
|
||||||
if role == "tool":
|
|
||||||
return "assistant"
|
|
||||||
return "user"
|
|
||||||
@ -1,158 +0,0 @@
|
|||||||
"""LLM helpers for Memory Gateway summaries."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
import httpx
|
|
||||||
|
|
||||||
from .config import get_config
|
|
||||||
|
|
||||||
|
|
||||||
class LLMConfigurationError(RuntimeError):
|
|
||||||
"""Raised when LLM summarization is requested but not configured."""
|
|
||||||
|
|
||||||
|
|
||||||
class LLMSummaryError(RuntimeError):
|
|
||||||
"""Raised when the LLM response cannot be used."""
|
|
||||||
|
|
||||||
|
|
||||||
def _llm_settings() -> dict[str, Any]:
|
|
||||||
config = get_config()
|
|
||||||
llm_config = getattr(config, "llm", None)
|
|
||||||
|
|
||||||
base_url = (
|
|
||||||
os.environ.get("MEMORY_GATEWAY_LLM_BASE_URL")
|
|
||||||
or os.environ.get("OPENAI_BASE_URL")
|
|
||||||
or getattr(llm_config, "base_url", "")
|
|
||||||
or "https://api.openai.com/v1"
|
|
||||||
).rstrip("/")
|
|
||||||
api_key = (
|
|
||||||
os.environ.get("MEMORY_GATEWAY_LLM_API_KEY")
|
|
||||||
or os.environ.get("OPENAI_API_KEY")
|
|
||||||
or getattr(llm_config, "api_key", "")
|
|
||||||
)
|
|
||||||
model = (
|
|
||||||
os.environ.get("MEMORY_GATEWAY_LLM_MODEL")
|
|
||||||
or os.environ.get("OPENAI_MODEL")
|
|
||||||
or getattr(llm_config, "model", "")
|
|
||||||
)
|
|
||||||
timeout = int(os.environ.get("MEMORY_GATEWAY_LLM_TIMEOUT") or getattr(llm_config, "timeout", 60))
|
|
||||||
max_input_chars = int(os.environ.get("MEMORY_GATEWAY_LLM_MAX_INPUT_CHARS") or getattr(llm_config, "max_input_chars", 24000))
|
|
||||||
return {
|
|
||||||
"base_url": base_url,
|
|
||||||
"api_key": api_key,
|
|
||||||
"model": model,
|
|
||||||
"timeout": timeout,
|
|
||||||
"max_input_chars": max_input_chars,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _extract_json(text: str) -> dict[str, Any]:
|
|
||||||
text = text.strip()
|
|
||||||
if text.startswith("```"):
|
|
||||||
text = re.sub(r"^```(?:json)?\s*", "", text)
|
|
||||||
text = re.sub(r"\s*```$", "", text)
|
|
||||||
try:
|
|
||||||
return json.loads(text)
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
match = re.search(r"\{.*\}", text, flags=re.S)
|
|
||||||
if not match:
|
|
||||||
raise LLMSummaryError("LLM did not return JSON") from None
|
|
||||||
return json.loads(match.group(0))
|
|
||||||
|
|
||||||
|
|
||||||
def _coerce_string_list(value: Any, limit: int = 12) -> list[str]:
|
|
||||||
if not isinstance(value, list):
|
|
||||||
return []
|
|
||||||
items: list[str] = []
|
|
||||||
for item in value:
|
|
||||||
if item is None:
|
|
||||||
continue
|
|
||||||
text = str(item).strip()
|
|
||||||
if text and text not in items:
|
|
||||||
items.append(text[:300])
|
|
||||||
if len(items) >= limit:
|
|
||||||
break
|
|
||||||
return items
|
|
||||||
|
|
||||||
|
|
||||||
async def summarize_with_llm(
|
|
||||||
content: str,
|
|
||||||
*,
|
|
||||||
title: str | None = None,
|
|
||||||
summary_hint: str | None = None,
|
|
||||||
tags: list[str] | None = None,
|
|
||||||
max_summary_chars: int = 800,
|
|
||||||
purpose: str = "generic knowledge memory",
|
|
||||||
) -> dict[str, Any]:
|
|
||||||
"""Summarize content using an OpenAI-compatible chat completions API."""
|
|
||||||
settings = _llm_settings()
|
|
||||||
if not settings["model"]:
|
|
||||||
raise LLMConfigurationError("LLM model is not configured. Set MEMORY_GATEWAY_LLM_MODEL or llm.model.")
|
|
||||||
if not settings["api_key"] and not settings["base_url"].startswith(("http://127.0.0.1", "http://localhost")):
|
|
||||||
raise LLMConfigurationError("LLM API key is not configured. Set MEMORY_GATEWAY_LLM_API_KEY or OPENAI_API_KEY.")
|
|
||||||
|
|
||||||
trimmed = content[: settings["max_input_chars"]]
|
|
||||||
tag_text = ", ".join(tags or [])
|
|
||||||
system_prompt = (
|
|
||||||
"You are a precise knowledge curator. Summarize input into reusable memory. "
|
|
||||||
"Return only valid JSON with these keys: title, summary, key_points, tags. "
|
|
||||||
"summary must be concise but specific; key_points must be reusable, evidence-based bullets. "
|
|
||||||
"Do not invent facts not present in the input. Preserve important identifiers, paths, URLs, IPs, IDs, and verdicts."
|
|
||||||
)
|
|
||||||
user_prompt = f"""
|
|
||||||
Purpose: {purpose}
|
|
||||||
Provided title: {title or ''}
|
|
||||||
Provided summary hint: {summary_hint or ''}
|
|
||||||
Provided tags: {tag_text}
|
|
||||||
Max summary characters: {max_summary_chars}
|
|
||||||
|
|
||||||
Content:
|
|
||||||
{trimmed}
|
|
||||||
""".strip()
|
|
||||||
|
|
||||||
headers = {"Content-Type": "application/json"}
|
|
||||||
if settings["api_key"]:
|
|
||||||
headers["Authorization"] = f"Bearer {settings['api_key']}"
|
|
||||||
|
|
||||||
payload = {
|
|
||||||
"model": settings["model"],
|
|
||||||
"messages": [
|
|
||||||
{"role": "system", "content": system_prompt},
|
|
||||||
{"role": "user", "content": user_prompt},
|
|
||||||
],
|
|
||||||
"temperature": 0.2,
|
|
||||||
"response_format": {"type": "json_object"},
|
|
||||||
}
|
|
||||||
async with httpx.AsyncClient(timeout=settings["timeout"]) as client:
|
|
||||||
response = await client.post(f"{settings['base_url']}/chat/completions", headers=headers, json=payload)
|
|
||||||
response.raise_for_status()
|
|
||||||
data = response.json()
|
|
||||||
|
|
||||||
try:
|
|
||||||
content_text = data["choices"][0]["message"]["content"]
|
|
||||||
except (KeyError, IndexError, TypeError) as exc:
|
|
||||||
raise LLMSummaryError(f"Unexpected LLM response shape: {data}") from exc
|
|
||||||
|
|
||||||
parsed = _extract_json(content_text)
|
|
||||||
merged_tags = []
|
|
||||||
for tag in [*(tags or []), *_coerce_string_list(parsed.get("tags"), limit=8)]:
|
|
||||||
tag = str(tag).strip()
|
|
||||||
if tag and tag not in merged_tags:
|
|
||||||
merged_tags.append(tag)
|
|
||||||
|
|
||||||
summary = str(parsed.get("summary") or "").strip()
|
|
||||||
return {
|
|
||||||
"title": str(parsed.get("title") or title or "Untitled summary").strip()[:160],
|
|
||||||
"summary": summary[:max(120, max_summary_chars)],
|
|
||||||
"key_points": _coerce_string_list(parsed.get("key_points"), limit=10),
|
|
||||||
"tags": merged_tags,
|
|
||||||
"llm": {
|
|
||||||
"provider": "openai-compatible",
|
|
||||||
"base_url": settings["base_url"],
|
|
||||||
"model": settings["model"],
|
|
||||||
},
|
|
||||||
}
|
|
||||||
@ -1,135 +0,0 @@
|
|||||||
"""MCP tool definitions for the generic Memory Gateway contract.
|
|
||||||
|
|
||||||
The legacy MCP endpoint in server.py remains available. These definitions are
|
|
||||||
the target v1 tool contract for Nanobot, Hermes Agent, OpenClaw, and other
|
|
||||||
agent frameworks.
|
|
||||||
"""
|
|
||||||
|
|
||||||
MEMORY_GATEWAY_MCP_TOOLS = [
|
|
||||||
{
|
|
||||||
"name": "memory_search",
|
|
||||||
"description": "Search accessible memories with user/agent/workspace/session isolation.",
|
|
||||||
"inputSchema": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"user_id": {"type": "string"},
|
|
||||||
"agent_id": {"type": "string"},
|
|
||||||
"workspace_id": {"type": "string"},
|
|
||||||
"session_id": {"type": "string"},
|
|
||||||
"query": {"type": "string"},
|
|
||||||
"namespaces": {"type": "array", "items": {"type": "string"}},
|
|
||||||
"limit": {"type": "integer", "default": 10},
|
|
||||||
},
|
|
||||||
"required": ["user_id", "query"],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "memory_upsert",
|
|
||||||
"description": "Create or update a memory record after ACL and namespace routing.",
|
|
||||||
"inputSchema": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"user_id": {"type": "string"},
|
|
||||||
"agent_id": {"type": "string"},
|
|
||||||
"workspace_id": {"type": "string"},
|
|
||||||
"session_id": {"type": "string"},
|
|
||||||
"namespace": {"type": "string"},
|
|
||||||
"memory_type": {"type": "string"},
|
|
||||||
"content": {"type": "string"},
|
|
||||||
"summary": {"type": "string"},
|
|
||||||
"tags": {"type": "array", "items": {"type": "string"}},
|
|
||||||
"importance": {"type": "number"},
|
|
||||||
"confidence": {"type": "number"},
|
|
||||||
"visibility": {"type": "string"},
|
|
||||||
},
|
|
||||||
"required": ["user_id", "content"],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "memory_append_episode",
|
|
||||||
"description": "Append temporary episode/session memory without automatically promoting it.",
|
|
||||||
"inputSchema": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"user_id": {"type": "string"},
|
|
||||||
"agent_id": {"type": "string"},
|
|
||||||
"workspace_id": {"type": "string"},
|
|
||||||
"session_id": {"type": "string"},
|
|
||||||
"content": {"type": "string"},
|
|
||||||
"events": {"type": "array", "items": {"type": "object"}},
|
|
||||||
"tags": {"type": "array", "items": {"type": "string"}},
|
|
||||||
},
|
|
||||||
"required": ["user_id", "session_id", "content"],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "memory_commit_session",
|
|
||||||
"description": "Promote selected session memories into long-term memory via consolidation.",
|
|
||||||
"inputSchema": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"user_id": {"type": "string"},
|
|
||||||
"agent_id": {"type": "string"},
|
|
||||||
"workspace_id": {"type": "string"},
|
|
||||||
"session_id": {"type": "string"},
|
|
||||||
"promote": {"type": "boolean", "default": True},
|
|
||||||
"min_importance": {"type": "number", "default": 0.6},
|
|
||||||
},
|
|
||||||
"required": ["user_id", "session_id"],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "memory_get_profile",
|
|
||||||
"description": "Get the effective user profile memory.",
|
|
||||||
"inputSchema": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {"user_id": {"type": "string"}},
|
|
||||||
"required": ["user_id"],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "memory_list_namespaces",
|
|
||||||
"description": "List namespaces visible to the current user/agent/workspace context.",
|
|
||||||
"inputSchema": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"user_id": {"type": "string"},
|
|
||||||
"agent_id": {"type": "string"},
|
|
||||||
"workspace_id": {"type": "string"},
|
|
||||||
"session_id": {"type": "string"},
|
|
||||||
},
|
|
||||||
"required": ["user_id"],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "memory_delete",
|
|
||||||
"description": "Delete or archive a memory record if the caller has access.",
|
|
||||||
"inputSchema": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"user_id": {"type": "string"},
|
|
||||||
"agent_id": {"type": "string"},
|
|
||||||
"workspace_id": {"type": "string"},
|
|
||||||
"memory_id": {"type": "string"},
|
|
||||||
},
|
|
||||||
"required": ["user_id", "memory_id"],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "memory_feedback",
|
|
||||||
"description": "Attach quality feedback to a memory record for pruning/merge decisions.",
|
|
||||||
"inputSchema": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"user_id": {"type": "string"},
|
|
||||||
"agent_id": {"type": "string"},
|
|
||||||
"workspace_id": {"type": "string"},
|
|
||||||
"memory_id": {"type": "string"},
|
|
||||||
"feedback": {"type": "string"},
|
|
||||||
"comment": {"type": "string"},
|
|
||||||
},
|
|
||||||
"required": ["user_id", "memory_id", "feedback"],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
]
|
|
||||||
|
|
||||||
@ -1,115 +0,0 @@
|
|||||||
"""Namespace construction and access checks for Memory Gateway."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
|
|
||||||
from .schemas import AccessContext, MemoryRecord, NamespaceInfo, Visibility
|
|
||||||
|
|
||||||
|
|
||||||
def user_profile_namespace(user_id: str) -> str:
|
|
||||||
return f"user/{user_id}/profile"
|
|
||||||
|
|
||||||
|
|
||||||
def user_preferences_namespace(user_id: str) -> str:
|
|
||||||
return f"user/{user_id}/preferences"
|
|
||||||
|
|
||||||
|
|
||||||
def user_long_term_namespace(user_id: str) -> str:
|
|
||||||
return f"user/{user_id}/long_term"
|
|
||||||
|
|
||||||
|
|
||||||
def agent_memory_namespace(agent_id: str) -> str:
|
|
||||||
return f"agent/{agent_id}/memory"
|
|
||||||
|
|
||||||
|
|
||||||
def workspace_shared_namespace(workspace_id: str) -> str:
|
|
||||||
return f"workspace/{workspace_id}/shared"
|
|
||||||
|
|
||||||
|
|
||||||
def session_episodic_namespace(session_id: str) -> str:
|
|
||||||
return f"session/{session_id}/episodic"
|
|
||||||
|
|
||||||
|
|
||||||
def global_public_namespace() -> str:
|
|
||||||
return "global/public"
|
|
||||||
|
|
||||||
|
|
||||||
def default_namespace_for_context(ctx: AccessContext, visibility: Visibility) -> str:
|
|
||||||
if visibility == Visibility.AGENT_ONLY and ctx.agent_id:
|
|
||||||
return agent_memory_namespace(ctx.agent_id)
|
|
||||||
if visibility == Visibility.WORKSPACE_SHARED and ctx.workspace_id:
|
|
||||||
return workspace_shared_namespace(ctx.workspace_id)
|
|
||||||
if ctx.session_id:
|
|
||||||
return session_episodic_namespace(ctx.session_id)
|
|
||||||
return user_long_term_namespace(ctx.user_id)
|
|
||||||
|
|
||||||
|
|
||||||
def can_access_memory(ctx: AccessContext, memory: MemoryRecord) -> bool:
|
|
||||||
if memory.expires_at and memory.expires_at <= datetime.now(timezone.utc):
|
|
||||||
return False
|
|
||||||
if memory.visibility == Visibility.GLOBAL:
|
|
||||||
return True
|
|
||||||
if memory.visibility == Visibility.PRIVATE:
|
|
||||||
return memory.user_id == ctx.user_id
|
|
||||||
if memory.visibility == Visibility.AGENT_ONLY:
|
|
||||||
return memory.user_id == ctx.user_id and memory.agent_id == ctx.agent_id
|
|
||||||
if memory.visibility == Visibility.WORKSPACE_SHARED:
|
|
||||||
return memory.workspace_id is not None and memory.workspace_id == ctx.workspace_id
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def visible_namespaces(ctx: AccessContext) -> list[NamespaceInfo]:
|
|
||||||
namespaces = [
|
|
||||||
NamespaceInfo(
|
|
||||||
namespace=user_profile_namespace(ctx.user_id),
|
|
||||||
owner_user_id=ctx.user_id,
|
|
||||||
visibility=Visibility.PRIVATE,
|
|
||||||
description="用户 profile 与稳定偏好",
|
|
||||||
),
|
|
||||||
NamespaceInfo(
|
|
||||||
namespace=user_preferences_namespace(ctx.user_id),
|
|
||||||
owner_user_id=ctx.user_id,
|
|
||||||
visibility=Visibility.PRIVATE,
|
|
||||||
description="用户显式偏好",
|
|
||||||
),
|
|
||||||
NamespaceInfo(
|
|
||||||
namespace=user_long_term_namespace(ctx.user_id),
|
|
||||||
owner_user_id=ctx.user_id,
|
|
||||||
visibility=Visibility.PRIVATE,
|
|
||||||
description="用户长期记忆",
|
|
||||||
),
|
|
||||||
NamespaceInfo(
|
|
||||||
namespace=global_public_namespace(),
|
|
||||||
visibility=Visibility.GLOBAL,
|
|
||||||
description="全局公开知识",
|
|
||||||
),
|
|
||||||
]
|
|
||||||
if ctx.agent_id:
|
|
||||||
namespaces.append(
|
|
||||||
NamespaceInfo(
|
|
||||||
namespace=agent_memory_namespace(ctx.agent_id),
|
|
||||||
owner_user_id=ctx.user_id,
|
|
||||||
visibility=Visibility.AGENT_ONLY,
|
|
||||||
description="指定 agent 私有经验",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
if ctx.workspace_id:
|
|
||||||
namespaces.append(
|
|
||||||
NamespaceInfo(
|
|
||||||
namespace=workspace_shared_namespace(ctx.workspace_id),
|
|
||||||
owner_user_id=ctx.user_id,
|
|
||||||
visibility=Visibility.WORKSPACE_SHARED,
|
|
||||||
description="workspace / project 共享记忆",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
if ctx.session_id:
|
|
||||||
namespaces.append(
|
|
||||||
NamespaceInfo(
|
|
||||||
namespace=session_episodic_namespace(ctx.session_id),
|
|
||||||
owner_user_id=ctx.user_id,
|
|
||||||
visibility=Visibility.PRIVATE,
|
|
||||||
description="session 临时 episodic memory",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return namespaces
|
|
||||||
|
|
||||||
@ -1,77 +0,0 @@
|
|||||||
"""Obsidian review draft writer."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import re
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from .config import get_config
|
|
||||||
from .schemas import MemoryRecord
|
|
||||||
|
|
||||||
|
|
||||||
def _slugify(value: str, fallback: str) -> str:
|
|
||||||
slug = re.sub(r"[^a-zA-Z0-9\u4e00-\u9fff_-]+", "-", value.lower()).strip("-")
|
|
||||||
slug = re.sub(r"-+", "-", slug)[:80].strip("-")
|
|
||||||
return slug or fallback
|
|
||||||
|
|
||||||
|
|
||||||
def write_review_draft(memory: MemoryRecord, reason: str, conflict_ids: list[str] | None = None) -> Path:
|
|
||||||
config = get_config()
|
|
||||||
review_dir = getattr(config.obsidian, "review_dir", "Reviews/Queue")
|
|
||||||
vault_path = Path(config.obsidian.vault_path)
|
|
||||||
target_dir = vault_path / review_dir
|
|
||||||
target_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
|
|
||||||
title = memory.summary or memory.content[:80] or memory.id
|
|
||||||
filename = f"{_slugify(title, memory.id)}-{memory.id}.md"
|
|
||||||
path = target_dir / filename
|
|
||||||
conflict_ids = conflict_ids or []
|
|
||||||
|
|
||||||
content = "\n".join(
|
|
||||||
[
|
|
||||||
"---",
|
|
||||||
"type: memory_review",
|
|
||||||
f"memory_id: {memory.id}",
|
|
||||||
f"user_id: {memory.user_id}",
|
|
||||||
f"agent_id: {memory.agent_id or ''}",
|
|
||||||
f"workspace_id: {memory.workspace_id or ''}",
|
|
||||||
f"namespace: {memory.namespace}",
|
|
||||||
f"visibility: {memory.visibility.value}",
|
|
||||||
f"importance: {memory.importance}",
|
|
||||||
f"confidence: {memory.confidence}",
|
|
||||||
f"reason: {reason}",
|
|
||||||
f"created_at: {datetime.now(timezone.utc).isoformat()}",
|
|
||||||
"tags:",
|
|
||||||
" - memory/review",
|
|
||||||
" - source/everos",
|
|
||||||
"---",
|
|
||||||
"",
|
|
||||||
f"# Memory Review - {title}",
|
|
||||||
"",
|
|
||||||
"## Candidate",
|
|
||||||
"",
|
|
||||||
memory.content,
|
|
||||||
"",
|
|
||||||
"## Summary",
|
|
||||||
"",
|
|
||||||
memory.summary or "",
|
|
||||||
"",
|
|
||||||
"## Proposed Action",
|
|
||||||
"",
|
|
||||||
"- [ ] Accept",
|
|
||||||
"- [ ] Edit",
|
|
||||||
"- [ ] Reject",
|
|
||||||
"- [ ] Merge",
|
|
||||||
"- [ ] Archive",
|
|
||||||
"",
|
|
||||||
"## Conflict IDs",
|
|
||||||
"",
|
|
||||||
"\n".join(f"- {memory_id}" for memory_id in conflict_ids) if conflict_ids else "- none",
|
|
||||||
"",
|
|
||||||
"## Notes",
|
|
||||||
"",
|
|
||||||
]
|
|
||||||
)
|
|
||||||
path.write_text(content, encoding="utf-8")
|
|
||||||
return path
|
|
||||||
|
|
||||||
@ -1,25 +0,0 @@
|
|||||||
"""Human-review backend skeleton for Memory Gateway v2.
|
|
||||||
|
|
||||||
Obsidian remains a human-in-the-loop review backend only. This skeleton does
|
|
||||||
not write files or call external APIs; it preserves the adapter contract until
|
|
||||||
the review draft integration is explicitly designed.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
from .backend_contracts import BackendOperation, BackendResultStatus, BackendWriteResult
|
|
||||||
from .schemas_v2 import BackendType
|
|
||||||
|
|
||||||
|
|
||||||
class ObsidianReviewClient:
|
|
||||||
def create_review_draft_v2(self, payload: dict[str, Any]) -> BackendWriteResult:
|
|
||||||
"""Return a skipped review-draft result until the real adapter exists."""
|
|
||||||
return BackendWriteResult(
|
|
||||||
backend_type=BackendType.OBSIDIAN,
|
|
||||||
operation=BackendOperation.CREATE_REVIEW_DRAFT,
|
|
||||||
status=BackendResultStatus.SKIPPED,
|
|
||||||
native_id=payload.get("event_id") or payload.get("gateway_id"),
|
|
||||||
retryable=False,
|
|
||||||
metadata={"reason": "obsidian_review_adapter_not_configured"},
|
|
||||||
)
|
|
||||||
@ -1,522 +0,0 @@
|
|||||||
"""OpenViking client wrapper used by Memory Gateway."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import logging
|
|
||||||
import mimetypes
|
|
||||||
import tempfile
|
|
||||||
from json import JSONDecodeError
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any, Optional
|
|
||||||
|
|
||||||
import httpx
|
|
||||||
|
|
||||||
from .backend_contracts import BackendCommitResult, BackendOperation, BackendResultStatus, BackendRetrieveResult, BackendWriteResult
|
|
||||||
from .backend_normalization import (
|
|
||||||
map_backend_error_to_retryable,
|
|
||||||
normalize_openviking_commit_response,
|
|
||||||
normalize_openviking_ingest_response,
|
|
||||||
normalize_openviking_retrieve_response,
|
|
||||||
)
|
|
||||||
from .config import get_config
|
|
||||||
from .schemas_v2 import BackendType
|
|
||||||
from .types import MemoryEntry, ResourceEntry, SearchResult
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
class OpenVikingClient:
|
|
||||||
"""Thin async client for the OpenViking HTTP API."""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
base_url: Optional[str] = None,
|
|
||||||
api_key: Optional[str] = None,
|
|
||||||
timeout: int | None = None,
|
|
||||||
account: str = "default",
|
|
||||||
user: str = "default",
|
|
||||||
enabled: bool | None = None,
|
|
||||||
mode: str | None = None,
|
|
||||||
verify_ssl: bool | None = None,
|
|
||||||
ingest_path: str | None = None,
|
|
||||||
transport: httpx.AsyncBaseTransport | None = None,
|
|
||||||
):
|
|
||||||
self.config = get_config()
|
|
||||||
self.base_url = base_url if base_url is not None else self.config.openviking.url
|
|
||||||
self.api_key = api_key if api_key is not None else (self.config.openviking.api_key or "your-secret-root-key")
|
|
||||||
self.timeout = timeout if timeout is not None else self.config.openviking.timeout
|
|
||||||
self.account = account
|
|
||||||
self.user = user
|
|
||||||
self.enabled = self.config.openviking.enabled if enabled is None else enabled
|
|
||||||
self.mode = mode or self.config.openviking.mode
|
|
||||||
self.verify_ssl = self.config.openviking.verify_ssl if verify_ssl is None else verify_ssl
|
|
||||||
self.ingest_path = ingest_path or self.config.openviking.ingest_path
|
|
||||||
self.transport = transport
|
|
||||||
self._client: Optional[httpx.AsyncClient] = None
|
|
||||||
|
|
||||||
def _get_headers(self) -> dict[str, str]:
|
|
||||||
headers = {}
|
|
||||||
if self.api_key:
|
|
||||||
headers["X-API-Key"] = self.api_key
|
|
||||||
headers["Authorization"] = f"Bearer {self.api_key}"
|
|
||||||
headers["X-OpenViking-Account"] = self.account
|
|
||||||
headers["X-OpenViking-User"] = self.user
|
|
||||||
return headers
|
|
||||||
|
|
||||||
async def _get_client(self) -> httpx.AsyncClient:
|
|
||||||
if self._client is None:
|
|
||||||
self._client = httpx.AsyncClient(
|
|
||||||
base_url=self.base_url,
|
|
||||||
headers=self._get_headers(),
|
|
||||||
timeout=self.timeout,
|
|
||||||
verify=self.verify_ssl,
|
|
||||||
transport=self.transport,
|
|
||||||
)
|
|
||||||
return self._client
|
|
||||||
|
|
||||||
async def close(self):
|
|
||||||
if self._client:
|
|
||||||
await self._client.aclose()
|
|
||||||
self._client = None
|
|
||||||
|
|
||||||
async def health_check(self) -> dict[str, Any]:
|
|
||||||
client = await self._get_client()
|
|
||||||
try:
|
|
||||||
response = await client.get("/health")
|
|
||||||
response.raise_for_status()
|
|
||||||
return response.json()
|
|
||||||
except httpx.HTTPError as e:
|
|
||||||
logger.error(f"OpenViking 健康检查失败: {e}")
|
|
||||||
return {"status": "error", "message": str(e)}
|
|
||||||
|
|
||||||
async def ingest_conversation_turn(self, payload: dict[str, Any]) -> BackendWriteResult:
|
|
||||||
"""v2 adapter placeholder for OpenViking session archive ingestion.
|
|
||||||
|
|
||||||
Mapping spec: `backend_adapter_mapping.AdapterMappingSpec` maps
|
|
||||||
OpenViking ingest_turn to this method and requires BackendWriteResult.
|
|
||||||
Payloads must contain only control-plane fields; conversation content
|
|
||||||
is not persisted by the Gateway control-plane store.
|
|
||||||
|
|
||||||
TODO(v2): bind this to OpenViking's stable session/message archive API
|
|
||||||
once that contract is finalized. Until then the gateway records a
|
|
||||||
skipped backend ref instead of inventing an unstable HTTP contract.
|
|
||||||
"""
|
|
||||||
runtime_payload = self._build_ingest_payload(payload)
|
|
||||||
if self._use_real_api:
|
|
||||||
return await self._ingest_conversation_turn_real(runtime_payload)
|
|
||||||
raw = {
|
|
||||||
"status": "skipped",
|
|
||||||
"session_id": runtime_payload.get("session_id"),
|
|
||||||
"uri": f"viking://sessions/{runtime_payload.get('session_id')}",
|
|
||||||
"metadata": {
|
|
||||||
"reason": "openviking_v2_ingest_adapter_not_configured",
|
|
||||||
"schema_version": "openviking.fixture.ingest.v2",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
return self._normalize_ingest_response(raw)
|
|
||||||
|
|
||||||
@property
|
|
||||||
def _use_real_api(self) -> bool:
|
|
||||||
# Real ingest is strictly gated by mode=real. The legacy `enabled`
|
|
||||||
# field is retained for config compatibility, but must not trigger
|
|
||||||
# network traffic by itself.
|
|
||||||
return self.mode == "real"
|
|
||||||
|
|
||||||
async def _ingest_conversation_turn_real(self, runtime_payload: dict[str, Any]) -> BackendWriteResult:
|
|
||||||
if not self.base_url:
|
|
||||||
return self._failed_ingest_result(
|
|
||||||
error_code="config_error",
|
|
||||||
error_message="OpenViking real ingest is enabled but base_url is missing",
|
|
||||||
retryable=False,
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
client = await self._get_client()
|
|
||||||
response = await client.post(
|
|
||||||
self._format_ingest_path(runtime_payload),
|
|
||||||
json=runtime_payload,
|
|
||||||
)
|
|
||||||
if response.status_code >= 400:
|
|
||||||
return self._failed_ingest_result(
|
|
||||||
error_code=f"http_{response.status_code}",
|
|
||||||
error_message=f"OpenViking ingest failed with HTTP {response.status_code}",
|
|
||||||
retryable=self._map_error(response),
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
raw = response.json()
|
|
||||||
except (JSONDecodeError, ValueError):
|
|
||||||
return self._failed_ingest_result(
|
|
||||||
error_code="invalid_json",
|
|
||||||
error_message="OpenViking ingest returned invalid JSON",
|
|
||||||
retryable=True,
|
|
||||||
)
|
|
||||||
if not isinstance(raw, dict):
|
|
||||||
return self._failed_ingest_result(
|
|
||||||
error_code="unexpected_response",
|
|
||||||
error_message="OpenViking ingest returned an unexpected response shape",
|
|
||||||
retryable=True,
|
|
||||||
)
|
|
||||||
return self._normalize_ingest_response(raw)
|
|
||||||
except httpx.TimeoutException as exc:
|
|
||||||
return self._failed_ingest_result("timeout", self._safe_error_message(exc), retryable=self._map_error(exc))
|
|
||||||
except httpx.RequestError as exc:
|
|
||||||
return self._failed_ingest_result("network_error", self._safe_error_message(exc), retryable=self._map_error(exc))
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
return self._failed_ingest_result("unexpected_error", self._safe_error_message(exc), retryable=self._map_error(exc))
|
|
||||||
|
|
||||||
async def commit_session_v2(self, payload: dict[str, Any]) -> BackendCommitResult:
|
|
||||||
"""v2 adapter placeholder for OpenViking session commit.
|
|
||||||
|
|
||||||
Mapping spec: commit_session returns BackendCommitResult and should
|
|
||||||
produce a native session/archive ref once the real API is stable.
|
|
||||||
"""
|
|
||||||
runtime_payload = self._build_commit_payload(payload)
|
|
||||||
raw = {
|
|
||||||
"status": "success",
|
|
||||||
"session_id": runtime_payload.get("session_id"),
|
|
||||||
"metadata": {
|
|
||||||
"reason": "openviking_v2_commit_fixture",
|
|
||||||
"schema_version": "openviking.fixture.commit.v2",
|
|
||||||
},
|
|
||||||
"result": {
|
|
||||||
"refs": [
|
|
||||||
{
|
|
||||||
"type": "session_summary",
|
|
||||||
"id": f"ov_session_summary:{runtime_payload.get('session_id')}",
|
|
||||||
"uri": f"viking://sessions/{runtime_payload.get('session_id')}/summary",
|
|
||||||
"metadata": {"schema_version": "openviking.fixture.ref.v2"},
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
}
|
|
||||||
return self._normalize_commit_response(raw)
|
|
||||||
|
|
||||||
async def retrieve_context_v2(self, payload: dict[str, Any]) -> BackendRetrieveResult:
|
|
||||||
"""
|
|
||||||
Calls OpenViking native API to retrieve context.
|
|
||||||
Uses POST /search
|
|
||||||
"""
|
|
||||||
if not self._use_real_api:
|
|
||||||
return BackendRetrieveResult(
|
|
||||||
backend_type=BackendType.OPENVIKING,
|
|
||||||
operation=BackendOperation.RETRIEVE_CONTEXT,
|
|
||||||
status=BackendResultStatus.SKIPPED,
|
|
||||||
items=[],
|
|
||||||
metadata={"reason": "openviking_retrieve_requires_real_mode"},
|
|
||||||
)
|
|
||||||
|
|
||||||
query = payload.get("query", "")
|
|
||||||
session_id = payload.get("session_id")
|
|
||||||
|
|
||||||
request_data = {"query": query, "limit": 10}
|
|
||||||
if session_id:
|
|
||||||
request_data["session_id"] = session_id
|
|
||||||
|
|
||||||
try:
|
|
||||||
client = await self._get_client()
|
|
||||||
response = await client.post("/api/v1/search/search", json=request_data)
|
|
||||||
|
|
||||||
if response.status_code >= 400:
|
|
||||||
return BackendRetrieveResult(
|
|
||||||
backend_type=BackendType.OPENVIKING,
|
|
||||||
operation=BackendOperation.RETRIEVE_CONTEXT,
|
|
||||||
status=BackendResultStatus.FAILED,
|
|
||||||
items=[],
|
|
||||||
error_code=f"http_{response.status_code}",
|
|
||||||
error_message=f"OpenViking search failed: {response.text}",
|
|
||||||
retryable=False
|
|
||||||
)
|
|
||||||
|
|
||||||
return self._normalize_retrieve_response(response.json())
|
|
||||||
except Exception as exc:
|
|
||||||
return BackendRetrieveResult(
|
|
||||||
backend_type=BackendType.OPENVIKING,
|
|
||||||
operation=BackendOperation.RETRIEVE_CONTEXT,
|
|
||||||
status=BackendResultStatus.FAILED,
|
|
||||||
items=[],
|
|
||||||
error_code="request_error",
|
|
||||||
error_message=str(exc),
|
|
||||||
retryable=True
|
|
||||||
)
|
|
||||||
|
|
||||||
def _build_ingest_payload(self, payload: dict[str, Any]) -> dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Build payload for native OpenViking AddMessageRequest.
|
|
||||||
OpenViking only expects role and content, and maybe metadata.
|
|
||||||
"""
|
|
||||||
return {
|
|
||||||
"role": payload.get("role", "user"),
|
|
||||||
"content": payload.get("content", ""),
|
|
||||||
"metadata": payload.get("metadata", {}),
|
|
||||||
"session_id": payload.get("session_id") # kept so format_ingest_path can use it
|
|
||||||
}
|
|
||||||
|
|
||||||
def _format_ingest_path(self, payload: dict[str, Any]) -> str:
|
|
||||||
session_id = str(payload.get("session_id") or "unknown")
|
|
||||||
return self.ingest_path.format(session_id=session_id)
|
|
||||||
|
|
||||||
def _build_commit_payload(self, payload: dict[str, Any]) -> dict[str, Any]:
|
|
||||||
return dict(payload)
|
|
||||||
|
|
||||||
def _normalize_ingest_response(self, raw: dict[str, Any]) -> BackendWriteResult:
|
|
||||||
return normalize_openviking_ingest_response(raw)
|
|
||||||
|
|
||||||
def _normalize_commit_response(self, raw: dict[str, Any]) -> BackendCommitResult:
|
|
||||||
return normalize_openviking_commit_response(raw)
|
|
||||||
|
|
||||||
def _normalize_retrieve_response(self, raw: dict[str, Any]) -> BackendRetrieveResult:
|
|
||||||
return normalize_openviking_retrieve_response(raw)
|
|
||||||
|
|
||||||
def _map_error(self, exc_or_response: Any) -> bool:
|
|
||||||
status_code = getattr(exc_or_response, "status_code", None)
|
|
||||||
error_code = getattr(exc_or_response, "error_code", None)
|
|
||||||
error_message = str(exc_or_response) if exc_or_response is not None else None
|
|
||||||
return map_backend_error_to_retryable(
|
|
||||||
BackendType.OPENVIKING,
|
|
||||||
status_code=status_code,
|
|
||||||
error_code=error_code,
|
|
||||||
error_message=error_message,
|
|
||||||
)
|
|
||||||
|
|
||||||
def _failed_ingest_result(self, error_code: str, error_message: str, retryable: bool) -> BackendWriteResult:
|
|
||||||
return BackendWriteResult(
|
|
||||||
backend_type=BackendType.OPENVIKING,
|
|
||||||
operation=BackendOperation.INGEST_TURN,
|
|
||||||
status=BackendResultStatus.FAILED,
|
|
||||||
retryable=retryable,
|
|
||||||
error_code=error_code,
|
|
||||||
error_message=error_message,
|
|
||||||
metadata={"error_code": error_code},
|
|
||||||
)
|
|
||||||
|
|
||||||
def _safe_error_message(self, exc: Exception) -> str:
|
|
||||||
return exc.__class__.__name__
|
|
||||||
|
|
||||||
async def search(
|
|
||||||
self,
|
|
||||||
query: str,
|
|
||||||
namespace: Optional[str] = None,
|
|
||||||
limit: Optional[int] = None,
|
|
||||||
uri: Optional[str] = None,
|
|
||||||
) -> SearchResult:
|
|
||||||
"""Semantic search against OpenViking resources/memories."""
|
|
||||||
client = await self._get_client()
|
|
||||||
|
|
||||||
payload: dict[str, Any] = {"query": query}
|
|
||||||
if limit:
|
|
||||||
payload["limit"] = limit
|
|
||||||
|
|
||||||
if uri:
|
|
||||||
payload["target_uri"] = uri
|
|
||||||
elif namespace:
|
|
||||||
payload["target_uri"] = f"viking://{namespace}"
|
|
||||||
|
|
||||||
try:
|
|
||||||
response = await client.post("/api/v1/search/search", json=payload)
|
|
||||||
response.raise_for_status()
|
|
||||||
data = response.json()
|
|
||||||
|
|
||||||
if data.get("status") != "ok":
|
|
||||||
logger.warning(f"搜索返回错误: {data.get('error')}")
|
|
||||||
return SearchResult(results=[], total=0)
|
|
||||||
|
|
||||||
result = data.get("result", {})
|
|
||||||
memories = result.get("memories", [])
|
|
||||||
resources = result.get("resources", [])
|
|
||||||
|
|
||||||
all_results = []
|
|
||||||
for m in memories + resources:
|
|
||||||
all_results.append(
|
|
||||||
{
|
|
||||||
"uri": m.get("uri"),
|
|
||||||
"abstract": m.get("abstract"),
|
|
||||||
"score": m.get("score"),
|
|
||||||
"context_type": m.get("context_type"),
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
return SearchResult(results=all_results, total=result.get("total", len(all_results)))
|
|
||||||
except httpx.HTTPError as e:
|
|
||||||
logger.error(f"搜索失败: {e}")
|
|
||||||
return SearchResult(results=[], total=0)
|
|
||||||
|
|
||||||
async def add_memory(
|
|
||||||
self,
|
|
||||||
content: str,
|
|
||||||
namespace: Optional[str] = None,
|
|
||||||
memory_type: str = "general",
|
|
||||||
) -> dict[str, Any]:
|
|
||||||
"""Add memory via session commit flow."""
|
|
||||||
client = await self._get_client()
|
|
||||||
ns = namespace or self.config.memory.default_namespace or "user/default/memories"
|
|
||||||
|
|
||||||
try:
|
|
||||||
response = await client.post("/api/v1/sessions")
|
|
||||||
response.raise_for_status()
|
|
||||||
session_data = response.json()
|
|
||||||
|
|
||||||
if session_data.get("status") != "ok":
|
|
||||||
return session_data
|
|
||||||
|
|
||||||
session_id = session_data["result"]["session_id"]
|
|
||||||
message_response = await client.post(
|
|
||||||
f"/api/v1/sessions/{session_id}/messages",
|
|
||||||
json={
|
|
||||||
"role": "user",
|
|
||||||
"content": f"[{ns}/{memory_type}] {content}",
|
|
||||||
},
|
|
||||||
)
|
|
||||||
message_response.raise_for_status()
|
|
||||||
commit_response = await client.post(f"/api/v1/sessions/{session_id}/commit")
|
|
||||||
commit_response.raise_for_status()
|
|
||||||
return commit_response.json()
|
|
||||||
except httpx.HTTPError as e:
|
|
||||||
logger.error(f"添加记忆失败: {e}")
|
|
||||||
raise
|
|
||||||
|
|
||||||
async def _upload_temp_file(self, file_path: str | Path) -> str:
|
|
||||||
client = await self._get_client()
|
|
||||||
file_path = Path(file_path)
|
|
||||||
mime_type = mimetypes.guess_type(file_path.name)[0] or "application/octet-stream"
|
|
||||||
|
|
||||||
with file_path.open("rb") as f:
|
|
||||||
response = await client.post(
|
|
||||||
"/api/v1/resources/temp_upload",
|
|
||||||
files={"file": (file_path.name, f, mime_type)},
|
|
||||||
)
|
|
||||||
response.raise_for_status()
|
|
||||||
data = response.json()
|
|
||||||
result = data.get("result", {})
|
|
||||||
if "temp_path" in result:
|
|
||||||
return result["temp_path"]
|
|
||||||
if "temp_file_id" in result:
|
|
||||||
return result["temp_file_id"]
|
|
||||||
raise KeyError(f"Unexpected temp upload response: {data}")
|
|
||||||
|
|
||||||
async def add_resource(
|
|
||||||
self,
|
|
||||||
uri: str,
|
|
||||||
content: str,
|
|
||||||
resource_type: str = "text",
|
|
||||||
wait: bool = False,
|
|
||||||
) -> dict[str, Any]:
|
|
||||||
"""Add a text/json resource by uploading a temporary file first.
|
|
||||||
|
|
||||||
OpenViking HTTP API does not accept raw `uri + content` directly. The
|
|
||||||
client must upload a temp file and then create the resource with `to`.
|
|
||||||
"""
|
|
||||||
client = await self._get_client()
|
|
||||||
suffix_map = {
|
|
||||||
"json": ".json",
|
|
||||||
"text": ".txt",
|
|
||||||
"markdown": ".md",
|
|
||||||
"md": ".md",
|
|
||||||
}
|
|
||||||
suffix = suffix_map.get(resource_type, ".txt")
|
|
||||||
|
|
||||||
with tempfile.NamedTemporaryFile("w", encoding="utf-8", suffix=suffix, delete=False) as tmp:
|
|
||||||
tmp.write(content)
|
|
||||||
tmp_path = Path(tmp.name)
|
|
||||||
|
|
||||||
try:
|
|
||||||
temp_ref = await self._upload_temp_file(tmp_path)
|
|
||||||
payload = {
|
|
||||||
"temp_path": temp_ref,
|
|
||||||
"to": uri,
|
|
||||||
"wait": wait,
|
|
||||||
"strict": False,
|
|
||||||
}
|
|
||||||
response = await client.post("/api/v1/resources", json=payload)
|
|
||||||
if response.status_code >= 400:
|
|
||||||
logger.error("添加资源失败响应: %s", response.text)
|
|
||||||
response.raise_for_status()
|
|
||||||
return response.json()
|
|
||||||
except httpx.HTTPError as e:
|
|
||||||
logger.error(f"添加资源失败: {e}")
|
|
||||||
raise
|
|
||||||
finally:
|
|
||||||
tmp_path.unlink(missing_ok=True)
|
|
||||||
|
|
||||||
async def list_memories(
|
|
||||||
self,
|
|
||||||
namespace: Optional[str] = None,
|
|
||||||
memory_type: Optional[str] = None,
|
|
||||||
limit: Optional[int] = None,
|
|
||||||
) -> list[MemoryEntry]:
|
|
||||||
client = await self._get_client()
|
|
||||||
|
|
||||||
ns = namespace or "user/default/memories"
|
|
||||||
if memory_type:
|
|
||||||
ns = f"{ns}/{memory_type}"
|
|
||||||
|
|
||||||
try:
|
|
||||||
response = await client.post(
|
|
||||||
"/api/v1/search/search",
|
|
||||||
json={"query": "", "target_uri": f"viking://{ns}", "limit": limit or 10},
|
|
||||||
)
|
|
||||||
response.raise_for_status()
|
|
||||||
data = response.json()
|
|
||||||
|
|
||||||
if data.get("status") == "ok":
|
|
||||||
result = data.get("result", {})
|
|
||||||
memories = result.get("memories", [])
|
|
||||||
return [
|
|
||||||
MemoryEntry(
|
|
||||||
id=m.get("uri", ""),
|
|
||||||
content=m.get("abstract", ""),
|
|
||||||
namespace=ns,
|
|
||||||
memory_type=memory_type or "general",
|
|
||||||
)
|
|
||||||
for m in memories
|
|
||||||
]
|
|
||||||
return []
|
|
||||||
except httpx.HTTPError as e:
|
|
||||||
logger.error(f"列出记忆失败: {e}")
|
|
||||||
return []
|
|
||||||
|
|
||||||
async def list_resources(
|
|
||||||
self,
|
|
||||||
namespace: Optional[str] = None,
|
|
||||||
limit: Optional[int] = None,
|
|
||||||
) -> list[ResourceEntry]:
|
|
||||||
client = await self._get_client()
|
|
||||||
|
|
||||||
uri = f"viking://{namespace}" if namespace else "viking://resources"
|
|
||||||
try:
|
|
||||||
response = await client.post(
|
|
||||||
"/api/v1/search/search",
|
|
||||||
json={"query": "", "target_uri": uri, "limit": limit or 10},
|
|
||||||
)
|
|
||||||
response.raise_for_status()
|
|
||||||
data = response.json()
|
|
||||||
|
|
||||||
if data.get("status") == "ok":
|
|
||||||
result = data.get("result", {})
|
|
||||||
resources = result.get("resources", [])
|
|
||||||
return [
|
|
||||||
ResourceEntry(
|
|
||||||
uri=r.get("uri", ""),
|
|
||||||
content=r.get("abstract", ""),
|
|
||||||
resource_type="text",
|
|
||||||
)
|
|
||||||
for r in resources
|
|
||||||
]
|
|
||||||
return []
|
|
||||||
except httpx.HTTPError as e:
|
|
||||||
logger.error(f"列出资源失败: {e}")
|
|
||||||
return []
|
|
||||||
|
|
||||||
|
|
||||||
_client: Optional[OpenVikingClient] = None
|
|
||||||
|
|
||||||
|
|
||||||
async def get_openviking_client() -> OpenVikingClient:
|
|
||||||
global _client
|
|
||||||
if _client is None:
|
|
||||||
_client = OpenVikingClient()
|
|
||||||
return _client
|
|
||||||
|
|
||||||
|
|
||||||
async def close_openviking_client():
|
|
||||||
global _client
|
|
||||||
if _client:
|
|
||||||
await _client.close()
|
|
||||||
_client = None
|
|
||||||
File diff suppressed because it is too large
Load Diff
@ -1,226 +0,0 @@
|
|||||||
"""Core schemas for the generic Memory Gateway v1 API."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from enum import Enum
|
|
||||||
from typing import Any, Literal, Optional
|
|
||||||
from uuid import uuid4
|
|
||||||
|
|
||||||
from pydantic import BaseModel, Field
|
|
||||||
|
|
||||||
|
|
||||||
def utc_now() -> datetime:
|
|
||||||
return datetime.now(timezone.utc)
|
|
||||||
|
|
||||||
|
|
||||||
class Visibility(str, Enum):
|
|
||||||
PRIVATE = "private"
|
|
||||||
AGENT_ONLY = "agent-only"
|
|
||||||
WORKSPACE_SHARED = "workspace-shared"
|
|
||||||
GLOBAL = "global"
|
|
||||||
|
|
||||||
|
|
||||||
class MemoryType(str, Enum):
|
|
||||||
PROFILE = "profile"
|
|
||||||
PREFERENCE = "preference"
|
|
||||||
FACT = "fact"
|
|
||||||
DECISION = "decision"
|
|
||||||
SUMMARY = "summary"
|
|
||||||
EPISODIC = "episodic"
|
|
||||||
PROCEDURE = "procedure"
|
|
||||||
EXPERIENCE = "experience"
|
|
||||||
KNOWLEDGE = "knowledge"
|
|
||||||
|
|
||||||
|
|
||||||
class SourceType(str, Enum):
|
|
||||||
CONVERSATION = "conversation"
|
|
||||||
TASK = "task"
|
|
||||||
AGENT = "agent"
|
|
||||||
OBSIDIAN = "obsidian"
|
|
||||||
OPENVIKING = "openviking"
|
|
||||||
EVEROS = "everos"
|
|
||||||
MANUAL = "manual"
|
|
||||||
|
|
||||||
|
|
||||||
class UserRecord(BaseModel):
|
|
||||||
id: str = Field(default_factory=lambda: f"user_{uuid4().hex[:12]}")
|
|
||||||
display_name: str
|
|
||||||
status: Literal["active", "disabled"] = "active"
|
|
||||||
profile_namespace: Optional[str] = None
|
|
||||||
preferences: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
created_at: datetime = Field(default_factory=utc_now)
|
|
||||||
updated_at: datetime = Field(default_factory=utc_now)
|
|
||||||
|
|
||||||
|
|
||||||
class AgentRecord(BaseModel):
|
|
||||||
id: str
|
|
||||||
name: str
|
|
||||||
framework: str
|
|
||||||
owner_user_id: Optional[str] = None
|
|
||||||
created_at: datetime = Field(default_factory=utc_now)
|
|
||||||
|
|
||||||
|
|
||||||
class WorkspaceRecord(BaseModel):
|
|
||||||
id: str
|
|
||||||
name: str
|
|
||||||
owner_user_id: str
|
|
||||||
member_user_ids: list[str] = Field(default_factory=list)
|
|
||||||
allowed_agent_ids: list[str] = Field(default_factory=list)
|
|
||||||
created_at: datetime = Field(default_factory=utc_now)
|
|
||||||
|
|
||||||
|
|
||||||
class SessionRecord(BaseModel):
|
|
||||||
id: str = Field(default_factory=lambda: f"sess_{uuid4().hex[:12]}")
|
|
||||||
user_id: str
|
|
||||||
agent_id: Optional[str] = None
|
|
||||||
workspace_id: Optional[str] = None
|
|
||||||
status: Literal["open", "committed", "expired"] = "open"
|
|
||||||
expires_at: Optional[datetime] = None
|
|
||||||
created_at: datetime = Field(default_factory=utc_now)
|
|
||||||
updated_at: datetime = Field(default_factory=utc_now)
|
|
||||||
|
|
||||||
|
|
||||||
class ACLRule(BaseModel):
|
|
||||||
visibility: Visibility = Visibility.PRIVATE
|
|
||||||
allowed_user_ids: list[str] = Field(default_factory=list)
|
|
||||||
allowed_agent_ids: list[str] = Field(default_factory=list)
|
|
||||||
allowed_workspace_ids: list[str] = Field(default_factory=list)
|
|
||||||
|
|
||||||
|
|
||||||
class MemoryRecord(BaseModel):
|
|
||||||
id: str = Field(default_factory=lambda: f"mem_{uuid4().hex[:16]}")
|
|
||||||
user_id: str
|
|
||||||
agent_id: Optional[str] = None
|
|
||||||
workspace_id: Optional[str] = None
|
|
||||||
session_id: Optional[str] = None
|
|
||||||
namespace: str
|
|
||||||
memory_type: MemoryType = MemoryType.FACT
|
|
||||||
content: str
|
|
||||||
summary: Optional[str] = None
|
|
||||||
tags: list[str] = Field(default_factory=list)
|
|
||||||
importance: float = Field(default=0.5, ge=0, le=1)
|
|
||||||
confidence: float = Field(default=0.8, ge=0, le=1)
|
|
||||||
visibility: Visibility = Visibility.PRIVATE
|
|
||||||
acl: ACLRule = Field(default_factory=ACLRule)
|
|
||||||
source: SourceType = SourceType.MANUAL
|
|
||||||
source_ref: Optional[str] = None
|
|
||||||
embedding_ref: Optional[str] = None
|
|
||||||
created_at: datetime = Field(default_factory=utc_now)
|
|
||||||
updated_at: datetime = Field(default_factory=utc_now)
|
|
||||||
expires_at: Optional[datetime] = None
|
|
||||||
archived_at: Optional[datetime] = None
|
|
||||||
version: int = 1
|
|
||||||
|
|
||||||
|
|
||||||
class EpisodeRecord(BaseModel):
|
|
||||||
id: str = Field(default_factory=lambda: f"epi_{uuid4().hex[:16]}")
|
|
||||||
user_id: str
|
|
||||||
agent_id: Optional[str] = None
|
|
||||||
workspace_id: Optional[str] = None
|
|
||||||
session_id: str
|
|
||||||
namespace: str
|
|
||||||
content: str
|
|
||||||
summary: Optional[str] = None
|
|
||||||
events: list[dict[str, Any]] = Field(default_factory=list)
|
|
||||||
tags: list[str] = Field(default_factory=list)
|
|
||||||
source: SourceType = SourceType.CONVERSATION
|
|
||||||
created_at: datetime = Field(default_factory=utc_now)
|
|
||||||
expires_at: Optional[datetime] = None
|
|
||||||
|
|
||||||
|
|
||||||
class ProfileRecord(BaseModel):
|
|
||||||
id: str = Field(default_factory=lambda: f"profile_{uuid4().hex[:12]}")
|
|
||||||
user_id: str
|
|
||||||
namespace: str
|
|
||||||
display_name: Optional[str] = None
|
|
||||||
stable_facts: list[str] = Field(default_factory=list)
|
|
||||||
preferences: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
working_style: list[str] = Field(default_factory=list)
|
|
||||||
updated_from_memory_ids: list[str] = Field(default_factory=list)
|
|
||||||
version: int = 1
|
|
||||||
updated_at: datetime = Field(default_factory=utc_now)
|
|
||||||
|
|
||||||
|
|
||||||
class AuditLog(BaseModel):
|
|
||||||
id: str = Field(default_factory=lambda: f"audit_{uuid4().hex[:16]}")
|
|
||||||
actor_user_id: Optional[str] = None
|
|
||||||
actor_agent_id: Optional[str] = None
|
|
||||||
action: str
|
|
||||||
target_type: str
|
|
||||||
target_id: Optional[str] = None
|
|
||||||
namespace: Optional[str] = None
|
|
||||||
decision: Literal["allow", "deny"] = "allow"
|
|
||||||
reason: Optional[str] = None
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
created_at: datetime = Field(default_factory=utc_now)
|
|
||||||
|
|
||||||
|
|
||||||
class AccessContext(BaseModel):
|
|
||||||
user_id: str
|
|
||||||
agent_id: Optional[str] = None
|
|
||||||
workspace_id: Optional[str] = None
|
|
||||||
session_id: Optional[str] = None
|
|
||||||
|
|
||||||
|
|
||||||
class CreateUserRequest(BaseModel):
|
|
||||||
display_name: str
|
|
||||||
user_id: Optional[str] = None
|
|
||||||
preferences: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class MemorySearchRequest(AccessContext):
|
|
||||||
query: str
|
|
||||||
namespaces: list[str] = Field(default_factory=list)
|
|
||||||
memory_types: list[MemoryType] = Field(default_factory=list)
|
|
||||||
tags: list[str] = Field(default_factory=list)
|
|
||||||
limit: int = Field(default=10, ge=1, le=100)
|
|
||||||
|
|
||||||
|
|
||||||
class MemoryUpsertRequest(AccessContext):
|
|
||||||
namespace: Optional[str] = None
|
|
||||||
memory_type: MemoryType = MemoryType.FACT
|
|
||||||
content: str
|
|
||||||
summary: Optional[str] = None
|
|
||||||
tags: list[str] = Field(default_factory=list)
|
|
||||||
importance: float = Field(default=0.5, ge=0, le=1)
|
|
||||||
confidence: float = Field(default=0.8, ge=0, le=1)
|
|
||||||
visibility: Visibility = Visibility.PRIVATE
|
|
||||||
source: SourceType = SourceType.MANUAL
|
|
||||||
expires_at: Optional[datetime] = None
|
|
||||||
|
|
||||||
|
|
||||||
class MemoryPatchRequest(BaseModel):
|
|
||||||
content: Optional[str] = None
|
|
||||||
summary: Optional[str] = None
|
|
||||||
tags: Optional[list[str]] = None
|
|
||||||
importance: Optional[float] = Field(default=None, ge=0, le=1)
|
|
||||||
confidence: Optional[float] = Field(default=None, ge=0, le=1)
|
|
||||||
visibility: Optional[Visibility] = None
|
|
||||||
expires_at: Optional[datetime] = None
|
|
||||||
|
|
||||||
|
|
||||||
class EpisodeAppendRequest(AccessContext):
|
|
||||||
content: str
|
|
||||||
namespace: Optional[str] = None
|
|
||||||
events: list[dict[str, Any]] = Field(default_factory=list)
|
|
||||||
tags: list[str] = Field(default_factory=list)
|
|
||||||
source: SourceType = SourceType.CONVERSATION
|
|
||||||
expires_at: Optional[datetime] = None
|
|
||||||
|
|
||||||
|
|
||||||
class CommitSessionRequest(AccessContext):
|
|
||||||
promote: bool = True
|
|
||||||
min_importance: float = Field(default=0.6, ge=0, le=1)
|
|
||||||
target_namespace: Optional[str] = None
|
|
||||||
|
|
||||||
|
|
||||||
class MemoryFeedbackRequest(AccessContext):
|
|
||||||
feedback: Literal["useful", "not_useful", "incorrect", "duplicate", "outdated"]
|
|
||||||
comment: Optional[str] = None
|
|
||||||
|
|
||||||
|
|
||||||
class NamespaceInfo(BaseModel):
|
|
||||||
namespace: str
|
|
||||||
owner_user_id: Optional[str] = None
|
|
||||||
visibility: Visibility
|
|
||||||
description: str
|
|
||||||
@ -1,228 +0,0 @@
|
|||||||
"""Schemas for the Memory Gateway v2 control-plane API."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from datetime import datetime
|
|
||||||
from enum import Enum
|
|
||||||
from typing import Any, Literal, Optional
|
|
||||||
from uuid import uuid4
|
|
||||||
|
|
||||||
from pydantic import BaseModel, Field
|
|
||||||
|
|
||||||
from .schemas import utc_now
|
|
||||||
|
|
||||||
|
|
||||||
class OperationStatus(str, Enum):
|
|
||||||
ACCEPTED = "accepted"
|
|
||||||
RUNNING = "running"
|
|
||||||
SUCCESS = "success"
|
|
||||||
PARTIAL_SUCCESS = "partial_success"
|
|
||||||
FAILED = "failed"
|
|
||||||
PENDING = "pending"
|
|
||||||
SKIPPED = "skipped"
|
|
||||||
|
|
||||||
|
|
||||||
class BackendRefStatus(str, Enum):
|
|
||||||
PENDING = "pending"
|
|
||||||
SUCCESS = "success"
|
|
||||||
FAILED = "failed"
|
|
||||||
SKIPPED = "skipped"
|
|
||||||
|
|
||||||
|
|
||||||
class BackendType(str, Enum):
|
|
||||||
OPENVIKING = "openviking"
|
|
||||||
EVEROS = "everos"
|
|
||||||
OBSIDIAN = "obsidian"
|
|
||||||
|
|
||||||
|
|
||||||
class MemoryRefType(str, Enum):
|
|
||||||
SESSION_ARCHIVE = "session_archive"
|
|
||||||
CONTEXT_RESOURCE = "context_resource"
|
|
||||||
MESSAGE_MEMORY = "message_memory"
|
|
||||||
EPISODIC_MEMORY = "episodic_memory"
|
|
||||||
PROFILE = "profile"
|
|
||||||
LONG_TERM_MEMORY = "long_term_memory"
|
|
||||||
DRAFT_REVIEW = "draft_review"
|
|
||||||
|
|
||||||
|
|
||||||
class TraceContext(BaseModel):
|
|
||||||
trace_id: Optional[str] = None
|
|
||||||
span_id: Optional[str] = None
|
|
||||||
parent_span_id: Optional[str] = None
|
|
||||||
request_id: Optional[str] = None
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class IngestPolicy(BaseModel):
|
|
||||||
allow_openviking: bool = True
|
|
||||||
allow_everos: bool = True
|
|
||||||
allow_obsidian_review: bool = False
|
|
||||||
redact_sensitive: bool = True
|
|
||||||
require_human_review: bool = False
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class IngestRequest(BaseModel):
|
|
||||||
workspace_id: str
|
|
||||||
user_id: str
|
|
||||||
agent_id: str
|
|
||||||
session_id: str
|
|
||||||
turn_id: str
|
|
||||||
request_id: Optional[str] = None
|
|
||||||
idempotency_key: Optional[str] = None
|
|
||||||
namespace: str
|
|
||||||
source_type: str = "conversation"
|
|
||||||
source_event_id: Optional[str] = None
|
|
||||||
role: Literal["system", "user", "assistant", "tool", "agent"] = "user"
|
|
||||||
content: str
|
|
||||||
policy: IngestPolicy = Field(default_factory=IngestPolicy)
|
|
||||||
trace: TraceContext = Field(default_factory=TraceContext)
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class MemoryRef(BaseModel):
|
|
||||||
id: str = Field(default_factory=lambda: f"ref_{uuid4().hex[:16]}")
|
|
||||||
gateway_id: str
|
|
||||||
workspace_id: str
|
|
||||||
user_id: str
|
|
||||||
agent_id: Optional[str] = None
|
|
||||||
session_id: Optional[str] = None
|
|
||||||
turn_id: Optional[str] = None
|
|
||||||
namespace: Optional[str] = None
|
|
||||||
backend_type: BackendType
|
|
||||||
ref_type: MemoryRefType
|
|
||||||
native_id: Optional[str] = None
|
|
||||||
native_uri: Optional[str] = None
|
|
||||||
provenance_id: Optional[str] = None
|
|
||||||
idempotency_key: Optional[str] = None
|
|
||||||
content_hash: Optional[str] = None
|
|
||||||
source_type: Optional[str] = None
|
|
||||||
source_event_id: Optional[str] = None
|
|
||||||
status: BackendRefStatus = BackendRefStatus.PENDING
|
|
||||||
error_message: Optional[str] = None
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
created_at: datetime = Field(default_factory=utc_now)
|
|
||||||
updated_at: datetime = Field(default_factory=utc_now)
|
|
||||||
|
|
||||||
|
|
||||||
class MemoryRefView(MemoryRef):
|
|
||||||
pass
|
|
||||||
|
|
||||||
|
|
||||||
class IngestResponse(BaseModel):
|
|
||||||
status: OperationStatus
|
|
||||||
gateway_id: str
|
|
||||||
provenance_id: str
|
|
||||||
request_id: Optional[str] = None
|
|
||||||
turn_id: str
|
|
||||||
refs: list[MemoryRefView] = Field(default_factory=list)
|
|
||||||
errors: list[str] = Field(default_factory=list)
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class CommitRequest(BaseModel):
|
|
||||||
workspace_id: str
|
|
||||||
user_id: str
|
|
||||||
agent_id: Optional[str] = None
|
|
||||||
namespace: Optional[str] = None
|
|
||||||
request_id: Optional[str] = None
|
|
||||||
idempotency_key: Optional[str] = None
|
|
||||||
policy: IngestPolicy = Field(default_factory=IngestPolicy)
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class CommitResponse(BaseModel):
|
|
||||||
status: OperationStatus = OperationStatus.ACCEPTED
|
|
||||||
job_id: str
|
|
||||||
session_id: str
|
|
||||||
message: str = "commit accepted"
|
|
||||||
refs: list[MemoryRefView] = Field(default_factory=list)
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class OutboxSummary(BaseModel):
|
|
||||||
total_events: int = 0
|
|
||||||
pending_events: int = 0
|
|
||||||
processing_events: int = 0
|
|
||||||
success_events: int = 0
|
|
||||||
skipped_events: int = 0
|
|
||||||
dead_letter_events: int = 0
|
|
||||||
|
|
||||||
|
|
||||||
class CommitJobView(BaseModel):
|
|
||||||
job_id: str
|
|
||||||
workspace_id: str
|
|
||||||
user_id: str
|
|
||||||
agent_id: Optional[str] = None
|
|
||||||
session_id: str
|
|
||||||
namespace: Optional[str] = None
|
|
||||||
status: OperationStatus
|
|
||||||
created_refs_count: int = 0
|
|
||||||
error_message: Optional[str] = None
|
|
||||||
created_at: datetime
|
|
||||||
updated_at: datetime
|
|
||||||
started_at: Optional[datetime] = None
|
|
||||||
finished_at: Optional[datetime] = None
|
|
||||||
outbox_summary: OutboxSummary = Field(default_factory=OutboxSummary)
|
|
||||||
|
|
||||||
|
|
||||||
class OutboxProcessResponse(BaseModel):
|
|
||||||
status: OperationStatus
|
|
||||||
worker_id: str
|
|
||||||
processed_count: int = 0
|
|
||||||
outbox_summary: OutboxSummary = Field(default_factory=OutboxSummary)
|
|
||||||
|
|
||||||
|
|
||||||
class RetrieveRequest(BaseModel):
|
|
||||||
workspace_id: str
|
|
||||||
user_id: str
|
|
||||||
agent_id: Optional[str] = None
|
|
||||||
session_id: Optional[str] = None
|
|
||||||
namespace: Optional[str] = None
|
|
||||||
query: str
|
|
||||||
limit: int = Field(default=10, ge=1, le=100)
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class ContextItem(BaseModel):
|
|
||||||
text: Optional[str] = None
|
|
||||||
source_backend: BackendType
|
|
||||||
ref_id: Optional[str] = None
|
|
||||||
score: float = 0.0
|
|
||||||
memory_type: Optional[str] = None
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class ContextConflict(BaseModel):
|
|
||||||
ref_ids: list[str] = Field(default_factory=list)
|
|
||||||
reason: str
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class RetrieveResponse(BaseModel):
|
|
||||||
status: OperationStatus
|
|
||||||
items: list[ContextItem] = Field(default_factory=list)
|
|
||||||
refs: list[MemoryRefView] = Field(default_factory=list)
|
|
||||||
conflicts: list[ContextConflict] = Field(default_factory=list)
|
|
||||||
trace_id: Optional[str] = None
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class FeedbackRequest(BaseModel):
|
|
||||||
workspace_id: str
|
|
||||||
user_id: str
|
|
||||||
agent_id: Optional[str] = None
|
|
||||||
session_id: Optional[str] = None
|
|
||||||
namespace: Optional[str] = None
|
|
||||||
memory_ref_id: Optional[str] = None
|
|
||||||
feedback_type: Literal["useful", "not_useful", "incorrect", "duplicate", "outdated", "review_approved", "review_rejected"]
|
|
||||||
comment: Optional[str] = None
|
|
||||||
source_type: str = "manual"
|
|
||||||
source_event_id: Optional[str] = None
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class FeedbackResponse(BaseModel):
|
|
||||||
status: OperationStatus
|
|
||||||
feedback_id: str
|
|
||||||
memory_ref_id: Optional[str] = None
|
|
||||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
|
||||||
@ -1,786 +0,0 @@
|
|||||||
"""Memory Gateway MCP Server.
|
|
||||||
|
|
||||||
通用 Memory Gateway 服务，为 AI agent / harness 提供统一的 OpenViking 记忆检索、总结和知识沉淀入口。
|
|
||||||
"""
|
|
||||||
import asyncio
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
import logging
|
|
||||||
import re
|
|
||||||
import tempfile
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from contextlib import asynccontextmanager
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any, Optional
|
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, FastAPI, File, Form, Header, HTTPException, Request, UploadFile, status
|
|
||||||
from fastapi.responses import JSONResponse
|
|
||||||
from fastapi.middleware.cors import CORSMiddleware
|
|
||||||
from mcp.server import Server
|
|
||||||
from mcp.types import TextContent, Tool
|
|
||||||
from sse_starlette import EventSourceResponse
|
|
||||||
|
|
||||||
from .config import get_config, set_config, Config
|
|
||||||
from .openviking_client import get_openviking_client, close_openviking_client
|
|
||||||
from .document_ingest import convert_file_to_markdown, save_markdown_to_obsidian, slugify
|
|
||||||
from .llm import LLMConfigurationError, LLMSummaryError, summarize_with_llm
|
|
||||||
from .mcp_tools_v1 import MEMORY_GATEWAY_MCP_TOOLS
|
|
||||||
from .schemas import (
|
|
||||||
AccessContext,
|
|
||||||
CommitSessionRequest,
|
|
||||||
EpisodeAppendRequest,
|
|
||||||
MemoryFeedbackRequest,
|
|
||||||
MemorySearchRequest,
|
|
||||||
MemoryUpsertRequest,
|
|
||||||
)
|
|
||||||
from .services import service as v1_service
|
|
||||||
from .types import SearchRequest, AddMemoryRequest, AddResourceRequest, CommitSummaryRequest
|
|
||||||
|
|
||||||
# 配置日志
|
|
||||||
logging.basicConfig(
|
|
||||||
level=logging.INFO,
|
|
||||||
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
|
||||||
)
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
# 创建 MCP Server
|
|
||||||
mcp_server = Server("memory-gateway")
|
|
||||||
|
|
||||||
|
|
||||||
@mcp_server.list_tools()
|
|
||||||
async def list_tools() -> list[Tool]:
|
|
||||||
"""列出可用的 MCP 工具"""
|
|
||||||
legacy_tools = [
|
|
||||||
Tool(
|
|
||||||
name="search",
|
|
||||||
description="语义搜索记忆和资源",
|
|
||||||
inputSchema={
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"query": {"type": "string", "description": "搜索查询"},
|
|
||||||
"namespace": {"type": "string", "description": "命名空间（可选）"},
|
|
||||||
"limit": {"type": "integer", "description": "返回结果数量（默认10）"},
|
|
||||||
"uri": {"type": "string", "description": "资源 URI（可选）"},
|
|
||||||
},
|
|
||||||
"required": ["query"],
|
|
||||||
},
|
|
||||||
),
|
|
||||||
Tool(
|
|
||||||
name="add_memory",
|
|
||||||
description="添加新记忆",
|
|
||||||
inputSchema={
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"content": {"type": "string", "description": "记忆内容"},
|
|
||||||
"namespace": {"type": "string", "description": "命名空间（可选）"},
|
|
||||||
"memory_type": {"type": "string", "description": "记忆类型（默认general）"},
|
|
||||||
},
|
|
||||||
"required": ["content"],
|
|
||||||
},
|
|
||||||
),
|
|
||||||
Tool(
|
|
||||||
name="add_resource",
|
|
||||||
description="添加资源",
|
|
||||||
inputSchema={
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"uri": {"type": "string", "description": "资源 URI"},
|
|
||||||
"content": {"type": "string", "description": "资源内容"},
|
|
||||||
"resource_type": {"type": "string", "description": "资源类型（默认text）"},
|
|
||||||
},
|
|
||||||
"required": ["uri", "content"],
|
|
||||||
},
|
|
||||||
),
|
|
||||||
Tool(
|
|
||||||
name="commit_summary",
|
|
||||||
description="总结一段通用内容并按需沉淀为 OpenViking memory/resource",
|
|
||||||
inputSchema={
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"content": {"type": "string", "description": "需要总结和沉淀的原文内容"},
|
|
||||||
"title": {"type": "string", "description": "标题（可选）"},
|
|
||||||
"summary": {"type": "string", "description": "人工提供的摘要（可选）"},
|
|
||||||
"namespace": {"type": "string", "description": "OpenViking memory namespace（可选）"},
|
|
||||||
"memory_type": {"type": "string", "description": "记忆类型，默认 summary"},
|
|
||||||
"tags": {"type": "array", "items": {"type": "string"}, "description": "标签列表"},
|
|
||||||
"source": {"type": "string", "description": "来源说明或外部链接"},
|
|
||||||
"resource_uri": {"type": "string", "description": "写入 resource 的 URI（可选）"},
|
|
||||||
"resource_type": {"type": "string", "description": "资源类型，默认 json"},
|
|
||||||
"persist_as": {"type": "string", "enum": ["memory", "resource", "both", "none"], "description": "沉淀方式"},
|
|
||||||
"max_summary_chars": {"type": "integer", "description": "摘要最大长度"},
|
|
||||||
},
|
|
||||||
"required": ["content"],
|
|
||||||
},
|
|
||||||
),
|
|
||||||
Tool(
|
|
||||||
name="get_status",
|
|
||||||
description="检查系统状态",
|
|
||||||
inputSchema={
|
|
||||||
"type": "object",
|
|
||||||
"properties": {},
|
|
||||||
},
|
|
||||||
),
|
|
||||||
Tool(
|
|
||||||
name="list_memories",
|
|
||||||
description="列出已存储的记忆",
|
|
||||||
inputSchema={
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"namespace": {"type": "string", "description": "命名空间（可选）"},
|
|
||||||
"memory_type": {"type": "string", "description": "记忆类型（可选）"},
|
|
||||||
"limit": {"type": "integer", "description": "返回数量（默认10）"},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
),
|
|
||||||
Tool(
|
|
||||||
name="list_resources",
|
|
||||||
description="列出已存储的资源",
|
|
||||||
inputSchema={
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"namespace": {"type": "string", "description": "命名空间（可选）"},
|
|
||||||
"limit": {"type": "integer", "description": "返回数量（默认10）"},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
),
|
|
||||||
]
|
|
||||||
v1_tools = [
|
|
||||||
Tool(
|
|
||||||
name=definition["name"],
|
|
||||||
description=definition["description"],
|
|
||||||
inputSchema=definition["inputSchema"],
|
|
||||||
)
|
|
||||||
for definition in MEMORY_GATEWAY_MCP_TOOLS
|
|
||||||
]
|
|
||||||
return legacy_tools + v1_tools
|
|
||||||
|
|
||||||
|
|
||||||
@mcp_server.call_tool()
|
|
||||||
async def call_tool(name: str, arguments: Any) -> list[TextContent]:
|
|
||||||
"""调用 MCP 工具"""
|
|
||||||
try:
|
|
||||||
if name.startswith("memory_"):
|
|
||||||
result = await call_v1_memory_tool(name, arguments or {})
|
|
||||||
return [TextContent(type="text", text=json.dumps(result, ensure_ascii=False, default=str))]
|
|
||||||
|
|
||||||
ov_client = await get_openviking_client()
|
|
||||||
|
|
||||||
if name == "search":
|
|
||||||
result = await ov_client.search(
|
|
||||||
query=arguments.get("query"),
|
|
||||||
namespace=arguments.get("namespace"),
|
|
||||||
limit=arguments.get("limit"),
|
|
||||||
uri=arguments.get("uri"),
|
|
||||||
)
|
|
||||||
return [TextContent(type="text", text=str(result.results))]
|
|
||||||
|
|
||||||
elif name == "add_memory":
|
|
||||||
result = await ov_client.add_memory(
|
|
||||||
content=arguments.get("content"),
|
|
||||||
namespace=arguments.get("namespace"),
|
|
||||||
memory_type=arguments.get("memory_type", "general"),
|
|
||||||
)
|
|
||||||
return [TextContent(type="text", text=str(result))]
|
|
||||||
|
|
||||||
elif name == "add_resource":
|
|
||||||
result = await ov_client.add_resource(
|
|
||||||
uri=arguments.get("uri"),
|
|
||||||
content=arguments.get("content"),
|
|
||||||
resource_type=arguments.get("resource_type", "text"),
|
|
||||||
)
|
|
||||||
return [TextContent(type="text", text=str(result))]
|
|
||||||
|
|
||||||
elif name == "commit_summary":
|
|
||||||
request = CommitSummaryRequest(**arguments)
|
|
||||||
result = await commit_summary_to_openviking(request)
|
|
||||||
return [TextContent(type="text", text=json.dumps(result, ensure_ascii=False))]
|
|
||||||
|
|
||||||
elif name == "get_status":
|
|
||||||
ov_status = await ov_client.health_check()
|
|
||||||
return [TextContent(type="text", text=f"Memory Gateway: OK\nOpenViking: {ov_status}")]
|
|
||||||
|
|
||||||
elif name == "list_memories":
|
|
||||||
memories = await ov_client.list_memories(
|
|
||||||
namespace=arguments.get("namespace"),
|
|
||||||
memory_type=arguments.get("memory_type"),
|
|
||||||
limit=arguments.get("limit"),
|
|
||||||
)
|
|
||||||
return [TextContent(type="text", text=str([m.model_dump() for m in memories]))]
|
|
||||||
|
|
||||||
elif name == "list_resources":
|
|
||||||
resources = await ov_client.list_resources(
|
|
||||||
namespace=arguments.get("namespace"),
|
|
||||||
limit=arguments.get("limit"),
|
|
||||||
)
|
|
||||||
return [TextContent(type="text", text=str([r.model_dump() for r in resources]))]
|
|
||||||
|
|
||||||
else:
|
|
||||||
raise ValueError(f"Unknown tool: {name}")
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"工具执行失败: {e}")
|
|
||||||
return [TextContent(type="text", text=f"Error: {str(e)}")]
|
|
||||||
|
|
||||||
|
|
||||||
async def call_v1_memory_tool(name: str, arguments: dict[str, Any]) -> dict[str, Any]:
|
|
||||||
"""Dispatch v1 Memory Gateway MCP tools to the same service used by /v1."""
|
|
||||||
if name == "memory_search":
|
|
||||||
return _jsonable(await v1_service.search_memory_with_openviking(MemorySearchRequest(**arguments)))
|
|
||||||
if name == "memory_upsert":
|
|
||||||
return v1_service.upsert_memory(MemoryUpsertRequest(**arguments)).model_dump(mode="json")
|
|
||||||
if name == "memory_append_episode":
|
|
||||||
return v1_service.append_episode(EpisodeAppendRequest(**arguments)).model_dump(mode="json")
|
|
||||||
if name == "memory_commit_session":
|
|
||||||
session_id = arguments.get("session_id")
|
|
||||||
if not session_id:
|
|
||||||
raise ValueError("session_id is required")
|
|
||||||
return _jsonable(v1_service.commit_session(session_id, CommitSessionRequest(**arguments)))
|
|
||||||
if name == "memory_get_profile":
|
|
||||||
return v1_service.get_profile(arguments["user_id"]).model_dump(mode="json")
|
|
||||||
if name == "memory_list_namespaces":
|
|
||||||
return {
|
|
||||||
"namespaces": [
|
|
||||||
item.model_dump(mode="json")
|
|
||||||
for item in v1_service.list_namespaces(
|
|
||||||
AccessContext(
|
|
||||||
user_id=arguments["user_id"],
|
|
||||||
agent_id=arguments.get("agent_id"),
|
|
||||||
workspace_id=arguments.get("workspace_id"),
|
|
||||||
session_id=arguments.get("session_id"),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
]
|
|
||||||
}
|
|
||||||
if name == "memory_delete":
|
|
||||||
return v1_service.delete_memory(
|
|
||||||
arguments["memory_id"],
|
|
||||||
AccessContext(
|
|
||||||
user_id=arguments["user_id"],
|
|
||||||
agent_id=arguments.get("agent_id"),
|
|
||||||
workspace_id=arguments.get("workspace_id"),
|
|
||||||
session_id=arguments.get("session_id"),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
if name == "memory_feedback":
|
|
||||||
return v1_service.add_feedback(arguments["memory_id"], MemoryFeedbackRequest(**arguments))
|
|
||||||
raise ValueError(f"Unknown v1 memory tool: {name}")
|
|
||||||
|
|
||||||
|
|
||||||
def _jsonable(value: Any) -> Any:
|
|
||||||
if hasattr(value, "model_dump"):
|
|
||||||
return value.model_dump(mode="json")
|
|
||||||
if isinstance(value, list):
|
|
||||||
return [_jsonable(item) for item in value]
|
|
||||||
if isinstance(value, dict):
|
|
||||||
return {key: _jsonable(item) for key, item in value.items()}
|
|
||||||
return value
|
|
||||||
|
|
||||||
|
|
||||||
@asynccontextmanager
|
|
||||||
async def lifespan(app: FastAPI):
|
|
||||||
"""应用生命周期管理"""
|
|
||||||
logger.info("Memory Gateway 启动中...")
|
|
||||||
config = get_config()
|
|
||||||
logger.info(f"配置加载完成: {config.server.host}:{config.server.port}")
|
|
||||||
logger.info(f"OpenViking 后端: {config.openviking.url}")
|
|
||||||
|
|
||||||
# 测试 OpenViking 连接
|
|
||||||
try:
|
|
||||||
ov_client = await get_openviking_client()
|
|
||||||
status = await ov_client.health_check()
|
|
||||||
logger.info(f"OpenViking 连接状态: {status}")
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning(f"OpenViking 连接失败: {e}")
|
|
||||||
|
|
||||||
yield
|
|
||||||
|
|
||||||
logger.info("Memory Gateway 关闭中...")
|
|
||||||
await close_openviking_client()
|
|
||||||
|
|
||||||
|
|
||||||
def verify_api_key(x_api_key: Optional[str] = Header(default=None)) -> None:
|
|
||||||
"""在配置了 API Key 时校验请求头。"""
|
|
||||||
expected_key = get_config().server.api_key
|
|
||||||
if not expected_key:
|
|
||||||
return
|
|
||||||
if x_api_key != expected_key:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
|
||||||
detail="Invalid or missing API key",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
_SENTENCE_RE = re.compile(r"(?<=[。！？.!?])\s+")
|
|
||||||
_WORD_RE = re.compile(r"[^a-zA-Z0-9\u4e00-\u9fff_-]+")
|
|
||||||
|
|
||||||
|
|
||||||
def _normalize_whitespace(value: str) -> str:
|
|
||||||
return re.sub(r"\s+", " ", value).strip()
|
|
||||||
|
|
||||||
|
|
||||||
def _slugify(value: str, fallback: str) -> str:
|
|
||||||
slug = _WORD_RE.sub("-", value.lower()).strip("-")
|
|
||||||
slug = re.sub(r"-+", "-", slug)[:80].strip("-")
|
|
||||||
return slug or fallback
|
|
||||||
|
|
||||||
|
|
||||||
def _derive_title(content: str, title: Optional[str]) -> str:
|
|
||||||
if title and title.strip():
|
|
||||||
return title.strip()
|
|
||||||
for line in content.splitlines():
|
|
||||||
line = line.strip("# -*\t")
|
|
||||||
if line:
|
|
||||||
return line[:120]
|
|
||||||
return "Untitled summary"
|
|
||||||
|
|
||||||
|
|
||||||
def _derive_summary(content: str, provided: Optional[str], max_chars: int) -> str:
|
|
||||||
if provided and provided.strip():
|
|
||||||
return provided.strip()[:max_chars]
|
|
||||||
|
|
||||||
normalized = _normalize_whitespace(content)
|
|
||||||
if not normalized:
|
|
||||||
return ""
|
|
||||||
|
|
||||||
sentences = [part.strip() for part in _SENTENCE_RE.split(normalized) if part.strip()]
|
|
||||||
if not sentences:
|
|
||||||
return normalized[:max_chars]
|
|
||||||
|
|
||||||
summary = " ".join(sentences[:3])
|
|
||||||
return summary[:max_chars]
|
|
||||||
|
|
||||||
|
|
||||||
def _extract_key_points(content: str, limit: int = 8) -> list[str]:
|
|
||||||
points: list[str] = []
|
|
||||||
for raw_line in content.splitlines():
|
|
||||||
line = raw_line.strip()
|
|
||||||
if not line:
|
|
||||||
continue
|
|
||||||
stripped = re.sub(r"^(?:[-*•]\s*|\d+[.、)]\s*)", "", line).strip()
|
|
||||||
if not stripped:
|
|
||||||
continue
|
|
||||||
is_structured = line.startswith(("-", "*", "•")) or re.match(r"^\d+[.、)]\s+", line)
|
|
||||||
has_signal = any(token in stripped.lower() for token in [
|
|
||||||
"verdict", "result", "finding", "evidence", "action", "risk", "ioc",
|
|
||||||
"结论", "结果", "证据", "建议", "动作", "风险", "命中", "关联",
|
|
||||||
])
|
|
||||||
if is_structured or has_signal:
|
|
||||||
point = _normalize_whitespace(stripped)
|
|
||||||
if point and point not in points:
|
|
||||||
points.append(point[:240])
|
|
||||||
if len(points) >= limit:
|
|
||||||
break
|
|
||||||
|
|
||||||
if points:
|
|
||||||
return points
|
|
||||||
|
|
||||||
summary = _derive_summary(content, None, 500)
|
|
||||||
return [summary] if summary else []
|
|
||||||
|
|
||||||
|
|
||||||
def _render_memory_text(artifact: dict[str, Any]) -> str:
|
|
||||||
lines = [
|
|
||||||
f"Title: {artifact['title']}",
|
|
||||||
f"Summary: {artifact['summary']}",
|
|
||||||
]
|
|
||||||
if artifact.get("tags"):
|
|
||||||
lines.append("Tags: " + ", ".join(artifact["tags"]))
|
|
||||||
if artifact.get("source"):
|
|
||||||
lines.append("Source: " + artifact["source"])
|
|
||||||
if artifact.get("key_points"):
|
|
||||||
lines.append("Key points:")
|
|
||||||
lines.extend(f"- {point}" for point in artifact["key_points"])
|
|
||||||
return "\n".join(lines)
|
|
||||||
|
|
||||||
|
|
||||||
def _default_summary_resource_uri(request: CommitSummaryRequest, title: str) -> str:
|
|
||||||
namespace = (request.namespace or get_config().memory.default_namespace or "general").strip("/")
|
|
||||||
memory_type = (request.memory_type or "summary").strip("/")
|
|
||||||
digest = hashlib.sha1(request.content.encode("utf-8")).hexdigest()[:12]
|
|
||||||
slug = _slugify(title, digest)
|
|
||||||
return f"viking://resources/{namespace}/{memory_type}/{slug}-{digest}.json"
|
|
||||||
|
|
||||||
|
|
||||||
async def build_summary_artifact(request: CommitSummaryRequest) -> dict[str, Any]:
|
|
||||||
max_chars = max(120, min(request.max_summary_chars, 4000))
|
|
||||||
llm_result = await summarize_with_llm(
|
|
||||||
request.content,
|
|
||||||
title=request.title,
|
|
||||||
summary_hint=request.summary,
|
|
||||||
tags=request.tags,
|
|
||||||
max_summary_chars=max_chars,
|
|
||||||
purpose=request.purpose or "generic knowledge memory",
|
|
||||||
)
|
|
||||||
title = llm_result.get("title") or _derive_title(request.content, request.title)
|
|
||||||
return {
|
|
||||||
"schema_version": "memory-gateway.summary.v1",
|
|
||||||
"id": hashlib.sha1(request.content.encode("utf-8")).hexdigest()[:16],
|
|
||||||
"title": title,
|
|
||||||
"summary": llm_result.get("summary", ""),
|
|
||||||
"key_points": llm_result.get("key_points", []),
|
|
||||||
"tags": llm_result.get("tags", request.tags),
|
|
||||||
"source": request.source,
|
|
||||||
"namespace": request.namespace or get_config().memory.default_namespace,
|
|
||||||
"memory_type": request.memory_type or "summary",
|
|
||||||
"created_at": datetime.now(timezone.utc).isoformat(),
|
|
||||||
"content": request.content,
|
|
||||||
"llm": llm_result.get("llm"),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
async def commit_summary_to_openviking(request: CommitSummaryRequest) -> dict[str, Any]:
|
|
||||||
artifact = await build_summary_artifact(request)
|
|
||||||
ov_client = await get_openviking_client()
|
|
||||||
|
|
||||||
memory_result: Optional[dict[str, Any]] = None
|
|
||||||
resource_result: Optional[dict[str, Any]] = None
|
|
||||||
|
|
||||||
if request.persist_as in {"memory", "both"}:
|
|
||||||
memory_result = await ov_client.add_memory(
|
|
||||||
content=_render_memory_text(artifact),
|
|
||||||
namespace=artifact["namespace"],
|
|
||||||
memory_type=artifact["memory_type"],
|
|
||||||
)
|
|
||||||
|
|
||||||
if request.persist_as in {"resource", "both"}:
|
|
||||||
resource_uri = request.resource_uri or _default_summary_resource_uri(request, artifact["title"])
|
|
||||||
artifact["resource_uri"] = resource_uri
|
|
||||||
resource_result = await ov_client.add_resource(
|
|
||||||
uri=resource_uri,
|
|
||||||
content=json.dumps(artifact, ensure_ascii=False, indent=2),
|
|
||||||
resource_type=request.resource_type or "json",
|
|
||||||
)
|
|
||||||
|
|
||||||
return {
|
|
||||||
"status": "ok",
|
|
||||||
"artifact": artifact,
|
|
||||||
"memory_result": memory_result,
|
|
||||||
"resource_result": resource_result,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
# FastAPI 应用
|
|
||||||
app = FastAPI(title="Memory Gateway", version="0.1.0", lifespan=lifespan)
|
|
||||||
|
|
||||||
app.add_middleware(
|
|
||||||
CORSMiddleware,
|
|
||||||
allow_origins=["*"],
|
|
||||||
allow_credentials=True,
|
|
||||||
allow_methods=["*"],
|
|
||||||
allow_headers=["*"],
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@app.get("/health", dependencies=[Depends(verify_api_key)])
|
|
||||||
async def health_check():
|
|
||||||
"""健康检查"""
|
|
||||||
try:
|
|
||||||
ov_client = await get_openviking_client()
|
|
||||||
ov_status = await ov_client.health_check()
|
|
||||||
everos_status = v1_service.everos_health()
|
|
||||||
return {
|
|
||||||
"status": "ok",
|
|
||||||
"gateway": "memory-gateway",
|
|
||||||
"openviking": ov_status,
|
|
||||||
"everos": everos_status,
|
|
||||||
}
|
|
||||||
except Exception as e:
|
|
||||||
return {
|
|
||||||
"status": "degraded",
|
|
||||||
"gateway": "memory-gateway",
|
|
||||||
"error": str(e),
|
|
||||||
}
|
|
||||||
|
|
||||||
mcp_router = APIRouter()
|
|
||||||
|
|
||||||
|
|
||||||
async def mcp_server_events(request: Request, _: None = Depends(verify_api_key)):
|
|
||||||
"""MCP Server-Sent Events 端点 - 使用 stdio 模式模拟"""
|
|
||||||
async def event_generator():
|
|
||||||
# 发送初始化消息
|
|
||||||
yield {"event": "initialize", "data": json.dumps({"protocolVersion": "2024-11-05"})}
|
|
||||||
|
|
||||||
# 保持连接
|
|
||||||
try:
|
|
||||||
while True:
|
|
||||||
await asyncio.sleep(30)
|
|
||||||
yield {"event": "ping", "data": ""}
|
|
||||||
except asyncio.CancelledError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
return EventSourceResponse(event_generator())
|
|
||||||
|
|
||||||
|
|
||||||
mcp_router.add_api_route("/sse", mcp_server_events, methods=["GET"])
|
|
||||||
|
|
||||||
|
|
||||||
# MCP JSON-RPC 端点（简化实现）
|
|
||||||
async def mcp_rpc(request: Request, _: None = Depends(verify_api_key)):
|
|
||||||
"""处理 MCP JSON-RPC 请求"""
|
|
||||||
body = await request.json()
|
|
||||||
|
|
||||||
method = body.get("method")
|
|
||||||
params = body.get("params", {})
|
|
||||||
msg_id = body.get("id")
|
|
||||||
|
|
||||||
try:
|
|
||||||
if method == "tools/list":
|
|
||||||
tools = await list_tools()
|
|
||||||
result = {
|
|
||||||
"tools": [
|
|
||||||
{
|
|
||||||
"name": t.name,
|
|
||||||
"description": t.description,
|
|
||||||
"inputSchema": t.inputSchema,
|
|
||||||
}
|
|
||||||
for t in tools
|
|
||||||
]
|
|
||||||
}
|
|
||||||
elif method == "tools/call":
|
|
||||||
tool_name = params.get("name")
|
|
||||||
tool_args = params.get("arguments", {})
|
|
||||||
result_content = await call_tool_tool(tool_name, tool_args)
|
|
||||||
result = {"content": [c.model_dump() for c in result_content]}
|
|
||||||
else:
|
|
||||||
return JSONResponse(
|
|
||||||
status_code=400,
|
|
||||||
content={"jsonrpc": "2.0", "error": {"code": -32601, "message": f"Method not found: {method}"}, "id": msg_id}
|
|
||||||
)
|
|
||||||
|
|
||||||
return {"jsonrpc": "2.0", "result": result, "id": msg_id}
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"MCP RPC 错误: {e}")
|
|
||||||
return JSONResponse(
|
|
||||||
status_code=500,
|
|
||||||
content={"jsonrpc": "2.0", "error": {"code": -32603, "message": str(e)}, "id": msg_id}
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
async def call_tool_tool(name: str, arguments: dict) -> list[TextContent]:
|
|
||||||
"""调用工具的内部函数"""
|
|
||||||
return await call_tool(name, arguments)
|
|
||||||
|
|
||||||
|
|
||||||
mcp_router.add_api_route("/rpc", mcp_rpc, methods=["POST"])
|
|
||||||
|
|
||||||
|
|
||||||
# 注册 MCP 路由
|
|
||||||
app.include_router(mcp_router, prefix="/mcp", tags=["mcp"])
|
|
||||||
|
|
||||||
# Generic Memory Gateway v1 routes are imported lazily here to avoid changing
|
|
||||||
# the existing legacy /api and /mcp startup path.
|
|
||||||
from .api_v1 import router as api_v1_router # noqa: E402
|
|
||||||
from .api_v2 import router as api_v2_router # noqa: E402
|
|
||||||
|
|
||||||
app.include_router(api_v1_router)
|
|
||||||
app.include_router(api_v2_router)
|
|
||||||
|
|
||||||
|
|
||||||
@app.post("/api/search", dependencies=[Depends(verify_api_key)])
|
|
||||||
async def api_search(request: SearchRequest):
|
|
||||||
"""REST API: 搜索"""
|
|
||||||
ov_client = await get_openviking_client()
|
|
||||||
result = await ov_client.search(
|
|
||||||
query=request.query,
|
|
||||||
namespace=request.namespace or get_config().memory.default_namespace,
|
|
||||||
limit=request.limit or get_config().memory.search_limit,
|
|
||||||
uri=request.uri,
|
|
||||||
)
|
|
||||||
return {"results": result.results, "total": result.total}
|
|
||||||
|
|
||||||
|
|
||||||
@app.post("/api/memory", dependencies=[Depends(verify_api_key)])
|
|
||||||
async def api_add_memory(request: AddMemoryRequest):
|
|
||||||
"""REST API: 添加记忆"""
|
|
||||||
ov_client = await get_openviking_client()
|
|
||||||
result = await ov_client.add_memory(
|
|
||||||
content=request.content,
|
|
||||||
namespace=request.namespace or get_config().memory.default_namespace,
|
|
||||||
memory_type=request.memory_type,
|
|
||||||
)
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
@app.post("/api/resource", dependencies=[Depends(verify_api_key)])
|
|
||||||
async def api_add_resource(request: AddResourceRequest):
|
|
||||||
"""REST API: 添加资源"""
|
|
||||||
ov_client = await get_openviking_client()
|
|
||||||
result = await ov_client.add_resource(
|
|
||||||
uri=request.uri,
|
|
||||||
content=request.content,
|
|
||||||
resource_type=request.resource_type,
|
|
||||||
)
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
@app.post("/api/summary", dependencies=[Depends(verify_api_key)])
|
|
||||||
async def api_commit_summary(request: CommitSummaryRequest):
|
|
||||||
"""REST API: 通用内容 LLM 总结与记忆沉淀。"""
|
|
||||||
try:
|
|
||||||
return await commit_summary_to_openviking(request)
|
|
||||||
except LLMConfigurationError as exc:
|
|
||||||
raise HTTPException(status_code=503, detail=str(exc)) from exc
|
|
||||||
except (LLMSummaryError, Exception) as exc:
|
|
||||||
if isinstance(exc, HTTPException):
|
|
||||||
raise
|
|
||||||
raise HTTPException(status_code=502, detail=f"LLM summary failed: {exc}") from exc
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_tags(tags: str | None) -> list[str]:
|
|
||||||
if not tags:
|
|
||||||
return []
|
|
||||||
return [tag.strip() for tag in re.split(r"[,\n]", tags) if tag.strip()]
|
|
||||||
|
|
||||||
|
|
||||||
def _default_knowledge_uri(namespace: str, knowledge_type: str, title: str, content: str) -> str:
|
|
||||||
digest = hashlib.sha1(content.encode("utf-8")).hexdigest()[:12]
|
|
||||||
return f"viking://resources/{namespace.strip('/')}/knowledge/{knowledge_type.strip('/')}/{slugify(title, digest)}-{digest}.json"
|
|
||||||
|
|
||||||
|
|
||||||
@app.post("/api/knowledge/upload", dependencies=[Depends(verify_api_key)])
|
|
||||||
async def api_upload_knowledge(
|
|
||||||
file: UploadFile = File(...),
|
|
||||||
title: Optional[str] = Form(default=None),
|
|
||||||
namespace: str = Form(default="memory-gateway"),
|
|
||||||
knowledge_type: str = Form(default="knowledge"),
|
|
||||||
tags: str = Form(default=""),
|
|
||||||
source: Optional[str] = Form(default=None),
|
|
||||||
obsidian_dir: Optional[str] = Form(default=None),
|
|
||||||
resource_uri: Optional[str] = Form(default=None),
|
|
||||||
persist_as: str = Form(default="resource"),
|
|
||||||
max_summary_chars: int = Form(default=1000),
|
|
||||||
):
|
|
||||||
"""Upload a document, convert it to Markdown, save to Obsidian, summarize with LLM, and commit to OpenViking."""
|
|
||||||
if persist_as not in {"memory", "resource", "both", "none"}:
|
|
||||||
raise HTTPException(status_code=422, detail="persist_as must be one of memory/resource/both/none")
|
|
||||||
|
|
||||||
original_name = file.filename or "uploaded-document"
|
|
||||||
suffix = Path(original_name).suffix or ".bin"
|
|
||||||
with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as tmp:
|
|
||||||
tmp.write(await file.read())
|
|
||||||
tmp_path = Path(tmp.name)
|
|
||||||
|
|
||||||
try:
|
|
||||||
markdown = await asyncio.to_thread(convert_file_to_markdown, tmp_path)
|
|
||||||
except RuntimeError as exc:
|
|
||||||
tmp_path.unlink(missing_ok=True)
|
|
||||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
tmp_path.unlink(missing_ok=True)
|
|
||||||
raise HTTPException(status_code=500, detail=f"Document conversion failed: {exc}") from exc
|
|
||||||
finally:
|
|
||||||
tmp_path.unlink(missing_ok=True)
|
|
||||||
|
|
||||||
parsed_tags = _parse_tags(tags)
|
|
||||||
effective_title = title or Path(original_name).stem or "Uploaded knowledge"
|
|
||||||
request = CommitSummaryRequest(
|
|
||||||
content=markdown,
|
|
||||||
title=effective_title,
|
|
||||||
namespace=namespace,
|
|
||||||
memory_type=knowledge_type,
|
|
||||||
tags=parsed_tags,
|
|
||||||
source=source or original_name,
|
|
||||||
persist_as="none",
|
|
||||||
max_summary_chars=max_summary_chars,
|
|
||||||
purpose=f"knowledge upload: {knowledge_type}",
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
artifact = await build_summary_artifact(request)
|
|
||||||
except LLMConfigurationError as exc:
|
|
||||||
raise HTTPException(status_code=503, detail=str(exc)) from exc
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
raise HTTPException(status_code=502, detail=f"LLM summary failed: {exc}") from exc
|
|
||||||
|
|
||||||
config = get_config()
|
|
||||||
relative_dir = obsidian_dir or getattr(config.obsidian, "knowledge_dir", "01_Knowledge/Uploaded")
|
|
||||||
obsidian_path = save_markdown_to_obsidian(
|
|
||||||
vault_path=config.obsidian.vault_path,
|
|
||||||
relative_dir=relative_dir,
|
|
||||||
title=artifact["title"],
|
|
||||||
markdown=markdown,
|
|
||||||
source_filename=original_name,
|
|
||||||
tags=artifact.get("tags", []),
|
|
||||||
knowledge_type=knowledge_type,
|
|
||||||
summary=artifact.get("summary"),
|
|
||||||
)
|
|
||||||
|
|
||||||
artifact.update(
|
|
||||||
{
|
|
||||||
"schema_version": "memory-gateway.knowledge_upload.v1",
|
|
||||||
"knowledge_type": knowledge_type,
|
|
||||||
"source_filename": original_name,
|
|
||||||
"obsidian_path": str(obsidian_path),
|
|
||||||
"obsidian_relative_path": str(obsidian_path.relative_to(config.obsidian.vault_path)),
|
|
||||||
"markdown_content": markdown,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
ov_client = await get_openviking_client()
|
|
||||||
memory_result: Optional[dict[str, Any]] = None
|
|
||||||
resource_result: Optional[dict[str, Any]] = None
|
|
||||||
if persist_as in {"memory", "both"}:
|
|
||||||
memory_result = await ov_client.add_memory(
|
|
||||||
content=_render_memory_text(artifact),
|
|
||||||
namespace=namespace,
|
|
||||||
memory_type=knowledge_type,
|
|
||||||
)
|
|
||||||
if persist_as in {"resource", "both"}:
|
|
||||||
final_uri = resource_uri or _default_knowledge_uri(namespace, knowledge_type, artifact["title"], markdown)
|
|
||||||
artifact["resource_uri"] = final_uri
|
|
||||||
resource_result = await ov_client.add_resource(
|
|
||||||
uri=final_uri,
|
|
||||||
content=json.dumps(artifact, ensure_ascii=False, indent=2),
|
|
||||||
resource_type="json",
|
|
||||||
)
|
|
||||||
|
|
||||||
return {
|
|
||||||
"status": "ok",
|
|
||||||
"artifact": artifact,
|
|
||||||
"markdown_chars": len(markdown),
|
|
||||||
"obsidian_path": str(obsidian_path),
|
|
||||||
"memory_result": memory_result,
|
|
||||||
"resource_result": resource_result,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def create_app(config: Optional[Config] = None) -> FastAPI:
|
|
||||||
"""创建 FastAPI 应用"""
|
|
||||||
if config:
|
|
||||||
set_config(config)
|
|
||||||
return app
|
|
||||||
|
|
||||||
|
|
||||||
# 入口点
|
|
||||||
def main():
|
|
||||||
"""主入口"""
|
|
||||||
import argparse
|
|
||||||
import uvicorn
|
|
||||||
|
|
||||||
parser = argparse.ArgumentParser(description="Memory Gateway MCP Server")
|
|
||||||
parser.add_argument("--config", default="config.yaml", help="配置文件路径")
|
|
||||||
parser.add_argument("--host", default=None, help="监听地址")
|
|
||||||
parser.add_argument("--port", type=int, default=None, help="监听端口")
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
# 加载配置
|
|
||||||
from .config import load_config as load
|
|
||||||
config = load(args.config)
|
|
||||||
if args.host:
|
|
||||||
config.server.host = args.host
|
|
||||||
if args.port:
|
|
||||||
config.server.port = args.port
|
|
||||||
set_config(config)
|
|
||||||
|
|
||||||
# 启动服务
|
|
||||||
uvicorn.run(
|
|
||||||
app,
|
|
||||||
host=config.server.host,
|
|
||||||
port=config.server.port,
|
|
||||||
log_level=config.logging.level.lower(),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@ -1,15 +0,0 @@
|
|||||||
"""Small auth bridge used by the modular v1 router."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import Optional
|
|
||||||
|
|
||||||
from fastapi import Header, HTTPException, status
|
|
||||||
|
|
||||||
from .config import get_config
|
|
||||||
|
|
||||||
|
|
||||||
def verify_api_key_compat(x_api_key: Optional[str] = Header(default=None)) -> None:
|
|
||||||
expected_key = get_config().server.api_key
|
|
||||||
if expected_key and x_api_key != expected_key:
|
|
||||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid or missing API key")
|
|
||||||
|
|
||||||
@ -1,366 +0,0 @@
|
|||||||
"""Application services for the generic Memory Gateway v1 API."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
|
|
||||||
from fastapi import HTTPException, status
|
|
||||||
|
|
||||||
from .config import get_config
|
|
||||||
from .everos_client import EverOSError, EverOSClient
|
|
||||||
from .namespace import can_access_memory, default_namespace_for_context, user_long_term_namespace, visible_namespaces
|
|
||||||
from .openviking_client import get_openviking_client
|
|
||||||
from .repositories import MetadataRepository, repository
|
|
||||||
from .schemas import (
|
|
||||||
AccessContext,
|
|
||||||
AuditLog,
|
|
||||||
CommitSessionRequest,
|
|
||||||
CreateUserRequest,
|
|
||||||
EpisodeAppendRequest,
|
|
||||||
EpisodeRecord,
|
|
||||||
MemoryFeedbackRequest,
|
|
||||||
MemoryPatchRequest,
|
|
||||||
MemoryRecord,
|
|
||||||
MemorySearchRequest,
|
|
||||||
MemoryType,
|
|
||||||
MemoryUpsertRequest,
|
|
||||||
NamespaceInfo,
|
|
||||||
ProfileRecord,
|
|
||||||
SourceType,
|
|
||||||
UserRecord,
|
|
||||||
Visibility,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class ConsolidationResult:
|
|
||||||
session_id: str
|
|
||||||
episodes: int
|
|
||||||
candidates: list[MemoryRecord] = field(default_factory=list)
|
|
||||||
promoted: list[MemoryRecord] = field(default_factory=list)
|
|
||||||
duplicates: list[dict] = field(default_factory=list)
|
|
||||||
review_drafts: list[str] = field(default_factory=list)
|
|
||||||
conflicts: list[dict] = field(default_factory=list)
|
|
||||||
|
|
||||||
|
|
||||||
class MemoryGatewayService:
|
|
||||||
def __init__(self, repo: MetadataRepository = repository, everos_client: EverOSClient | None = None) -> None:
|
|
||||||
self.repo = repo
|
|
||||||
self.everos_client = everos_client
|
|
||||||
|
|
||||||
def create_user(self, request: CreateUserRequest) -> UserRecord:
|
|
||||||
user = UserRecord(
|
|
||||||
id=request.user_id or UserRecord(display_name=request.display_name).id,
|
|
||||||
display_name=request.display_name,
|
|
||||||
preferences=request.preferences,
|
|
||||||
)
|
|
||||||
user.profile_namespace = f"user/{user.id}/profile"
|
|
||||||
self.repo.create_user(user)
|
|
||||||
self._audit("create_user", "user", user.id, namespace=user.profile_namespace, actor_user_id=user.id)
|
|
||||||
return user
|
|
||||||
|
|
||||||
def get_user(self, user_id: str) -> UserRecord:
|
|
||||||
user = self.repo.get_user(user_id)
|
|
||||||
if not user:
|
|
||||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="User not found")
|
|
||||||
return user
|
|
||||||
|
|
||||||
def search_memory(self, request: MemorySearchRequest) -> dict:
|
|
||||||
ctx = AccessContext(**request.model_dump(include={"user_id", "agent_id", "workspace_id", "session_id"}))
|
|
||||||
query = request.query.lower().strip()
|
|
||||||
results = []
|
|
||||||
for memory in self.repo.list_memories():
|
|
||||||
if not can_access_memory(ctx, memory):
|
|
||||||
continue
|
|
||||||
if request.namespaces and memory.namespace not in request.namespaces:
|
|
||||||
continue
|
|
||||||
if request.memory_types and memory.memory_type not in request.memory_types:
|
|
||||||
continue
|
|
||||||
if request.tags and not set(request.tags).intersection(memory.tags):
|
|
||||||
continue
|
|
||||||
haystack = " ".join([memory.content, memory.summary or "", " ".join(memory.tags)]).lower()
|
|
||||||
if query and query not in haystack:
|
|
||||||
continue
|
|
||||||
score = self._score(memory, query)
|
|
||||||
results.append({"memory": memory, "score": score})
|
|
||||||
results.sort(key=lambda item: item["score"], reverse=True)
|
|
||||||
return {"results": results[: request.limit], "total": len(results)}
|
|
||||||
|
|
||||||
async def search_memory_with_openviking(self, request: MemorySearchRequest) -> dict:
|
|
||||||
"""Search local metadata first, then fan out to OpenViking for visible namespaces."""
|
|
||||||
ctx = AccessContext(**request.model_dump(include={"user_id", "agent_id", "workspace_id", "session_id"}))
|
|
||||||
local = self.search_memory(request)
|
|
||||||
visible = {namespace.namespace for namespace in visible_namespaces(ctx)}
|
|
||||||
requested = set(request.namespaces) if request.namespaces else visible
|
|
||||||
allowed_namespaces = sorted(requested.intersection(visible))
|
|
||||||
|
|
||||||
openviking_results = []
|
|
||||||
if allowed_namespaces and request.query.strip():
|
|
||||||
try:
|
|
||||||
ov_client = await get_openviking_client()
|
|
||||||
per_namespace_limit = max(1, min(request.limit, 10))
|
|
||||||
for namespace in allowed_namespaces:
|
|
||||||
result = await ov_client.search(
|
|
||||||
query=request.query,
|
|
||||||
namespace=namespace,
|
|
||||||
limit=per_namespace_limit,
|
|
||||||
)
|
|
||||||
for item in result.results:
|
|
||||||
item = dict(item)
|
|
||||||
item["namespace"] = namespace
|
|
||||||
item["source"] = "openviking"
|
|
||||||
openviking_results.append(item)
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
self._audit(
|
|
||||||
"openviking_search_failed",
|
|
||||||
"search",
|
|
||||||
None,
|
|
||||||
actor_user_id=request.user_id,
|
|
||||||
actor_agent_id=request.agent_id,
|
|
||||||
metadata={"error": str(exc)},
|
|
||||||
)
|
|
||||||
|
|
||||||
self._audit(
|
|
||||||
"memory_search",
|
|
||||||
"memory",
|
|
||||||
None,
|
|
||||||
actor_user_id=request.user_id,
|
|
||||||
actor_agent_id=request.agent_id,
|
|
||||||
metadata={"query": request.query, "namespaces": allowed_namespaces, "openviking_results": len(openviking_results)},
|
|
||||||
)
|
|
||||||
return {
|
|
||||||
"results": local["results"] + [{"openviking": item, "score": item.get("score", 0)} for item in openviking_results],
|
|
||||||
"total": local["total"] + len(openviking_results),
|
|
||||||
"local_total": local["total"],
|
|
||||||
"openviking_total": len(openviking_results),
|
|
||||||
"searched_namespaces": allowed_namespaces,
|
|
||||||
}
|
|
||||||
|
|
||||||
def upsert_memory(self, request: MemoryUpsertRequest) -> MemoryRecord:
|
|
||||||
ctx = AccessContext(**request.model_dump(include={"user_id", "agent_id", "workspace_id", "session_id"}))
|
|
||||||
namespace = request.namespace or default_namespace_for_context(ctx, request.visibility)
|
|
||||||
memory = MemoryRecord(
|
|
||||||
user_id=request.user_id,
|
|
||||||
agent_id=request.agent_id,
|
|
||||||
workspace_id=request.workspace_id,
|
|
||||||
session_id=request.session_id,
|
|
||||||
namespace=namespace,
|
|
||||||
memory_type=request.memory_type,
|
|
||||||
content=request.content,
|
|
||||||
summary=request.summary,
|
|
||||||
tags=request.tags,
|
|
||||||
importance=request.importance,
|
|
||||||
confidence=request.confidence,
|
|
||||||
visibility=request.visibility,
|
|
||||||
source=request.source,
|
|
||||||
expires_at=request.expires_at,
|
|
||||||
)
|
|
||||||
self.repo.upsert_memory(memory)
|
|
||||||
self._audit("upsert_memory", "memory", memory.id, namespace=memory.namespace, actor_user_id=request.user_id, actor_agent_id=request.agent_id)
|
|
||||||
return memory
|
|
||||||
|
|
||||||
def get_memory(self, memory_id: str, ctx: AccessContext) -> MemoryRecord:
|
|
||||||
memory = self.repo.get_memory(memory_id)
|
|
||||||
if not memory:
|
|
||||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Memory not found")
|
|
||||||
if not can_access_memory(ctx, memory):
|
|
||||||
self._audit("get_memory", "memory", memory_id, namespace=memory.namespace, actor_user_id=ctx.user_id, actor_agent_id=ctx.agent_id, decision="deny")
|
|
||||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Memory access denied")
|
|
||||||
return memory
|
|
||||||
|
|
||||||
def patch_memory(self, memory_id: str, ctx: AccessContext, patch: MemoryPatchRequest) -> MemoryRecord:
|
|
||||||
memory = self.get_memory(memory_id, ctx)
|
|
||||||
updates = patch.model_dump(exclude_unset=True)
|
|
||||||
for key, value in updates.items():
|
|
||||||
setattr(memory, key, value)
|
|
||||||
memory.updated_at = datetime.now(timezone.utc)
|
|
||||||
memory.version += 1
|
|
||||||
self.repo.upsert_memory(memory)
|
|
||||||
self._audit("patch_memory", "memory", memory.id, namespace=memory.namespace, actor_user_id=ctx.user_id, actor_agent_id=ctx.agent_id)
|
|
||||||
return memory
|
|
||||||
|
|
||||||
def delete_memory(self, memory_id: str, ctx: AccessContext) -> dict:
|
|
||||||
memory = self.get_memory(memory_id, ctx)
|
|
||||||
deleted = self.repo.delete_memory(memory_id)
|
|
||||||
self._audit("delete_memory", "memory", memory_id, namespace=memory.namespace, actor_user_id=ctx.user_id, actor_agent_id=ctx.agent_id)
|
|
||||||
return {"deleted": deleted, "id": memory_id}
|
|
||||||
|
|
||||||
def append_episode(self, request: EpisodeAppendRequest) -> EpisodeRecord:
|
|
||||||
ctx = AccessContext(**request.model_dump(include={"user_id", "agent_id", "workspace_id", "session_id"}))
|
|
||||||
episode = EpisodeRecord(
|
|
||||||
user_id=request.user_id,
|
|
||||||
agent_id=request.agent_id,
|
|
||||||
workspace_id=request.workspace_id,
|
|
||||||
session_id=request.session_id or "default",
|
|
||||||
namespace=request.namespace or default_namespace_for_context(ctx, Visibility.PRIVATE),
|
|
||||||
content=request.content,
|
|
||||||
events=request.events,
|
|
||||||
tags=request.tags,
|
|
||||||
source=request.source,
|
|
||||||
expires_at=request.expires_at,
|
|
||||||
)
|
|
||||||
self.repo.append_episode(episode)
|
|
||||||
self._audit("append_episode", "episode", episode.id, namespace=episode.namespace, actor_user_id=request.user_id, actor_agent_id=request.agent_id)
|
|
||||||
return episode
|
|
||||||
|
|
||||||
def commit_session(self, session_id: str, request: CommitSessionRequest) -> dict:
|
|
||||||
episodes = self.repo.list_session_episodes(session_id)
|
|
||||||
backend = "disabled"
|
|
||||||
error: str | None = None
|
|
||||||
if request.promote:
|
|
||||||
ctx = AccessContext(
|
|
||||||
user_id=request.user_id,
|
|
||||||
agent_id=request.agent_id,
|
|
||||||
workspace_id=request.workspace_id,
|
|
||||||
session_id=session_id,
|
|
||||||
)
|
|
||||||
target_namespace = request.target_namespace or user_long_term_namespace(request.user_id)
|
|
||||||
config = get_config().everos
|
|
||||||
if config.enabled:
|
|
||||||
try:
|
|
||||||
external_result = (self.everos_client or EverOSClient()).consolidate_session(
|
|
||||||
session_id=session_id,
|
|
||||||
ctx=ctx,
|
|
||||||
episodes=episodes,
|
|
||||||
existing_memories=list(self.repo.list_memories()),
|
|
||||||
min_importance=request.min_importance,
|
|
||||||
target_namespace=target_namespace,
|
|
||||||
)
|
|
||||||
result = self._persist_external_consolidation(external_result, ctx, session_id)
|
|
||||||
backend = "external"
|
|
||||||
except EverOSError as exc:
|
|
||||||
error = str(exc)
|
|
||||||
self._audit(
|
|
||||||
"everos_commit_failed",
|
|
||||||
"session",
|
|
||||||
session_id,
|
|
||||||
actor_user_id=request.user_id,
|
|
||||||
actor_agent_id=request.agent_id,
|
|
||||||
decision="deny",
|
|
||||||
metadata={"error": error},
|
|
||||||
)
|
|
||||||
raise HTTPException(status_code=status.HTTP_502_BAD_GATEWAY, detail=f"EverOS failed: {error}") from exc
|
|
||||||
else:
|
|
||||||
result = None
|
|
||||||
backend = "disabled"
|
|
||||||
else:
|
|
||||||
result = None
|
|
||||||
self._audit("commit_session", "session", session_id, actor_user_id=request.user_id, actor_agent_id=request.agent_id)
|
|
||||||
if not result:
|
|
||||||
return {"session_id": session_id, "episodes": len(episodes), "promoted": [], "everos_backend": backend}
|
|
||||||
return {
|
|
||||||
"everos_backend": backend,
|
|
||||||
"everos_error": error,
|
|
||||||
"session_id": session_id,
|
|
||||||
"episodes": result.episodes,
|
|
||||||
"candidates": result.candidates,
|
|
||||||
"promoted": result.promoted,
|
|
||||||
"duplicates": result.duplicates,
|
|
||||||
"conflicts": result.conflicts,
|
|
||||||
"review_drafts": result.review_drafts,
|
|
||||||
}
|
|
||||||
|
|
||||||
def everos_health(self) -> dict:
|
|
||||||
config = get_config().everos
|
|
||||||
if not config.enabled:
|
|
||||||
return {"status": "disabled", "url": config.url}
|
|
||||||
return (self.everos_client or EverOSClient()).health()
|
|
||||||
|
|
||||||
def _persist_external_consolidation(self, external_result: dict, ctx: AccessContext, session_id: str):
|
|
||||||
result = ConsolidationResult(
|
|
||||||
session_id=session_id,
|
|
||||||
episodes=external_result.get("episodes") or len(self.repo.list_session_episodes(session_id)),
|
|
||||||
duplicates=external_result.get("duplicates", []),
|
|
||||||
conflicts=external_result.get("conflicts", []),
|
|
||||||
review_drafts=external_result.get("review_drafts", []),
|
|
||||||
)
|
|
||||||
for item in external_result.get("candidates", []):
|
|
||||||
memory = self._memory_from_external(item, ctx, session_id)
|
|
||||||
if memory:
|
|
||||||
result.candidates.append(memory)
|
|
||||||
for item in external_result.get("promoted", []):
|
|
||||||
memory = self._memory_from_external(item, ctx, session_id)
|
|
||||||
if memory:
|
|
||||||
self.repo.upsert_memory(memory)
|
|
||||||
result.promoted.append(memory)
|
|
||||||
if all(candidate.id != memory.id for candidate in result.candidates):
|
|
||||||
result.candidates.append(memory)
|
|
||||||
return result
|
|
||||||
|
|
||||||
def _memory_from_external(self, item: dict, ctx: AccessContext, session_id: str) -> MemoryRecord | None:
|
|
||||||
if not isinstance(item, dict):
|
|
||||||
return None
|
|
||||||
data = dict(item)
|
|
||||||
data.setdefault("user_id", ctx.user_id)
|
|
||||||
data.setdefault("agent_id", ctx.agent_id)
|
|
||||||
data.setdefault("workspace_id", ctx.workspace_id)
|
|
||||||
data.setdefault("session_id", session_id)
|
|
||||||
data.setdefault("namespace", default_namespace_for_context(ctx, Visibility.PRIVATE))
|
|
||||||
data.setdefault("memory_type", MemoryType.SUMMARY.value)
|
|
||||||
data.setdefault("content", data.get("text") or data.get("summary") or "")
|
|
||||||
data.setdefault("summary", data.get("content", "")[:180])
|
|
||||||
data.setdefault("tags", ["everos-external"])
|
|
||||||
data.setdefault("importance", 0.7)
|
|
||||||
data.setdefault("confidence", 0.65)
|
|
||||||
data.setdefault("visibility", Visibility.PRIVATE.value)
|
|
||||||
data.setdefault("source", SourceType.EVEROS.value)
|
|
||||||
if not data["content"]:
|
|
||||||
return None
|
|
||||||
return MemoryRecord.model_validate(data)
|
|
||||||
|
|
||||||
def get_profile(self, user_id: str) -> ProfileRecord:
|
|
||||||
profile = self.repo.get_profile(user_id)
|
|
||||||
if not profile:
|
|
||||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Profile not found")
|
|
||||||
return profile
|
|
||||||
|
|
||||||
def add_feedback(self, memory_id: str, request: MemoryFeedbackRequest) -> dict:
|
|
||||||
ctx = AccessContext(**request.model_dump(include={"user_id", "agent_id", "workspace_id", "session_id"}))
|
|
||||||
memory = self.get_memory(memory_id, ctx)
|
|
||||||
self._audit(
|
|
||||||
f"feedback:{request.feedback}",
|
|
||||||
"memory",
|
|
||||||
memory.id,
|
|
||||||
namespace=memory.namespace,
|
|
||||||
actor_user_id=request.user_id,
|
|
||||||
actor_agent_id=request.agent_id,
|
|
||||||
metadata={"comment": request.comment},
|
|
||||||
)
|
|
||||||
return {"status": "ok", "memory_id": memory_id, "feedback": request.feedback}
|
|
||||||
|
|
||||||
def list_namespaces(self, ctx: AccessContext) -> list[NamespaceInfo]:
|
|
||||||
return visible_namespaces(ctx)
|
|
||||||
|
|
||||||
def list_audit(self, limit: int = 100) -> list[AuditLog]:
|
|
||||||
return self.repo.list_audit(limit)
|
|
||||||
|
|
||||||
def _score(self, memory: MemoryRecord, query: str) -> float:
|
|
||||||
lexical = 1.0 if query and query in memory.content.lower() else 0.2
|
|
||||||
return lexical + memory.importance + memory.confidence
|
|
||||||
|
|
||||||
def _audit(
|
|
||||||
self,
|
|
||||||
action: str,
|
|
||||||
target_type: str,
|
|
||||||
target_id: str | None,
|
|
||||||
namespace: str | None = None,
|
|
||||||
actor_user_id: str | None = None,
|
|
||||||
actor_agent_id: str | None = None,
|
|
||||||
decision: str = "allow",
|
|
||||||
metadata: dict | None = None,
|
|
||||||
) -> None:
|
|
||||||
self.repo.add_audit(
|
|
||||||
AuditLog(
|
|
||||||
actor_user_id=actor_user_id,
|
|
||||||
actor_agent_id=actor_agent_id,
|
|
||||||
action=action,
|
|
||||||
target_type=target_type,
|
|
||||||
target_id=target_id,
|
|
||||||
namespace=namespace,
|
|
||||||
decision=decision, # type: ignore[arg-type]
|
|
||||||
metadata=metadata or {},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
service = MemoryGatewayService()
|
|
||||||
File diff suppressed because it is too large
Load Diff
@ -1,2 +0,0 @@
|
|||||||
"""Skill skeletons for Memory Gateway processing units."""
|
|
||||||
|
|
||||||
@ -1,21 +0,0 @@
|
|||||||
"""Shared skill contracts."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class SkillResult:
|
|
||||||
status: str
|
|
||||||
output: dict[str, Any] = field(default_factory=dict)
|
|
||||||
writes_long_term_memory: bool = False
|
|
||||||
|
|
||||||
|
|
||||||
class MemorySkill:
|
|
||||||
name = "memory_skill"
|
|
||||||
writes_long_term_memory = False
|
|
||||||
|
|
||||||
async def run(self, payload: dict[str, Any]) -> SkillResult:
|
|
||||||
raise NotImplementedError
|
|
||||||
|
|
||||||
@ -1,9 +0,0 @@
|
|||||||
from .base import MemorySkill, SkillResult
|
|
||||||
|
|
||||||
|
|
||||||
class ClassifyMemorySkill(MemorySkill):
|
|
||||||
name = "classify_memory_skill"
|
|
||||||
|
|
||||||
async def run(self, payload: dict) -> SkillResult:
|
|
||||||
return SkillResult(status="ok", output={"memory_type": payload.get("memory_type", "fact"), "visibility": payload.get("visibility", "private")})
|
|
||||||
|
|
||||||
@ -1,10 +0,0 @@
|
|||||||
from .base import MemorySkill, SkillResult
|
|
||||||
|
|
||||||
|
|
||||||
class CommitMemorySkill(MemorySkill):
|
|
||||||
name = "commit_memory_skill"
|
|
||||||
writes_long_term_memory = True
|
|
||||||
|
|
||||||
async def run(self, payload: dict) -> SkillResult:
|
|
||||||
return SkillResult(status="ok", output={"committed": payload}, writes_long_term_memory=True)
|
|
||||||
|
|
||||||
@ -1,9 +0,0 @@
|
|||||||
from .base import MemorySkill, SkillResult
|
|
||||||
|
|
||||||
|
|
||||||
class ExportToObsidianSkill(MemorySkill):
|
|
||||||
name = "export_to_obsidian_skill"
|
|
||||||
|
|
||||||
async def run(self, payload: dict) -> SkillResult:
|
|
||||||
return SkillResult(status="ok", output={"draft_path": payload.get("draft_path")})
|
|
||||||
|
|
||||||
@ -1,11 +0,0 @@
|
|||||||
from .base import MemorySkill, SkillResult
|
|
||||||
|
|
||||||
|
|
||||||
class ExtractMemorySkill(MemorySkill):
|
|
||||||
name = "extract_memory_skill"
|
|
||||||
|
|
||||||
async def run(self, payload: dict) -> SkillResult:
|
|
||||||
text = payload.get("content", "")
|
|
||||||
candidates = [{"content": text, "confidence": 0.5}] if text else []
|
|
||||||
return SkillResult(status="ok", output={"candidates": candidates})
|
|
||||||
|
|
||||||
@ -1,10 +0,0 @@
|
|||||||
from .base import MemorySkill, SkillResult
|
|
||||||
|
|
||||||
|
|
||||||
class ImportFromObsidianSkill(MemorySkill):
|
|
||||||
name = "import_from_obsidian_skill"
|
|
||||||
writes_long_term_memory = True
|
|
||||||
|
|
||||||
async def run(self, payload: dict) -> SkillResult:
|
|
||||||
return SkillResult(status="ok", output={"imported_path": payload.get("path")}, writes_long_term_memory=True)
|
|
||||||
|
|
||||||
@ -1,9 +0,0 @@
|
|||||||
from .base import MemorySkill, SkillResult
|
|
||||||
|
|
||||||
|
|
||||||
class IngestSkill(MemorySkill):
|
|
||||||
name = "ingest_skill"
|
|
||||||
|
|
||||||
async def run(self, payload: dict) -> SkillResult:
|
|
||||||
return SkillResult(status="ok", output={"normalized": payload})
|
|
||||||
|
|
||||||
@ -1,10 +0,0 @@
|
|||||||
from .base import MemorySkill, SkillResult
|
|
||||||
|
|
||||||
|
|
||||||
class MergeMemorySkill(MemorySkill):
|
|
||||||
name = "merge_memory_skill"
|
|
||||||
writes_long_term_memory = True
|
|
||||||
|
|
||||||
async def run(self, payload: dict) -> SkillResult:
|
|
||||||
return SkillResult(status="ok", output={"merged": payload.get("memory_ids", [])}, writes_long_term_memory=True)
|
|
||||||
|
|
||||||
@ -1,10 +0,0 @@
|
|||||||
from .base import MemorySkill, SkillResult
|
|
||||||
|
|
||||||
|
|
||||||
class PruneMemorySkill(MemorySkill):
|
|
||||||
name = "prune_memory_skill"
|
|
||||||
writes_long_term_memory = True
|
|
||||||
|
|
||||||
async def run(self, payload: dict) -> SkillResult:
|
|
||||||
return SkillResult(status="ok", output={"pruned": payload.get("memory_ids", [])}, writes_long_term_memory=True)
|
|
||||||
|
|
||||||
@ -1,9 +0,0 @@
|
|||||||
from .base import MemorySkill, SkillResult
|
|
||||||
|
|
||||||
|
|
||||||
class RetrieveContextSkill(MemorySkill):
|
|
||||||
name = "retrieve_context_skill"
|
|
||||||
|
|
||||||
async def run(self, payload: dict) -> SkillResult:
|
|
||||||
return SkillResult(status="ok", output={"query": payload.get("query"), "contexts": []})
|
|
||||||
|
|
||||||
@ -1,10 +0,0 @@
|
|||||||
from .base import MemorySkill, SkillResult
|
|
||||||
|
|
||||||
|
|
||||||
class SummarizeEpisodeSkill(MemorySkill):
|
|
||||||
name = "summarize_episode_skill"
|
|
||||||
|
|
||||||
async def run(self, payload: dict) -> SkillResult:
|
|
||||||
content = payload.get("content", "")
|
|
||||||
return SkillResult(status="ok", output={"summary": content[:500]})
|
|
||||||
|
|
||||||
@ -1,158 +0,0 @@
|
|||||||
"""类型定义"""
|
|
||||||
from typing import Optional, Any, Literal
|
|
||||||
from pydantic import BaseModel, Field
|
|
||||||
|
|
||||||
|
|
||||||
class ServerConfig(BaseModel):
|
|
||||||
"""服务器配置"""
|
|
||||||
host: str = "0.0.0.0"
|
|
||||||
port: int = 1934
|
|
||||||
api_key: str = ""
|
|
||||||
|
|
||||||
|
|
||||||
class OpenVikingConfig(BaseModel):
|
|
||||||
"""OpenViking 后端配置"""
|
|
||||||
enabled: bool = False
|
|
||||||
mode: Literal["offline", "skeleton", "real"] = "offline"
|
|
||||||
url: str = "http://localhost:1933"
|
|
||||||
api_key: str = ""
|
|
||||||
timeout: int = 30
|
|
||||||
verify_ssl: bool = True
|
|
||||||
ingest_path: str = "/api/v1/sessions/{session_id}/messages"
|
|
||||||
|
|
||||||
|
|
||||||
class EverOSConfig(BaseModel):
|
|
||||||
"""External EverOS memory service configuration."""
|
|
||||||
enabled: bool = False
|
|
||||||
mode: Literal["offline", "skeleton", "real"] = "offline"
|
|
||||||
url: str = "http://127.0.0.1:1995"
|
|
||||||
api_key: str = ""
|
|
||||||
timeout: int = 30
|
|
||||||
verify_ssl: bool = True
|
|
||||||
health_path: str = "/health"
|
|
||||||
ingest_path: str = "/api/v1/memories"
|
|
||||||
search_path: str = "/api/v1/memories/search"
|
|
||||||
flush_path: str = "/api/v1/memories/flush"
|
|
||||||
retrieve_method: Literal["keyword", "vector", "hybrid", "rrf", "agentic"] = "keyword"
|
|
||||||
|
|
||||||
class MemoryConfig(BaseModel):
|
|
||||||
"""记忆配置"""
|
|
||||||
default_namespace: str = "memory-gateway"
|
|
||||||
search_limit: int = 10
|
|
||||||
|
|
||||||
|
|
||||||
class LLMConfig(BaseModel):
|
|
||||||
"""LLM 配置，用于通用总结和知识沉淀。"""
|
|
||||||
base_url: str = "https://api.openai.com/v1"
|
|
||||||
api_key: str = ""
|
|
||||||
model: str = ""
|
|
||||||
timeout: int = 60
|
|
||||||
max_input_chars: int = 24000
|
|
||||||
|
|
||||||
|
|
||||||
class ObsidianConfig(BaseModel):
|
|
||||||
"""Obsidian Vault 配置。"""
|
|
||||||
vault_path: str = "/home/tom/memory-gateway/obsidian-vault"
|
|
||||||
knowledge_dir: str = "01_Knowledge/Uploaded"
|
|
||||||
review_dir: str = "Reviews/Queue"
|
|
||||||
|
|
||||||
|
|
||||||
class StorageConfig(BaseModel):
|
|
||||||
"""Metadata storage configuration."""
|
|
||||||
backend: Literal["sqlite", "memory"] = "sqlite"
|
|
||||||
sqlite_path: str = "/home/tom/memory-gateway/memory_gateway.sqlite3"
|
|
||||||
|
|
||||||
|
|
||||||
class LoggingConfig(BaseModel):
|
|
||||||
"""日志配置"""
|
|
||||||
level: str = "INFO"
|
|
||||||
format: str = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
|
||||||
|
|
||||||
|
|
||||||
class Config(BaseModel):
|
|
||||||
"""完整配置"""
|
|
||||||
def __init__(self, **data: Any) -> None:
|
|
||||||
super().__init__(**data)
|
|
||||||
|
|
||||||
server: ServerConfig = Field(default_factory=ServerConfig)
|
|
||||||
openviking: OpenVikingConfig = Field(default_factory=OpenVikingConfig)
|
|
||||||
everos: EverOSConfig = Field(default_factory=EverOSConfig)
|
|
||||||
memory: MemoryConfig = Field(default_factory=MemoryConfig)
|
|
||||||
logging: LoggingConfig = Field(default_factory=LoggingConfig)
|
|
||||||
llm: LLMConfig = Field(default_factory=LLMConfig)
|
|
||||||
obsidian: ObsidianConfig = Field(default_factory=ObsidianConfig)
|
|
||||||
storage: StorageConfig = Field(default_factory=StorageConfig)
|
|
||||||
|
|
||||||
class SearchRequest(BaseModel):
|
|
||||||
"""搜索请求"""
|
|
||||||
query: str
|
|
||||||
namespace: Optional[str] = None
|
|
||||||
limit: Optional[int] = None
|
|
||||||
uri: Optional[str] = None
|
|
||||||
|
|
||||||
|
|
||||||
class AddMemoryRequest(BaseModel):
|
|
||||||
"""添加记忆请求"""
|
|
||||||
content: str
|
|
||||||
namespace: Optional[str] = None
|
|
||||||
memory_type: Optional[str] = "general"
|
|
||||||
|
|
||||||
|
|
||||||
class AddResourceRequest(BaseModel):
|
|
||||||
"""添加资源请求"""
|
|
||||||
uri: str
|
|
||||||
content: str
|
|
||||||
resource_type: Optional[str] = "text"
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
class CommitSummaryRequest(BaseModel):
|
|
||||||
"""通用总结与沉淀请求。
|
|
||||||
|
|
||||||
该模型用于任意场景把一段高价值内容总结后
|
|
||||||
写入 OpenViking memory、resource，或两者同时写入。
|
|
||||||
"""
|
|
||||||
content: str
|
|
||||||
title: Optional[str] = None
|
|
||||||
summary: Optional[str] = None
|
|
||||||
purpose: Optional[str] = "generic knowledge memory"
|
|
||||||
namespace: Optional[str] = None
|
|
||||||
memory_type: Optional[str] = "summary"
|
|
||||||
tags: list[str] = Field(default_factory=list)
|
|
||||||
source: Optional[str] = None
|
|
||||||
resource_uri: Optional[str] = None
|
|
||||||
resource_type: Optional[str] = "json"
|
|
||||||
persist_as: Literal["memory", "resource", "both", "none"] = "both"
|
|
||||||
max_summary_chars: int = 600
|
|
||||||
|
|
||||||
|
|
||||||
class CommitSummaryResponse(BaseModel):
|
|
||||||
"""通用总结与沉淀响应。"""
|
|
||||||
status: str
|
|
||||||
artifact: dict[str, Any]
|
|
||||||
memory_result: Optional[dict[str, Any]] = None
|
|
||||||
resource_result: Optional[dict[str, Any]] = None
|
|
||||||
|
|
||||||
|
|
||||||
class SearchResult(BaseModel):
|
|
||||||
"""搜索结果"""
|
|
||||||
results: list[dict[str, Any]]
|
|
||||||
total: int
|
|
||||||
|
|
||||||
|
|
||||||
class MemoryEntry(BaseModel):
|
|
||||||
"""记忆条目"""
|
|
||||||
id: str
|
|
||||||
content: str
|
|
||||||
namespace: str
|
|
||||||
memory_type: str
|
|
||||||
created_at: Optional[str] = None
|
|
||||||
|
|
||||||
|
|
||||||
class ResourceEntry(BaseModel):
|
|
||||||
"""资源条目"""
|
|
||||||
uri: str
|
|
||||||
content: str
|
|
||||||
resource_type: str
|
|
||||||
created_at: Optional[str] = None
|
|
||||||
@ -1,49 +0,0 @@
|
|||||||
"""Lightweight v2 outbox worker entrypoint.
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
python -m memory_gateway.worker_v2 --limit 100 --worker-id local-worker --lease-seconds 300
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import asyncio
|
|
||||||
import json
|
|
||||||
from typing import Sequence
|
|
||||||
from uuid import uuid4
|
|
||||||
|
|
||||||
from .services_v2 import v2_service
|
|
||||||
|
|
||||||
|
|
||||||
def build_parser() -> argparse.ArgumentParser:
|
|
||||||
parser = argparse.ArgumentParser(description="Process Memory Gateway v2 outbox events once.")
|
|
||||||
parser.add_argument("--limit", type=int, default=100, help="Maximum pending events to claim and process.")
|
|
||||||
parser.add_argument("--worker-id", default=None, help="Stable worker id recorded in outbox lease fields.")
|
|
||||||
parser.add_argument("--lease-seconds", type=int, default=300, help="Lease duration for claimed events.")
|
|
||||||
return parser
|
|
||||||
|
|
||||||
|
|
||||||
async def run_once(limit: int, worker_id: str | None, lease_seconds: int) -> dict[str, object]:
|
|
||||||
worker_id = worker_id or f"worker_{uuid4().hex[:12]}"
|
|
||||||
response = await v2_service.process_pending_outbox_events_summary(
|
|
||||||
limit=limit,
|
|
||||||
worker_id=worker_id,
|
|
||||||
lease_seconds=lease_seconds,
|
|
||||||
)
|
|
||||||
return response.model_dump(mode="json")
|
|
||||||
|
|
||||||
|
|
||||||
def main(argv: Sequence[str] | None = None) -> int:
|
|
||||||
args = build_parser().parse_args(argv)
|
|
||||||
payload = asyncio.run(
|
|
||||||
run_once(
|
|
||||||
limit=args.limit,
|
|
||||||
worker_id=args.worker_id,
|
|
||||||
lease_seconds=args.lease_seconds,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
print(json.dumps(payload, ensure_ascii=False, sort_keys=True))
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
1
memory_system_api/__init__.py
Normal file
1
memory_system_api/__init__.py
Normal file
@ -0,0 +1 @@
|
|||||||
|
"""Lightweight Memory System API package."""
|
||||||
69
memory_system_api/api.py
Normal file
69
memory_system_api/api.py
Normal file
@ -0,0 +1,69 @@
|
|||||||
|
"""FastAPI router for the lightweight Memory System API."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
||||||
|
|
||||||
|
from .auth import verify_api_key
|
||||||
|
from .schemas import MessageIngestRequest, SearchRequest, SessionUserRequest
|
||||||
|
from .service import MemorySystemService
|
||||||
|
|
||||||
|
|
||||||
|
router = APIRouter(
|
||||||
|
prefix="/memory-system",
|
||||||
|
tags=["memory-system"],
|
||||||
|
dependencies=[Depends(verify_api_key)],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def get_service() -> MemorySystemService:
|
||||||
|
return MemorySystemService()
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/health")
|
||||||
|
async def health(service: MemorySystemService = Depends(get_service)):
|
||||||
|
return await service.health()
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/messages")
|
||||||
|
async def ingest_messages(request: MessageIngestRequest, service: MemorySystemService = Depends(get_service)):
|
||||||
|
try:
|
||||||
|
return await service.ingest_messages(request)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail=str(exc)) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/sessions/{session_id}/commit")
|
||||||
|
async def commit_session(
|
||||||
|
session_id: str,
|
||||||
|
request: SessionUserRequest,
|
||||||
|
service: MemorySystemService = Depends(get_service),
|
||||||
|
):
|
||||||
|
return await service.commit_session(request.user_id, session_id)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/sessions/{session_id}/extract")
|
||||||
|
async def extract_session(
|
||||||
|
session_id: str,
|
||||||
|
request: SessionUserRequest,
|
||||||
|
service: MemorySystemService = Depends(get_service),
|
||||||
|
):
|
||||||
|
return await service.extract_session(request.user_id, session_id)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/openviking/tasks/{task_id}")
|
||||||
|
async def get_openviking_task(
|
||||||
|
task_id: str,
|
||||||
|
user_id: str = Query(min_length=1),
|
||||||
|
service: MemorySystemService = Depends(get_service),
|
||||||
|
):
|
||||||
|
return await service.get_openviking_task(user_id, task_id)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/search")
|
||||||
|
async def search(request: SearchRequest, service: MemorySystemService = Depends(get_service)):
|
||||||
|
return await service.search(request)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/users/{user_id}/profile")
|
||||||
|
async def get_profile(user_id: str, service: MemorySystemService = Depends(get_service)):
|
||||||
|
return await service.get_profile(user_id)
|
||||||
12
memory_system_api/auth.py
Normal file
12
memory_system_api/auth.py
Normal file
@ -0,0 +1,12 @@
|
|||||||
|
"""API key auth for Memory System API."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from fastapi import Header, HTTPException, status
|
||||||
|
|
||||||
|
from .config import get_config
|
||||||
|
|
||||||
|
|
||||||
|
def verify_api_key(x_api_key: str | None = Header(default=None)) -> None:
|
||||||
|
expected = get_config().server.api_key
|
||||||
|
if expected and x_api_key != expected:
|
||||||
|
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid API key")
|
||||||
210
memory_system_api/clients.py
Normal file
210
memory_system_api/clients.py
Normal file
@ -0,0 +1,210 @@
|
|||||||
|
"""Async clients for OpenViking and EverOS used by the lightweight API."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
from .config import get_config
|
||||||
|
from .store import OpenVikingUserKeyStore
|
||||||
|
|
||||||
|
|
||||||
|
class OpenVikingMemorySystemClient:
|
||||||
|
def __init__(self, store: OpenVikingUserKeyStore | None = None) -> None:
|
||||||
|
config = get_config()
|
||||||
|
self.base_url = config.openviking.url.rstrip("/")
|
||||||
|
self.root_key = config.openviking.api_key or "your-secret-root-key"
|
||||||
|
self.timeout = config.openviking.timeout
|
||||||
|
self.verify_ssl = config.openviking.verify_ssl
|
||||||
|
self.store = store or OpenVikingUserKeyStore(config.storage.sqlite_path)
|
||||||
|
|
||||||
|
async def health(self) -> dict[str, Any]:
|
||||||
|
async with self._client(self.root_key) as client:
|
||||||
|
response = await client.get("/health")
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
async def ensure_user(self, user_id: str) -> str:
|
||||||
|
existing = self.store.get_user_key(user_id)
|
||||||
|
if existing:
|
||||||
|
return existing
|
||||||
|
|
||||||
|
async with self._client(self.root_key) as client:
|
||||||
|
response = await client.post(
|
||||||
|
"/api/v1/admin/accounts",
|
||||||
|
json={"account_id": user_id, "admin_user_id": user_id},
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
data = response.json()
|
||||||
|
|
||||||
|
user_key = self._extract_user_key(data)
|
||||||
|
if not user_key:
|
||||||
|
raise RuntimeError("OpenViking did not return user_key")
|
||||||
|
self.store.save_user_key(user_id, user_key)
|
||||||
|
return user_key
|
||||||
|
|
||||||
|
async def ensure_session(self, user_key: str, session_id: str) -> dict[str, Any]:
|
||||||
|
async with self._client(user_key) as client:
|
||||||
|
response = await client.post("/api/v1/sessions", json={"session_id": session_id})
|
||||||
|
if response.status_code in {409, 422}:
|
||||||
|
return {"session_id": session_id, "status": "exists"}
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
async def append_message(self, user_key: str, session_id: str, role: str, content: str) -> dict[str, Any]:
|
||||||
|
async with self._client(user_key) as client:
|
||||||
|
response = await client.post(
|
||||||
|
f"/api/v1/sessions/{session_id}/messages",
|
||||||
|
json={"role": role, "content": content},
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
async def commit_session(self, user_key: str, session_id: str) -> dict[str, Any]:
|
||||||
|
async with self._client(user_key) as client:
|
||||||
|
response = await client.post(f"/api/v1/sessions/{session_id}/commit")
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
async def extract_session(self, user_key: str, session_id: str) -> dict[str, Any]:
|
||||||
|
async with self._client(user_key) as client:
|
||||||
|
response = await client.post(f"/api/v1/sessions/{session_id}/extract")
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
async def get_task(self, user_key: str, task_id: str) -> dict[str, Any]:
|
||||||
|
async with self._client(user_key) as client:
|
||||||
|
response = await client.get(f"/api/v1/tasks/{task_id}")
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
async def find(self, user_key: str, user_id: str, query: str, limit: int) -> dict[str, Any]:
|
||||||
|
async with self._client(user_key) as client:
|
||||||
|
response = await client.post(
|
||||||
|
"/api/v1/search/find",
|
||||||
|
json={
|
||||||
|
"query": query,
|
||||||
|
"target_uri": f"viking://user/{user_id}/memories/",
|
||||||
|
"limit": limit,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
async def search(self, user_key: str, session_id: str | None, query: str, limit: int) -> dict[str, Any]:
|
||||||
|
payload: dict[str, Any] = {"query": query, "limit": limit}
|
||||||
|
if session_id:
|
||||||
|
payload["session_id"] = session_id
|
||||||
|
async with self._client(user_key) as client:
|
||||||
|
response = await client.post("/api/v1/search/search", json=payload)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
def _client(self, api_key: str) -> httpx.AsyncClient:
|
||||||
|
return httpx.AsyncClient(
|
||||||
|
base_url=self.base_url,
|
||||||
|
headers={"X-API-Key": api_key, "Content-Type": "application/json"},
|
||||||
|
timeout=self.timeout,
|
||||||
|
verify=self.verify_ssl,
|
||||||
|
)
|
||||||
|
|
||||||
|
def _extract_user_key(self, data: dict[str, Any]) -> str | None:
|
||||||
|
result = data.get("result") if isinstance(data.get("result"), dict) else data
|
||||||
|
value = result.get("user_key") if isinstance(result, dict) else None
|
||||||
|
return str(value) if value else None
|
||||||
|
|
||||||
|
|
||||||
|
class EverOSMemorySystemClient:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
config = get_config()
|
||||||
|
self.base_url = config.everos.url.rstrip("/")
|
||||||
|
self.api_key = config.everos.api_key
|
||||||
|
self.timeout = config.everos.timeout
|
||||||
|
self.verify_ssl = config.everos.verify_ssl
|
||||||
|
self.health_path = config.everos.health_path
|
||||||
|
|
||||||
|
async def health(self) -> dict[str, Any]:
|
||||||
|
async with self._client() as client:
|
||||||
|
response = await client.get(self.health_path)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
async def append_message(self, user_id: str, session_id: str, role: str, content: str) -> dict[str, Any]:
|
||||||
|
async with self._client() as client:
|
||||||
|
response = await client.post(
|
||||||
|
"/api/v1/memories",
|
||||||
|
json=self.build_message_payload(user_id=user_id, session_id=session_id, role=role, content=content),
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
def build_message_payload(self, user_id: str, session_id: str, role: str, content: str) -> dict[str, Any]:
|
||||||
|
everos_role = "assistant" if role == "assistant" else "user"
|
||||||
|
sender_id = "assistant" if everos_role == "assistant" else user_id
|
||||||
|
timestamp = int(datetime.now(timezone.utc).timestamp() * 1000)
|
||||||
|
return {
|
||||||
|
"user_id": user_id,
|
||||||
|
"session_id": session_id,
|
||||||
|
"messages": [
|
||||||
|
{
|
||||||
|
"message_id": f"msg_{timestamp}",
|
||||||
|
"timestamp": timestamp,
|
||||||
|
"sender_id": sender_id,
|
||||||
|
"sender_name": sender_id,
|
||||||
|
"role": everos_role,
|
||||||
|
"content": content,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
async def flush(self, user_id: str, session_id: str) -> dict[str, Any]:
|
||||||
|
async with self._client() as client:
|
||||||
|
response = await client.post("/api/v1/memories/flush", json={"user_id": user_id, "session_id": session_id})
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
async def search(self, user_id: str, session_id: str | None, query: str, method: str, limit: int) -> dict[str, Any]:
|
||||||
|
filters: dict[str, Any] = {"user_id": user_id}
|
||||||
|
if session_id:
|
||||||
|
filters["session_id"] = session_id
|
||||||
|
async with self._client() as client:
|
||||||
|
response = await client.post(
|
||||||
|
"/api/v1/memories/search",
|
||||||
|
json={
|
||||||
|
"query": query,
|
||||||
|
"method": method,
|
||||||
|
"memory_types": ["episodic_memory", "profile", "raw_message"],
|
||||||
|
"filters": filters,
|
||||||
|
"top_k": limit,
|
||||||
|
"include_original_data": True,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
async def get_profile(self, user_id: str) -> dict[str, Any]:
|
||||||
|
async with self._client() as client:
|
||||||
|
response = await client.post(
|
||||||
|
"/api/v1/memories/get",
|
||||||
|
json={
|
||||||
|
"memory_type": "profile",
|
||||||
|
"filters": {"user_id": user_id},
|
||||||
|
"page": 1,
|
||||||
|
"page_size": 20,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
def _client(self) -> httpx.AsyncClient:
|
||||||
|
headers = {"Content-Type": "application/json"}
|
||||||
|
if self.api_key:
|
||||||
|
headers["X-API-Key"] = self.api_key
|
||||||
|
headers["Authorization"] = f"Bearer {self.api_key}"
|
||||||
|
return httpx.AsyncClient(
|
||||||
|
base_url=self.base_url,
|
||||||
|
headers=headers,
|
||||||
|
timeout=self.timeout,
|
||||||
|
verify=self.verify_ssl,
|
||||||
|
)
|
||||||
100
memory_system_api/config.py
Normal file
100
memory_system_api/config.py
Normal file
@ -0,0 +1,100 @@
|
|||||||
|
"""Configuration loading for Memory System API."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Literal
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
|
||||||
|
class ServerConfig(BaseModel):
|
||||||
|
host: str = "127.0.0.1"
|
||||||
|
port: int = 1934
|
||||||
|
api_key: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
class OpenVikingConfig(BaseModel):
|
||||||
|
url: str = "http://127.0.0.1:1933"
|
||||||
|
api_key: str = ""
|
||||||
|
timeout: int = 30
|
||||||
|
verify_ssl: bool = True
|
||||||
|
|
||||||
|
|
||||||
|
class EverOSConfig(BaseModel):
|
||||||
|
url: str = "http://127.0.0.1:1995"
|
||||||
|
api_key: str = ""
|
||||||
|
timeout: int = 30
|
||||||
|
verify_ssl: bool = True
|
||||||
|
health_path: str = "/health"
|
||||||
|
|
||||||
|
|
||||||
|
class StorageConfig(BaseModel):
|
||||||
|
sqlite_path: str = "/home/tom/memory-gateway/memory_system_api.sqlite3"
|
||||||
|
|
||||||
|
|
||||||
|
class LoggingConfig(BaseModel):
|
||||||
|
level: str = "INFO"
|
||||||
|
|
||||||
|
|
||||||
|
class Config(BaseModel):
|
||||||
|
server: ServerConfig = Field(default_factory=ServerConfig)
|
||||||
|
openviking: OpenVikingConfig = Field(default_factory=OpenVikingConfig)
|
||||||
|
everos: EverOSConfig = Field(default_factory=EverOSConfig)
|
||||||
|
storage: StorageConfig = Field(default_factory=StorageConfig)
|
||||||
|
logging: LoggingConfig = Field(default_factory=LoggingConfig)
|
||||||
|
|
||||||
|
|
||||||
|
_config: Config | None = None
|
||||||
|
|
||||||
|
|
||||||
|
def load_config(config_path: str | None = None) -> Config:
|
||||||
|
path = Path(config_path or os.environ.get("MEMORY_SYSTEM_CONFIG", "config.yaml"))
|
||||||
|
if not path.exists():
|
||||||
|
return _apply_env_overrides(Config())
|
||||||
|
with path.open("r", encoding="utf-8") as handle:
|
||||||
|
data = yaml.safe_load(handle) or {}
|
||||||
|
config = Config(
|
||||||
|
server=ServerConfig(**data.get("server", {})),
|
||||||
|
openviking=OpenVikingConfig(**data.get("openviking", {})),
|
||||||
|
everos=EverOSConfig(**data.get("everos", {})),
|
||||||
|
storage=StorageConfig(**data.get("storage", {})),
|
||||||
|
logging=LoggingConfig(**data.get("logging", {})),
|
||||||
|
)
|
||||||
|
return _apply_env_overrides(config)
|
||||||
|
|
||||||
|
|
||||||
|
def get_config() -> Config:
|
||||||
|
global _config
|
||||||
|
if _config is None:
|
||||||
|
_config = load_config()
|
||||||
|
return _config
|
||||||
|
|
||||||
|
|
||||||
|
def set_config(config: Config) -> None:
|
||||||
|
global _config
|
||||||
|
_config = config
|
||||||
|
|
||||||
|
|
||||||
|
def _apply_env_overrides(config: Config) -> Config:
|
||||||
|
updates: dict[str, dict[str, Any]] = {
|
||||||
|
"server": _env_updates("MEMORY_SYSTEM_SERVER", {"API_KEY": "api_key", "HOST": "host", "PORT": "port"}),
|
||||||
|
"openviking": _env_updates("OPENVIKING", {"URL": "url", "BASE_URL": "url", "API_KEY": "api_key", "TIMEOUT": "timeout"}),
|
||||||
|
"everos": _env_updates("EVEROS", {"URL": "url", "BASE_URL": "url", "API_KEY": "api_key", "TIMEOUT": "timeout"}),
|
||||||
|
"storage": _env_updates("MEMORY_SYSTEM_STORAGE", {"SQLITE_PATH": "sqlite_path"}),
|
||||||
|
}
|
||||||
|
for section, values in updates.items():
|
||||||
|
if values:
|
||||||
|
setattr(config, section, getattr(config, section).model_copy(update=values))
|
||||||
|
return config
|
||||||
|
|
||||||
|
|
||||||
|
def _env_updates(prefix: str, mapping: dict[str, str]) -> dict[str, Any]:
|
||||||
|
values: dict[str, Any] = {}
|
||||||
|
for env_name, field_name in mapping.items():
|
||||||
|
raw = os.environ.get(f"{prefix}_{env_name}")
|
||||||
|
if raw is None:
|
||||||
|
continue
|
||||||
|
values[field_name] = int(raw) if field_name in {"port", "timeout"} else raw
|
||||||
|
return values
|
||||||
64
memory_system_api/schemas.py
Normal file
64
memory_system_api/schemas.py
Normal file
@ -0,0 +1,64 @@
|
|||||||
|
"""Schemas for the lightweight Memory System API."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any, Literal
|
||||||
|
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
|
||||||
|
OperationStatus = Literal["success", "partial_success", "failed"]
|
||||||
|
|
||||||
|
|
||||||
|
class MessageIngestRequest(BaseModel):
|
||||||
|
user_id: str = Field(min_length=1)
|
||||||
|
session_id: str = Field(min_length=1)
|
||||||
|
user_message: str | None = None
|
||||||
|
assistant_message: str | None = None
|
||||||
|
timestamp: int | None = None
|
||||||
|
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class SessionUserRequest(BaseModel):
|
||||||
|
user_id: str = Field(min_length=1)
|
||||||
|
|
||||||
|
|
||||||
|
class SearchRequest(BaseModel):
|
||||||
|
user_id: str = Field(min_length=1)
|
||||||
|
session_id: str | None = None
|
||||||
|
query: str = Field(min_length=1)
|
||||||
|
use_llm: bool = False
|
||||||
|
limit: int = Field(default=10, ge=1, le=100)
|
||||||
|
|
||||||
|
|
||||||
|
class BackendStatus(BaseModel):
|
||||||
|
status: OperationStatus
|
||||||
|
result: Any = None
|
||||||
|
error: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class MessageIngestResponse(BaseModel):
|
||||||
|
status: OperationStatus
|
||||||
|
message_count: int
|
||||||
|
backends: dict[str, BackendStatus]
|
||||||
|
|
||||||
|
|
||||||
|
class CommitResponse(BaseModel):
|
||||||
|
status: OperationStatus
|
||||||
|
backends: dict[str, BackendStatus]
|
||||||
|
|
||||||
|
|
||||||
|
class ExtractResponse(BaseModel):
|
||||||
|
status: OperationStatus
|
||||||
|
backends: dict[str, BackendStatus]
|
||||||
|
|
||||||
|
|
||||||
|
class SearchResponse(BaseModel):
|
||||||
|
status: OperationStatus
|
||||||
|
items: list[dict[str, Any]] = Field(default_factory=list)
|
||||||
|
backends: dict[str, BackendStatus]
|
||||||
|
|
||||||
|
|
||||||
|
class ProfileResponse(BaseModel):
|
||||||
|
status: OperationStatus
|
||||||
|
profile: Any = None
|
||||||
|
backends: dict[str, BackendStatus]
|
||||||
49
memory_system_api/server.py
Normal file
49
memory_system_api/server.py
Normal file
@ -0,0 +1,49 @@
|
|||||||
|
"""Standalone FastAPI server for Memory System API."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from fastapi import FastAPI
|
||||||
|
from fastapi.middleware.cors import CORSMiddleware
|
||||||
|
|
||||||
|
from .api import router
|
||||||
|
from .config import Config, load_config, set_config
|
||||||
|
|
||||||
|
|
||||||
|
app = FastAPI(title="Memory System API", version="0.1.0")
|
||||||
|
app.add_middleware(
|
||||||
|
CORSMiddleware,
|
||||||
|
allow_origins=["*"],
|
||||||
|
allow_credentials=True,
|
||||||
|
allow_methods=["*"],
|
||||||
|
allow_headers=["*"],
|
||||||
|
)
|
||||||
|
app.include_router(router)
|
||||||
|
|
||||||
|
|
||||||
|
def create_app(config: Config | None = None) -> FastAPI:
|
||||||
|
if config:
|
||||||
|
set_config(config)
|
||||||
|
return app
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
import argparse
|
||||||
|
import uvicorn
|
||||||
|
|
||||||
|
parser = argparse.ArgumentParser(description="Memory System API")
|
||||||
|
parser.add_argument("--config", default="config.yaml", help="Config file path")
|
||||||
|
parser.add_argument("--host", default=None, help="Bind host")
|
||||||
|
parser.add_argument("--port", type=int, default=None, help="Bind port")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
config = load_config(args.config)
|
||||||
|
if args.host:
|
||||||
|
config.server.host = args.host
|
||||||
|
if args.port:
|
||||||
|
config.server.port = args.port
|
||||||
|
set_config(config)
|
||||||
|
|
||||||
|
uvicorn.run(app, host=config.server.host, port=config.server.port, log_level=config.logging.level.lower())
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
165
memory_system_api/service.py
Normal file
165
memory_system_api/service.py
Normal file
@ -0,0 +1,165 @@
|
|||||||
|
"""Orchestration for the lightweight Memory System API."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
from typing import Any, Awaitable, Callable
|
||||||
|
|
||||||
|
from .clients import EverOSMemorySystemClient, OpenVikingMemorySystemClient
|
||||||
|
from .schemas import (
|
||||||
|
BackendStatus,
|
||||||
|
CommitResponse,
|
||||||
|
ExtractResponse,
|
||||||
|
MessageIngestRequest,
|
||||||
|
MessageIngestResponse,
|
||||||
|
ProfileResponse,
|
||||||
|
SearchRequest,
|
||||||
|
SearchResponse,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class MemorySystemService:
|
||||||
|
def __init__(self, openviking: Any | None = None, everos: Any | None = None) -> None:
|
||||||
|
self.openviking = openviking or OpenVikingMemorySystemClient()
|
||||||
|
self.everos = everos or EverOSMemorySystemClient()
|
||||||
|
|
||||||
|
async def ingest_messages(self, request: MessageIngestRequest) -> MessageIngestResponse:
|
||||||
|
messages = self._messages_from_request(request)
|
||||||
|
if not messages:
|
||||||
|
raise ValueError("at least one message is required")
|
||||||
|
|
||||||
|
user_key = await self.openviking.ensure_user(request.user_id)
|
||||||
|
await self.openviking.ensure_session(user_key, request.session_id)
|
||||||
|
|
||||||
|
print("user_key:", user_key) # Debugging line to check the user_key value
|
||||||
|
|
||||||
|
async def write_openviking() -> list[dict[str, Any]]:
|
||||||
|
results = []
|
||||||
|
for message in messages:
|
||||||
|
results.append(
|
||||||
|
await self.openviking.append_message(user_key, request.session_id, message["role"], message["content"])
|
||||||
|
)
|
||||||
|
return results
|
||||||
|
|
||||||
|
async def write_everos() -> list[dict[str, Any]]:
|
||||||
|
results = []
|
||||||
|
for message in messages:
|
||||||
|
results.append(
|
||||||
|
await self.everos.append_message(request.user_id, request.session_id, message["role"], message["content"])
|
||||||
|
)
|
||||||
|
return results
|
||||||
|
|
||||||
|
backends = await self._run_backends(openviking=write_openviking, everos=write_everos)
|
||||||
|
return MessageIngestResponse(
|
||||||
|
status=self._aggregate_status(backends),
|
||||||
|
message_count=len(messages),
|
||||||
|
backends=backends,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def commit_session(self, user_id: str, session_id: str) -> CommitResponse:
|
||||||
|
user_key = await self.openviking.ensure_user(user_id)
|
||||||
|
|
||||||
|
async def commit_openviking() -> dict[str, Any]:
|
||||||
|
return await self.openviking.commit_session(user_key, session_id)
|
||||||
|
|
||||||
|
async def flush_everos() -> dict[str, Any]:
|
||||||
|
return await self.everos.flush(user_id, session_id)
|
||||||
|
|
||||||
|
backends = await self._run_backends(openviking=commit_openviking, everos=flush_everos)
|
||||||
|
return CommitResponse(status=self._aggregate_status(backends), backends=backends)
|
||||||
|
|
||||||
|
async def extract_session(self, user_id: str, session_id: str) -> ExtractResponse:
|
||||||
|
user_key = await self.openviking.ensure_user(user_id)
|
||||||
|
backends = {
|
||||||
|
"openviking": await self._capture(lambda: self.openviking.extract_session(user_key, session_id)),
|
||||||
|
}
|
||||||
|
return ExtractResponse(status=self._aggregate_status(backends), backends=backends)
|
||||||
|
|
||||||
|
async def get_openviking_task(self, user_id: str, task_id: str) -> dict[str, Any]:
|
||||||
|
user_key = await self.openviking.ensure_user(user_id)
|
||||||
|
return await self.openviking.get_task(user_key, task_id)
|
||||||
|
|
||||||
|
async def search(self, request: SearchRequest) -> SearchResponse:
|
||||||
|
user_key = await self.openviking.ensure_user(request.user_id)
|
||||||
|
everos_method = "agentic" if request.use_llm else "hybrid"
|
||||||
|
|
||||||
|
async def search_openviking() -> dict[str, Any]:
|
||||||
|
if request.use_llm:
|
||||||
|
return await self.openviking.search(user_key, request.session_id, request.query, request.limit)
|
||||||
|
return await self.openviking.find(user_key, request.user_id, request.query, request.limit)
|
||||||
|
|
||||||
|
async def search_everos() -> dict[str, Any]:
|
||||||
|
return await self.everos.search(
|
||||||
|
request.user_id,
|
||||||
|
request.session_id,
|
||||||
|
request.query,
|
||||||
|
everos_method,
|
||||||
|
request.limit,
|
||||||
|
)
|
||||||
|
|
||||||
|
backends = await self._run_backends(openviking=search_openviking, everos=search_everos)
|
||||||
|
items = self._merge_search_items(backends)
|
||||||
|
return SearchResponse(status=self._aggregate_status(backends), items=items[: request.limit], backends=backends)
|
||||||
|
|
||||||
|
async def get_profile(self, user_id: str) -> ProfileResponse:
|
||||||
|
backends = {"everos": await self._capture(lambda: self.everos.get_profile(user_id))}
|
||||||
|
profile = backends["everos"].result if backends["everos"].status == "success" else None
|
||||||
|
return ProfileResponse(status=self._aggregate_status(backends), profile=profile, backends=backends)
|
||||||
|
|
||||||
|
async def health(self) -> dict[str, Any]:
|
||||||
|
backends = await self._run_backends(openviking=self.openviking.health, everos=self.everos.health)
|
||||||
|
return {"status": self._aggregate_status(backends), "backends": backends}
|
||||||
|
|
||||||
|
def _messages_from_request(self, request: MessageIngestRequest) -> list[dict[str, str]]:
|
||||||
|
messages = []
|
||||||
|
if request.user_message:
|
||||||
|
messages.append({"role": "user", "content": request.user_message})
|
||||||
|
if request.assistant_message:
|
||||||
|
messages.append({"role": "assistant", "content": request.assistant_message})
|
||||||
|
return messages
|
||||||
|
|
||||||
|
async def _run_backends(self, **calls: Callable[[], Awaitable[Any]]) -> dict[str, BackendStatus]:
|
||||||
|
names = list(calls)
|
||||||
|
results = await asyncio.gather(*(self._capture(calls[name]) for name in names))
|
||||||
|
return dict(zip(names, results))
|
||||||
|
|
||||||
|
async def _capture(self, call: Callable[[], Awaitable[Any]]) -> BackendStatus:
|
||||||
|
try:
|
||||||
|
return BackendStatus(status="success", result=await call())
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
return BackendStatus(status="failed", error=str(exc))
|
||||||
|
|
||||||
|
def _aggregate_status(self, backends: dict[str, BackendStatus]) -> str:
|
||||||
|
statuses = {backend.status for backend in backends.values()}
|
||||||
|
if statuses == {"success"}:
|
||||||
|
return "success"
|
||||||
|
if "success" in statuses:
|
||||||
|
return "partial_success"
|
||||||
|
return "failed"
|
||||||
|
|
||||||
|
def _merge_search_items(self, backends: dict[str, BackendStatus]) -> list[dict[str, Any]]:
|
||||||
|
items: list[dict[str, Any]] = []
|
||||||
|
for backend_name, backend in backends.items():
|
||||||
|
if backend.status != "success":
|
||||||
|
continue
|
||||||
|
items.extend(self._items_from_backend_result(backend_name, backend.result))
|
||||||
|
return items
|
||||||
|
|
||||||
|
def _items_from_backend_result(self, backend_name: str, result: Any) -> list[dict[str, Any]]:
|
||||||
|
if isinstance(result, dict) and isinstance(result.get("items"), list):
|
||||||
|
return [self._with_backend(backend_name, item) for item in result["items"] if isinstance(item, dict)]
|
||||||
|
data = result.get("data") if isinstance(result, dict) and isinstance(result.get("data"), dict) else result
|
||||||
|
if not isinstance(data, dict):
|
||||||
|
return []
|
||||||
|
if isinstance(data.get("result"), dict):
|
||||||
|
data = data["result"]
|
||||||
|
raw_items: list[dict[str, Any]] = []
|
||||||
|
for key in ("memories", "resources", "episodes", "profiles", "raw_messages"):
|
||||||
|
values = data.get(key)
|
||||||
|
if isinstance(values, list):
|
||||||
|
raw_items.extend(item for item in values if isinstance(item, dict))
|
||||||
|
return [self._with_backend(backend_name, item) for item in raw_items]
|
||||||
|
|
||||||
|
def _with_backend(self, backend_name: str, item: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
if "source_backend" in item:
|
||||||
|
return item
|
||||||
|
return {"source_backend": backend_name, **item}
|
||||||
53
memory_system_api/store.py
Normal file
53
memory_system_api/store.py
Normal file
@ -0,0 +1,53 @@
|
|||||||
|
"""Small SQLite store for OpenViking user keys."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sqlite3
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
class OpenVikingUserKeyStore:
|
||||||
|
def __init__(self, sqlite_path: str) -> None:
|
||||||
|
self.sqlite_path = sqlite_path
|
||||||
|
self._ensure_table()
|
||||||
|
|
||||||
|
def get_user_key(self, user_id: str) -> str | None:
|
||||||
|
with self._connect() as conn:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT user_key FROM memory_system_openviking_users WHERE user_id = ?",
|
||||||
|
(user_id,),
|
||||||
|
).fetchone()
|
||||||
|
return str(row[0]) if row else None
|
||||||
|
|
||||||
|
def save_user_key(self, user_id: str, user_key: str) -> None:
|
||||||
|
now = datetime.now(timezone.utc).isoformat()
|
||||||
|
with self._connect() as conn:
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO memory_system_openviking_users (user_id, account_id, user_key, created_at, updated_at)
|
||||||
|
VALUES (?, ?, ?, ?, ?)
|
||||||
|
ON CONFLICT(user_id) DO UPDATE SET
|
||||||
|
user_key = excluded.user_key,
|
||||||
|
updated_at = excluded.updated_at
|
||||||
|
""",
|
||||||
|
(user_id, user_id, user_key, now, now),
|
||||||
|
)
|
||||||
|
|
||||||
|
def _ensure_table(self) -> None:
|
||||||
|
path = Path(self.sqlite_path)
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
with self._connect() as conn:
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
CREATE TABLE IF NOT EXISTS memory_system_openviking_users (
|
||||||
|
user_id TEXT PRIMARY KEY,
|
||||||
|
account_id TEXT NOT NULL,
|
||||||
|
user_key TEXT NOT NULL,
|
||||||
|
created_at TEXT NOT NULL,
|
||||||
|
updated_at TEXT NOT NULL
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
|
||||||
|
def _connect(self) -> sqlite3.Connection:
|
||||||
|
return sqlite3.connect(self.sqlite_path)
|
||||||
@ -1,18 +0,0 @@
|
|||||||
---
|
|
||||||
title: Generic Memory Gateway Upload Test
|
|
||||||
knowledge_type: reference
|
|
||||||
source_filename: memory-gateway-generic-upload.txt
|
|
||||||
created_at: 2026-04-29T09:53:46.674688+00:00
|
|
||||||
tags: [generic, memory-gateway, upload-test, agent-workflow, knowledge-management, openviking]
|
|
||||||
summary: "A domain-neutral document describing a generic agent memory workflow for uploading reference documents. The workflow involves retrieving relevant context, summarizing final conclusions, uploading reference documents, and committing reusable knowledge to OpenViking."
|
|
||||||
---
|
|
||||||
|
|
||||||
# Generic Memory Gateway Upload Test
|
|
||||||
|
|
||||||
# Generic Memory Gateway Upload Test
|
|
||||||
|
|
||||||
This document describes a generic agent memory workflow:
|
|
||||||
- retrieve relevant context
|
|
||||||
- summarize final conclusions
|
|
||||||
- upload reference documents
|
|
||||||
- commit reusable knowledge to OpenViking
|
|
||||||
@ -1,14 +0,0 @@
|
|||||||
---
|
|
||||||
title: Memory Gateway Migration Upload Script Retry
|
|
||||||
knowledge_type: migration_note
|
|
||||||
source_filename: memory-gateway-migration-upload.txt
|
|
||||||
created_at: 2026-04-29T10:01:47.830369+00:00
|
|
||||||
tags: [migration, memory-gateway, script-retry]
|
|
||||||
summary: "Verification document confirming that document upload functionality works correctly after migrating the Memory Gateway project to /home/tom/memory-gateway. This serves as a test of the upload script following the project relocation."
|
|
||||||
---
|
|
||||||
|
|
||||||
# Memory Gateway Migration Upload Script Retry
|
|
||||||
|
|
||||||
# Memory Gateway Migration Upload
|
|
||||||
|
|
||||||
This document verifies that document upload works after moving the project to /home/tom/memory-gateway.
|
|
||||||
@ -1,14 +0,0 @@
|
|||||||
---
|
|
||||||
title: Memory Gateway Migration Upload
|
|
||||||
knowledge_type: migration_note
|
|
||||||
source_filename: memory-gateway-migration-upload.txt
|
|
||||||
created_at: 2026-04-29T10:01:29.858006+00:00
|
|
||||||
tags: [migration, memory-gateway, verification, document-upload]
|
|
||||||
summary: "Document upload functionality verified as working after migrating the Memory Gateway project to /home/tom/memory-gateway. This serves as a verification test confirming the migration was successful."
|
|
||||||
---
|
|
||||||
|
|
||||||
# Memory Gateway Migration Upload
|
|
||||||
|
|
||||||
# Memory Gateway Migration Upload
|
|
||||||
|
|
||||||
This document verifies that document upload works after moving the project to /home/tom/memory-gateway.
|
|
||||||
@ -1,22 +0,0 @@
|
|||||||
---
|
|
||||||
type: agent_experience
|
|
||||||
agent_id:
|
|
||||||
visibility: agent-only
|
|
||||||
tags:
|
|
||||||
- memory/agent-experience
|
|
||||||
---
|
|
||||||
|
|
||||||
# Agent Experience - {{agent_id}}
|
|
||||||
|
|
||||||
## What Worked
|
|
||||||
|
|
||||||
-
|
|
||||||
|
|
||||||
## What Failed
|
|
||||||
|
|
||||||
-
|
|
||||||
|
|
||||||
## Tooling Notes
|
|
||||||
|
|
||||||
-
|
|
||||||
|
|
||||||
@ -1,26 +0,0 @@
|
|||||||
# {{title}}
|
|
||||||
|
|
||||||
---
|
|
||||||
type: knowledge
|
|
||||||
source: {{source}}
|
|
||||||
created: {{date}}
|
|
||||||
tags:
|
|
||||||
- memory-gateway
|
|
||||||
---
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
简要说明这份知识对后续 agent / harness 的可复用价值。
|
|
||||||
|
|
||||||
## Key Points
|
|
||||||
|
|
||||||
-
|
|
||||||
|
|
||||||
## Usage Notes
|
|
||||||
|
|
||||||
说明 agent 在什么场景下应该检索或引用这份知识。
|
|
||||||
|
|
||||||
## Source
|
|
||||||
|
|
||||||
- 原始来源：
|
|
||||||
- OpenViking URI：
|
|
||||||
@ -1,33 +0,0 @@
|
|||||||
---
|
|
||||||
type: long_term_memory
|
|
||||||
memory_id:
|
|
||||||
user_id:
|
|
||||||
workspace_id:
|
|
||||||
visibility: private
|
|
||||||
importance:
|
|
||||||
confidence:
|
|
||||||
source:
|
|
||||||
tags:
|
|
||||||
- memory/long-term
|
|
||||||
---
|
|
||||||
|
|
||||||
# {{summary}}
|
|
||||||
|
|
||||||
## Memory
|
|
||||||
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
|
|
||||||
## Evidence
|
|
||||||
|
|
||||||
- Source:
|
|
||||||
- Created:
|
|
||||||
- Version:
|
|
||||||
|
|
||||||
## Review
|
|
||||||
|
|
||||||
- Status: pending
|
|
||||||
- Reviewer:
|
|
||||||
- Decision:
|
|
||||||
|
|
||||||
@ -1,23 +0,0 @@
|
|||||||
---
|
|
||||||
type: memory_review
|
|
||||||
review_status: pending
|
|
||||||
tags:
|
|
||||||
- memory/review
|
|
||||||
---
|
|
||||||
|
|
||||||
# Memory Review - {{memory_id}}
|
|
||||||
|
|
||||||
## Candidate
|
|
||||||
|
|
||||||
|
|
||||||
## Proposed Action
|
|
||||||
|
|
||||||
- [ ] Accept
|
|
||||||
- [ ] Edit
|
|
||||||
- [ ] Reject
|
|
||||||
- [ ] Merge
|
|
||||||
- [ ] Archive
|
|
||||||
|
|
||||||
## Reason
|
|
||||||
|
|
||||||
|
|
||||||
@ -1,28 +0,0 @@
|
|||||||
---
|
|
||||||
type: user_profile
|
|
||||||
user_id:
|
|
||||||
visibility: private
|
|
||||||
tags:
|
|
||||||
- memory/profile
|
|
||||||
- visibility/private
|
|
||||||
---
|
|
||||||
|
|
||||||
# User Profile - {{user_id}}
|
|
||||||
|
|
||||||
## Stable Facts
|
|
||||||
|
|
||||||
-
|
|
||||||
|
|
||||||
## Preferences
|
|
||||||
|
|
||||||
-
|
|
||||||
|
|
||||||
## Working Style
|
|
||||||
|
|
||||||
-
|
|
||||||
|
|
||||||
## Evidence
|
|
||||||
|
|
||||||
| Memory ID | Evidence | Confidence | Updated |
|
|
||||||
|---|---|---:|---|
|
|
||||||
|
|
||||||
@ -1,23 +0,0 @@
|
|||||||
---
|
|
||||||
type: workspace_memory
|
|
||||||
workspace_id:
|
|
||||||
visibility: workspace-shared
|
|
||||||
tags:
|
|
||||||
- memory/workspace
|
|
||||||
- visibility/workspace-shared
|
|
||||||
---
|
|
||||||
|
|
||||||
# Workspace Memory - {{workspace_id}}
|
|
||||||
|
|
||||||
## Shared Decisions
|
|
||||||
|
|
||||||
-
|
|
||||||
|
|
||||||
## Project Knowledge
|
|
||||||
|
|
||||||
-
|
|
||||||
|
|
||||||
## Reusable Context
|
|
||||||
|
|
||||||
-
|
|
||||||
|
|
||||||
@ -1,15 +0,0 @@
|
|||||||
# Obsidian Vault
|
|
||||||
|
|
||||||
这个目录用于保存 Memory Gateway 的 Markdown 知识沉淀。
|
|
||||||
|
|
||||||
原则：
|
|
||||||
|
|
||||||
- 只存高价值、可人工维护的知识和总结。
|
|
||||||
- 不存全量原始资料。
|
|
||||||
- 不存密钥、凭证、私人敏感信息或无需长期保留的聊天流水。
|
|
||||||
- 上传文档默认进入 `01_Knowledge/Uploaded/`，再由 Memory Gateway 总结并写入 OpenViking。
|
|
||||||
|
|
||||||
当前结构：
|
|
||||||
|
|
||||||
- `01_Knowledge/Uploaded/`：上传文档转换后的 Markdown。
|
|
||||||
- `05_Templates/`：通用知识笔记模板。
|
|
||||||
@ -1,154 +0,0 @@
|
|||||||
# Memory Gateway Agent Plugin
|
|
||||||
|
|
||||||
This plugin is an adapter for the existing Memory Gateway. It is not Memory Gateway core and it does not import core service, repository, or server modules.
|
|
||||||
|
|
||||||
The plugin calls the existing HTTP API:
|
|
||||||
|
|
||||||
- `POST /v1/memory/search`
|
|
||||||
- `POST /v1/episodes`
|
|
||||||
- `POST /v1/sessions/{session_id}/commit`
|
|
||||||
- `POST /v1/memory`
|
|
||||||
- `POST /v1/memory/{memory_id}/feedback`
|
|
||||||
|
|
||||||
## Configuration
|
|
||||||
|
|
||||||
Environment variables:
|
|
||||||
|
|
||||||
- `MEMORY_GATEWAY_URL`, default `http://127.0.0.1:1934`
|
|
||||||
- `MEMORY_GATEWAY_API_KEY`, optional
|
|
||||||
- `MEMORY_GATEWAY_DEFAULT_USER_ID`, optional
|
|
||||||
- `MEMORY_GATEWAY_DEFAULT_AGENT_ID`, optional
|
|
||||||
- `MEMORY_GATEWAY_DEFAULT_WORKSPACE_ID`, optional
|
|
||||||
- `MEMORY_GATEWAY_AUTO_SEARCH`, default `true`
|
|
||||||
- `MEMORY_GATEWAY_AUTO_APPEND_EPISODE`, default `true`
|
|
||||||
- `MEMORY_GATEWAY_AUTO_COMMIT_SESSION`, default `false`
|
|
||||||
- `MEMORY_GATEWAY_REVIEW_MODE`, default `true`
|
|
||||||
- `MEMORY_GATEWAY_PLUGIN_DEBUG_RAW`, default `false`
|
|
||||||
- `MEMORY_GATEWAY_PLUGIN_TRACE_HOOKS`, default `false`
|
|
||||||
|
|
||||||
If an API key is configured, the plugin sends `X-API-Key`. It never logs the API key.
|
|
||||||
|
|
||||||
## Validation Status
|
|
||||||
|
|
||||||
| Check | Status |
|
|
||||||
| --- | --- |
|
|
||||||
| Mock unit tests | passed |
|
|
||||||
| Hermes plugin discovery | passed |
|
|
||||||
| Hermes tool registration | passed |
|
|
||||||
| Hermes hook registration | passed |
|
|
||||||
| Gateway E2E | passed |
|
|
||||||
| PluginManager.invoke_hook probe | passed |
|
|
||||||
| Real Hermes interactive session | passed |
|
|
||||||
| OpenClaw runtime validation | pending |
|
|
||||||
|
|
||||||
## Hermes
|
|
||||||
|
|
||||||
Use `hermes.plugin.yaml` as the Hermes-facing manifest. The entrypoint is:
|
|
||||||
|
|
||||||
```text
|
|
||||||
__init__:register
|
|
||||||
```
|
|
||||||
|
|
||||||
The plugin attempts to register tools and best-effort hooks. If the Hermes runtime does not expose hook registration, it still works in tools-only mode.
|
|
||||||
|
|
||||||
Install locally:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mkdir -p ~/.hermes/plugins
|
|
||||||
cd /opt/memory-gateway
|
|
||||||
ln -s "$(pwd)/plugins/memory-gateway-agent" ~/.hermes/plugins/memory-gateway-agent
|
|
||||||
hermes plugins enable memory-gateway-agent
|
|
||||||
hermes plugins list
|
|
||||||
hermes tools list
|
|
||||||
```
|
|
||||||
|
|
||||||
Example runtime configuration:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
export MEMORY_GATEWAY_URL=http://127.0.0.1:1934
|
|
||||||
export MEMORY_GATEWAY_API_KEY=
|
|
||||||
export MEMORY_GATEWAY_AUTO_SEARCH=true
|
|
||||||
export MEMORY_GATEWAY_AUTO_APPEND_EPISODE=true
|
|
||||||
export MEMORY_GATEWAY_AUTO_COMMIT_SESSION=false
|
|
||||||
```
|
|
||||||
|
|
||||||
## OpenClaw
|
|
||||||
|
|
||||||
`openclaw.plugin.yaml` is a best-effort draft manifest. Adjust field names to the actual OpenClaw runtime schema before production use.
|
|
||||||
|
|
||||||
## Tools-Only Mode
|
|
||||||
|
|
||||||
Agent runtimes can call:
|
|
||||||
|
|
||||||
- `memory_search`
|
|
||||||
- `memory_append_episode`
|
|
||||||
- `memory_commit_session`
|
|
||||||
- `memory_upsert`
|
|
||||||
- `memory_feedback`
|
|
||||||
|
|
||||||
Tools-only mode does not automatically remember anything. The agent policy must decide when to call tools.
|
|
||||||
|
|
||||||
## Lifecycle Hooks
|
|
||||||
|
|
||||||
Best-effort hooks:
|
|
||||||
|
|
||||||
- `on_session_start`: initializes session memory context without writing long-term memory.
|
|
||||||
- `pre_llm_call`: searches memory and returns compact memory context.
|
|
||||||
- `post_llm_call`: appends a safe candidate episode when policy allows it.
|
|
||||||
- `on_session_end`: commits session only when `MEMORY_GATEWAY_AUTO_COMMIT_SESSION=true`.
|
|
||||||
|
|
||||||
Plugin support for hooks depends on the agent runtime context API. This plugin should be described as tools-only plus best-effort hooks unless the target runtime has been verified.
|
|
||||||
|
|
||||||
Verified boundaries:
|
|
||||||
|
|
||||||
- Tools-only is usable through the registered `memory_gateway` toolset.
|
|
||||||
- `pre_llm_call` automatic search has passed hook-probe and real `hermes chat -Q -q` validation.
|
|
||||||
- `post_llm_call` automatic candidate episode append has passed hook-probe and real `hermes chat -Q -q` validation.
|
|
||||||
- `on_session_end` auto commit is off by default, stays off when `MEMORY_GATEWAY_AUTO_COMMIT_SESSION=false`, and commits when `MEMORY_GATEWAY_AUTO_COMMIT_SESSION=true`.
|
|
||||||
|
|
||||||
## Defaults
|
|
||||||
|
|
||||||
- Automatic search can be enabled.
|
|
||||||
- Automatic append episode can be enabled.
|
|
||||||
- Automatic commit is disabled by default.
|
|
||||||
- Automatic direct long-term upsert is disabled by default.
|
|
||||||
|
|
||||||
## Safety And Privacy
|
|
||||||
|
|
||||||
The plugin rejects memory writes containing passwords, API keys, bearer tokens, cookies, private keys, SSH keys, one-time verification codes, large logs, full raw transcripts, and chain-of-thought.
|
|
||||||
|
|
||||||
The plugin writes summarized candidate episodes. It does not store full raw conversations. Long-term memory should normally be produced by `memory_commit_session`, allowing Memory Gateway and EverOS to deduplicate, detect conflicts, and route review drafts.
|
|
||||||
|
|
||||||
Direct long-term `memory_upsert` is high risk and is not called automatically. If a user asks to forget or delete a memory, the agent should call `memory_feedback` or a delete-capable tool instead of silently keeping the memory.
|
|
||||||
|
|
||||||
Script output is redacted by default: no API key, headers, cookies, tokens, or raw result payloads are printed. Set `MEMORY_GATEWAY_PLUGIN_DEBUG_RAW=true` only for local debugging with non-sensitive test data.
|
|
||||||
|
|
||||||
Hook trace is disabled by default. Set `MEMORY_GATEWAY_PLUGIN_TRACE_HOOKS=true` to write minimal hook events to `plugins/memory-gateway-agent/.tmp/hook_trace.log`. Trace entries contain hook name, timestamp, shortened session id, Gateway action, and status only; they do not include user or assistant message bodies.
|
|
||||||
|
|
||||||
## Cleanup Test Data
|
|
||||||
|
|
||||||
Integration tests use:
|
|
||||||
|
|
||||||
- `user_id=test_user_memory_gateway_plugin`
|
|
||||||
- tags such as `integration_test`, `plugin`, and `safe_to_delete`
|
|
||||||
|
|
||||||
Run cleanup:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd /opt/memory-gateway
|
|
||||||
PYTHONPATH=plugins/memory-gateway-agent python plugins/memory-gateway-agent/scripts/cleanup_test_memories.py
|
|
||||||
```
|
|
||||||
|
|
||||||
The cleanup script refuses non-`test_user_` users. It first tries `DELETE /v1/memory/{memory_id}` for local test memories. If deletion fails, it falls back to `memory_feedback` with `incorrect`. Current cleanup is limited by the search API: it can only clean local `MemoryRecord` rows returned by search, not arbitrary OpenViking context rows.
|
|
||||||
|
|
||||||
## Local Smoke Test
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd /opt/memory-gateway
|
|
||||||
PYTHONPATH=plugins/memory-gateway-agent python plugins/memory-gateway-agent/scripts/health.py
|
|
||||||
PYTHONPATH=plugins/memory-gateway-agent python plugins/memory-gateway-agent/scripts/smoke_test.py
|
|
||||||
PYTHONPATH=plugins/memory-gateway-agent python plugins/memory-gateway-agent/scripts/hermes_smoke_check.py
|
|
||||||
PYTHONPATH=plugins/memory-gateway-agent python plugins/memory-gateway-agent/scripts/gateway_e2e_check.py
|
|
||||||
PYTHONPATH=plugins/memory-gateway-agent python plugins/memory-gateway-agent/scripts/hermes_hook_probe.py
|
|
||||||
PYTHONPATH=plugins/memory-gateway-agent python plugins/memory-gateway-agent/scripts/hermes_interactive_session_check.py
|
|
||||||
```
|
|
||||||
@ -1,120 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
try:
|
|
||||||
from . import schemas, tools
|
|
||||||
from .memory_gateway_plugin.config import load_config
|
|
||||||
from .memory_gateway_plugin import lifecycle
|
|
||||||
from .memory_gateway_plugin.trace import trace_hook
|
|
||||||
except ImportError:
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
_PLUGIN_ROOT = Path(__file__).resolve().parent
|
|
||||||
if str(_PLUGIN_ROOT) not in sys.path:
|
|
||||||
sys.path.insert(0, str(_PLUGIN_ROOT))
|
|
||||||
|
|
||||||
import schemas # type: ignore[no-redef]
|
|
||||||
import tools # type: ignore[no-redef]
|
|
||||||
from memory_gateway_plugin.config import load_config # type: ignore[no-redef]
|
|
||||||
from memory_gateway_plugin import lifecycle # type: ignore[no-redef]
|
|
||||||
from memory_gateway_plugin.trace import trace_hook # type: ignore[no-redef]
|
|
||||||
|
|
||||||
TOOLSET = "memory_gateway"
|
|
||||||
_LAST_USER_MESSAGES: dict[str, str] = {}
|
|
||||||
|
|
||||||
|
|
||||||
def _context_from_kwargs(kwargs: dict[str, Any]) -> dict[str, Any]:
|
|
||||||
cfg = load_config()
|
|
||||||
return {
|
|
||||||
"user_id": kwargs.get("user_id") or kwargs.get("user") or cfg.default_user_id,
|
|
||||||
"agent_id": kwargs.get("agent_id") or kwargs.get("agent") or cfg.default_agent_id or "hermes_agent",
|
|
||||||
"workspace_id": kwargs.get("workspace_id") or kwargs.get("workspace") or cfg.default_workspace_id,
|
|
||||||
"session_id": kwargs.get("session_id") or kwargs.get("task_id") or "",
|
|
||||||
"user_message": kwargs.get("user_message") or kwargs.get("prompt") or kwargs.get("message") or "",
|
|
||||||
"assistant_response": kwargs.get("assistant_response") or kwargs.get("response") or "",
|
|
||||||
"conversation_history": kwargs.get("conversation_history") or [],
|
|
||||||
"model": kwargs.get("model") or "",
|
|
||||||
"platform": kwargs.get("platform") or "",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def on_session_start(**kwargs: Any) -> dict[str, Any]:
|
|
||||||
session_id = kwargs.get("session_id") or kwargs.get("task_id") or ""
|
|
||||||
trace_hook("on_session_start", session_id=str(session_id), gateway_action="", gateway_called=False, ok=True)
|
|
||||||
return {
|
|
||||||
"status": "ok",
|
|
||||||
"memory_gateway": "session_initialized",
|
|
||||||
"session_id": session_id,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def pre_llm_call(**kwargs: Any) -> dict[str, str]:
|
|
||||||
context = _context_from_kwargs(kwargs)
|
|
||||||
if context.get("session_id") and context.get("user_message"):
|
|
||||||
_LAST_USER_MESSAGES[context["session_id"]] = context["user_message"]
|
|
||||||
result = lifecycle.on_conversation_start(context)
|
|
||||||
trace_hook(
|
|
||||||
"pre_llm_call",
|
|
||||||
session_id=context.get("session_id", ""),
|
|
||||||
gateway_action="memory_search",
|
|
||||||
gateway_called=bool(result.get("raw")),
|
|
||||||
ok=bool(result.get("ok")),
|
|
||||||
reason=str(result.get("error") or result.get("reason") or ""),
|
|
||||||
)
|
|
||||||
if not result.get("ok") or not result.get("memory_context"):
|
|
||||||
return {}
|
|
||||||
return {"context": "Relevant Memory Gateway context:\n" + result["memory_context"]}
|
|
||||||
|
|
||||||
|
|
||||||
def post_llm_call(**kwargs: Any) -> dict[str, Any] | None:
|
|
||||||
context = _context_from_kwargs(kwargs)
|
|
||||||
if not context.get("user_message") and context.get("session_id"):
|
|
||||||
context["user_message"] = _LAST_USER_MESSAGES.get(context["session_id"], "")
|
|
||||||
result = lifecycle.after_user_message(context)
|
|
||||||
trace_hook(
|
|
||||||
"post_llm_call",
|
|
||||||
session_id=context.get("session_id", ""),
|
|
||||||
gateway_action="append_episode",
|
|
||||||
gateway_called=bool(result.get("raw")),
|
|
||||||
ok=bool(result.get("ok")),
|
|
||||||
reason=str(result.get("error") or result.get("reason") or ""),
|
|
||||||
)
|
|
||||||
if result.get("ok"):
|
|
||||||
return None
|
|
||||||
return {"memory_gateway_error": result.get("error") or result.get("reason") or "append_failed"}
|
|
||||||
|
|
||||||
|
|
||||||
def on_session_end(**kwargs: Any) -> dict[str, Any] | None:
|
|
||||||
context = _context_from_kwargs(kwargs)
|
|
||||||
result = lifecycle.on_session_end(context)
|
|
||||||
trace_hook(
|
|
||||||
"on_session_end",
|
|
||||||
session_id=context.get("session_id", ""),
|
|
||||||
gateway_action="commit_session",
|
|
||||||
gateway_called=bool(result.get("raw")),
|
|
||||||
ok=bool(result.get("ok")),
|
|
||||||
reason=str(result.get("error") or result.get("reason") or ""),
|
|
||||||
)
|
|
||||||
if context.get("session_id"):
|
|
||||||
_LAST_USER_MESSAGES.pop(context["session_id"], None)
|
|
||||||
if result.get("ok"):
|
|
||||||
return None
|
|
||||||
return {"memory_gateway_error": result.get("error") or "commit_failed"}
|
|
||||||
|
|
||||||
|
|
||||||
def register(ctx: Any) -> None:
|
|
||||||
for name, schema in schemas.TOOL_SCHEMAS.items():
|
|
||||||
ctx.register_tool(
|
|
||||||
name=name,
|
|
||||||
toolset=TOOLSET,
|
|
||||||
schema=schema,
|
|
||||||
handler=tools.HANDLERS[name],
|
|
||||||
)
|
|
||||||
|
|
||||||
if hasattr(ctx, "register_hook"):
|
|
||||||
ctx.register_hook("on_session_start", on_session_start)
|
|
||||||
ctx.register_hook("pre_llm_call", pre_llm_call)
|
|
||||||
ctx.register_hook("post_llm_call", post_llm_call)
|
|
||||||
ctx.register_hook("on_session_end", on_session_end)
|
|
||||||
@ -1,41 +0,0 @@
|
|||||||
name: memory-gateway-agent
|
|
||||||
runtime: hermes
|
|
||||||
version: 0.1.0
|
|
||||||
description: Hermes plugin adapter for Memory Gateway v1. Provides tools-only mode plus best-effort lifecycle hooks.
|
|
||||||
entrypoint: register
|
|
||||||
provides_tools:
|
|
||||||
- memory_search
|
|
||||||
- memory_append_episode
|
|
||||||
- memory_commit_session
|
|
||||||
- memory_upsert
|
|
||||||
- memory_feedback
|
|
||||||
provides_hooks:
|
|
||||||
- on_session_start
|
|
||||||
- pre_llm_call
|
|
||||||
- post_llm_call
|
|
||||||
- on_session_end
|
|
||||||
env:
|
|
||||||
MEMORY_GATEWAY_URL: http://127.0.0.1:1934
|
|
||||||
MEMORY_GATEWAY_AUTO_SEARCH: "true"
|
|
||||||
MEMORY_GATEWAY_AUTO_APPEND_EPISODE: "true"
|
|
||||||
MEMORY_GATEWAY_AUTO_COMMIT_SESSION: "false"
|
|
||||||
tools:
|
|
||||||
memory_search:
|
|
||||||
description: Search Memory Gateway with user/agent/workspace/session ACL.
|
|
||||||
memory_append_episode:
|
|
||||||
description: Append a safe summarized candidate episode.
|
|
||||||
memory_commit_session:
|
|
||||||
description: Ask Gateway/EverOS to consolidate session episodes.
|
|
||||||
memory_upsert:
|
|
||||||
description: Upsert a stable memory through Gateway.
|
|
||||||
memory_feedback:
|
|
||||||
description: Send feedback for a memory record.
|
|
||||||
hooks:
|
|
||||||
on_session_start: __init__:on_session_start
|
|
||||||
pre_llm_call: __init__:pre_llm_call
|
|
||||||
post_llm_call: __init__:post_llm_call
|
|
||||||
on_session_end: __init__:on_session_end
|
|
||||||
notes:
|
|
||||||
- Hooks are best-effort and depend on the Hermes runtime context API.
|
|
||||||
- Without hook support, the plugin remains usable as tools-only.
|
|
||||||
- This plugin does not store full raw conversations.
|
|
||||||
@ -1,74 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import logging
|
|
||||||
from typing import Any, Callable
|
|
||||||
|
|
||||||
from . import lifecycle
|
|
||||||
from .tools import memory_append_episode, memory_commit_session, memory_feedback, memory_search, memory_upsert
|
|
||||||
|
|
||||||
_logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
__all__ = [
|
|
||||||
"register",
|
|
||||||
"memory_search",
|
|
||||||
"memory_append_episode",
|
|
||||||
"memory_commit_session",
|
|
||||||
"memory_upsert",
|
|
||||||
"memory_feedback",
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
TOOLS: dict[str, Callable[..., dict[str, Any]]] = {
|
|
||||||
"memory_search": memory_search,
|
|
||||||
"memory_append_episode": memory_append_episode,
|
|
||||||
"memory_commit_session": memory_commit_session,
|
|
||||||
"memory_upsert": memory_upsert,
|
|
||||||
"memory_feedback": memory_feedback,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _try_call(target: Any, method_names: list[str], *args: Any, **kwargs: Any) -> bool:
|
|
||||||
for name in method_names:
|
|
||||||
method = getattr(target, name, None)
|
|
||||||
if callable(method):
|
|
||||||
try:
|
|
||||||
method(*args, **kwargs)
|
|
||||||
return True
|
|
||||||
except TypeError:
|
|
||||||
try:
|
|
||||||
method(args[0], args[1])
|
|
||||||
return True
|
|
||||||
except Exception as exc:
|
|
||||||
_logger.debug("[_try_call] %s(%s, %s) failed: %s", name, args, kwargs, exc)
|
|
||||||
return False
|
|
||||||
except Exception as exc:
|
|
||||||
_logger.debug("[_try_call] %s(%s, %s) failed: %s", name, args, kwargs, exc)
|
|
||||||
return False
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def register(ctx: Any) -> dict[str, Any]:
|
|
||||||
registered_tools: list[str] = []
|
|
||||||
registered_hooks: list[str] = []
|
|
||||||
|
|
||||||
for name, func in TOOLS.items():
|
|
||||||
if _try_call(ctx, ["register_tool", "add_tool", "tool"], name, func):
|
|
||||||
registered_tools.append(name)
|
|
||||||
|
|
||||||
hook_map = {
|
|
||||||
"pre_llm_call": lifecycle.on_conversation_start,
|
|
||||||
"post_llm_call": lifecycle.after_user_message,
|
|
||||||
"session_end": lifecycle.on_session_end,
|
|
||||||
"after_task_complete": lifecycle.after_task_complete,
|
|
||||||
}
|
|
||||||
for name, func in hook_map.items():
|
|
||||||
if _try_call(ctx, ["register_hook", "add_hook", "hook"], name, func):
|
|
||||||
registered_hooks.append(name)
|
|
||||||
|
|
||||||
return {
|
|
||||||
"ok": True,
|
|
||||||
"mode": "tools-and-hooks" if registered_hooks else "tools-only" if registered_tools else "manual",
|
|
||||||
"registered_tools": registered_tools,
|
|
||||||
"registered_hooks": registered_hooks,
|
|
||||||
}
|
|
||||||
|
|
||||||
@ -1,101 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import logging
|
|
||||||
import time
|
|
||||||
import urllib.error
|
|
||||||
import urllib.request
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
from .config import PluginConfig, load_config
|
|
||||||
|
|
||||||
_logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
def _short_error(value: Any, max_chars: int = 500) -> str:
|
|
||||||
text = str(value).replace("\n", " ").strip()
|
|
||||||
return text[:max_chars]
|
|
||||||
|
|
||||||
|
|
||||||
class MemoryGatewayClient:
|
|
||||||
def __init__(self, config: PluginConfig | None = None) -> None:
|
|
||||||
self.config = config or load_config()
|
|
||||||
|
|
||||||
def _headers(self) -> dict[str, str]:
|
|
||||||
headers = {"Content-Type": "application/json"}
|
|
||||||
if self.config.api_key:
|
|
||||||
headers["X-API-Key"] = self.config.api_key
|
|
||||||
return headers
|
|
||||||
|
|
||||||
def _post(self, endpoint: str, payload: dict[str, Any], retries: int = 3, backoff: float = 1.0) -> dict[str, Any]:
|
|
||||||
url = self.config.gateway_url.rstrip("/") + endpoint
|
|
||||||
body = json.dumps(payload, ensure_ascii=False).encode("utf-8")
|
|
||||||
|
|
||||||
last_error: Exception | None = None
|
|
||||||
for attempt in range(retries):
|
|
||||||
request = urllib.request.Request(url, data=body, headers=self._headers(), method="POST")
|
|
||||||
try:
|
|
||||||
with urllib.request.urlopen(request, timeout=self.config.timeout) as response:
|
|
||||||
raw = response.read().decode("utf-8")
|
|
||||||
data = json.loads(raw) if raw else {}
|
|
||||||
return {
|
|
||||||
"ok": True,
|
|
||||||
"status_code": getattr(response, "status", 200),
|
|
||||||
"endpoint": endpoint,
|
|
||||||
"data": data,
|
|
||||||
}
|
|
||||||
except urllib.error.HTTPError as exc:
|
|
||||||
# Typically, client errors (4xx) shouldn't be retried unless specifically handled.
|
|
||||||
# Since HTTPError is a subclass of URLError, we catch it first.
|
|
||||||
if exc.code < 500 and exc.code != 429:
|
|
||||||
try:
|
|
||||||
body_text = exc.read().decode("utf-8")
|
|
||||||
except Exception:
|
|
||||||
body_text = exc.reason
|
|
||||||
_logger.error(f"HTTPError in _post to {endpoint}: {exc.code} {body_text}")
|
|
||||||
return {
|
|
||||||
"ok": False,
|
|
||||||
"status_code": exc.code,
|
|
||||||
"endpoint": endpoint,
|
|
||||||
"error": _short_error(body_text),
|
|
||||||
}
|
|
||||||
last_error = exc
|
|
||||||
except (urllib.error.URLError, TimeoutError, OSError) as exc:
|
|
||||||
last_error = exc
|
|
||||||
except Exception as exc:
|
|
||||||
_logger.error("Unexpected error in _post to %s: %s", endpoint, exc, exc_info=True)
|
|
||||||
return {
|
|
||||||
"ok": False,
|
|
||||||
"status_code": None,
|
|
||||||
"endpoint": endpoint,
|
|
||||||
"error": _short_error(exc),
|
|
||||||
}
|
|
||||||
|
|
||||||
if attempt < retries - 1:
|
|
||||||
time.sleep(backoff * (2 ** attempt))
|
|
||||||
|
|
||||||
# Exhausted retries
|
|
||||||
error_msg = str(last_error) if last_error else "Max retries exceeded"
|
|
||||||
_logger.error("Failed _post to %s after %d attempts. Last error: %s", endpoint, retries, last_error)
|
|
||||||
return {
|
|
||||||
"ok": False,
|
|
||||||
"status_code": None,
|
|
||||||
"endpoint": endpoint,
|
|
||||||
"error": error_msg,
|
|
||||||
}
|
|
||||||
|
|
||||||
def search_memory(self, payload: dict[str, Any]) -> dict[str, Any]:
|
|
||||||
return self._post("/v1/memory/search", payload)
|
|
||||||
|
|
||||||
def append_episode(self, payload: dict[str, Any]) -> dict[str, Any]:
|
|
||||||
return self._post("/v1/episodes", payload)
|
|
||||||
|
|
||||||
def commit_session(self, session_id: str, payload: dict[str, Any]) -> dict[str, Any]:
|
|
||||||
return self._post(f"/v1/sessions/{session_id}/commit", payload)
|
|
||||||
|
|
||||||
def upsert_memory(self, payload: dict[str, Any]) -> dict[str, Any]:
|
|
||||||
return self._post("/v1/memory", payload)
|
|
||||||
|
|
||||||
def send_feedback(self, memory_id: str, payload: dict[str, Any]) -> dict[str, Any]:
|
|
||||||
return self._post(f"/v1/memory/{memory_id}/feedback", payload)
|
|
||||||
|
|
||||||
@ -1,50 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
from dataclasses import dataclass
|
|
||||||
|
|
||||||
|
|
||||||
def _env_bool(name: str, default: bool) -> bool:
|
|
||||||
value = os.environ.get(name)
|
|
||||||
if value is None:
|
|
||||||
return default
|
|
||||||
return value.strip().lower() in {"1", "true", "yes", "on"}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class PluginConfig:
|
|
||||||
gateway_url: str = "http://127.0.0.1:1934"
|
|
||||||
api_key: str = ""
|
|
||||||
default_user_id: str = ""
|
|
||||||
default_agent_id: str = ""
|
|
||||||
default_workspace_id: str = ""
|
|
||||||
auto_search: bool = True
|
|
||||||
auto_append_episode: bool = True
|
|
||||||
auto_commit_session: bool = False
|
|
||||||
review_mode: bool = True
|
|
||||||
timeout: int = 30
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def from_env(cls) -> "PluginConfig":
|
|
||||||
try:
|
|
||||||
timeout_val = int(os.environ.get("MEMORY_GATEWAY_TIMEOUT", "30"))
|
|
||||||
except ValueError:
|
|
||||||
timeout_val = 30
|
|
||||||
|
|
||||||
return cls(
|
|
||||||
gateway_url=os.environ.get("MEMORY_GATEWAY_URL", cls.gateway_url).rstrip("/"),
|
|
||||||
api_key=os.environ.get("MEMORY_GATEWAY_API_KEY", ""),
|
|
||||||
default_user_id=os.environ.get("MEMORY_GATEWAY_DEFAULT_USER_ID", ""),
|
|
||||||
default_agent_id=os.environ.get("MEMORY_GATEWAY_DEFAULT_AGENT_ID", ""),
|
|
||||||
default_workspace_id=os.environ.get("MEMORY_GATEWAY_DEFAULT_WORKSPACE_ID", ""),
|
|
||||||
auto_search=_env_bool("MEMORY_GATEWAY_AUTO_SEARCH", True),
|
|
||||||
auto_append_episode=_env_bool("MEMORY_GATEWAY_AUTO_APPEND_EPISODE", True),
|
|
||||||
auto_commit_session=_env_bool("MEMORY_GATEWAY_AUTO_COMMIT_SESSION", False),
|
|
||||||
review_mode=_env_bool("MEMORY_GATEWAY_REVIEW_MODE", True),
|
|
||||||
timeout=timeout_val,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def load_config() -> PluginConfig:
|
|
||||||
return PluginConfig.from_env()
|
|
||||||
|
|
||||||
@ -1,109 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
from .client import MemoryGatewayClient
|
|
||||||
from .config import PluginConfig, load_config
|
|
||||||
from .policy import build_episode_summary, should_append_episode, should_commit_session, should_search_memory
|
|
||||||
from .tools import memory_append_episode, memory_commit_session, memory_search
|
|
||||||
|
|
||||||
|
|
||||||
def _get(context: dict[str, Any], key: str, default: str = "") -> str:
|
|
||||||
value = context.get(key, default)
|
|
||||||
return "" if value is None else str(value)
|
|
||||||
|
|
||||||
|
|
||||||
def compact_memory_context(search_result: dict[str, Any], limit: int = 5) -> str:
|
|
||||||
if not search_result.get("ok"):
|
|
||||||
return ""
|
|
||||||
data = search_result.get("data", {})
|
|
||||||
rows = []
|
|
||||||
for item in data.get("results", [])[:limit]:
|
|
||||||
memory = item.get("memory") or item.get("openviking") or {}
|
|
||||||
summary = memory.get("summary") or memory.get("abstract") or memory.get("content") or ""
|
|
||||||
namespace = memory.get("namespace", "")
|
|
||||||
memory_id = memory.get("id") or memory.get("uri") or ""
|
|
||||||
if summary:
|
|
||||||
rows.append(f"- {memory_id} [{namespace}]: {summary[:240]}")
|
|
||||||
return "\n".join(rows)
|
|
||||||
|
|
||||||
|
|
||||||
def on_conversation_start(context: dict[str, Any], client: MemoryGatewayClient | None = None, config: PluginConfig | None = None) -> dict[str, Any]:
|
|
||||||
cfg = config or load_config()
|
|
||||||
user_message = _get(context, "user_message") or _get(context, "query")
|
|
||||||
if not should_search_memory(user_message, context, cfg):
|
|
||||||
return {"ok": True, "memory_context": ""}
|
|
||||||
user_id = _get(context, "user_id", cfg.default_user_id)
|
|
||||||
if not user_id:
|
|
||||||
return {"ok": False, "error": "user_id_required"}
|
|
||||||
try:
|
|
||||||
limit_val = int(context.get("limit", 5))
|
|
||||||
except (ValueError, TypeError):
|
|
||||||
limit_val = 5
|
|
||||||
|
|
||||||
result = memory_search(
|
|
||||||
query=user_message,
|
|
||||||
user_id=user_id,
|
|
||||||
agent_id=_get(context, "agent_id", cfg.default_agent_id),
|
|
||||||
workspace_id=_get(context, "workspace_id", cfg.default_workspace_id),
|
|
||||||
session_id=_get(context, "session_id"),
|
|
||||||
limit=limit_val,
|
|
||||||
client=client,
|
|
||||||
)
|
|
||||||
return {"ok": result.get("ok", False), "memory_context": compact_memory_context(result), "raw": result}
|
|
||||||
|
|
||||||
|
|
||||||
def after_user_message(context: dict[str, Any], client: MemoryGatewayClient | None = None, config: PluginConfig | None = None) -> dict[str, Any]:
|
|
||||||
cfg = config or load_config()
|
|
||||||
user_message = _get(context, "user_message")
|
|
||||||
assistant_response = _get(context, "assistant_response")
|
|
||||||
if not should_append_episode(user_message, assistant_response, context, cfg):
|
|
||||||
return {"ok": True, "appended": False, "reason": "policy_skip"}
|
|
||||||
user_id = _get(context, "user_id", cfg.default_user_id)
|
|
||||||
session_id = _get(context, "session_id")
|
|
||||||
if not user_id or not session_id:
|
|
||||||
return {"ok": False, "error": "user_id_and_session_id_required"}
|
|
||||||
summary = build_episode_summary(user_message, assistant_response, context)
|
|
||||||
result = memory_append_episode(
|
|
||||||
user_id=user_id,
|
|
||||||
agent_id=_get(context, "agent_id", cfg.default_agent_id),
|
|
||||||
workspace_id=_get(context, "workspace_id", cfg.default_workspace_id),
|
|
||||||
session_id=session_id,
|
|
||||||
episode_summary=summary,
|
|
||||||
tags=["plugin-candidate"],
|
|
||||||
client=client,
|
|
||||||
)
|
|
||||||
return {"ok": result.get("ok", False), "appended": result.get("ok", False), "raw": result}
|
|
||||||
|
|
||||||
|
|
||||||
def after_task_complete(context: dict[str, Any], client: MemoryGatewayClient | None = None, config: PluginConfig | None = None) -> dict[str, Any]:
|
|
||||||
return _maybe_commit(context, client, config)
|
|
||||||
|
|
||||||
|
|
||||||
def on_session_end(context: dict[str, Any], client: MemoryGatewayClient | None = None, config: PluginConfig | None = None) -> dict[str, Any]:
|
|
||||||
return _maybe_commit(context, client, config)
|
|
||||||
|
|
||||||
|
|
||||||
def _maybe_commit(context: dict[str, Any], client: MemoryGatewayClient | None, config: PluginConfig | None) -> dict[str, Any]:
|
|
||||||
cfg = config or load_config()
|
|
||||||
if not should_commit_session(context, cfg):
|
|
||||||
return {"ok": True, "committed": False, "reason": "auto_commit_disabled"}
|
|
||||||
user_id = _get(context, "user_id", cfg.default_user_id)
|
|
||||||
session_id = _get(context, "session_id")
|
|
||||||
if not user_id or not session_id:
|
|
||||||
return {"ok": False, "error": "user_id_and_session_id_required"}
|
|
||||||
try:
|
|
||||||
min_importance_val = float(context.get("min_importance", 0.6))
|
|
||||||
except (ValueError, TypeError):
|
|
||||||
min_importance_val = 0.6
|
|
||||||
|
|
||||||
result = memory_commit_session(
|
|
||||||
user_id=user_id,
|
|
||||||
agent_id=_get(context, "agent_id", cfg.default_agent_id),
|
|
||||||
workspace_id=_get(context, "workspace_id", cfg.default_workspace_id),
|
|
||||||
session_id=session_id,
|
|
||||||
min_importance=min_importance_val,
|
|
||||||
client=client,
|
|
||||||
)
|
|
||||||
return {"ok": result.get("ok", False), "committed": result.get("ok", False), "raw": result}
|
|
||||||
|
|
||||||
@ -1,86 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
|
|
||||||
SENSITIVE_KEYS = ("api_key", "apikey", "authorization", "token", "cookie", "secret", "password", "x-api-key")
|
|
||||||
|
|
||||||
|
|
||||||
def debug_raw_enabled() -> bool:
|
|
||||||
return os.environ.get("MEMORY_GATEWAY_PLUGIN_DEBUG_RAW", "").strip().lower() in {"1", "true", "yes", "on"}
|
|
||||||
|
|
||||||
|
|
||||||
def short_id(value: Any, prefix: int = 8, suffix: int = 4) -> str:
|
|
||||||
text = "" if value is None else str(value)
|
|
||||||
if len(text) <= prefix + suffix + 3:
|
|
||||||
return text
|
|
||||||
return f"{text[:prefix]}...{text[-suffix:]}"
|
|
||||||
|
|
||||||
|
|
||||||
import re
|
|
||||||
|
|
||||||
def redact(value: Any) -> Any:
|
|
||||||
if isinstance(value, dict):
|
|
||||||
return {
|
|
||||||
key: ("<redacted>" if key.lower() in SENSITIVE_KEYS else redact(item))
|
|
||||||
for key, item in value.items()
|
|
||||||
}
|
|
||||||
if isinstance(value, list):
|
|
||||||
return [redact(item) for item in value]
|
|
||||||
if isinstance(value, str):
|
|
||||||
lowered = value.lower()
|
|
||||||
sensitive_markers = ("api_key=", "password=", "token=", "bearer ", "cookie:", "private key")
|
|
||||||
if any(marker in lowered for marker in sensitive_markers):
|
|
||||||
return "<redacted>"
|
|
||||||
return value
|
|
||||||
|
|
||||||
|
|
||||||
def summarize_data(data: Any) -> Any:
|
|
||||||
if debug_raw_enabled():
|
|
||||||
return redact(data)
|
|
||||||
if isinstance(data, list):
|
|
||||||
return {"count": len(data)}
|
|
||||||
if not isinstance(data, dict):
|
|
||||||
return data
|
|
||||||
if "results" in data:
|
|
||||||
return {
|
|
||||||
"count": len(data.get("results") or []),
|
|
||||||
"total": data.get("total"),
|
|
||||||
"local_total": data.get("local_total"),
|
|
||||||
"openviking_total": data.get("openviking_total"),
|
|
||||||
"searched_namespaces": data.get("searched_namespaces", []),
|
|
||||||
}
|
|
||||||
if "id" in data:
|
|
||||||
return {
|
|
||||||
"id": short_id(data.get("id")),
|
|
||||||
"namespace": data.get("namespace"),
|
|
||||||
"memory_type": data.get("memory_type"),
|
|
||||||
"source": data.get("source"),
|
|
||||||
}
|
|
||||||
if "memory_id" in data:
|
|
||||||
return {"status": data.get("status"), "memory_id": short_id(data.get("memory_id")), "feedback": data.get("feedback")}
|
|
||||||
if "promoted" in data or "consolidation" in data:
|
|
||||||
return {
|
|
||||||
"status": data.get("status"),
|
|
||||||
"promoted_count": len(data.get("promoted") or []),
|
|
||||||
"archived_count": len(data.get("archived_episode_ids") or []),
|
|
||||||
"consolidation_status": (data.get("consolidation") or {}).get("status") if isinstance(data.get("consolidation"), dict) else None,
|
|
||||||
}
|
|
||||||
allowed = {"ok", "status", "gateway", "service", "version", "healthy", "endpoint", "status_code", "error", "count"}
|
|
||||||
return {key: redact(value) for key, value in data.items() if key in allowed}
|
|
||||||
|
|
||||||
|
|
||||||
def summarize_result(result: dict[str, Any]) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"ok": bool(result.get("ok")),
|
|
||||||
"endpoint": result.get("endpoint"),
|
|
||||||
"status_code": result.get("status_code"),
|
|
||||||
"error": redact(result.get("error", "")),
|
|
||||||
"data": summarize_data(result.get("data")),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def dumps_safe(payload: Any, *, indent: int = 2) -> str:
|
|
||||||
return json.dumps(redact(payload), ensure_ascii=False, indent=indent, default=str)
|
|
||||||
@ -1,59 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import re
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
from .config import PluginConfig, load_config
|
|
||||||
from .safety import validate_memory_write
|
|
||||||
|
|
||||||
|
|
||||||
REMEMBER_RE = re.compile(r"记住|请保存|remember this|save this|keep in memory", re.I)
|
|
||||||
STABLE_SIGNAL_RE = re.compile(
|
|
||||||
r"偏好|长期|约束|架构决策|决策|结论|workflow|工作流|preference|constraint|decision|always|以后都|project fact",
|
|
||||||
re.I,
|
|
||||||
)
|
|
||||||
SMALL_TALK_RE = re.compile(r"^\s*(你好|hi|hello|谢谢|thanks|ok|好的|收到|再见)[。.!！\s\w]*$", re.I)
|
|
||||||
|
|
||||||
|
|
||||||
def should_search_memory(user_message: str, context: dict[str, Any] | None = None, config: PluginConfig | None = None) -> bool:
|
|
||||||
cfg = config or load_config()
|
|
||||||
if not cfg.auto_search:
|
|
||||||
return False
|
|
||||||
return bool(user_message and user_message.strip())
|
|
||||||
|
|
||||||
|
|
||||||
def should_append_episode(
|
|
||||||
user_message: str,
|
|
||||||
assistant_response: str = "",
|
|
||||||
context: dict[str, Any] | None = None,
|
|
||||||
config: PluginConfig | None = None,
|
|
||||||
) -> bool:
|
|
||||||
cfg = config or load_config()
|
|
||||||
if not cfg.auto_append_episode:
|
|
||||||
return False
|
|
||||||
combined = "\n".join(part for part in [user_message, assistant_response] if part)
|
|
||||||
if not combined.strip() or SMALL_TALK_RE.match(combined.strip()):
|
|
||||||
return False
|
|
||||||
if not validate_memory_write(combined)["allowed"]:
|
|
||||||
return False
|
|
||||||
return bool(REMEMBER_RE.search(combined) or STABLE_SIGNAL_RE.search(combined))
|
|
||||||
|
|
||||||
|
|
||||||
def build_episode_summary(user_message: str, assistant_response: str = "", context: dict[str, Any] | None = None) -> str:
|
|
||||||
parts = []
|
|
||||||
if REMEMBER_RE.search(user_message or ""):
|
|
||||||
parts.append(f"用户明确要求记住：{user_message.strip()}")
|
|
||||||
elif user_message:
|
|
||||||
parts.append(f"用户输入中的可复用信息：{user_message.strip()}")
|
|
||||||
if assistant_response and STABLE_SIGNAL_RE.search(assistant_response):
|
|
||||||
parts.append(f"助手结论：{assistant_response.strip()}")
|
|
||||||
summary = " ".join(parts).strip()
|
|
||||||
return summary[:1000]
|
|
||||||
|
|
||||||
|
|
||||||
def should_commit_session(context: dict[str, Any] | None = None, config: PluginConfig | None = None) -> bool:
|
|
||||||
cfg = config or load_config()
|
|
||||||
if cfg.auto_commit_session:
|
|
||||||
return True
|
|
||||||
return bool((context or {}).get("force_commit"))
|
|
||||||
|
|
||||||
@ -1,97 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import re
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
|
|
||||||
SECRET_PATTERNS = [
|
|
||||||
r"\bpassword\s*[:=]",
|
|
||||||
r"\bapi[_-]?key\s*[:=]",
|
|
||||||
r"\btoken\s*[:=]",
|
|
||||||
r"\bsecret\s*[:=]",
|
|
||||||
r"\bbearer\s+[a-z0-9._\-]{12,}",
|
|
||||||
r"\bcookie\s*[:=]",
|
|
||||||
r"\bsession[_ -]?id\s*[:=]",
|
|
||||||
r"-----BEGIN [A-Z ]*PRIVATE KEY-----",
|
|
||||||
r"\bssh-rsa\s+[a-z0-9+/=]{40,}",
|
|
||||||
r"\bone[- ]?time (?:password|code)\b",
|
|
||||||
r"\botp\s*[:=]?\s*\d{4,8}\b",
|
|
||||||
r"\b验证码\s*[:：]?\s*\d{4,8}\b",
|
|
||||||
]
|
|
||||||
|
|
||||||
CHAT_LINE_RE = re.compile(r"^\s*(user|assistant|system|用户|助手|模型|human|ai)\s*[:：]", re.I)
|
|
||||||
LOG_LINE_RE = re.compile(r"\b(ERROR|WARN|INFO|DEBUG|TRACE)\b|^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}")
|
|
||||||
CHAIN_OF_THOUGHT_RE = re.compile(r"chain[- ]of[- ]thought|逐步推理|隐藏推理|internal reasoning", re.I)
|
|
||||||
|
|
||||||
|
|
||||||
def detect_secret(content: str) -> tuple[bool, str]:
|
|
||||||
for pattern in SECRET_PATTERNS:
|
|
||||||
if re.search(pattern, content, re.I):
|
|
||||||
return True, "secret_like_content"
|
|
||||||
return False, ""
|
|
||||||
|
|
||||||
|
|
||||||
def detect_raw_transcript(content: str) -> tuple[bool, str]:
|
|
||||||
lines = [line for line in content.splitlines() if line.strip()]
|
|
||||||
chat_lines = sum(1 for line in lines if CHAT_LINE_RE.search(line))
|
|
||||||
if chat_lines >= 4:
|
|
||||||
return True, "raw_chat_transcript"
|
|
||||||
if "完整原始对话" in content or "full transcript" in content.lower():
|
|
||||||
return True, "raw_chat_transcript"
|
|
||||||
return False, ""
|
|
||||||
|
|
||||||
|
|
||||||
def detect_large_log(content: str) -> tuple[bool, str]:
|
|
||||||
lines = [line for line in content.splitlines() if line.strip()]
|
|
||||||
log_lines = sum(1 for line in lines if LOG_LINE_RE.search(line))
|
|
||||||
if len(content) > 4000 or len(lines) > 40 or log_lines >= 8:
|
|
||||||
return True, "large_or_raw_log"
|
|
||||||
return False, ""
|
|
||||||
|
|
||||||
|
|
||||||
def detect_low_value_memory(content: str) -> tuple[bool, str]:
|
|
||||||
normalized = re.sub(r"\s+", " ", content).strip().lower()
|
|
||||||
stable_signal = re.search(r"记住|偏好|长期|决策|结论|约束|preference|remember|decision|constraint", normalized, re.I)
|
|
||||||
if stable_signal:
|
|
||||||
return False, ""
|
|
||||||
if len(normalized) < 12:
|
|
||||||
return True, "too_short"
|
|
||||||
small_talk = {
|
|
||||||
"hi",
|
|
||||||
"hello",
|
|
||||||
"thanks",
|
|
||||||
"thank you",
|
|
||||||
"ok",
|
|
||||||
"好的",
|
|
||||||
"谢谢",
|
|
||||||
"你好",
|
|
||||||
"收到",
|
|
||||||
"再见",
|
|
||||||
}
|
|
||||||
if normalized in small_talk:
|
|
||||||
return True, "small_talk"
|
|
||||||
return False, ""
|
|
||||||
|
|
||||||
|
|
||||||
def sanitize_memory_content(content: str) -> str:
|
|
||||||
sanitized = content.strip()
|
|
||||||
sanitized = re.sub(r"\b(password|api[_-]?key|token|secret)\s*[:=]\s*\S+", r"\1=<redacted>", sanitized, flags=re.I)
|
|
||||||
sanitized = re.sub(r"\bbearer\s+[a-z0-9._\-]{12,}", "Bearer <redacted>", sanitized, flags=re.I)
|
|
||||||
sanitized = re.sub(r"-----BEGIN [A-Z ]*PRIVATE KEY-----.*?-----END [A-Z ]*PRIVATE KEY-----", "<redacted-private-key>", sanitized, flags=re.I | re.S)
|
|
||||||
return sanitized
|
|
||||||
|
|
||||||
|
|
||||||
def validate_memory_write(content: str, *, allow_low_value: bool = False) -> dict[str, Any]:
|
|
||||||
if not content or not content.strip():
|
|
||||||
return {"allowed": False, "reason": "empty_content", "sanitized_content": ""}
|
|
||||||
checks = [detect_secret, detect_raw_transcript, detect_large_log]
|
|
||||||
for check in checks:
|
|
||||||
blocked, reason = check(content)
|
|
||||||
if blocked:
|
|
||||||
return {"allowed": False, "reason": reason, "sanitized_content": ""}
|
|
||||||
if CHAIN_OF_THOUGHT_RE.search(content):
|
|
||||||
return {"allowed": False, "reason": "chain_of_thought", "sanitized_content": ""}
|
|
||||||
low_value, reason = detect_low_value_memory(content)
|
|
||||||
if low_value and not allow_low_value:
|
|
||||||
return {"allowed": False, "reason": reason, "sanitized_content": ""}
|
|
||||||
return {"allowed": True, "reason": "ok", "sanitized_content": sanitize_memory_content(content)}
|
|
||||||
@ -1,14 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class AgentContext:
|
|
||||||
user_id: str
|
|
||||||
agent_id: str = ""
|
|
||||||
workspace_id: str = ""
|
|
||||||
session_id: str = ""
|
|
||||||
metadata: dict[str, Any] = field(default_factory=dict)
|
|
||||||
|
|
||||||
@ -1,163 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
from .client import MemoryGatewayClient
|
|
||||||
from .config import PluginConfig, load_config
|
|
||||||
from .safety import validate_memory_write
|
|
||||||
|
|
||||||
|
|
||||||
FEEDBACK_MAP = {
|
|
||||||
"confirm": "useful",
|
|
||||||
"correct": "useful",
|
|
||||||
"useful": "useful",
|
|
||||||
"delete": "incorrect",
|
|
||||||
"reject": "incorrect",
|
|
||||||
"incorrect": "incorrect",
|
|
||||||
"duplicate": "duplicate",
|
|
||||||
"outdated": "outdated",
|
|
||||||
"not_useful": "not_useful",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _client(client: MemoryGatewayClient | None = None) -> MemoryGatewayClient:
|
|
||||||
return client or MemoryGatewayClient()
|
|
||||||
|
|
||||||
|
|
||||||
def _context_payload(user_id: str, agent_id: str = "", workspace_id: str = "", session_id: str = "") -> dict[str, Any]:
|
|
||||||
payload: dict[str, Any] = {"user_id": user_id}
|
|
||||||
if agent_id:
|
|
||||||
payload["agent_id"] = agent_id
|
|
||||||
if workspace_id:
|
|
||||||
payload["workspace_id"] = workspace_id
|
|
||||||
if session_id:
|
|
||||||
payload["session_id"] = session_id
|
|
||||||
return payload
|
|
||||||
|
|
||||||
|
|
||||||
def memory_search(
|
|
||||||
query: str,
|
|
||||||
user_id: str,
|
|
||||||
agent_id: str = "",
|
|
||||||
workspace_id: str = "",
|
|
||||||
session_id: str = "",
|
|
||||||
namespaces: list[str] | None = None,
|
|
||||||
memory_types: list[str] | None = None,
|
|
||||||
tags: list[str] | None = None,
|
|
||||||
limit: int = 5,
|
|
||||||
client: MemoryGatewayClient | None = None,
|
|
||||||
) -> dict[str, Any]:
|
|
||||||
if not query or not query.strip():
|
|
||||||
return {"ok": False, "error": "query_required"}
|
|
||||||
payload = _context_payload(user_id, agent_id, workspace_id, session_id)
|
|
||||||
payload.update(
|
|
||||||
{
|
|
||||||
"query": query.strip(),
|
|
||||||
"namespaces": namespaces or [],
|
|
||||||
"memory_types": memory_types or [],
|
|
||||||
"tags": tags or [],
|
|
||||||
"limit": limit,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
return _client(client).search_memory(payload)
|
|
||||||
|
|
||||||
|
|
||||||
def memory_append_episode(
|
|
||||||
user_id: str,
|
|
||||||
agent_id: str,
|
|
||||||
session_id: str,
|
|
||||||
content: str = "",
|
|
||||||
episode_summary: str = "",
|
|
||||||
workspace_id: str = "",
|
|
||||||
source: str = "conversation",
|
|
||||||
tags: list[str] | None = None,
|
|
||||||
importance: float | None = None,
|
|
||||||
confidence: float | None = None,
|
|
||||||
client: MemoryGatewayClient | None = None,
|
|
||||||
) -> dict[str, Any]:
|
|
||||||
candidate = (episode_summary or content or "").strip()
|
|
||||||
validation = validate_memory_write(candidate)
|
|
||||||
if not validation["allowed"]:
|
|
||||||
return {"ok": False, "error": "memory_write_rejected", "reason": validation["reason"]}
|
|
||||||
payload = _context_payload(user_id, agent_id, workspace_id, session_id)
|
|
||||||
payload.update({"content": validation["sanitized_content"], "tags": tags or [], "source": source})
|
|
||||||
if importance is not None:
|
|
||||||
payload["events"] = [{"type": "importance_hint", "value": importance}]
|
|
||||||
if confidence is not None:
|
|
||||||
payload.setdefault("events", []).append({"type": "confidence_hint", "value": confidence})
|
|
||||||
return _client(client).append_episode(payload)
|
|
||||||
|
|
||||||
|
|
||||||
def memory_commit_session(
|
|
||||||
user_id: str,
|
|
||||||
agent_id: str,
|
|
||||||
session_id: str,
|
|
||||||
workspace_id: str = "",
|
|
||||||
promote: bool = True,
|
|
||||||
min_importance: float = 0.6,
|
|
||||||
client: MemoryGatewayClient | None = None,
|
|
||||||
) -> dict[str, Any]:
|
|
||||||
payload = _context_payload(user_id, agent_id, workspace_id, session_id)
|
|
||||||
payload.update({"promote": promote, "min_importance": min_importance})
|
|
||||||
return _client(client).commit_session(session_id, payload)
|
|
||||||
|
|
||||||
|
|
||||||
def memory_upsert(
|
|
||||||
user_id: str,
|
|
||||||
agent_id: str,
|
|
||||||
content: str,
|
|
||||||
workspace_id: str = "",
|
|
||||||
namespace: str = "",
|
|
||||||
memory_type: str = "fact",
|
|
||||||
summary: str = "",
|
|
||||||
tags: list[str] | None = None,
|
|
||||||
importance: float = 0.5,
|
|
||||||
confidence: float = 0.8,
|
|
||||||
visibility: str = "private",
|
|
||||||
source: str = "agent",
|
|
||||||
client: MemoryGatewayClient | None = None,
|
|
||||||
) -> dict[str, Any]:
|
|
||||||
validation = validate_memory_write(content)
|
|
||||||
if not validation["allowed"]:
|
|
||||||
return {"ok": False, "error": "memory_write_rejected", "reason": validation["reason"]}
|
|
||||||
payload = _context_payload(user_id, agent_id, workspace_id)
|
|
||||||
payload.update(
|
|
||||||
{
|
|
||||||
"namespace": namespace or None,
|
|
||||||
"memory_type": memory_type,
|
|
||||||
"content": validation["sanitized_content"],
|
|
||||||
"summary": summary or None,
|
|
||||||
"tags": tags or [],
|
|
||||||
"importance": importance,
|
|
||||||
"confidence": confidence,
|
|
||||||
"visibility": visibility,
|
|
||||||
"source": source,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
return _client(client).upsert_memory(payload)
|
|
||||||
|
|
||||||
|
|
||||||
def memory_feedback(
|
|
||||||
user_id: str,
|
|
||||||
agent_id: str,
|
|
||||||
memory_id: str,
|
|
||||||
feedback: str,
|
|
||||||
workspace_id: str = "",
|
|
||||||
session_id: str = "",
|
|
||||||
comment: str = "",
|
|
||||||
client: MemoryGatewayClient | None = None,
|
|
||||||
) -> dict[str, Any]:
|
|
||||||
mapped_feedback = FEEDBACK_MAP.get(feedback, feedback)
|
|
||||||
payload = _context_payload(user_id, agent_id, workspace_id, session_id)
|
|
||||||
payload.update({"feedback": mapped_feedback, "comment": comment or None})
|
|
||||||
return _client(client).send_feedback(memory_id, payload)
|
|
||||||
|
|
||||||
|
|
||||||
def default_context(config: PluginConfig | None = None) -> dict[str, str]:
|
|
||||||
cfg = config or load_config()
|
|
||||||
return {
|
|
||||||
"user_id": cfg.default_user_id,
|
|
||||||
"agent_id": cfg.default_agent_id,
|
|
||||||
"workspace_id": cfg.default_workspace_id,
|
|
||||||
}
|
|
||||||
|
|
||||||
@ -1,45 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
from .output import redact, short_id
|
|
||||||
|
|
||||||
|
|
||||||
def trace_enabled() -> bool:
|
|
||||||
return os.environ.get("MEMORY_GATEWAY_PLUGIN_TRACE_HOOKS", "").strip().lower() in {"1", "true", "yes", "on"}
|
|
||||||
|
|
||||||
|
|
||||||
def trace_path() -> Path:
|
|
||||||
return Path(__file__).resolve().parents[1] / ".tmp" / "hook_trace.log"
|
|
||||||
|
|
||||||
|
|
||||||
def trace_hook(
|
|
||||||
hook_name: str,
|
|
||||||
*,
|
|
||||||
session_id: str = "",
|
|
||||||
gateway_action: str = "",
|
|
||||||
gateway_called: bool = False,
|
|
||||||
ok: bool | None = None,
|
|
||||||
audit_delta: int | None = None,
|
|
||||||
reason: str = "",
|
|
||||||
) -> None:
|
|
||||||
if not trace_enabled():
|
|
||||||
return
|
|
||||||
path = trace_path()
|
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
payload: dict[str, Any] = {
|
|
||||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
|
||||||
"hook": hook_name,
|
|
||||||
"session_id": short_id(session_id),
|
|
||||||
"gateway_action": gateway_action,
|
|
||||||
"gateway_called": gateway_called,
|
|
||||||
"ok": ok,
|
|
||||||
"audit_delta": audit_delta,
|
|
||||||
"reason": reason[:160] if reason else "",
|
|
||||||
}
|
|
||||||
with path.open("a", encoding="utf-8") as handle:
|
|
||||||
handle.write(json.dumps(redact(payload), ensure_ascii=False, default=str) + "\n")
|
|
||||||
@ -1,26 +0,0 @@
|
|||||||
name: memory-gateway-agent
|
|
||||||
runtime: openclaw
|
|
||||||
version: 0.1.0
|
|
||||||
description: Draft OpenClaw plugin manifest for Memory Gateway v1. Adjust field names to the actual OpenClaw runtime schema before production use.
|
|
||||||
entrypoint: memory_gateway_plugin:register
|
|
||||||
config:
|
|
||||||
gateway_url: ${MEMORY_GATEWAY_URL:-http://127.0.0.1:1934}
|
|
||||||
api_key_env: MEMORY_GATEWAY_API_KEY
|
|
||||||
tools:
|
|
||||||
- name: memory_search
|
|
||||||
- name: memory_append_episode
|
|
||||||
- name: memory_commit_session
|
|
||||||
- name: memory_upsert
|
|
||||||
- name: memory_feedback
|
|
||||||
hooks:
|
|
||||||
- name: pre_llm_call
|
|
||||||
handler: memory_gateway_plugin.lifecycle:on_conversation_start
|
|
||||||
- name: post_llm_call
|
|
||||||
handler: memory_gateway_plugin.lifecycle:after_user_message
|
|
||||||
- name: session_end
|
|
||||||
handler: memory_gateway_plugin.lifecycle:on_session_end
|
|
||||||
safety:
|
|
||||||
stores_full_raw_conversation: false
|
|
||||||
rejects_secrets: true
|
|
||||||
long_term_commit_via_everos: true
|
|
||||||
|
|
||||||
@ -1,55 +0,0 @@
|
|||||||
name: memory-gateway-agent
|
|
||||||
version: 0.1.0
|
|
||||||
description: Generic AI Agent plugin adapter for the existing Memory Gateway v1 HTTP API.
|
|
||||||
type: agent-plugin
|
|
||||||
provides_tools:
|
|
||||||
- memory_search
|
|
||||||
- memory_append_episode
|
|
||||||
- memory_commit_session
|
|
||||||
- memory_upsert
|
|
||||||
- memory_feedback
|
|
||||||
provides_hooks:
|
|
||||||
- pre_llm_call
|
|
||||||
- post_llm_call
|
|
||||||
- session_end
|
|
||||||
- after_task_complete
|
|
||||||
requires_env: []
|
|
||||||
entrypoint: register
|
|
||||||
transport:
|
|
||||||
type: http
|
|
||||||
target: ${MEMORY_GATEWAY_URL:-http://127.0.0.1:1934}
|
|
||||||
privacy:
|
|
||||||
stores_full_raw_conversation: false
|
|
||||||
writes_long_term_directly_by_default: false
|
|
||||||
auto_commit_session_default: false
|
|
||||||
configuration:
|
|
||||||
MEMORY_GATEWAY_URL:
|
|
||||||
default: http://127.0.0.1:1934
|
|
||||||
MEMORY_GATEWAY_API_KEY:
|
|
||||||
secret: true
|
|
||||||
required: false
|
|
||||||
MEMORY_GATEWAY_DEFAULT_USER_ID:
|
|
||||||
required: false
|
|
||||||
MEMORY_GATEWAY_DEFAULT_AGENT_ID:
|
|
||||||
required: false
|
|
||||||
MEMORY_GATEWAY_DEFAULT_WORKSPACE_ID:
|
|
||||||
required: false
|
|
||||||
MEMORY_GATEWAY_AUTO_SEARCH:
|
|
||||||
default: "true"
|
|
||||||
MEMORY_GATEWAY_AUTO_APPEND_EPISODE:
|
|
||||||
default: "true"
|
|
||||||
MEMORY_GATEWAY_AUTO_COMMIT_SESSION:
|
|
||||||
default: "false"
|
|
||||||
MEMORY_GATEWAY_REVIEW_MODE:
|
|
||||||
default: "true"
|
|
||||||
tools:
|
|
||||||
- memory_search
|
|
||||||
- memory_append_episode
|
|
||||||
- memory_commit_session
|
|
||||||
- memory_upsert
|
|
||||||
- memory_feedback
|
|
||||||
hooks:
|
|
||||||
- pre_llm_call
|
|
||||||
- post_llm_call
|
|
||||||
- session_end
|
|
||||||
- after_task_complete
|
|
||||||
@ -1,34 +0,0 @@
|
|||||||
# Memory Gateway Agent Policy
|
|
||||||
|
|
||||||
Use Memory Gateway as a shared memory adapter. It is not a transcript store.
|
|
||||||
|
|
||||||
At conversation start:
|
|
||||||
|
|
||||||
- Search memory when previous context may matter.
|
|
||||||
- Use `memory_search` with the current `user_id`, `agent_id`, `workspace_id`, and `session_id`.
|
|
||||||
- Inject only compact relevant memory summaries into the working context.
|
|
||||||
|
|
||||||
During a task:
|
|
||||||
|
|
||||||
- Write only candidate episode summaries with `memory_append_episode`.
|
|
||||||
- Save stable preferences, long-term project facts, architecture decisions, durable constraints, reusable workflows, and completed task conclusions.
|
|
||||||
- Do not save complete raw conversations, chain-of-thought, large logs, one-time values, or secrets.
|
|
||||||
|
|
||||||
At task or session completion:
|
|
||||||
|
|
||||||
- Use `memory_commit_session` to let Memory Gateway and EverOS decide what can be promoted.
|
|
||||||
- Do not promote all episodes directly to long-term memory.
|
|
||||||
- Conflicting or high-value memories should enter review rather than overwrite existing memory.
|
|
||||||
|
|
||||||
When the user says to forget or reject memory:
|
|
||||||
|
|
||||||
- Use `memory_feedback` with `incorrect`, `outdated`, or `not_useful`.
|
|
||||||
- Use delete-capable tools only when the runtime exposes them and access control allows it.
|
|
||||||
|
|
||||||
Default automation:
|
|
||||||
|
|
||||||
- Auto search may be enabled.
|
|
||||||
- Auto append episode may be enabled for safe summaries.
|
|
||||||
- Auto commit is disabled by default.
|
|
||||||
- Auto direct long-term upsert is disabled by default.
|
|
||||||
|
|
||||||
@ -1,24 +0,0 @@
|
|||||||
# Memory Gateway Safety Filter
|
|
||||||
|
|
||||||
The plugin must reject memory writes that contain:
|
|
||||||
|
|
||||||
- passwords
|
|
||||||
- API keys
|
|
||||||
- tokens
|
|
||||||
- secrets
|
|
||||||
- bearer tokens
|
|
||||||
- cookies
|
|
||||||
- session IDs
|
|
||||||
- private keys
|
|
||||||
- SSH keys
|
|
||||||
- one-time passwords or verification codes
|
|
||||||
- large raw logs
|
|
||||||
- full chat transcripts
|
|
||||||
- chain-of-thought or hidden reasoning
|
|
||||||
- unconfirmed sensitive personal attributes
|
|
||||||
- low-value temporary chatter
|
|
||||||
|
|
||||||
The plugin stores summaries rather than raw messages. If a message is useful but contains sensitive detail, redact the sensitive detail before writing. If redaction would remove the meaning, reject the write.
|
|
||||||
|
|
||||||
Long-term memory should normally be created by session commit and EverOS consolidation, not by direct upsert.
|
|
||||||
|
|
||||||
@ -1,107 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
|
|
||||||
MEMORY_SEARCH = {
|
|
||||||
"name": "memory_search",
|
|
||||||
"description": "Search accessible Memory Gateway records for the current user/agent context.",
|
|
||||||
"parameters": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"query": {"type": "string", "description": "Search query. Must not be empty."},
|
|
||||||
"user_id": {"type": "string", "description": "Memory Gateway user id."},
|
|
||||||
"agent_id": {"type": "string", "description": "Calling agent id, for ACL and namespace routing."},
|
|
||||||
"workspace_id": {"type": "string", "description": "Optional workspace/project id."},
|
|
||||||
"session_id": {"type": "string", "description": "Optional session id for session-scoped context."},
|
|
||||||
"namespaces": {"type": "array", "items": {"type": "string"}, "description": "Optional namespace filters."},
|
|
||||||
"memory_types": {"type": "array", "items": {"type": "string"}, "description": "Optional memory type filters."},
|
|
||||||
"tags": {"type": "array", "items": {"type": "string"}, "description": "Optional tag filters."},
|
|
||||||
"limit": {"type": "integer", "description": "Maximum result count.", "default": 5},
|
|
||||||
},
|
|
||||||
"required": ["query", "user_id", "agent_id"],
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
MEMORY_APPEND_EPISODE = {
|
|
||||||
"name": "memory_append_episode",
|
|
||||||
"description": "Append a safe summarized candidate episode. Does not save full raw conversation or directly promote long-term memory.",
|
|
||||||
"parameters": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"content": {"type": "string", "description": "Safe summarized episode content. Do not pass raw transcripts."},
|
|
||||||
"episode_summary": {"type": "string", "description": "Optional prebuilt summary. Used instead of content when provided."},
|
|
||||||
"user_id": {"type": "string", "description": "Memory Gateway user id."},
|
|
||||||
"agent_id": {"type": "string", "description": "Calling agent id."},
|
|
||||||
"workspace_id": {"type": "string", "description": "Optional workspace/project id."},
|
|
||||||
"session_id": {"type": "string", "description": "Current session id."},
|
|
||||||
"tags": {"type": "array", "items": {"type": "string"}, "description": "Optional tags."},
|
|
||||||
"source": {"type": "string", "description": "Source label.", "default": "conversation"},
|
|
||||||
},
|
|
||||||
"required": ["content", "user_id", "agent_id", "session_id"],
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
MEMORY_COMMIT_SESSION = {
|
|
||||||
"name": "memory_commit_session",
|
|
||||||
"description": "Commit a session through Memory Gateway and EverOS. Promotes only what consolidation accepts.",
|
|
||||||
"parameters": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"user_id": {"type": "string", "description": "Memory Gateway user id."},
|
|
||||||
"agent_id": {"type": "string", "description": "Calling agent id."},
|
|
||||||
"workspace_id": {"type": "string", "description": "Optional workspace/project id."},
|
|
||||||
"session_id": {"type": "string", "description": "Session id to commit."},
|
|
||||||
"promote": {"type": "boolean", "description": "Whether promotion is allowed.", "default": True},
|
|
||||||
"min_importance": {"type": "number", "description": "Minimum importance threshold.", "default": 0.6},
|
|
||||||
},
|
|
||||||
"required": ["user_id", "agent_id", "session_id"],
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
MEMORY_UPSERT = {
|
|
||||||
"name": "memory_upsert",
|
|
||||||
"description": "High-risk direct memory write. Use only for stable, concise, user-approved long-term memory; do not call automatically.",
|
|
||||||
"parameters": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"user_id": {"type": "string", "description": "Memory Gateway user id."},
|
|
||||||
"agent_id": {"type": "string", "description": "Calling agent id."},
|
|
||||||
"workspace_id": {"type": "string", "description": "Optional workspace/project id."},
|
|
||||||
"namespace": {"type": "string", "description": "Optional explicit namespace, e.g. user/{user_id}/long_term."},
|
|
||||||
"memory_type": {"type": "string", "description": "Memory type, e.g. preference, decision, fact, procedure."},
|
|
||||||
"content": {"type": "string", "description": "Stable memory content. Do not pass full raw conversation."},
|
|
||||||
"summary": {"type": "string", "description": "Optional concise summary."},
|
|
||||||
"tags": {"type": "array", "items": {"type": "string"}, "description": "Optional tags."},
|
|
||||||
"importance": {"type": "number", "description": "Importance score 0..1.", "default": 0.5},
|
|
||||||
"confidence": {"type": "number", "description": "Confidence score 0..1.", "default": 0.8},
|
|
||||||
"visibility": {"type": "string", "description": "Memory visibility.", "default": "private"},
|
|
||||||
},
|
|
||||||
"required": ["user_id", "agent_id", "content", "memory_type"],
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
MEMORY_FEEDBACK = {
|
|
||||||
"name": "memory_feedback",
|
|
||||||
"description": "Send quality feedback for an existing memory record.",
|
|
||||||
"parameters": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"memory_id": {"type": "string", "description": "Memory id to mark."},
|
|
||||||
"user_id": {"type": "string", "description": "Memory Gateway user id."},
|
|
||||||
"agent_id": {"type": "string", "description": "Calling agent id."},
|
|
||||||
"workspace_id": {"type": "string", "description": "Optional workspace/project id."},
|
|
||||||
"session_id": {"type": "string", "description": "Optional session id."},
|
|
||||||
"feedback": {"type": "string", "description": "Feedback, e.g. confirm, correct, delete, reject, incorrect, duplicate, outdated."},
|
|
||||||
"comment": {"type": "string", "description": "Optional feedback comment."},
|
|
||||||
},
|
|
||||||
"required": ["memory_id", "user_id", "agent_id", "feedback"],
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
TOOL_SCHEMAS = {
|
|
||||||
"memory_search": MEMORY_SEARCH,
|
|
||||||
"memory_append_episode": MEMORY_APPEND_EPISODE,
|
|
||||||
"memory_commit_session": MEMORY_COMMIT_SESSION,
|
|
||||||
"memory_upsert": MEMORY_UPSERT,
|
|
||||||
"memory_feedback": MEMORY_FEEDBACK,
|
|
||||||
}
|
|
||||||
|
|
||||||
@ -1,187 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
import urllib.error
|
|
||||||
import urllib.parse
|
|
||||||
import urllib.request
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
PLUGIN_ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
if str(PLUGIN_ROOT) not in sys.path:
|
|
||||||
sys.path.insert(0, str(PLUGIN_ROOT))
|
|
||||||
|
|
||||||
from memory_gateway_plugin.output import dumps_safe, short_id, summarize_data
|
|
||||||
|
|
||||||
|
|
||||||
USER_ID = "test_user_memory_gateway_plugin"
|
|
||||||
AGENT_ID = "test_hermes_memory_gateway_plugin"
|
|
||||||
WORKSPACE_ID = "test_workspace_memory_gateway_plugin"
|
|
||||||
SESSION_ID = "test_session_memory_gateway_plugin_001"
|
|
||||||
|
|
||||||
|
|
||||||
def _assert_test_user(user_id: str) -> None:
|
|
||||||
if not user_id.startswith("test_user_"):
|
|
||||||
raise ValueError("cleanup_refuses_non_test_user")
|
|
||||||
|
|
||||||
|
|
||||||
def _gateway_url() -> str:
|
|
||||||
return os.environ.get("MEMORY_GATEWAY_URL", "http://127.0.0.1:1934").rstrip("/")
|
|
||||||
|
|
||||||
|
|
||||||
def _api_key() -> str:
|
|
||||||
return os.environ.get("MEMORY_GATEWAY_API_KEY", "")
|
|
||||||
|
|
||||||
|
|
||||||
def _request(method: str, endpoint: str, payload: dict[str, Any] | None = None) -> dict[str, Any]:
|
|
||||||
headers = {"Content-Type": "application/json"}
|
|
||||||
if _api_key():
|
|
||||||
headers["X-API-Key"] = _api_key()
|
|
||||||
body = None if payload is None else json.dumps(payload, ensure_ascii=False).encode("utf-8")
|
|
||||||
req = urllib.request.Request(_gateway_url() + endpoint, data=body, headers=headers, method=method)
|
|
||||||
try:
|
|
||||||
with urllib.request.urlopen(req, timeout=15) as response:
|
|
||||||
raw = response.read().decode("utf-8")
|
|
||||||
return {"ok": True, "status_code": getattr(response, "status", 200), "data": json.loads(raw) if raw else {}}
|
|
||||||
except urllib.error.HTTPError as exc:
|
|
||||||
return {"ok": False, "status_code": exc.code, "error": str(exc.reason)[:300]}
|
|
||||||
except Exception as exc:
|
|
||||||
return {"ok": False, "status_code": None, "error": str(exc)[:300]}
|
|
||||||
|
|
||||||
|
|
||||||
def _search_candidates() -> dict[str, Any]:
|
|
||||||
return _request(
|
|
||||||
"POST",
|
|
||||||
"/v1/memory/search",
|
|
||||||
{
|
|
||||||
"query": "integration_test plugin safe_to_delete Memory Gateway plugin",
|
|
||||||
"user_id": USER_ID,
|
|
||||||
"agent_id": AGENT_ID,
|
|
||||||
"workspace_id": WORKSPACE_ID,
|
|
||||||
"session_id": SESSION_ID,
|
|
||||||
"tags": ["integration_test"],
|
|
||||||
"limit": 100,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _audit_candidate_ids() -> list[str]:
|
|
||||||
result = _request("GET", "/v1/audit?limit=1000")
|
|
||||||
if not result.get("ok"):
|
|
||||||
return []
|
|
||||||
ids: list[str] = []
|
|
||||||
for row in result.get("data") or []:
|
|
||||||
if row.get("actor_user_id") != USER_ID:
|
|
||||||
continue
|
|
||||||
if row.get("actor_agent_id") not in {AGENT_ID, None, ""}:
|
|
||||||
continue
|
|
||||||
if row.get("target_type") == "memory" and row.get("action") in {"upsert_memory", "feedback:incorrect", "feedback:duplicate", "feedback:outdated"}:
|
|
||||||
target_id = row.get("target_id")
|
|
||||||
if target_id and target_id not in ids:
|
|
||||||
ids.append(target_id)
|
|
||||||
return ids
|
|
||||||
|
|
||||||
|
|
||||||
def _memory_from_result(item: dict[str, Any]) -> dict[str, Any] | None:
|
|
||||||
memory = item.get("memory")
|
|
||||||
if isinstance(memory, dict) and memory.get("id"):
|
|
||||||
return memory
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _is_cleanup_candidate(memory: dict[str, Any]) -> bool:
|
|
||||||
if memory.get("user_id") != USER_ID:
|
|
||||||
return False
|
|
||||||
tags = set(memory.get("tags") or [])
|
|
||||||
return bool(tags.intersection({"integration_test", "safe_to_delete", "plugin"}))
|
|
||||||
|
|
||||||
|
|
||||||
def _delete_memory(memory_id: str) -> dict[str, Any]:
|
|
||||||
query = urllib.parse.urlencode({"user_id": USER_ID, "agent_id": AGENT_ID, "workspace_id": WORKSPACE_ID, "session_id": SESSION_ID})
|
|
||||||
return _request("DELETE", f"/v1/memory/{urllib.parse.quote(memory_id)}?{query}")
|
|
||||||
|
|
||||||
|
|
||||||
def _feedback_memory(memory_id: str) -> dict[str, Any]:
|
|
||||||
return _request(
|
|
||||||
"POST",
|
|
||||||
f"/v1/memory/{urllib.parse.quote(memory_id)}/feedback",
|
|
||||||
{
|
|
||||||
"user_id": USER_ID,
|
|
||||||
"agent_id": AGENT_ID,
|
|
||||||
"workspace_id": WORKSPACE_ID,
|
|
||||||
"session_id": SESSION_ID,
|
|
||||||
"feedback": "incorrect",
|
|
||||||
"comment": "cleanup marker for integration test memory",
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def run(user_id: str = USER_ID) -> dict[str, Any]:
|
|
||||||
_assert_test_user(user_id)
|
|
||||||
if user_id != USER_ID:
|
|
||||||
return {"ok": False, "error": "script_is_scoped_to_fixed_test_user", "user_id": user_id}
|
|
||||||
|
|
||||||
search = _search_candidates()
|
|
||||||
if not search.get("ok"):
|
|
||||||
return {"ok": False, "search": {"ok": False, "status_code": search.get("status_code"), "error": search.get("error")}, "deleted": 0, "feedback_marked": 0, "skipped": 0}
|
|
||||||
|
|
||||||
rows = (search.get("data") or {}).get("results") or []
|
|
||||||
memory_ids = _audit_candidate_ids()
|
|
||||||
deleted = 0
|
|
||||||
feedback_marked = 0
|
|
||||||
skipped = 0
|
|
||||||
unable: list[dict[str, Any]] = []
|
|
||||||
touched: list[str] = []
|
|
||||||
|
|
||||||
for item in rows:
|
|
||||||
memory = _memory_from_result(item)
|
|
||||||
if not memory or not _is_cleanup_candidate(memory):
|
|
||||||
skipped += 1
|
|
||||||
continue
|
|
||||||
memory_id = memory["id"]
|
|
||||||
if memory_id not in memory_ids:
|
|
||||||
memory_ids.append(memory_id)
|
|
||||||
|
|
||||||
for memory_id in memory_ids:
|
|
||||||
deletion = _delete_memory(memory_id)
|
|
||||||
if deletion.get("ok"):
|
|
||||||
deleted += 1
|
|
||||||
touched.append(short_id(memory_id))
|
|
||||||
continue
|
|
||||||
if deletion.get("status_code") == 404:
|
|
||||||
skipped += 1
|
|
||||||
continue
|
|
||||||
feedback = _feedback_memory(memory_id)
|
|
||||||
if feedback.get("ok"):
|
|
||||||
feedback_marked += 1
|
|
||||||
touched.append(short_id(memory_id))
|
|
||||||
else:
|
|
||||||
unable.append({"memory_id": short_id(memory_id), "delete_status": deletion.get("status_code"), "feedback_status": feedback.get("status_code"), "reason": feedback.get("error") or deletion.get("error")})
|
|
||||||
|
|
||||||
return {
|
|
||||||
"ok": not unable,
|
|
||||||
"search": {"ok": True, "status_code": search.get("status_code"), "data": summarize_data(search.get("data"))},
|
|
||||||
"deleted": deleted,
|
|
||||||
"feedback_marked": feedback_marked,
|
|
||||||
"skipped": skipped,
|
|
||||||
"unable_count": len(unable),
|
|
||||||
"unable": unable,
|
|
||||||
"touched_memory_ids": touched,
|
|
||||||
"limitation": "search API returns local MemoryRecord rows plus OpenViking context; cleanup only deletes local MemoryRecord rows for the fixed test user.",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
try:
|
|
||||||
result = run(os.environ.get("MEMORY_GATEWAY_CLEANUP_USER_ID", USER_ID))
|
|
||||||
except ValueError as exc:
|
|
||||||
result = {"ok": False, "error": str(exc), "deleted": 0, "feedback_marked": 0, "skipped": 0}
|
|
||||||
print(dumps_safe(result))
|
|
||||||
return 0 if result.get("ok") else 1
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
@ -1,218 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
import urllib.error
|
|
||||||
import urllib.request
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
PLUGIN_ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
if str(PLUGIN_ROOT) not in sys.path:
|
|
||||||
sys.path.insert(0, str(PLUGIN_ROOT))
|
|
||||||
|
|
||||||
from memory_gateway_plugin.client import MemoryGatewayClient
|
|
||||||
from memory_gateway_plugin.config import PluginConfig
|
|
||||||
from memory_gateway_plugin.output import debug_raw_enabled, dumps_safe, redact, short_id, summarize_data, summarize_result
|
|
||||||
from memory_gateway_plugin.tools import (
|
|
||||||
memory_append_episode,
|
|
||||||
memory_commit_session,
|
|
||||||
memory_feedback,
|
|
||||||
memory_search,
|
|
||||||
memory_upsert,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
USER_ID = "test_user_memory_gateway_plugin"
|
|
||||||
AGENT_ID = "test_hermes_memory_gateway_plugin"
|
|
||||||
WORKSPACE_ID = "test_workspace_memory_gateway_plugin"
|
|
||||||
SESSION_ID = "test_session_memory_gateway_plugin_001"
|
|
||||||
|
|
||||||
|
|
||||||
def _short(value: Any, max_chars: int = 700) -> str:
|
|
||||||
text = json.dumps(redact(value), ensure_ascii=False, default=str)
|
|
||||||
return text[:max_chars]
|
|
||||||
|
|
||||||
|
|
||||||
def _summary_data(data: dict[str, Any] | None) -> dict[str, Any]:
|
|
||||||
return summarize_data(data) if isinstance(summarize_data(data), dict) else {}
|
|
||||||
|
|
||||||
|
|
||||||
def _result_detail(result: dict[str, Any]) -> str:
|
|
||||||
return _short(summarize_result(result))
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class Step:
|
|
||||||
name: str
|
|
||||||
ok: bool
|
|
||||||
endpoint: str = ""
|
|
||||||
status_code: int | None = None
|
|
||||||
detail: str = ""
|
|
||||||
data: dict[str, Any] | None = None
|
|
||||||
|
|
||||||
def to_dict(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"name": self.name,
|
|
||||||
"ok": self.ok,
|
|
||||||
"endpoint": self.endpoint,
|
|
||||||
"status_code": self.status_code,
|
|
||||||
"detail": self.detail,
|
|
||||||
"data": _summary_data(self.data),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _request(method: str, url: str, payload: dict[str, Any] | None = None, api_key: str = "") -> dict[str, Any]:
|
|
||||||
headers = {"Content-Type": "application/json"}
|
|
||||||
if api_key:
|
|
||||||
headers["X-API-Key"] = api_key
|
|
||||||
body = None if payload is None else json.dumps(payload, ensure_ascii=False).encode("utf-8")
|
|
||||||
req = urllib.request.Request(url, data=body, headers=headers, method=method)
|
|
||||||
try:
|
|
||||||
with urllib.request.urlopen(req, timeout=10) as response:
|
|
||||||
raw = response.read().decode("utf-8")
|
|
||||||
return {"ok": True, "status_code": getattr(response, "status", 200), "data": json.loads(raw) if raw else {}}
|
|
||||||
except urllib.error.HTTPError as exc:
|
|
||||||
try:
|
|
||||||
body_text = exc.read().decode("utf-8")
|
|
||||||
except Exception:
|
|
||||||
body_text = str(exc.reason)
|
|
||||||
return {"ok": False, "status_code": exc.code, "error": body_text[:700]}
|
|
||||||
except Exception as exc:
|
|
||||||
return {"ok": False, "status_code": None, "error": str(exc)[:700]}
|
|
||||||
|
|
||||||
|
|
||||||
def _health(config: PluginConfig) -> Step:
|
|
||||||
endpoint = "/health"
|
|
||||||
result = _request("GET", config.gateway_url.rstrip("/") + endpoint, api_key=config.api_key)
|
|
||||||
return Step("health", bool(result.get("ok")), endpoint, result.get("status_code"), _result_detail(result), result.get("data"))
|
|
||||||
|
|
||||||
|
|
||||||
def _ensure_user(config: PluginConfig) -> Step:
|
|
||||||
endpoint = "/v1/users"
|
|
||||||
result = _request(
|
|
||||||
"POST",
|
|
||||||
config.gateway_url.rstrip("/") + endpoint,
|
|
||||||
{"user_id": USER_ID, "display_name": "Memory Gateway Plugin Integration Test", "preferences": {"purpose": "integration_test"}},
|
|
||||||
api_key=config.api_key,
|
|
||||||
)
|
|
||||||
ok = bool(result.get("ok")) or result.get("status_code") in {200, 201, 409}
|
|
||||||
return Step("ensure_user", ok, endpoint, result.get("status_code"), _result_detail(result), result.get("data"))
|
|
||||||
|
|
||||||
|
|
||||||
def _client(config: PluginConfig) -> MemoryGatewayClient:
|
|
||||||
return MemoryGatewayClient(config)
|
|
||||||
|
|
||||||
|
|
||||||
def run() -> dict[str, Any]:
|
|
||||||
config = PluginConfig.from_env()
|
|
||||||
client = _client(config)
|
|
||||||
steps: list[Step] = []
|
|
||||||
|
|
||||||
steps.append(_health(config))
|
|
||||||
if not steps[-1].ok:
|
|
||||||
return {"ok": False, "steps": [s.to_dict() for s in steps]}
|
|
||||||
|
|
||||||
steps.append(_ensure_user(config))
|
|
||||||
|
|
||||||
search_1 = memory_search(
|
|
||||||
query="Memory Gateway plugin integration test",
|
|
||||||
user_id=USER_ID,
|
|
||||||
agent_id=AGENT_ID,
|
|
||||||
workspace_id=WORKSPACE_ID,
|
|
||||||
session_id=SESSION_ID,
|
|
||||||
limit=5,
|
|
||||||
client=client,
|
|
||||||
)
|
|
||||||
steps.append(Step("memory_search_initial", bool(search_1.get("ok")), "/v1/memory/search", search_1.get("status_code"), _result_detail(search_1), search_1.get("data")))
|
|
||||||
|
|
||||||
episode = memory_append_episode(
|
|
||||||
user_id=USER_ID,
|
|
||||||
agent_id=AGENT_ID,
|
|
||||||
workspace_id=WORKSPACE_ID,
|
|
||||||
session_id=SESSION_ID,
|
|
||||||
content="Integration test: user prefers Memory Gateway plugin to store only summarized episodes, not raw transcripts.",
|
|
||||||
tags=["integration_test", "plugin"],
|
|
||||||
source="agent",
|
|
||||||
importance=0.2,
|
|
||||||
confidence=0.5,
|
|
||||||
client=client,
|
|
||||||
)
|
|
||||||
steps.append(Step("memory_append_episode", bool(episode.get("ok")), "/v1/episodes", episode.get("status_code"), _result_detail(episode), episode.get("data")))
|
|
||||||
|
|
||||||
commit = memory_commit_session(
|
|
||||||
user_id=USER_ID,
|
|
||||||
agent_id=AGENT_ID,
|
|
||||||
workspace_id=WORKSPACE_ID,
|
|
||||||
session_id=SESSION_ID,
|
|
||||||
promote=True,
|
|
||||||
min_importance=0.1,
|
|
||||||
client=client,
|
|
||||||
)
|
|
||||||
commit_detail = _result_detail(commit)
|
|
||||||
if commit.get("ok") and not (commit.get("data") or {}).get("promoted"):
|
|
||||||
commit_detail += " | promotion may be empty while commit endpoint succeeded"
|
|
||||||
steps.append(Step("memory_commit_session", bool(commit.get("ok")), f"/v1/sessions/{SESSION_ID}/commit", commit.get("status_code"), commit_detail, commit.get("data")))
|
|
||||||
|
|
||||||
upsert = memory_upsert(
|
|
||||||
user_id=USER_ID,
|
|
||||||
agent_id=AGENT_ID,
|
|
||||||
workspace_id=WORKSPACE_ID,
|
|
||||||
namespace=f"user/{USER_ID}/long_term",
|
|
||||||
memory_type="preference",
|
|
||||||
content="Integration test memory: this should be removable or clearly tagged as test data.",
|
|
||||||
tags=["integration_test", "plugin", "safe_to_delete"],
|
|
||||||
importance=0.1,
|
|
||||||
confidence=0.5,
|
|
||||||
source="agent",
|
|
||||||
client=client,
|
|
||||||
)
|
|
||||||
steps.append(Step("memory_upsert", bool(upsert.get("ok")), "/v1/memory", upsert.get("status_code"), _result_detail(upsert), upsert.get("data")))
|
|
||||||
|
|
||||||
search_2 = memory_search(
|
|
||||||
query="Integration test memory summarized episodes raw transcripts",
|
|
||||||
user_id=USER_ID,
|
|
||||||
agent_id=AGENT_ID,
|
|
||||||
workspace_id=WORKSPACE_ID,
|
|
||||||
session_id=SESSION_ID,
|
|
||||||
limit=10,
|
|
||||||
client=client,
|
|
||||||
)
|
|
||||||
result_count = len((search_2.get("data") or {}).get("results", []))
|
|
||||||
detail = _result_detail(search_2)
|
|
||||||
if search_2.get("ok") and result_count == 0:
|
|
||||||
detail += " | search succeeded but returned no results; indexing or OpenViking sync may be asynchronous"
|
|
||||||
steps.append(Step("memory_search_after_write", bool(search_2.get("ok")), "/v1/memory/search", search_2.get("status_code"), detail, search_2.get("data")))
|
|
||||||
|
|
||||||
memory_id = ((upsert.get("data") or {}).get("memory") or upsert.get("data") or {}).get("id")
|
|
||||||
if memory_id:
|
|
||||||
feedback = memory_feedback(
|
|
||||||
user_id=USER_ID,
|
|
||||||
agent_id=AGENT_ID,
|
|
||||||
workspace_id=WORKSPACE_ID,
|
|
||||||
session_id=SESSION_ID,
|
|
||||||
memory_id=memory_id,
|
|
||||||
feedback="reject",
|
|
||||||
comment="Integration test cleanup marker; safe to ignore/delete.",
|
|
||||||
client=client,
|
|
||||||
)
|
|
||||||
steps.append(Step("memory_feedback", bool(feedback.get("ok")), f"/v1/memory/{memory_id}/feedback", feedback.get("status_code"), _result_detail(feedback), feedback.get("data")))
|
|
||||||
else:
|
|
||||||
steps.append(Step("memory_feedback", False, "/v1/memory/{memory_id}/feedback", None, "skipped because memory_upsert did not return memory id"))
|
|
||||||
|
|
||||||
required = {"health", "memory_search_initial", "memory_append_episode", "memory_commit_session", "memory_upsert", "memory_search_after_write", "memory_feedback"}
|
|
||||||
ok = all(step.ok for step in steps if step.name in required)
|
|
||||||
return {"ok": ok, "gateway_url": config.gateway_url, "debug_raw": debug_raw_enabled(), "test_identity": {"user_id": USER_ID, "agent_id": AGENT_ID, "workspace_id": WORKSPACE_ID, "session_id": SESSION_ID}, "steps": [s.to_dict() for s in steps]}
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
result = run()
|
|
||||||
print(dumps_safe(result))
|
|
||||||
return 0 if result.get("ok") else 1
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
@ -1,32 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import sys
|
|
||||||
import urllib.error
|
|
||||||
import urllib.request
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
PLUGIN_ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
if str(PLUGIN_ROOT) not in sys.path:
|
|
||||||
sys.path.insert(0, str(PLUGIN_ROOT))
|
|
||||||
|
|
||||||
from memory_gateway_plugin.config import load_config
|
|
||||||
from memory_gateway_plugin.output import dumps_safe, summarize_data
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
config = load_config()
|
|
||||||
request = urllib.request.Request(config.gateway_url.rstrip("/") + "/health", method="GET")
|
|
||||||
if config.api_key:
|
|
||||||
request.add_header("X-API-Key", config.api_key)
|
|
||||||
try:
|
|
||||||
with urllib.request.urlopen(request, timeout=config.timeout) as response:
|
|
||||||
payload = json.loads(response.read().decode("utf-8"))
|
|
||||||
print(dumps_safe({"ok": True, "endpoint": "/health", "status_code": getattr(response, "status", 200), "data": summarize_data(payload)}))
|
|
||||||
except urllib.error.URLError as exc:
|
|
||||||
print(dumps_safe({"ok": False, "endpoint": "/health", "status_code": None, "error": str(exc)[:300]}))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@ -1,174 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
import urllib.error
|
|
||||||
import urllib.request
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
|
|
||||||
USER_ID = "test_user_memory_gateway_plugin"
|
|
||||||
AGENT_ID = "test_hermes_memory_gateway_plugin"
|
|
||||||
WORKSPACE_ID = "test_workspace_memory_gateway_plugin"
|
|
||||||
SESSION_ID = "test_session_memory_gateway_plugin_001"
|
|
||||||
|
|
||||||
|
|
||||||
def _ensure_paths() -> None:
|
|
||||||
plugin_root = Path(__file__).resolve().parents[1]
|
|
||||||
hermes_repo = Path(os.environ.get("HERMES_REPO", "/home/tom/.hermes/hermes-agent"))
|
|
||||||
hermes_cli = hermes_repo / "hermes_cli"
|
|
||||||
for path in [plugin_root, hermes_repo, hermes_cli]:
|
|
||||||
if str(path) not in sys.path:
|
|
||||||
sys.path.insert(0, str(path))
|
|
||||||
|
|
||||||
|
|
||||||
_ensure_paths()
|
|
||||||
|
|
||||||
from memory_gateway_plugin.output import dumps_safe, summarize_data
|
|
||||||
|
|
||||||
|
|
||||||
def _request(method: str, url: str, payload: dict[str, Any] | None = None, api_key: str = "") -> dict[str, Any]:
|
|
||||||
headers = {"Content-Type": "application/json"}
|
|
||||||
if api_key:
|
|
||||||
headers["X-API-Key"] = api_key
|
|
||||||
body = None if payload is None else json.dumps(payload, ensure_ascii=False).encode("utf-8")
|
|
||||||
req = urllib.request.Request(url, data=body, headers=headers, method=method)
|
|
||||||
try:
|
|
||||||
with urllib.request.urlopen(req, timeout=10) as response:
|
|
||||||
raw = response.read().decode("utf-8")
|
|
||||||
return {"ok": True, "status_code": getattr(response, "status", 200), "data": json.loads(raw) if raw else {}}
|
|
||||||
except urllib.error.HTTPError as exc:
|
|
||||||
try:
|
|
||||||
body_text = exc.read().decode("utf-8")
|
|
||||||
except Exception:
|
|
||||||
body_text = str(exc.reason)
|
|
||||||
return {"ok": False, "status_code": exc.code, "error": body_text[:500]}
|
|
||||||
except Exception as exc:
|
|
||||||
return {"ok": False, "status_code": None, "error": str(exc)[:500]}
|
|
||||||
|
|
||||||
|
|
||||||
def _ensure_user() -> dict[str, Any]:
|
|
||||||
gateway_url = os.environ.get("MEMORY_GATEWAY_URL", "http://127.0.0.1:1934").rstrip("/")
|
|
||||||
api_key = os.environ.get("MEMORY_GATEWAY_API_KEY", "")
|
|
||||||
return _request(
|
|
||||||
"POST",
|
|
||||||
gateway_url + "/v1/users",
|
|
||||||
{"user_id": USER_ID, "display_name": "Memory Gateway Hook Probe", "preferences": {"purpose": "hook_probe"}},
|
|
||||||
api_key=api_key,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _audit_count(action: str) -> int:
|
|
||||||
gateway_url = os.environ.get("MEMORY_GATEWAY_URL", "http://127.0.0.1:1934").rstrip("/")
|
|
||||||
api_key = os.environ.get("MEMORY_GATEWAY_API_KEY", "")
|
|
||||||
result = _request("GET", gateway_url + "/v1/audit?limit=1000", api_key=api_key)
|
|
||||||
if not result.get("ok"):
|
|
||||||
return -1
|
|
||||||
rows = result.get("data") or []
|
|
||||||
return sum(
|
|
||||||
1
|
|
||||||
for row in rows
|
|
||||||
if row.get("action") == action
|
|
||||||
and row.get("actor_user_id") == USER_ID
|
|
||||||
and row.get("actor_agent_id") == AGENT_ID
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _hook_report(manager: Any, hook_name: str, payload: dict[str, Any], audit_action: str = "") -> dict[str, Any]:
|
|
||||||
registered = hook_name in getattr(manager, "_hooks", {}) and bool(manager._hooks[hook_name])
|
|
||||||
before = _audit_count(audit_action) if audit_action else -1
|
|
||||||
try:
|
|
||||||
result = manager.invoke_hook(hook_name, **payload)
|
|
||||||
after = _audit_count(audit_action) if audit_action else -1
|
|
||||||
return {
|
|
||||||
"registered": registered,
|
|
||||||
"invoked": True,
|
|
||||||
"result_type": type(result).__name__,
|
|
||||||
"result": summarize_data(result),
|
|
||||||
"audit_action": audit_action,
|
|
||||||
"audit_delta": (after - before) if before >= 0 and after >= 0 else None,
|
|
||||||
"error": "",
|
|
||||||
}
|
|
||||||
except Exception as exc:
|
|
||||||
return {
|
|
||||||
"registered": registered,
|
|
||||||
"invoked": False,
|
|
||||||
"result_type": "",
|
|
||||||
"result": None,
|
|
||||||
"audit_action": audit_action,
|
|
||||||
"audit_delta": None,
|
|
||||||
"error": str(exc)[:500],
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def run(auto_commit: bool = False) -> dict[str, Any]:
|
|
||||||
os.environ.setdefault("MEMORY_GATEWAY_URL", "http://127.0.0.1:1934")
|
|
||||||
os.environ["MEMORY_GATEWAY_DEFAULT_USER_ID"] = USER_ID
|
|
||||||
os.environ["MEMORY_GATEWAY_DEFAULT_AGENT_ID"] = AGENT_ID
|
|
||||||
os.environ["MEMORY_GATEWAY_DEFAULT_WORKSPACE_ID"] = WORKSPACE_ID
|
|
||||||
os.environ["MEMORY_GATEWAY_AUTO_COMMIT_SESSION"] = "true" if auto_commit else "false"
|
|
||||||
|
|
||||||
from plugins import PluginManager
|
|
||||||
|
|
||||||
ensure_user = _ensure_user()
|
|
||||||
manager = PluginManager()
|
|
||||||
manager.discover_and_load()
|
|
||||||
|
|
||||||
base = {
|
|
||||||
"user_id": USER_ID,
|
|
||||||
"agent_id": AGENT_ID,
|
|
||||||
"workspace_id": WORKSPACE_ID,
|
|
||||||
"session_id": SESSION_ID,
|
|
||||||
"task_id": SESSION_ID,
|
|
||||||
"model": "hook-probe",
|
|
||||||
"platform": "cli",
|
|
||||||
}
|
|
||||||
hooks = {
|
|
||||||
"on_session_start": dict(base),
|
|
||||||
"pre_llm_call": {
|
|
||||||
**base,
|
|
||||||
"user_message": "Memory Gateway plugin integration test memory preference",
|
|
||||||
"conversation_history": [],
|
|
||||||
"is_first_turn": True,
|
|
||||||
},
|
|
||||||
"post_llm_call": {
|
|
||||||
**base,
|
|
||||||
"user_message": "请记住：Memory Gateway plugin hook probe 偏好保存简短摘要型 episode。",
|
|
||||||
"assistant_response": "已记录为候选摘要，后续由 session commit 判断是否提升为长期记忆。",
|
|
||||||
},
|
|
||||||
"on_session_end": dict(base),
|
|
||||||
}
|
|
||||||
|
|
||||||
audit_actions = {
|
|
||||||
"pre_llm_call": "memory_search",
|
|
||||||
"post_llm_call": "append_episode",
|
|
||||||
"on_session_end": "commit_session",
|
|
||||||
}
|
|
||||||
reports = {name: _hook_report(manager, name, payload, audit_actions.get(name, "")) for name, payload in hooks.items()}
|
|
||||||
plugin = manager._plugins.get("memory-gateway-agent")
|
|
||||||
return {
|
|
||||||
"ok": all(item["registered"] and item["invoked"] for item in reports.values()),
|
|
||||||
"auto_commit": auto_commit,
|
|
||||||
"ensure_user": {"ok": ensure_user.get("ok"), "status_code": ensure_user.get("status_code"), "data": summarize_data(ensure_user.get("data"))},
|
|
||||||
"plugin": {
|
|
||||||
"enabled": bool(plugin and plugin.enabled),
|
|
||||||
"tools_registered": sorted(getattr(plugin, "tools_registered", []) if plugin else []),
|
|
||||||
"hooks_registered": sorted(getattr(plugin, "hooks_registered", []) if plugin else []),
|
|
||||||
"error": getattr(plugin, "error", None) if plugin else "plugin_not_found",
|
|
||||||
},
|
|
||||||
"hooks": reports,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
auto_commit = os.environ.get("MEMORY_GATEWAY_AUTO_COMMIT_SESSION", "").strip().lower() in {"1", "true", "yes", "on"}
|
|
||||||
result = run(auto_commit=auto_commit)
|
|
||||||
print(dumps_safe(result))
|
|
||||||
return 0 if result.get("ok") else 1
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
@ -1,147 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import shutil
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
import urllib.error
|
|
||||||
import urllib.request
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
PLUGIN_ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
if str(PLUGIN_ROOT) not in sys.path:
|
|
||||||
sys.path.insert(0, str(PLUGIN_ROOT))
|
|
||||||
|
|
||||||
from memory_gateway_plugin.output import dumps_safe, short_id, summarize_data
|
|
||||||
|
|
||||||
|
|
||||||
USER_ID = "test_user_memory_gateway_plugin"
|
|
||||||
AGENT_ID = "test_hermes_memory_gateway_plugin"
|
|
||||||
WORKSPACE_ID = "test_workspace_memory_gateway_plugin"
|
|
||||||
SESSION_ID = "test_session_memory_gateway_plugin_interactive_002"
|
|
||||||
PROMPT = "Please remember this integration test preference: Memory Gateway plugin should store summarized episodes, not raw transcripts."
|
|
||||||
|
|
||||||
|
|
||||||
def _request(method: str, url: str, payload: dict[str, Any] | None = None, api_key: str = "") -> dict[str, Any]:
|
|
||||||
headers = {"Content-Type": "application/json"}
|
|
||||||
if api_key:
|
|
||||||
headers["X-API-Key"] = api_key
|
|
||||||
body = None if payload is None else json.dumps(payload, ensure_ascii=False).encode("utf-8")
|
|
||||||
req = urllib.request.Request(url, data=body, headers=headers, method=method)
|
|
||||||
try:
|
|
||||||
with urllib.request.urlopen(req, timeout=10) as response:
|
|
||||||
raw = response.read().decode("utf-8")
|
|
||||||
return {"ok": True, "status_code": getattr(response, "status", 200), "data": json.loads(raw) if raw else {}}
|
|
||||||
except urllib.error.HTTPError as exc:
|
|
||||||
return {"ok": False, "status_code": exc.code, "error": str(exc.reason)[:300]}
|
|
||||||
except Exception as exc:
|
|
||||||
return {"ok": False, "status_code": None, "error": str(exc)[:300]}
|
|
||||||
|
|
||||||
|
|
||||||
def _gateway_url() -> str:
|
|
||||||
return os.environ.get("MEMORY_GATEWAY_URL", "http://127.0.0.1:1934").rstrip("/")
|
|
||||||
|
|
||||||
|
|
||||||
def _api_key() -> str:
|
|
||||||
return os.environ.get("MEMORY_GATEWAY_API_KEY", "")
|
|
||||||
|
|
||||||
|
|
||||||
def _audit_counts() -> dict[str, int]:
|
|
||||||
result = _request("GET", _gateway_url() + "/v1/audit?limit=1000", api_key=_api_key())
|
|
||||||
rows = result.get("data") or []
|
|
||||||
actions = {"memory_search": 0, "append_episode": 0, "commit_session": 0}
|
|
||||||
for row in rows:
|
|
||||||
if row.get("actor_user_id") != USER_ID or row.get("actor_agent_id") != AGENT_ID:
|
|
||||||
continue
|
|
||||||
action = row.get("action")
|
|
||||||
if action in actions:
|
|
||||||
actions[action] += 1
|
|
||||||
return actions
|
|
||||||
|
|
||||||
|
|
||||||
def _run_cmd(args: list[str], timeout: int = 20, env: dict[str, str] | None = None) -> dict[str, Any]:
|
|
||||||
try:
|
|
||||||
completed = subprocess.run(args, capture_output=True, text=True, timeout=timeout, env=env, check=False)
|
|
||||||
return {"ok": completed.returncode == 0, "returncode": completed.returncode, "stdout_chars": len(completed.stdout), "stderr_chars": len(completed.stderr)}
|
|
||||||
except FileNotFoundError:
|
|
||||||
return {"ok": False, "returncode": None, "error": "command_not_found"}
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
return {"ok": False, "returncode": None, "error": "timeout"}
|
|
||||||
|
|
||||||
|
|
||||||
def _manual_instructions(reason: str) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"mode": "manual",
|
|
||||||
"reason": reason,
|
|
||||||
"commands": [
|
|
||||||
"hermes plugins list",
|
|
||||||
"hermes tools list",
|
|
||||||
"MEMORY_GATEWAY_URL=http://127.0.0.1:1934 MEMORY_GATEWAY_DEFAULT_USER_ID=test_user_memory_gateway_plugin MEMORY_GATEWAY_DEFAULT_AGENT_ID=test_hermes_memory_gateway_plugin MEMORY_GATEWAY_DEFAULT_WORKSPACE_ID=test_workspace_memory_gateway_plugin MEMORY_GATEWAY_AUTO_COMMIT_SESSION=false hermes chat -Q -q 'Please remember this integration test preference: Memory Gateway plugin should store summarized episodes, not raw transcripts.' --source memory-gateway-plugin-test --toolsets memory_gateway",
|
|
||||||
"python plugins/memory-gateway-agent/scripts/hermes_interactive_session_check.py",
|
|
||||||
],
|
|
||||||
"expected": [
|
|
||||||
"Gateway audit memory_search count increases for the test user/agent.",
|
|
||||||
"Gateway audit append_episode count increases for the test user/agent.",
|
|
||||||
"commit_session count does not increase while MEMORY_GATEWAY_AUTO_COMMIT_SESSION=false.",
|
|
||||||
],
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def run() -> dict[str, Any]:
|
|
||||||
hermes = shutil.which("hermes") or "/home/tom/.local/bin/hermes"
|
|
||||||
plugin_list = _run_cmd([hermes, "plugins", "list"], timeout=10)
|
|
||||||
tools_list = _run_cmd([hermes, "tools", "list"], timeout=10)
|
|
||||||
health = _request("GET", _gateway_url() + "/health", api_key=_api_key())
|
|
||||||
if not health.get("ok"):
|
|
||||||
return {"ok": False, "mode": "blocked", "plugin_list": plugin_list, "tools_list": tools_list, "gateway_health": {"ok": False, "status_code": health.get("status_code"), "error": health.get("error")}, "manual": _manual_instructions("gateway_unhealthy")}
|
|
||||||
|
|
||||||
before = _audit_counts()
|
|
||||||
env = os.environ.copy()
|
|
||||||
env.update(
|
|
||||||
{
|
|
||||||
"MEMORY_GATEWAY_URL": _gateway_url(),
|
|
||||||
"MEMORY_GATEWAY_DEFAULT_USER_ID": USER_ID,
|
|
||||||
"MEMORY_GATEWAY_DEFAULT_AGENT_ID": AGENT_ID,
|
|
||||||
"MEMORY_GATEWAY_DEFAULT_WORKSPACE_ID": WORKSPACE_ID,
|
|
||||||
"MEMORY_GATEWAY_AUTO_SEARCH": "true",
|
|
||||||
"MEMORY_GATEWAY_AUTO_APPEND_EPISODE": "true",
|
|
||||||
"MEMORY_GATEWAY_AUTO_COMMIT_SESSION": os.environ.get("MEMORY_GATEWAY_AUTO_COMMIT_SESSION", "false"),
|
|
||||||
}
|
|
||||||
)
|
|
||||||
chat = _run_cmd(
|
|
||||||
[hermes, "chat", "-Q", "-q", PROMPT, "--source", "memory-gateway-plugin-test", "--toolsets", "memory_gateway"],
|
|
||||||
timeout=int(os.environ.get("MEMORY_GATEWAY_PLUGIN_CHAT_TIMEOUT", "180")),
|
|
||||||
env=env,
|
|
||||||
)
|
|
||||||
after = _audit_counts()
|
|
||||||
delta = {key: after.get(key, 0) - before.get(key, 0) for key in before}
|
|
||||||
auto_commit = env["MEMORY_GATEWAY_AUTO_COMMIT_SESSION"].strip().lower() in {"1", "true", "yes", "on"}
|
|
||||||
expected_commit = delta.get("commit_session", 0) > 0 if auto_commit else delta.get("commit_session", 0) == 0
|
|
||||||
passed = chat.get("ok") and delta.get("memory_search", 0) > 0 and delta.get("append_episode", 0) > 0 and expected_commit
|
|
||||||
return {
|
|
||||||
"ok": bool(passed),
|
|
||||||
"mode": "auto" if chat.get("ok") else "manual",
|
|
||||||
"plugin_list": plugin_list,
|
|
||||||
"tools_list": tools_list,
|
|
||||||
"gateway_health": {"ok": True, "status_code": health.get("status_code"), "data": summarize_data(health.get("data"))},
|
|
||||||
"chat": chat,
|
|
||||||
"auto_commit": auto_commit,
|
|
||||||
"audit_before": before,
|
|
||||||
"audit_after": after,
|
|
||||||
"audit_delta": delta,
|
|
||||||
"test_identity": {"user_id": USER_ID, "agent_id": AGENT_ID, "workspace_id": WORKSPACE_ID, "session_id": short_id(SESSION_ID)},
|
|
||||||
"manual": None if chat.get("ok") else _manual_instructions(chat.get("error", "chat_command_failed")),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
result = run()
|
|
||||||
print(dumps_safe(result))
|
|
||||||
return 0 if result.get("ok") else 1
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user