diff --git a/README.md b/README.md index e4412f9..73c3db2 100644 --- a/README.md +++ b/README.md @@ -1,129 +1,50 @@ -# Memory Gateway +# Memory System API -Memory Gateway 是一个本地 memory/context gateway，用统一的 HTTP、MCP 和 Hermes skill 入口，把上层 agent 的记忆写入、上下文检索、会话提交和知识沉淀路由到 OpenViking、EverOS、SQLite metadata store 和可选 Obsidian vault。 +Memory System API is a lightweight HTTP facade over two memory backends: -当前项目的主线是 **OpenViking + EverOS 双后端**： +- OpenViking stores session conversation memory. +- EverOS stores user profile and episodic memory. -- OpenViking 运行在 `127.0.0.1:1933`，负责 session/resource/context 层能力。 -- EverOS/EverCore 运行在 `127.0.0.1:1995`，负责长期记忆、profile 和检索。 -- Memory Gateway 默认运行在 `127.0.0.1:1934`，提供统一 API、认证、metadata、outbox 和 adapter 编排。 +The caller only sends `user_id`, `session_id`, and optional `user_message` / `assistant_message`. +The API creates or reuses the OpenViking user key, writes messages to both backends, and exposes simple endpoints for commit, immediate extraction, search, and profile reads. -## 核心能力 +## Endpoints -- `/v2/conversations/ingest`：把一轮对话写入 OpenViking 和 EverOS，并在本地保存 `memory_refs` 控制面引用。 -- `/v2/context/retrieve`：实际调用 OpenViking / EverOS 的 retrieve 接口，把两个后端返回的上下文合并到 `items`。 -- `/v2/conversations/{session_id}/commit`：创建 commit job 和 outbox events，用于异步生成长期 ref。 -- `/v2/admin/outbox/process`：处理 pending outbox，生成 OpenViking session ref、EverOS profile ref、EverOS long-term ref。 -- `/v2/memory/refs`：查询本地保存的后端引用元数据。 -- `/v1/*`：保留基础用户、memory、episode、session commit、audit 和 EverOS health 能力。 -- `/api/*`：保留旧版搜索、写 memory/resource、LLM summary、文档上传到 Obsidian/OpenViking 的兼容接口。 -- `/mcp/rpc` 与 `/mcp/sse`：提供 MCP 调用入口。 - -## 架构 - -```mermaid -flowchart LR - Agent[Agent / Hermes / MCP Client] --> Gateway[Memory Gateway] - Gateway --> SQLite[(SQLite metadata)] - Gateway --> OpenViking[OpenViking API] - Gateway --> EverOS[EverOS / EverCore API] - Gateway --> Obsidian[Obsidian Markdown Vault] - Gateway --> LLM[OpenAI-compatible LLM] - - SQLite --> Refs[memory_refs / outbox_events / commit_jobs] - OpenViking --> OVStore[(session / resource / context index)] - EverOS --> EverStore[(memory / profile / long-term store)] -``` - -Memory Gateway 不把完整对话正文长期保存到 SQLite。SQLite 主要保存控制面 metadata：用户、episode、audit、`memory_refs`、`commit_jobs`、`outbox_events`。真正的记忆正文和检索上下文在 OpenViking / EverOS 后端中。 - -## 项目结构 +Base URL: ```text -memory_gateway/ - api_v1.py # v1 REST API - api_v2.py # v2 workflow API - server.py # FastAPI + MCP + legacy /api entrypoint - services.py # v1 service - services_v2.py # v2 ingest/retrieve/commit/outbox orchestration - openviking_client.py # OpenViking adapter - everos_client.py # EverOS adapter - repositories.py # in-memory / SQLite metadata repository - schemas.py # v1 schemas - schemas_v2.py # v2 schemas - backend_contracts.py # backend adapter result contracts - backend_normalization.py # backend response normalization - backend_ref_mapping.py # native ref type -> MemoryRefType mapping - obsidian_review.py # Obsidian review draft support -integrations/hermes/memory-gateway/ - SKILL.md - scripts/ -plugins/memory-gateway-agent/ -tests/ -config.example.yaml -pyproject.toml +http://127.0.0.1:1934 ``` -## 安装 +Routes: -要求 Python 3.10+。 +- `GET /memory-system/health` +- `POST /memory-system/messages` +- `POST /memory-system/sessions/{session_id}/commit` +- `POST /memory-system/sessions/{session_id}/extract` +- `GET /memory-system/openviking/tasks/{task_id}?user_id=...` +- `POST /memory-system/search` +- `GET /memory-system/users/{user_id}/profile` + +## Install ```bash cd /home/tom/memory-gateway -python3 -m venv .venv +python -m venv .venv source .venv/bin/activate pip install -U pip pip install -e ".[dev]" ``` -如果使用 `uv`： +## Configure + +Copy the example config and edit backend URLs or keys as needed: ```bash -cd /home/tom/memory-gateway -uv sync --extra dev -``` - -## 依赖服务 - -### OpenViking - -参考 `/home/tom/OpenViking/CONTRIBUTING.md`。当前约定启动方式： - -```bash -openviking-server --host 127.0.0.1 --port 1933 -``` - -配置文件： - -```text -/home/tom/.openviking/ov.conf -``` - -### EverOS / EverCore - -参考 `/home/tom/EverOS/methods/EverCore/docs/installation/SETUP.md`。当前约定启动方式： - -```bash -cd /home/tom/EverOS/methods/EverCore -uv run python src/run.py --port 1995 -``` - -配置文件： - -```text -/home/tom/EverOS/methods/EverCore/.env -``` - -## 配置 - -复制示例配置： - -```bash -cd /home/tom/memory-gateway cp config.example.yaml config.yaml ``` -核心配置示例： +Important fields: ```yaml server: @@ -133,242 +54,88 @@ server: openviking: url: "http://127.0.0.1:1933" - api_key: "" - timeout: 30 + api_key: "your-secret-root-key" everos: - enabled: true - mode: "real" url: "http://127.0.0.1:1995" - api_key: "" - timeout: 30 - health_path: "/health" - ingest_path: "/api/v1/memories" - search_path: "/api/v1/memories/search" - flush_path: "/api/v1/memories/flush" - retrieve_method: "keyword" - -storage: - backend: "sqlite" - sqlite_path: "/home/tom/memory-gateway/memory_gateway.sqlite3" ``` -也可以用环境变量覆盖后端配置，例如： +If `server.api_key` is set, clients must send `X-API-Key`. + +## Start + +Start OpenViking and EverOS first, then run: ```bash -export OPENVIKING_URL=http://127.0.0.1:1933 -export EVEROS_URL=http://127.0.0.1:1995 -export EVEROS_MODE=real +python -m memory_system_api.server --config config.yaml --host 127.0.0.1 --port 1934 ``` -## 启动 +## Real Test Flow + +Health: ```bash -cd /home/tom/memory-gateway -source .venv/bin/activate -python -m memory_gateway.server --config config.yaml +curl -s http://127.0.0.1:1934/memory-system/health ``` -也可以显式指定 host/port： +Write user and assistant messages: ```bash -python -m memory_gateway.server --config config.yaml --host 127.0.0.1 --port 1934 -``` - -健康检查： - -```bash -curl http://127.0.0.1:1934/health -curl http://127.0.0.1:1934/v1/everos/health -``` - -如果设置了 `server.api_key`，请求需要带： - -```bash --H "X-API-Key: " -``` - -## v2 工作流 - -### 1. Ingest 一轮对话 - -```bash -curl -s http://127.0.0.1:1934/v2/conversations/ingest \ - -H 'Content-Type: application/json' \ +curl -s -X POST http://127.0.0.1:1934/memory-system/messages \ + -H "Content-Type: application/json" \ -d '{ - "workspace_id": "ws_1", - "user_id": "user_a", - "agent_id": "agent_cli", - "session_id": "sess_1", - "turn_id": "turn_1", - "namespace": "workspace/ws_1/user/user_a", - "role": "user", - "content": "Remember that the demo environment uses EverOS and OpenViking.", - "metadata": {"channel": "manual-test"} + "user_id": "real_user_001", + "session_id": "real_sess_001", + "user_message": "我喜欢喝拿铁，不喜欢美式。", + "assistant_message": "好的，我会记住你的咖啡偏好。" }' ``` -结果中的 `refs` 是本地 `memory_refs` 控制面引用，通常包括： - -- OpenViking `session_archive` ref -- EverOS `message_memory` ref - -这些 refs 保存的是 native id/uri、状态、hash、trace 等 metadata，不是完整记忆正文。 - -### 2. Retrieve 上下文 +Commit OpenViking and flush EverOS: ```bash -curl -s http://127.0.0.1:1934/v2/context/retrieve \ - -H 'Content-Type: application/json' \ +curl -s -X POST http://127.0.0.1:1934/memory-system/sessions/real_sess_001/commit \ + -H "Content-Type: application/json" \ + -d '{"user_id": "real_user_001"}' +``` + +Search without LLM planning: + +```bash +curl -s -X POST http://127.0.0.1:1934/memory-system/search \ + -H "Content-Type: application/json" \ -d '{ - "workspace_id": "ws_1", - "user_id": "user_a", - "agent_id": "agent_cli", - "session_id": "sess_1", - "namespace": "workspace/ws_1/user/user_a", - "query": "EverOS OpenViking demo environment", - "limit": 5, - "metadata": {"trace_id": "trace_manual_1"} + "user_id": "real_user_001", + "session_id": "real_sess_001", + "query": "我喜欢喝什么咖啡？", + "use_llm": false, + "limit": 10 }' ``` -返回结构重点： - -- `items`：真实上下文，由 OpenViking / EverOS retrieve 返回后合并，包含 `text`、`source_backend`、`ref_id`、`score`、`memory_type`。 -- `refs`：本地已有的 `memory_refs` 视图，用于追踪哪些后端引用已保存。 -- `metadata.backend_results`：每个后端 retrieve 的状态、返回数量和错误信息。 - -### 3. Commit 一个 session +Search with LLM planning: ```bash -curl -s http://127.0.0.1:1934/v2/conversations/sess_1/commit \ - -H 'Content-Type: application/json' \ +curl -s -X POST http://127.0.0.1:1934/memory-system/search \ + -H "Content-Type: application/json" \ -d '{ - "workspace_id": "ws_1", - "user_id": "user_a", - "agent_id": "agent_cli", - "namespace": "workspace/ws_1/user/user_a" + "user_id": "real_user_001", + "session_id": "real_sess_001", + "query": "我的偏好是什么？", + "use_llm": true, + "limit": 10 }' ``` -该接口只创建 commit job 和 outbox events，不直接执行长期记忆生成。返回中会有 `job_id` 和 `metadata.gateway_id`。 - -### 4. Process outbox +Read EverOS profile: ```bash -curl -s -X POST 'http://127.0.0.1:1934/v2/admin/outbox/process?limit=20' +curl -s http://127.0.0.1:1934/memory-system/users/real_user_001/profile ``` -处理成功后会生成长期 refs： - -- OpenViking `session_archive` ref：session archive / summary 的 native 引用。 -- EverOS `profile` ref：用户 profile 的 native 引用。 -- EverOS `long_term_memory` ref：session 提炼出的长期记忆 native 引用。 - -这些 ref 保存在 SQLite 的 `memory_refs` 表中。 - -### 5. 查看 refs 和 job +## Development Checks ```bash -curl -s 'http://127.0.0.1:1934/v2/memory/refs?workspace_id=ws_1&user_id=user_a&session_id=sess_1&limit=20' - -curl -s http://127.0.0.1:1934/v2/jobs/ +python -m pytest -q +python -m compileall -q memory_system_api tests ``` - -SQLite 默认路径取决于配置，例如： - -```text -/home/tom/memory-gateway/memory_gateway.sqlite3 -``` - -主要表： - -- `memory_refs` -- `outbox_events` -- `commit_jobs` -- `audit_logs` -- `users` / `memories` / `episodes` / `profiles` - -## v1 和 legacy API - -v1 保留用户隔离、namespace、visibility/ACL、episode、session commit、audit 等基础能力： - -```text -POST /v1/users -GET /v1/users/{user_id} -POST /v1/memory/search -POST /v1/memory -GET /v1/memory/{memory_id} -PATCH /v1/memory/{memory_id} -DELETE /v1/memory/{memory_id} -POST /v1/episodes -POST /v1/sessions/{session_id}/commit -GET /v1/users/{user_id}/profile -POST /v1/memory/{memory_id}/feedback -GET /v1/namespaces -GET /v1/audit -GET /v1/everos/health -``` - -旧 `/api/*` 接口仍保留： - -```text -POST /api/search -POST /api/memory -POST /api/resource -POST /api/summary -POST /api/knowledge/upload -``` - -## MCP / Hermes - -MCP endpoints： - -```text -POST /mcp/rpc -GET /mcp/sse -``` - -Hermes skill 位于： - -```text -integrations/hermes/memory-gateway/ -``` - -常用脚本示例： - -```bash -python integrations/hermes/memory-gateway/scripts/everos_health.py -python integrations/hermes/memory-gateway/scripts/memory_commit_session.py --help -``` - -## 开发与验证 - -运行测试： - -```bash -cd /home/tom/memory-gateway -PYTHONPATH=/home/tom/memory-gateway pytest -q -``` - -编译检查： - -```bash -python -m compileall -q memory_gateway tests integrations/hermes/memory-gateway plugins/memory-gateway-agent -``` - -Ruff 已在 `pyproject.toml` 中配置。如果本地环境安装了 ruff： - -```bash -python -m ruff check . -``` - -当前仓库不要求真实 OpenViking / EverOS 服务才能跑单元测试；真实服务流程需要先启动 `127.0.0.1:1933` 和 `127.0.0.1:1995`。 - -## 设计约束 - -- SQLite 保存控制面 metadata，不作为长期记忆正文数据库。 -- `refs` 是后端 native 对象引用，不等于上下文正文。 -- `retrieve.items` 才是运行时上下文内容。 -- `commit` 只创建 job/outbox；长期 refs 由 outbox process 生成。 -- 默认只绑定本机地址；远程暴露时必须设置 API key、TLS 和网络访问控制。 diff --git a/config.example.yaml b/config.example.yaml index d0c3d43..e63756e 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -1,68 +1,23 @@ -# Memory Gateway 配置示例。 -# 复制为 config.yaml 并根据实际服务器路径、端口和密钥修改。 -# 不要提交 config.yaml；它应包含本机/服务器密钥。 - -# Memory Gateway 服务配置 server: - # 本机测试可用 127.0.0.1；需要远程调用时使用 0.0.0.0 并配置防火墙/反向代理。 host: "127.0.0.1" - # REST API、MCP RPC 和 SSE 共用端口。 port: 1934 - # 强烈建议生产/远程调用时设置；客户端通过 X-API-Key 传入。 api_key: "" -# OpenViking 后端配置 openviking: - # OpenViking 服务器地址。Memory Gateway 通过它检索 context/resource/memory。 url: "http://127.0.0.1:1933" - # OpenViking API Key。按 OpenViking 实际配置填写。 - api_key: "" + api_key: "your-secret-root-key" timeout: 30 + verify_ssl: true -# EverOS / EverCore 后台长期记忆整理服务 everos: - enabled: true - mode: "real" - # 指向 /home/tom/EverOS/methods/EverCore 启动的 API。 url: "http://127.0.0.1:1995" api_key: "" timeout: 30 + verify_ssl: true health_path: "/health" - ingest_path: "/api/v1/memories" - search_path: "/api/v1/memories/search" - flush_path: "/api/v1/memories/flush" - retrieve_method: "keyword" -# 记忆配置 -memory: - # 旧 /api/* 接口使用的默认命名空间。v1 API 会按 user/agent/workspace/session 自动展开 namespace。 - default_namespace: "memory-gateway" - search_limit: 10 +storage: + sqlite_path: "/home/tom/memory-gateway/memory_system_api.sqlite3" -# 日志配置 logging: level: "INFO" - format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s" - -# LLM 配置：用于 /api/summary 和 /api/knowledge/upload -# 兼容 OpenAI Chat Completions API；也可指向本地 vLLM / Ollama OpenAI-compatible endpoint。 -llm: - base_url: "https://api.openai.com/v1" - api_key: "" - model: "" - timeout: 60 - max_input_chars: 24000 - -# Obsidian Vault 配置。 -# 服务端不要求安装 Obsidian 桌面应用；这里本质上是一个 Markdown vault 目录。 -obsidian: - vault_path: "/opt/memory-gateway/obsidian-vault" - knowledge_dir: "01_Knowledge/Uploaded" - review_dir: "Reviews/Queue" - -# v1 metadata storage。 -# SQLite 保存 users、memories、episodes、profiles、audit，是用户隔离和 ACL 判断的主要 metadata store。 -# Use "memory" only for isolated unit tests. -storage: - backend: "sqlite" - sqlite_path: "/opt/memory-gateway/data/memory_gateway.sqlite3" diff --git a/docs/generic-memory-gateway-design.md b/docs/generic-memory-gateway-design.md deleted file mode 100644 index 340204c..0000000 --- a/docs/generic-memory-gateway-design.md +++ /dev/null @@ -1,767 +0,0 @@ -# 通用 Memory Gateway 方案与 POC 骨架 - -本文基于当前仓库的轻量 FastAPI + MCP + OpenViking + Obsidian 能力扩展，不把系统设计成重平台。第一阶段目标是先跑通多用户隔离、namespace routing、记忆检索、写入、session commit 和人工 review 草稿，后续再替换持久化、向量索引和 EverOS worker。 - -## A. 总体架构图 - -```mermaid -flowchart TB - subgraph Agents["Agent Frameworks"] - Nanobot[Nanobot] - Hermes[Hermes Agent] - OpenClaw[OpenClaw] - Other[Other Agents] - end - - subgraph Gateway["Memory Gateway"] - HTTP[HTTP API /v1] - MCP[MCP tools] - Auth[Auth / API Key / Future Login] - ACL[ACL & Visibility Policy] - Router[Namespace Router] - Audit[Audit Log] - Retrieval[Retrieval Orchestrator] - Writeback[Writeback Orchestrator] - end - - subgraph Skills["Skills Layer"] - Ingest[ingest] - Extract[extract] - Classify[classify] - Retrieve[retrieve] - Commit[commit] - Merge[merge] - Prune[prune] - Summarize[summarize] - end - - subgraph OpenViking["OpenViking"] - OVFS[context filesystem] - OVMem[memory] - OVRes[resources] - OVSkills[skills] - OVWorkspace[workspace] - end - - subgraph EverOS["EverOS"] - LTE[long-term extraction] - Consolidation[consolidation] - Decay[decay] - Dedup[dedup] - Profile[profile evolution] - end - - subgraph Obsidian["Obsidian"] - Vault[human editable memory vault] - Reviews[review queue] - Profiles[profiles] - LongTerm[long-term notes] - end - - subgraph Storage["Storage"] - DB[(metadata DB)] - Vector[(vector index)] - Files[(object / file storage)] - end - - Nanobot --> HTTP - Hermes --> MCP - OpenClaw --> HTTP - Other --> HTTP - Other --> MCP - - HTTP --> Auth --> ACL --> Router - MCP --> Auth - Router --> Retrieval - Router --> Writeback - ACL --> Audit - - Retrieval --> Skills - Writeback --> Skills - Skills --> OpenViking - Skills --> EverOS - Skills --> Obsidian - - Gateway --> DB - Gateway --> Vector - Gateway --> Files - OpenViking --> DB - OpenViking --> Vector - Obsidian --> Files - EverOS --> DB - EverOS --> Vector -``` - -## B. 核心数据模型 - -代码骨架见 `memory_gateway/schemas.py`。核心模型如下。 - -### User - -```json -{ - "id": "user_tom", - "display_name": "Tom", - "status": "active", - "profile_namespace": "user/user_tom/profile", - "preferences": {"language": "zh-CN"}, - "created_at": "2026-04-30T10:00:00Z", - "updated_at": "2026-04-30T10:00:00Z" -} -``` - -### Agent - -```json -{ - "id": "agent_hermes_default", - "name": "Hermes Default Agent", - "framework": "hermes", - "owner_user_id": "user_tom", - "created_at": "2026-04-30T10:00:00Z" -} -``` - -### Workspace - -```json -{ - "id": "ws_memory_gateway", - "name": "Memory Gateway POC", - "owner_user_id": "user_tom", - "member_user_ids": ["user_tom"], - "allowed_agent_ids": ["agent_hermes_default"] -} -``` - -### Session - -```json -{ - "id": "sess_20260430_001", - "user_id": "user_tom", - "agent_id": "agent_hermes_default", - "workspace_id": "ws_memory_gateway", - "status": "open", - "expires_at": "2026-05-07T10:00:00Z" -} -``` - -### MemoryRecord - -```json -{ - "id": "mem_abc123", - "user_id": "user_tom", - "agent_id": "agent_hermes_default", - "workspace_id": "ws_memory_gateway", - "session_id": "sess_20260430_001", - "namespace": "user/user_tom/long_term", - "memory_type": "preference", - "content": "用户偏好中文输出，结构化但不要过度平台化。", - "summary": "中文、结构化、轻量 POC 优先。", - "tags": ["preference", "style"], - "importance": 0.8, - "confidence": 0.9, - "visibility": "private", - "source": "conversation", - "created_at": "2026-04-30T10:00:00Z", - "updated_at": "2026-04-30T10:00:00Z", - "expires_at": null, - "version": 1 -} -``` - -### EpisodeRecord - -短期过程记录，默认不进入 Obsidian，不自动成为长期记忆。 - -```json -{ - "id": "epi_abc123", - "user_id": "user_tom", - "agent_id": "agent_hermes_default", - "workspace_id": "ws_memory_gateway", - "session_id": "sess_20260430_001", - "namespace": "session/sess_20260430_001/episodic", - "content": "本轮讨论了 Memory Gateway POC 范围。", - "summary": "确认 POC 优先做隔离、检索、写入和整理。", - "events": [], - "tags": ["design"] -} -``` - -### ProfileRecord - -```json -{ - "id": "profile_user_tom", - "user_id": "user_tom", - "namespace": "user/user_tom/profile", - "display_name": "Tom", - "stable_facts": ["正在设计通用 Memory Gateway"], - "preferences": {"language": "Chinese"}, - "working_style": ["偏好可落地 POC"], - "updated_from_memory_ids": ["mem_abc123"], - "version": 3 -} -``` - -### ACL / Visibility - -`visibility` 四档： - -- `private`：仅 `user_id` 相同可读写。 -- `agent-only`：同一 `user_id` 且同一 `agent_id` 可读写。 -- `workspace-shared`：在同一 `workspace_id` 且通过 workspace membership 授权后可读。 -- `global`：可公开检索，只能由受信任 actor 写入。 - -### AuditLog - -```json -{ - "id": "audit_abc123", - "actor_user_id": "user_tom", - "actor_agent_id": "agent_hermes_default", - "action": "memory_search", - "target_type": "memory", - "target_id": "mem_abc123", - "namespace": "user/user_tom/long_term", - "decision": "allow", - "reason": "private owner", - "created_at": "2026-04-30T10:00:00Z" -} -``` - -## C. Namespace 与隔离设计 - -推荐 namespace： - -```text -user/{user_id}/profile -user/{user_id}/preferences -user/{user_id}/long_term -agent/{agent_id}/memory -workspace/{workspace_id}/shared -session/{session_id}/episodic -global/public -``` - -隔离规则： - -- 用户隔离：所有 `user/{user_id}/...` 默认只允许同一 `user_id` 访问。Gateway 先校验 actor，再把 namespace 映射到 OpenViking URI。 -- Agent 隔离：`agent/{agent_id}/memory` 用于某个 agent 的工具经验、失败教训、prompt working notes。默认 `agent-only`。 -- Workspace 共享：`workspace/{workspace_id}/shared` 必须检查用户是否属于 workspace，agent 是否在 `allowed_agent_ids` 内。 -- Session 过期：`session/{session_id}/episodic` 必须有 TTL。过期后不可检索；只保留必要 audit。 -- 可跨 agent 共享：用户显式确认的 profile、preferences、user long_term、workspace shared、global public。 -- 不可跨 agent 共享：agent-only memory、未 commit 的 session episodic、低置信度候选记忆、含敏感凭据或临时日志的内容。 - -OpenViking URI 映射： - -```text -viking://memory/user/{user_id}/long_term/{memory_id}.json -viking://resources/workspace/{workspace_id}/shared/{slug}.md -viking://skills/memory-gateway/{skill_name} -``` - -## D. API 设计 - -第一阶段代码已挂载 `/v1` router，见 `memory_gateway/api_v1.py`。 - -### POST /v1/users - -Request: - -```json -{"user_id": "user_tom", "display_name": "Tom", "preferences": {"language": "zh-CN"}} -``` - -Response: - -```json -{"id": "user_tom", "display_name": "Tom", "profile_namespace": "user/user_tom/profile", "status": "active"} -``` - -### GET /v1/users/{user_id} - -Response: - -```json -{"id": "user_tom", "display_name": "Tom", "status": "active"} -``` - -### POST /v1/memory/search - -Request: - -```json -{ - "user_id": "user_tom", - "agent_id": "agent_hermes_default", - "workspace_id": "ws_memory_gateway", - "query": "中文输出偏好", - "namespaces": ["user/user_tom/long_term"], - "limit": 5 -} -``` - -Response: - -```json -{ - "results": [ - { - "memory": { - "id": "mem_abc123", - "namespace": "user/user_tom/long_term", - "summary": "中文、结构化、轻量 POC 优先。" - }, - "score": 2.7 - } - ], - "total": 1 -} -``` - -### POST /v1/memory - -Request: - -```json -{ - "user_id": "user_tom", - "agent_id": "agent_hermes_default", - "workspace_id": "ws_memory_gateway", - "memory_type": "preference", - "content": "用户偏好中文输出。", - "summary": "中文输出偏好", - "tags": ["preference"], - "importance": 0.8, - "confidence": 0.9, - "visibility": "private", - "source": "manual" -} -``` - -Response: - -```json -{"id": "mem_abc123", "namespace": "user/user_tom/long_term", "version": 1} -``` - -### GET /v1/memory/{memory_id} - -Request query: - -```text -?user_id=user_tom&agent_id=agent_hermes_default&workspace_id=ws_memory_gateway -``` - -Response: - -```json -{"id": "mem_abc123", "content": "用户偏好中文输出。", "visibility": "private"} -``` - -### PATCH /v1/memory/{memory_id} - -Request: - -```json -{"summary": "用户偏好中文、结构化、少废话。", "importance": 0.9} -``` - -Response: - -```json -{"id": "mem_abc123", "version": 2, "importance": 0.9} -``` - -### DELETE /v1/memory/{memory_id} - -Response: - -```json -{"deleted": true, "id": "mem_abc123"} -``` - -### POST /v1/episodes - -Request: - -```json -{ - "user_id": "user_tom", - "agent_id": "agent_hermes_default", - "workspace_id": "ws_memory_gateway", - "session_id": "sess_001", - "content": "本轮完成了 namespace 和 ACL 设计。", - "tags": ["design"] -} -``` - -Response: - -```json -{"id": "epi_abc123", "namespace": "session/sess_001/episodic"} -``` - -### POST /v1/sessions/{session_id}/commit - -Request: - -```json -{ - "user_id": "user_tom", - "agent_id": "agent_hermes_default", - "workspace_id": "ws_memory_gateway", - "promote": true, - "min_importance": 0.6, - "target_namespace": "user/user_tom/long_term" -} -``` - -Response: - -```json -{"session_id": "sess_001", "episodes": 3, "promoted": [{"id": "mem_def456"}]} -``` - -### GET /v1/users/{user_id}/profile - -Response: - -```json -{"user_id": "user_tom", "namespace": "user/user_tom/profile", "preferences": {"language": "zh-CN"}} -``` - -### POST /v1/memory/{memory_id}/feedback - -Request: - -```json -{"user_id": "user_tom", "feedback": "incorrect", "comment": "这是一次临时偏好，不应长期保留。"} -``` - -Response: - -```json -{"status": "ok", "memory_id": "mem_abc123", "feedback": "incorrect"} -``` - -### GET /v1/namespaces - -Request query: - -```text -?user_id=user_tom&agent_id=agent_hermes_default&workspace_id=ws_memory_gateway&session_id=sess_001 -``` - -Response: - -```json -[ - {"namespace": "user/user_tom/profile", "visibility": "private"}, - {"namespace": "agent/agent_hermes_default/memory", "visibility": "agent-only"}, - {"namespace": "workspace/ws_memory_gateway/shared", "visibility": "workspace-shared"} -] -``` - -### GET /v1/audit - -Response: - -```json -[{"action": "upsert_memory", "target_type": "memory", "decision": "allow"}] -``` - -### MCP tools - -目标 v1 tools 见 `memory_gateway/mcp_tools_v1.py`： - -- `memory_search` -- `memory_upsert` -- `memory_append_episode` -- `memory_commit_session` -- `memory_get_profile` -- `memory_list_namespaces` -- `memory_delete` -- `memory_feedback` - -示例 MCP call: - -```json -{ - "name": "memory_search", - "arguments": { - "user_id": "user_tom", - "agent_id": "agent_hermes_default", - "workspace_id": "ws_memory_gateway", - "query": "项目 POC 决策", - "limit": 5 - } -} -``` - -## E. Skills 设计 - -代码骨架位于 `memory_gateway/skills/`。 - -| Skill | 功能 | 输入 | 输出 | 触发时机 | 组件 | 写长期记忆 | -|---|---|---|---|---|---|---| -| `ingest_skill` | 标准化对话、文件、任务事件 | raw text/file/events | normalized payload | agent 写入 episode 前 | Gateway, file storage | 否 | -| `extract_memory_skill` | 从 episode/session 抽取候选记忆 | episode/session content | memory candidates | session commit / worker 定时 | LLM, EverOS | 否 | -| `classify_memory_skill` | 判断 memory_type、visibility、namespace | candidate memory | classification | 写入前 | ACL, namespace router | 否 | -| `retrieve_context_skill` | 聚合用户、agent、workspace 上下文 | query + context ids | ranked contexts | agent 调用前 | OpenViking, vector index | 否 | -| `commit_memory_skill` | 写入长期记忆 | MemoryRecord | stored record | 人工确认或 commit 通过 | DB, OpenViking | 是 | -| `summarize_episode_skill` | 压缩 episode | episode content | summary | session commit | LLM | 否 | -| `merge_memory_skill` | 合并重复或相近记忆 | memory ids | merged memory | EverOS 整理 | DB, vector index | 是 | -| `prune_memory_skill` | 衰减、归档、删除低质记忆 | policy + memory ids | archived/deleted list | 定时 worker | EverOS | 是 | -| `export_to_obsidian_skill` | 生成 Obsidian review draft | high-value memory | markdown draft | 高价值或需人工确认 | Obsidian | 否 | -| `import_from_obsidian_skill` | 从人工维护笔记导入记忆 | markdown path | MemoryRecord | vault sync | Obsidian, OpenViking | 是 | - -## F. Obsidian Vault 设计 - -推荐目录： - -```text -obsidian-vault/ -├── Users/ -│ └── {user_id}/ -│ ├── Profile.md -│ ├── Preferences.md -│ └── LongTerm/ -├── Agents/ -│ └── {agent_id}/Experience.md -├── Workspaces/ -│ └── {workspace_id}/Shared.md -├── Memories/ -│ ├── LongTerm/ -│ └── Archived/ -├── Profiles/ -├── Reviews/ -│ ├── Queue/ -│ ├── Accepted/ -│ └── Rejected/ -├── Exports/ -└── Templates/ -``` - -进入 Obsidian 的内容： - -- 人工可维护 profile、preferences、长期总结。 -- 高价值 workspace 知识、项目决策、复用经验。 -- EverOS 标记为 `needs_review` 的长期记忆草稿。 - -不进入 Obsidian 的内容： - -- 全量原始对话。 -- 高频工具日志、临时 session trace。 -- 低置信度候选记忆。 -- 敏感凭据、token、临时错误栈。 - -标签体系： - -```text -#memory/profile -#memory/preference -#memory/long-term -#memory/workspace -#memory/agent-experience -#memory/review -#memory/conflict -#memory/deprecated -#source/everos -#source/manual -#visibility/private -#visibility/workspace-shared -``` - -模板文件已加入 `obsidian-vault/05_Templates/`。 - -## G. OpenViking 设计 - -OpenViking 作为统一 context 层，Gateway 不要求 agent 直接理解 OpenViking 内部结构。 - -组织方式： - -```text -viking://memory/user/{user_id}/profile -viking://memory/user/{user_id}/preferences -viking://memory/user/{user_id}/long_term -viking://memory/agent/{agent_id}/memory -viking://memory/workspace/{workspace_id}/shared -viking://resources/user/{user_id}/obsidian/{note_id}.md -viking://skills/memory-gateway/{skill_name} -``` - -检索路径： - -1. Agent 调用 Gateway `/v1/memory/search` 或 MCP `memory_search`。 -2. Gateway 执行 Auth、ACL、namespace expansion。 -3. Gateway 查询 metadata DB 和 vector index，必要时调用 OpenViking search。 -4. 返回统一 `MemoryRecord` 或 context chunk，不暴露底层差异。 - -同步： - -- Obsidian accepted note 通过 `import_from_obsidian_skill` 写回 Gateway，再同步 OpenViking resource。 -- EverOS consolidation 后写入 `user/{user_id}/long_term` 或 `workspace/{workspace_id}/shared`。 -- Gateway 保存 `source_ref`，避免 OpenViking 与 Obsidian 互相重复导入。 - -## H. EverOS 设计 - -输入来源： - -- `EpisodeRecord`：对话片段、任务执行摘要、agent 过程事件。 -- `SessionRecord`：session commit 包。 -- `MemoryFeedback`：incorrect、duplicate、outdated 等反馈。 -- Obsidian review 结果：accepted/rejected/edited。 - -整理流程： - -1. 抽取：从 episode 中提炼候选事实、偏好、决策、经验。 -2. 打分：根据重要性、稳定性、重复出现次数、来源可信度打分。 -3. 去重：按 semantic hash + embedding 相似度查找近似 MemoryRecord。 -4. 合并：相同事实合并 evidence；更高置信度覆盖低置信度。 -5. 冲突检测：同一 subject 的相反陈述标记 `needs_review`，不自动覆盖。 -6. 衰减：长时间未命中且低反馈的记忆降低 importance。 -7. 归档：过期、错误、低置信度、被人工拒绝的记忆转 archived。 -8. profile evolution：只有稳定、重复、高置信偏好进入 ProfileRecord。 - -污染控制： - -- session 临时内容不直接提升为长期记忆。 -- LLM 抽取结果默认是 candidate，需阈值或人工确认。 -- 每条长期记忆保留 source、confidence、version、feedback。 -- 对 profile 更新采用 evidence count，禁止一次对话永久改写强偏好。 - -## I. 工程目录结构 - -当前仓库保留 `memory_gateway/` 包名，目标结构如下： - -```text -memory-gateway/ -├── memory_gateway/ -│ ├── api_v1.py # v1 HTTP API -│ ├── mcp_tools_v1.py # v1 MCP tool contract -│ ├── schemas.py # User/Memory/Episode/Profile/ACL/Audit -│ ├── namespace.py # namespace builder + ACL helpers -│ ├── services.py # orchestration service -│ ├── repositories.py # POC in-memory repo; later DB repo -│ ├── security/ # future auth, RBAC, audit policy -│ ├── skills/ -│ │ ├── ingest_skill.py -│ │ ├── extract_memory_skill.py -│ │ ├── classify_memory_skill.py -│ │ ├── retrieve_context_skill.py -│ │ ├── commit_memory_skill.py -│ │ ├── summarize_episode_skill.py -│ │ ├── merge_memory_skill.py -│ │ ├── prune_memory_skill.py -│ │ ├── export_to_obsidian_skill.py -│ │ └── import_from_obsidian_skill.py -│ ├── adapters/ -│ │ ├── openviking.py -│ │ ├── everos.py -│ │ └── obsidian.py -│ └── workers/ -│ └── everos_worker.py -├── obsidian-vault/ -├── integrations/ -│ ├── nanobot/ -│ ├── hermes/ -│ └── openclaw/ -└── tests/ -``` - -如果未来迁移到更标准的 `app/`，可把 `memory_gateway/api_v1.py` 对应到 `app/api`，`schemas.py` 对应到 `app/schemas`，`services.py` 对应到 `app/services`。 - -## J. 2 到 4 周 POC 实施计划 - -第一周： - -- 完成 `/v1/users`、`/v1/memory`、`/v1/memory/search`、`/v1/episodes`。 -- 实现 namespace router、visibility、基础 audit。 -- 存储先用 SQLite 或当前内存 repo，搜索先用 lexical，OpenViking 作为可选后端。 - -第二周： - -- 接入 OpenViking URI 写入和检索。 -- 实现 `retrieve_context_skill`、`commit_memory_skill`、`summarize_episode_skill`。 -- 给 Hermes/Nanobot/OpenClaw 提供最小 client 示例。 - -第三周： - -- 加 EverOS worker 原型：session commit、candidate extraction、dedup、merge。 -- 增加 feedback 流程：incorrect、duplicate、outdated 影响 prune/merge。 -- 生成 Obsidian review draft，而不是直接写入最终知识库。 - -第四周： - -- Obsidian import/export 双向同步。 -- 增加 profile evolution 的阈值和 evidence 机制。 -- 补充权限测试、污染测试、重复记忆测试、跨 agent 检索测试。 - -先做： - -- 用户隔离、namespace、memory CRUD、episode append、session commit、basic search、audit。 - -暂不做： - -- 完整登录系统、复杂 RBAC、多租户计费、实时同步、复杂 UI、全量向量数据库治理。 - -POC 成功指标： - -- 不同 `user_id` 之间无法互相读写 private memory。 -- 同一 workspace 的共享记忆可被授权 agent 检索。 -- session 记忆不会自动污染长期记忆。 -- 10 条重复候选能合并到 1 到 2 条长期记忆。 -- 错误反馈后，该记忆不再进入默认 retrieval。 -- Hermes/Nanobot/OpenClaw 至少两个框架能通过统一 API 调用。 - -## K. 推荐默认方案 - -第一阶段最合理默认方案： - -- FastAPI 提供 `/v1` 统一 HTTP API。 -- MCP 先保留现有 `/mcp/rpc`，新增 `memory_gateway/mcp_tools_v1.py` 作为目标 contract。 -- 存储使用 SQLite metadata + 本地文件存 object；当前代码先用 in-memory repo 验证接口。 -- 搜索先用 OpenViking search + 简单 lexical fallback；向量索引第二阶段引入。 -- Obsidian 只保存人工可读的高价值长期记忆和 review draft。 -- EverOS 第一阶段不做独立大系统，只做 worker 模块：extract、dedup、merge、prune、profile update。 - -第一阶段实现 API： - -- `POST /v1/users` -- `GET /v1/users/{user_id}` -- `POST /v1/memory/search` -- `POST /v1/memory` -- `GET /v1/memory/{memory_id}` -- `POST /v1/episodes` -- `POST /v1/sessions/{session_id}/commit` -- `GET /v1/users/{user_id}/profile` -- `GET /v1/namespaces` - -第一阶段实现 skills： - -- `ingest_skill` -- `summarize_episode_skill` -- `retrieve_context_skill` -- `commit_memory_skill` -- `export_to_obsidian_skill` - -第二阶段再补： - -- `extract_memory_skill` -- `classify_memory_skill` -- `merge_memory_skill` -- `prune_memory_skill` -- `import_from_obsidian_skill` -- 更完整的 EverOS consolidation 和 profile evolution。 - -角色分工： - -- Obsidian 第一阶段：review draft、人类确认 profile/长期知识。第二阶段：双向同步。 -- OpenViking 第一阶段：统一 context/resource 检索入口。第二阶段：承载多 namespace context filesystem 和 skill registry。 -- EverOS 第一阶段：session commit worker。第二阶段：长期记忆治理、衰减、冲突检测、profile evolution。 - diff --git a/docs/openviking_adapter_config.md b/docs/openviking_adapter_config.md deleted file mode 100644 index 0fa68e6..0000000 --- a/docs/openviking_adapter_config.md +++ /dev/null @@ -1,93 +0,0 @@ -# OpenViking Adapter Config - -## Overview - -Memory Gateway v2 keeps the OpenViking ingest adapter in `offline` / `skeleton` -mode by default. In the default configuration it does not send any HTTP -requests. - -## Modes - -### Offline - -`mode: offline` - -The adapter must not touch the network. It returns fixture-backed normalized -results through the existing skeleton path. - -### Skeleton - -`mode: skeleton` - -This behaves like `offline` for now. It keeps the same normalized result path -without sending HTTP requests. - -### Real - -Real mode is enabled only when: - -- `mode: real` - -When real mode is active, the adapter may send an HTTP request for OpenViking -ingest only. Commit and retrieve remain offline/skeleton in the current phase. -The legacy `enabled` field is retained for config compatibility, but it does -not open the network path by itself. - -## Config Fields - -- `base_url` - The OpenViking API base URL. -- `api_key` - Token used only for request headers. -- `timeout` - Request timeout in seconds. -- `verify_ssl` - TLS verification toggle for the real HTTP path. -- `ingest_path` - Configurable ingest endpoint path template. The current placeholder is - `/api/v1/sessions/{session_id}/messages`. - -## Example Config - -### Offline Example - -```yaml -openviking: - enabled: false - mode: offline - url: http://localhost:1933 - timeout: 30 - verify_ssl: true -``` - -### Real Example - -```yaml -openviking: - enabled: false - mode: real - url: https://openviking.example.internal - api_key: YOUR_OPENVIKING_TOKEN - timeout: 30 - verify_ssl: true - ingest_path: /api/v1/sessions/{session_id}/messages -``` - -## Security - -Runtime ingest requests may temporarily include `content` while the current -request is in flight. Memory Gateway does not persist `content`, -`raw_request`, `messages`, or `transcript` into SQLite metadata, outbox -payloads, or audit summaries. - -`api_key` / tokens are used only in request headers. They do not belong in: - -- adapter result metadata -- audit summaries -- persisted MemoryRef metadata -- error messages - -## Notes - -The current ingest endpoint path is still a configurable placeholder. It should -be calibrated once the real OpenViking API contract is stable. diff --git a/integrations/hermes/memory-gateway/SKILL.md b/integrations/hermes/memory-gateway/SKILL.md deleted file mode 100644 index 759a18b..0000000 --- a/integrations/hermes/memory-gateway/SKILL.md +++ /dev/null @@ -1,290 +0,0 @@ ---- -name: memory-gateway -description: Use this skill when Hermes needs shared long-term memory, user-scoped preferences/profile, workspace memory, session episode capture, Memory Gateway retrieval, OpenViking context search, Obsidian document upload/review, or session commit through the standalone EverOS service. This skill is domain-neutral. -version: 3.1.0 -metadata: - hermes: - tags: [memory, memory-gateway, openviking, obsidian, everos, long-term-memory, retrieval, agent-context] ---- - -# Memory Gateway - -Use this skill as Hermes' generic memory layer. It connects Hermes to the local Memory Gateway at `http://127.0.0.1:1934`. - -The gateway provides: - -- v1 user/agent/workspace/session aware memory APIs backed by SQLite metadata. -- ACL and namespace routing before retrieval. -- OpenViking fan-out search for visible namespaces. -- Session episode capture and commit through the standalone EverOS HTTP service, with Gateway local fallback only when configured. -- Obsidian review drafts for high-value or conflicting long-term memory candidates. -- Legacy summary/document upload endpoints for LLM summarization and Obsidian knowledge ingestion. - -## Environment - -Defaults: - -- Memory Gateway URL: `http://127.0.0.1:1934` -- EverOS URL through Gateway config: `http://127.0.0.1:1995` -- Obsidian vault: `/home/tom/memory-gateway/obsidian-vault` -- Default review queue: `/home/tom/memory-gateway/obsidian-vault/Reviews/Queue` - -Optional env vars: - -- `MEMORY_GATEWAY_URL` -- `MEMORY_GATEWAY_API_KEY` -- `MEMORY_GATEWAY_OBSIDIAN_VAULT` - -## Recommended Hermes Workflow - -For normal agent work: - -1. Search memory before answering if prior context may matter. -2. Append important session episodes while working. -3. Commit the session at the end so EverOS can promote stable memories. -4. Use feedback to mark incorrect, duplicate, outdated, or useful memories. -5. Upload documents only when they are reusable knowledge, not raw noisy logs. - -Do not write full transcripts to long-term memory. Use episodes for temporary process capture and commit only stable conclusions. - -## v1 Memory Commands - -### Check EverOS - -```bash -python /home/tom/.hermes/skills/memory-gateway/scripts/everos_health.py -``` - -Expected healthy response includes `status: ok` and `response.service: everos-local`. - -### Create User - -```bash -python /home/tom/.hermes/skills/memory-gateway/scripts/memory_create_user.py \ - --user-id user_tom \ - --display-name "Tom" \ - --preference language=zh-CN -``` - -### Search ACL-Aware Memory - -```bash -python /home/tom/.hermes/skills/memory-gateway/scripts/memory_search.py \ - --user-id user_tom \ - --agent-id agent_hermes \ - --workspace-id ws_memory_gateway \ - --query "namespace ACL decision" \ - --limit 5 -``` - -Equivalent backward-compatible command: - -```bash -python /home/tom/.hermes/skills/memory-gateway/scripts/retrieve_memory.py \ - --user-id user_tom \ - --agent-id agent_hermes \ - --workspace-id ws_memory_gateway \ - --query "namespace ACL decision" \ - --limit 5 -``` - -If `retrieve_memory.py` is called without `--user-id`, it falls back to the legacy `/api/search` endpoint. - -### Upsert Long-Term Memory - -Use this only for stable, concise, reusable memory. - -```bash -python /home/tom/.hermes/skills/memory-gateway/scripts/memory_upsert.py \ - --user-id user_tom \ - --agent-id agent_hermes \ - --workspace-id ws_memory_gateway \ - --memory-type preference \ - --visibility private \ - --importance 0.8 \ - --confidence 0.9 \ - --tag preference \ - --summary "中文、结构化、轻量 POC 优先" \ - --text "用户偏好中文输出，结构化但不要过度工程化。" -``` - -### Append Session Episode - -Use this during a task to record useful process notes without immediately polluting long-term memory. - -```bash -python /home/tom/.hermes/skills/memory-gateway/scripts/memory_append_episode.py \ - --user-id user_tom \ - --agent-id agent_hermes \ - --workspace-id ws_memory_gateway \ - --session-id sess_demo \ - --tag decision \ - --text "结论：这个项目必须保留用户隔离和 namespace ACL。" -``` - -### Commit Session Through EverOS - -This asks Memory Gateway to call the standalone EverOS service configured in `config.yaml`. -For local POC the default service is `http://127.0.0.1:1995`. If `everos.fallback_to_local` is true and the service is unavailable, Gateway returns `everos_backend: local-fallback`. - -- extracts candidate memories from session episodes -- deduplicates exact repeated candidates -- detects simple conflicts -- promotes normal stable memories into SQLite long-term memory -- sends high-value or conflicting candidates to Obsidian review drafts - -```bash -python /home/tom/.hermes/skills/memory-gateway/scripts/memory_commit_session.py \ - --user-id user_tom \ - --agent-id agent_hermes \ - --workspace-id ws_memory_gateway \ - --session-id sess_demo \ - --min-importance 0.6 -``` - -Review drafts are written under: - -```text -/home/tom/memory-gateway/obsidian-vault/Reviews/Queue/ -``` - -### Get Profile - -```bash -python /home/tom/.hermes/skills/memory-gateway/scripts/memory_get_profile.py \ - --user-id user_tom -``` - -### List Visible Namespaces - -```bash -python /home/tom/.hermes/skills/memory-gateway/scripts/memory_list_namespaces.py \ - --user-id user_tom \ - --agent-id agent_hermes \ - --workspace-id ws_memory_gateway \ - --session-id sess_demo -``` - -### Patch Memory - -```bash -python /home/tom/.hermes/skills/memory-gateway/scripts/memory_patch.py \ - --memory-id mem_xxx \ - --user-id user_tom \ - --agent-id agent_hermes \ - --workspace-id ws_memory_gateway \ - --summary "用户偏好中文、结构化、少废话。" \ - --importance 0.9 \ - --tag preference \ - --tag confirmed -``` - -### Feedback - -```bash -python /home/tom/.hermes/skills/memory-gateway/scripts/memory_feedback.py \ - --memory-id mem_xxx \ - --user-id user_tom \ - --agent-id agent_hermes \ - --workspace-id ws_memory_gateway \ - --feedback incorrect \ - --comment "这是临时偏好，不应长期保留。" -``` - -### Delete Memory - -```bash -python /home/tom/.hermes/skills/memory-gateway/scripts/memory_delete.py \ - --memory-id mem_xxx \ - --user-id user_tom \ - --agent-id agent_hermes \ - --workspace-id ws_memory_gateway -``` - -## Knowledge And Obsidian Commands - -### Summarize And Commit Via Legacy LLM Endpoint - -Use this for high-value text that should become an OpenViking resource or summarized memory. - -```bash -python /home/tom/.hermes/skills/memory-gateway/scripts/commit_summary.py \ - --title "Project decision summary" \ - --namespace memory-gateway \ - --memory-type decision \ - --tag project --tag decision \ - --persist-as resource \ - --text "" -``` - -This calls `POST /api/summary`. - -### Upload Document As Knowledge - -```bash -python /home/tom/.hermes/skills/memory-gateway/scripts/upload_knowledge.py \ - --file /path/to/document.pdf \ - --title "Design Notes" \ - --namespace memory-gateway \ - --knowledge-type design_doc \ - --tags project,design,reference \ - --persist-as resource -``` - -This calls `POST /api/knowledge/upload`: document -> MarkItDown Markdown -> Obsidian note -> LLM summary -> OpenViking resource. - -### Search Local Obsidian Notes - -```bash -python /home/tom/.hermes/skills/memory-gateway/scripts/search_obsidian.py \ - --query "design notes memory gateway" \ - --limit 5 -``` - -## MCP Tool Names - -The gateway also exposes these v1 tools through `/mcp/rpc`: - -- `memory_search` -- `memory_upsert` -- `memory_append_episode` -- `memory_commit_session` -- `memory_get_profile` -- `memory_list_namespaces` -- `memory_delete` -- `memory_feedback` - -Use MCP tools when Hermes has an MCP bridge available. Use the scripts above when Hermes runs skills as shell commands. - -## Output Template - -When using this skill, answer with: - -```markdown -## Answer - - -## Memory References -- `` — `` — why it matters - -## Obsidian Review -- `` — why it needs review - -## Memory Action -- searched: yes/no -- appended_episode: yes/no -- committed_session: yes/no -- promoted_memory_count: -- review_draft_count: -``` - -## Guardrails - -- Do not store raw noisy data as long-term memory. -- Use `memory_append_episode.py` for temporary process notes. -- Use `memory_commit_session.py` at task end to let EverOS decide what should persist. -- Use `memory_upsert.py` directly only for stable, concise, user-approved memory. -- Do not commit secrets, credentials, tokens, private keys, or unnecessary personal data. -- If content is sensitive, summarize and redact before committing. -- High-value or conflicting candidates should go to Obsidian review drafts before becoming durable memory. -- Always report whether retrieval, episode append, session commit, or upload actually succeeded. diff --git a/integrations/hermes/memory-gateway/scripts/_client.py b/integrations/hermes/memory-gateway/scripts/_client.py deleted file mode 100755 index b346ae2..0000000 --- a/integrations/hermes/memory-gateway/scripts/_client.py +++ /dev/null @@ -1,52 +0,0 @@ -from __future__ import annotations - -import json -import os -import urllib.parse -import urllib.request -from typing import Any - -DEFAULT_GATEWAY_URL = os.environ.get("MEMORY_GATEWAY_URL", "http://127.0.0.1:1934") -DEFAULT_GATEWAY_API_KEY = os.environ.get("MEMORY_GATEWAY_API_KEY", "") - - -def post_json(path: str, payload: dict[str, Any], gateway_url: str = DEFAULT_GATEWAY_URL, api_key: str = DEFAULT_GATEWAY_API_KEY, timeout: int = 120) -> dict[str, Any]: - data = json.dumps(payload, ensure_ascii=False).encode("utf-8") - req = urllib.request.Request(gateway_url.rstrip("/") + path, data=data, method="POST") - req.add_header("Content-Type", "application/json") - if api_key: - req.add_header("X-API-Key", api_key) - with urllib.request.urlopen(req, timeout=timeout) as resp: - return json.loads(resp.read().decode("utf-8")) - - -def get_json(path: str, params: dict[str, Any] | None = None, gateway_url: str = DEFAULT_GATEWAY_URL, api_key: str = DEFAULT_GATEWAY_API_KEY, timeout: int = 120) -> dict[str, Any] | list[Any]: - query = urllib.parse.urlencode({k: v for k, v in (params or {}).items() if v not in (None, "")}) - url = gateway_url.rstrip("/") + path + (f"?{query}" if query else "") - req = urllib.request.Request(url, method="GET") - if api_key: - req.add_header("X-API-Key", api_key) - with urllib.request.urlopen(req, timeout=timeout) as resp: - return json.loads(resp.read().decode("utf-8")) - - -def patch_json(path: str, payload: dict[str, Any], params: dict[str, Any] | None = None, gateway_url: str = DEFAULT_GATEWAY_URL, api_key: str = DEFAULT_GATEWAY_API_KEY, timeout: int = 120) -> dict[str, Any]: - query = urllib.parse.urlencode({k: v for k, v in (params or {}).items() if v not in (None, "")}) - url = gateway_url.rstrip("/") + path + (f"?{query}" if query else "") - data = json.dumps(payload, ensure_ascii=False).encode("utf-8") - req = urllib.request.Request(url, data=data, method="PATCH") - req.add_header("Content-Type", "application/json") - if api_key: - req.add_header("X-API-Key", api_key) - with urllib.request.urlopen(req, timeout=timeout) as resp: - return json.loads(resp.read().decode("utf-8")) - - -def delete_json(path: str, params: dict[str, Any] | None = None, gateway_url: str = DEFAULT_GATEWAY_URL, api_key: str = DEFAULT_GATEWAY_API_KEY, timeout: int = 120) -> dict[str, Any]: - query = urllib.parse.urlencode({k: v for k, v in (params or {}).items() if v not in (None, "")}) - url = gateway_url.rstrip("/") + path + (f"?{query}" if query else "") - req = urllib.request.Request(url, method="DELETE") - if api_key: - req.add_header("X-API-Key", api_key) - with urllib.request.urlopen(req, timeout=timeout) as resp: - return json.loads(resp.read().decode("utf-8")) diff --git a/integrations/hermes/memory-gateway/scripts/commit_summary.py b/integrations/hermes/memory-gateway/scripts/commit_summary.py deleted file mode 100755 index eb6640e..0000000 --- a/integrations/hermes/memory-gateway/scripts/commit_summary.py +++ /dev/null @@ -1,53 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import json -import sys -from pathlib import Path -from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, post_json - - -def load_text(args: argparse.Namespace) -> str: - if args.file: - return Path(args.file).read_text(encoding="utf-8") - if args.text: - return args.text - return sys.stdin.read().strip() - - -def main() -> None: - parser = argparse.ArgumentParser(description="Summarize arbitrary content with the Gateway LLM and commit it as memory/resource.") - parser.add_argument("--text", help="Text to summarize; stdin is used if omitted") - parser.add_argument("--file", help="File containing text to summarize") - parser.add_argument("--title", default="") - parser.add_argument("--summary", default="", help="Optional summary hint") - parser.add_argument("--namespace", default="memory-gateway") - parser.add_argument("--memory-type", default="summary") - parser.add_argument("--tag", action="append", default=[]) - parser.add_argument("--source", default="hermes:memory-gateway") - parser.add_argument("--resource-uri", default="") - parser.add_argument("--persist-as", choices=["memory", "resource", "both", "none"], default="resource") - parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL) - parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY) - args = parser.parse_args() - - content = load_text(args) - if not content: - parser.error("No content provided via --text, --file, or stdin") - payload = { - "content": content, - "title": args.title or None, - "summary": args.summary or None, - "namespace": args.namespace, - "memory_type": args.memory_type, - "tags": args.tag, - "source": args.source, - "resource_uri": args.resource_uri or None, - "persist_as": args.persist_as, - } - print(json.dumps(post_json("/api/summary", payload, args.gateway_url, args.api_key), ensure_ascii=False, indent=2)) - - -if __name__ == "__main__": - main() diff --git a/integrations/hermes/memory-gateway/scripts/everos_health.py b/integrations/hermes/memory-gateway/scripts/everos_health.py deleted file mode 100644 index 66c2004..0000000 --- a/integrations/hermes/memory-gateway/scripts/everos_health.py +++ /dev/null @@ -1,21 +0,0 @@ -#!/usr/bin/env python3 -"""Check EverOS health through Memory Gateway.""" - -from __future__ import annotations - -import argparse -import json - -from _client import get_json - - -def main() -> None: - parser = argparse.ArgumentParser(description="Check EverOS health through Memory Gateway.") - parser.add_argument("--gateway-url", default="http://127.0.0.1:1934") - parser.add_argument("--api-key", default=None) - args = parser.parse_args() - print(json.dumps(get_json("/v1/everos/health", gateway_url=args.gateway_url, api_key=args.api_key), ensure_ascii=False, indent=2)) - - -if __name__ == "__main__": - main() diff --git a/integrations/hermes/memory-gateway/scripts/memory_append_episode.py b/integrations/hermes/memory-gateway/scripts/memory_append_episode.py deleted file mode 100644 index e51c65b..0000000 --- a/integrations/hermes/memory-gateway/scripts/memory_append_episode.py +++ /dev/null @@ -1,54 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import json -import sys -from pathlib import Path - -from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, post_json - - -def load_content(args: argparse.Namespace) -> str: - if args.file: - return Path(args.file).read_text(encoding="utf-8") - if args.text: - return args.text - return sys.stdin.read().strip() - - -def main() -> None: - parser = argparse.ArgumentParser(description="Append session episode memory without directly promoting it.") - parser.add_argument("--user-id", required=True) - parser.add_argument("--session-id", required=True) - parser.add_argument("--agent-id", default="") - parser.add_argument("--workspace-id", default="") - parser.add_argument("--namespace", default="") - parser.add_argument("--text", default="") - parser.add_argument("--file", default="") - parser.add_argument("--tag", action="append", default=[]) - parser.add_argument("--source", default="conversation") - parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL) - parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY) - args = parser.parse_args() - - content = load_content(args) - if not content: - parser.error("No episode content provided via --text, --file, or stdin") - - payload = { - "user_id": args.user_id, - "agent_id": args.agent_id or None, - "workspace_id": args.workspace_id or None, - "session_id": args.session_id, - "namespace": args.namespace or None, - "content": content, - "tags": args.tag, - "source": args.source, - } - print(json.dumps(post_json("/v1/episodes", payload, args.gateway_url, args.api_key), ensure_ascii=False, indent=2)) - - -if __name__ == "__main__": - main() - diff --git a/integrations/hermes/memory-gateway/scripts/memory_commit_session.py b/integrations/hermes/memory-gateway/scripts/memory_commit_session.py deleted file mode 100644 index ea0aab7..0000000 --- a/integrations/hermes/memory-gateway/scripts/memory_commit_session.py +++ /dev/null @@ -1,37 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import json - -from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, post_json - - -def main() -> None: - parser = argparse.ArgumentParser(description="Commit a session through the minimal EverOS consolidation worker.") - parser.add_argument("--user-id", required=True) - parser.add_argument("--session-id", required=True) - parser.add_argument("--agent-id", default="") - parser.add_argument("--workspace-id", default="") - parser.add_argument("--target-namespace", default="") - parser.add_argument("--min-importance", type=float, default=0.6) - parser.add_argument("--no-promote", action="store_true") - parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL) - parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY) - args = parser.parse_args() - - payload = { - "user_id": args.user_id, - "agent_id": args.agent_id or None, - "workspace_id": args.workspace_id or None, - "session_id": args.session_id, - "promote": not args.no_promote, - "min_importance": args.min_importance, - "target_namespace": args.target_namespace or None, - } - print(json.dumps(post_json(f"/v1/sessions/{args.session_id}/commit", payload, args.gateway_url, args.api_key), ensure_ascii=False, indent=2)) - - -if __name__ == "__main__": - main() - diff --git a/integrations/hermes/memory-gateway/scripts/memory_create_user.py b/integrations/hermes/memory-gateway/scripts/memory_create_user.py deleted file mode 100644 index f4bf9a9..0000000 --- a/integrations/hermes/memory-gateway/scripts/memory_create_user.py +++ /dev/null @@ -1,36 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import json - -from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, post_json - - -def main() -> None: - parser = argparse.ArgumentParser(description="Create or replace a Memory Gateway v1 user.") - parser.add_argument("--user-id", required=True) - parser.add_argument("--display-name", required=True) - parser.add_argument("--preference", action="append", default=[], help="Preference as key=value; repeatable") - parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL) - parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY) - args = parser.parse_args() - - preferences = {} - for item in args.preference: - if "=" not in item: - parser.error(f"Invalid --preference {item!r}; expected key=value") - key, value = item.split("=", 1) - preferences[key.strip()] = value.strip() - - payload = { - "user_id": args.user_id, - "display_name": args.display_name, - "preferences": preferences, - } - print(json.dumps(post_json("/v1/users", payload, args.gateway_url, args.api_key), ensure_ascii=False, indent=2)) - - -if __name__ == "__main__": - main() - diff --git a/integrations/hermes/memory-gateway/scripts/memory_delete.py b/integrations/hermes/memory-gateway/scripts/memory_delete.py deleted file mode 100644 index da95e3c..0000000 --- a/integrations/hermes/memory-gateway/scripts/memory_delete.py +++ /dev/null @@ -1,31 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import json - -from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, delete_json - - -def main() -> None: - parser = argparse.ArgumentParser(description="Delete a MemoryRecord if the caller has access.") - parser.add_argument("--memory-id", required=True) - parser.add_argument("--user-id", required=True) - parser.add_argument("--agent-id", default="") - parser.add_argument("--workspace-id", default="") - parser.add_argument("--session-id", default="") - parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL) - parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY) - args = parser.parse_args() - params = { - "user_id": args.user_id, - "agent_id": args.agent_id, - "workspace_id": args.workspace_id, - "session_id": args.session_id, - } - print(json.dumps(delete_json(f"/v1/memory/{args.memory_id}", params=params, gateway_url=args.gateway_url, api_key=args.api_key), ensure_ascii=False, indent=2)) - - -if __name__ == "__main__": - main() - diff --git a/integrations/hermes/memory-gateway/scripts/memory_feedback.py b/integrations/hermes/memory-gateway/scripts/memory_feedback.py deleted file mode 100644 index bfe73dd..0000000 --- a/integrations/hermes/memory-gateway/scripts/memory_feedback.py +++ /dev/null @@ -1,35 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import json - -from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, post_json - - -def main() -> None: - parser = argparse.ArgumentParser(description="Attach quality feedback to a MemoryRecord.") - parser.add_argument("--memory-id", required=True) - parser.add_argument("--user-id", required=True) - parser.add_argument("--agent-id", default="") - parser.add_argument("--workspace-id", default="") - parser.add_argument("--session-id", default="") - parser.add_argument("--feedback", required=True, choices=["useful", "not_useful", "incorrect", "duplicate", "outdated"]) - parser.add_argument("--comment", default="") - parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL) - parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY) - args = parser.parse_args() - payload = { - "user_id": args.user_id, - "agent_id": args.agent_id or None, - "workspace_id": args.workspace_id or None, - "session_id": args.session_id or None, - "feedback": args.feedback, - "comment": args.comment or None, - } - print(json.dumps(post_json(f"/v1/memory/{args.memory_id}/feedback", payload, args.gateway_url, args.api_key), ensure_ascii=False, indent=2)) - - -if __name__ == "__main__": - main() - diff --git a/integrations/hermes/memory-gateway/scripts/memory_get_profile.py b/integrations/hermes/memory-gateway/scripts/memory_get_profile.py deleted file mode 100644 index 2dd5915..0000000 --- a/integrations/hermes/memory-gateway/scripts/memory_get_profile.py +++ /dev/null @@ -1,21 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import json - -from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, get_json - - -def main() -> None: - parser = argparse.ArgumentParser(description="Get a user's Memory Gateway profile.") - parser.add_argument("--user-id", required=True) - parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL) - parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY) - args = parser.parse_args() - print(json.dumps(get_json(f"/v1/users/{args.user_id}/profile", gateway_url=args.gateway_url, api_key=args.api_key), ensure_ascii=False, indent=2)) - - -if __name__ == "__main__": - main() - diff --git a/integrations/hermes/memory-gateway/scripts/memory_list_namespaces.py b/integrations/hermes/memory-gateway/scripts/memory_list_namespaces.py deleted file mode 100644 index acecf65..0000000 --- a/integrations/hermes/memory-gateway/scripts/memory_list_namespaces.py +++ /dev/null @@ -1,30 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import json - -from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, get_json - - -def main() -> None: - parser = argparse.ArgumentParser(description="List namespaces visible to a user/agent/workspace/session context.") - parser.add_argument("--user-id", required=True) - parser.add_argument("--agent-id", default="") - parser.add_argument("--workspace-id", default="") - parser.add_argument("--session-id", default="") - parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL) - parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY) - args = parser.parse_args() - params = { - "user_id": args.user_id, - "agent_id": args.agent_id, - "workspace_id": args.workspace_id, - "session_id": args.session_id, - } - print(json.dumps(get_json("/v1/namespaces", params=params, gateway_url=args.gateway_url, api_key=args.api_key), ensure_ascii=False, indent=2)) - - -if __name__ == "__main__": - main() - diff --git a/integrations/hermes/memory-gateway/scripts/memory_patch.py b/integrations/hermes/memory-gateway/scripts/memory_patch.py deleted file mode 100644 index ebc26d4..0000000 --- a/integrations/hermes/memory-gateway/scripts/memory_patch.py +++ /dev/null @@ -1,54 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import json - -from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, patch_json - - -def main() -> None: - parser = argparse.ArgumentParser(description="Patch a MemoryRecord.") - parser.add_argument("--memory-id", required=True) - parser.add_argument("--user-id", required=True) - parser.add_argument("--agent-id", default="") - parser.add_argument("--workspace-id", default="") - parser.add_argument("--session-id", default="") - parser.add_argument("--content", default="") - parser.add_argument("--summary", default="") - parser.add_argument("--tag", action="append", default=None) - parser.add_argument("--importance", type=float, default=None) - parser.add_argument("--confidence", type=float, default=None) - parser.add_argument("--visibility", choices=["private", "agent-only", "workspace-shared", "global"], default=None) - parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL) - parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY) - args = parser.parse_args() - - payload = {} - if args.content: - payload["content"] = args.content - if args.summary: - payload["summary"] = args.summary - if args.tag is not None: - payload["tags"] = args.tag - if args.importance is not None: - payload["importance"] = args.importance - if args.confidence is not None: - payload["confidence"] = args.confidence - if args.visibility: - payload["visibility"] = args.visibility - if not payload: - parser.error("No patch fields provided") - - params = { - "user_id": args.user_id, - "agent_id": args.agent_id, - "workspace_id": args.workspace_id, - "session_id": args.session_id, - } - print(json.dumps(patch_json(f"/v1/memory/{args.memory_id}", payload, params=params, gateway_url=args.gateway_url, api_key=args.api_key), ensure_ascii=False, indent=2)) - - -if __name__ == "__main__": - main() - diff --git a/integrations/hermes/memory-gateway/scripts/memory_search.py b/integrations/hermes/memory-gateway/scripts/memory_search.py deleted file mode 100644 index 188fb86..0000000 --- a/integrations/hermes/memory-gateway/scripts/memory_search.py +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import json - -from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, post_json - - -def main() -> None: - parser = argparse.ArgumentParser(description="Search v1 Memory Gateway with user/agent/workspace/session ACL.") - parser.add_argument("--query", required=True) - parser.add_argument("--user-id", required=True) - parser.add_argument("--agent-id", default="") - parser.add_argument("--workspace-id", default="") - parser.add_argument("--session-id", default="") - parser.add_argument("--namespace", action="append", default=[], help="Allowed namespace to search; repeatable") - parser.add_argument("--memory-type", action="append", default=[], help="Memory type filter; repeatable") - parser.add_argument("--tag", action="append", default=[], help="Tag filter; repeatable") - parser.add_argument("--limit", type=int, default=5) - parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL) - parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY) - args = parser.parse_args() - - payload = { - "user_id": args.user_id, - "agent_id": args.agent_id or None, - "workspace_id": args.workspace_id or None, - "session_id": args.session_id or None, - "query": args.query, - "namespaces": args.namespace, - "memory_types": args.memory_type, - "tags": args.tag, - "limit": args.limit, - } - print(json.dumps(post_json("/v1/memory/search", payload, args.gateway_url, args.api_key), ensure_ascii=False, indent=2)) - - -if __name__ == "__main__": - main() - diff --git a/integrations/hermes/memory-gateway/scripts/memory_upsert.py b/integrations/hermes/memory-gateway/scripts/memory_upsert.py deleted file mode 100644 index e2b5443..0000000 --- a/integrations/hermes/memory-gateway/scripts/memory_upsert.py +++ /dev/null @@ -1,64 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import json -import sys -from pathlib import Path - -from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, post_json - - -def load_content(args: argparse.Namespace) -> str: - if args.file: - return Path(args.file).read_text(encoding="utf-8") - if args.text: - return args.text - return sys.stdin.read().strip() - - -def main() -> None: - parser = argparse.ArgumentParser(description="Create a v1 MemoryRecord through Memory Gateway.") - parser.add_argument("--user-id", required=True) - parser.add_argument("--agent-id", default="") - parser.add_argument("--workspace-id", default="") - parser.add_argument("--session-id", default="") - parser.add_argument("--namespace", default="") - parser.add_argument("--memory-type", default="fact") - parser.add_argument("--text", default="") - parser.add_argument("--file", default="") - parser.add_argument("--summary", default="") - parser.add_argument("--tag", action="append", default=[]) - parser.add_argument("--importance", type=float, default=0.5) - parser.add_argument("--confidence", type=float, default=0.8) - parser.add_argument("--visibility", choices=["private", "agent-only", "workspace-shared", "global"], default="private") - parser.add_argument("--source", default="manual") - parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL) - parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY) - args = parser.parse_args() - - content = load_content(args) - if not content: - parser.error("No memory content provided via --text, --file, or stdin") - - payload = { - "user_id": args.user_id, - "agent_id": args.agent_id or None, - "workspace_id": args.workspace_id or None, - "session_id": args.session_id or None, - "namespace": args.namespace or None, - "memory_type": args.memory_type, - "content": content, - "summary": args.summary or None, - "tags": args.tag, - "importance": args.importance, - "confidence": args.confidence, - "visibility": args.visibility, - "source": args.source, - } - print(json.dumps(post_json("/v1/memory", payload, args.gateway_url, args.api_key), ensure_ascii=False, indent=2)) - - -if __name__ == "__main__": - main() - diff --git a/integrations/hermes/memory-gateway/scripts/retrieve_memory.py b/integrations/hermes/memory-gateway/scripts/retrieve_memory.py deleted file mode 100755 index 14abe8f..0000000 --- a/integrations/hermes/memory-gateway/scripts/retrieve_memory.py +++ /dev/null @@ -1,45 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import json -from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL, post_json - - -def main() -> None: - parser = argparse.ArgumentParser(description="Retrieve memory/resources from Memory Gateway. Defaults to v1 ACL-aware search when --user-id is provided.") - parser.add_argument("--query", required=True, help="Search query") - parser.add_argument("--uri", default="", help="Optional OpenViking URI scope, e.g. viking://resources/project") - parser.add_argument("--namespace", default="", help="Optional namespace if URI is not provided") - parser.add_argument("--user-id", default="", help="Use v1 ACL-aware search when provided") - parser.add_argument("--agent-id", default="") - parser.add_argument("--workspace-id", default="") - parser.add_argument("--session-id", default="") - parser.add_argument("--limit", type=int, default=5) - parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL) - parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY) - args = parser.parse_args() - - if args.user_id: - payload = { - "user_id": args.user_id, - "agent_id": args.agent_id or None, - "workspace_id": args.workspace_id or None, - "session_id": args.session_id or None, - "query": args.query, - "namespaces": [args.namespace] if args.namespace else [], - "limit": args.limit, - } - result = post_json("/v1/memory/search", payload, args.gateway_url, args.api_key) - else: - payload = {"query": args.query, "limit": args.limit} - if args.uri: - payload["uri"] = args.uri - if args.namespace: - payload["namespace"] = args.namespace - result = post_json("/api/search", payload, args.gateway_url, args.api_key) - print(json.dumps(result, ensure_ascii=False, indent=2)) - - -if __name__ == "__main__": - main() diff --git a/integrations/hermes/memory-gateway/scripts/search_obsidian.py b/integrations/hermes/memory-gateway/scripts/search_obsidian.py deleted file mode 100755 index b56e5f0..0000000 --- a/integrations/hermes/memory-gateway/scripts/search_obsidian.py +++ /dev/null @@ -1,55 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import json -import os -import re -from pathlib import Path - -DEFAULT_VAULT = os.environ.get("MEMORY_GATEWAY_OBSIDIAN_VAULT", "/home/tom/memory-gateway/obsidian-vault") - - -def tokenize(query: str) -> list[str]: - return [t.lower() for t in re.split(r"[^\w\u4e00-\u9fff.-]+", query) if len(t.strip()) > 1] - - -def main() -> None: - parser = argparse.ArgumentParser(description="Search local Obsidian Markdown notes by keyword.") - parser.add_argument("--query", required=True) - parser.add_argument("--vault-root", default=DEFAULT_VAULT) - parser.add_argument("--limit", type=int, default=5) - args = parser.parse_args() - - root = Path(args.vault_root) - tokens = tokenize(args.query) - results = [] - for file in root.rglob("*.md"): - try: - text = file.read_text(encoding="utf-8") - except UnicodeDecodeError: - continue - haystack = (file.name + "\n" + text).lower() - matched = [token for token in tokens if token in haystack] - if not matched: - continue - summary = "" - for line in text.splitlines(): - line = line.strip("# -\t") - if len(line) > 30: - summary = line[:240] - break - results.append({ - "score": len(matched) * 10 + min(len(matched), 10), - "file_name": file.name, - "relative_path": str(file.relative_to(root)), - "absolute_path": str(file), - "matched_terms": matched, - "summary": summary, - }) - results.sort(key=lambda item: item["score"], reverse=True) - print(json.dumps({"query": args.query, "vault_root": str(root), "matched_docs": results[:args.limit]}, ensure_ascii=False, indent=2)) - - -if __name__ == "__main__": - main() diff --git a/integrations/hermes/memory-gateway/scripts/upload_knowledge.py b/integrations/hermes/memory-gateway/scripts/upload_knowledge.py deleted file mode 100755 index af15d5d..0000000 --- a/integrations/hermes/memory-gateway/scripts/upload_knowledge.py +++ /dev/null @@ -1,65 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import json -import mimetypes -import urllib.request -from pathlib import Path -from _client import DEFAULT_GATEWAY_API_KEY, DEFAULT_GATEWAY_URL - - -def multipart_upload(url: str, fields: dict[str, str], file_path: Path, api_key: str = "") -> dict: - boundary = "----memorygatewayboundary" - body = bytearray() - for name, value in fields.items(): - if value == "": - continue - body.extend(f"--{boundary}\r\n".encode()) - body.extend(f'Content-Disposition: form-data; name="{name}"\r\n\r\n{value}\r\n'.encode()) - body.extend(f"--{boundary}\r\n".encode()) - mime = mimetypes.guess_type(file_path.name)[0] or "application/octet-stream" - body.extend(f'Content-Disposition: form-data; name="file"; filename="{file_path.name}"\r\n'.encode()) - body.extend(f"Content-Type: {mime}\r\n\r\n".encode()) - body.extend(file_path.read_bytes()) - body.extend(b"\r\n") - body.extend(f"--{boundary}--\r\n".encode()) - req = urllib.request.Request(url, data=bytes(body), method="POST") - req.add_header("Content-Type", f"multipart/form-data; boundary={boundary}") - if api_key: - req.add_header("X-API-Key", api_key) - with urllib.request.urlopen(req, timeout=180) as resp: - return json.loads(resp.read().decode("utf-8")) - - -def main() -> None: - parser = argparse.ArgumentParser(description="Upload a document, convert to Markdown, save to Obsidian, summarize with LLM, and commit to OpenViking.") - parser.add_argument("--file", required=True) - parser.add_argument("--title", default="") - parser.add_argument("--namespace", default="memory-gateway") - parser.add_argument("--knowledge-type", default="knowledge") - parser.add_argument("--tags", default="") - parser.add_argument("--source", default="") - parser.add_argument("--obsidian-dir", default="") - parser.add_argument("--resource-uri", default="") - parser.add_argument("--persist-as", choices=["memory", "resource", "both", "none"], default="resource") - parser.add_argument("--gateway-url", default=DEFAULT_GATEWAY_URL) - parser.add_argument("--api-key", default=DEFAULT_GATEWAY_API_KEY) - args = parser.parse_args() - - fields = { - "title": args.title, - "namespace": args.namespace, - "knowledge_type": args.knowledge_type, - "tags": args.tags, - "source": args.source, - "obsidian_dir": args.obsidian_dir, - "resource_uri": args.resource_uri, - "persist_as": args.persist_as, - } - result = multipart_upload(args.gateway_url.rstrip("/") + "/api/knowledge/upload", fields, Path(args.file), args.api_key) - print(json.dumps(result, ensure_ascii=False, indent=2)) - - -if __name__ == "__main__": - main() diff --git a/memory_gateway/__init__.py b/memory_gateway/__init__.py deleted file mode 100644 index 02500d0..0000000 --- a/memory_gateway/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Memory Gateway 核心模块""" diff --git a/memory_gateway/api_v1.py b/memory_gateway/api_v1.py deleted file mode 100644 index 9dafd8b..0000000 --- a/memory_gateway/api_v1.py +++ /dev/null @@ -1,115 +0,0 @@ -"""Generic Memory Gateway v1 HTTP API.""" -from __future__ import annotations - -from typing import Optional - -from fastapi import APIRouter, Depends, Query - -from .schemas import ( - AccessContext, - CommitSessionRequest, - CreateUserRequest, - EpisodeAppendRequest, - MemoryFeedbackRequest, - MemoryPatchRequest, - MemorySearchRequest, - MemoryUpsertRequest, -) -from .server_auth import verify_api_key_compat -from .services import service - -router = APIRouter(prefix="/v1", tags=["memory-v1"], dependencies=[Depends(verify_api_key_compat)]) - - -@router.post("/users") -async def create_user(request: CreateUserRequest): - return service.create_user(request) - - -@router.get("/users/{user_id}") -async def get_user(user_id: str): - return service.get_user(user_id) - - -@router.post("/memory/search") -async def search_memory(request: MemorySearchRequest): - return await service.search_memory_with_openviking(request) - - -@router.post("/memory") -async def upsert_memory(request: MemoryUpsertRequest): - return service.upsert_memory(request) - - -@router.get("/memory/{memory_id}") -async def get_memory( - memory_id: str, - user_id: str = Query(...), - agent_id: Optional[str] = Query(default=None), - workspace_id: Optional[str] = Query(default=None), - session_id: Optional[str] = Query(default=None), -): - return service.get_memory(memory_id, AccessContext(user_id=user_id, agent_id=agent_id, workspace_id=workspace_id, session_id=session_id)) - - -@router.patch("/memory/{memory_id}") -async def patch_memory( - memory_id: str, - patch: MemoryPatchRequest, - user_id: str = Query(...), - agent_id: Optional[str] = Query(default=None), - workspace_id: Optional[str] = Query(default=None), - session_id: Optional[str] = Query(default=None), -): - return service.patch_memory(memory_id, AccessContext(user_id=user_id, agent_id=agent_id, workspace_id=workspace_id, session_id=session_id), patch) - - -@router.delete("/memory/{memory_id}") -async def delete_memory( - memory_id: str, - user_id: str = Query(...), - agent_id: Optional[str] = Query(default=None), - workspace_id: Optional[str] = Query(default=None), - session_id: Optional[str] = Query(default=None), -): - return service.delete_memory(memory_id, AccessContext(user_id=user_id, agent_id=agent_id, workspace_id=workspace_id, session_id=session_id)) - - -@router.post("/episodes") -async def append_episode(request: EpisodeAppendRequest): - return service.append_episode(request) - - -@router.post("/sessions/{session_id}/commit") -async def commit_session(session_id: str, request: CommitSessionRequest): - return service.commit_session(session_id, request) - - -@router.get("/users/{user_id}/profile") -async def get_profile(user_id: str): - return service.get_profile(user_id) - - -@router.post("/memory/{memory_id}/feedback") -async def memory_feedback(memory_id: str, request: MemoryFeedbackRequest): - return service.add_feedback(memory_id, request) - - -@router.get("/namespaces") -async def list_namespaces( - user_id: str = Query(...), - agent_id: Optional[str] = Query(default=None), - workspace_id: Optional[str] = Query(default=None), - session_id: Optional[str] = Query(default=None), -): - return service.list_namespaces(AccessContext(user_id=user_id, agent_id=agent_id, workspace_id=workspace_id, session_id=session_id)) - - -@router.get("/audit") -async def list_audit(limit: int = Query(default=100, ge=1, le=1000)): - return service.list_audit(limit) - - -@router.get("/everos/health") -async def everos_health(): - return service.everos_health() diff --git a/memory_gateway/api_v2.py b/memory_gateway/api_v2.py deleted file mode 100644 index dff080c..0000000 --- a/memory_gateway/api_v2.py +++ /dev/null @@ -1,90 +0,0 @@ -"""Memory Gateway v2 workflow API.""" -from __future__ import annotations - -from typing import Optional - -from fastapi import APIRouter, Depends, Query - -from .schemas_v2 import ( - BackendRefStatus, - BackendType, - CommitJobView, - CommitRequest, - CommitResponse, - FeedbackRequest, - FeedbackResponse, - IngestRequest, - IngestResponse, - MemoryRefType, - MemoryRefView, - OutboxProcessResponse, - RetrieveRequest, - RetrieveResponse, -) -from .server_auth import verify_api_key_compat -from .services_v2 import v2_service - -router = APIRouter(prefix="/v2", tags=["memory-v2"], dependencies=[Depends(verify_api_key_compat)]) - - -@router.post("/conversations/ingest", response_model=IngestResponse) -async def ingest_conversation(request: IngestRequest): - return await v2_service.ingest_conversation_turn(request) - - -@router.post("/conversations/{session_id}/commit", response_model=CommitResponse) -async def commit_conversation(session_id: str, request: CommitRequest): - return await v2_service.commit_session(session_id, request) - - -@router.get("/jobs/{job_id}", response_model=CommitJobView) -async def get_commit_job(job_id: str): - return v2_service.get_commit_job_view(job_id) - - -@router.post("/context/retrieve", response_model=RetrieveResponse) -async def retrieve_context(request: RetrieveRequest): - return await v2_service.retrieve_context(request) - - -@router.get("/memory/refs", response_model=list[MemoryRefView]) -async def list_memory_refs( - workspace_id: Optional[str] = Query(default=None), - user_id: Optional[str] = Query(default=None), - agent_id: Optional[str] = Query(default=None), - session_id: Optional[str] = Query(default=None), - namespace: Optional[str] = Query(default=None), - backend_type: Optional[BackendType] = Query(default=None), - ref_type: Optional[MemoryRefType] = Query(default=None), - status: Optional[BackendRefStatus] = Query(default=None), - limit: int = Query(default=100, ge=1, le=1000), -): - return v2_service.list_memory_refs( - workspace_id=workspace_id, - user_id=user_id, - agent_id=agent_id, - session_id=session_id, - namespace=namespace, - backend_type=backend_type, - ref_type=ref_type, - status=status, - limit=limit, - ) - - -@router.post("/memory/feedback", response_model=FeedbackResponse) -async def memory_feedback(request: FeedbackRequest): - return await v2_service.record_memory_feedback(request) - - -@router.post("/admin/outbox/process", response_model=OutboxProcessResponse, tags=["memory-v2-admin"]) -async def process_outbox( - limit: int = Query(default=100, ge=1, le=1000), - worker_id: Optional[str] = Query(default=None), - lease_seconds: int = Query(default=300, ge=1, le=3600), -): - return await v2_service.process_pending_outbox_events_summary( - limit=limit, - worker_id=worker_id, - lease_seconds=lease_seconds, - ) diff --git a/memory_gateway/backend_adapter_mapping.py b/memory_gateway/backend_adapter_mapping.py deleted file mode 100644 index 123ef50..0000000 --- a/memory_gateway/backend_adapter_mapping.py +++ /dev/null @@ -1,129 +0,0 @@ -"""Contract-first mapping spec for future v2 backend adapters. - -This module intentionally does not call OpenViking, EverOS, or Obsidian. -It documents the stable Gateway control-plane fields that may be persisted in -outbox payload refs, SQLite metadata_json, audit summaries, and related control -records. It is not a validator for transient runtime adapter request objects: -the Gateway may pass conversation content to a backend during the current -request lifecycle, but must not persist that content in SQLite/outbox/audit -control-plane records. -""" -from __future__ import annotations - -from typing import Final, NamedTuple - -from .backend_contracts import BackendCommitResult, BackendOperation, BackendRetrieveResult, BackendWriteResult -from .schemas_v2 import BackendType - - -CONTROL_PLANE_PERSISTED_PAYLOAD_FIELDS: Final[frozenset[str]] = frozenset( - { - "event_id", - "gateway_id", - "workspace_id", - "user_id", - "agent_id", - "session_id", - "turn_id", - "namespace", - "source_type", - "source_event_id", - "backend_type", - "operation", - "payload_ref", - "metadata", - "trace", - "role", - } -) -CONTROL_PLANE_PAYLOAD_FIELDS: Final[frozenset[str]] = CONTROL_PLANE_PERSISTED_PAYLOAD_FIELDS - -DISALLOWED_PAYLOAD_FIELDS: Final[frozenset[str]] = frozenset( - { - "content", - "raw_request", - "messages", - "conversation", - "transcript", - } -) - - -class AdapterMappingSpec(NamedTuple): - backend_type: BackendType - operation: BackendOperation - adapter_method: str - backend_capability: str - result_model: type[BackendWriteResult] | type[BackendCommitResult] | type[BackendRetrieveResult] - allowed_payload_fields: frozenset[str] = CONTROL_PLANE_PERSISTED_PAYLOAD_FIELDS - - -ADAPTER_MAPPING_SPECS: Final[tuple[AdapterMappingSpec, ...]] = ( - AdapterMappingSpec( - backend_type=BackendType.OPENVIKING, - operation=BackendOperation.INGEST_TURN, - adapter_method="ingest_conversation_turn", - backend_capability="session archive append / resource context organization", - result_model=BackendWriteResult, - ), - AdapterMappingSpec( - backend_type=BackendType.OPENVIKING, - operation=BackendOperation.COMMIT_SESSION, - adapter_method="commit_session_v2", - backend_capability="session commit and session archive ref creation", - result_model=BackendCommitResult, - ), - AdapterMappingSpec( - backend_type=BackendType.OPENVIKING, - operation=BackendOperation.RETRIEVE_CONTEXT, - adapter_method="retrieve_context_v2", - backend_capability="runtime session/resource context retrieval", - result_model=BackendRetrieveResult, - ), - AdapterMappingSpec( - backend_type=BackendType.EVEROS, - operation=BackendOperation.INGEST_TURN, - adapter_method="ingest_message", - backend_capability="message-level memory ingestion", - result_model=BackendWriteResult, - ), - AdapterMappingSpec( - backend_type=BackendType.EVEROS, - operation=BackendOperation.COMMIT_SESSION, - adapter_method="extract_profile_long_term_v2", - backend_capability="episodic/profile/long-term extraction", - result_model=BackendCommitResult, - ), - AdapterMappingSpec( - backend_type=BackendType.EVEROS, - operation=BackendOperation.RETRIEVE_CONTEXT, - adapter_method="retrieve_context_v2", - backend_capability="episodic/profile/long-term memory retrieval", - result_model=BackendRetrieveResult, - ), - AdapterMappingSpec( - backend_type=BackendType.OBSIDIAN, - operation=BackendOperation.CREATE_REVIEW_DRAFT, - adapter_method="create_review_draft_v2", - backend_capability="human review draft creation for high-risk/high-conflict candidates", - result_model=BackendWriteResult, - ), -) - - -def get_adapter_mapping_spec(backend_type: BackendType, operation: BackendOperation) -> AdapterMappingSpec: - for spec in ADAPTER_MAPPING_SPECS: - if spec.backend_type == backend_type and spec.operation == operation: - return spec - raise KeyError(f"No v2 adapter mapping for {backend_type.value}:{operation.value}") - - -def validate_control_plane_payload(payload: dict[str, object]) -> None: - """Validate only persisted control-plane payloads, not runtime adapter requests.""" - blocked = sorted(DISALLOWED_PAYLOAD_FIELDS.intersection(payload)) - if blocked: - raise ValueError(f"Control-plane persisted payload includes disallowed content fields: {', '.join(blocked)}") - - -def validate_control_plane_persisted_payload(payload: dict[str, object]) -> None: - validate_control_plane_payload(payload) diff --git a/memory_gateway/backend_contracts.py b/memory_gateway/backend_contracts.py deleted file mode 100644 index 4d8554f..0000000 --- a/memory_gateway/backend_contracts.py +++ /dev/null @@ -1,134 +0,0 @@ -"""Backend adapter contracts for Memory Gateway v2.""" -from __future__ import annotations - -from datetime import datetime -from enum import Enum -from typing import Any, Optional -from uuid import uuid4 - -from pydantic import BaseModel, Field - -from .schemas import utc_now -from .schemas_v2 import BackendType, MemoryRefType, OperationStatus - - -class BackendOperation(str, Enum): - INGEST_TURN = "ingest_turn" - COMMIT_SESSION = "commit_session" - RETRIEVE_CONTEXT = "retrieve_context" - CREATE_REVIEW_DRAFT = "create_review_draft" - - -class BackendResultStatus(str, Enum): - SUCCESS = "success" - FAILED = "failed" - SKIPPED = "skipped" - PENDING = "pending" - - -class OutboxEventStatus(str, Enum): - PENDING = "pending" - PROCESSING = "processing" - SUCCESS = "success" - SKIPPED = "skipped" - FAILED = "failed" - DEAD_LETTER = "dead_letter" - - -class BackendWriteResult(BaseModel): - backend_type: BackendType - operation: BackendOperation - status: BackendResultStatus - native_id: Optional[str] = None - native_uri: Optional[str] = None - retryable: bool = False - error_code: Optional[str] = None - error_message: Optional[str] = None - latency_ms: Optional[float] = None - metadata: dict[str, Any] = Field(default_factory=dict) - - -class BackendProducedRef(BaseModel): - ref_type: MemoryRefType - native_id: Optional[str] = None - native_uri: Optional[str] = None - metadata: dict[str, Any] = Field(default_factory=dict) - - -class BackendCommitResult(BaseModel): - backend_type: BackendType - operation: BackendOperation = BackendOperation.COMMIT_SESSION - status: BackendResultStatus - native_id: Optional[str] = None - native_uri: Optional[str] = None - retryable: bool = False - error_code: Optional[str] = None - error_message: Optional[str] = None - latency_ms: Optional[float] = None - created_refs: list[str] = Field(default_factory=list) - refs: list[BackendProducedRef] = Field(default_factory=list) - metadata: dict[str, Any] = Field(default_factory=dict) - - -class BackendRetrieveItem(BaseModel): - text: Optional[str] = None - source_backend: BackendType - ref_id: Optional[str] = None - score: float = 0.0 - memory_type: Optional[str] = None - metadata: dict[str, Any] = Field(default_factory=dict) - - -class BackendRetrieveResult(BaseModel): - backend_type: BackendType - operation: BackendOperation = BackendOperation.RETRIEVE_CONTEXT - status: BackendResultStatus - native_id: Optional[str] = None - native_uri: Optional[str] = None - retryable: bool = False - error_code: Optional[str] = None - error_message: Optional[str] = None - latency_ms: Optional[float] = None - items: list[BackendRetrieveItem] = Field(default_factory=list) - metadata: dict[str, Any] = Field(default_factory=dict) - - -class OutboxEvent(BaseModel): - id: str = Field(default_factory=lambda: f"outbox_{uuid4().hex[:16]}") - event_type: str - gateway_id: str - workspace_id: str - user_id: str - agent_id: Optional[str] = None - session_id: Optional[str] = None - backend_type: BackendType - operation: BackendOperation - payload_ref: Optional[str] = None - status: OutboxEventStatus = OutboxEventStatus.PENDING - attempt_count: int = 0 - max_attempts: int = 3 - next_retry_at: Optional[datetime] = None - last_error: Optional[str] = None - locked_by: Optional[str] = None - locked_at: Optional[datetime] = None - lease_expires_at: Optional[datetime] = None - metadata: dict[str, Any] = Field(default_factory=dict) - created_at: datetime = Field(default_factory=utc_now) - updated_at: datetime = Field(default_factory=utc_now) - - -class CommitJob(BaseModel): - job_id: str = Field(default_factory=lambda: f"job_{uuid4().hex[:16]}") - workspace_id: str - user_id: str - agent_id: Optional[str] = None - session_id: str - namespace: Optional[str] = None - status: OperationStatus = OperationStatus.ACCEPTED - requested_by: Optional[str] = None - created_refs_count: int = 0 - error_message: Optional[str] = None - created_at: datetime = Field(default_factory=utc_now) - updated_at: datetime = Field(default_factory=utc_now) - started_at: Optional[datetime] = None - finished_at: Optional[datetime] = None diff --git a/memory_gateway/backend_normalization.py b/memory_gateway/backend_normalization.py deleted file mode 100644 index 95f8e6d..0000000 --- a/memory_gateway/backend_normalization.py +++ /dev/null @@ -1,261 +0,0 @@ -"""Offline response normalization helpers for future v2 backend adapters.""" -from __future__ import annotations - -from typing import Any - -from .backend_contracts import ( - BackendCommitResult, - BackendOperation, - BackendProducedRef, - BackendResultStatus, - BackendRetrieveItem, - BackendRetrieveResult, - BackendWriteResult, -) -from .backend_ref_mapping import map_backend_ref_type -from .schemas_v2 import BackendType - - -SAFE_METADATA_KEYS = { - "backend_request_id", - "request_id", - "trace_id", - "latency_ms", - "schema_version", - "source_channel", - "reason", - "original_ref_type", - "confidence", - "score", - "version", - "created_at", -} -BLOCKED_METADATA_KEYS = {"content", "raw_request", "messages", "conversation", "transcript"} - - -def safe_backend_metadata(metadata: dict[str, Any] | None) -> dict[str, Any]: - if not metadata: - return {} - safe: dict[str, Any] = {} - for key, value in metadata.items(): - if key in BLOCKED_METADATA_KEYS or key not in SAFE_METADATA_KEYS: - continue - if isinstance(value, (str, int, float, bool)) or value is None: - safe[key] = value - return safe - - -def normalize_openviking_commit_response(raw: dict[str, Any]) -> BackendCommitResult: - status = _result_status(raw) - refs = [_produced_ref(BackendType.OPENVIKING, item) for item in _extract_ref_items(raw)] - return BackendCommitResult( - backend_type=BackendType.OPENVIKING, - operation=BackendOperation.COMMIT_SESSION, - status=status, - native_id=raw.get("native_id") or raw.get("session_id"), - native_uri=raw.get("native_uri") or raw.get("uri"), - retryable=_retryable_from_raw(BackendType.OPENVIKING, raw), - error_code=raw.get("error_code"), - error_message=raw.get("error") or raw.get("error_message"), - latency_ms=raw.get("latency_ms"), - refs=refs, - metadata=safe_backend_metadata(raw.get("metadata") or raw), - ) - - -def normalize_everos_commit_response(raw: dict[str, Any]) -> BackendCommitResult: - status = _result_status(raw) - refs = [_produced_ref(BackendType.EVEROS, item) for item in _extract_ref_items(raw)] - return BackendCommitResult( - backend_type=BackendType.EVEROS, - operation=BackendOperation.COMMIT_SESSION, - status=status, - native_id=raw.get("native_id") or raw.get("session_id"), - native_uri=raw.get("native_uri") or raw.get("uri"), - retryable=_retryable_from_raw(BackendType.EVEROS, raw), - error_code=raw.get("error_code"), - error_message=raw.get("error") or raw.get("error_message"), - latency_ms=raw.get("latency_ms"), - refs=refs, - metadata=safe_backend_metadata(raw.get("metadata") or raw), - ) - - -def normalize_openviking_ingest_response(raw: dict[str, Any]) -> BackendWriteResult: - return _write_result(BackendType.OPENVIKING, raw) - - -def normalize_everos_ingest_response(raw: dict[str, Any]) -> BackendWriteResult: - return _write_result(BackendType.EVEROS, raw) - - -def normalize_openviking_retrieve_response(raw: dict[str, Any]) -> BackendRetrieveResult: - return _retrieve_result(BackendType.OPENVIKING, raw) - - -def normalize_everos_retrieve_response(raw: dict[str, Any]) -> BackendRetrieveResult: - return _retrieve_result(BackendType.EVEROS, raw) - - -def map_backend_error_to_retryable( - backend_type: BackendType, - status_code: int | None = None, - error_code: str | None = None, - error_message: str | None = None, -) -> bool: - """Map backend errors into retryable/non-retryable categories. - - Unknown errors default to retryable because adapter contracts are still - unstable and transient backend/API rollout failures are more likely during - integration. - """ - if status_code in {429, 500, 502, 503, 504}: - return True - if status_code in {400, 401, 403, 404, 422}: - return False - text = f"{error_code or ''} {error_message or ''}".lower() - if "timeout" in text or "network_error" in text or "connection" in text: - return True - if "validation" in text or "unauthorized" in text or "forbidden" in text or "not_found" in text: - return False - return True - - -def _write_result(backend_type: BackendType, raw: dict[str, Any]) -> BackendWriteResult: - data = raw.get("result") or raw.get("data") or {} - if not isinstance(data, dict): - data = {} - native_id = ( - raw.get("native_id") - or raw.get("id") - or raw.get("memory_id") - or raw.get("request_id") - or raw.get("session_id") - or data.get("native_id") - or data.get("id") - or data.get("memory_id") - or data.get("request_id") - or data.get("session_id") - ) - native_uri = ( - raw.get("native_uri") - or raw.get("uri") - or raw.get("url") - or data.get("native_uri") - or data.get("uri") - or data.get("url") - ) - if not native_uri and backend_type == BackendType.OPENVIKING and native_id: - native_uri = f"viking://sessions/{native_id}" - return BackendWriteResult( - backend_type=backend_type, - operation=BackendOperation.INGEST_TURN, - status=_result_status(raw), - native_id=native_id, - native_uri=native_uri, - retryable=_retryable_from_raw(backend_type, raw), - error_code=raw.get("error_code") or raw.get("code"), - error_message=raw.get("error") or raw.get("error_message") or raw.get("message"), - latency_ms=raw.get("latency_ms"), - metadata=safe_backend_metadata(raw.get("metadata") or raw), - ) - - -def _retrieve_result(backend_type: BackendType, raw: dict[str, Any]) -> BackendRetrieveResult: - if not isinstance(raw, dict) or not raw: - return BackendRetrieveResult( - backend_type=backend_type, - operation=BackendOperation.RETRIEVE_CONTEXT, - status=BackendResultStatus.SKIPPED, - metadata={"reason": "malformed_or_empty_response"}, - ) - return BackendRetrieveResult( - backend_type=backend_type, - operation=BackendOperation.RETRIEVE_CONTEXT, - status=_result_status(raw), - native_id=raw.get("native_id") or raw.get("session_id"), - native_uri=raw.get("native_uri") or raw.get("uri"), - retryable=_retryable_from_raw(backend_type, raw), - error_code=raw.get("error_code") or raw.get("code"), - error_message=raw.get("error") or raw.get("error_message") or raw.get("message"), - latency_ms=raw.get("latency_ms"), - items=[_retrieve_item(backend_type, item) for item in _extract_retrieve_items(raw)], - metadata=safe_backend_metadata(raw.get("metadata") or raw), - ) - - -def _retrieve_item(backend_type: BackendType, item: dict[str, Any]) -> BackendRetrieveItem: - metadata = safe_backend_metadata(item.get("metadata") if isinstance(item.get("metadata"), dict) else item) - return BackendRetrieveItem( - text=item.get("text") or item.get("summary") or item.get("abstract"), - source_backend=backend_type, - ref_id=item.get("ref_id") or item.get("id") or item.get("memory_id") or item.get("profile_id") or item.get("session_id") or item.get("uri"), - score=float(item.get("score") or 0.0), - memory_type=item.get("memory_type") or item.get("ref_type") or item.get("type") or item.get("kind"), - metadata=metadata, - ) - - -def _produced_ref(backend_type: BackendType, item: dict[str, Any]) -> BackendProducedRef: - ref_type, mapping_metadata = map_backend_ref_type(backend_type, item.get("ref_type") or item.get("type") or item.get("kind")) - metadata = { - **safe_backend_metadata(item.get("metadata") if isinstance(item.get("metadata"), dict) else item), - **mapping_metadata, - } - return BackendProducedRef( - ref_type=ref_type, - native_id=item.get("native_id") or item.get("id") or item.get("memory_id") or item.get("profile_id") or item.get("session_id"), - native_uri=item.get("native_uri") or item.get("uri") or item.get("url"), - metadata=metadata, - ) - - -def _extract_ref_items(raw: dict[str, Any]) -> list[dict[str, Any]]: - data = raw.get("result") or raw.get("data") or raw - candidates = ( - data.get("refs") - or data.get("produced_refs") - or data.get("created_refs") - or data.get("memories") - or data.get("items") - or [] - ) - return [item for item in candidates if isinstance(item, dict)] - - -def _extract_retrieve_items(raw: dict[str, Any]) -> list[dict[str, Any]]: - data = raw.get("result") or raw.get("data") or raw - if not isinstance(data, dict): - return [] - candidates = ( - data.get("items") - or data.get("results") - or data.get("memories") - or data.get("resources") - or data.get("contexts") - or [] - ) - return [item for item in candidates if isinstance(item, dict)] - - -def _result_status(raw: dict[str, Any]) -> BackendResultStatus: - status = str(raw.get("status") or "success").lower() - if status in {"ok", "created", "accepted"}: - return BackendResultStatus.SUCCESS - try: - return BackendResultStatus(status) - except ValueError: - return BackendResultStatus.SUCCESS if not raw.get("error") and not raw.get("error_message") else BackendResultStatus.FAILED - - -def _retryable_from_raw(backend_type: BackendType, raw: dict[str, Any]) -> bool: - if "retryable" in raw: - return bool(raw["retryable"]) - if raw.get("error") or raw.get("error_message") or raw.get("error_code") or raw.get("status_code"): - return map_backend_error_to_retryable( - backend_type, - status_code=raw.get("status_code"), - error_code=raw.get("error_code"), - error_message=raw.get("error") or raw.get("error_message"), - ) - return False diff --git a/memory_gateway/backend_ref_mapping.py b/memory_gateway/backend_ref_mapping.py deleted file mode 100644 index 178cfb2..0000000 --- a/memory_gateway/backend_ref_mapping.py +++ /dev/null @@ -1,66 +0,0 @@ -"""Backend-specific ref type mapping for Memory Gateway v2.""" -from __future__ import annotations - -from .schemas_v2 import BackendType, MemoryRefType - - -OPENVIKING_REF_TYPE_MAP = { - "session_archive": MemoryRefType.SESSION_ARCHIVE, - "context_resource": MemoryRefType.CONTEXT_RESOURCE, - "resource": MemoryRefType.CONTEXT_RESOURCE, - "session_summary": MemoryRefType.SESSION_ARCHIVE, -} - -EVEROS_REF_TYPE_MAP = { - "message_memory": MemoryRefType.MESSAGE_MEMORY, - "episodic_memory": MemoryRefType.EPISODIC_MEMORY, - "episode": MemoryRefType.EPISODIC_MEMORY, - "profile": MemoryRefType.PROFILE, - "long_term_memory": MemoryRefType.LONG_TERM_MEMORY, - "memory": MemoryRefType.LONG_TERM_MEMORY, - "preference": MemoryRefType.PROFILE, -} - -OBSIDIAN_REF_TYPE_MAP = { - "review_draft": MemoryRefType.DRAFT_REVIEW, - "draft_review": MemoryRefType.DRAFT_REVIEW, -} - - -def map_backend_ref_type( - backend_type: BackendType, - backend_ref_type: str | None, -) -> tuple[MemoryRefType, dict[str, str]]: - """Map backend-native ref type to a Gateway MemoryRefType. - - Unknown values fall back to a backend-appropriate default and preserve the - original value in returned metadata for later inspection. - """ - raw_type = (backend_ref_type or "").strip() - normalized = raw_type.lower() - - if backend_type == BackendType.OPENVIKING: - mapped = OPENVIKING_REF_TYPE_MAP.get(normalized, MemoryRefType.SESSION_ARCHIVE) - elif backend_type == BackendType.EVEROS: - mapped = EVEROS_REF_TYPE_MAP.get(normalized, MemoryRefType.LONG_TERM_MEMORY) - elif backend_type == BackendType.OBSIDIAN: - mapped = OBSIDIAN_REF_TYPE_MAP.get(normalized, MemoryRefType.DRAFT_REVIEW) - else: - mapped = MemoryRefType.LONG_TERM_MEMORY - - metadata: dict[str, str] = {} - if raw_type and raw_type not in {mapped.value, normalized}: - metadata["original_ref_type"] = raw_type - elif raw_type and normalized not in _known_backend_ref_types(backend_type): - metadata["original_ref_type"] = raw_type - return mapped, metadata - - -def _known_backend_ref_types(backend_type: BackendType) -> set[str]: - if backend_type == BackendType.OPENVIKING: - return set(OPENVIKING_REF_TYPE_MAP) - if backend_type == BackendType.EVEROS: - return set(EVEROS_REF_TYPE_MAP) - if backend_type == BackendType.OBSIDIAN: - return set(OBSIDIAN_REF_TYPE_MAP) - return set() diff --git a/memory_gateway/config.py b/memory_gateway/config.py deleted file mode 100644 index ad233e8..0000000 --- a/memory_gateway/config.py +++ /dev/null @@ -1,102 +0,0 @@ -"""配置加载模块""" -import os -from pathlib import Path -from typing import Optional - -import yaml -from pydantic import ValidationError - -from .types import Config, ServerConfig, OpenVikingConfig, EverOSConfig, MemoryConfig, LoggingConfig, LLMConfig, ObsidianConfig, StorageConfig - - -def load_config(config_path: Optional[str] = None) -> Config: - """加载配置文件""" - if config_path is None: - config_path = os.environ.get("MEMORY_GATEWAY_CONFIG", "config.yaml") - - config_file = Path(config_path) - - if not config_file.exists(): - # 返回默认配置 - return _apply_env_overrides(Config()) - - try: - with open(config_file, "r", encoding="utf-8") as f: - data = yaml.safe_load(f) - - if data is None: - return _apply_env_overrides(Config()) - - config = Config( - server=ServerConfig(**data.get("server", {})), - openviking=OpenVikingConfig(**data.get("openviking", {})), - everos=EverOSConfig(**data.get("everos", {})), - memory=MemoryConfig(**data.get("memory", {})), - logging=LoggingConfig(**data.get("logging", {})), - llm=LLMConfig(**data.get("llm", {})), - obsidian=ObsidianConfig(**data.get("obsidian", {})), - storage=StorageConfig(**data.get("storage", {})), - ) - return _apply_env_overrides(config) - except (ValidationError, yaml.YAMLError) as e: - print(f"配置文件解析错误: {e}") - return _apply_env_overrides(Config()) - - -def get_config() -> Config: - """获取全局配置（单例）""" - global _config - if _config is None: - _config = load_config() - return _config - - -def set_config(config: Config) -> None: - """设置全局配置""" - global _config - _config = config - - -_config: Optional[Config] = None - - -def _apply_env_overrides(config: Config) -> Config: - openviking_updates = _backend_env_updates("OPENVIKING") - everos_updates = _backend_env_updates("EVEROS") - if openviking_updates: - config.openviking = config.openviking.model_copy(update=openviking_updates) - if everos_updates: - config.everos = config.everos.model_copy(update=everos_updates) - return config - - -def _backend_env_updates(prefix: str) -> dict: - updates = {} - env_map = { - "ENABLED": "enabled", - "MODE": "mode", - "BASE_URL": "url", - "URL": "url", - "API_KEY": "api_key", - "TOKEN": "api_key", - "TIMEOUT": "timeout", - "TIMEOUT_SECONDS": "timeout", - "VERIFY_SSL": "verify_ssl", - "INGEST_PATH": "ingest_path", - "SEARCH_PATH": "search_path", - "FLUSH_PATH": "flush_path", - "RETRIEVE_METHOD": "retrieve_method", - } - for env_name, field_name in env_map.items(): - value = os.environ.get(f"{prefix}_{env_name}") - if value is None: - continue - if field_name == "enabled": - updates[field_name] = value.lower() in {"1", "true", "yes", "on"} - elif field_name == "timeout": - updates[field_name] = int(value) - elif field_name == "verify_ssl": - updates[field_name] = value.lower() not in {"0", "false", "no", "off"} - else: - updates[field_name] = value - return updates diff --git a/memory_gateway/document_ingest.py b/memory_gateway/document_ingest.py deleted file mode 100644 index 1d2feda..0000000 --- a/memory_gateway/document_ingest.py +++ /dev/null @@ -1,87 +0,0 @@ -"""Document ingestion helpers for Memory Gateway.""" -from __future__ import annotations - -import re -from datetime import datetime, timezone -from pathlib import Path - - -def slugify(value: str, fallback: str = "document") -> str: - slug = re.sub(r"[^a-zA-Z0-9\u4e00-\u9fff_-]+", "-", value.lower()).strip("-") - slug = re.sub(r"-+", "-", slug)[:100].strip("-") - return slug or fallback - - -def convert_file_to_markdown(file_path: str | Path) -> str: - """Convert a local document to Markdown using Microsoft MarkItDown.""" - try: - from markitdown import MarkItDown - except ModuleNotFoundError as exc: - raise RuntimeError("markitdown is not installed. Install with: pip install 'markitdown[all]'") from exc - - file_path = Path(file_path) - converter = MarkItDown(enable_plugins=False) - if hasattr(converter, "convert_local"): - result = converter.convert_local(str(file_path)) - else: - result = converter.convert(str(file_path)) - markdown = getattr(result, "text_content", "") or "" - if not markdown.strip(): - raise RuntimeError("Document conversion produced empty Markdown") - return markdown - - -def build_markdown_note( - *, - title: str, - markdown: str, - source_filename: str, - tags: list[str], - knowledge_type: str, - summary: str | None = None, -) -> str: - tag_text = ", ".join(tags) - frontmatter = [ - "---", - f"title: {title}", - f"knowledge_type: {knowledge_type}", - f"source_filename: {source_filename}", - f"created_at: {datetime.now(timezone.utc).isoformat()}", - f"tags: [{tag_text}]" if tag_text else "tags: []", - ] - if summary: - escaped = summary.replace('"', '\\"') - frontmatter.append(f'summary: "{escaped}"') - frontmatter.extend(["---", "", f"# {title}", "", markdown.strip(), ""]) - return "\n".join(frontmatter) - - -def save_markdown_to_obsidian( - *, - vault_path: str | Path, - relative_dir: str, - title: str, - markdown: str, - source_filename: str, - tags: list[str], - knowledge_type: str, - summary: str | None = None, -) -> Path: - vault = Path(vault_path) - target_dir = vault / relative_dir.strip("/") - target_dir.mkdir(parents=True, exist_ok=True) - digest = slugify(source_filename.rsplit(".", 1)[0] or title) - note_name = f"{slugify(title, digest)}.md" - target = target_dir / note_name - target.write_text( - build_markdown_note( - title=title, - markdown=markdown, - source_filename=source_filename, - tags=tags, - knowledge_type=knowledge_type, - summary=summary, - ), - encoding="utf-8", - ) - return target diff --git a/memory_gateway/everos_client.py b/memory_gateway/everos_client.py deleted file mode 100644 index 2f5da74..0000000 --- a/memory_gateway/everos_client.py +++ /dev/null @@ -1,496 +0,0 @@ -"""Client for the external EverOS memory service.""" -from __future__ import annotations - -from datetime import datetime, timezone -from json import JSONDecodeError -from typing import Any - -import httpx - -from .backend_contracts import BackendCommitResult, BackendOperation, BackendResultStatus, BackendRetrieveResult, BackendWriteResult -from .backend_normalization import ( - map_backend_error_to_retryable, - normalize_everos_commit_response, - normalize_everos_ingest_response, - normalize_everos_retrieve_response, -) -from .config import get_config -from .schemas import AccessContext, EpisodeRecord, MemoryRecord -from .schemas_v2 import BackendType - - -class EverOSError(RuntimeError): - """Raised when the external EverOS service cannot process a request.""" - - -class EverOSClient: - """Small HTTP client with a tolerant response normalizer. - - The deployed EverOS API may evolve independently from Memory Gateway. - Gateway sends a stable payload and accepts several common response shapes: - `result`, `data`, or the raw top-level object with `candidates/promoted`. - """ - - def __init__( - self, - base_url: str | None = None, - api_key: str | None = None, - timeout: int | None = None, - enabled: bool | None = None, - mode: str | None = None, - verify_ssl: bool | None = None, - health_path: str | None = None, - ingest_path: str | None = None, - search_path: str | None = None, - flush_path: str | None = None, - retrieve_method: str | None = None, - transport: httpx.BaseTransport | None = None, - ) -> None: - config = get_config().everos - self.base_url = (base_url if base_url is not None else config.url).rstrip("/") - self.api_key = api_key if api_key is not None else config.api_key - self.timeout = timeout or config.timeout - self.enabled = config.enabled if enabled is None else enabled - self.mode = mode or config.mode - self.verify_ssl = config.verify_ssl if verify_ssl is None else verify_ssl - self.health_path = health_path or config.health_path - self.ingest_path = ingest_path or config.ingest_path - self.search_path = search_path or config.search_path - self.flush_path = flush_path or config.flush_path - self.retrieve_method = retrieve_method or config.retrieve_method - self.transport = transport - - def _headers(self) -> dict[str, str]: - headers = {"Content-Type": "application/json"} - if self.api_key: - headers["X-API-Key"] = self.api_key - headers["Authorization"] = f"Bearer {self.api_key}" - return headers - - def health(self) -> dict[str, Any]: - url = self.base_url + self.health_path - try: - health_timeout = httpx.Timeout(min(self.timeout, 2.0), connect=min(self.timeout, 0.5)) - with httpx.Client(timeout=health_timeout, headers=self._headers()) as client: - response = client.get(url) - response.raise_for_status() - return {"status": "ok", "url": self.base_url, "response": response.json()} - except Exception as exc: # noqa: BLE001 - return {"status": "error", "url": self.base_url, "error": str(exc)} - - def ingest_message(self, payload: dict[str, Any]) -> BackendWriteResult: - """Write one Gateway turn to EverOS.""" - runtime_payload = self._build_ingest_payload(payload) - if self._use_real_api: - return self._ingest_message_real(runtime_payload) - raw = { - "status": "skipped", - "memory_id": (runtime_payload.get("messages") or [{}])[0].get("message_id"), - "metadata": { - "reason": "everos_v2_ingest_adapter_not_configured", - "schema_version": "everos.fixture.ingest.v2", - }, - } - return self._normalize_ingest_response(raw) - - @property - def _use_real_api(self) -> bool: - # Real ingest is strictly gated by mode=real. The legacy `enabled` - # field is retained for config compatibility, but must not trigger - # network traffic by itself. - return self.mode == "real" - - def _ingest_message_real(self, runtime_payload: dict[str, Any]) -> BackendWriteResult: - if not self.base_url: - return self._failed_ingest_result( - error_code="config_error", - error_message="EverOS real ingest is enabled but base_url is missing", - retryable=False, - ) - try: - with httpx.Client( - base_url=self.base_url, - headers=self._headers(), - timeout=self.timeout, - verify=self.verify_ssl, - transport=self.transport, - ) as client: - response = client.post(self.ingest_path, json=runtime_payload) - if response.status_code >= 400: - return self._failed_ingest_result( - error_code=f"http_{response.status_code}", - error_message=f"EverOS ingest failed with HTTP {response.status_code}", - retryable=self._map_error(response), - ) - try: - raw = response.json() - except (JSONDecodeError, ValueError): - return self._failed_ingest_result( - error_code="invalid_json", - error_message="EverOS ingest returned invalid JSON", - retryable=True, - ) - if not isinstance(raw, dict): - return self._failed_ingest_result( - error_code="unexpected_response", - error_message="EverOS ingest returned an unexpected response shape", - retryable=True, - ) - return self._normalize_ingest_response(raw) - except httpx.TimeoutException as exc: - return self._failed_ingest_result("timeout", self._safe_error_message(exc), retryable=self._map_error(exc)) - except httpx.RequestError as exc: - return self._failed_ingest_result("network_error", self._safe_error_message(exc), retryable=self._map_error(exc)) - except Exception as exc: # noqa: BLE001 - return self._failed_ingest_result("unexpected_error", self._safe_error_message(exc), retryable=self._map_error(exc)) - - def extract_profile_long_term_v2(self, payload: dict[str, Any]) -> BackendCommitResult: - """v2 adapter placeholder for profile / long-term extraction. - - Mapping spec: commit_session returns BackendCommitResult and should - produce native episodic/profile/long-term refs once the real API is stable. - """ - runtime_payload = self._build_commit_payload(payload) - raw = { - "status": "success", - "session_id": runtime_payload.get("session_id"), - "metadata": { - "reason": "everos_v2_commit_fixture", - "schema_version": "everos.fixture.commit.v2", - }, - "data": { - "produced_refs": [ - { - "ref_type": "profile", - "profile_id": f"everos_profile:{runtime_payload.get('user_id') or 'unknown'}", - "metadata": {"schema_version": "everos.fixture.profile.v2"}, - }, - { - "ref_type": "long_term_memory", - "memory_id": f"everos_long_term:{runtime_payload.get('session_id')}", - "metadata": {"schema_version": "everos.fixture.long_term.v2"}, - }, - ] - }, - } - return self._normalize_commit_response(raw) - - def retrieve_context_v2(self, payload: dict[str, Any]) -> BackendRetrieveResult: - """ - Calls EverOS native API to retrieve memories. - """ - if not self._use_real_api: - return BackendRetrieveResult( - backend_type=BackendType.EVEROS, - operation=BackendOperation.RETRIEVE_CONTEXT, - status=BackendResultStatus.SKIPPED, - items=[], - metadata={"reason": "everos_retrieve_requires_real_mode"}, - ) - - query = payload.get("query", "") - user_id = payload.get("user_id", "") - try: - with httpx.Client( - base_url=self.base_url, - headers=self._headers(), - timeout=self.timeout, - verify=self.verify_ssl, - transport=self.transport, - ) as client: - resp = client.post( - self.search_path, - json={ - "query": query, - "method": self.retrieve_method, - "memory_types": ["episodic_memory", "profile", "raw_message"], - "top_k": payload.get("limit", 10), - "filters": self._search_filters(user_id=user_id, session_id=payload.get("session_id")), - }, - ) - if resp.status_code >= 400: - return BackendRetrieveResult( - backend_type=BackendType.EVEROS, - operation=BackendOperation.RETRIEVE_CONTEXT, - status=BackendResultStatus.FAILED, - items=[], - error_code=f"http_{resp.status_code}", - error_message=f"EverOS retrieve failed: {resp.text}", - retryable=False - ) - - items = self._items_from_search_response(resp.json()) - - raw = { - "status": "success", - "data": { - "items": items - } - } - return self._normalize_retrieve_response(raw) - except Exception as exc: - return BackendRetrieveResult( - backend_type=BackendType.EVEROS, - operation=BackendOperation.RETRIEVE_CONTEXT, - status=BackendResultStatus.FAILED, - items=[], - error_code="request_error", - error_message=str(exc), - retryable=True - ) - - def _build_ingest_payload(self, payload: dict[str, Any]) -> dict[str, Any]: - """ - Builds the payload according to EverOS native message schema. - """ - return { - "user_id": payload.get("user_id") or "gateway_user", - "session_id": payload.get("session_id"), - "messages": [ - { - "message_id": payload.get("turn_id") or f"msg_{int(datetime.now(timezone.utc).timestamp() * 1000)}", - "sender_id": payload.get("user_id") or "gateway_user", - "sender_name": payload.get("user_id") or "gateway_user", - "role": self._everos_role(payload.get("role", "user")), - "timestamp": self._timestamp_ms(payload), - "content": payload.get("content", ""), - } - ], - } - - def _build_commit_payload(self, payload: dict[str, Any]) -> dict[str, Any]: - return dict(payload) - - def _normalize_ingest_response(self, raw: dict[str, Any]) -> BackendWriteResult: - return normalize_everos_ingest_response(raw) - - def _normalize_commit_response(self, raw: dict[str, Any]) -> BackendCommitResult: - return normalize_everos_commit_response(raw) - - def _normalize_retrieve_response(self, raw: dict[str, Any]) -> BackendRetrieveResult: - return normalize_everos_retrieve_response(raw) - - def _map_error(self, exc_or_response: Any) -> bool: - status_code = getattr(exc_or_response, "status_code", None) - error_code = getattr(exc_or_response, "error_code", None) - error_message = str(exc_or_response) if exc_or_response is not None else None - return map_backend_error_to_retryable( - BackendType.EVEROS, - status_code=status_code, - error_code=error_code, - error_message=error_message, - ) - - def _failed_ingest_result(self, error_code: str, error_message: str, retryable: bool) -> BackendWriteResult: - return BackendWriteResult( - backend_type=BackendType.EVEROS, - operation=BackendOperation.INGEST_TURN, - status=BackendResultStatus.FAILED, - retryable=retryable, - error_code=error_code, - error_message=error_message, - metadata={"error_code": error_code}, - ) - - def _safe_error_message(self, exc: Exception) -> str: - return exc.__class__.__name__ - - def consolidate_session( - self, - session_id: str, - ctx: AccessContext, - episodes: list[EpisodeRecord], - existing_memories: list[MemoryRecord], - min_importance: float, - target_namespace: str | None, - ) -> dict[str, Any]: - if not self.base_url: - raise EverOSError("EverOS real mode requires base_url") - - user_id = ctx.user_id or "gateway_user" - agent_id = ctx.agent_id or "gateway_agent" - - with httpx.Client( - base_url=self.base_url, - timeout=self.timeout, - headers=self._headers(), - verify=self.verify_ssl, - transport=self.transport, - ) as client: - for episode in episodes: - self._memorize_episode(client, episode=episode, session_id=session_id, user_id=user_id, agent_id=agent_id) - self._flush_session(client, session_id=session_id, user_id=user_id) - promoted = self._fetch_session_memories(client, session_id=session_id, user_id=user_id, target_namespace=target_namespace) - - return { - "backend": "external", - "service_url": self.base_url, - "endpoint": self.ingest_path, - "raw": {"result": {"memories": promoted}}, - "session_id": session_id, - "episodes": len(episodes), - "candidates": promoted, - "promoted": promoted, - "duplicates": [], - "conflicts": [], - "review_drafts": [], - } - - def _memorize_episode( - self, - client: httpx.Client, - *, - episode: dict[str, Any], - session_id: str, - user_id: str, - agent_id: str, - ) -> None: - episode_data = episode.model_dump(mode="json") if hasattr(episode, "model_dump") else dict(episode) - episode_id = str(episode_data.get("id") or f"epi_{int(datetime.now(timezone.utc).timestamp())}") - sender = agent_id if episode_data.get("source") == "agent" else user_id - role = "assistant" if sender == agent_id else "user" - created_at = episode_data.get("created_at") or datetime.now(timezone.utc).isoformat() - payload = { - "user_id": user_id, - "session_id": session_id, - "messages": [ - { - "message_id": episode_id, - "sender_id": sender, - "sender_name": sender, - "role": role, - "timestamp": self._datetime_to_ms(created_at), - "content": episode_data.get("content") or "", - } - ], - } - response = client.post(self.ingest_path, json=payload) - response.raise_for_status() - - def _flush_session(self, client: httpx.Client, *, session_id: str, user_id: str) -> None: - response = client.post(self.flush_path, json={"user_id": user_id, "session_id": session_id}) - response.raise_for_status() - - def _fetch_session_memories( - self, - client: httpx.Client, - *, - session_id: str, - user_id: str, - target_namespace: str | None, - ) -> list[dict[str, Any]]: - response = client.post( - self.search_path, - json={ - "query": "memory", - "method": self.retrieve_method, - "memory_types": ["episodic_memory"], - "top_k": 20, - "filters": self._search_filters(user_id=user_id, session_id=session_id), - }, - ) - response.raise_for_status() - memories = self._items_from_search_response(response.json()) - normalized: list[dict[str, Any]] = [] - for index, memory in enumerate(memories, start=1): - content = memory.get("text") or memory.get("content") or memory.get("summary") or "" - if not content: - continue - normalized.append( - { - "id": memory.get("memory_id") or memory.get("id") or f"everos_{session_id}_{index}", - "namespace": target_namespace or f"user/{user_id}/long_term", - "memory_type": memory.get("memory_type") or "episodic_memory", - "content": content, - "summary": memory.get("summary") or content[:180], - "tags": ["everos-real", "memory-gateway"], - "importance": 0.7, - "confidence": 0.7, - "source": "everos", - "source_ref": memory.get("memory_id") or memory.get("id"), - } - ) - return normalized - - def _items_from_search_response(self, payload: dict[str, Any]) -> list[dict[str, Any]]: - data = payload.get("data") if isinstance(payload.get("data"), dict) else payload - items: list[dict[str, Any]] = [] - for memory_type, key in ( - ("episodic_memory", "episodes"), - ("profile", "profiles"), - ("raw_message", "raw_messages"), - ): - for item in data.get(key, []) or []: - if isinstance(item, dict): - items.append({**item, "memory_type": item.get("memory_type") or memory_type, "text": self._memory_text(item)}) - agent_memory = data.get("agent_memory") or {} - if isinstance(agent_memory, dict): - for item in agent_memory.get("cases", []) or []: - if isinstance(item, dict): - items.append({**item, "memory_type": "agent_case", "text": self._memory_text(item)}) - for item in agent_memory.get("skills", []) or []: - if isinstance(item, dict): - items.append({**item, "memory_type": "agent_skill", "text": self._memory_text(item)}) - return items - - def _memory_text(self, item: dict[str, Any]) -> str: - content_items = item.get("content_items") - if isinstance(content_items, list): - content_text = "\n".join( - str(content.get("text") or content.get("content") or "") - for content in content_items - if isinstance(content, dict) - ).strip() - else: - content_text = "" - profile_data = item.get("profile_data") - if isinstance(profile_data, dict): - profile_text = str(profile_data) - else: - profile_text = "" - return ( - item.get("episode") - or item.get("summary") - or item.get("subject") - or item.get("atomic_fact") - or item.get("task_intent") - or item.get("approach") - or item.get("content") - or content_text - or item.get("description") - or profile_text - or "" - ) - - def _search_filters(self, *, user_id: str | None, session_id: str | None = None) -> dict[str, Any]: - filters: dict[str, Any] = {"user_id": user_id or "gateway_user"} - if session_id: - filters["session_id"] = session_id - return filters - - def _timestamp_ms(self, payload: dict[str, Any]) -> int: - trace = payload.get("trace") if isinstance(payload.get("trace"), dict) else {} - timestamp = trace.get("timestamp") or payload.get("created_at") - if timestamp: - return self._datetime_to_ms(timestamp) - return int(datetime.now(timezone.utc).timestamp() * 1000) - - def _datetime_to_ms(self, value: Any) -> int: - if isinstance(value, (int, float)): - return int(value if value > 1_000_000_000_000 else value * 1000) - if isinstance(value, str): - text = value.replace("Z", "+00:00") - try: - return int(datetime.fromisoformat(text).timestamp() * 1000) - except ValueError: - return int(datetime.now(timezone.utc).timestamp() * 1000) - if isinstance(value, datetime): - return int(value.timestamp() * 1000) - return int(datetime.now(timezone.utc).timestamp() * 1000) - - def _everos_role(self, role: str) -> str: - if role in {"assistant", "agent"}: - return "assistant" - if role == "tool": - return "assistant" - return "user" diff --git a/memory_gateway/llm.py b/memory_gateway/llm.py deleted file mode 100644 index f17bd29..0000000 --- a/memory_gateway/llm.py +++ /dev/null @@ -1,158 +0,0 @@ -"""LLM helpers for Memory Gateway summaries.""" -from __future__ import annotations - -import json -import os -import re -from typing import Any - -import httpx - -from .config import get_config - - -class LLMConfigurationError(RuntimeError): - """Raised when LLM summarization is requested but not configured.""" - - -class LLMSummaryError(RuntimeError): - """Raised when the LLM response cannot be used.""" - - -def _llm_settings() -> dict[str, Any]: - config = get_config() - llm_config = getattr(config, "llm", None) - - base_url = ( - os.environ.get("MEMORY_GATEWAY_LLM_BASE_URL") - or os.environ.get("OPENAI_BASE_URL") - or getattr(llm_config, "base_url", "") - or "https://api.openai.com/v1" - ).rstrip("/") - api_key = ( - os.environ.get("MEMORY_GATEWAY_LLM_API_KEY") - or os.environ.get("OPENAI_API_KEY") - or getattr(llm_config, "api_key", "") - ) - model = ( - os.environ.get("MEMORY_GATEWAY_LLM_MODEL") - or os.environ.get("OPENAI_MODEL") - or getattr(llm_config, "model", "") - ) - timeout = int(os.environ.get("MEMORY_GATEWAY_LLM_TIMEOUT") or getattr(llm_config, "timeout", 60)) - max_input_chars = int(os.environ.get("MEMORY_GATEWAY_LLM_MAX_INPUT_CHARS") or getattr(llm_config, "max_input_chars", 24000)) - return { - "base_url": base_url, - "api_key": api_key, - "model": model, - "timeout": timeout, - "max_input_chars": max_input_chars, - } - - -def _extract_json(text: str) -> dict[str, Any]: - text = text.strip() - if text.startswith("```"): - text = re.sub(r"^```(?:json)?\s*", "", text) - text = re.sub(r"\s*```$", "", text) - try: - return json.loads(text) - except json.JSONDecodeError: - match = re.search(r"\{.*\}", text, flags=re.S) - if not match: - raise LLMSummaryError("LLM did not return JSON") from None - return json.loads(match.group(0)) - - -def _coerce_string_list(value: Any, limit: int = 12) -> list[str]: - if not isinstance(value, list): - return [] - items: list[str] = [] - for item in value: - if item is None: - continue - text = str(item).strip() - if text and text not in items: - items.append(text[:300]) - if len(items) >= limit: - break - return items - - -async def summarize_with_llm( - content: str, - *, - title: str | None = None, - summary_hint: str | None = None, - tags: list[str] | None = None, - max_summary_chars: int = 800, - purpose: str = "generic knowledge memory", -) -> dict[str, Any]: - """Summarize content using an OpenAI-compatible chat completions API.""" - settings = _llm_settings() - if not settings["model"]: - raise LLMConfigurationError("LLM model is not configured. Set MEMORY_GATEWAY_LLM_MODEL or llm.model.") - if not settings["api_key"] and not settings["base_url"].startswith(("http://127.0.0.1", "http://localhost")): - raise LLMConfigurationError("LLM API key is not configured. Set MEMORY_GATEWAY_LLM_API_KEY or OPENAI_API_KEY.") - - trimmed = content[: settings["max_input_chars"]] - tag_text = ", ".join(tags or []) - system_prompt = ( - "You are a precise knowledge curator. Summarize input into reusable memory. " - "Return only valid JSON with these keys: title, summary, key_points, tags. " - "summary must be concise but specific; key_points must be reusable, evidence-based bullets. " - "Do not invent facts not present in the input. Preserve important identifiers, paths, URLs, IPs, IDs, and verdicts." - ) - user_prompt = f""" -Purpose: {purpose} -Provided title: {title or ''} -Provided summary hint: {summary_hint or ''} -Provided tags: {tag_text} -Max summary characters: {max_summary_chars} - -Content: -{trimmed} -""".strip() - - headers = {"Content-Type": "application/json"} - if settings["api_key"]: - headers["Authorization"] = f"Bearer {settings['api_key']}" - - payload = { - "model": settings["model"], - "messages": [ - {"role": "system", "content": system_prompt}, - {"role": "user", "content": user_prompt}, - ], - "temperature": 0.2, - "response_format": {"type": "json_object"}, - } - async with httpx.AsyncClient(timeout=settings["timeout"]) as client: - response = await client.post(f"{settings['base_url']}/chat/completions", headers=headers, json=payload) - response.raise_for_status() - data = response.json() - - try: - content_text = data["choices"][0]["message"]["content"] - except (KeyError, IndexError, TypeError) as exc: - raise LLMSummaryError(f"Unexpected LLM response shape: {data}") from exc - - parsed = _extract_json(content_text) - merged_tags = [] - for tag in [*(tags or []), *_coerce_string_list(parsed.get("tags"), limit=8)]: - tag = str(tag).strip() - if tag and tag not in merged_tags: - merged_tags.append(tag) - - summary = str(parsed.get("summary") or "").strip() - return { - "title": str(parsed.get("title") or title or "Untitled summary").strip()[:160], - "summary": summary[:max(120, max_summary_chars)], - "key_points": _coerce_string_list(parsed.get("key_points"), limit=10), - "tags": merged_tags, - "llm": { - "provider": "openai-compatible", - "base_url": settings["base_url"], - "model": settings["model"], - }, - } diff --git a/memory_gateway/mcp_tools_v1.py b/memory_gateway/mcp_tools_v1.py deleted file mode 100644 index 05e39f3..0000000 --- a/memory_gateway/mcp_tools_v1.py +++ /dev/null @@ -1,135 +0,0 @@ -"""MCP tool definitions for the generic Memory Gateway contract. - -The legacy MCP endpoint in server.py remains available. These definitions are -the target v1 tool contract for Nanobot, Hermes Agent, OpenClaw, and other -agent frameworks. -""" - -MEMORY_GATEWAY_MCP_TOOLS = [ - { - "name": "memory_search", - "description": "Search accessible memories with user/agent/workspace/session isolation.", - "inputSchema": { - "type": "object", - "properties": { - "user_id": {"type": "string"}, - "agent_id": {"type": "string"}, - "workspace_id": {"type": "string"}, - "session_id": {"type": "string"}, - "query": {"type": "string"}, - "namespaces": {"type": "array", "items": {"type": "string"}}, - "limit": {"type": "integer", "default": 10}, - }, - "required": ["user_id", "query"], - }, - }, - { - "name": "memory_upsert", - "description": "Create or update a memory record after ACL and namespace routing.", - "inputSchema": { - "type": "object", - "properties": { - "user_id": {"type": "string"}, - "agent_id": {"type": "string"}, - "workspace_id": {"type": "string"}, - "session_id": {"type": "string"}, - "namespace": {"type": "string"}, - "memory_type": {"type": "string"}, - "content": {"type": "string"}, - "summary": {"type": "string"}, - "tags": {"type": "array", "items": {"type": "string"}}, - "importance": {"type": "number"}, - "confidence": {"type": "number"}, - "visibility": {"type": "string"}, - }, - "required": ["user_id", "content"], - }, - }, - { - "name": "memory_append_episode", - "description": "Append temporary episode/session memory without automatically promoting it.", - "inputSchema": { - "type": "object", - "properties": { - "user_id": {"type": "string"}, - "agent_id": {"type": "string"}, - "workspace_id": {"type": "string"}, - "session_id": {"type": "string"}, - "content": {"type": "string"}, - "events": {"type": "array", "items": {"type": "object"}}, - "tags": {"type": "array", "items": {"type": "string"}}, - }, - "required": ["user_id", "session_id", "content"], - }, - }, - { - "name": "memory_commit_session", - "description": "Promote selected session memories into long-term memory via consolidation.", - "inputSchema": { - "type": "object", - "properties": { - "user_id": {"type": "string"}, - "agent_id": {"type": "string"}, - "workspace_id": {"type": "string"}, - "session_id": {"type": "string"}, - "promote": {"type": "boolean", "default": True}, - "min_importance": {"type": "number", "default": 0.6}, - }, - "required": ["user_id", "session_id"], - }, - }, - { - "name": "memory_get_profile", - "description": "Get the effective user profile memory.", - "inputSchema": { - "type": "object", - "properties": {"user_id": {"type": "string"}}, - "required": ["user_id"], - }, - }, - { - "name": "memory_list_namespaces", - "description": "List namespaces visible to the current user/agent/workspace context.", - "inputSchema": { - "type": "object", - "properties": { - "user_id": {"type": "string"}, - "agent_id": {"type": "string"}, - "workspace_id": {"type": "string"}, - "session_id": {"type": "string"}, - }, - "required": ["user_id"], - }, - }, - { - "name": "memory_delete", - "description": "Delete or archive a memory record if the caller has access.", - "inputSchema": { - "type": "object", - "properties": { - "user_id": {"type": "string"}, - "agent_id": {"type": "string"}, - "workspace_id": {"type": "string"}, - "memory_id": {"type": "string"}, - }, - "required": ["user_id", "memory_id"], - }, - }, - { - "name": "memory_feedback", - "description": "Attach quality feedback to a memory record for pruning/merge decisions.", - "inputSchema": { - "type": "object", - "properties": { - "user_id": {"type": "string"}, - "agent_id": {"type": "string"}, - "workspace_id": {"type": "string"}, - "memory_id": {"type": "string"}, - "feedback": {"type": "string"}, - "comment": {"type": "string"}, - }, - "required": ["user_id", "memory_id", "feedback"], - }, - }, -] - diff --git a/memory_gateway/namespace.py b/memory_gateway/namespace.py deleted file mode 100644 index 9226a61..0000000 --- a/memory_gateway/namespace.py +++ /dev/null @@ -1,115 +0,0 @@ -"""Namespace construction and access checks for Memory Gateway.""" -from __future__ import annotations - -from datetime import datetime, timezone - -from .schemas import AccessContext, MemoryRecord, NamespaceInfo, Visibility - - -def user_profile_namespace(user_id: str) -> str: - return f"user/{user_id}/profile" - - -def user_preferences_namespace(user_id: str) -> str: - return f"user/{user_id}/preferences" - - -def user_long_term_namespace(user_id: str) -> str: - return f"user/{user_id}/long_term" - - -def agent_memory_namespace(agent_id: str) -> str: - return f"agent/{agent_id}/memory" - - -def workspace_shared_namespace(workspace_id: str) -> str: - return f"workspace/{workspace_id}/shared" - - -def session_episodic_namespace(session_id: str) -> str: - return f"session/{session_id}/episodic" - - -def global_public_namespace() -> str: - return "global/public" - - -def default_namespace_for_context(ctx: AccessContext, visibility: Visibility) -> str: - if visibility == Visibility.AGENT_ONLY and ctx.agent_id: - return agent_memory_namespace(ctx.agent_id) - if visibility == Visibility.WORKSPACE_SHARED and ctx.workspace_id: - return workspace_shared_namespace(ctx.workspace_id) - if ctx.session_id: - return session_episodic_namespace(ctx.session_id) - return user_long_term_namespace(ctx.user_id) - - -def can_access_memory(ctx: AccessContext, memory: MemoryRecord) -> bool: - if memory.expires_at and memory.expires_at <= datetime.now(timezone.utc): - return False - if memory.visibility == Visibility.GLOBAL: - return True - if memory.visibility == Visibility.PRIVATE: - return memory.user_id == ctx.user_id - if memory.visibility == Visibility.AGENT_ONLY: - return memory.user_id == ctx.user_id and memory.agent_id == ctx.agent_id - if memory.visibility == Visibility.WORKSPACE_SHARED: - return memory.workspace_id is not None and memory.workspace_id == ctx.workspace_id - return False - - -def visible_namespaces(ctx: AccessContext) -> list[NamespaceInfo]: - namespaces = [ - NamespaceInfo( - namespace=user_profile_namespace(ctx.user_id), - owner_user_id=ctx.user_id, - visibility=Visibility.PRIVATE, - description="用户 profile 与稳定偏好", - ), - NamespaceInfo( - namespace=user_preferences_namespace(ctx.user_id), - owner_user_id=ctx.user_id, - visibility=Visibility.PRIVATE, - description="用户显式偏好", - ), - NamespaceInfo( - namespace=user_long_term_namespace(ctx.user_id), - owner_user_id=ctx.user_id, - visibility=Visibility.PRIVATE, - description="用户长期记忆", - ), - NamespaceInfo( - namespace=global_public_namespace(), - visibility=Visibility.GLOBAL, - description="全局公开知识", - ), - ] - if ctx.agent_id: - namespaces.append( - NamespaceInfo( - namespace=agent_memory_namespace(ctx.agent_id), - owner_user_id=ctx.user_id, - visibility=Visibility.AGENT_ONLY, - description="指定 agent 私有经验", - ) - ) - if ctx.workspace_id: - namespaces.append( - NamespaceInfo( - namespace=workspace_shared_namespace(ctx.workspace_id), - owner_user_id=ctx.user_id, - visibility=Visibility.WORKSPACE_SHARED, - description="workspace / project 共享记忆", - ) - ) - if ctx.session_id: - namespaces.append( - NamespaceInfo( - namespace=session_episodic_namespace(ctx.session_id), - owner_user_id=ctx.user_id, - visibility=Visibility.PRIVATE, - description="session 临时 episodic memory", - ) - ) - return namespaces - diff --git a/memory_gateway/obsidian_review.py b/memory_gateway/obsidian_review.py deleted file mode 100644 index c12ba44..0000000 --- a/memory_gateway/obsidian_review.py +++ /dev/null @@ -1,77 +0,0 @@ -"""Obsidian review draft writer.""" -from __future__ import annotations - -import re -from datetime import datetime, timezone -from pathlib import Path - -from .config import get_config -from .schemas import MemoryRecord - - -def _slugify(value: str, fallback: str) -> str: - slug = re.sub(r"[^a-zA-Z0-9\u4e00-\u9fff_-]+", "-", value.lower()).strip("-") - slug = re.sub(r"-+", "-", slug)[:80].strip("-") - return slug or fallback - - -def write_review_draft(memory: MemoryRecord, reason: str, conflict_ids: list[str] | None = None) -> Path: - config = get_config() - review_dir = getattr(config.obsidian, "review_dir", "Reviews/Queue") - vault_path = Path(config.obsidian.vault_path) - target_dir = vault_path / review_dir - target_dir.mkdir(parents=True, exist_ok=True) - - title = memory.summary or memory.content[:80] or memory.id - filename = f"{_slugify(title, memory.id)}-{memory.id}.md" - path = target_dir / filename - conflict_ids = conflict_ids or [] - - content = "\n".join( - [ - "---", - "type: memory_review", - f"memory_id: {memory.id}", - f"user_id: {memory.user_id}", - f"agent_id: {memory.agent_id or ''}", - f"workspace_id: {memory.workspace_id or ''}", - f"namespace: {memory.namespace}", - f"visibility: {memory.visibility.value}", - f"importance: {memory.importance}", - f"confidence: {memory.confidence}", - f"reason: {reason}", - f"created_at: {datetime.now(timezone.utc).isoformat()}", - "tags:", - " - memory/review", - " - source/everos", - "---", - "", - f"# Memory Review - {title}", - "", - "## Candidate", - "", - memory.content, - "", - "## Summary", - "", - memory.summary or "", - "", - "## Proposed Action", - "", - "- [ ] Accept", - "- [ ] Edit", - "- [ ] Reject", - "- [ ] Merge", - "- [ ] Archive", - "", - "## Conflict IDs", - "", - "\n".join(f"- {memory_id}" for memory_id in conflict_ids) if conflict_ids else "- none", - "", - "## Notes", - "", - ] - ) - path.write_text(content, encoding="utf-8") - return path - diff --git a/memory_gateway/obsidian_review_client.py b/memory_gateway/obsidian_review_client.py deleted file mode 100644 index aa9f366..0000000 --- a/memory_gateway/obsidian_review_client.py +++ /dev/null @@ -1,25 +0,0 @@ -"""Human-review backend skeleton for Memory Gateway v2. - -Obsidian remains a human-in-the-loop review backend only. This skeleton does -not write files or call external APIs; it preserves the adapter contract until -the review draft integration is explicitly designed. -""" -from __future__ import annotations - -from typing import Any - -from .backend_contracts import BackendOperation, BackendResultStatus, BackendWriteResult -from .schemas_v2 import BackendType - - -class ObsidianReviewClient: - def create_review_draft_v2(self, payload: dict[str, Any]) -> BackendWriteResult: - """Return a skipped review-draft result until the real adapter exists.""" - return BackendWriteResult( - backend_type=BackendType.OBSIDIAN, - operation=BackendOperation.CREATE_REVIEW_DRAFT, - status=BackendResultStatus.SKIPPED, - native_id=payload.get("event_id") or payload.get("gateway_id"), - retryable=False, - metadata={"reason": "obsidian_review_adapter_not_configured"}, - ) diff --git a/memory_gateway/openviking_client.py b/memory_gateway/openviking_client.py deleted file mode 100644 index d714c48..0000000 --- a/memory_gateway/openviking_client.py +++ /dev/null @@ -1,522 +0,0 @@ -"""OpenViking client wrapper used by Memory Gateway.""" -from __future__ import annotations - -import logging -import mimetypes -import tempfile -from json import JSONDecodeError -from pathlib import Path -from typing import Any, Optional - -import httpx - -from .backend_contracts import BackendCommitResult, BackendOperation, BackendResultStatus, BackendRetrieveResult, BackendWriteResult -from .backend_normalization import ( - map_backend_error_to_retryable, - normalize_openviking_commit_response, - normalize_openviking_ingest_response, - normalize_openviking_retrieve_response, -) -from .config import get_config -from .schemas_v2 import BackendType -from .types import MemoryEntry, ResourceEntry, SearchResult - -logger = logging.getLogger(__name__) - - -class OpenVikingClient: - """Thin async client for the OpenViking HTTP API.""" - - def __init__( - self, - base_url: Optional[str] = None, - api_key: Optional[str] = None, - timeout: int | None = None, - account: str = "default", - user: str = "default", - enabled: bool | None = None, - mode: str | None = None, - verify_ssl: bool | None = None, - ingest_path: str | None = None, - transport: httpx.AsyncBaseTransport | None = None, - ): - self.config = get_config() - self.base_url = base_url if base_url is not None else self.config.openviking.url - self.api_key = api_key if api_key is not None else (self.config.openviking.api_key or "your-secret-root-key") - self.timeout = timeout if timeout is not None else self.config.openviking.timeout - self.account = account - self.user = user - self.enabled = self.config.openviking.enabled if enabled is None else enabled - self.mode = mode or self.config.openviking.mode - self.verify_ssl = self.config.openviking.verify_ssl if verify_ssl is None else verify_ssl - self.ingest_path = ingest_path or self.config.openviking.ingest_path - self.transport = transport - self._client: Optional[httpx.AsyncClient] = None - - def _get_headers(self) -> dict[str, str]: - headers = {} - if self.api_key: - headers["X-API-Key"] = self.api_key - headers["Authorization"] = f"Bearer {self.api_key}" - headers["X-OpenViking-Account"] = self.account - headers["X-OpenViking-User"] = self.user - return headers - - async def _get_client(self) -> httpx.AsyncClient: - if self._client is None: - self._client = httpx.AsyncClient( - base_url=self.base_url, - headers=self._get_headers(), - timeout=self.timeout, - verify=self.verify_ssl, - transport=self.transport, - ) - return self._client - - async def close(self): - if self._client: - await self._client.aclose() - self._client = None - - async def health_check(self) -> dict[str, Any]: - client = await self._get_client() - try: - response = await client.get("/health") - response.raise_for_status() - return response.json() - except httpx.HTTPError as e: - logger.error(f"OpenViking 健康检查失败: {e}") - return {"status": "error", "message": str(e)} - - async def ingest_conversation_turn(self, payload: dict[str, Any]) -> BackendWriteResult: - """v2 adapter placeholder for OpenViking session archive ingestion. - - Mapping spec: `backend_adapter_mapping.AdapterMappingSpec` maps - OpenViking ingest_turn to this method and requires BackendWriteResult. - Payloads must contain only control-plane fields; conversation content - is not persisted by the Gateway control-plane store. - - TODO(v2): bind this to OpenViking's stable session/message archive API - once that contract is finalized. Until then the gateway records a - skipped backend ref instead of inventing an unstable HTTP contract. - """ - runtime_payload = self._build_ingest_payload(payload) - if self._use_real_api: - return await self._ingest_conversation_turn_real(runtime_payload) - raw = { - "status": "skipped", - "session_id": runtime_payload.get("session_id"), - "uri": f"viking://sessions/{runtime_payload.get('session_id')}", - "metadata": { - "reason": "openviking_v2_ingest_adapter_not_configured", - "schema_version": "openviking.fixture.ingest.v2", - }, - } - return self._normalize_ingest_response(raw) - - @property - def _use_real_api(self) -> bool: - # Real ingest is strictly gated by mode=real. The legacy `enabled` - # field is retained for config compatibility, but must not trigger - # network traffic by itself. - return self.mode == "real" - - async def _ingest_conversation_turn_real(self, runtime_payload: dict[str, Any]) -> BackendWriteResult: - if not self.base_url: - return self._failed_ingest_result( - error_code="config_error", - error_message="OpenViking real ingest is enabled but base_url is missing", - retryable=False, - ) - try: - client = await self._get_client() - response = await client.post( - self._format_ingest_path(runtime_payload), - json=runtime_payload, - ) - if response.status_code >= 400: - return self._failed_ingest_result( - error_code=f"http_{response.status_code}", - error_message=f"OpenViking ingest failed with HTTP {response.status_code}", - retryable=self._map_error(response), - ) - try: - raw = response.json() - except (JSONDecodeError, ValueError): - return self._failed_ingest_result( - error_code="invalid_json", - error_message="OpenViking ingest returned invalid JSON", - retryable=True, - ) - if not isinstance(raw, dict): - return self._failed_ingest_result( - error_code="unexpected_response", - error_message="OpenViking ingest returned an unexpected response shape", - retryable=True, - ) - return self._normalize_ingest_response(raw) - except httpx.TimeoutException as exc: - return self._failed_ingest_result("timeout", self._safe_error_message(exc), retryable=self._map_error(exc)) - except httpx.RequestError as exc: - return self._failed_ingest_result("network_error", self._safe_error_message(exc), retryable=self._map_error(exc)) - except Exception as exc: # noqa: BLE001 - return self._failed_ingest_result("unexpected_error", self._safe_error_message(exc), retryable=self._map_error(exc)) - - async def commit_session_v2(self, payload: dict[str, Any]) -> BackendCommitResult: - """v2 adapter placeholder for OpenViking session commit. - - Mapping spec: commit_session returns BackendCommitResult and should - produce a native session/archive ref once the real API is stable. - """ - runtime_payload = self._build_commit_payload(payload) - raw = { - "status": "success", - "session_id": runtime_payload.get("session_id"), - "metadata": { - "reason": "openviking_v2_commit_fixture", - "schema_version": "openviking.fixture.commit.v2", - }, - "result": { - "refs": [ - { - "type": "session_summary", - "id": f"ov_session_summary:{runtime_payload.get('session_id')}", - "uri": f"viking://sessions/{runtime_payload.get('session_id')}/summary", - "metadata": {"schema_version": "openviking.fixture.ref.v2"}, - } - ] - }, - } - return self._normalize_commit_response(raw) - - async def retrieve_context_v2(self, payload: dict[str, Any]) -> BackendRetrieveResult: - """ - Calls OpenViking native API to retrieve context. - Uses POST /search - """ - if not self._use_real_api: - return BackendRetrieveResult( - backend_type=BackendType.OPENVIKING, - operation=BackendOperation.RETRIEVE_CONTEXT, - status=BackendResultStatus.SKIPPED, - items=[], - metadata={"reason": "openviking_retrieve_requires_real_mode"}, - ) - - query = payload.get("query", "") - session_id = payload.get("session_id") - - request_data = {"query": query, "limit": 10} - if session_id: - request_data["session_id"] = session_id - - try: - client = await self._get_client() - response = await client.post("/api/v1/search/search", json=request_data) - - if response.status_code >= 400: - return BackendRetrieveResult( - backend_type=BackendType.OPENVIKING, - operation=BackendOperation.RETRIEVE_CONTEXT, - status=BackendResultStatus.FAILED, - items=[], - error_code=f"http_{response.status_code}", - error_message=f"OpenViking search failed: {response.text}", - retryable=False - ) - - return self._normalize_retrieve_response(response.json()) - except Exception as exc: - return BackendRetrieveResult( - backend_type=BackendType.OPENVIKING, - operation=BackendOperation.RETRIEVE_CONTEXT, - status=BackendResultStatus.FAILED, - items=[], - error_code="request_error", - error_message=str(exc), - retryable=True - ) - - def _build_ingest_payload(self, payload: dict[str, Any]) -> dict[str, Any]: - """ - Build payload for native OpenViking AddMessageRequest. - OpenViking only expects role and content, and maybe metadata. - """ - return { - "role": payload.get("role", "user"), - "content": payload.get("content", ""), - "metadata": payload.get("metadata", {}), - "session_id": payload.get("session_id") # kept so format_ingest_path can use it - } - - def _format_ingest_path(self, payload: dict[str, Any]) -> str: - session_id = str(payload.get("session_id") or "unknown") - return self.ingest_path.format(session_id=session_id) - - def _build_commit_payload(self, payload: dict[str, Any]) -> dict[str, Any]: - return dict(payload) - - def _normalize_ingest_response(self, raw: dict[str, Any]) -> BackendWriteResult: - return normalize_openviking_ingest_response(raw) - - def _normalize_commit_response(self, raw: dict[str, Any]) -> BackendCommitResult: - return normalize_openviking_commit_response(raw) - - def _normalize_retrieve_response(self, raw: dict[str, Any]) -> BackendRetrieveResult: - return normalize_openviking_retrieve_response(raw) - - def _map_error(self, exc_or_response: Any) -> bool: - status_code = getattr(exc_or_response, "status_code", None) - error_code = getattr(exc_or_response, "error_code", None) - error_message = str(exc_or_response) if exc_or_response is not None else None - return map_backend_error_to_retryable( - BackendType.OPENVIKING, - status_code=status_code, - error_code=error_code, - error_message=error_message, - ) - - def _failed_ingest_result(self, error_code: str, error_message: str, retryable: bool) -> BackendWriteResult: - return BackendWriteResult( - backend_type=BackendType.OPENVIKING, - operation=BackendOperation.INGEST_TURN, - status=BackendResultStatus.FAILED, - retryable=retryable, - error_code=error_code, - error_message=error_message, - metadata={"error_code": error_code}, - ) - - def _safe_error_message(self, exc: Exception) -> str: - return exc.__class__.__name__ - - async def search( - self, - query: str, - namespace: Optional[str] = None, - limit: Optional[int] = None, - uri: Optional[str] = None, - ) -> SearchResult: - """Semantic search against OpenViking resources/memories.""" - client = await self._get_client() - - payload: dict[str, Any] = {"query": query} - if limit: - payload["limit"] = limit - - if uri: - payload["target_uri"] = uri - elif namespace: - payload["target_uri"] = f"viking://{namespace}" - - try: - response = await client.post("/api/v1/search/search", json=payload) - response.raise_for_status() - data = response.json() - - if data.get("status") != "ok": - logger.warning(f"搜索返回错误: {data.get('error')}") - return SearchResult(results=[], total=0) - - result = data.get("result", {}) - memories = result.get("memories", []) - resources = result.get("resources", []) - - all_results = [] - for m in memories + resources: - all_results.append( - { - "uri": m.get("uri"), - "abstract": m.get("abstract"), - "score": m.get("score"), - "context_type": m.get("context_type"), - } - ) - - return SearchResult(results=all_results, total=result.get("total", len(all_results))) - except httpx.HTTPError as e: - logger.error(f"搜索失败: {e}") - return SearchResult(results=[], total=0) - - async def add_memory( - self, - content: str, - namespace: Optional[str] = None, - memory_type: str = "general", - ) -> dict[str, Any]: - """Add memory via session commit flow.""" - client = await self._get_client() - ns = namespace or self.config.memory.default_namespace or "user/default/memories" - - try: - response = await client.post("/api/v1/sessions") - response.raise_for_status() - session_data = response.json() - - if session_data.get("status") != "ok": - return session_data - - session_id = session_data["result"]["session_id"] - message_response = await client.post( - f"/api/v1/sessions/{session_id}/messages", - json={ - "role": "user", - "content": f"[{ns}/{memory_type}] {content}", - }, - ) - message_response.raise_for_status() - commit_response = await client.post(f"/api/v1/sessions/{session_id}/commit") - commit_response.raise_for_status() - return commit_response.json() - except httpx.HTTPError as e: - logger.error(f"添加记忆失败: {e}") - raise - - async def _upload_temp_file(self, file_path: str | Path) -> str: - client = await self._get_client() - file_path = Path(file_path) - mime_type = mimetypes.guess_type(file_path.name)[0] or "application/octet-stream" - - with file_path.open("rb") as f: - response = await client.post( - "/api/v1/resources/temp_upload", - files={"file": (file_path.name, f, mime_type)}, - ) - response.raise_for_status() - data = response.json() - result = data.get("result", {}) - if "temp_path" in result: - return result["temp_path"] - if "temp_file_id" in result: - return result["temp_file_id"] - raise KeyError(f"Unexpected temp upload response: {data}") - - async def add_resource( - self, - uri: str, - content: str, - resource_type: str = "text", - wait: bool = False, - ) -> dict[str, Any]: - """Add a text/json resource by uploading a temporary file first. - - OpenViking HTTP API does not accept raw `uri + content` directly. The - client must upload a temp file and then create the resource with `to`. - """ - client = await self._get_client() - suffix_map = { - "json": ".json", - "text": ".txt", - "markdown": ".md", - "md": ".md", - } - suffix = suffix_map.get(resource_type, ".txt") - - with tempfile.NamedTemporaryFile("w", encoding="utf-8", suffix=suffix, delete=False) as tmp: - tmp.write(content) - tmp_path = Path(tmp.name) - - try: - temp_ref = await self._upload_temp_file(tmp_path) - payload = { - "temp_path": temp_ref, - "to": uri, - "wait": wait, - "strict": False, - } - response = await client.post("/api/v1/resources", json=payload) - if response.status_code >= 400: - logger.error("添加资源失败响应: %s", response.text) - response.raise_for_status() - return response.json() - except httpx.HTTPError as e: - logger.error(f"添加资源失败: {e}") - raise - finally: - tmp_path.unlink(missing_ok=True) - - async def list_memories( - self, - namespace: Optional[str] = None, - memory_type: Optional[str] = None, - limit: Optional[int] = None, - ) -> list[MemoryEntry]: - client = await self._get_client() - - ns = namespace or "user/default/memories" - if memory_type: - ns = f"{ns}/{memory_type}" - - try: - response = await client.post( - "/api/v1/search/search", - json={"query": "", "target_uri": f"viking://{ns}", "limit": limit or 10}, - ) - response.raise_for_status() - data = response.json() - - if data.get("status") == "ok": - result = data.get("result", {}) - memories = result.get("memories", []) - return [ - MemoryEntry( - id=m.get("uri", ""), - content=m.get("abstract", ""), - namespace=ns, - memory_type=memory_type or "general", - ) - for m in memories - ] - return [] - except httpx.HTTPError as e: - logger.error(f"列出记忆失败: {e}") - return [] - - async def list_resources( - self, - namespace: Optional[str] = None, - limit: Optional[int] = None, - ) -> list[ResourceEntry]: - client = await self._get_client() - - uri = f"viking://{namespace}" if namespace else "viking://resources" - try: - response = await client.post( - "/api/v1/search/search", - json={"query": "", "target_uri": uri, "limit": limit or 10}, - ) - response.raise_for_status() - data = response.json() - - if data.get("status") == "ok": - result = data.get("result", {}) - resources = result.get("resources", []) - return [ - ResourceEntry( - uri=r.get("uri", ""), - content=r.get("abstract", ""), - resource_type="text", - ) - for r in resources - ] - return [] - except httpx.HTTPError as e: - logger.error(f"列出资源失败: {e}") - return [] - - -_client: Optional[OpenVikingClient] = None - - -async def get_openviking_client() -> OpenVikingClient: - global _client - if _client is None: - _client = OpenVikingClient() - return _client - - -async def close_openviking_client(): - global _client - if _client: - await _client.close() - _client = None diff --git a/memory_gateway/repositories.py b/memory_gateway/repositories.py deleted file mode 100644 index a6e0fa1..0000000 --- a/memory_gateway/repositories.py +++ /dev/null @@ -1,1103 +0,0 @@ -"""Metadata repositories for Memory Gateway. - -SQLite is the default POC store. The in-memory implementation is retained for -small isolated tests and for cases where persistence is explicitly disabled. -""" -from __future__ import annotations - -import json -import sqlite3 -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Iterable, Optional, Protocol - -from .backend_contracts import BackendOperation, CommitJob, OutboxEvent, OutboxEventStatus -from .config import get_config -from .schemas import AuditLog, EpisodeRecord, MemoryRecord, ProfileRecord, UserRecord -from .schemas_v2 import BackendRefStatus, BackendType, MemoryRef, MemoryRefType - - -class MetadataRepository(Protocol): - def create_user(self, user: UserRecord) -> UserRecord: ... - def get_user(self, user_id: str) -> Optional[UserRecord]: ... - def upsert_memory(self, memory: MemoryRecord) -> MemoryRecord: ... - def get_memory(self, memory_id: str) -> Optional[MemoryRecord]: ... - def delete_memory(self, memory_id: str) -> bool: ... - def list_memories(self) -> Iterable[MemoryRecord]: ... - def append_episode(self, episode: EpisodeRecord) -> EpisodeRecord: ... - def list_session_episodes(self, session_id: str) -> list[EpisodeRecord]: ... - def get_profile(self, user_id: str) -> Optional[ProfileRecord]: ... - def upsert_profile(self, profile: ProfileRecord) -> ProfileRecord: ... - def add_audit(self, audit: AuditLog) -> AuditLog: ... - def list_audit(self, limit: int = 100) -> list[AuditLog]: ... - def save_memory_ref(self, ref: MemoryRef) -> MemoryRef: ... - def get_memory_ref(self, ref_id: str) -> MemoryRef | None: ... - def list_memory_refs( - self, - gateway_id: str | None = None, - workspace_id: str | None = None, - user_id: str | None = None, - agent_id: str | None = None, - session_id: str | None = None, - namespace: str | None = None, - backend_type: BackendType | str | None = None, - ref_type: MemoryRefType | str | None = None, - status: BackendRefStatus | str | None = None, - limit: int = 100, - ) -> list[MemoryRef]: ... - def save_outbox_event(self, event: OutboxEvent) -> OutboxEvent: ... - def list_outbox_events( - self, - status: OutboxEventStatus | str | None = None, - backend_type: BackendType | str | None = None, - operation: BackendOperation | str | None = None, - gateway_id: str | None = None, - payload_ref: str | None = None, - limit: int = 100, - ) -> list[OutboxEvent]: ... - def list_outbox_events_by_job(self, job_id: str, limit: int = 100) -> list[OutboxEvent]: ... - def claim_outbox_event(self, event_id: str) -> OutboxEvent | None: ... - def claim_pending_outbox_events( - self, - limit: int, - worker_id: str, - lease_seconds: int, - ) -> list[OutboxEvent]: ... - def release_expired_processing_events(self, now: datetime | None = None) -> list[OutboxEvent]: ... - def update_outbox_event_status( - self, - event_id: str, - status: OutboxEventStatus | str, - last_error: str | None = None, - ) -> OutboxEvent | None: ... - def save_commit_job(self, job: CommitJob) -> CommitJob: ... - def get_commit_job(self, job_id: str) -> CommitJob | None: ... - def update_commit_job_status( - self, - job_id: str, - status: str, - error_message: str | None = None, - created_refs_count: int | None = None, - ) -> CommitJob | None: ... - def count_memory_refs( - self, - gateway_id: str | None = None, - session_id: str | None = None, - status: BackendRefStatus | str | None = None, - ) -> int: ... - - -def _json_dump_model(model) -> str: - return json.dumps(model.model_dump(mode="json"), ensure_ascii=False) - - -def _json_load_model(model_cls, payload: str): - return model_cls.model_validate(json.loads(payload)) - - -def _enum_value(value): - return value.value if hasattr(value, "value") else value - - -def _safe_timedelta(seconds: int) -> timedelta: - return timedelta(seconds=max(1, int(seconds))) - - -class InMemoryRepository: - def __init__(self) -> None: - self.users: dict[str, UserRecord] = {} - self.memories: dict[str, MemoryRecord] = {} - self.episodes: dict[str, EpisodeRecord] = {} - self.profiles: dict[str, ProfileRecord] = {} - self.audit_logs: list[AuditLog] = [] - self.memory_refs: dict[str, MemoryRef] = {} - self.outbox_events: dict[str, OutboxEvent] = {} - self.commit_jobs: dict[str, CommitJob] = {} - - def create_user(self, user: UserRecord) -> UserRecord: - now = datetime.now(timezone.utc) - user.created_at = now - user.updated_at = now - self.users[user.id] = user - self.profiles.setdefault( - user.id, - ProfileRecord(user_id=user.id, namespace=user.profile_namespace or f"user/{user.id}/profile"), - ) - return user - - def get_user(self, user_id: str) -> Optional[UserRecord]: - return self.users.get(user_id) - - def upsert_memory(self, memory: MemoryRecord) -> MemoryRecord: - now = datetime.now(timezone.utc) - existing = self.memories.get(memory.id) - if existing: - memory.version = existing.version + 1 - memory.created_at = existing.created_at - memory.updated_at = now - self.memories[memory.id] = memory - return memory - - def get_memory(self, memory_id: str) -> Optional[MemoryRecord]: - return self.memories.get(memory_id) - - def delete_memory(self, memory_id: str) -> bool: - return self.memories.pop(memory_id, None) is not None - - def list_memories(self) -> Iterable[MemoryRecord]: - return list(self.memories.values()) - - def append_episode(self, episode: EpisodeRecord) -> EpisodeRecord: - self.episodes[episode.id] = episode - return episode - - def list_session_episodes(self, session_id: str) -> list[EpisodeRecord]: - return [episode for episode in self.episodes.values() if episode.session_id == session_id] - - def get_profile(self, user_id: str) -> Optional[ProfileRecord]: - return self.profiles.get(user_id) - - def upsert_profile(self, profile: ProfileRecord) -> ProfileRecord: - profile.updated_at = datetime.now(timezone.utc) - profile.version += 1 - self.profiles[profile.user_id] = profile - return profile - - def add_audit(self, audit: AuditLog) -> AuditLog: - self.audit_logs.append(audit) - return audit - - def list_audit(self, limit: int = 100) -> list[AuditLog]: - return self.audit_logs[-limit:] - - def save_memory_ref(self, ref: MemoryRef) -> MemoryRef: - now = datetime.now(timezone.utc) - existing = self.memory_refs.get(ref.id) - if existing: - ref.created_at = existing.created_at - ref.updated_at = now - self.memory_refs[ref.id] = ref - return ref - - def get_memory_ref(self, ref_id: str) -> MemoryRef | None: - return self.memory_refs.get(ref_id) - - def list_memory_refs( - self, - gateway_id: str | None = None, - workspace_id: str | None = None, - user_id: str | None = None, - agent_id: str | None = None, - session_id: str | None = None, - namespace: str | None = None, - backend_type: BackendType | str | None = None, - ref_type: MemoryRefType | str | None = None, - status: BackendRefStatus | str | None = None, - limit: int = 100, - ) -> list[MemoryRef]: - refs = list(self.memory_refs.values()) - - def matches(ref: MemoryRef) -> bool: - return ( - (workspace_id is None or ref.workspace_id == workspace_id) - and (gateway_id is None or ref.gateway_id == gateway_id) - and (user_id is None or ref.user_id == user_id) - and (agent_id is None or ref.agent_id == agent_id) - and (session_id is None or ref.session_id == session_id) - and (namespace is None or ref.namespace == namespace) - and (backend_type is None or ref.backend_type.value == _enum_value(backend_type)) - and (ref_type is None or ref.ref_type.value == _enum_value(ref_type)) - and (status is None or ref.status.value == _enum_value(status)) - ) - - refs = [ref for ref in refs if matches(ref)] - refs.sort(key=lambda ref: ref.created_at, reverse=True) - return refs[:limit] - - def save_outbox_event(self, event: OutboxEvent) -> OutboxEvent: - now = datetime.now(timezone.utc) - existing = self.outbox_events.get(event.id) - if existing: - event.created_at = existing.created_at - event.updated_at = now - self.outbox_events[event.id] = event - return event - - def list_outbox_events( - self, - status: OutboxEventStatus | str | None = None, - backend_type: BackendType | str | None = None, - operation: BackendOperation | str | None = None, - gateway_id: str | None = None, - payload_ref: str | None = None, - limit: int = 100, - ) -> list[OutboxEvent]: - events = list(self.outbox_events.values()) - - def matches(event: OutboxEvent) -> bool: - return ( - (status is None or event.status.value == _enum_value(status)) - and (backend_type is None or event.backend_type.value == _enum_value(backend_type)) - and (operation is None or event.operation.value == _enum_value(operation)) - and (gateway_id is None or event.gateway_id == gateway_id) - and (payload_ref is None or event.payload_ref == payload_ref) - ) - - events = [event for event in events if matches(event)] - events.sort(key=lambda event: event.created_at, reverse=True) - return events[:limit] - - def list_outbox_events_by_job(self, job_id: str, limit: int = 100) -> list[OutboxEvent]: - return self.list_outbox_events(payload_ref=f"commit_job:{job_id}", limit=limit) - - def claim_outbox_event(self, event_id: str) -> OutboxEvent | None: - event = self.outbox_events.get(event_id) - now = datetime.now(timezone.utc) - if not event or event.status != OutboxEventStatus.PENDING: - return None - if event.next_retry_at and event.next_retry_at > now: - return None - event.status = OutboxEventStatus.PROCESSING - event.locked_by = "inline" - event.locked_at = now - event.lease_expires_at = now + _safe_timedelta(300) - event.updated_at = now - self.outbox_events[event.id] = event - return event - - def claim_pending_outbox_events( - self, - limit: int, - worker_id: str, - lease_seconds: int, - ) -> list[OutboxEvent]: - now = datetime.now(timezone.utc) - candidates = [ - event - for event in self.outbox_events.values() - if event.status == OutboxEventStatus.PENDING - and (event.next_retry_at is None or event.next_retry_at <= now) - ] - candidates.sort(key=lambda event: event.created_at) - claimed: list[OutboxEvent] = [] - for event in candidates[:limit]: - event.status = OutboxEventStatus.PROCESSING - event.locked_by = worker_id - event.locked_at = now - event.lease_expires_at = now + _safe_timedelta(lease_seconds) - event.updated_at = now - self.outbox_events[event.id] = event - claimed.append(event) - return claimed - - def release_expired_processing_events(self, now: datetime | None = None) -> list[OutboxEvent]: - now = now or datetime.now(timezone.utc) - released: list[OutboxEvent] = [] - for event in list(self.outbox_events.values()): - if ( - event.status == OutboxEventStatus.PROCESSING - and event.lease_expires_at is not None - and event.lease_expires_at <= now - ): - event.status = OutboxEventStatus.PENDING - event.locked_by = None - event.locked_at = None - event.lease_expires_at = None - event.updated_at = now - self.outbox_events[event.id] = event - released.append(event) - return released - - def update_outbox_event_status( - self, - event_id: str, - status: OutboxEventStatus | str, - last_error: str | None = None, - ) -> OutboxEvent | None: - event = self.outbox_events.get(event_id) - if not event: - return None - event.status = OutboxEventStatus(_enum_value(status)) - event.last_error = last_error - event.updated_at = datetime.now(timezone.utc) - if event.status != OutboxEventStatus.PROCESSING: - event.locked_by = None - event.locked_at = None - event.lease_expires_at = None - if event.status in {OutboxEventStatus.FAILED, OutboxEventStatus.DEAD_LETTER}: - event.attempt_count += 1 - self.outbox_events[event.id] = event - return event - - def save_commit_job(self, job: CommitJob) -> CommitJob: - now = datetime.now(timezone.utc) - existing = self.commit_jobs.get(job.job_id) - if existing: - job.created_at = existing.created_at - job.updated_at = now - self.commit_jobs[job.job_id] = job - return job - - def count_memory_refs( - self, - gateway_id: str | None = None, - session_id: str | None = None, - status: BackendRefStatus | str | None = None, - ) -> int: - return len(self.list_memory_refs(gateway_id=gateway_id, session_id=session_id, status=status, limit=100000)) - - def get_commit_job(self, job_id: str) -> CommitJob | None: - return self.commit_jobs.get(job_id) - - def update_commit_job_status( - self, - job_id: str, - status: str, - error_message: str | None = None, - created_refs_count: int | None = None, - ) -> CommitJob | None: - job = self.commit_jobs.get(job_id) - if not job: - return None - from .schemas_v2 import OperationStatus - - job.status = OperationStatus(_enum_value(status)) - job.error_message = error_message - if created_refs_count is not None: - job.created_refs_count = created_refs_count - now = datetime.now(timezone.utc) - job.updated_at = now - if job.status.value == "running" and job.started_at is None: - job.started_at = now - if job.status.value in {"success", "failed", "partial_success"}: - job.finished_at = now - self.commit_jobs[job.job_id] = job - return job - - -class SQLiteRepository: - def __init__(self, db_path: str | Path) -> None: - self.db_path = Path(db_path) - self.db_path.parent.mkdir(parents=True, exist_ok=True) - self._init_schema() - - def _connect(self) -> sqlite3.Connection: - conn = sqlite3.connect(self.db_path) - conn.row_factory = sqlite3.Row - return conn - - def _init_schema(self) -> None: - with self._connect() as conn: - conn.executescript( - """ - CREATE TABLE IF NOT EXISTS users ( - id TEXT PRIMARY KEY, - payload TEXT NOT NULL, - updated_at TEXT NOT NULL - ); - CREATE TABLE IF NOT EXISTS profiles ( - user_id TEXT PRIMARY KEY, - payload TEXT NOT NULL, - updated_at TEXT NOT NULL - ); - CREATE TABLE IF NOT EXISTS memories ( - id TEXT PRIMARY KEY, - user_id TEXT NOT NULL, - agent_id TEXT, - workspace_id TEXT, - session_id TEXT, - namespace TEXT NOT NULL, - memory_type TEXT NOT NULL, - visibility TEXT NOT NULL, - importance REAL NOT NULL, - confidence REAL NOT NULL, - expires_at TEXT, - archived_at TEXT, - payload TEXT NOT NULL, - updated_at TEXT NOT NULL - ); - CREATE INDEX IF NOT EXISTS idx_memories_user ON memories(user_id); - CREATE INDEX IF NOT EXISTS idx_memories_namespace ON memories(namespace); - CREATE INDEX IF NOT EXISTS idx_memories_workspace ON memories(workspace_id); - CREATE TABLE IF NOT EXISTS episodes ( - id TEXT PRIMARY KEY, - user_id TEXT NOT NULL, - agent_id TEXT, - workspace_id TEXT, - session_id TEXT NOT NULL, - namespace TEXT NOT NULL, - payload TEXT NOT NULL, - created_at TEXT NOT NULL - ); - CREATE INDEX IF NOT EXISTS idx_episodes_session ON episodes(session_id); - CREATE TABLE IF NOT EXISTS audit_logs ( - id TEXT PRIMARY KEY, - actor_user_id TEXT, - actor_agent_id TEXT, - action TEXT NOT NULL, - target_type TEXT NOT NULL, - target_id TEXT, - namespace TEXT, - decision TEXT NOT NULL, - payload TEXT NOT NULL, - created_at TEXT NOT NULL - ); - CREATE INDEX IF NOT EXISTS idx_audit_created ON audit_logs(created_at); - CREATE TABLE IF NOT EXISTS memory_refs ( - id TEXT PRIMARY KEY, - gateway_id TEXT NOT NULL, - workspace_id TEXT NOT NULL, - user_id TEXT NOT NULL, - agent_id TEXT, - session_id TEXT, - turn_id TEXT, - namespace TEXT, - backend_type TEXT NOT NULL, - ref_type TEXT NOT NULL, - native_id TEXT, - native_uri TEXT, - provenance_id TEXT, - idempotency_key TEXT, - content_hash TEXT, - source_type TEXT, - source_event_id TEXT, - status TEXT NOT NULL, - error_message TEXT, - metadata_json TEXT NOT NULL, - payload TEXT NOT NULL, - created_at TEXT NOT NULL, - updated_at TEXT NOT NULL - ); - CREATE INDEX IF NOT EXISTS idx_memory_refs_gateway ON memory_refs(gateway_id); - CREATE INDEX IF NOT EXISTS idx_memory_refs_scope ON memory_refs(workspace_id, user_id, agent_id, session_id); - CREATE INDEX IF NOT EXISTS idx_memory_refs_backend ON memory_refs(backend_type, ref_type, status); - CREATE INDEX IF NOT EXISTS idx_memory_refs_namespace ON memory_refs(namespace); - CREATE TABLE IF NOT EXISTS outbox_events ( - id TEXT PRIMARY KEY, - event_type TEXT NOT NULL, - gateway_id TEXT NOT NULL, - workspace_id TEXT NOT NULL, - user_id TEXT NOT NULL, - agent_id TEXT, - session_id TEXT, - backend_type TEXT NOT NULL, - operation TEXT NOT NULL, - payload_ref TEXT, - status TEXT NOT NULL, - attempt_count INTEGER NOT NULL, - max_attempts INTEGER NOT NULL, - next_retry_at TEXT, - last_error TEXT, - locked_by TEXT, - locked_at TEXT, - lease_expires_at TEXT, - metadata_json TEXT NOT NULL, - payload TEXT NOT NULL, - created_at TEXT NOT NULL, - updated_at TEXT NOT NULL - ); - CREATE INDEX IF NOT EXISTS idx_outbox_events_status ON outbox_events(status, next_retry_at); - CREATE INDEX IF NOT EXISTS idx_outbox_events_backend ON outbox_events(backend_type, operation); - CREATE INDEX IF NOT EXISTS idx_outbox_events_gateway ON outbox_events(gateway_id); - CREATE TABLE IF NOT EXISTS commit_jobs ( - job_id TEXT PRIMARY KEY, - workspace_id TEXT NOT NULL, - user_id TEXT NOT NULL, - agent_id TEXT, - session_id TEXT NOT NULL, - namespace TEXT, - status TEXT NOT NULL, - requested_by TEXT, - created_refs_count INTEGER NOT NULL, - error_message TEXT, - payload TEXT NOT NULL, - created_at TEXT NOT NULL, - updated_at TEXT NOT NULL, - started_at TEXT, - finished_at TEXT - ); - CREATE INDEX IF NOT EXISTS idx_commit_jobs_session ON commit_jobs(session_id); - CREATE INDEX IF NOT EXISTS idx_commit_jobs_status ON commit_jobs(status); - """ - ) - self._ensure_memory_ref_columns(conn) - self._ensure_outbox_event_columns(conn) - conn.execute( - """ - DELETE FROM memory_refs - WHERE rowid NOT IN ( - SELECT MAX(rowid) - FROM memory_refs - GROUP BY gateway_id, backend_type, ref_type - ) - """ - ) - conn.execute( - """ - DROP INDEX IF EXISTS uq_memory_refs_gateway_backend_ref_type - """ - ) - - def _ensure_memory_ref_columns(self, conn: sqlite3.Connection) -> None: - columns = {row["name"] for row in conn.execute("PRAGMA table_info(memory_refs)").fetchall()} - additions = { - "idempotency_key": "TEXT", - "content_hash": "TEXT", - } - for column, column_type in additions.items(): - if column not in columns: - conn.execute(f"ALTER TABLE memory_refs ADD COLUMN {column} {column_type}") - - def _ensure_outbox_event_columns(self, conn: sqlite3.Connection) -> None: - columns = {row["name"] for row in conn.execute("PRAGMA table_info(outbox_events)").fetchall()} - additions = { - "locked_by": "TEXT", - "locked_at": "TEXT", - "lease_expires_at": "TEXT", - } - for column, column_type in additions.items(): - if column not in columns: - conn.execute(f"ALTER TABLE outbox_events ADD COLUMN {column} {column_type}") - - def create_user(self, user: UserRecord) -> UserRecord: - now = datetime.now(timezone.utc) - user.created_at = user.created_at or now - user.updated_at = now - with self._connect() as conn: - conn.execute( - "INSERT OR REPLACE INTO users(id, payload, updated_at) VALUES (?, ?, ?)", - (user.id, _json_dump_model(user), user.updated_at.isoformat()), - ) - self.upsert_profile(ProfileRecord(user_id=user.id, namespace=user.profile_namespace or f"user/{user.id}/profile")) - return user - - def get_user(self, user_id: str) -> Optional[UserRecord]: - with self._connect() as conn: - row = conn.execute("SELECT payload FROM users WHERE id = ?", (user_id,)).fetchone() - return _json_load_model(UserRecord, row["payload"]) if row else None - - def upsert_memory(self, memory: MemoryRecord) -> MemoryRecord: - existing = self.get_memory(memory.id) - now = datetime.now(timezone.utc) - if existing: - memory.version = existing.version + 1 - memory.created_at = existing.created_at - memory.updated_at = now - with self._connect() as conn: - conn.execute( - """ - INSERT OR REPLACE INTO memories( - id, user_id, agent_id, workspace_id, session_id, namespace, - memory_type, visibility, importance, confidence, expires_at, - archived_at, payload, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - memory.id, - memory.user_id, - memory.agent_id, - memory.workspace_id, - memory.session_id, - memory.namespace, - memory.memory_type.value, - memory.visibility.value, - memory.importance, - memory.confidence, - memory.expires_at.isoformat() if memory.expires_at else None, - memory.archived_at.isoformat() if memory.archived_at else None, - _json_dump_model(memory), - memory.updated_at.isoformat(), - ), - ) - return memory - - def get_memory(self, memory_id: str) -> Optional[MemoryRecord]: - with self._connect() as conn: - row = conn.execute("SELECT payload FROM memories WHERE id = ?", (memory_id,)).fetchone() - return _json_load_model(MemoryRecord, row["payload"]) if row else None - - def delete_memory(self, memory_id: str) -> bool: - with self._connect() as conn: - cursor = conn.execute("DELETE FROM memories WHERE id = ?", (memory_id,)) - return cursor.rowcount > 0 - - def list_memories(self) -> Iterable[MemoryRecord]: - with self._connect() as conn: - rows = conn.execute("SELECT payload FROM memories").fetchall() - return [_json_load_model(MemoryRecord, row["payload"]) for row in rows] - - def append_episode(self, episode: EpisodeRecord) -> EpisodeRecord: - with self._connect() as conn: - conn.execute( - """ - INSERT OR REPLACE INTO episodes( - id, user_id, agent_id, workspace_id, session_id, namespace, payload, created_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - episode.id, - episode.user_id, - episode.agent_id, - episode.workspace_id, - episode.session_id, - episode.namespace, - _json_dump_model(episode), - episode.created_at.isoformat(), - ), - ) - return episode - - def list_session_episodes(self, session_id: str) -> list[EpisodeRecord]: - with self._connect() as conn: - rows = conn.execute( - "SELECT payload FROM episodes WHERE session_id = ? ORDER BY created_at ASC", - (session_id,), - ).fetchall() - return [_json_load_model(EpisodeRecord, row["payload"]) for row in rows] - - def get_profile(self, user_id: str) -> Optional[ProfileRecord]: - with self._connect() as conn: - row = conn.execute("SELECT payload FROM profiles WHERE user_id = ?", (user_id,)).fetchone() - return _json_load_model(ProfileRecord, row["payload"]) if row else None - - def upsert_profile(self, profile: ProfileRecord) -> ProfileRecord: - profile.updated_at = datetime.now(timezone.utc) - with self._connect() as conn: - conn.execute( - "INSERT OR REPLACE INTO profiles(user_id, payload, updated_at) VALUES (?, ?, ?)", - (profile.user_id, _json_dump_model(profile), profile.updated_at.isoformat()), - ) - return profile - - def add_audit(self, audit: AuditLog) -> AuditLog: - with self._connect() as conn: - conn.execute( - """ - INSERT OR REPLACE INTO audit_logs( - id, actor_user_id, actor_agent_id, action, target_type, target_id, - namespace, decision, payload, created_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - audit.id, - audit.actor_user_id, - audit.actor_agent_id, - audit.action, - audit.target_type, - audit.target_id, - audit.namespace, - audit.decision, - _json_dump_model(audit), - audit.created_at.isoformat(), - ), - ) - return audit - - def list_audit(self, limit: int = 100) -> list[AuditLog]: - with self._connect() as conn: - rows = conn.execute( - "SELECT payload FROM audit_logs ORDER BY created_at DESC LIMIT ?", - (limit,), - ).fetchall() - return [_json_load_model(AuditLog, row["payload"]) for row in rows] - - def save_memory_ref(self, ref: MemoryRef) -> MemoryRef: - existing = None - with self._connect() as conn: - row = conn.execute("SELECT payload FROM memory_refs WHERE id = ?", (ref.id,)).fetchone() - if row: - existing = _json_load_model(MemoryRef, row["payload"]) - now = datetime.now(timezone.utc) - if existing: - ref.created_at = existing.created_at - ref.updated_at = now - with self._connect() as conn: - conn.execute( - """ - INSERT OR REPLACE INTO memory_refs( - id, gateway_id, workspace_id, user_id, agent_id, session_id, turn_id, - namespace, backend_type, ref_type, native_id, native_uri, provenance_id, - idempotency_key, content_hash, source_type, source_event_id, status, error_message, metadata_json, - payload, created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - ref.id, - ref.gateway_id, - ref.workspace_id, - ref.user_id, - ref.agent_id, - ref.session_id, - ref.turn_id, - ref.namespace, - ref.backend_type.value, - ref.ref_type.value, - ref.native_id, - ref.native_uri, - ref.provenance_id, - ref.idempotency_key, - ref.content_hash, - ref.source_type, - ref.source_event_id, - ref.status.value, - ref.error_message, - json.dumps(ref.metadata, ensure_ascii=False), - _json_dump_model(ref), - ref.created_at.isoformat(), - ref.updated_at.isoformat(), - ), - ) - return ref - - def get_memory_ref(self, ref_id: str) -> MemoryRef | None: - with self._connect() as conn: - row = conn.execute("SELECT payload FROM memory_refs WHERE id = ?", (ref_id,)).fetchone() - return _json_load_model(MemoryRef, row["payload"]) if row else None - - def list_memory_refs( - self, - gateway_id: str | None = None, - workspace_id: str | None = None, - user_id: str | None = None, - agent_id: str | None = None, - session_id: str | None = None, - namespace: str | None = None, - backend_type: BackendType | str | None = None, - ref_type: MemoryRefType | str | None = None, - status: BackendRefStatus | str | None = None, - limit: int = 100, - ) -> list[MemoryRef]: - clauses: list[str] = [] - params: list[str | int] = [] - filters = { - "gateway_id": gateway_id, - "workspace_id": workspace_id, - "user_id": user_id, - "agent_id": agent_id, - "session_id": session_id, - "namespace": namespace, - "backend_type": _enum_value(backend_type) if backend_type is not None else None, - "ref_type": _enum_value(ref_type) if ref_type is not None else None, - "status": _enum_value(status) if status is not None else None, - } - for key, value in filters.items(): - if value is not None: - clauses.append(f"{key} = ?") - params.append(value) - where = f"WHERE {' AND '.join(clauses)}" if clauses else "" - params.append(limit) - with self._connect() as conn: - rows = conn.execute( - f"SELECT payload FROM memory_refs {where} ORDER BY created_at DESC LIMIT ?", - params, - ).fetchall() - return [_json_load_model(MemoryRef, row["payload"]) for row in rows] - - def save_outbox_event(self, event: OutboxEvent) -> OutboxEvent: - existing = None - with self._connect() as conn: - row = conn.execute("SELECT payload FROM outbox_events WHERE id = ?", (event.id,)).fetchone() - if row: - existing = _json_load_model(OutboxEvent, row["payload"]) - now = datetime.now(timezone.utc) - if existing: - event.created_at = existing.created_at - event.updated_at = now - with self._connect() as conn: - conn.execute( - """ - INSERT OR REPLACE INTO outbox_events( - id, event_type, gateway_id, workspace_id, user_id, agent_id, session_id, - backend_type, operation, payload_ref, status, attempt_count, max_attempts, - next_retry_at, last_error, locked_by, locked_at, lease_expires_at, - metadata_json, payload, created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - event.id, - event.event_type, - event.gateway_id, - event.workspace_id, - event.user_id, - event.agent_id, - event.session_id, - event.backend_type.value, - event.operation.value, - event.payload_ref, - event.status.value, - event.attempt_count, - event.max_attempts, - event.next_retry_at.isoformat() if event.next_retry_at else None, - event.last_error, - event.locked_by, - event.locked_at.isoformat() if event.locked_at else None, - event.lease_expires_at.isoformat() if event.lease_expires_at else None, - json.dumps(event.metadata, ensure_ascii=False), - _json_dump_model(event), - event.created_at.isoformat(), - event.updated_at.isoformat(), - ), - ) - return event - - def list_outbox_events( - self, - status: OutboxEventStatus | str | None = None, - backend_type: BackendType | str | None = None, - operation: BackendOperation | str | None = None, - gateway_id: str | None = None, - payload_ref: str | None = None, - limit: int = 100, - ) -> list[OutboxEvent]: - clauses: list[str] = [] - params: list[str | int] = [] - filters = { - "status": _enum_value(status) if status is not None else None, - "backend_type": _enum_value(backend_type) if backend_type is not None else None, - "operation": _enum_value(operation) if operation is not None else None, - "gateway_id": gateway_id, - "payload_ref": payload_ref, - } - for key, value in filters.items(): - if value is not None: - clauses.append(f"{key} = ?") - params.append(value) - where = f"WHERE {' AND '.join(clauses)}" if clauses else "" - params.append(limit) - with self._connect() as conn: - rows = conn.execute( - f"SELECT payload FROM outbox_events {where} ORDER BY created_at DESC LIMIT ?", - params, - ).fetchall() - return [_json_load_model(OutboxEvent, row["payload"]) for row in rows] - - def list_outbox_events_by_job(self, job_id: str, limit: int = 100) -> list[OutboxEvent]: - return self.list_outbox_events(payload_ref=f"commit_job:{job_id}", limit=limit) - - def claim_outbox_event(self, event_id: str) -> OutboxEvent | None: - with self._connect() as conn: - row = conn.execute("SELECT payload FROM outbox_events WHERE id = ?", (event_id,)).fetchone() - if not row: - return None - event = _json_load_model(OutboxEvent, row["payload"]) - now = datetime.now(timezone.utc) - if event.status != OutboxEventStatus.PENDING: - return None - if event.next_retry_at and event.next_retry_at > now: - return None - event.status = OutboxEventStatus.PROCESSING - event.locked_by = "inline" - event.locked_at = now - event.lease_expires_at = now + _safe_timedelta(300) - event.updated_at = now - with self._connect() as conn: - cursor = conn.execute( - """ - UPDATE outbox_events - SET status = ?, locked_by = ?, locked_at = ?, lease_expires_at = ?, - payload = ?, metadata_json = ?, updated_at = ? - WHERE id = ? - AND status = ? - AND (next_retry_at IS NULL OR next_retry_at <= ?) - """, - ( - event.status.value, - event.locked_by, - event.locked_at.isoformat() if event.locked_at else None, - event.lease_expires_at.isoformat() if event.lease_expires_at else None, - _json_dump_model(event), - json.dumps(event.metadata, ensure_ascii=False), - event.updated_at.isoformat(), - event.id, - OutboxEventStatus.PENDING.value, - now.isoformat(), - ), - ) - return event if cursor.rowcount else None - - def claim_pending_outbox_events( - self, - limit: int, - worker_id: str, - lease_seconds: int, - ) -> list[OutboxEvent]: - now = datetime.now(timezone.utc) - now_iso = now.isoformat() - with self._connect() as conn: - rows = conn.execute( - """ - SELECT payload FROM outbox_events - WHERE status = ? - AND (next_retry_at IS NULL OR next_retry_at <= ?) - ORDER BY created_at ASC - LIMIT ? - """, - (OutboxEventStatus.PENDING.value, now_iso, limit), - ).fetchall() - claimed: list[OutboxEvent] = [] - with self._connect() as conn: - for row in rows: - event = _json_load_model(OutboxEvent, row["payload"]) - if event.status != OutboxEventStatus.PENDING: - continue - event.status = OutboxEventStatus.PROCESSING - event.locked_by = worker_id - event.locked_at = now - event.lease_expires_at = now + _safe_timedelta(lease_seconds) - event.updated_at = now - cursor = conn.execute( - """ - UPDATE outbox_events - SET status = ?, locked_by = ?, locked_at = ?, lease_expires_at = ?, - payload = ?, metadata_json = ?, updated_at = ? - WHERE id = ? - AND status = ? - AND (next_retry_at IS NULL OR next_retry_at <= ?) - """, - ( - event.status.value, - event.locked_by, - event.locked_at.isoformat() if event.locked_at else None, - event.lease_expires_at.isoformat() if event.lease_expires_at else None, - _json_dump_model(event), - json.dumps(event.metadata, ensure_ascii=False), - event.updated_at.isoformat(), - event.id, - OutboxEventStatus.PENDING.value, - now_iso, - ), - ) - if cursor.rowcount: - claimed.append(event) - return claimed - - def release_expired_processing_events(self, now: datetime | None = None) -> list[OutboxEvent]: - now = now or datetime.now(timezone.utc) - with self._connect() as conn: - rows = conn.execute( - """ - SELECT payload FROM outbox_events - WHERE status = ? - AND lease_expires_at IS NOT NULL - AND lease_expires_at <= ? - """, - (OutboxEventStatus.PROCESSING.value, now.isoformat()), - ).fetchall() - released: list[OutboxEvent] = [] - for row in rows: - event = _json_load_model(OutboxEvent, row["payload"]) - event.status = OutboxEventStatus.PENDING - event.locked_by = None - event.locked_at = None - event.lease_expires_at = None - event.updated_at = now - released.append(self.save_outbox_event(event)) - return released - - def update_outbox_event_status( - self, - event_id: str, - status: OutboxEventStatus | str, - last_error: str | None = None, - ) -> OutboxEvent | None: - with self._connect() as conn: - row = conn.execute("SELECT payload FROM outbox_events WHERE id = ?", (event_id,)).fetchone() - if not row: - return None - event = _json_load_model(OutboxEvent, row["payload"]) - event.status = OutboxEventStatus(_enum_value(status)) - event.last_error = last_error - event.updated_at = datetime.now(timezone.utc) - if event.status != OutboxEventStatus.PROCESSING: - event.locked_by = None - event.locked_at = None - event.lease_expires_at = None - if event.status in {OutboxEventStatus.FAILED, OutboxEventStatus.DEAD_LETTER}: - event.attempt_count += 1 - return self.save_outbox_event(event) - - def save_commit_job(self, job: CommitJob) -> CommitJob: - existing = None - with self._connect() as conn: - row = conn.execute("SELECT payload FROM commit_jobs WHERE job_id = ?", (job.job_id,)).fetchone() - if row: - existing = _json_load_model(CommitJob, row["payload"]) - now = datetime.now(timezone.utc) - if existing: - job.created_at = existing.created_at - job.updated_at = now - with self._connect() as conn: - conn.execute( - """ - INSERT OR REPLACE INTO commit_jobs( - job_id, workspace_id, user_id, agent_id, session_id, namespace, - status, requested_by, created_refs_count, error_message, payload, - created_at, updated_at, started_at, finished_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - job.job_id, - job.workspace_id, - job.user_id, - job.agent_id, - job.session_id, - job.namespace, - job.status.value, - job.requested_by, - job.created_refs_count, - job.error_message, - _json_dump_model(job), - job.created_at.isoformat(), - job.updated_at.isoformat(), - job.started_at.isoformat() if job.started_at else None, - job.finished_at.isoformat() if job.finished_at else None, - ), - ) - return job - - def get_commit_job(self, job_id: str) -> CommitJob | None: - with self._connect() as conn: - row = conn.execute("SELECT payload FROM commit_jobs WHERE job_id = ?", (job_id,)).fetchone() - return _json_load_model(CommitJob, row["payload"]) if row else None - - def update_commit_job_status( - self, - job_id: str, - status: str, - error_message: str | None = None, - created_refs_count: int | None = None, - ) -> CommitJob | None: - from .schemas_v2 import OperationStatus - - job = self.get_commit_job(job_id) - if not job: - return None - job.status = OperationStatus(_enum_value(status)) - job.error_message = error_message - if created_refs_count is not None: - job.created_refs_count = created_refs_count - now = datetime.now(timezone.utc) - job.updated_at = now - if job.status.value == "running" and job.started_at is None: - job.started_at = now - if job.status.value in {"success", "failed", "partial_success"}: - job.finished_at = now - return self.save_commit_job(job) - - def count_memory_refs( - self, - gateway_id: str | None = None, - session_id: str | None = None, - status: BackendRefStatus | str | None = None, - ) -> int: - return len(self.list_memory_refs(gateway_id=gateway_id, session_id=session_id, status=status, limit=100000)) - - -def build_repository() -> MetadataRepository: - config = get_config() - if config.storage.backend == "memory": - return InMemoryRepository() - return SQLiteRepository(config.storage.sqlite_path) - - -repository = build_repository() diff --git a/memory_gateway/schemas.py b/memory_gateway/schemas.py deleted file mode 100644 index ab5398a..0000000 --- a/memory_gateway/schemas.py +++ /dev/null @@ -1,226 +0,0 @@ -"""Core schemas for the generic Memory Gateway v1 API.""" -from __future__ import annotations - -from datetime import datetime, timezone -from enum import Enum -from typing import Any, Literal, Optional -from uuid import uuid4 - -from pydantic import BaseModel, Field - - -def utc_now() -> datetime: - return datetime.now(timezone.utc) - - -class Visibility(str, Enum): - PRIVATE = "private" - AGENT_ONLY = "agent-only" - WORKSPACE_SHARED = "workspace-shared" - GLOBAL = "global" - - -class MemoryType(str, Enum): - PROFILE = "profile" - PREFERENCE = "preference" - FACT = "fact" - DECISION = "decision" - SUMMARY = "summary" - EPISODIC = "episodic" - PROCEDURE = "procedure" - EXPERIENCE = "experience" - KNOWLEDGE = "knowledge" - - -class SourceType(str, Enum): - CONVERSATION = "conversation" - TASK = "task" - AGENT = "agent" - OBSIDIAN = "obsidian" - OPENVIKING = "openviking" - EVEROS = "everos" - MANUAL = "manual" - - -class UserRecord(BaseModel): - id: str = Field(default_factory=lambda: f"user_{uuid4().hex[:12]}") - display_name: str - status: Literal["active", "disabled"] = "active" - profile_namespace: Optional[str] = None - preferences: dict[str, Any] = Field(default_factory=dict) - created_at: datetime = Field(default_factory=utc_now) - updated_at: datetime = Field(default_factory=utc_now) - - -class AgentRecord(BaseModel): - id: str - name: str - framework: str - owner_user_id: Optional[str] = None - created_at: datetime = Field(default_factory=utc_now) - - -class WorkspaceRecord(BaseModel): - id: str - name: str - owner_user_id: str - member_user_ids: list[str] = Field(default_factory=list) - allowed_agent_ids: list[str] = Field(default_factory=list) - created_at: datetime = Field(default_factory=utc_now) - - -class SessionRecord(BaseModel): - id: str = Field(default_factory=lambda: f"sess_{uuid4().hex[:12]}") - user_id: str - agent_id: Optional[str] = None - workspace_id: Optional[str] = None - status: Literal["open", "committed", "expired"] = "open" - expires_at: Optional[datetime] = None - created_at: datetime = Field(default_factory=utc_now) - updated_at: datetime = Field(default_factory=utc_now) - - -class ACLRule(BaseModel): - visibility: Visibility = Visibility.PRIVATE - allowed_user_ids: list[str] = Field(default_factory=list) - allowed_agent_ids: list[str] = Field(default_factory=list) - allowed_workspace_ids: list[str] = Field(default_factory=list) - - -class MemoryRecord(BaseModel): - id: str = Field(default_factory=lambda: f"mem_{uuid4().hex[:16]}") - user_id: str - agent_id: Optional[str] = None - workspace_id: Optional[str] = None - session_id: Optional[str] = None - namespace: str - memory_type: MemoryType = MemoryType.FACT - content: str - summary: Optional[str] = None - tags: list[str] = Field(default_factory=list) - importance: float = Field(default=0.5, ge=0, le=1) - confidence: float = Field(default=0.8, ge=0, le=1) - visibility: Visibility = Visibility.PRIVATE - acl: ACLRule = Field(default_factory=ACLRule) - source: SourceType = SourceType.MANUAL - source_ref: Optional[str] = None - embedding_ref: Optional[str] = None - created_at: datetime = Field(default_factory=utc_now) - updated_at: datetime = Field(default_factory=utc_now) - expires_at: Optional[datetime] = None - archived_at: Optional[datetime] = None - version: int = 1 - - -class EpisodeRecord(BaseModel): - id: str = Field(default_factory=lambda: f"epi_{uuid4().hex[:16]}") - user_id: str - agent_id: Optional[str] = None - workspace_id: Optional[str] = None - session_id: str - namespace: str - content: str - summary: Optional[str] = None - events: list[dict[str, Any]] = Field(default_factory=list) - tags: list[str] = Field(default_factory=list) - source: SourceType = SourceType.CONVERSATION - created_at: datetime = Field(default_factory=utc_now) - expires_at: Optional[datetime] = None - - -class ProfileRecord(BaseModel): - id: str = Field(default_factory=lambda: f"profile_{uuid4().hex[:12]}") - user_id: str - namespace: str - display_name: Optional[str] = None - stable_facts: list[str] = Field(default_factory=list) - preferences: dict[str, Any] = Field(default_factory=dict) - working_style: list[str] = Field(default_factory=list) - updated_from_memory_ids: list[str] = Field(default_factory=list) - version: int = 1 - updated_at: datetime = Field(default_factory=utc_now) - - -class AuditLog(BaseModel): - id: str = Field(default_factory=lambda: f"audit_{uuid4().hex[:16]}") - actor_user_id: Optional[str] = None - actor_agent_id: Optional[str] = None - action: str - target_type: str - target_id: Optional[str] = None - namespace: Optional[str] = None - decision: Literal["allow", "deny"] = "allow" - reason: Optional[str] = None - metadata: dict[str, Any] = Field(default_factory=dict) - created_at: datetime = Field(default_factory=utc_now) - - -class AccessContext(BaseModel): - user_id: str - agent_id: Optional[str] = None - workspace_id: Optional[str] = None - session_id: Optional[str] = None - - -class CreateUserRequest(BaseModel): - display_name: str - user_id: Optional[str] = None - preferences: dict[str, Any] = Field(default_factory=dict) - - -class MemorySearchRequest(AccessContext): - query: str - namespaces: list[str] = Field(default_factory=list) - memory_types: list[MemoryType] = Field(default_factory=list) - tags: list[str] = Field(default_factory=list) - limit: int = Field(default=10, ge=1, le=100) - - -class MemoryUpsertRequest(AccessContext): - namespace: Optional[str] = None - memory_type: MemoryType = MemoryType.FACT - content: str - summary: Optional[str] = None - tags: list[str] = Field(default_factory=list) - importance: float = Field(default=0.5, ge=0, le=1) - confidence: float = Field(default=0.8, ge=0, le=1) - visibility: Visibility = Visibility.PRIVATE - source: SourceType = SourceType.MANUAL - expires_at: Optional[datetime] = None - - -class MemoryPatchRequest(BaseModel): - content: Optional[str] = None - summary: Optional[str] = None - tags: Optional[list[str]] = None - importance: Optional[float] = Field(default=None, ge=0, le=1) - confidence: Optional[float] = Field(default=None, ge=0, le=1) - visibility: Optional[Visibility] = None - expires_at: Optional[datetime] = None - - -class EpisodeAppendRequest(AccessContext): - content: str - namespace: Optional[str] = None - events: list[dict[str, Any]] = Field(default_factory=list) - tags: list[str] = Field(default_factory=list) - source: SourceType = SourceType.CONVERSATION - expires_at: Optional[datetime] = None - - -class CommitSessionRequest(AccessContext): - promote: bool = True - min_importance: float = Field(default=0.6, ge=0, le=1) - target_namespace: Optional[str] = None - - -class MemoryFeedbackRequest(AccessContext): - feedback: Literal["useful", "not_useful", "incorrect", "duplicate", "outdated"] - comment: Optional[str] = None - - -class NamespaceInfo(BaseModel): - namespace: str - owner_user_id: Optional[str] = None - visibility: Visibility - description: str diff --git a/memory_gateway/schemas_v2.py b/memory_gateway/schemas_v2.py deleted file mode 100644 index d078c50..0000000 --- a/memory_gateway/schemas_v2.py +++ /dev/null @@ -1,228 +0,0 @@ -"""Schemas for the Memory Gateway v2 control-plane API.""" -from __future__ import annotations - -from datetime import datetime -from enum import Enum -from typing import Any, Literal, Optional -from uuid import uuid4 - -from pydantic import BaseModel, Field - -from .schemas import utc_now - - -class OperationStatus(str, Enum): - ACCEPTED = "accepted" - RUNNING = "running" - SUCCESS = "success" - PARTIAL_SUCCESS = "partial_success" - FAILED = "failed" - PENDING = "pending" - SKIPPED = "skipped" - - -class BackendRefStatus(str, Enum): - PENDING = "pending" - SUCCESS = "success" - FAILED = "failed" - SKIPPED = "skipped" - - -class BackendType(str, Enum): - OPENVIKING = "openviking" - EVEROS = "everos" - OBSIDIAN = "obsidian" - - -class MemoryRefType(str, Enum): - SESSION_ARCHIVE = "session_archive" - CONTEXT_RESOURCE = "context_resource" - MESSAGE_MEMORY = "message_memory" - EPISODIC_MEMORY = "episodic_memory" - PROFILE = "profile" - LONG_TERM_MEMORY = "long_term_memory" - DRAFT_REVIEW = "draft_review" - - -class TraceContext(BaseModel): - trace_id: Optional[str] = None - span_id: Optional[str] = None - parent_span_id: Optional[str] = None - request_id: Optional[str] = None - metadata: dict[str, Any] = Field(default_factory=dict) - - -class IngestPolicy(BaseModel): - allow_openviking: bool = True - allow_everos: bool = True - allow_obsidian_review: bool = False - redact_sensitive: bool = True - require_human_review: bool = False - metadata: dict[str, Any] = Field(default_factory=dict) - - -class IngestRequest(BaseModel): - workspace_id: str - user_id: str - agent_id: str - session_id: str - turn_id: str - request_id: Optional[str] = None - idempotency_key: Optional[str] = None - namespace: str - source_type: str = "conversation" - source_event_id: Optional[str] = None - role: Literal["system", "user", "assistant", "tool", "agent"] = "user" - content: str - policy: IngestPolicy = Field(default_factory=IngestPolicy) - trace: TraceContext = Field(default_factory=TraceContext) - metadata: dict[str, Any] = Field(default_factory=dict) - - -class MemoryRef(BaseModel): - id: str = Field(default_factory=lambda: f"ref_{uuid4().hex[:16]}") - gateway_id: str - workspace_id: str - user_id: str - agent_id: Optional[str] = None - session_id: Optional[str] = None - turn_id: Optional[str] = None - namespace: Optional[str] = None - backend_type: BackendType - ref_type: MemoryRefType - native_id: Optional[str] = None - native_uri: Optional[str] = None - provenance_id: Optional[str] = None - idempotency_key: Optional[str] = None - content_hash: Optional[str] = None - source_type: Optional[str] = None - source_event_id: Optional[str] = None - status: BackendRefStatus = BackendRefStatus.PENDING - error_message: Optional[str] = None - metadata: dict[str, Any] = Field(default_factory=dict) - created_at: datetime = Field(default_factory=utc_now) - updated_at: datetime = Field(default_factory=utc_now) - - -class MemoryRefView(MemoryRef): - pass - - -class IngestResponse(BaseModel): - status: OperationStatus - gateway_id: str - provenance_id: str - request_id: Optional[str] = None - turn_id: str - refs: list[MemoryRefView] = Field(default_factory=list) - errors: list[str] = Field(default_factory=list) - metadata: dict[str, Any] = Field(default_factory=dict) - - -class CommitRequest(BaseModel): - workspace_id: str - user_id: str - agent_id: Optional[str] = None - namespace: Optional[str] = None - request_id: Optional[str] = None - idempotency_key: Optional[str] = None - policy: IngestPolicy = Field(default_factory=IngestPolicy) - metadata: dict[str, Any] = Field(default_factory=dict) - - -class CommitResponse(BaseModel): - status: OperationStatus = OperationStatus.ACCEPTED - job_id: str - session_id: str - message: str = "commit accepted" - refs: list[MemoryRefView] = Field(default_factory=list) - metadata: dict[str, Any] = Field(default_factory=dict) - - -class OutboxSummary(BaseModel): - total_events: int = 0 - pending_events: int = 0 - processing_events: int = 0 - success_events: int = 0 - skipped_events: int = 0 - dead_letter_events: int = 0 - - -class CommitJobView(BaseModel): - job_id: str - workspace_id: str - user_id: str - agent_id: Optional[str] = None - session_id: str - namespace: Optional[str] = None - status: OperationStatus - created_refs_count: int = 0 - error_message: Optional[str] = None - created_at: datetime - updated_at: datetime - started_at: Optional[datetime] = None - finished_at: Optional[datetime] = None - outbox_summary: OutboxSummary = Field(default_factory=OutboxSummary) - - -class OutboxProcessResponse(BaseModel): - status: OperationStatus - worker_id: str - processed_count: int = 0 - outbox_summary: OutboxSummary = Field(default_factory=OutboxSummary) - - -class RetrieveRequest(BaseModel): - workspace_id: str - user_id: str - agent_id: Optional[str] = None - session_id: Optional[str] = None - namespace: Optional[str] = None - query: str - limit: int = Field(default=10, ge=1, le=100) - metadata: dict[str, Any] = Field(default_factory=dict) - - -class ContextItem(BaseModel): - text: Optional[str] = None - source_backend: BackendType - ref_id: Optional[str] = None - score: float = 0.0 - memory_type: Optional[str] = None - metadata: dict[str, Any] = Field(default_factory=dict) - - -class ContextConflict(BaseModel): - ref_ids: list[str] = Field(default_factory=list) - reason: str - metadata: dict[str, Any] = Field(default_factory=dict) - - -class RetrieveResponse(BaseModel): - status: OperationStatus - items: list[ContextItem] = Field(default_factory=list) - refs: list[MemoryRefView] = Field(default_factory=list) - conflicts: list[ContextConflict] = Field(default_factory=list) - trace_id: Optional[str] = None - metadata: dict[str, Any] = Field(default_factory=dict) - - -class FeedbackRequest(BaseModel): - workspace_id: str - user_id: str - agent_id: Optional[str] = None - session_id: Optional[str] = None - namespace: Optional[str] = None - memory_ref_id: Optional[str] = None - feedback_type: Literal["useful", "not_useful", "incorrect", "duplicate", "outdated", "review_approved", "review_rejected"] - comment: Optional[str] = None - source_type: str = "manual" - source_event_id: Optional[str] = None - metadata: dict[str, Any] = Field(default_factory=dict) - - -class FeedbackResponse(BaseModel): - status: OperationStatus - feedback_id: str - memory_ref_id: Optional[str] = None - metadata: dict[str, Any] = Field(default_factory=dict) diff --git a/memory_gateway/server.py b/memory_gateway/server.py deleted file mode 100644 index a1c4f38..0000000 --- a/memory_gateway/server.py +++ /dev/null @@ -1,786 +0,0 @@ -"""Memory Gateway MCP Server. - -通用 Memory Gateway 服务，为 AI agent / harness 提供统一的 OpenViking 记忆检索、总结和知识沉淀入口。 -""" -import asyncio -import hashlib -import json -import logging -import re -import tempfile -from datetime import datetime, timezone -from contextlib import asynccontextmanager -from pathlib import Path -from typing import Any, Optional - -from fastapi import APIRouter, Depends, FastAPI, File, Form, Header, HTTPException, Request, UploadFile, status -from fastapi.responses import JSONResponse -from fastapi.middleware.cors import CORSMiddleware -from mcp.server import Server -from mcp.types import TextContent, Tool -from sse_starlette import EventSourceResponse - -from .config import get_config, set_config, Config -from .openviking_client import get_openviking_client, close_openviking_client -from .document_ingest import convert_file_to_markdown, save_markdown_to_obsidian, slugify -from .llm import LLMConfigurationError, LLMSummaryError, summarize_with_llm -from .mcp_tools_v1 import MEMORY_GATEWAY_MCP_TOOLS -from .schemas import ( - AccessContext, - CommitSessionRequest, - EpisodeAppendRequest, - MemoryFeedbackRequest, - MemorySearchRequest, - MemoryUpsertRequest, -) -from .services import service as v1_service -from .types import SearchRequest, AddMemoryRequest, AddResourceRequest, CommitSummaryRequest - -# 配置日志 -logging.basicConfig( - level=logging.INFO, - format="%(asctime)s - %(name)s - %(levelname)s - %(message)s" -) -logger = logging.getLogger(__name__) - - -# 创建 MCP Server -mcp_server = Server("memory-gateway") - - -@mcp_server.list_tools() -async def list_tools() -> list[Tool]: - """列出可用的 MCP 工具""" - legacy_tools = [ - Tool( - name="search", - description="语义搜索记忆和资源", - inputSchema={ - "type": "object", - "properties": { - "query": {"type": "string", "description": "搜索查询"}, - "namespace": {"type": "string", "description": "命名空间（可选）"}, - "limit": {"type": "integer", "description": "返回结果数量（默认10）"}, - "uri": {"type": "string", "description": "资源 URI（可选）"}, - }, - "required": ["query"], - }, - ), - Tool( - name="add_memory", - description="添加新记忆", - inputSchema={ - "type": "object", - "properties": { - "content": {"type": "string", "description": "记忆内容"}, - "namespace": {"type": "string", "description": "命名空间（可选）"}, - "memory_type": {"type": "string", "description": "记忆类型（默认general）"}, - }, - "required": ["content"], - }, - ), - Tool( - name="add_resource", - description="添加资源", - inputSchema={ - "type": "object", - "properties": { - "uri": {"type": "string", "description": "资源 URI"}, - "content": {"type": "string", "description": "资源内容"}, - "resource_type": {"type": "string", "description": "资源类型（默认text）"}, - }, - "required": ["uri", "content"], - }, - ), - Tool( - name="commit_summary", - description="总结一段通用内容并按需沉淀为 OpenViking memory/resource", - inputSchema={ - "type": "object", - "properties": { - "content": {"type": "string", "description": "需要总结和沉淀的原文内容"}, - "title": {"type": "string", "description": "标题（可选）"}, - "summary": {"type": "string", "description": "人工提供的摘要（可选）"}, - "namespace": {"type": "string", "description": "OpenViking memory namespace（可选）"}, - "memory_type": {"type": "string", "description": "记忆类型，默认 summary"}, - "tags": {"type": "array", "items": {"type": "string"}, "description": "标签列表"}, - "source": {"type": "string", "description": "来源说明或外部链接"}, - "resource_uri": {"type": "string", "description": "写入 resource 的 URI（可选）"}, - "resource_type": {"type": "string", "description": "资源类型，默认 json"}, - "persist_as": {"type": "string", "enum": ["memory", "resource", "both", "none"], "description": "沉淀方式"}, - "max_summary_chars": {"type": "integer", "description": "摘要最大长度"}, - }, - "required": ["content"], - }, - ), - Tool( - name="get_status", - description="检查系统状态", - inputSchema={ - "type": "object", - "properties": {}, - }, - ), - Tool( - name="list_memories", - description="列出已存储的记忆", - inputSchema={ - "type": "object", - "properties": { - "namespace": {"type": "string", "description": "命名空间（可选）"}, - "memory_type": {"type": "string", "description": "记忆类型（可选）"}, - "limit": {"type": "integer", "description": "返回数量（默认10）"}, - }, - }, - ), - Tool( - name="list_resources", - description="列出已存储的资源", - inputSchema={ - "type": "object", - "properties": { - "namespace": {"type": "string", "description": "命名空间（可选）"}, - "limit": {"type": "integer", "description": "返回数量（默认10）"}, - }, - }, - ), - ] - v1_tools = [ - Tool( - name=definition["name"], - description=definition["description"], - inputSchema=definition["inputSchema"], - ) - for definition in MEMORY_GATEWAY_MCP_TOOLS - ] - return legacy_tools + v1_tools - - -@mcp_server.call_tool() -async def call_tool(name: str, arguments: Any) -> list[TextContent]: - """调用 MCP 工具""" - try: - if name.startswith("memory_"): - result = await call_v1_memory_tool(name, arguments or {}) - return [TextContent(type="text", text=json.dumps(result, ensure_ascii=False, default=str))] - - ov_client = await get_openviking_client() - - if name == "search": - result = await ov_client.search( - query=arguments.get("query"), - namespace=arguments.get("namespace"), - limit=arguments.get("limit"), - uri=arguments.get("uri"), - ) - return [TextContent(type="text", text=str(result.results))] - - elif name == "add_memory": - result = await ov_client.add_memory( - content=arguments.get("content"), - namespace=arguments.get("namespace"), - memory_type=arguments.get("memory_type", "general"), - ) - return [TextContent(type="text", text=str(result))] - - elif name == "add_resource": - result = await ov_client.add_resource( - uri=arguments.get("uri"), - content=arguments.get("content"), - resource_type=arguments.get("resource_type", "text"), - ) - return [TextContent(type="text", text=str(result))] - - elif name == "commit_summary": - request = CommitSummaryRequest(**arguments) - result = await commit_summary_to_openviking(request) - return [TextContent(type="text", text=json.dumps(result, ensure_ascii=False))] - - elif name == "get_status": - ov_status = await ov_client.health_check() - return [TextContent(type="text", text=f"Memory Gateway: OK\nOpenViking: {ov_status}")] - - elif name == "list_memories": - memories = await ov_client.list_memories( - namespace=arguments.get("namespace"), - memory_type=arguments.get("memory_type"), - limit=arguments.get("limit"), - ) - return [TextContent(type="text", text=str([m.model_dump() for m in memories]))] - - elif name == "list_resources": - resources = await ov_client.list_resources( - namespace=arguments.get("namespace"), - limit=arguments.get("limit"), - ) - return [TextContent(type="text", text=str([r.model_dump() for r in resources]))] - - else: - raise ValueError(f"Unknown tool: {name}") - - except Exception as e: - logger.error(f"工具执行失败: {e}") - return [TextContent(type="text", text=f"Error: {str(e)}")] - - -async def call_v1_memory_tool(name: str, arguments: dict[str, Any]) -> dict[str, Any]: - """Dispatch v1 Memory Gateway MCP tools to the same service used by /v1.""" - if name == "memory_search": - return _jsonable(await v1_service.search_memory_with_openviking(MemorySearchRequest(**arguments))) - if name == "memory_upsert": - return v1_service.upsert_memory(MemoryUpsertRequest(**arguments)).model_dump(mode="json") - if name == "memory_append_episode": - return v1_service.append_episode(EpisodeAppendRequest(**arguments)).model_dump(mode="json") - if name == "memory_commit_session": - session_id = arguments.get("session_id") - if not session_id: - raise ValueError("session_id is required") - return _jsonable(v1_service.commit_session(session_id, CommitSessionRequest(**arguments))) - if name == "memory_get_profile": - return v1_service.get_profile(arguments["user_id"]).model_dump(mode="json") - if name == "memory_list_namespaces": - return { - "namespaces": [ - item.model_dump(mode="json") - for item in v1_service.list_namespaces( - AccessContext( - user_id=arguments["user_id"], - agent_id=arguments.get("agent_id"), - workspace_id=arguments.get("workspace_id"), - session_id=arguments.get("session_id"), - ) - ) - ] - } - if name == "memory_delete": - return v1_service.delete_memory( - arguments["memory_id"], - AccessContext( - user_id=arguments["user_id"], - agent_id=arguments.get("agent_id"), - workspace_id=arguments.get("workspace_id"), - session_id=arguments.get("session_id"), - ), - ) - if name == "memory_feedback": - return v1_service.add_feedback(arguments["memory_id"], MemoryFeedbackRequest(**arguments)) - raise ValueError(f"Unknown v1 memory tool: {name}") - - -def _jsonable(value: Any) -> Any: - if hasattr(value, "model_dump"): - return value.model_dump(mode="json") - if isinstance(value, list): - return [_jsonable(item) for item in value] - if isinstance(value, dict): - return {key: _jsonable(item) for key, item in value.items()} - return value - - -@asynccontextmanager -async def lifespan(app: FastAPI): - """应用生命周期管理""" - logger.info("Memory Gateway 启动中...") - config = get_config() - logger.info(f"配置加载完成: {config.server.host}:{config.server.port}") - logger.info(f"OpenViking 后端: {config.openviking.url}") - - # 测试 OpenViking 连接 - try: - ov_client = await get_openviking_client() - status = await ov_client.health_check() - logger.info(f"OpenViking 连接状态: {status}") - except Exception as e: - logger.warning(f"OpenViking 连接失败: {e}") - - yield - - logger.info("Memory Gateway 关闭中...") - await close_openviking_client() - - -def verify_api_key(x_api_key: Optional[str] = Header(default=None)) -> None: - """在配置了 API Key 时校验请求头。""" - expected_key = get_config().server.api_key - if not expected_key: - return - if x_api_key != expected_key: - raise HTTPException( - status_code=status.HTTP_401_UNAUTHORIZED, - detail="Invalid or missing API key", - ) - - -_SENTENCE_RE = re.compile(r"(?<=[。！？.!?])\s+") -_WORD_RE = re.compile(r"[^a-zA-Z0-9\u4e00-\u9fff_-]+") - - -def _normalize_whitespace(value: str) -> str: - return re.sub(r"\s+", " ", value).strip() - - -def _slugify(value: str, fallback: str) -> str: - slug = _WORD_RE.sub("-", value.lower()).strip("-") - slug = re.sub(r"-+", "-", slug)[:80].strip("-") - return slug or fallback - - -def _derive_title(content: str, title: Optional[str]) -> str: - if title and title.strip(): - return title.strip() - for line in content.splitlines(): - line = line.strip("# -*\t") - if line: - return line[:120] - return "Untitled summary" - - -def _derive_summary(content: str, provided: Optional[str], max_chars: int) -> str: - if provided and provided.strip(): - return provided.strip()[:max_chars] - - normalized = _normalize_whitespace(content) - if not normalized: - return "" - - sentences = [part.strip() for part in _SENTENCE_RE.split(normalized) if part.strip()] - if not sentences: - return normalized[:max_chars] - - summary = " ".join(sentences[:3]) - return summary[:max_chars] - - -def _extract_key_points(content: str, limit: int = 8) -> list[str]: - points: list[str] = [] - for raw_line in content.splitlines(): - line = raw_line.strip() - if not line: - continue - stripped = re.sub(r"^(?:[-*•]\s*|\d+[.、)]\s*)", "", line).strip() - if not stripped: - continue - is_structured = line.startswith(("-", "*", "•")) or re.match(r"^\d+[.、)]\s+", line) - has_signal = any(token in stripped.lower() for token in [ - "verdict", "result", "finding", "evidence", "action", "risk", "ioc", - "结论", "结果", "证据", "建议", "动作", "风险", "命中", "关联", - ]) - if is_structured or has_signal: - point = _normalize_whitespace(stripped) - if point and point not in points: - points.append(point[:240]) - if len(points) >= limit: - break - - if points: - return points - - summary = _derive_summary(content, None, 500) - return [summary] if summary else [] - - -def _render_memory_text(artifact: dict[str, Any]) -> str: - lines = [ - f"Title: {artifact['title']}", - f"Summary: {artifact['summary']}", - ] - if artifact.get("tags"): - lines.append("Tags: " + ", ".join(artifact["tags"])) - if artifact.get("source"): - lines.append("Source: " + artifact["source"]) - if artifact.get("key_points"): - lines.append("Key points:") - lines.extend(f"- {point}" for point in artifact["key_points"]) - return "\n".join(lines) - - -def _default_summary_resource_uri(request: CommitSummaryRequest, title: str) -> str: - namespace = (request.namespace or get_config().memory.default_namespace or "general").strip("/") - memory_type = (request.memory_type or "summary").strip("/") - digest = hashlib.sha1(request.content.encode("utf-8")).hexdigest()[:12] - slug = _slugify(title, digest) - return f"viking://resources/{namespace}/{memory_type}/{slug}-{digest}.json" - - -async def build_summary_artifact(request: CommitSummaryRequest) -> dict[str, Any]: - max_chars = max(120, min(request.max_summary_chars, 4000)) - llm_result = await summarize_with_llm( - request.content, - title=request.title, - summary_hint=request.summary, - tags=request.tags, - max_summary_chars=max_chars, - purpose=request.purpose or "generic knowledge memory", - ) - title = llm_result.get("title") or _derive_title(request.content, request.title) - return { - "schema_version": "memory-gateway.summary.v1", - "id": hashlib.sha1(request.content.encode("utf-8")).hexdigest()[:16], - "title": title, - "summary": llm_result.get("summary", ""), - "key_points": llm_result.get("key_points", []), - "tags": llm_result.get("tags", request.tags), - "source": request.source, - "namespace": request.namespace or get_config().memory.default_namespace, - "memory_type": request.memory_type or "summary", - "created_at": datetime.now(timezone.utc).isoformat(), - "content": request.content, - "llm": llm_result.get("llm"), - } - - -async def commit_summary_to_openviking(request: CommitSummaryRequest) -> dict[str, Any]: - artifact = await build_summary_artifact(request) - ov_client = await get_openviking_client() - - memory_result: Optional[dict[str, Any]] = None - resource_result: Optional[dict[str, Any]] = None - - if request.persist_as in {"memory", "both"}: - memory_result = await ov_client.add_memory( - content=_render_memory_text(artifact), - namespace=artifact["namespace"], - memory_type=artifact["memory_type"], - ) - - if request.persist_as in {"resource", "both"}: - resource_uri = request.resource_uri or _default_summary_resource_uri(request, artifact["title"]) - artifact["resource_uri"] = resource_uri - resource_result = await ov_client.add_resource( - uri=resource_uri, - content=json.dumps(artifact, ensure_ascii=False, indent=2), - resource_type=request.resource_type or "json", - ) - - return { - "status": "ok", - "artifact": artifact, - "memory_result": memory_result, - "resource_result": resource_result, - } - - -# FastAPI 应用 -app = FastAPI(title="Memory Gateway", version="0.1.0", lifespan=lifespan) - -app.add_middleware( - CORSMiddleware, - allow_origins=["*"], - allow_credentials=True, - allow_methods=["*"], - allow_headers=["*"], -) - - -@app.get("/health", dependencies=[Depends(verify_api_key)]) -async def health_check(): - """健康检查""" - try: - ov_client = await get_openviking_client() - ov_status = await ov_client.health_check() - everos_status = v1_service.everos_health() - return { - "status": "ok", - "gateway": "memory-gateway", - "openviking": ov_status, - "everos": everos_status, - } - except Exception as e: - return { - "status": "degraded", - "gateway": "memory-gateway", - "error": str(e), - } - -mcp_router = APIRouter() - - -async def mcp_server_events(request: Request, _: None = Depends(verify_api_key)): - """MCP Server-Sent Events 端点 - 使用 stdio 模式模拟""" - async def event_generator(): - # 发送初始化消息 - yield {"event": "initialize", "data": json.dumps({"protocolVersion": "2024-11-05"})} - - # 保持连接 - try: - while True: - await asyncio.sleep(30) - yield {"event": "ping", "data": ""} - except asyncio.CancelledError: - pass - - return EventSourceResponse(event_generator()) - - -mcp_router.add_api_route("/sse", mcp_server_events, methods=["GET"]) - - -# MCP JSON-RPC 端点（简化实现） -async def mcp_rpc(request: Request, _: None = Depends(verify_api_key)): - """处理 MCP JSON-RPC 请求""" - body = await request.json() - - method = body.get("method") - params = body.get("params", {}) - msg_id = body.get("id") - - try: - if method == "tools/list": - tools = await list_tools() - result = { - "tools": [ - { - "name": t.name, - "description": t.description, - "inputSchema": t.inputSchema, - } - for t in tools - ] - } - elif method == "tools/call": - tool_name = params.get("name") - tool_args = params.get("arguments", {}) - result_content = await call_tool_tool(tool_name, tool_args) - result = {"content": [c.model_dump() for c in result_content]} - else: - return JSONResponse( - status_code=400, - content={"jsonrpc": "2.0", "error": {"code": -32601, "message": f"Method not found: {method}"}, "id": msg_id} - ) - - return {"jsonrpc": "2.0", "result": result, "id": msg_id} - - except Exception as e: - logger.error(f"MCP RPC 错误: {e}") - return JSONResponse( - status_code=500, - content={"jsonrpc": "2.0", "error": {"code": -32603, "message": str(e)}, "id": msg_id} - ) - - -async def call_tool_tool(name: str, arguments: dict) -> list[TextContent]: - """调用工具的内部函数""" - return await call_tool(name, arguments) - - -mcp_router.add_api_route("/rpc", mcp_rpc, methods=["POST"]) - - -# 注册 MCP 路由 -app.include_router(mcp_router, prefix="/mcp", tags=["mcp"]) - -# Generic Memory Gateway v1 routes are imported lazily here to avoid changing -# the existing legacy /api and /mcp startup path. -from .api_v1 import router as api_v1_router # noqa: E402 -from .api_v2 import router as api_v2_router # noqa: E402 - -app.include_router(api_v1_router) -app.include_router(api_v2_router) - - -@app.post("/api/search", dependencies=[Depends(verify_api_key)]) -async def api_search(request: SearchRequest): - """REST API: 搜索""" - ov_client = await get_openviking_client() - result = await ov_client.search( - query=request.query, - namespace=request.namespace or get_config().memory.default_namespace, - limit=request.limit or get_config().memory.search_limit, - uri=request.uri, - ) - return {"results": result.results, "total": result.total} - - -@app.post("/api/memory", dependencies=[Depends(verify_api_key)]) -async def api_add_memory(request: AddMemoryRequest): - """REST API: 添加记忆""" - ov_client = await get_openviking_client() - result = await ov_client.add_memory( - content=request.content, - namespace=request.namespace or get_config().memory.default_namespace, - memory_type=request.memory_type, - ) - return result - - -@app.post("/api/resource", dependencies=[Depends(verify_api_key)]) -async def api_add_resource(request: AddResourceRequest): - """REST API: 添加资源""" - ov_client = await get_openviking_client() - result = await ov_client.add_resource( - uri=request.uri, - content=request.content, - resource_type=request.resource_type, - ) - return result - - -@app.post("/api/summary", dependencies=[Depends(verify_api_key)]) -async def api_commit_summary(request: CommitSummaryRequest): - """REST API: 通用内容 LLM 总结与记忆沉淀。""" - try: - return await commit_summary_to_openviking(request) - except LLMConfigurationError as exc: - raise HTTPException(status_code=503, detail=str(exc)) from exc - except (LLMSummaryError, Exception) as exc: - if isinstance(exc, HTTPException): - raise - raise HTTPException(status_code=502, detail=f"LLM summary failed: {exc}") from exc - - -def _parse_tags(tags: str | None) -> list[str]: - if not tags: - return [] - return [tag.strip() for tag in re.split(r"[,\n]", tags) if tag.strip()] - - -def _default_knowledge_uri(namespace: str, knowledge_type: str, title: str, content: str) -> str: - digest = hashlib.sha1(content.encode("utf-8")).hexdigest()[:12] - return f"viking://resources/{namespace.strip('/')}/knowledge/{knowledge_type.strip('/')}/{slugify(title, digest)}-{digest}.json" - - -@app.post("/api/knowledge/upload", dependencies=[Depends(verify_api_key)]) -async def api_upload_knowledge( - file: UploadFile = File(...), - title: Optional[str] = Form(default=None), - namespace: str = Form(default="memory-gateway"), - knowledge_type: str = Form(default="knowledge"), - tags: str = Form(default=""), - source: Optional[str] = Form(default=None), - obsidian_dir: Optional[str] = Form(default=None), - resource_uri: Optional[str] = Form(default=None), - persist_as: str = Form(default="resource"), - max_summary_chars: int = Form(default=1000), -): - """Upload a document, convert it to Markdown, save to Obsidian, summarize with LLM, and commit to OpenViking.""" - if persist_as not in {"memory", "resource", "both", "none"}: - raise HTTPException(status_code=422, detail="persist_as must be one of memory/resource/both/none") - - original_name = file.filename or "uploaded-document" - suffix = Path(original_name).suffix or ".bin" - with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as tmp: - tmp.write(await file.read()) - tmp_path = Path(tmp.name) - - try: - markdown = await asyncio.to_thread(convert_file_to_markdown, tmp_path) - except RuntimeError as exc: - tmp_path.unlink(missing_ok=True) - raise HTTPException(status_code=500, detail=str(exc)) from exc - except Exception as exc: # noqa: BLE001 - tmp_path.unlink(missing_ok=True) - raise HTTPException(status_code=500, detail=f"Document conversion failed: {exc}") from exc - finally: - tmp_path.unlink(missing_ok=True) - - parsed_tags = _parse_tags(tags) - effective_title = title or Path(original_name).stem or "Uploaded knowledge" - request = CommitSummaryRequest( - content=markdown, - title=effective_title, - namespace=namespace, - memory_type=knowledge_type, - tags=parsed_tags, - source=source or original_name, - persist_as="none", - max_summary_chars=max_summary_chars, - purpose=f"knowledge upload: {knowledge_type}", - ) - try: - artifact = await build_summary_artifact(request) - except LLMConfigurationError as exc: - raise HTTPException(status_code=503, detail=str(exc)) from exc - except Exception as exc: # noqa: BLE001 - raise HTTPException(status_code=502, detail=f"LLM summary failed: {exc}") from exc - - config = get_config() - relative_dir = obsidian_dir or getattr(config.obsidian, "knowledge_dir", "01_Knowledge/Uploaded") - obsidian_path = save_markdown_to_obsidian( - vault_path=config.obsidian.vault_path, - relative_dir=relative_dir, - title=artifact["title"], - markdown=markdown, - source_filename=original_name, - tags=artifact.get("tags", []), - knowledge_type=knowledge_type, - summary=artifact.get("summary"), - ) - - artifact.update( - { - "schema_version": "memory-gateway.knowledge_upload.v1", - "knowledge_type": knowledge_type, - "source_filename": original_name, - "obsidian_path": str(obsidian_path), - "obsidian_relative_path": str(obsidian_path.relative_to(config.obsidian.vault_path)), - "markdown_content": markdown, - } - ) - - ov_client = await get_openviking_client() - memory_result: Optional[dict[str, Any]] = None - resource_result: Optional[dict[str, Any]] = None - if persist_as in {"memory", "both"}: - memory_result = await ov_client.add_memory( - content=_render_memory_text(artifact), - namespace=namespace, - memory_type=knowledge_type, - ) - if persist_as in {"resource", "both"}: - final_uri = resource_uri or _default_knowledge_uri(namespace, knowledge_type, artifact["title"], markdown) - artifact["resource_uri"] = final_uri - resource_result = await ov_client.add_resource( - uri=final_uri, - content=json.dumps(artifact, ensure_ascii=False, indent=2), - resource_type="json", - ) - - return { - "status": "ok", - "artifact": artifact, - "markdown_chars": len(markdown), - "obsidian_path": str(obsidian_path), - "memory_result": memory_result, - "resource_result": resource_result, - } - - -def create_app(config: Optional[Config] = None) -> FastAPI: - """创建 FastAPI 应用""" - if config: - set_config(config) - return app - - -# 入口点 -def main(): - """主入口""" - import argparse - import uvicorn - - parser = argparse.ArgumentParser(description="Memory Gateway MCP Server") - parser.add_argument("--config", default="config.yaml", help="配置文件路径") - parser.add_argument("--host", default=None, help="监听地址") - parser.add_argument("--port", type=int, default=None, help="监听端口") - args = parser.parse_args() - - # 加载配置 - from .config import load_config as load - config = load(args.config) - if args.host: - config.server.host = args.host - if args.port: - config.server.port = args.port - set_config(config) - - # 启动服务 - uvicorn.run( - app, - host=config.server.host, - port=config.server.port, - log_level=config.logging.level.lower(), - ) - - -if __name__ == "__main__": - main() diff --git a/memory_gateway/server_auth.py b/memory_gateway/server_auth.py deleted file mode 100644 index 3970a51..0000000 --- a/memory_gateway/server_auth.py +++ /dev/null @@ -1,15 +0,0 @@ -"""Small auth bridge used by the modular v1 router.""" -from __future__ import annotations - -from typing import Optional - -from fastapi import Header, HTTPException, status - -from .config import get_config - - -def verify_api_key_compat(x_api_key: Optional[str] = Header(default=None)) -> None: - expected_key = get_config().server.api_key - if expected_key and x_api_key != expected_key: - raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid or missing API key") - diff --git a/memory_gateway/services.py b/memory_gateway/services.py deleted file mode 100644 index 10412ad..0000000 --- a/memory_gateway/services.py +++ /dev/null @@ -1,366 +0,0 @@ -"""Application services for the generic Memory Gateway v1 API.""" -from __future__ import annotations - -from dataclasses import dataclass, field -from datetime import datetime, timezone - -from fastapi import HTTPException, status - -from .config import get_config -from .everos_client import EverOSError, EverOSClient -from .namespace import can_access_memory, default_namespace_for_context, user_long_term_namespace, visible_namespaces -from .openviking_client import get_openviking_client -from .repositories import MetadataRepository, repository -from .schemas import ( - AccessContext, - AuditLog, - CommitSessionRequest, - CreateUserRequest, - EpisodeAppendRequest, - EpisodeRecord, - MemoryFeedbackRequest, - MemoryPatchRequest, - MemoryRecord, - MemorySearchRequest, - MemoryType, - MemoryUpsertRequest, - NamespaceInfo, - ProfileRecord, - SourceType, - UserRecord, - Visibility, -) - - -@dataclass -class ConsolidationResult: - session_id: str - episodes: int - candidates: list[MemoryRecord] = field(default_factory=list) - promoted: list[MemoryRecord] = field(default_factory=list) - duplicates: list[dict] = field(default_factory=list) - review_drafts: list[str] = field(default_factory=list) - conflicts: list[dict] = field(default_factory=list) - - -class MemoryGatewayService: - def __init__(self, repo: MetadataRepository = repository, everos_client: EverOSClient | None = None) -> None: - self.repo = repo - self.everos_client = everos_client - - def create_user(self, request: CreateUserRequest) -> UserRecord: - user = UserRecord( - id=request.user_id or UserRecord(display_name=request.display_name).id, - display_name=request.display_name, - preferences=request.preferences, - ) - user.profile_namespace = f"user/{user.id}/profile" - self.repo.create_user(user) - self._audit("create_user", "user", user.id, namespace=user.profile_namespace, actor_user_id=user.id) - return user - - def get_user(self, user_id: str) -> UserRecord: - user = self.repo.get_user(user_id) - if not user: - raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="User not found") - return user - - def search_memory(self, request: MemorySearchRequest) -> dict: - ctx = AccessContext(**request.model_dump(include={"user_id", "agent_id", "workspace_id", "session_id"})) - query = request.query.lower().strip() - results = [] - for memory in self.repo.list_memories(): - if not can_access_memory(ctx, memory): - continue - if request.namespaces and memory.namespace not in request.namespaces: - continue - if request.memory_types and memory.memory_type not in request.memory_types: - continue - if request.tags and not set(request.tags).intersection(memory.tags): - continue - haystack = " ".join([memory.content, memory.summary or "", " ".join(memory.tags)]).lower() - if query and query not in haystack: - continue - score = self._score(memory, query) - results.append({"memory": memory, "score": score}) - results.sort(key=lambda item: item["score"], reverse=True) - return {"results": results[: request.limit], "total": len(results)} - - async def search_memory_with_openviking(self, request: MemorySearchRequest) -> dict: - """Search local metadata first, then fan out to OpenViking for visible namespaces.""" - ctx = AccessContext(**request.model_dump(include={"user_id", "agent_id", "workspace_id", "session_id"})) - local = self.search_memory(request) - visible = {namespace.namespace for namespace in visible_namespaces(ctx)} - requested = set(request.namespaces) if request.namespaces else visible - allowed_namespaces = sorted(requested.intersection(visible)) - - openviking_results = [] - if allowed_namespaces and request.query.strip(): - try: - ov_client = await get_openviking_client() - per_namespace_limit = max(1, min(request.limit, 10)) - for namespace in allowed_namespaces: - result = await ov_client.search( - query=request.query, - namespace=namespace, - limit=per_namespace_limit, - ) - for item in result.results: - item = dict(item) - item["namespace"] = namespace - item["source"] = "openviking" - openviking_results.append(item) - except Exception as exc: # noqa: BLE001 - self._audit( - "openviking_search_failed", - "search", - None, - actor_user_id=request.user_id, - actor_agent_id=request.agent_id, - metadata={"error": str(exc)}, - ) - - self._audit( - "memory_search", - "memory", - None, - actor_user_id=request.user_id, - actor_agent_id=request.agent_id, - metadata={"query": request.query, "namespaces": allowed_namespaces, "openviking_results": len(openviking_results)}, - ) - return { - "results": local["results"] + [{"openviking": item, "score": item.get("score", 0)} for item in openviking_results], - "total": local["total"] + len(openviking_results), - "local_total": local["total"], - "openviking_total": len(openviking_results), - "searched_namespaces": allowed_namespaces, - } - - def upsert_memory(self, request: MemoryUpsertRequest) -> MemoryRecord: - ctx = AccessContext(**request.model_dump(include={"user_id", "agent_id", "workspace_id", "session_id"})) - namespace = request.namespace or default_namespace_for_context(ctx, request.visibility) - memory = MemoryRecord( - user_id=request.user_id, - agent_id=request.agent_id, - workspace_id=request.workspace_id, - session_id=request.session_id, - namespace=namespace, - memory_type=request.memory_type, - content=request.content, - summary=request.summary, - tags=request.tags, - importance=request.importance, - confidence=request.confidence, - visibility=request.visibility, - source=request.source, - expires_at=request.expires_at, - ) - self.repo.upsert_memory(memory) - self._audit("upsert_memory", "memory", memory.id, namespace=memory.namespace, actor_user_id=request.user_id, actor_agent_id=request.agent_id) - return memory - - def get_memory(self, memory_id: str, ctx: AccessContext) -> MemoryRecord: - memory = self.repo.get_memory(memory_id) - if not memory: - raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Memory not found") - if not can_access_memory(ctx, memory): - self._audit("get_memory", "memory", memory_id, namespace=memory.namespace, actor_user_id=ctx.user_id, actor_agent_id=ctx.agent_id, decision="deny") - raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Memory access denied") - return memory - - def patch_memory(self, memory_id: str, ctx: AccessContext, patch: MemoryPatchRequest) -> MemoryRecord: - memory = self.get_memory(memory_id, ctx) - updates = patch.model_dump(exclude_unset=True) - for key, value in updates.items(): - setattr(memory, key, value) - memory.updated_at = datetime.now(timezone.utc) - memory.version += 1 - self.repo.upsert_memory(memory) - self._audit("patch_memory", "memory", memory.id, namespace=memory.namespace, actor_user_id=ctx.user_id, actor_agent_id=ctx.agent_id) - return memory - - def delete_memory(self, memory_id: str, ctx: AccessContext) -> dict: - memory = self.get_memory(memory_id, ctx) - deleted = self.repo.delete_memory(memory_id) - self._audit("delete_memory", "memory", memory_id, namespace=memory.namespace, actor_user_id=ctx.user_id, actor_agent_id=ctx.agent_id) - return {"deleted": deleted, "id": memory_id} - - def append_episode(self, request: EpisodeAppendRequest) -> EpisodeRecord: - ctx = AccessContext(**request.model_dump(include={"user_id", "agent_id", "workspace_id", "session_id"})) - episode = EpisodeRecord( - user_id=request.user_id, - agent_id=request.agent_id, - workspace_id=request.workspace_id, - session_id=request.session_id or "default", - namespace=request.namespace or default_namespace_for_context(ctx, Visibility.PRIVATE), - content=request.content, - events=request.events, - tags=request.tags, - source=request.source, - expires_at=request.expires_at, - ) - self.repo.append_episode(episode) - self._audit("append_episode", "episode", episode.id, namespace=episode.namespace, actor_user_id=request.user_id, actor_agent_id=request.agent_id) - return episode - - def commit_session(self, session_id: str, request: CommitSessionRequest) -> dict: - episodes = self.repo.list_session_episodes(session_id) - backend = "disabled" - error: str | None = None - if request.promote: - ctx = AccessContext( - user_id=request.user_id, - agent_id=request.agent_id, - workspace_id=request.workspace_id, - session_id=session_id, - ) - target_namespace = request.target_namespace or user_long_term_namespace(request.user_id) - config = get_config().everos - if config.enabled: - try: - external_result = (self.everos_client or EverOSClient()).consolidate_session( - session_id=session_id, - ctx=ctx, - episodes=episodes, - existing_memories=list(self.repo.list_memories()), - min_importance=request.min_importance, - target_namespace=target_namespace, - ) - result = self._persist_external_consolidation(external_result, ctx, session_id) - backend = "external" - except EverOSError as exc: - error = str(exc) - self._audit( - "everos_commit_failed", - "session", - session_id, - actor_user_id=request.user_id, - actor_agent_id=request.agent_id, - decision="deny", - metadata={"error": error}, - ) - raise HTTPException(status_code=status.HTTP_502_BAD_GATEWAY, detail=f"EverOS failed: {error}") from exc - else: - result = None - backend = "disabled" - else: - result = None - self._audit("commit_session", "session", session_id, actor_user_id=request.user_id, actor_agent_id=request.agent_id) - if not result: - return {"session_id": session_id, "episodes": len(episodes), "promoted": [], "everos_backend": backend} - return { - "everos_backend": backend, - "everos_error": error, - "session_id": session_id, - "episodes": result.episodes, - "candidates": result.candidates, - "promoted": result.promoted, - "duplicates": result.duplicates, - "conflicts": result.conflicts, - "review_drafts": result.review_drafts, - } - - def everos_health(self) -> dict: - config = get_config().everos - if not config.enabled: - return {"status": "disabled", "url": config.url} - return (self.everos_client or EverOSClient()).health() - - def _persist_external_consolidation(self, external_result: dict, ctx: AccessContext, session_id: str): - result = ConsolidationResult( - session_id=session_id, - episodes=external_result.get("episodes") or len(self.repo.list_session_episodes(session_id)), - duplicates=external_result.get("duplicates", []), - conflicts=external_result.get("conflicts", []), - review_drafts=external_result.get("review_drafts", []), - ) - for item in external_result.get("candidates", []): - memory = self._memory_from_external(item, ctx, session_id) - if memory: - result.candidates.append(memory) - for item in external_result.get("promoted", []): - memory = self._memory_from_external(item, ctx, session_id) - if memory: - self.repo.upsert_memory(memory) - result.promoted.append(memory) - if all(candidate.id != memory.id for candidate in result.candidates): - result.candidates.append(memory) - return result - - def _memory_from_external(self, item: dict, ctx: AccessContext, session_id: str) -> MemoryRecord | None: - if not isinstance(item, dict): - return None - data = dict(item) - data.setdefault("user_id", ctx.user_id) - data.setdefault("agent_id", ctx.agent_id) - data.setdefault("workspace_id", ctx.workspace_id) - data.setdefault("session_id", session_id) - data.setdefault("namespace", default_namespace_for_context(ctx, Visibility.PRIVATE)) - data.setdefault("memory_type", MemoryType.SUMMARY.value) - data.setdefault("content", data.get("text") or data.get("summary") or "") - data.setdefault("summary", data.get("content", "")[:180]) - data.setdefault("tags", ["everos-external"]) - data.setdefault("importance", 0.7) - data.setdefault("confidence", 0.65) - data.setdefault("visibility", Visibility.PRIVATE.value) - data.setdefault("source", SourceType.EVEROS.value) - if not data["content"]: - return None - return MemoryRecord.model_validate(data) - - def get_profile(self, user_id: str) -> ProfileRecord: - profile = self.repo.get_profile(user_id) - if not profile: - raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Profile not found") - return profile - - def add_feedback(self, memory_id: str, request: MemoryFeedbackRequest) -> dict: - ctx = AccessContext(**request.model_dump(include={"user_id", "agent_id", "workspace_id", "session_id"})) - memory = self.get_memory(memory_id, ctx) - self._audit( - f"feedback:{request.feedback}", - "memory", - memory.id, - namespace=memory.namespace, - actor_user_id=request.user_id, - actor_agent_id=request.agent_id, - metadata={"comment": request.comment}, - ) - return {"status": "ok", "memory_id": memory_id, "feedback": request.feedback} - - def list_namespaces(self, ctx: AccessContext) -> list[NamespaceInfo]: - return visible_namespaces(ctx) - - def list_audit(self, limit: int = 100) -> list[AuditLog]: - return self.repo.list_audit(limit) - - def _score(self, memory: MemoryRecord, query: str) -> float: - lexical = 1.0 if query and query in memory.content.lower() else 0.2 - return lexical + memory.importance + memory.confidence - - def _audit( - self, - action: str, - target_type: str, - target_id: str | None, - namespace: str | None = None, - actor_user_id: str | None = None, - actor_agent_id: str | None = None, - decision: str = "allow", - metadata: dict | None = None, - ) -> None: - self.repo.add_audit( - AuditLog( - actor_user_id=actor_user_id, - actor_agent_id=actor_agent_id, - action=action, - target_type=target_type, - target_id=target_id, - namespace=namespace, - decision=decision, # type: ignore[arg-type] - metadata=metadata or {}, - ) - ) - - -service = MemoryGatewayService() diff --git a/memory_gateway/services_v2.py b/memory_gateway/services_v2.py deleted file mode 100644 index 0e31c1b..0000000 --- a/memory_gateway/services_v2.py +++ /dev/null @@ -1,1046 +0,0 @@ -"""Service orchestration for the Memory Gateway v2 control plane.""" -from __future__ import annotations - -import hashlib -from datetime import datetime, timedelta, timezone -from typing import Any, Awaitable, Callable -from uuid import uuid4 - -from fastapi import HTTPException, status - -from .backend_contracts import ( - BackendOperation, - BackendCommitResult, - BackendProducedRef, - BackendRetrieveResult, - BackendResultStatus, - BackendWriteResult, - CommitJob, - OutboxEvent, - OutboxEventStatus, -) -from .everos_client import EverOSClient -from .openviking_client import get_openviking_client -from .repositories import MetadataRepository, repository -from .schemas import AuditLog -from .schemas_v2 import ( - BackendRefStatus, - BackendType, - CommitJobView, - CommitRequest, - CommitResponse, - ContextItem, - FeedbackRequest, - FeedbackResponse, - IngestRequest, - IngestResponse, - MemoryRef, - MemoryRefView, - MemoryRefType, - OperationStatus, - OutboxProcessResponse, - OutboxSummary, - RetrieveRequest, - RetrieveResponse, -) - - -OpenVikingClientFactory = Callable[[], Awaitable[Any]] - - -class MemoryGatewayV2Service: - def __init__( - self, - repo: MetadataRepository = repository, - openviking_client_factory: OpenVikingClientFactory = get_openviking_client, - everos_client: Any | None = None, - ) -> None: - self.repo = repo - self.openviking_client_factory = openviking_client_factory - self.everos_client = everos_client - - async def ingest_conversation_turn(self, request: IngestRequest) -> IngestResponse: - normalized = self._normalize_ingest_request(request) - gateway_id = self._build_gateway_id(normalized) - provenance_id = self._build_provenance_id(normalized, gateway_id) - content_hash = self._content_hash(normalized.content) - - self._check_namespace_access(normalized) - payload = self._apply_safety_policy(normalized) - - refs: list[MemoryRef] = [] - if normalized.policy.allow_openviking: - refs.append( - await self._write_openviking_turn( - normalized, - payload, - gateway_id=gateway_id, - provenance_id=provenance_id, - content_hash=content_hash, - ) - ) - else: - refs.append( - self._save_ref( - normalized, - gateway_id, - provenance_id, - BackendType.OPENVIKING, - MemoryRefType.SESSION_ARCHIVE, - BackendRefStatus.SKIPPED, - content_hash=content_hash, - metadata=self._control_metadata(normalized, content_hash, {"reason": "policy_disabled"}), - ) - ) - - if normalized.policy.allow_everos: - refs.append( - await self._write_everos_message( - normalized, - payload, - gateway_id=gateway_id, - provenance_id=provenance_id, - content_hash=content_hash, - ) - ) - else: - refs.append( - self._save_ref( - normalized, - gateway_id, - provenance_id, - BackendType.EVEROS, - MemoryRefType.MESSAGE_MEMORY, - BackendRefStatus.SKIPPED, - content_hash=content_hash, - metadata=self._control_metadata(normalized, content_hash, {"reason": "policy_disabled"}), - ) - ) - - status_value = self._aggregate_ref_status(refs) - errors = [ref.error_message for ref in refs if ref.error_message] - self.repo.add_audit( - AuditLog( - actor_user_id=normalized.user_id, - actor_agent_id=normalized.agent_id, - action="v2_ingest_conversation_turn", - target_type="conversation_turn", - target_id=normalized.turn_id, - namespace=normalized.namespace, - metadata={ - "gateway_id": gateway_id, - "provenance_id": provenance_id, - "idempotency_basis": self._idempotency_basis(normalized), - "content_hash": content_hash, - "status": status_value.value, - "source_type": normalized.source_type, - "source_event_id": normalized.source_event_id, - }, - ) - ) - return IngestResponse( - status=status_value, - gateway_id=gateway_id, - provenance_id=provenance_id, - request_id=normalized.request_id, - turn_id=normalized.turn_id, - refs=self._view_refs(refs), - errors=errors, - metadata={"backend_count": len(refs)}, - ) - - async def commit_session(self, session_id: str, request: CommitRequest) -> CommitResponse: - # TODO(v2): add a worker that consumes these outbox events and writes - # resulting backend refs. This method intentionally only records - # control-plane intent. - job_id = f"job_{uuid4().hex[:16]}" - gateway_id = self._commit_gateway_id(session_id, request) - job = CommitJob( - job_id=job_id, - workspace_id=request.workspace_id, - user_id=request.user_id, - agent_id=request.agent_id, - session_id=session_id, - namespace=request.namespace, - requested_by=request.user_id, - ) - self.repo.save_commit_job(job) - self._create_commit_outbox_events(gateway_id, job, request) - self.repo.add_audit( - AuditLog( - actor_user_id=request.user_id, - actor_agent_id=request.agent_id, - action="v2_commit_session_accepted", - target_type="session", - target_id=session_id, - namespace=request.namespace, - metadata={ - "job_id": job_id, - "gateway_id": gateway_id, - "workspace_id": request.workspace_id, - "outbox_events": 2, - }, - ) - ) - return CommitResponse( - job_id=job_id, - session_id=session_id, - metadata={"gateway_id": gateway_id, "outbox_events": 2}, - ) - - async def retrieve_context(self, request: RetrieveRequest) -> RetrieveResponse: - payload = { - "workspace_id": request.workspace_id, - "user_id": request.user_id, - "agent_id": request.agent_id, - "session_id": request.session_id, - "namespace": request.namespace, - "query": request.query, - "limit": request.limit, - "metadata": request.metadata, - } - results = [ - await self._retrieve_openviking_context(payload), - await self._retrieve_everos_context(payload), - ] - items = self._merge_retrieve_items(results, limit=request.limit) - refs = self.repo.list_memory_refs( - workspace_id=request.workspace_id, - user_id=request.user_id, - agent_id=request.agent_id, - session_id=request.session_id, - namespace=request.namespace, - limit=request.limit, - ) - trace_id = request.metadata.get("trace_id") if request.metadata else None - return RetrieveResponse( - status=OperationStatus.SUCCESS, - items=items, - refs=self._view_refs(refs), - conflicts=[], - trace_id=trace_id, - metadata=self._retrieve_metadata(results), - ) - - async def record_memory_feedback(self, request: FeedbackRequest) -> FeedbackResponse: - # TODO(v2): persist review/feedback state in a dedicated table and route - # accepted corrections back to the owning backend adapter. - feedback_id = f"fb_{uuid4().hex[:16]}" - self.repo.add_audit( - AuditLog( - actor_user_id=request.user_id, - actor_agent_id=request.agent_id, - action=f"v2_feedback:{request.feedback_type}", - target_type="memory_ref", - target_id=request.memory_ref_id, - namespace=request.namespace, - metadata={ - "feedback_id": feedback_id, - "workspace_id": request.workspace_id, - "comment": request.comment, - "source_type": request.source_type, - "source_event_id": request.source_event_id, - }, - ) - ) - return FeedbackResponse( - status=OperationStatus.ACCEPTED, - feedback_id=feedback_id, - memory_ref_id=request.memory_ref_id, - ) - - async def process_pending_outbox_events( - self, - limit: int = 100, - worker_id: str | None = None, - lease_seconds: int = 300, - ) -> list[OutboxEvent]: - worker_id = worker_id or f"worker_{uuid4().hex[:12]}" - self.repo.release_expired_processing_events() - events = self.repo.claim_pending_outbox_events( - limit=limit, - worker_id=worker_id, - lease_seconds=lease_seconds, - ) - processed: list[OutboxEvent] = [] - for event in events: - try: - result = await self._execute_outbox_event(event) - except Exception as exc: # noqa: BLE001 - result = BackendCommitResult( - backend_type=event.backend_type, - operation=event.operation, - status=BackendResultStatus.FAILED, - retryable=True, - error_code="adapter_exception", - error_message=str(exc), - ) - processed.append(self._apply_outbox_result(event, result)) - return processed - - async def process_pending_outbox_events_summary( - self, - limit: int = 100, - worker_id: str | None = None, - lease_seconds: int = 300, - ) -> OutboxProcessResponse: - worker_id = worker_id or f"worker_{uuid4().hex[:12]}" - processed = await self.process_pending_outbox_events( - limit=limit, - worker_id=worker_id, - lease_seconds=lease_seconds, - ) - return OutboxProcessResponse( - status=OperationStatus.SUCCESS, - worker_id=worker_id, - processed_count=len(processed), - outbox_summary=self._outbox_summary(), - ) - - async def process_commit_job(self, job_id: str) -> CommitJob: - job = self.repo.get_commit_job(job_id) - if not job: - raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Commit job not found") - - self.repo.update_commit_job_status(job_id, OperationStatus.RUNNING.value) - self.repo.release_expired_processing_events() - events = self.repo.list_outbox_events_by_job(job_id) - for event in events: - if event.status == OutboxEventStatus.PENDING: - await self.process_outbox_event(event.id) - - events = self.repo.list_outbox_events_by_job(job_id) - final_status = self._aggregate_commit_job_status(events) - created_refs_count = self.repo.count_memory_refs(session_id=job.session_id, status=BackendRefStatus.SUCCESS) - error_message = self._commit_job_error_message(events) - updated = self.repo.update_commit_job_status( - job_id, - final_status.value, - error_message=error_message, - created_refs_count=created_refs_count, - ) - if not updated: - raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Commit job not found") - return updated - - def get_commit_job_view(self, job_id: str) -> CommitJobView: - job = self.repo.get_commit_job(job_id) - if not job: - raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Commit job not found") - return CommitJobView( - job_id=job.job_id, - workspace_id=job.workspace_id, - user_id=job.user_id, - agent_id=job.agent_id, - session_id=job.session_id, - namespace=job.namespace, - status=job.status, - created_refs_count=job.created_refs_count, - error_message=job.error_message, - created_at=job.created_at, - updated_at=job.updated_at, - started_at=job.started_at, - finished_at=job.finished_at, - outbox_summary=self._outbox_summary(self.repo.list_outbox_events_by_job(job_id)), - ) - - async def process_outbox_event(self, event_id: str) -> OutboxEvent | None: - event = self.repo.claim_outbox_event(event_id) - if not event: - return None - try: - result = await self._execute_outbox_event(event) - except Exception as exc: # noqa: BLE001 - result = BackendCommitResult( - backend_type=event.backend_type, - operation=event.operation, - status=BackendResultStatus.FAILED, - retryable=True, - error_code="adapter_exception", - error_message=str(exc), - ) - return self._apply_outbox_result(event, result) - - def list_memory_refs( - self, - workspace_id: str | None = None, - user_id: str | None = None, - agent_id: str | None = None, - session_id: str | None = None, - namespace: str | None = None, - backend_type: BackendType | str | None = None, - ref_type: MemoryRefType | str | None = None, - status: BackendRefStatus | str | None = None, - limit: int = 100, - ) -> list[MemoryRef]: - return self.repo.list_memory_refs( - workspace_id=workspace_id, - user_id=user_id, - agent_id=agent_id, - session_id=session_id, - namespace=namespace, - backend_type=backend_type, - ref_type=ref_type, - status=status, - limit=limit, - ) - - async def _retrieve_openviking_context(self, payload: dict[str, Any]) -> BackendRetrieveResult: - try: - client = await self.openviking_client_factory() - if not hasattr(client, "retrieve_context_v2"): - return BackendRetrieveResult( - backend_type=BackendType.OPENVIKING, - status=BackendResultStatus.SKIPPED, - metadata={"reason": "adapter_method_missing"}, - ) - result = client.retrieve_context_v2(payload) - if hasattr(result, "__await__"): - result = await result - return result - except Exception as exc: # noqa: BLE001 - return BackendRetrieveResult( - backend_type=BackendType.OPENVIKING, - status=BackendResultStatus.FAILED, - error_code="adapter_exception", - error_message=str(exc), - retryable=True, - ) - - async def _retrieve_everos_context(self, payload: dict[str, Any]) -> BackendRetrieveResult: - try: - client = self.everos_client or EverOSClient() - if not hasattr(client, "retrieve_context_v2"): - return BackendRetrieveResult( - backend_type=BackendType.EVEROS, - status=BackendResultStatus.SKIPPED, - metadata={"reason": "adapter_method_missing"}, - ) - result = client.retrieve_context_v2(payload) - if hasattr(result, "__await__"): - result = await result - return result - except Exception as exc: # noqa: BLE001 - return BackendRetrieveResult( - backend_type=BackendType.EVEROS, - status=BackendResultStatus.FAILED, - error_code="adapter_exception", - error_message=str(exc), - retryable=True, - ) - - def _merge_retrieve_items(self, results: list[BackendRetrieveResult], limit: int) -> list[ContextItem]: - items: list[ContextItem] = [] - for result in results: - if result.status != BackendResultStatus.SUCCESS: - continue - for item in result.items: - items.append( - ContextItem( - text=item.text, - source_backend=item.source_backend, - ref_id=item.ref_id, - score=item.score, - memory_type=item.memory_type, - metadata=item.metadata, - ) - ) - items.sort(key=lambda item: item.score, reverse=True) - return items[:limit] - - def _retrieve_metadata(self, results: list[BackendRetrieveResult]) -> dict[str, Any]: - return { - "backend_results": [ - { - "backend_type": result.backend_type.value, - "status": result.status.value, - "items": len(result.items), - "error_code": result.error_code, - "error_message": result.error_message, - } - for result in results - ] - } - - async def _execute_outbox_event(self, event: OutboxEvent) -> BackendCommitResult | BackendWriteResult: - payload = self._outbox_payload(event) - if event.operation != BackendOperation.COMMIT_SESSION: - return BackendCommitResult( - backend_type=event.backend_type, - operation=event.operation, - status=BackendResultStatus.SKIPPED, - metadata={"reason": "unsupported_operation"}, - ) - if event.backend_type == BackendType.OPENVIKING: - client = await self.openviking_client_factory() - if not hasattr(client, "commit_session_v2"): - return BackendCommitResult( - backend_type=BackendType.OPENVIKING, - operation=BackendOperation.COMMIT_SESSION, - status=BackendResultStatus.SKIPPED, - metadata={"reason": "adapter_method_missing"}, - ) - result = await client.commit_session_v2(payload) - return result - if event.backend_type == BackendType.EVEROS: - client = self.everos_client or EverOSClient() - if not hasattr(client, "extract_profile_long_term_v2"): - return BackendCommitResult( - backend_type=event.backend_type, - operation=BackendOperation.COMMIT_SESSION, - status=BackendResultStatus.SKIPPED, - metadata={"reason": "adapter_method_missing"}, - ) - result = client.extract_profile_long_term_v2(payload) - if hasattr(result, "__await__"): - result = await result - return result - return BackendCommitResult( - backend_type=event.backend_type, - operation=event.operation, - status=BackendResultStatus.SKIPPED, - metadata={"reason": "unsupported_backend"}, - ) - - def _outbox_payload(self, event: OutboxEvent) -> dict[str, Any]: - return { - "event_id": event.id, - "gateway_id": event.gateway_id, - "workspace_id": event.workspace_id, - "user_id": event.user_id, - "agent_id": event.agent_id, - "session_id": event.session_id, - "backend_type": event.backend_type.value, - "operation": event.operation.value, - "payload_ref": event.payload_ref, - "metadata": self._safe_control_metadata(event.metadata), - } - - def _apply_outbox_result(self, event: OutboxEvent, result: BackendCommitResult | BackendWriteResult) -> OutboxEvent: - result_data = self._backend_result_to_dict(result) - safe_result = self._backend_control_metadata(result_data) - event.metadata = self._safe_control_metadata({**event.metadata, "backend_result": safe_result}) - event.last_error = result.error_message - event.updated_at = datetime.now(timezone.utc) - - if result.status == BackendResultStatus.SUCCESS: - self._write_commit_memory_refs(event, result) - event.status = OutboxEventStatus.SUCCESS - self._clear_outbox_lock(event) - return self.repo.save_outbox_event(event) - - if result.status == BackendResultStatus.SKIPPED: - event.status = OutboxEventStatus.SKIPPED - self._clear_outbox_lock(event) - return self.repo.save_outbox_event(event) - - if result.status == BackendResultStatus.FAILED: - return self._handle_failed_outbox_event(event, result) - - event.status = OutboxEventStatus.PENDING - self._clear_outbox_lock(event) - return self.repo.save_outbox_event(event) - - def _handle_failed_outbox_event(self, event: OutboxEvent, result: BackendCommitResult | BackendWriteResult) -> OutboxEvent: - event.attempt_count += 1 - event.last_error = result.error_message - if result.retryable and event.attempt_count < event.max_attempts: - event.status = OutboxEventStatus.PENDING - event.next_retry_at = datetime.now(timezone.utc) + timedelta(seconds=min(60, 2**event.attempt_count)) - else: - event.status = OutboxEventStatus.DEAD_LETTER - event.next_retry_at = None - self._clear_outbox_lock(event) - return self.repo.save_outbox_event(event) - - def _clear_outbox_lock(self, event: OutboxEvent) -> None: - event.locked_by = None - event.locked_at = None - event.lease_expires_at = None - - def _write_commit_memory_refs(self, event: OutboxEvent, result: BackendCommitResult | BackendWriteResult) -> list[MemoryRef]: - produced_refs = result.refs if isinstance(result, BackendCommitResult) and result.refs else [] - if produced_refs: - refs: list[MemoryRef] = [] - for index, produced_ref in enumerate(produced_refs): - saved = self._write_commit_memory_ref(event, result, produced_ref, index=index) - if saved: - refs.append(saved) - return refs - - fallback = BackendProducedRef( - ref_type=self._commit_ref_type(event, result), - native_id=result.native_id, - native_uri=result.native_uri, - metadata={}, - ) - saved = self._write_commit_memory_ref(event, result, fallback, index=0) - return [saved] if saved else [] - - def _write_commit_memory_ref( - self, - event: OutboxEvent, - result: BackendCommitResult | BackendWriteResult, - produced_ref: BackendProducedRef, - index: int, - ) -> MemoryRef | None: - stable_key = self._produced_ref_stable_key(produced_ref, index) - if not produced_ref.native_id and not produced_ref.native_uri and not stable_key: - return None - ref_type = produced_ref.ref_type - ref_id = self._memory_ref_id(event.gateway_id, event.backend_type, ref_type, produced_ref.native_id, produced_ref.native_uri, stable_key) - existing = self.repo.get_memory_ref(ref_id) - safe_produced_metadata = self._safe_control_metadata(produced_ref.metadata) - ref = MemoryRef( - id=ref_id, - gateway_id=event.gateway_id, - workspace_id=event.workspace_id, - user_id=event.user_id, - agent_id=event.agent_id, - session_id=event.session_id, - namespace=event.metadata.get("namespace"), - backend_type=event.backend_type, - ref_type=ref_type, - native_id=produced_ref.native_id, - native_uri=produced_ref.native_uri, - provenance_id="prov_" - + hashlib.sha256(f"{event.id}|{ref_type.value}|{produced_ref.native_id}|{produced_ref.native_uri}".encode("utf-8")).hexdigest()[:24], - source_type="commit_session", - source_event_id=event.id, - status=BackendRefStatus.SUCCESS, - error_message=None, - metadata={ - "schema_version": "memory-gateway.commit-ref.v2", - "job_id": self._job_id_from_payload_ref(event.payload_ref), - "outbox_event_id": event.id, - "operation": event.operation.value, - "produced_ref_index": index, - "stable_key": stable_key, - "produced_ref": safe_produced_metadata, - "backend_result": self._backend_control_metadata(self._backend_result_to_dict(result)), - }, - ) - if existing: - ref.created_at = existing.created_at - return self.repo.save_memory_ref(ref) - - def _commit_ref_type(self, event: OutboxEvent, result: BackendCommitResult | BackendWriteResult) -> MemoryRefType: - requested = result.metadata.get("ref_type") if isinstance(result.metadata, dict) else None - if requested: - try: - return MemoryRefType(requested) - except ValueError: - pass - if event.backend_type == BackendType.OPENVIKING: - return MemoryRefType.SESSION_ARCHIVE - if event.backend_type == BackendType.EVEROS: - return MemoryRefType.LONG_TERM_MEMORY - return MemoryRefType.DRAFT_REVIEW - - def _job_id_from_payload_ref(self, payload_ref: str | None) -> str | None: - if payload_ref and payload_ref.startswith("commit_job:"): - return payload_ref.split(":", 1)[1] - return None - - def _aggregate_commit_job_status(self, events: list[OutboxEvent]) -> OperationStatus: - if not events: - return OperationStatus.FAILED - statuses = {event.status for event in events} - if statuses.issubset({OutboxEventStatus.SUCCESS, OutboxEventStatus.SKIPPED}): - return OperationStatus.SUCCESS - if statuses == {OutboxEventStatus.DEAD_LETTER} or statuses == {OutboxEventStatus.FAILED}: - return OperationStatus.FAILED - if OutboxEventStatus.PENDING in statuses or OutboxEventStatus.PROCESSING in statuses: - if OutboxEventStatus.SUCCESS in statuses: - return OperationStatus.PARTIAL_SUCCESS - return OperationStatus.FAILED - if OutboxEventStatus.SUCCESS in statuses: - return OperationStatus.PARTIAL_SUCCESS - return OperationStatus.FAILED - - def _commit_job_error_message(self, events: list[OutboxEvent]) -> str | None: - errors = [event.last_error for event in events if event.last_error] - return "; ".join(errors) if errors else None - - def _outbox_summary(self, events: list[OutboxEvent] | None = None) -> OutboxSummary: - events = events if events is not None else self.repo.list_outbox_events(limit=100000) - counts = {status: 0 for status in OutboxEventStatus} - for event in events: - counts[event.status] = counts.get(event.status, 0) + 1 - return OutboxSummary( - total_events=len(events), - pending_events=counts.get(OutboxEventStatus.PENDING, 0), - processing_events=counts.get(OutboxEventStatus.PROCESSING, 0), - success_events=counts.get(OutboxEventStatus.SUCCESS, 0), - skipped_events=counts.get(OutboxEventStatus.SKIPPED, 0), - dead_letter_events=counts.get(OutboxEventStatus.DEAD_LETTER, 0), - ) - - def _safe_control_metadata(self, metadata: dict[str, Any] | None) -> dict[str, Any]: - if not metadata: - return {} - blocked = {"content", "raw_request", "messages", "conversation", "transcript"} - safe: dict[str, Any] = {} - for key, value in metadata.items(): - if key in blocked: - continue - if isinstance(value, dict): - nested = self._safe_control_metadata(value) - if nested: - safe[key] = nested - elif isinstance(value, (str, int, float, bool)) or value is None: - safe[key] = value - elif isinstance(value, list): - safe[key] = [item for item in value if isinstance(item, (str, int, float, bool))] - return safe - - def _normalize_ingest_request(self, request: IngestRequest) -> IngestRequest: - data = request.model_copy(deep=True) - data.namespace = data.namespace.strip("/") - data.content = data.content.strip() - if not data.namespace: - raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail="namespace is required") - if not data.content: - raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail="content is required") - return data - - def _build_gateway_id(self, request: IngestRequest) -> str: - seed = self._idempotency_basis(request) - return "gw_" + hashlib.sha256(seed.encode("utf-8")).hexdigest()[:24] - - def _build_provenance_id(self, request: IngestRequest, gateway_id: str) -> str: - seed = f"{gateway_id}|{self._idempotency_basis(request)}" - return "prov_" + hashlib.sha256(seed.encode("utf-8")).hexdigest()[:24] - - def _idempotency_basis(self, request: IngestRequest) -> str: - if request.idempotency_key: - return f"idempotency_key:{request.workspace_id}:{request.idempotency_key}" - if request.source_event_id: - return f"source_event_id:{request.workspace_id}:{request.source_type}:{request.source_event_id}" - return f"turn:{request.workspace_id}:{request.session_id}:{request.turn_id}" - - def _content_hash(self, content: str) -> str: - return hashlib.sha256(content.encode("utf-8")).hexdigest() - - def _check_namespace_access(self, request: IngestRequest) -> None: - # TODO(v2): enforce workspace/user/agent namespace ACL tree. - if not request.workspace_id or not request.user_id or not request.agent_id: - raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="workspace, user, and agent are required") - - def _apply_safety_policy(self, request: IngestRequest) -> dict[str, Any]: - # TODO(v2): apply configurable redaction/safety rules before fan-out. - return { - "workspace_id": request.workspace_id, - "user_id": request.user_id, - "agent_id": request.agent_id, - "session_id": request.session_id, - "turn_id": request.turn_id, - "namespace": request.namespace, - "source_type": request.source_type, - "source_event_id": request.source_event_id, - "role": request.role, - "content": request.content, - "metadata": request.metadata, - "trace": request.trace.model_dump(mode="json"), - } - - async def _write_openviking_turn( - self, - request: IngestRequest, - payload: dict[str, Any], - gateway_id: str, - provenance_id: str, - content_hash: str, - ) -> MemoryRef: - try: - client = await self.openviking_client_factory() - if not hasattr(client, "ingest_conversation_turn"): - return self._save_ref( - request, - gateway_id, - provenance_id, - BackendType.OPENVIKING, - MemoryRefType.SESSION_ARCHIVE, - BackendRefStatus.SKIPPED, - content_hash=content_hash, - metadata=self._control_metadata(request, content_hash, {"reason": "adapter_method_missing"}), - ) - result = await client.ingest_conversation_turn(payload) - return self._ref_from_backend_result( - request, - gateway_id, - provenance_id, - BackendType.OPENVIKING, - MemoryRefType.SESSION_ARCHIVE, - result, - content_hash, - ) - except Exception as exc: # noqa: BLE001 - return self._save_ref( - request, - gateway_id, - provenance_id, - BackendType.OPENVIKING, - MemoryRefType.SESSION_ARCHIVE, - BackendRefStatus.FAILED, - content_hash=content_hash, - error_message=str(exc), - metadata=self._control_metadata(request, content_hash), - ) - - async def _write_everos_message( - self, - request: IngestRequest, - payload: dict[str, Any], - gateway_id: str, - provenance_id: str, - content_hash: str, - ) -> MemoryRef: - try: - client = self.everos_client or EverOSClient() - if not hasattr(client, "ingest_message"): - return self._save_ref( - request, - gateway_id, - provenance_id, - BackendType.EVEROS, - MemoryRefType.MESSAGE_MEMORY, - BackendRefStatus.SKIPPED, - content_hash=content_hash, - metadata=self._control_metadata(request, content_hash, {"reason": "adapter_method_missing"}), - ) - result = client.ingest_message(payload) - if hasattr(result, "__await__"): - result = await result - return self._ref_from_backend_result( - request, - gateway_id, - provenance_id, - BackendType.EVEROS, - MemoryRefType.MESSAGE_MEMORY, - result, - content_hash, - ) - except Exception as exc: # noqa: BLE001 - return self._save_ref( - request, - gateway_id, - provenance_id, - BackendType.EVEROS, - MemoryRefType.MESSAGE_MEMORY, - BackendRefStatus.FAILED, - content_hash=content_hash, - error_message=str(exc), - metadata=self._control_metadata(request, content_hash), - ) - - def _ref_from_backend_result( - self, - request: IngestRequest, - gateway_id: str, - provenance_id: str, - backend_type: BackendType, - ref_type: MemoryRefType, - result: Any, - content_hash: str, - ) -> MemoryRef: - data = self._backend_result_to_dict(result) - raw_status = str(data.get("status") or "success") - ref_status = BackendRefStatus.SUCCESS - if raw_status in {BackendRefStatus.PENDING.value, BackendRefStatus.FAILED.value, BackendRefStatus.SKIPPED.value}: - ref_status = BackendRefStatus(raw_status) - native_id = data.get("native_id") or data.get("id") or data.get("memory_id") or data.get("session_id") - native_uri = data.get("native_uri") or data.get("uri") or data.get("url") - return self._save_ref( - request, - gateway_id, - provenance_id, - backend_type, - ref_type, - ref_status, - native_id=native_id, - native_uri=native_uri, - error_message=data.get("error") or data.get("error_message"), - content_hash=content_hash, - metadata=self._control_metadata( - request, - content_hash, - {"backend_response": self._backend_control_metadata(data)}, - ), - ) - - def _save_ref( - self, - request: IngestRequest, - gateway_id: str, - provenance_id: str, - backend_type: BackendType, - ref_type: MemoryRefType, - ref_status: BackendRefStatus, - native_id: str | None = None, - native_uri: str | None = None, - error_message: str | None = None, - content_hash: str | None = None, - metadata: dict[str, Any] | None = None, - ) -> MemoryRef: - ref_id = self._memory_ref_id(gateway_id, backend_type, ref_type) - existing = self.repo.get_memory_ref(ref_id) - ref = MemoryRef( - id=ref_id, - gateway_id=gateway_id, - workspace_id=request.workspace_id, - user_id=request.user_id, - agent_id=request.agent_id, - session_id=request.session_id, - turn_id=request.turn_id, - namespace=request.namespace, - backend_type=backend_type, - ref_type=ref_type, - native_id=native_id, - native_uri=native_uri, - provenance_id=provenance_id, - idempotency_key=request.idempotency_key, - content_hash=content_hash, - source_type=request.source_type, - source_event_id=request.source_event_id, - status=ref_status, - error_message=error_message, - metadata=metadata or {}, - ) - if existing: - ref.created_at = existing.created_at - return self.repo.save_memory_ref(ref) - - def _memory_ref_id( - self, - gateway_id: str, - backend_type: BackendType, - ref_type: MemoryRefType, - native_id: str | None = None, - native_uri: str | None = None, - stable_key: str | None = None, - ) -> str: - native_key = f"|{native_id or ''}|{native_uri or ''}|{stable_key or ''}" if native_id or native_uri or stable_key else "" - seed = f"{gateway_id}|{backend_type.value}|{ref_type.value}{native_key}" - return "ref_" + hashlib.sha256(seed.encode("utf-8")).hexdigest()[:24] - - def _produced_ref_stable_key(self, produced_ref: BackendProducedRef, index: int) -> str | None: - for key in ("stable_key", "backend_ref_key", "idempotency_key"): - value = produced_ref.metadata.get(key) - if isinstance(value, str) and value: - return value - if not produced_ref.native_id and not produced_ref.native_uri: - return f"produced_ref_index:{index}" - return None - - def _aggregate_ref_status(self, refs: list[MemoryRef]) -> OperationStatus: - if not refs: - return OperationStatus.SKIPPED - statuses = {ref.status for ref in refs} - if statuses == {BackendRefStatus.SUCCESS}: - return OperationStatus.SUCCESS - if BackendRefStatus.SUCCESS in statuses and (BackendRefStatus.FAILED in statuses or BackendRefStatus.SKIPPED in statuses): - return OperationStatus.PARTIAL_SUCCESS - if statuses == {BackendRefStatus.FAILED}: - return OperationStatus.FAILED - if BackendRefStatus.FAILED in statuses: - return OperationStatus.PARTIAL_SUCCESS - if statuses == {BackendRefStatus.SKIPPED}: - return OperationStatus.SKIPPED - return OperationStatus.PENDING - - def _view_refs(self, refs: list[MemoryRef]) -> list[MemoryRefView]: - return [MemoryRefView.model_validate(ref.model_dump(mode="json")) for ref in refs] - - def _backend_control_metadata(self, data: dict[str, Any]) -> dict[str, Any]: - allowed_keys = { - "status", - "reason", - "native_id", - "native_uri", - "id", - "uri", - "url", - "memory_id", - "session_id", - "retryable", - "error_code", - "error", - "error_message", - "latency_ms", - } - metadata = {key: value for key, value in data.items() if key in allowed_keys} - nested_metadata = data.get("metadata") - if isinstance(nested_metadata, dict): - for key in ("reason", "backend_request_id", "latency_ms", "schema_version"): - if key in nested_metadata: - metadata[key] = nested_metadata[key] - return metadata - - def _control_metadata( - self, - request: IngestRequest, - content_hash: str, - extra: dict[str, Any] | None = None, - ) -> dict[str, Any]: - metadata: dict[str, Any] = { - "schema_version": "memory-gateway.control-ref.v2", - "idempotency_basis": self._idempotency_basis(request), - "content_hash": content_hash, - } - source_channel = request.metadata.get("source_channel") or request.metadata.get("channel") - if source_channel: - metadata["source_channel"] = source_channel - if request.trace.trace_id: - metadata["trace_id"] = request.trace.trace_id - if request.trace.request_id: - metadata["trace_request_id"] = request.trace.request_id - if extra: - metadata.update(extra) - return metadata - - def _backend_result_to_dict(self, result: Any) -> dict[str, Any]: - if isinstance(result, BackendWriteResult): - data = result.model_dump(mode="json") - if result.status == BackendResultStatus.FAILED and result.retryable: - data["retryable"] = True - return data - if hasattr(result, "model_dump"): - return result.model_dump(mode="json") - return result if isinstance(result, dict) else {"raw": str(result)} - - def _commit_gateway_id(self, session_id: str, request: CommitRequest) -> str: - basis = request.idempotency_key or request.request_id or session_id - seed = f"commit:{request.workspace_id}:{session_id}:{basis}" - return "gwc_" + hashlib.sha256(seed.encode("utf-8")).hexdigest()[:24] - - def _create_commit_outbox_events(self, gateway_id: str, job: CommitJob, request: CommitRequest) -> None: - metadata = { - "schema_version": "memory-gateway.outbox.v2", - "job_id": job.job_id, - "namespace": request.namespace, - "idempotency_key": request.idempotency_key, - "request_id": request.request_id, - } - for backend_type in (BackendType.OPENVIKING, BackendType.EVEROS): - event = OutboxEvent( - id=self._outbox_event_id(gateway_id, backend_type, BackendOperation.COMMIT_SESSION), - event_type="commit_session", - gateway_id=gateway_id, - workspace_id=request.workspace_id, - user_id=request.user_id, - agent_id=request.agent_id, - session_id=job.session_id, - backend_type=backend_type, - operation=BackendOperation.COMMIT_SESSION, - payload_ref=f"commit_job:{job.job_id}", - metadata=metadata, - ) - self.repo.save_outbox_event(event) - - def _outbox_event_id(self, gateway_id: str, backend_type: BackendType, operation: BackendOperation) -> str: - seed = f"{gateway_id}|{backend_type.value}|{operation.value}" - return "outbox_" + hashlib.sha256(seed.encode("utf-8")).hexdigest()[:24] - - -v2_service = MemoryGatewayV2Service() diff --git a/memory_gateway/skills/__init__.py b/memory_gateway/skills/__init__.py deleted file mode 100644 index 072e22b..0000000 --- a/memory_gateway/skills/__init__.py +++ /dev/null @@ -1,2 +0,0 @@ -"""Skill skeletons for Memory Gateway processing units.""" - diff --git a/memory_gateway/skills/base.py b/memory_gateway/skills/base.py deleted file mode 100644 index a028047..0000000 --- a/memory_gateway/skills/base.py +++ /dev/null @@ -1,21 +0,0 @@ -"""Shared skill contracts.""" -from __future__ import annotations - -from dataclasses import dataclass, field -from typing import Any - - -@dataclass -class SkillResult: - status: str - output: dict[str, Any] = field(default_factory=dict) - writes_long_term_memory: bool = False - - -class MemorySkill: - name = "memory_skill" - writes_long_term_memory = False - - async def run(self, payload: dict[str, Any]) -> SkillResult: - raise NotImplementedError - diff --git a/memory_gateway/skills/classify_memory_skill.py b/memory_gateway/skills/classify_memory_skill.py deleted file mode 100644 index cbb7d74..0000000 --- a/memory_gateway/skills/classify_memory_skill.py +++ /dev/null @@ -1,9 +0,0 @@ -from .base import MemorySkill, SkillResult - - -class ClassifyMemorySkill(MemorySkill): - name = "classify_memory_skill" - - async def run(self, payload: dict) -> SkillResult: - return SkillResult(status="ok", output={"memory_type": payload.get("memory_type", "fact"), "visibility": payload.get("visibility", "private")}) - diff --git a/memory_gateway/skills/commit_memory_skill.py b/memory_gateway/skills/commit_memory_skill.py deleted file mode 100644 index 13fb405..0000000 --- a/memory_gateway/skills/commit_memory_skill.py +++ /dev/null @@ -1,10 +0,0 @@ -from .base import MemorySkill, SkillResult - - -class CommitMemorySkill(MemorySkill): - name = "commit_memory_skill" - writes_long_term_memory = True - - async def run(self, payload: dict) -> SkillResult: - return SkillResult(status="ok", output={"committed": payload}, writes_long_term_memory=True) - diff --git a/memory_gateway/skills/export_to_obsidian_skill.py b/memory_gateway/skills/export_to_obsidian_skill.py deleted file mode 100644 index cbd2204..0000000 --- a/memory_gateway/skills/export_to_obsidian_skill.py +++ /dev/null @@ -1,9 +0,0 @@ -from .base import MemorySkill, SkillResult - - -class ExportToObsidianSkill(MemorySkill): - name = "export_to_obsidian_skill" - - async def run(self, payload: dict) -> SkillResult: - return SkillResult(status="ok", output={"draft_path": payload.get("draft_path")}) - diff --git a/memory_gateway/skills/extract_memory_skill.py b/memory_gateway/skills/extract_memory_skill.py deleted file mode 100644 index 2a0692c..0000000 --- a/memory_gateway/skills/extract_memory_skill.py +++ /dev/null @@ -1,11 +0,0 @@ -from .base import MemorySkill, SkillResult - - -class ExtractMemorySkill(MemorySkill): - name = "extract_memory_skill" - - async def run(self, payload: dict) -> SkillResult: - text = payload.get("content", "") - candidates = [{"content": text, "confidence": 0.5}] if text else [] - return SkillResult(status="ok", output={"candidates": candidates}) - diff --git a/memory_gateway/skills/import_from_obsidian_skill.py b/memory_gateway/skills/import_from_obsidian_skill.py deleted file mode 100644 index c30e80b..0000000 --- a/memory_gateway/skills/import_from_obsidian_skill.py +++ /dev/null @@ -1,10 +0,0 @@ -from .base import MemorySkill, SkillResult - - -class ImportFromObsidianSkill(MemorySkill): - name = "import_from_obsidian_skill" - writes_long_term_memory = True - - async def run(self, payload: dict) -> SkillResult: - return SkillResult(status="ok", output={"imported_path": payload.get("path")}, writes_long_term_memory=True) - diff --git a/memory_gateway/skills/ingest_skill.py b/memory_gateway/skills/ingest_skill.py deleted file mode 100644 index 58dabdb..0000000 --- a/memory_gateway/skills/ingest_skill.py +++ /dev/null @@ -1,9 +0,0 @@ -from .base import MemorySkill, SkillResult - - -class IngestSkill(MemorySkill): - name = "ingest_skill" - - async def run(self, payload: dict) -> SkillResult: - return SkillResult(status="ok", output={"normalized": payload}) - diff --git a/memory_gateway/skills/merge_memory_skill.py b/memory_gateway/skills/merge_memory_skill.py deleted file mode 100644 index 3199322..0000000 --- a/memory_gateway/skills/merge_memory_skill.py +++ /dev/null @@ -1,10 +0,0 @@ -from .base import MemorySkill, SkillResult - - -class MergeMemorySkill(MemorySkill): - name = "merge_memory_skill" - writes_long_term_memory = True - - async def run(self, payload: dict) -> SkillResult: - return SkillResult(status="ok", output={"merged": payload.get("memory_ids", [])}, writes_long_term_memory=True) - diff --git a/memory_gateway/skills/prune_memory_skill.py b/memory_gateway/skills/prune_memory_skill.py deleted file mode 100644 index 0554b74..0000000 --- a/memory_gateway/skills/prune_memory_skill.py +++ /dev/null @@ -1,10 +0,0 @@ -from .base import MemorySkill, SkillResult - - -class PruneMemorySkill(MemorySkill): - name = "prune_memory_skill" - writes_long_term_memory = True - - async def run(self, payload: dict) -> SkillResult: - return SkillResult(status="ok", output={"pruned": payload.get("memory_ids", [])}, writes_long_term_memory=True) - diff --git a/memory_gateway/skills/retrieve_context_skill.py b/memory_gateway/skills/retrieve_context_skill.py deleted file mode 100644 index 5e3dcc0..0000000 --- a/memory_gateway/skills/retrieve_context_skill.py +++ /dev/null @@ -1,9 +0,0 @@ -from .base import MemorySkill, SkillResult - - -class RetrieveContextSkill(MemorySkill): - name = "retrieve_context_skill" - - async def run(self, payload: dict) -> SkillResult: - return SkillResult(status="ok", output={"query": payload.get("query"), "contexts": []}) - diff --git a/memory_gateway/skills/summarize_episode_skill.py b/memory_gateway/skills/summarize_episode_skill.py deleted file mode 100644 index d6e0ad0..0000000 --- a/memory_gateway/skills/summarize_episode_skill.py +++ /dev/null @@ -1,10 +0,0 @@ -from .base import MemorySkill, SkillResult - - -class SummarizeEpisodeSkill(MemorySkill): - name = "summarize_episode_skill" - - async def run(self, payload: dict) -> SkillResult: - content = payload.get("content", "") - return SkillResult(status="ok", output={"summary": content[:500]}) - diff --git a/memory_gateway/types.py b/memory_gateway/types.py deleted file mode 100644 index 1ff29ba..0000000 --- a/memory_gateway/types.py +++ /dev/null @@ -1,158 +0,0 @@ -"""类型定义""" -from typing import Optional, Any, Literal -from pydantic import BaseModel, Field - - -class ServerConfig(BaseModel): - """服务器配置""" - host: str = "0.0.0.0" - port: int = 1934 - api_key: str = "" - - -class OpenVikingConfig(BaseModel): - """OpenViking 后端配置""" - enabled: bool = False - mode: Literal["offline", "skeleton", "real"] = "offline" - url: str = "http://localhost:1933" - api_key: str = "" - timeout: int = 30 - verify_ssl: bool = True - ingest_path: str = "/api/v1/sessions/{session_id}/messages" - - -class EverOSConfig(BaseModel): - """External EverOS memory service configuration.""" - enabled: bool = False - mode: Literal["offline", "skeleton", "real"] = "offline" - url: str = "http://127.0.0.1:1995" - api_key: str = "" - timeout: int = 30 - verify_ssl: bool = True - health_path: str = "/health" - ingest_path: str = "/api/v1/memories" - search_path: str = "/api/v1/memories/search" - flush_path: str = "/api/v1/memories/flush" - retrieve_method: Literal["keyword", "vector", "hybrid", "rrf", "agentic"] = "keyword" - -class MemoryConfig(BaseModel): - """记忆配置""" - default_namespace: str = "memory-gateway" - search_limit: int = 10 - - -class LLMConfig(BaseModel): - """LLM 配置，用于通用总结和知识沉淀。""" - base_url: str = "https://api.openai.com/v1" - api_key: str = "" - model: str = "" - timeout: int = 60 - max_input_chars: int = 24000 - - -class ObsidianConfig(BaseModel): - """Obsidian Vault 配置。""" - vault_path: str = "/home/tom/memory-gateway/obsidian-vault" - knowledge_dir: str = "01_Knowledge/Uploaded" - review_dir: str = "Reviews/Queue" - - -class StorageConfig(BaseModel): - """Metadata storage configuration.""" - backend: Literal["sqlite", "memory"] = "sqlite" - sqlite_path: str = "/home/tom/memory-gateway/memory_gateway.sqlite3" - - -class LoggingConfig(BaseModel): - """日志配置""" - level: str = "INFO" - format: str = "%(asctime)s - %(name)s - %(levelname)s - %(message)s" - - -class Config(BaseModel): - """完整配置""" - def __init__(self, **data: Any) -> None: - super().__init__(**data) - - server: ServerConfig = Field(default_factory=ServerConfig) - openviking: OpenVikingConfig = Field(default_factory=OpenVikingConfig) - everos: EverOSConfig = Field(default_factory=EverOSConfig) - memory: MemoryConfig = Field(default_factory=MemoryConfig) - logging: LoggingConfig = Field(default_factory=LoggingConfig) - llm: LLMConfig = Field(default_factory=LLMConfig) - obsidian: ObsidianConfig = Field(default_factory=ObsidianConfig) - storage: StorageConfig = Field(default_factory=StorageConfig) - -class SearchRequest(BaseModel): - """搜索请求""" - query: str - namespace: Optional[str] = None - limit: Optional[int] = None - uri: Optional[str] = None - - -class AddMemoryRequest(BaseModel): - """添加记忆请求""" - content: str - namespace: Optional[str] = None - memory_type: Optional[str] = "general" - - -class AddResourceRequest(BaseModel): - """添加资源请求""" - uri: str - content: str - resource_type: Optional[str] = "text" - - - - -class CommitSummaryRequest(BaseModel): - """通用总结与沉淀请求。 - - 该模型用于任意场景把一段高价值内容总结后 - 写入 OpenViking memory、resource，或两者同时写入。 - """ - content: str - title: Optional[str] = None - summary: Optional[str] = None - purpose: Optional[str] = "generic knowledge memory" - namespace: Optional[str] = None - memory_type: Optional[str] = "summary" - tags: list[str] = Field(default_factory=list) - source: Optional[str] = None - resource_uri: Optional[str] = None - resource_type: Optional[str] = "json" - persist_as: Literal["memory", "resource", "both", "none"] = "both" - max_summary_chars: int = 600 - - -class CommitSummaryResponse(BaseModel): - """通用总结与沉淀响应。""" - status: str - artifact: dict[str, Any] - memory_result: Optional[dict[str, Any]] = None - resource_result: Optional[dict[str, Any]] = None - - -class SearchResult(BaseModel): - """搜索结果""" - results: list[dict[str, Any]] - total: int - - -class MemoryEntry(BaseModel): - """记忆条目""" - id: str - content: str - namespace: str - memory_type: str - created_at: Optional[str] = None - - -class ResourceEntry(BaseModel): - """资源条目""" - uri: str - content: str - resource_type: str - created_at: Optional[str] = None diff --git a/memory_gateway/worker_v2.py b/memory_gateway/worker_v2.py deleted file mode 100644 index f2b782f..0000000 --- a/memory_gateway/worker_v2.py +++ /dev/null @@ -1,49 +0,0 @@ -"""Lightweight v2 outbox worker entrypoint. - -Usage: - python -m memory_gateway.worker_v2 --limit 100 --worker-id local-worker --lease-seconds 300 -""" -from __future__ import annotations - -import argparse -import asyncio -import json -from typing import Sequence -from uuid import uuid4 - -from .services_v2 import v2_service - - -def build_parser() -> argparse.ArgumentParser: - parser = argparse.ArgumentParser(description="Process Memory Gateway v2 outbox events once.") - parser.add_argument("--limit", type=int, default=100, help="Maximum pending events to claim and process.") - parser.add_argument("--worker-id", default=None, help="Stable worker id recorded in outbox lease fields.") - parser.add_argument("--lease-seconds", type=int, default=300, help="Lease duration for claimed events.") - return parser - - -async def run_once(limit: int, worker_id: str | None, lease_seconds: int) -> dict[str, object]: - worker_id = worker_id or f"worker_{uuid4().hex[:12]}" - response = await v2_service.process_pending_outbox_events_summary( - limit=limit, - worker_id=worker_id, - lease_seconds=lease_seconds, - ) - return response.model_dump(mode="json") - - -def main(argv: Sequence[str] | None = None) -> int: - args = build_parser().parse_args(argv) - payload = asyncio.run( - run_once( - limit=args.limit, - worker_id=args.worker_id, - lease_seconds=args.lease_seconds, - ) - ) - print(json.dumps(payload, ensure_ascii=False, sort_keys=True)) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/memory_system_api/__init__.py b/memory_system_api/__init__.py new file mode 100644 index 0000000..a61649f --- /dev/null +++ b/memory_system_api/__init__.py @@ -0,0 +1 @@ +"""Lightweight Memory System API package.""" diff --git a/memory_system_api/api.py b/memory_system_api/api.py new file mode 100644 index 0000000..b31a030 --- /dev/null +++ b/memory_system_api/api.py @@ -0,0 +1,69 @@ +"""FastAPI router for the lightweight Memory System API.""" +from __future__ import annotations + +from fastapi import APIRouter, Depends, HTTPException, Query, status + +from .auth import verify_api_key +from .schemas import MessageIngestRequest, SearchRequest, SessionUserRequest +from .service import MemorySystemService + + +router = APIRouter( + prefix="/memory-system", + tags=["memory-system"], + dependencies=[Depends(verify_api_key)], +) + + +def get_service() -> MemorySystemService: + return MemorySystemService() + + +@router.get("/health") +async def health(service: MemorySystemService = Depends(get_service)): + return await service.health() + + +@router.post("/messages") +async def ingest_messages(request: MessageIngestRequest, service: MemorySystemService = Depends(get_service)): + try: + return await service.ingest_messages(request) + except ValueError as exc: + raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail=str(exc)) from exc + + +@router.post("/sessions/{session_id}/commit") +async def commit_session( + session_id: str, + request: SessionUserRequest, + service: MemorySystemService = Depends(get_service), +): + return await service.commit_session(request.user_id, session_id) + + +@router.post("/sessions/{session_id}/extract") +async def extract_session( + session_id: str, + request: SessionUserRequest, + service: MemorySystemService = Depends(get_service), +): + return await service.extract_session(request.user_id, session_id) + + +@router.get("/openviking/tasks/{task_id}") +async def get_openviking_task( + task_id: str, + user_id: str = Query(min_length=1), + service: MemorySystemService = Depends(get_service), +): + return await service.get_openviking_task(user_id, task_id) + + +@router.post("/search") +async def search(request: SearchRequest, service: MemorySystemService = Depends(get_service)): + return await service.search(request) + + +@router.get("/users/{user_id}/profile") +async def get_profile(user_id: str, service: MemorySystemService = Depends(get_service)): + return await service.get_profile(user_id) diff --git a/memory_system_api/auth.py b/memory_system_api/auth.py new file mode 100644 index 0000000..e76b395 --- /dev/null +++ b/memory_system_api/auth.py @@ -0,0 +1,12 @@ +"""API key auth for Memory System API.""" +from __future__ import annotations + +from fastapi import Header, HTTPException, status + +from .config import get_config + + +def verify_api_key(x_api_key: str | None = Header(default=None)) -> None: + expected = get_config().server.api_key + if expected and x_api_key != expected: + raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid API key") diff --git a/memory_system_api/clients.py b/memory_system_api/clients.py new file mode 100644 index 0000000..4ee0232 --- /dev/null +++ b/memory_system_api/clients.py @@ -0,0 +1,210 @@ +"""Async clients for OpenViking and EverOS used by the lightweight API.""" +from __future__ import annotations + +from datetime import datetime, timezone +from typing import Any + +import httpx + +from .config import get_config +from .store import OpenVikingUserKeyStore + + +class OpenVikingMemorySystemClient: + def __init__(self, store: OpenVikingUserKeyStore | None = None) -> None: + config = get_config() + self.base_url = config.openviking.url.rstrip("/") + self.root_key = config.openviking.api_key or "your-secret-root-key" + self.timeout = config.openviking.timeout + self.verify_ssl = config.openviking.verify_ssl + self.store = store or OpenVikingUserKeyStore(config.storage.sqlite_path) + + async def health(self) -> dict[str, Any]: + async with self._client(self.root_key) as client: + response = await client.get("/health") + response.raise_for_status() + return response.json() + + async def ensure_user(self, user_id: str) -> str: + existing = self.store.get_user_key(user_id) + if existing: + return existing + + async with self._client(self.root_key) as client: + response = await client.post( + "/api/v1/admin/accounts", + json={"account_id": user_id, "admin_user_id": user_id}, + ) + response.raise_for_status() + data = response.json() + + user_key = self._extract_user_key(data) + if not user_key: + raise RuntimeError("OpenViking did not return user_key") + self.store.save_user_key(user_id, user_key) + return user_key + + async def ensure_session(self, user_key: str, session_id: str) -> dict[str, Any]: + async with self._client(user_key) as client: + response = await client.post("/api/v1/sessions", json={"session_id": session_id}) + if response.status_code in {409, 422}: + return {"session_id": session_id, "status": "exists"} + response.raise_for_status() + return response.json() + + async def append_message(self, user_key: str, session_id: str, role: str, content: str) -> dict[str, Any]: + async with self._client(user_key) as client: + response = await client.post( + f"/api/v1/sessions/{session_id}/messages", + json={"role": role, "content": content}, + ) + response.raise_for_status() + return response.json() + + async def commit_session(self, user_key: str, session_id: str) -> dict[str, Any]: + async with self._client(user_key) as client: + response = await client.post(f"/api/v1/sessions/{session_id}/commit") + response.raise_for_status() + return response.json() + + async def extract_session(self, user_key: str, session_id: str) -> dict[str, Any]: + async with self._client(user_key) as client: + response = await client.post(f"/api/v1/sessions/{session_id}/extract") + response.raise_for_status() + return response.json() + + async def get_task(self, user_key: str, task_id: str) -> dict[str, Any]: + async with self._client(user_key) as client: + response = await client.get(f"/api/v1/tasks/{task_id}") + response.raise_for_status() + return response.json() + + async def find(self, user_key: str, user_id: str, query: str, limit: int) -> dict[str, Any]: + async with self._client(user_key) as client: + response = await client.post( + "/api/v1/search/find", + json={ + "query": query, + "target_uri": f"viking://user/{user_id}/memories/", + "limit": limit, + }, + ) + response.raise_for_status() + return response.json() + + async def search(self, user_key: str, session_id: str | None, query: str, limit: int) -> dict[str, Any]: + payload: dict[str, Any] = {"query": query, "limit": limit} + if session_id: + payload["session_id"] = session_id + async with self._client(user_key) as client: + response = await client.post("/api/v1/search/search", json=payload) + response.raise_for_status() + return response.json() + + def _client(self, api_key: str) -> httpx.AsyncClient: + return httpx.AsyncClient( + base_url=self.base_url, + headers={"X-API-Key": api_key, "Content-Type": "application/json"}, + timeout=self.timeout, + verify=self.verify_ssl, + ) + + def _extract_user_key(self, data: dict[str, Any]) -> str | None: + result = data.get("result") if isinstance(data.get("result"), dict) else data + value = result.get("user_key") if isinstance(result, dict) else None + return str(value) if value else None + + +class EverOSMemorySystemClient: + def __init__(self) -> None: + config = get_config() + self.base_url = config.everos.url.rstrip("/") + self.api_key = config.everos.api_key + self.timeout = config.everos.timeout + self.verify_ssl = config.everos.verify_ssl + self.health_path = config.everos.health_path + + async def health(self) -> dict[str, Any]: + async with self._client() as client: + response = await client.get(self.health_path) + response.raise_for_status() + return response.json() + + async def append_message(self, user_id: str, session_id: str, role: str, content: str) -> dict[str, Any]: + async with self._client() as client: + response = await client.post( + "/api/v1/memories", + json=self.build_message_payload(user_id=user_id, session_id=session_id, role=role, content=content), + ) + response.raise_for_status() + return response.json() + + def build_message_payload(self, user_id: str, session_id: str, role: str, content: str) -> dict[str, Any]: + everos_role = "assistant" if role == "assistant" else "user" + sender_id = "assistant" if everos_role == "assistant" else user_id + timestamp = int(datetime.now(timezone.utc).timestamp() * 1000) + return { + "user_id": user_id, + "session_id": session_id, + "messages": [ + { + "message_id": f"msg_{timestamp}", + "timestamp": timestamp, + "sender_id": sender_id, + "sender_name": sender_id, + "role": everos_role, + "content": content, + } + ], + } + + async def flush(self, user_id: str, session_id: str) -> dict[str, Any]: + async with self._client() as client: + response = await client.post("/api/v1/memories/flush", json={"user_id": user_id, "session_id": session_id}) + response.raise_for_status() + return response.json() + + async def search(self, user_id: str, session_id: str | None, query: str, method: str, limit: int) -> dict[str, Any]: + filters: dict[str, Any] = {"user_id": user_id} + if session_id: + filters["session_id"] = session_id + async with self._client() as client: + response = await client.post( + "/api/v1/memories/search", + json={ + "query": query, + "method": method, + "memory_types": ["episodic_memory", "profile", "raw_message"], + "filters": filters, + "top_k": limit, + "include_original_data": True, + }, + ) + response.raise_for_status() + return response.json() + + async def get_profile(self, user_id: str) -> dict[str, Any]: + async with self._client() as client: + response = await client.post( + "/api/v1/memories/get", + json={ + "memory_type": "profile", + "filters": {"user_id": user_id}, + "page": 1, + "page_size": 20, + }, + ) + response.raise_for_status() + return response.json() + + def _client(self) -> httpx.AsyncClient: + headers = {"Content-Type": "application/json"} + if self.api_key: + headers["X-API-Key"] = self.api_key + headers["Authorization"] = f"Bearer {self.api_key}" + return httpx.AsyncClient( + base_url=self.base_url, + headers=headers, + timeout=self.timeout, + verify=self.verify_ssl, + ) diff --git a/memory_system_api/config.py b/memory_system_api/config.py new file mode 100644 index 0000000..6b74a51 --- /dev/null +++ b/memory_system_api/config.py @@ -0,0 +1,100 @@ +"""Configuration loading for Memory System API.""" +from __future__ import annotations + +import os +from pathlib import Path +from typing import Any, Literal + +import yaml +from pydantic import BaseModel, Field + + +class ServerConfig(BaseModel): + host: str = "127.0.0.1" + port: int = 1934 + api_key: str = "" + + +class OpenVikingConfig(BaseModel): + url: str = "http://127.0.0.1:1933" + api_key: str = "" + timeout: int = 30 + verify_ssl: bool = True + + +class EverOSConfig(BaseModel): + url: str = "http://127.0.0.1:1995" + api_key: str = "" + timeout: int = 30 + verify_ssl: bool = True + health_path: str = "/health" + + +class StorageConfig(BaseModel): + sqlite_path: str = "/home/tom/memory-gateway/memory_system_api.sqlite3" + + +class LoggingConfig(BaseModel): + level: str = "INFO" + + +class Config(BaseModel): + server: ServerConfig = Field(default_factory=ServerConfig) + openviking: OpenVikingConfig = Field(default_factory=OpenVikingConfig) + everos: EverOSConfig = Field(default_factory=EverOSConfig) + storage: StorageConfig = Field(default_factory=StorageConfig) + logging: LoggingConfig = Field(default_factory=LoggingConfig) + + +_config: Config | None = None + + +def load_config(config_path: str | None = None) -> Config: + path = Path(config_path or os.environ.get("MEMORY_SYSTEM_CONFIG", "config.yaml")) + if not path.exists(): + return _apply_env_overrides(Config()) + with path.open("r", encoding="utf-8") as handle: + data = yaml.safe_load(handle) or {} + config = Config( + server=ServerConfig(**data.get("server", {})), + openviking=OpenVikingConfig(**data.get("openviking", {})), + everos=EverOSConfig(**data.get("everos", {})), + storage=StorageConfig(**data.get("storage", {})), + logging=LoggingConfig(**data.get("logging", {})), + ) + return _apply_env_overrides(config) + + +def get_config() -> Config: + global _config + if _config is None: + _config = load_config() + return _config + + +def set_config(config: Config) -> None: + global _config + _config = config + + +def _apply_env_overrides(config: Config) -> Config: + updates: dict[str, dict[str, Any]] = { + "server": _env_updates("MEMORY_SYSTEM_SERVER", {"API_KEY": "api_key", "HOST": "host", "PORT": "port"}), + "openviking": _env_updates("OPENVIKING", {"URL": "url", "BASE_URL": "url", "API_KEY": "api_key", "TIMEOUT": "timeout"}), + "everos": _env_updates("EVEROS", {"URL": "url", "BASE_URL": "url", "API_KEY": "api_key", "TIMEOUT": "timeout"}), + "storage": _env_updates("MEMORY_SYSTEM_STORAGE", {"SQLITE_PATH": "sqlite_path"}), + } + for section, values in updates.items(): + if values: + setattr(config, section, getattr(config, section).model_copy(update=values)) + return config + + +def _env_updates(prefix: str, mapping: dict[str, str]) -> dict[str, Any]: + values: dict[str, Any] = {} + for env_name, field_name in mapping.items(): + raw = os.environ.get(f"{prefix}_{env_name}") + if raw is None: + continue + values[field_name] = int(raw) if field_name in {"port", "timeout"} else raw + return values diff --git a/memory_system_api/schemas.py b/memory_system_api/schemas.py new file mode 100644 index 0000000..b606f05 --- /dev/null +++ b/memory_system_api/schemas.py @@ -0,0 +1,64 @@ +"""Schemas for the lightweight Memory System API.""" +from __future__ import annotations + +from typing import Any, Literal + +from pydantic import BaseModel, Field + + +OperationStatus = Literal["success", "partial_success", "failed"] + + +class MessageIngestRequest(BaseModel): + user_id: str = Field(min_length=1) + session_id: str = Field(min_length=1) + user_message: str | None = None + assistant_message: str | None = None + timestamp: int | None = None + metadata: dict[str, Any] = Field(default_factory=dict) + + +class SessionUserRequest(BaseModel): + user_id: str = Field(min_length=1) + + +class SearchRequest(BaseModel): + user_id: str = Field(min_length=1) + session_id: str | None = None + query: str = Field(min_length=1) + use_llm: bool = False + limit: int = Field(default=10, ge=1, le=100) + + +class BackendStatus(BaseModel): + status: OperationStatus + result: Any = None + error: str | None = None + + +class MessageIngestResponse(BaseModel): + status: OperationStatus + message_count: int + backends: dict[str, BackendStatus] + + +class CommitResponse(BaseModel): + status: OperationStatus + backends: dict[str, BackendStatus] + + +class ExtractResponse(BaseModel): + status: OperationStatus + backends: dict[str, BackendStatus] + + +class SearchResponse(BaseModel): + status: OperationStatus + items: list[dict[str, Any]] = Field(default_factory=list) + backends: dict[str, BackendStatus] + + +class ProfileResponse(BaseModel): + status: OperationStatus + profile: Any = None + backends: dict[str, BackendStatus] diff --git a/memory_system_api/server.py b/memory_system_api/server.py new file mode 100644 index 0000000..eadd6c8 --- /dev/null +++ b/memory_system_api/server.py @@ -0,0 +1,49 @@ +"""Standalone FastAPI server for Memory System API.""" +from __future__ import annotations + +from fastapi import FastAPI +from fastapi.middleware.cors import CORSMiddleware + +from .api import router +from .config import Config, load_config, set_config + + +app = FastAPI(title="Memory System API", version="0.1.0") +app.add_middleware( + CORSMiddleware, + allow_origins=["*"], + allow_credentials=True, + allow_methods=["*"], + allow_headers=["*"], +) +app.include_router(router) + + +def create_app(config: Config | None = None) -> FastAPI: + if config: + set_config(config) + return app + + +def main() -> None: + import argparse + import uvicorn + + parser = argparse.ArgumentParser(description="Memory System API") + parser.add_argument("--config", default="config.yaml", help="Config file path") + parser.add_argument("--host", default=None, help="Bind host") + parser.add_argument("--port", type=int, default=None, help="Bind port") + args = parser.parse_args() + + config = load_config(args.config) + if args.host: + config.server.host = args.host + if args.port: + config.server.port = args.port + set_config(config) + + uvicorn.run(app, host=config.server.host, port=config.server.port, log_level=config.logging.level.lower()) + + +if __name__ == "__main__": + main() diff --git a/memory_system_api/service.py b/memory_system_api/service.py new file mode 100644 index 0000000..053fcfb --- /dev/null +++ b/memory_system_api/service.py @@ -0,0 +1,165 @@ +"""Orchestration for the lightweight Memory System API.""" +from __future__ import annotations + +import asyncio +from typing import Any, Awaitable, Callable + +from .clients import EverOSMemorySystemClient, OpenVikingMemorySystemClient +from .schemas import ( + BackendStatus, + CommitResponse, + ExtractResponse, + MessageIngestRequest, + MessageIngestResponse, + ProfileResponse, + SearchRequest, + SearchResponse, +) + + +class MemorySystemService: + def __init__(self, openviking: Any | None = None, everos: Any | None = None) -> None: + self.openviking = openviking or OpenVikingMemorySystemClient() + self.everos = everos or EverOSMemorySystemClient() + + async def ingest_messages(self, request: MessageIngestRequest) -> MessageIngestResponse: + messages = self._messages_from_request(request) + if not messages: + raise ValueError("at least one message is required") + + user_key = await self.openviking.ensure_user(request.user_id) + await self.openviking.ensure_session(user_key, request.session_id) + + print("user_key:", user_key) # Debugging line to check the user_key value + + async def write_openviking() -> list[dict[str, Any]]: + results = [] + for message in messages: + results.append( + await self.openviking.append_message(user_key, request.session_id, message["role"], message["content"]) + ) + return results + + async def write_everos() -> list[dict[str, Any]]: + results = [] + for message in messages: + results.append( + await self.everos.append_message(request.user_id, request.session_id, message["role"], message["content"]) + ) + return results + + backends = await self._run_backends(openviking=write_openviking, everos=write_everos) + return MessageIngestResponse( + status=self._aggregate_status(backends), + message_count=len(messages), + backends=backends, + ) + + async def commit_session(self, user_id: str, session_id: str) -> CommitResponse: + user_key = await self.openviking.ensure_user(user_id) + + async def commit_openviking() -> dict[str, Any]: + return await self.openviking.commit_session(user_key, session_id) + + async def flush_everos() -> dict[str, Any]: + return await self.everos.flush(user_id, session_id) + + backends = await self._run_backends(openviking=commit_openviking, everos=flush_everos) + return CommitResponse(status=self._aggregate_status(backends), backends=backends) + + async def extract_session(self, user_id: str, session_id: str) -> ExtractResponse: + user_key = await self.openviking.ensure_user(user_id) + backends = { + "openviking": await self._capture(lambda: self.openviking.extract_session(user_key, session_id)), + } + return ExtractResponse(status=self._aggregate_status(backends), backends=backends) + + async def get_openviking_task(self, user_id: str, task_id: str) -> dict[str, Any]: + user_key = await self.openviking.ensure_user(user_id) + return await self.openviking.get_task(user_key, task_id) + + async def search(self, request: SearchRequest) -> SearchResponse: + user_key = await self.openviking.ensure_user(request.user_id) + everos_method = "agentic" if request.use_llm else "hybrid" + + async def search_openviking() -> dict[str, Any]: + if request.use_llm: + return await self.openviking.search(user_key, request.session_id, request.query, request.limit) + return await self.openviking.find(user_key, request.user_id, request.query, request.limit) + + async def search_everos() -> dict[str, Any]: + return await self.everos.search( + request.user_id, + request.session_id, + request.query, + everos_method, + request.limit, + ) + + backends = await self._run_backends(openviking=search_openviking, everos=search_everos) + items = self._merge_search_items(backends) + return SearchResponse(status=self._aggregate_status(backends), items=items[: request.limit], backends=backends) + + async def get_profile(self, user_id: str) -> ProfileResponse: + backends = {"everos": await self._capture(lambda: self.everos.get_profile(user_id))} + profile = backends["everos"].result if backends["everos"].status == "success" else None + return ProfileResponse(status=self._aggregate_status(backends), profile=profile, backends=backends) + + async def health(self) -> dict[str, Any]: + backends = await self._run_backends(openviking=self.openviking.health, everos=self.everos.health) + return {"status": self._aggregate_status(backends), "backends": backends} + + def _messages_from_request(self, request: MessageIngestRequest) -> list[dict[str, str]]: + messages = [] + if request.user_message: + messages.append({"role": "user", "content": request.user_message}) + if request.assistant_message: + messages.append({"role": "assistant", "content": request.assistant_message}) + return messages + + async def _run_backends(self, **calls: Callable[[], Awaitable[Any]]) -> dict[str, BackendStatus]: + names = list(calls) + results = await asyncio.gather(*(self._capture(calls[name]) for name in names)) + return dict(zip(names, results)) + + async def _capture(self, call: Callable[[], Awaitable[Any]]) -> BackendStatus: + try: + return BackendStatus(status="success", result=await call()) + except Exception as exc: # noqa: BLE001 + return BackendStatus(status="failed", error=str(exc)) + + def _aggregate_status(self, backends: dict[str, BackendStatus]) -> str: + statuses = {backend.status for backend in backends.values()} + if statuses == {"success"}: + return "success" + if "success" in statuses: + return "partial_success" + return "failed" + + def _merge_search_items(self, backends: dict[str, BackendStatus]) -> list[dict[str, Any]]: + items: list[dict[str, Any]] = [] + for backend_name, backend in backends.items(): + if backend.status != "success": + continue + items.extend(self._items_from_backend_result(backend_name, backend.result)) + return items + + def _items_from_backend_result(self, backend_name: str, result: Any) -> list[dict[str, Any]]: + if isinstance(result, dict) and isinstance(result.get("items"), list): + return [self._with_backend(backend_name, item) for item in result["items"] if isinstance(item, dict)] + data = result.get("data") if isinstance(result, dict) and isinstance(result.get("data"), dict) else result + if not isinstance(data, dict): + return [] + if isinstance(data.get("result"), dict): + data = data["result"] + raw_items: list[dict[str, Any]] = [] + for key in ("memories", "resources", "episodes", "profiles", "raw_messages"): + values = data.get(key) + if isinstance(values, list): + raw_items.extend(item for item in values if isinstance(item, dict)) + return [self._with_backend(backend_name, item) for item in raw_items] + + def _with_backend(self, backend_name: str, item: dict[str, Any]) -> dict[str, Any]: + if "source_backend" in item: + return item + return {"source_backend": backend_name, **item} diff --git a/memory_system_api/store.py b/memory_system_api/store.py new file mode 100644 index 0000000..aa407e1 --- /dev/null +++ b/memory_system_api/store.py @@ -0,0 +1,53 @@ +"""Small SQLite store for OpenViking user keys.""" +from __future__ import annotations + +import sqlite3 +from datetime import datetime, timezone +from pathlib import Path + + +class OpenVikingUserKeyStore: + def __init__(self, sqlite_path: str) -> None: + self.sqlite_path = sqlite_path + self._ensure_table() + + def get_user_key(self, user_id: str) -> str | None: + with self._connect() as conn: + row = conn.execute( + "SELECT user_key FROM memory_system_openviking_users WHERE user_id = ?", + (user_id,), + ).fetchone() + return str(row[0]) if row else None + + def save_user_key(self, user_id: str, user_key: str) -> None: + now = datetime.now(timezone.utc).isoformat() + with self._connect() as conn: + conn.execute( + """ + INSERT INTO memory_system_openviking_users (user_id, account_id, user_key, created_at, updated_at) + VALUES (?, ?, ?, ?, ?) + ON CONFLICT(user_id) DO UPDATE SET + user_key = excluded.user_key, + updated_at = excluded.updated_at + """, + (user_id, user_id, user_key, now, now), + ) + + def _ensure_table(self) -> None: + path = Path(self.sqlite_path) + path.parent.mkdir(parents=True, exist_ok=True) + with self._connect() as conn: + conn.execute( + """ + CREATE TABLE IF NOT EXISTS memory_system_openviking_users ( + user_id TEXT PRIMARY KEY, + account_id TEXT NOT NULL, + user_key TEXT NOT NULL, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL + ) + """ + ) + + def _connect(self) -> sqlite3.Connection: + return sqlite3.connect(self.sqlite_path) diff --git a/obsidian-vault/01_Knowledge/Uploaded/generic-memory-gateway-upload-test.md b/obsidian-vault/01_Knowledge/Uploaded/generic-memory-gateway-upload-test.md deleted file mode 100644 index 9337e9d..0000000 --- a/obsidian-vault/01_Knowledge/Uploaded/generic-memory-gateway-upload-test.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Generic Memory Gateway Upload Test -knowledge_type: reference -source_filename: memory-gateway-generic-upload.txt -created_at: 2026-04-29T09:53:46.674688+00:00 -tags: [generic, memory-gateway, upload-test, agent-workflow, knowledge-management, openviking] -summary: "A domain-neutral document describing a generic agent memory workflow for uploading reference documents. The workflow involves retrieving relevant context, summarizing final conclusions, uploading reference documents, and committing reusable knowledge to OpenViking." ---- - -# Generic Memory Gateway Upload Test - -# Generic Memory Gateway Upload Test - -This document describes a generic agent memory workflow: -- retrieve relevant context -- summarize final conclusions -- upload reference documents -- commit reusable knowledge to OpenViking diff --git a/obsidian-vault/01_Knowledge/Uploaded/memory-gateway-migration-upload-script-retry.md b/obsidian-vault/01_Knowledge/Uploaded/memory-gateway-migration-upload-script-retry.md deleted file mode 100644 index 0efe64c..0000000 --- a/obsidian-vault/01_Knowledge/Uploaded/memory-gateway-migration-upload-script-retry.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -title: Memory Gateway Migration Upload Script Retry -knowledge_type: migration_note -source_filename: memory-gateway-migration-upload.txt -created_at: 2026-04-29T10:01:47.830369+00:00 -tags: [migration, memory-gateway, script-retry] -summary: "Verification document confirming that document upload functionality works correctly after migrating the Memory Gateway project to /home/tom/memory-gateway. This serves as a test of the upload script following the project relocation." ---- - -# Memory Gateway Migration Upload Script Retry - -# Memory Gateway Migration Upload - -This document verifies that document upload works after moving the project to /home/tom/memory-gateway. diff --git a/obsidian-vault/01_Knowledge/Uploaded/memory-gateway-migration-upload.md b/obsidian-vault/01_Knowledge/Uploaded/memory-gateway-migration-upload.md deleted file mode 100644 index fdb9e5e..0000000 --- a/obsidian-vault/01_Knowledge/Uploaded/memory-gateway-migration-upload.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -title: Memory Gateway Migration Upload -knowledge_type: migration_note -source_filename: memory-gateway-migration-upload.txt -created_at: 2026-04-29T10:01:29.858006+00:00 -tags: [migration, memory-gateway, verification, document-upload] -summary: "Document upload functionality verified as working after migrating the Memory Gateway project to /home/tom/memory-gateway. This serves as a verification test confirming the migration was successful." ---- - -# Memory Gateway Migration Upload - -# Memory Gateway Migration Upload - -This document verifies that document upload works after moving the project to /home/tom/memory-gateway. diff --git a/obsidian-vault/05_Templates/agent-experience-template.md b/obsidian-vault/05_Templates/agent-experience-template.md deleted file mode 100644 index 291b797..0000000 --- a/obsidian-vault/05_Templates/agent-experience-template.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -type: agent_experience -agent_id: -visibility: agent-only -tags: - - memory/agent-experience ---- - -# Agent Experience - {{agent_id}} - -## What Worked - -- - -## What Failed - -- - -## Tooling Notes - -- - diff --git a/obsidian-vault/05_Templates/knowledge-note-template.md b/obsidian-vault/05_Templates/knowledge-note-template.md deleted file mode 100644 index b5157e6..0000000 --- a/obsidian-vault/05_Templates/knowledge-note-template.md +++ /dev/null @@ -1,26 +0,0 @@ -# {{title}} - ---- -type: knowledge -source: {{source}} -created: {{date}} -tags: - - memory-gateway ---- - -## Summary - -简要说明这份知识对后续 agent / harness 的可复用价值。 - -## Key Points - -- - -## Usage Notes - -说明 agent 在什么场景下应该检索或引用这份知识。 - -## Source - -- 原始来源： -- OpenViking URI： diff --git a/obsidian-vault/05_Templates/long-term-memory-template.md b/obsidian-vault/05_Templates/long-term-memory-template.md deleted file mode 100644 index 6d5636a..0000000 --- a/obsidian-vault/05_Templates/long-term-memory-template.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -type: long_term_memory -memory_id: -user_id: -workspace_id: -visibility: private -importance: -confidence: -source: -tags: - - memory/long-term ---- - -# {{summary}} - -## Memory - - -## Context - - -## Evidence - -- Source: -- Created: -- Version: - -## Review - -- Status: pending -- Reviewer: -- Decision: - diff --git a/obsidian-vault/05_Templates/review-queue-template.md b/obsidian-vault/05_Templates/review-queue-template.md deleted file mode 100644 index c02ec6c..0000000 --- a/obsidian-vault/05_Templates/review-queue-template.md +++ /dev/null @@ -1,23 +0,0 @@ ---- -type: memory_review -review_status: pending -tags: - - memory/review ---- - -# Memory Review - {{memory_id}} - -## Candidate - - -## Proposed Action - -- [ ] Accept -- [ ] Edit -- [ ] Reject -- [ ] Merge -- [ ] Archive - -## Reason - - diff --git a/obsidian-vault/05_Templates/user-profile-template.md b/obsidian-vault/05_Templates/user-profile-template.md deleted file mode 100644 index fe30931..0000000 --- a/obsidian-vault/05_Templates/user-profile-template.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -type: user_profile -user_id: -visibility: private -tags: - - memory/profile - - visibility/private ---- - -# User Profile - {{user_id}} - -## Stable Facts - -- - -## Preferences - -- - -## Working Style - -- - -## Evidence - -| Memory ID | Evidence | Confidence | Updated | -|---|---|---:|---| - diff --git a/obsidian-vault/05_Templates/workspace-memory-template.md b/obsidian-vault/05_Templates/workspace-memory-template.md deleted file mode 100644 index 3aa52d1..0000000 --- a/obsidian-vault/05_Templates/workspace-memory-template.md +++ /dev/null @@ -1,23 +0,0 @@ ---- -type: workspace_memory -workspace_id: -visibility: workspace-shared -tags: - - memory/workspace - - visibility/workspace-shared ---- - -# Workspace Memory - {{workspace_id}} - -## Shared Decisions - -- - -## Project Knowledge - -- - -## Reusable Context - -- - diff --git a/obsidian-vault/README.md b/obsidian-vault/README.md deleted file mode 100644 index 3c47eb7..0000000 --- a/obsidian-vault/README.md +++ /dev/null @@ -1,15 +0,0 @@ -# Obsidian Vault - -这个目录用于保存 Memory Gateway 的 Markdown 知识沉淀。 - -原则： - -- 只存高价值、可人工维护的知识和总结。 -- 不存全量原始资料。 -- 不存密钥、凭证、私人敏感信息或无需长期保留的聊天流水。 -- 上传文档默认进入 `01_Knowledge/Uploaded/`，再由 Memory Gateway 总结并写入 OpenViking。 - -当前结构： - -- `01_Knowledge/Uploaded/`：上传文档转换后的 Markdown。 -- `05_Templates/`：通用知识笔记模板。 diff --git a/plugins/memory-gateway-agent/README.md b/plugins/memory-gateway-agent/README.md deleted file mode 100644 index d82d89b..0000000 --- a/plugins/memory-gateway-agent/README.md +++ /dev/null @@ -1,154 +0,0 @@ -# Memory Gateway Agent Plugin - -This plugin is an adapter for the existing Memory Gateway. It is not Memory Gateway core and it does not import core service, repository, or server modules. - -The plugin calls the existing HTTP API: - -- `POST /v1/memory/search` -- `POST /v1/episodes` -- `POST /v1/sessions/{session_id}/commit` -- `POST /v1/memory` -- `POST /v1/memory/{memory_id}/feedback` - -## Configuration - -Environment variables: - -- `MEMORY_GATEWAY_URL`, default `http://127.0.0.1:1934` -- `MEMORY_GATEWAY_API_KEY`, optional -- `MEMORY_GATEWAY_DEFAULT_USER_ID`, optional -- `MEMORY_GATEWAY_DEFAULT_AGENT_ID`, optional -- `MEMORY_GATEWAY_DEFAULT_WORKSPACE_ID`, optional -- `MEMORY_GATEWAY_AUTO_SEARCH`, default `true` -- `MEMORY_GATEWAY_AUTO_APPEND_EPISODE`, default `true` -- `MEMORY_GATEWAY_AUTO_COMMIT_SESSION`, default `false` -- `MEMORY_GATEWAY_REVIEW_MODE`, default `true` -- `MEMORY_GATEWAY_PLUGIN_DEBUG_RAW`, default `false` -- `MEMORY_GATEWAY_PLUGIN_TRACE_HOOKS`, default `false` - -If an API key is configured, the plugin sends `X-API-Key`. It never logs the API key. - -## Validation Status - -| Check | Status | -| --- | --- | -| Mock unit tests | passed | -| Hermes plugin discovery | passed | -| Hermes tool registration | passed | -| Hermes hook registration | passed | -| Gateway E2E | passed | -| PluginManager.invoke_hook probe | passed | -| Real Hermes interactive session | passed | -| OpenClaw runtime validation | pending | - -## Hermes - -Use `hermes.plugin.yaml` as the Hermes-facing manifest. The entrypoint is: - -```text -__init__:register -``` - -The plugin attempts to register tools and best-effort hooks. If the Hermes runtime does not expose hook registration, it still works in tools-only mode. - -Install locally: - -```bash -mkdir -p ~/.hermes/plugins -cd /opt/memory-gateway -ln -s "$(pwd)/plugins/memory-gateway-agent" ~/.hermes/plugins/memory-gateway-agent -hermes plugins enable memory-gateway-agent -hermes plugins list -hermes tools list -``` - -Example runtime configuration: - -```bash -export MEMORY_GATEWAY_URL=http://127.0.0.1:1934 -export MEMORY_GATEWAY_API_KEY= -export MEMORY_GATEWAY_AUTO_SEARCH=true -export MEMORY_GATEWAY_AUTO_APPEND_EPISODE=true -export MEMORY_GATEWAY_AUTO_COMMIT_SESSION=false -``` - -## OpenClaw - -`openclaw.plugin.yaml` is a best-effort draft manifest. Adjust field names to the actual OpenClaw runtime schema before production use. - -## Tools-Only Mode - -Agent runtimes can call: - -- `memory_search` -- `memory_append_episode` -- `memory_commit_session` -- `memory_upsert` -- `memory_feedback` - -Tools-only mode does not automatically remember anything. The agent policy must decide when to call tools. - -## Lifecycle Hooks - -Best-effort hooks: - -- `on_session_start`: initializes session memory context without writing long-term memory. -- `pre_llm_call`: searches memory and returns compact memory context. -- `post_llm_call`: appends a safe candidate episode when policy allows it. -- `on_session_end`: commits session only when `MEMORY_GATEWAY_AUTO_COMMIT_SESSION=true`. - -Plugin support for hooks depends on the agent runtime context API. This plugin should be described as tools-only plus best-effort hooks unless the target runtime has been verified. - -Verified boundaries: - -- Tools-only is usable through the registered `memory_gateway` toolset. -- `pre_llm_call` automatic search has passed hook-probe and real `hermes chat -Q -q` validation. -- `post_llm_call` automatic candidate episode append has passed hook-probe and real `hermes chat -Q -q` validation. -- `on_session_end` auto commit is off by default, stays off when `MEMORY_GATEWAY_AUTO_COMMIT_SESSION=false`, and commits when `MEMORY_GATEWAY_AUTO_COMMIT_SESSION=true`. - -## Defaults - -- Automatic search can be enabled. -- Automatic append episode can be enabled. -- Automatic commit is disabled by default. -- Automatic direct long-term upsert is disabled by default. - -## Safety And Privacy - -The plugin rejects memory writes containing passwords, API keys, bearer tokens, cookies, private keys, SSH keys, one-time verification codes, large logs, full raw transcripts, and chain-of-thought. - -The plugin writes summarized candidate episodes. It does not store full raw conversations. Long-term memory should normally be produced by `memory_commit_session`, allowing Memory Gateway and EverOS to deduplicate, detect conflicts, and route review drafts. - -Direct long-term `memory_upsert` is high risk and is not called automatically. If a user asks to forget or delete a memory, the agent should call `memory_feedback` or a delete-capable tool instead of silently keeping the memory. - -Script output is redacted by default: no API key, headers, cookies, tokens, or raw result payloads are printed. Set `MEMORY_GATEWAY_PLUGIN_DEBUG_RAW=true` only for local debugging with non-sensitive test data. - -Hook trace is disabled by default. Set `MEMORY_GATEWAY_PLUGIN_TRACE_HOOKS=true` to write minimal hook events to `plugins/memory-gateway-agent/.tmp/hook_trace.log`. Trace entries contain hook name, timestamp, shortened session id, Gateway action, and status only; they do not include user or assistant message bodies. - -## Cleanup Test Data - -Integration tests use: - -- `user_id=test_user_memory_gateway_plugin` -- tags such as `integration_test`, `plugin`, and `safe_to_delete` - -Run cleanup: - -```bash -cd /opt/memory-gateway -PYTHONPATH=plugins/memory-gateway-agent python plugins/memory-gateway-agent/scripts/cleanup_test_memories.py -``` - -The cleanup script refuses non-`test_user_` users. It first tries `DELETE /v1/memory/{memory_id}` for local test memories. If deletion fails, it falls back to `memory_feedback` with `incorrect`. Current cleanup is limited by the search API: it can only clean local `MemoryRecord` rows returned by search, not arbitrary OpenViking context rows. - -## Local Smoke Test - -```bash -cd /opt/memory-gateway -PYTHONPATH=plugins/memory-gateway-agent python plugins/memory-gateway-agent/scripts/health.py -PYTHONPATH=plugins/memory-gateway-agent python plugins/memory-gateway-agent/scripts/smoke_test.py -PYTHONPATH=plugins/memory-gateway-agent python plugins/memory-gateway-agent/scripts/hermes_smoke_check.py -PYTHONPATH=plugins/memory-gateway-agent python plugins/memory-gateway-agent/scripts/gateway_e2e_check.py -PYTHONPATH=plugins/memory-gateway-agent python plugins/memory-gateway-agent/scripts/hermes_hook_probe.py -PYTHONPATH=plugins/memory-gateway-agent python plugins/memory-gateway-agent/scripts/hermes_interactive_session_check.py -``` diff --git a/plugins/memory-gateway-agent/__init__.py b/plugins/memory-gateway-agent/__init__.py deleted file mode 100644 index 215b78a..0000000 --- a/plugins/memory-gateway-agent/__init__.py +++ /dev/null @@ -1,120 +0,0 @@ -from __future__ import annotations - -from typing import Any - -try: - from . import schemas, tools - from .memory_gateway_plugin.config import load_config - from .memory_gateway_plugin import lifecycle - from .memory_gateway_plugin.trace import trace_hook -except ImportError: - import sys - from pathlib import Path - - _PLUGIN_ROOT = Path(__file__).resolve().parent - if str(_PLUGIN_ROOT) not in sys.path: - sys.path.insert(0, str(_PLUGIN_ROOT)) - - import schemas # type: ignore[no-redef] - import tools # type: ignore[no-redef] - from memory_gateway_plugin.config import load_config # type: ignore[no-redef] - from memory_gateway_plugin import lifecycle # type: ignore[no-redef] - from memory_gateway_plugin.trace import trace_hook # type: ignore[no-redef] - -TOOLSET = "memory_gateway" -_LAST_USER_MESSAGES: dict[str, str] = {} - - -def _context_from_kwargs(kwargs: dict[str, Any]) -> dict[str, Any]: - cfg = load_config() - return { - "user_id": kwargs.get("user_id") or kwargs.get("user") or cfg.default_user_id, - "agent_id": kwargs.get("agent_id") or kwargs.get("agent") or cfg.default_agent_id or "hermes_agent", - "workspace_id": kwargs.get("workspace_id") or kwargs.get("workspace") or cfg.default_workspace_id, - "session_id": kwargs.get("session_id") or kwargs.get("task_id") or "", - "user_message": kwargs.get("user_message") or kwargs.get("prompt") or kwargs.get("message") or "", - "assistant_response": kwargs.get("assistant_response") or kwargs.get("response") or "", - "conversation_history": kwargs.get("conversation_history") or [], - "model": kwargs.get("model") or "", - "platform": kwargs.get("platform") or "", - } - - -def on_session_start(**kwargs: Any) -> dict[str, Any]: - session_id = kwargs.get("session_id") or kwargs.get("task_id") or "" - trace_hook("on_session_start", session_id=str(session_id), gateway_action="", gateway_called=False, ok=True) - return { - "status": "ok", - "memory_gateway": "session_initialized", - "session_id": session_id, - } - - -def pre_llm_call(**kwargs: Any) -> dict[str, str]: - context = _context_from_kwargs(kwargs) - if context.get("session_id") and context.get("user_message"): - _LAST_USER_MESSAGES[context["session_id"]] = context["user_message"] - result = lifecycle.on_conversation_start(context) - trace_hook( - "pre_llm_call", - session_id=context.get("session_id", ""), - gateway_action="memory_search", - gateway_called=bool(result.get("raw")), - ok=bool(result.get("ok")), - reason=str(result.get("error") or result.get("reason") or ""), - ) - if not result.get("ok") or not result.get("memory_context"): - return {} - return {"context": "Relevant Memory Gateway context:\n" + result["memory_context"]} - - -def post_llm_call(**kwargs: Any) -> dict[str, Any] | None: - context = _context_from_kwargs(kwargs) - if not context.get("user_message") and context.get("session_id"): - context["user_message"] = _LAST_USER_MESSAGES.get(context["session_id"], "") - result = lifecycle.after_user_message(context) - trace_hook( - "post_llm_call", - session_id=context.get("session_id", ""), - gateway_action="append_episode", - gateway_called=bool(result.get("raw")), - ok=bool(result.get("ok")), - reason=str(result.get("error") or result.get("reason") or ""), - ) - if result.get("ok"): - return None - return {"memory_gateway_error": result.get("error") or result.get("reason") or "append_failed"} - - -def on_session_end(**kwargs: Any) -> dict[str, Any] | None: - context = _context_from_kwargs(kwargs) - result = lifecycle.on_session_end(context) - trace_hook( - "on_session_end", - session_id=context.get("session_id", ""), - gateway_action="commit_session", - gateway_called=bool(result.get("raw")), - ok=bool(result.get("ok")), - reason=str(result.get("error") or result.get("reason") or ""), - ) - if context.get("session_id"): - _LAST_USER_MESSAGES.pop(context["session_id"], None) - if result.get("ok"): - return None - return {"memory_gateway_error": result.get("error") or "commit_failed"} - - -def register(ctx: Any) -> None: - for name, schema in schemas.TOOL_SCHEMAS.items(): - ctx.register_tool( - name=name, - toolset=TOOLSET, - schema=schema, - handler=tools.HANDLERS[name], - ) - - if hasattr(ctx, "register_hook"): - ctx.register_hook("on_session_start", on_session_start) - ctx.register_hook("pre_llm_call", pre_llm_call) - ctx.register_hook("post_llm_call", post_llm_call) - ctx.register_hook("on_session_end", on_session_end) diff --git a/plugins/memory-gateway-agent/hermes.plugin.yaml b/plugins/memory-gateway-agent/hermes.plugin.yaml deleted file mode 100644 index 8213ce0..0000000 --- a/plugins/memory-gateway-agent/hermes.plugin.yaml +++ /dev/null @@ -1,41 +0,0 @@ -name: memory-gateway-agent -runtime: hermes -version: 0.1.0 -description: Hermes plugin adapter for Memory Gateway v1. Provides tools-only mode plus best-effort lifecycle hooks. -entrypoint: register -provides_tools: - - memory_search - - memory_append_episode - - memory_commit_session - - memory_upsert - - memory_feedback -provides_hooks: - - on_session_start - - pre_llm_call - - post_llm_call - - on_session_end -env: - MEMORY_GATEWAY_URL: http://127.0.0.1:1934 - MEMORY_GATEWAY_AUTO_SEARCH: "true" - MEMORY_GATEWAY_AUTO_APPEND_EPISODE: "true" - MEMORY_GATEWAY_AUTO_COMMIT_SESSION: "false" -tools: - memory_search: - description: Search Memory Gateway with user/agent/workspace/session ACL. - memory_append_episode: - description: Append a safe summarized candidate episode. - memory_commit_session: - description: Ask Gateway/EverOS to consolidate session episodes. - memory_upsert: - description: Upsert a stable memory through Gateway. - memory_feedback: - description: Send feedback for a memory record. -hooks: - on_session_start: __init__:on_session_start - pre_llm_call: __init__:pre_llm_call - post_llm_call: __init__:post_llm_call - on_session_end: __init__:on_session_end -notes: - - Hooks are best-effort and depend on the Hermes runtime context API. - - Without hook support, the plugin remains usable as tools-only. - - This plugin does not store full raw conversations. diff --git a/plugins/memory-gateway-agent/memory_gateway_plugin/__init__.py b/plugins/memory-gateway-agent/memory_gateway_plugin/__init__.py deleted file mode 100644 index 8adb7bb..0000000 --- a/plugins/memory-gateway-agent/memory_gateway_plugin/__init__.py +++ /dev/null @@ -1,74 +0,0 @@ -from __future__ import annotations - -import logging -from typing import Any, Callable - -from . import lifecycle -from .tools import memory_append_episode, memory_commit_session, memory_feedback, memory_search, memory_upsert - -_logger = logging.getLogger(__name__) - -__all__ = [ - "register", - "memory_search", - "memory_append_episode", - "memory_commit_session", - "memory_upsert", - "memory_feedback", -] - - -TOOLS: dict[str, Callable[..., dict[str, Any]]] = { - "memory_search": memory_search, - "memory_append_episode": memory_append_episode, - "memory_commit_session": memory_commit_session, - "memory_upsert": memory_upsert, - "memory_feedback": memory_feedback, -} - - -def _try_call(target: Any, method_names: list[str], *args: Any, **kwargs: Any) -> bool: - for name in method_names: - method = getattr(target, name, None) - if callable(method): - try: - method(*args, **kwargs) - return True - except TypeError: - try: - method(args[0], args[1]) - return True - except Exception as exc: - _logger.debug("[_try_call] %s(%s, %s) failed: %s", name, args, kwargs, exc) - return False - except Exception as exc: - _logger.debug("[_try_call] %s(%s, %s) failed: %s", name, args, kwargs, exc) - return False - return False - - -def register(ctx: Any) -> dict[str, Any]: - registered_tools: list[str] = [] - registered_hooks: list[str] = [] - - for name, func in TOOLS.items(): - if _try_call(ctx, ["register_tool", "add_tool", "tool"], name, func): - registered_tools.append(name) - - hook_map = { - "pre_llm_call": lifecycle.on_conversation_start, - "post_llm_call": lifecycle.after_user_message, - "session_end": lifecycle.on_session_end, - "after_task_complete": lifecycle.after_task_complete, - } - for name, func in hook_map.items(): - if _try_call(ctx, ["register_hook", "add_hook", "hook"], name, func): - registered_hooks.append(name) - - return { - "ok": True, - "mode": "tools-and-hooks" if registered_hooks else "tools-only" if registered_tools else "manual", - "registered_tools": registered_tools, - "registered_hooks": registered_hooks, - } - diff --git a/plugins/memory-gateway-agent/memory_gateway_plugin/client.py b/plugins/memory-gateway-agent/memory_gateway_plugin/client.py deleted file mode 100644 index a6de036..0000000 --- a/plugins/memory-gateway-agent/memory_gateway_plugin/client.py +++ /dev/null @@ -1,101 +0,0 @@ -from __future__ import annotations - -import json -import logging -import time -import urllib.error -import urllib.request -from typing import Any - -from .config import PluginConfig, load_config - -_logger = logging.getLogger(__name__) - - -def _short_error(value: Any, max_chars: int = 500) -> str: - text = str(value).replace("\n", " ").strip() - return text[:max_chars] - - -class MemoryGatewayClient: - def __init__(self, config: PluginConfig | None = None) -> None: - self.config = config or load_config() - - def _headers(self) -> dict[str, str]: - headers = {"Content-Type": "application/json"} - if self.config.api_key: - headers["X-API-Key"] = self.config.api_key - return headers - - def _post(self, endpoint: str, payload: dict[str, Any], retries: int = 3, backoff: float = 1.0) -> dict[str, Any]: - url = self.config.gateway_url.rstrip("/") + endpoint - body = json.dumps(payload, ensure_ascii=False).encode("utf-8") - - last_error: Exception | None = None - for attempt in range(retries): - request = urllib.request.Request(url, data=body, headers=self._headers(), method="POST") - try: - with urllib.request.urlopen(request, timeout=self.config.timeout) as response: - raw = response.read().decode("utf-8") - data = json.loads(raw) if raw else {} - return { - "ok": True, - "status_code": getattr(response, "status", 200), - "endpoint": endpoint, - "data": data, - } - except urllib.error.HTTPError as exc: - # Typically, client errors (4xx) shouldn't be retried unless specifically handled. - # Since HTTPError is a subclass of URLError, we catch it first. - if exc.code < 500 and exc.code != 429: - try: - body_text = exc.read().decode("utf-8") - except Exception: - body_text = exc.reason - _logger.error(f"HTTPError in _post to {endpoint}: {exc.code} {body_text}") - return { - "ok": False, - "status_code": exc.code, - "endpoint": endpoint, - "error": _short_error(body_text), - } - last_error = exc - except (urllib.error.URLError, TimeoutError, OSError) as exc: - last_error = exc - except Exception as exc: - _logger.error("Unexpected error in _post to %s: %s", endpoint, exc, exc_info=True) - return { - "ok": False, - "status_code": None, - "endpoint": endpoint, - "error": _short_error(exc), - } - - if attempt < retries - 1: - time.sleep(backoff * (2 ** attempt)) - - # Exhausted retries - error_msg = str(last_error) if last_error else "Max retries exceeded" - _logger.error("Failed _post to %s after %d attempts. Last error: %s", endpoint, retries, last_error) - return { - "ok": False, - "status_code": None, - "endpoint": endpoint, - "error": error_msg, - } - - def search_memory(self, payload: dict[str, Any]) -> dict[str, Any]: - return self._post("/v1/memory/search", payload) - - def append_episode(self, payload: dict[str, Any]) -> dict[str, Any]: - return self._post("/v1/episodes", payload) - - def commit_session(self, session_id: str, payload: dict[str, Any]) -> dict[str, Any]: - return self._post(f"/v1/sessions/{session_id}/commit", payload) - - def upsert_memory(self, payload: dict[str, Any]) -> dict[str, Any]: - return self._post("/v1/memory", payload) - - def send_feedback(self, memory_id: str, payload: dict[str, Any]) -> dict[str, Any]: - return self._post(f"/v1/memory/{memory_id}/feedback", payload) - diff --git a/plugins/memory-gateway-agent/memory_gateway_plugin/config.py b/plugins/memory-gateway-agent/memory_gateway_plugin/config.py deleted file mode 100644 index 82f9162..0000000 --- a/plugins/memory-gateway-agent/memory_gateway_plugin/config.py +++ /dev/null @@ -1,50 +0,0 @@ -from __future__ import annotations - -import os -from dataclasses import dataclass - - -def _env_bool(name: str, default: bool) -> bool: - value = os.environ.get(name) - if value is None: - return default - return value.strip().lower() in {"1", "true", "yes", "on"} - - -@dataclass(frozen=True) -class PluginConfig: - gateway_url: str = "http://127.0.0.1:1934" - api_key: str = "" - default_user_id: str = "" - default_agent_id: str = "" - default_workspace_id: str = "" - auto_search: bool = True - auto_append_episode: bool = True - auto_commit_session: bool = False - review_mode: bool = True - timeout: int = 30 - - @classmethod - def from_env(cls) -> "PluginConfig": - try: - timeout_val = int(os.environ.get("MEMORY_GATEWAY_TIMEOUT", "30")) - except ValueError: - timeout_val = 30 - - return cls( - gateway_url=os.environ.get("MEMORY_GATEWAY_URL", cls.gateway_url).rstrip("/"), - api_key=os.environ.get("MEMORY_GATEWAY_API_KEY", ""), - default_user_id=os.environ.get("MEMORY_GATEWAY_DEFAULT_USER_ID", ""), - default_agent_id=os.environ.get("MEMORY_GATEWAY_DEFAULT_AGENT_ID", ""), - default_workspace_id=os.environ.get("MEMORY_GATEWAY_DEFAULT_WORKSPACE_ID", ""), - auto_search=_env_bool("MEMORY_GATEWAY_AUTO_SEARCH", True), - auto_append_episode=_env_bool("MEMORY_GATEWAY_AUTO_APPEND_EPISODE", True), - auto_commit_session=_env_bool("MEMORY_GATEWAY_AUTO_COMMIT_SESSION", False), - review_mode=_env_bool("MEMORY_GATEWAY_REVIEW_MODE", True), - timeout=timeout_val, - ) - - -def load_config() -> PluginConfig: - return PluginConfig.from_env() - diff --git a/plugins/memory-gateway-agent/memory_gateway_plugin/lifecycle.py b/plugins/memory-gateway-agent/memory_gateway_plugin/lifecycle.py deleted file mode 100644 index 8b7ea25..0000000 --- a/plugins/memory-gateway-agent/memory_gateway_plugin/lifecycle.py +++ /dev/null @@ -1,109 +0,0 @@ -from __future__ import annotations - -from typing import Any - -from .client import MemoryGatewayClient -from .config import PluginConfig, load_config -from .policy import build_episode_summary, should_append_episode, should_commit_session, should_search_memory -from .tools import memory_append_episode, memory_commit_session, memory_search - - -def _get(context: dict[str, Any], key: str, default: str = "") -> str: - value = context.get(key, default) - return "" if value is None else str(value) - - -def compact_memory_context(search_result: dict[str, Any], limit: int = 5) -> str: - if not search_result.get("ok"): - return "" - data = search_result.get("data", {}) - rows = [] - for item in data.get("results", [])[:limit]: - memory = item.get("memory") or item.get("openviking") or {} - summary = memory.get("summary") or memory.get("abstract") or memory.get("content") or "" - namespace = memory.get("namespace", "") - memory_id = memory.get("id") or memory.get("uri") or "" - if summary: - rows.append(f"- {memory_id} [{namespace}]: {summary[:240]}") - return "\n".join(rows) - - -def on_conversation_start(context: dict[str, Any], client: MemoryGatewayClient | None = None, config: PluginConfig | None = None) -> dict[str, Any]: - cfg = config or load_config() - user_message = _get(context, "user_message") or _get(context, "query") - if not should_search_memory(user_message, context, cfg): - return {"ok": True, "memory_context": ""} - user_id = _get(context, "user_id", cfg.default_user_id) - if not user_id: - return {"ok": False, "error": "user_id_required"} - try: - limit_val = int(context.get("limit", 5)) - except (ValueError, TypeError): - limit_val = 5 - - result = memory_search( - query=user_message, - user_id=user_id, - agent_id=_get(context, "agent_id", cfg.default_agent_id), - workspace_id=_get(context, "workspace_id", cfg.default_workspace_id), - session_id=_get(context, "session_id"), - limit=limit_val, - client=client, - ) - return {"ok": result.get("ok", False), "memory_context": compact_memory_context(result), "raw": result} - - -def after_user_message(context: dict[str, Any], client: MemoryGatewayClient | None = None, config: PluginConfig | None = None) -> dict[str, Any]: - cfg = config or load_config() - user_message = _get(context, "user_message") - assistant_response = _get(context, "assistant_response") - if not should_append_episode(user_message, assistant_response, context, cfg): - return {"ok": True, "appended": False, "reason": "policy_skip"} - user_id = _get(context, "user_id", cfg.default_user_id) - session_id = _get(context, "session_id") - if not user_id or not session_id: - return {"ok": False, "error": "user_id_and_session_id_required"} - summary = build_episode_summary(user_message, assistant_response, context) - result = memory_append_episode( - user_id=user_id, - agent_id=_get(context, "agent_id", cfg.default_agent_id), - workspace_id=_get(context, "workspace_id", cfg.default_workspace_id), - session_id=session_id, - episode_summary=summary, - tags=["plugin-candidate"], - client=client, - ) - return {"ok": result.get("ok", False), "appended": result.get("ok", False), "raw": result} - - -def after_task_complete(context: dict[str, Any], client: MemoryGatewayClient | None = None, config: PluginConfig | None = None) -> dict[str, Any]: - return _maybe_commit(context, client, config) - - -def on_session_end(context: dict[str, Any], client: MemoryGatewayClient | None = None, config: PluginConfig | None = None) -> dict[str, Any]: - return _maybe_commit(context, client, config) - - -def _maybe_commit(context: dict[str, Any], client: MemoryGatewayClient | None, config: PluginConfig | None) -> dict[str, Any]: - cfg = config or load_config() - if not should_commit_session(context, cfg): - return {"ok": True, "committed": False, "reason": "auto_commit_disabled"} - user_id = _get(context, "user_id", cfg.default_user_id) - session_id = _get(context, "session_id") - if not user_id or not session_id: - return {"ok": False, "error": "user_id_and_session_id_required"} - try: - min_importance_val = float(context.get("min_importance", 0.6)) - except (ValueError, TypeError): - min_importance_val = 0.6 - - result = memory_commit_session( - user_id=user_id, - agent_id=_get(context, "agent_id", cfg.default_agent_id), - workspace_id=_get(context, "workspace_id", cfg.default_workspace_id), - session_id=session_id, - min_importance=min_importance_val, - client=client, - ) - return {"ok": result.get("ok", False), "committed": result.get("ok", False), "raw": result} - diff --git a/plugins/memory-gateway-agent/memory_gateway_plugin/output.py b/plugins/memory-gateway-agent/memory_gateway_plugin/output.py deleted file mode 100644 index 202aed7..0000000 --- a/plugins/memory-gateway-agent/memory_gateway_plugin/output.py +++ /dev/null @@ -1,86 +0,0 @@ -from __future__ import annotations - -import json -import os -from typing import Any - - -SENSITIVE_KEYS = ("api_key", "apikey", "authorization", "token", "cookie", "secret", "password", "x-api-key") - - -def debug_raw_enabled() -> bool: - return os.environ.get("MEMORY_GATEWAY_PLUGIN_DEBUG_RAW", "").strip().lower() in {"1", "true", "yes", "on"} - - -def short_id(value: Any, prefix: int = 8, suffix: int = 4) -> str: - text = "" if value is None else str(value) - if len(text) <= prefix + suffix + 3: - return text - return f"{text[:prefix]}...{text[-suffix:]}" - - -import re - -def redact(value: Any) -> Any: - if isinstance(value, dict): - return { - key: ("" if key.lower() in SENSITIVE_KEYS else redact(item)) - for key, item in value.items() - } - if isinstance(value, list): - return [redact(item) for item in value] - if isinstance(value, str): - lowered = value.lower() - sensitive_markers = ("api_key=", "password=", "token=", "bearer ", "cookie:", "private key") - if any(marker in lowered for marker in sensitive_markers): - return "" - return value - - -def summarize_data(data: Any) -> Any: - if debug_raw_enabled(): - return redact(data) - if isinstance(data, list): - return {"count": len(data)} - if not isinstance(data, dict): - return data - if "results" in data: - return { - "count": len(data.get("results") or []), - "total": data.get("total"), - "local_total": data.get("local_total"), - "openviking_total": data.get("openviking_total"), - "searched_namespaces": data.get("searched_namespaces", []), - } - if "id" in data: - return { - "id": short_id(data.get("id")), - "namespace": data.get("namespace"), - "memory_type": data.get("memory_type"), - "source": data.get("source"), - } - if "memory_id" in data: - return {"status": data.get("status"), "memory_id": short_id(data.get("memory_id")), "feedback": data.get("feedback")} - if "promoted" in data or "consolidation" in data: - return { - "status": data.get("status"), - "promoted_count": len(data.get("promoted") or []), - "archived_count": len(data.get("archived_episode_ids") or []), - "consolidation_status": (data.get("consolidation") or {}).get("status") if isinstance(data.get("consolidation"), dict) else None, - } - allowed = {"ok", "status", "gateway", "service", "version", "healthy", "endpoint", "status_code", "error", "count"} - return {key: redact(value) for key, value in data.items() if key in allowed} - - -def summarize_result(result: dict[str, Any]) -> dict[str, Any]: - return { - "ok": bool(result.get("ok")), - "endpoint": result.get("endpoint"), - "status_code": result.get("status_code"), - "error": redact(result.get("error", "")), - "data": summarize_data(result.get("data")), - } - - -def dumps_safe(payload: Any, *, indent: int = 2) -> str: - return json.dumps(redact(payload), ensure_ascii=False, indent=indent, default=str) diff --git a/plugins/memory-gateway-agent/memory_gateway_plugin/policy.py b/plugins/memory-gateway-agent/memory_gateway_plugin/policy.py deleted file mode 100644 index 494b2f3..0000000 --- a/plugins/memory-gateway-agent/memory_gateway_plugin/policy.py +++ /dev/null @@ -1,59 +0,0 @@ -from __future__ import annotations - -import re -from typing import Any - -from .config import PluginConfig, load_config -from .safety import validate_memory_write - - -REMEMBER_RE = re.compile(r"记住|请保存|remember this|save this|keep in memory", re.I) -STABLE_SIGNAL_RE = re.compile( - r"偏好|长期|约束|架构决策|决策|结论|workflow|工作流|preference|constraint|decision|always|以后都|project fact", - re.I, -) -SMALL_TALK_RE = re.compile(r"^\s*(你好|hi|hello|谢谢|thanks|ok|好的|收到|再见)[。.!！\s\w]*$", re.I) - - -def should_search_memory(user_message: str, context: dict[str, Any] | None = None, config: PluginConfig | None = None) -> bool: - cfg = config or load_config() - if not cfg.auto_search: - return False - return bool(user_message and user_message.strip()) - - -def should_append_episode( - user_message: str, - assistant_response: str = "", - context: dict[str, Any] | None = None, - config: PluginConfig | None = None, -) -> bool: - cfg = config or load_config() - if not cfg.auto_append_episode: - return False - combined = "\n".join(part for part in [user_message, assistant_response] if part) - if not combined.strip() or SMALL_TALK_RE.match(combined.strip()): - return False - if not validate_memory_write(combined)["allowed"]: - return False - return bool(REMEMBER_RE.search(combined) or STABLE_SIGNAL_RE.search(combined)) - - -def build_episode_summary(user_message: str, assistant_response: str = "", context: dict[str, Any] | None = None) -> str: - parts = [] - if REMEMBER_RE.search(user_message or ""): - parts.append(f"用户明确要求记住：{user_message.strip()}") - elif user_message: - parts.append(f"用户输入中的可复用信息：{user_message.strip()}") - if assistant_response and STABLE_SIGNAL_RE.search(assistant_response): - parts.append(f"助手结论：{assistant_response.strip()}") - summary = " ".join(parts).strip() - return summary[:1000] - - -def should_commit_session(context: dict[str, Any] | None = None, config: PluginConfig | None = None) -> bool: - cfg = config or load_config() - if cfg.auto_commit_session: - return True - return bool((context or {}).get("force_commit")) - diff --git a/plugins/memory-gateway-agent/memory_gateway_plugin/safety.py b/plugins/memory-gateway-agent/memory_gateway_plugin/safety.py deleted file mode 100644 index 9745ea8..0000000 --- a/plugins/memory-gateway-agent/memory_gateway_plugin/safety.py +++ /dev/null @@ -1,97 +0,0 @@ -from __future__ import annotations - -import re -from typing import Any - - -SECRET_PATTERNS = [ - r"\bpassword\s*[:=]", - r"\bapi[_-]?key\s*[:=]", - r"\btoken\s*[:=]", - r"\bsecret\s*[:=]", - r"\bbearer\s+[a-z0-9._\-]{12,}", - r"\bcookie\s*[:=]", - r"\bsession[_ -]?id\s*[:=]", - r"-----BEGIN [A-Z ]*PRIVATE KEY-----", - r"\bssh-rsa\s+[a-z0-9+/=]{40,}", - r"\bone[- ]?time (?:password|code)\b", - r"\botp\s*[:=]?\s*\d{4,8}\b", - r"\b验证码\s*[:：]?\s*\d{4,8}\b", -] - -CHAT_LINE_RE = re.compile(r"^\s*(user|assistant|system|用户|助手|模型|human|ai)\s*[:：]", re.I) -LOG_LINE_RE = re.compile(r"\b(ERROR|WARN|INFO|DEBUG|TRACE)\b|^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}") -CHAIN_OF_THOUGHT_RE = re.compile(r"chain[- ]of[- ]thought|逐步推理|隐藏推理|internal reasoning", re.I) - - -def detect_secret(content: str) -> tuple[bool, str]: - for pattern in SECRET_PATTERNS: - if re.search(pattern, content, re.I): - return True, "secret_like_content" - return False, "" - - -def detect_raw_transcript(content: str) -> tuple[bool, str]: - lines = [line for line in content.splitlines() if line.strip()] - chat_lines = sum(1 for line in lines if CHAT_LINE_RE.search(line)) - if chat_lines >= 4: - return True, "raw_chat_transcript" - if "完整原始对话" in content or "full transcript" in content.lower(): - return True, "raw_chat_transcript" - return False, "" - - -def detect_large_log(content: str) -> tuple[bool, str]: - lines = [line for line in content.splitlines() if line.strip()] - log_lines = sum(1 for line in lines if LOG_LINE_RE.search(line)) - if len(content) > 4000 or len(lines) > 40 or log_lines >= 8: - return True, "large_or_raw_log" - return False, "" - - -def detect_low_value_memory(content: str) -> tuple[bool, str]: - normalized = re.sub(r"\s+", " ", content).strip().lower() - stable_signal = re.search(r"记住|偏好|长期|决策|结论|约束|preference|remember|decision|constraint", normalized, re.I) - if stable_signal: - return False, "" - if len(normalized) < 12: - return True, "too_short" - small_talk = { - "hi", - "hello", - "thanks", - "thank you", - "ok", - "好的", - "谢谢", - "你好", - "收到", - "再见", - } - if normalized in small_talk: - return True, "small_talk" - return False, "" - - -def sanitize_memory_content(content: str) -> str: - sanitized = content.strip() - sanitized = re.sub(r"\b(password|api[_-]?key|token|secret)\s*[:=]\s*\S+", r"\1=", sanitized, flags=re.I) - sanitized = re.sub(r"\bbearer\s+[a-z0-9._\-]{12,}", "Bearer ", sanitized, flags=re.I) - sanitized = re.sub(r"-----BEGIN [A-Z ]*PRIVATE KEY-----.*?-----END [A-Z ]*PRIVATE KEY-----", "", sanitized, flags=re.I | re.S) - return sanitized - - -def validate_memory_write(content: str, *, allow_low_value: bool = False) -> dict[str, Any]: - if not content or not content.strip(): - return {"allowed": False, "reason": "empty_content", "sanitized_content": ""} - checks = [detect_secret, detect_raw_transcript, detect_large_log] - for check in checks: - blocked, reason = check(content) - if blocked: - return {"allowed": False, "reason": reason, "sanitized_content": ""} - if CHAIN_OF_THOUGHT_RE.search(content): - return {"allowed": False, "reason": "chain_of_thought", "sanitized_content": ""} - low_value, reason = detect_low_value_memory(content) - if low_value and not allow_low_value: - return {"allowed": False, "reason": reason, "sanitized_content": ""} - return {"allowed": True, "reason": "ok", "sanitized_content": sanitize_memory_content(content)} diff --git a/plugins/memory-gateway-agent/memory_gateway_plugin/schemas.py b/plugins/memory-gateway-agent/memory_gateway_plugin/schemas.py deleted file mode 100644 index 7207a9f..0000000 --- a/plugins/memory-gateway-agent/memory_gateway_plugin/schemas.py +++ /dev/null @@ -1,14 +0,0 @@ -from __future__ import annotations - -from dataclasses import dataclass, field -from typing import Any - - -@dataclass -class AgentContext: - user_id: str - agent_id: str = "" - workspace_id: str = "" - session_id: str = "" - metadata: dict[str, Any] = field(default_factory=dict) - diff --git a/plugins/memory-gateway-agent/memory_gateway_plugin/tools.py b/plugins/memory-gateway-agent/memory_gateway_plugin/tools.py deleted file mode 100644 index 3417152..0000000 --- a/plugins/memory-gateway-agent/memory_gateway_plugin/tools.py +++ /dev/null @@ -1,163 +0,0 @@ -from __future__ import annotations - -from typing import Any - -from .client import MemoryGatewayClient -from .config import PluginConfig, load_config -from .safety import validate_memory_write - - -FEEDBACK_MAP = { - "confirm": "useful", - "correct": "useful", - "useful": "useful", - "delete": "incorrect", - "reject": "incorrect", - "incorrect": "incorrect", - "duplicate": "duplicate", - "outdated": "outdated", - "not_useful": "not_useful", -} - - -def _client(client: MemoryGatewayClient | None = None) -> MemoryGatewayClient: - return client or MemoryGatewayClient() - - -def _context_payload(user_id: str, agent_id: str = "", workspace_id: str = "", session_id: str = "") -> dict[str, Any]: - payload: dict[str, Any] = {"user_id": user_id} - if agent_id: - payload["agent_id"] = agent_id - if workspace_id: - payload["workspace_id"] = workspace_id - if session_id: - payload["session_id"] = session_id - return payload - - -def memory_search( - query: str, - user_id: str, - agent_id: str = "", - workspace_id: str = "", - session_id: str = "", - namespaces: list[str] | None = None, - memory_types: list[str] | None = None, - tags: list[str] | None = None, - limit: int = 5, - client: MemoryGatewayClient | None = None, -) -> dict[str, Any]: - if not query or not query.strip(): - return {"ok": False, "error": "query_required"} - payload = _context_payload(user_id, agent_id, workspace_id, session_id) - payload.update( - { - "query": query.strip(), - "namespaces": namespaces or [], - "memory_types": memory_types or [], - "tags": tags or [], - "limit": limit, - } - ) - return _client(client).search_memory(payload) - - -def memory_append_episode( - user_id: str, - agent_id: str, - session_id: str, - content: str = "", - episode_summary: str = "", - workspace_id: str = "", - source: str = "conversation", - tags: list[str] | None = None, - importance: float | None = None, - confidence: float | None = None, - client: MemoryGatewayClient | None = None, -) -> dict[str, Any]: - candidate = (episode_summary or content or "").strip() - validation = validate_memory_write(candidate) - if not validation["allowed"]: - return {"ok": False, "error": "memory_write_rejected", "reason": validation["reason"]} - payload = _context_payload(user_id, agent_id, workspace_id, session_id) - payload.update({"content": validation["sanitized_content"], "tags": tags or [], "source": source}) - if importance is not None: - payload["events"] = [{"type": "importance_hint", "value": importance}] - if confidence is not None: - payload.setdefault("events", []).append({"type": "confidence_hint", "value": confidence}) - return _client(client).append_episode(payload) - - -def memory_commit_session( - user_id: str, - agent_id: str, - session_id: str, - workspace_id: str = "", - promote: bool = True, - min_importance: float = 0.6, - client: MemoryGatewayClient | None = None, -) -> dict[str, Any]: - payload = _context_payload(user_id, agent_id, workspace_id, session_id) - payload.update({"promote": promote, "min_importance": min_importance}) - return _client(client).commit_session(session_id, payload) - - -def memory_upsert( - user_id: str, - agent_id: str, - content: str, - workspace_id: str = "", - namespace: str = "", - memory_type: str = "fact", - summary: str = "", - tags: list[str] | None = None, - importance: float = 0.5, - confidence: float = 0.8, - visibility: str = "private", - source: str = "agent", - client: MemoryGatewayClient | None = None, -) -> dict[str, Any]: - validation = validate_memory_write(content) - if not validation["allowed"]: - return {"ok": False, "error": "memory_write_rejected", "reason": validation["reason"]} - payload = _context_payload(user_id, agent_id, workspace_id) - payload.update( - { - "namespace": namespace or None, - "memory_type": memory_type, - "content": validation["sanitized_content"], - "summary": summary or None, - "tags": tags or [], - "importance": importance, - "confidence": confidence, - "visibility": visibility, - "source": source, - } - ) - return _client(client).upsert_memory(payload) - - -def memory_feedback( - user_id: str, - agent_id: str, - memory_id: str, - feedback: str, - workspace_id: str = "", - session_id: str = "", - comment: str = "", - client: MemoryGatewayClient | None = None, -) -> dict[str, Any]: - mapped_feedback = FEEDBACK_MAP.get(feedback, feedback) - payload = _context_payload(user_id, agent_id, workspace_id, session_id) - payload.update({"feedback": mapped_feedback, "comment": comment or None}) - return _client(client).send_feedback(memory_id, payload) - - -def default_context(config: PluginConfig | None = None) -> dict[str, str]: - cfg = config or load_config() - return { - "user_id": cfg.default_user_id, - "agent_id": cfg.default_agent_id, - "workspace_id": cfg.default_workspace_id, - } - diff --git a/plugins/memory-gateway-agent/memory_gateway_plugin/trace.py b/plugins/memory-gateway-agent/memory_gateway_plugin/trace.py deleted file mode 100644 index 0e2a707..0000000 --- a/plugins/memory-gateway-agent/memory_gateway_plugin/trace.py +++ /dev/null @@ -1,45 +0,0 @@ -from __future__ import annotations - -import json -import os -from datetime import datetime, timezone -from pathlib import Path -from typing import Any - -from .output import redact, short_id - - -def trace_enabled() -> bool: - return os.environ.get("MEMORY_GATEWAY_PLUGIN_TRACE_HOOKS", "").strip().lower() in {"1", "true", "yes", "on"} - - -def trace_path() -> Path: - return Path(__file__).resolve().parents[1] / ".tmp" / "hook_trace.log" - - -def trace_hook( - hook_name: str, - *, - session_id: str = "", - gateway_action: str = "", - gateway_called: bool = False, - ok: bool | None = None, - audit_delta: int | None = None, - reason: str = "", -) -> None: - if not trace_enabled(): - return - path = trace_path() - path.parent.mkdir(parents=True, exist_ok=True) - payload: dict[str, Any] = { - "timestamp": datetime.now(timezone.utc).isoformat(), - "hook": hook_name, - "session_id": short_id(session_id), - "gateway_action": gateway_action, - "gateway_called": gateway_called, - "ok": ok, - "audit_delta": audit_delta, - "reason": reason[:160] if reason else "", - } - with path.open("a", encoding="utf-8") as handle: - handle.write(json.dumps(redact(payload), ensure_ascii=False, default=str) + "\n") diff --git a/plugins/memory-gateway-agent/openclaw.plugin.yaml b/plugins/memory-gateway-agent/openclaw.plugin.yaml deleted file mode 100644 index e6bc1cf..0000000 --- a/plugins/memory-gateway-agent/openclaw.plugin.yaml +++ /dev/null @@ -1,26 +0,0 @@ -name: memory-gateway-agent -runtime: openclaw -version: 0.1.0 -description: Draft OpenClaw plugin manifest for Memory Gateway v1. Adjust field names to the actual OpenClaw runtime schema before production use. -entrypoint: memory_gateway_plugin:register -config: - gateway_url: ${MEMORY_GATEWAY_URL:-http://127.0.0.1:1934} - api_key_env: MEMORY_GATEWAY_API_KEY -tools: - - name: memory_search - - name: memory_append_episode - - name: memory_commit_session - - name: memory_upsert - - name: memory_feedback -hooks: - - name: pre_llm_call - handler: memory_gateway_plugin.lifecycle:on_conversation_start - - name: post_llm_call - handler: memory_gateway_plugin.lifecycle:after_user_message - - name: session_end - handler: memory_gateway_plugin.lifecycle:on_session_end -safety: - stores_full_raw_conversation: false - rejects_secrets: true - long_term_commit_via_everos: true - diff --git a/plugins/memory-gateway-agent/plugin.yaml b/plugins/memory-gateway-agent/plugin.yaml deleted file mode 100644 index 4da0b0b..0000000 --- a/plugins/memory-gateway-agent/plugin.yaml +++ /dev/null @@ -1,55 +0,0 @@ -name: memory-gateway-agent -version: 0.1.0 -description: Generic AI Agent plugin adapter for the existing Memory Gateway v1 HTTP API. -type: agent-plugin -provides_tools: - - memory_search - - memory_append_episode - - memory_commit_session - - memory_upsert - - memory_feedback -provides_hooks: - - pre_llm_call - - post_llm_call - - session_end - - after_task_complete -requires_env: [] -entrypoint: register -transport: - type: http - target: ${MEMORY_GATEWAY_URL:-http://127.0.0.1:1934} -privacy: - stores_full_raw_conversation: false - writes_long_term_directly_by_default: false - auto_commit_session_default: false -configuration: - MEMORY_GATEWAY_URL: - default: http://127.0.0.1:1934 - MEMORY_GATEWAY_API_KEY: - secret: true - required: false - MEMORY_GATEWAY_DEFAULT_USER_ID: - required: false - MEMORY_GATEWAY_DEFAULT_AGENT_ID: - required: false - MEMORY_GATEWAY_DEFAULT_WORKSPACE_ID: - required: false - MEMORY_GATEWAY_AUTO_SEARCH: - default: "true" - MEMORY_GATEWAY_AUTO_APPEND_EPISODE: - default: "true" - MEMORY_GATEWAY_AUTO_COMMIT_SESSION: - default: "false" - MEMORY_GATEWAY_REVIEW_MODE: - default: "true" -tools: - - memory_search - - memory_append_episode - - memory_commit_session - - memory_upsert - - memory_feedback -hooks: - - pre_llm_call - - post_llm_call - - session_end - - after_task_complete diff --git a/plugins/memory-gateway-agent/policies/memory_policy.md b/plugins/memory-gateway-agent/policies/memory_policy.md deleted file mode 100644 index 34c12e9..0000000 --- a/plugins/memory-gateway-agent/policies/memory_policy.md +++ /dev/null @@ -1,34 +0,0 @@ -# Memory Gateway Agent Policy - -Use Memory Gateway as a shared memory adapter. It is not a transcript store. - -At conversation start: - -- Search memory when previous context may matter. -- Use `memory_search` with the current `user_id`, `agent_id`, `workspace_id`, and `session_id`. -- Inject only compact relevant memory summaries into the working context. - -During a task: - -- Write only candidate episode summaries with `memory_append_episode`. -- Save stable preferences, long-term project facts, architecture decisions, durable constraints, reusable workflows, and completed task conclusions. -- Do not save complete raw conversations, chain-of-thought, large logs, one-time values, or secrets. - -At task or session completion: - -- Use `memory_commit_session` to let Memory Gateway and EverOS decide what can be promoted. -- Do not promote all episodes directly to long-term memory. -- Conflicting or high-value memories should enter review rather than overwrite existing memory. - -When the user says to forget or reject memory: - -- Use `memory_feedback` with `incorrect`, `outdated`, or `not_useful`. -- Use delete-capable tools only when the runtime exposes them and access control allows it. - -Default automation: - -- Auto search may be enabled. -- Auto append episode may be enabled for safe summaries. -- Auto commit is disabled by default. -- Auto direct long-term upsert is disabled by default. - diff --git a/plugins/memory-gateway-agent/policies/safety_filter.md b/plugins/memory-gateway-agent/policies/safety_filter.md deleted file mode 100644 index df630d5..0000000 --- a/plugins/memory-gateway-agent/policies/safety_filter.md +++ /dev/null @@ -1,24 +0,0 @@ -# Memory Gateway Safety Filter - -The plugin must reject memory writes that contain: - -- passwords -- API keys -- tokens -- secrets -- bearer tokens -- cookies -- session IDs -- private keys -- SSH keys -- one-time passwords or verification codes -- large raw logs -- full chat transcripts -- chain-of-thought or hidden reasoning -- unconfirmed sensitive personal attributes -- low-value temporary chatter - -The plugin stores summaries rather than raw messages. If a message is useful but contains sensitive detail, redact the sensitive detail before writing. If redaction would remove the meaning, reject the write. - -Long-term memory should normally be created by session commit and EverOS consolidation, not by direct upsert. - diff --git a/plugins/memory-gateway-agent/schemas.py b/plugins/memory-gateway-agent/schemas.py deleted file mode 100644 index 210b6fd..0000000 --- a/plugins/memory-gateway-agent/schemas.py +++ /dev/null @@ -1,107 +0,0 @@ -from __future__ import annotations - - -MEMORY_SEARCH = { - "name": "memory_search", - "description": "Search accessible Memory Gateway records for the current user/agent context.", - "parameters": { - "type": "object", - "properties": { - "query": {"type": "string", "description": "Search query. Must not be empty."}, - "user_id": {"type": "string", "description": "Memory Gateway user id."}, - "agent_id": {"type": "string", "description": "Calling agent id, for ACL and namespace routing."}, - "workspace_id": {"type": "string", "description": "Optional workspace/project id."}, - "session_id": {"type": "string", "description": "Optional session id for session-scoped context."}, - "namespaces": {"type": "array", "items": {"type": "string"}, "description": "Optional namespace filters."}, - "memory_types": {"type": "array", "items": {"type": "string"}, "description": "Optional memory type filters."}, - "tags": {"type": "array", "items": {"type": "string"}, "description": "Optional tag filters."}, - "limit": {"type": "integer", "description": "Maximum result count.", "default": 5}, - }, - "required": ["query", "user_id", "agent_id"], - }, -} - -MEMORY_APPEND_EPISODE = { - "name": "memory_append_episode", - "description": "Append a safe summarized candidate episode. Does not save full raw conversation or directly promote long-term memory.", - "parameters": { - "type": "object", - "properties": { - "content": {"type": "string", "description": "Safe summarized episode content. Do not pass raw transcripts."}, - "episode_summary": {"type": "string", "description": "Optional prebuilt summary. Used instead of content when provided."}, - "user_id": {"type": "string", "description": "Memory Gateway user id."}, - "agent_id": {"type": "string", "description": "Calling agent id."}, - "workspace_id": {"type": "string", "description": "Optional workspace/project id."}, - "session_id": {"type": "string", "description": "Current session id."}, - "tags": {"type": "array", "items": {"type": "string"}, "description": "Optional tags."}, - "source": {"type": "string", "description": "Source label.", "default": "conversation"}, - }, - "required": ["content", "user_id", "agent_id", "session_id"], - }, -} - -MEMORY_COMMIT_SESSION = { - "name": "memory_commit_session", - "description": "Commit a session through Memory Gateway and EverOS. Promotes only what consolidation accepts.", - "parameters": { - "type": "object", - "properties": { - "user_id": {"type": "string", "description": "Memory Gateway user id."}, - "agent_id": {"type": "string", "description": "Calling agent id."}, - "workspace_id": {"type": "string", "description": "Optional workspace/project id."}, - "session_id": {"type": "string", "description": "Session id to commit."}, - "promote": {"type": "boolean", "description": "Whether promotion is allowed.", "default": True}, - "min_importance": {"type": "number", "description": "Minimum importance threshold.", "default": 0.6}, - }, - "required": ["user_id", "agent_id", "session_id"], - }, -} - -MEMORY_UPSERT = { - "name": "memory_upsert", - "description": "High-risk direct memory write. Use only for stable, concise, user-approved long-term memory; do not call automatically.", - "parameters": { - "type": "object", - "properties": { - "user_id": {"type": "string", "description": "Memory Gateway user id."}, - "agent_id": {"type": "string", "description": "Calling agent id."}, - "workspace_id": {"type": "string", "description": "Optional workspace/project id."}, - "namespace": {"type": "string", "description": "Optional explicit namespace, e.g. user/{user_id}/long_term."}, - "memory_type": {"type": "string", "description": "Memory type, e.g. preference, decision, fact, procedure."}, - "content": {"type": "string", "description": "Stable memory content. Do not pass full raw conversation."}, - "summary": {"type": "string", "description": "Optional concise summary."}, - "tags": {"type": "array", "items": {"type": "string"}, "description": "Optional tags."}, - "importance": {"type": "number", "description": "Importance score 0..1.", "default": 0.5}, - "confidence": {"type": "number", "description": "Confidence score 0..1.", "default": 0.8}, - "visibility": {"type": "string", "description": "Memory visibility.", "default": "private"}, - }, - "required": ["user_id", "agent_id", "content", "memory_type"], - }, -} - -MEMORY_FEEDBACK = { - "name": "memory_feedback", - "description": "Send quality feedback for an existing memory record.", - "parameters": { - "type": "object", - "properties": { - "memory_id": {"type": "string", "description": "Memory id to mark."}, - "user_id": {"type": "string", "description": "Memory Gateway user id."}, - "agent_id": {"type": "string", "description": "Calling agent id."}, - "workspace_id": {"type": "string", "description": "Optional workspace/project id."}, - "session_id": {"type": "string", "description": "Optional session id."}, - "feedback": {"type": "string", "description": "Feedback, e.g. confirm, correct, delete, reject, incorrect, duplicate, outdated."}, - "comment": {"type": "string", "description": "Optional feedback comment."}, - }, - "required": ["memory_id", "user_id", "agent_id", "feedback"], - }, -} - -TOOL_SCHEMAS = { - "memory_search": MEMORY_SEARCH, - "memory_append_episode": MEMORY_APPEND_EPISODE, - "memory_commit_session": MEMORY_COMMIT_SESSION, - "memory_upsert": MEMORY_UPSERT, - "memory_feedback": MEMORY_FEEDBACK, -} - diff --git a/plugins/memory-gateway-agent/scripts/cleanup_test_memories.py b/plugins/memory-gateway-agent/scripts/cleanup_test_memories.py deleted file mode 100644 index 8fa8633..0000000 --- a/plugins/memory-gateway-agent/scripts/cleanup_test_memories.py +++ /dev/null @@ -1,187 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import json -import os -import sys -import urllib.error -import urllib.parse -import urllib.request -from pathlib import Path -from typing import Any - -PLUGIN_ROOT = Path(__file__).resolve().parents[1] -if str(PLUGIN_ROOT) not in sys.path: - sys.path.insert(0, str(PLUGIN_ROOT)) - -from memory_gateway_plugin.output import dumps_safe, short_id, summarize_data - - -USER_ID = "test_user_memory_gateway_plugin" -AGENT_ID = "test_hermes_memory_gateway_plugin" -WORKSPACE_ID = "test_workspace_memory_gateway_plugin" -SESSION_ID = "test_session_memory_gateway_plugin_001" - - -def _assert_test_user(user_id: str) -> None: - if not user_id.startswith("test_user_"): - raise ValueError("cleanup_refuses_non_test_user") - - -def _gateway_url() -> str: - return os.environ.get("MEMORY_GATEWAY_URL", "http://127.0.0.1:1934").rstrip("/") - - -def _api_key() -> str: - return os.environ.get("MEMORY_GATEWAY_API_KEY", "") - - -def _request(method: str, endpoint: str, payload: dict[str, Any] | None = None) -> dict[str, Any]: - headers = {"Content-Type": "application/json"} - if _api_key(): - headers["X-API-Key"] = _api_key() - body = None if payload is None else json.dumps(payload, ensure_ascii=False).encode("utf-8") - req = urllib.request.Request(_gateway_url() + endpoint, data=body, headers=headers, method=method) - try: - with urllib.request.urlopen(req, timeout=15) as response: - raw = response.read().decode("utf-8") - return {"ok": True, "status_code": getattr(response, "status", 200), "data": json.loads(raw) if raw else {}} - except urllib.error.HTTPError as exc: - return {"ok": False, "status_code": exc.code, "error": str(exc.reason)[:300]} - except Exception as exc: - return {"ok": False, "status_code": None, "error": str(exc)[:300]} - - -def _search_candidates() -> dict[str, Any]: - return _request( - "POST", - "/v1/memory/search", - { - "query": "integration_test plugin safe_to_delete Memory Gateway plugin", - "user_id": USER_ID, - "agent_id": AGENT_ID, - "workspace_id": WORKSPACE_ID, - "session_id": SESSION_ID, - "tags": ["integration_test"], - "limit": 100, - }, - ) - - -def _audit_candidate_ids() -> list[str]: - result = _request("GET", "/v1/audit?limit=1000") - if not result.get("ok"): - return [] - ids: list[str] = [] - for row in result.get("data") or []: - if row.get("actor_user_id") != USER_ID: - continue - if row.get("actor_agent_id") not in {AGENT_ID, None, ""}: - continue - if row.get("target_type") == "memory" and row.get("action") in {"upsert_memory", "feedback:incorrect", "feedback:duplicate", "feedback:outdated"}: - target_id = row.get("target_id") - if target_id and target_id not in ids: - ids.append(target_id) - return ids - - -def _memory_from_result(item: dict[str, Any]) -> dict[str, Any] | None: - memory = item.get("memory") - if isinstance(memory, dict) and memory.get("id"): - return memory - return None - - -def _is_cleanup_candidate(memory: dict[str, Any]) -> bool: - if memory.get("user_id") != USER_ID: - return False - tags = set(memory.get("tags") or []) - return bool(tags.intersection({"integration_test", "safe_to_delete", "plugin"})) - - -def _delete_memory(memory_id: str) -> dict[str, Any]: - query = urllib.parse.urlencode({"user_id": USER_ID, "agent_id": AGENT_ID, "workspace_id": WORKSPACE_ID, "session_id": SESSION_ID}) - return _request("DELETE", f"/v1/memory/{urllib.parse.quote(memory_id)}?{query}") - - -def _feedback_memory(memory_id: str) -> dict[str, Any]: - return _request( - "POST", - f"/v1/memory/{urllib.parse.quote(memory_id)}/feedback", - { - "user_id": USER_ID, - "agent_id": AGENT_ID, - "workspace_id": WORKSPACE_ID, - "session_id": SESSION_ID, - "feedback": "incorrect", - "comment": "cleanup marker for integration test memory", - }, - ) - - -def run(user_id: str = USER_ID) -> dict[str, Any]: - _assert_test_user(user_id) - if user_id != USER_ID: - return {"ok": False, "error": "script_is_scoped_to_fixed_test_user", "user_id": user_id} - - search = _search_candidates() - if not search.get("ok"): - return {"ok": False, "search": {"ok": False, "status_code": search.get("status_code"), "error": search.get("error")}, "deleted": 0, "feedback_marked": 0, "skipped": 0} - - rows = (search.get("data") or {}).get("results") or [] - memory_ids = _audit_candidate_ids() - deleted = 0 - feedback_marked = 0 - skipped = 0 - unable: list[dict[str, Any]] = [] - touched: list[str] = [] - - for item in rows: - memory = _memory_from_result(item) - if not memory or not _is_cleanup_candidate(memory): - skipped += 1 - continue - memory_id = memory["id"] - if memory_id not in memory_ids: - memory_ids.append(memory_id) - - for memory_id in memory_ids: - deletion = _delete_memory(memory_id) - if deletion.get("ok"): - deleted += 1 - touched.append(short_id(memory_id)) - continue - if deletion.get("status_code") == 404: - skipped += 1 - continue - feedback = _feedback_memory(memory_id) - if feedback.get("ok"): - feedback_marked += 1 - touched.append(short_id(memory_id)) - else: - unable.append({"memory_id": short_id(memory_id), "delete_status": deletion.get("status_code"), "feedback_status": feedback.get("status_code"), "reason": feedback.get("error") or deletion.get("error")}) - - return { - "ok": not unable, - "search": {"ok": True, "status_code": search.get("status_code"), "data": summarize_data(search.get("data"))}, - "deleted": deleted, - "feedback_marked": feedback_marked, - "skipped": skipped, - "unable_count": len(unable), - "unable": unable, - "touched_memory_ids": touched, - "limitation": "search API returns local MemoryRecord rows plus OpenViking context; cleanup only deletes local MemoryRecord rows for the fixed test user.", - } - - -def main() -> int: - try: - result = run(os.environ.get("MEMORY_GATEWAY_CLEANUP_USER_ID", USER_ID)) - except ValueError as exc: - result = {"ok": False, "error": str(exc), "deleted": 0, "feedback_marked": 0, "skipped": 0} - print(dumps_safe(result)) - return 0 if result.get("ok") else 1 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/plugins/memory-gateway-agent/scripts/gateway_e2e_check.py b/plugins/memory-gateway-agent/scripts/gateway_e2e_check.py deleted file mode 100644 index 8a8a24c..0000000 --- a/plugins/memory-gateway-agent/scripts/gateway_e2e_check.py +++ /dev/null @@ -1,218 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import json -import os -import sys -import urllib.error -import urllib.request -from dataclasses import dataclass -from pathlib import Path -from typing import Any - -PLUGIN_ROOT = Path(__file__).resolve().parents[1] -if str(PLUGIN_ROOT) not in sys.path: - sys.path.insert(0, str(PLUGIN_ROOT)) - -from memory_gateway_plugin.client import MemoryGatewayClient -from memory_gateway_plugin.config import PluginConfig -from memory_gateway_plugin.output import debug_raw_enabled, dumps_safe, redact, short_id, summarize_data, summarize_result -from memory_gateway_plugin.tools import ( - memory_append_episode, - memory_commit_session, - memory_feedback, - memory_search, - memory_upsert, -) - - -USER_ID = "test_user_memory_gateway_plugin" -AGENT_ID = "test_hermes_memory_gateway_plugin" -WORKSPACE_ID = "test_workspace_memory_gateway_plugin" -SESSION_ID = "test_session_memory_gateway_plugin_001" - - -def _short(value: Any, max_chars: int = 700) -> str: - text = json.dumps(redact(value), ensure_ascii=False, default=str) - return text[:max_chars] - - -def _summary_data(data: dict[str, Any] | None) -> dict[str, Any]: - return summarize_data(data) if isinstance(summarize_data(data), dict) else {} - - -def _result_detail(result: dict[str, Any]) -> str: - return _short(summarize_result(result)) - - -@dataclass -class Step: - name: str - ok: bool - endpoint: str = "" - status_code: int | None = None - detail: str = "" - data: dict[str, Any] | None = None - - def to_dict(self) -> dict[str, Any]: - return { - "name": self.name, - "ok": self.ok, - "endpoint": self.endpoint, - "status_code": self.status_code, - "detail": self.detail, - "data": _summary_data(self.data), - } - - -def _request(method: str, url: str, payload: dict[str, Any] | None = None, api_key: str = "") -> dict[str, Any]: - headers = {"Content-Type": "application/json"} - if api_key: - headers["X-API-Key"] = api_key - body = None if payload is None else json.dumps(payload, ensure_ascii=False).encode("utf-8") - req = urllib.request.Request(url, data=body, headers=headers, method=method) - try: - with urllib.request.urlopen(req, timeout=10) as response: - raw = response.read().decode("utf-8") - return {"ok": True, "status_code": getattr(response, "status", 200), "data": json.loads(raw) if raw else {}} - except urllib.error.HTTPError as exc: - try: - body_text = exc.read().decode("utf-8") - except Exception: - body_text = str(exc.reason) - return {"ok": False, "status_code": exc.code, "error": body_text[:700]} - except Exception as exc: - return {"ok": False, "status_code": None, "error": str(exc)[:700]} - - -def _health(config: PluginConfig) -> Step: - endpoint = "/health" - result = _request("GET", config.gateway_url.rstrip("/") + endpoint, api_key=config.api_key) - return Step("health", bool(result.get("ok")), endpoint, result.get("status_code"), _result_detail(result), result.get("data")) - - -def _ensure_user(config: PluginConfig) -> Step: - endpoint = "/v1/users" - result = _request( - "POST", - config.gateway_url.rstrip("/") + endpoint, - {"user_id": USER_ID, "display_name": "Memory Gateway Plugin Integration Test", "preferences": {"purpose": "integration_test"}}, - api_key=config.api_key, - ) - ok = bool(result.get("ok")) or result.get("status_code") in {200, 201, 409} - return Step("ensure_user", ok, endpoint, result.get("status_code"), _result_detail(result), result.get("data")) - - -def _client(config: PluginConfig) -> MemoryGatewayClient: - return MemoryGatewayClient(config) - - -def run() -> dict[str, Any]: - config = PluginConfig.from_env() - client = _client(config) - steps: list[Step] = [] - - steps.append(_health(config)) - if not steps[-1].ok: - return {"ok": False, "steps": [s.to_dict() for s in steps]} - - steps.append(_ensure_user(config)) - - search_1 = memory_search( - query="Memory Gateway plugin integration test", - user_id=USER_ID, - agent_id=AGENT_ID, - workspace_id=WORKSPACE_ID, - session_id=SESSION_ID, - limit=5, - client=client, - ) - steps.append(Step("memory_search_initial", bool(search_1.get("ok")), "/v1/memory/search", search_1.get("status_code"), _result_detail(search_1), search_1.get("data"))) - - episode = memory_append_episode( - user_id=USER_ID, - agent_id=AGENT_ID, - workspace_id=WORKSPACE_ID, - session_id=SESSION_ID, - content="Integration test: user prefers Memory Gateway plugin to store only summarized episodes, not raw transcripts.", - tags=["integration_test", "plugin"], - source="agent", - importance=0.2, - confidence=0.5, - client=client, - ) - steps.append(Step("memory_append_episode", bool(episode.get("ok")), "/v1/episodes", episode.get("status_code"), _result_detail(episode), episode.get("data"))) - - commit = memory_commit_session( - user_id=USER_ID, - agent_id=AGENT_ID, - workspace_id=WORKSPACE_ID, - session_id=SESSION_ID, - promote=True, - min_importance=0.1, - client=client, - ) - commit_detail = _result_detail(commit) - if commit.get("ok") and not (commit.get("data") or {}).get("promoted"): - commit_detail += " | promotion may be empty while commit endpoint succeeded" - steps.append(Step("memory_commit_session", bool(commit.get("ok")), f"/v1/sessions/{SESSION_ID}/commit", commit.get("status_code"), commit_detail, commit.get("data"))) - - upsert = memory_upsert( - user_id=USER_ID, - agent_id=AGENT_ID, - workspace_id=WORKSPACE_ID, - namespace=f"user/{USER_ID}/long_term", - memory_type="preference", - content="Integration test memory: this should be removable or clearly tagged as test data.", - tags=["integration_test", "plugin", "safe_to_delete"], - importance=0.1, - confidence=0.5, - source="agent", - client=client, - ) - steps.append(Step("memory_upsert", bool(upsert.get("ok")), "/v1/memory", upsert.get("status_code"), _result_detail(upsert), upsert.get("data"))) - - search_2 = memory_search( - query="Integration test memory summarized episodes raw transcripts", - user_id=USER_ID, - agent_id=AGENT_ID, - workspace_id=WORKSPACE_ID, - session_id=SESSION_ID, - limit=10, - client=client, - ) - result_count = len((search_2.get("data") or {}).get("results", [])) - detail = _result_detail(search_2) - if search_2.get("ok") and result_count == 0: - detail += " | search succeeded but returned no results; indexing or OpenViking sync may be asynchronous" - steps.append(Step("memory_search_after_write", bool(search_2.get("ok")), "/v1/memory/search", search_2.get("status_code"), detail, search_2.get("data"))) - - memory_id = ((upsert.get("data") or {}).get("memory") or upsert.get("data") or {}).get("id") - if memory_id: - feedback = memory_feedback( - user_id=USER_ID, - agent_id=AGENT_ID, - workspace_id=WORKSPACE_ID, - session_id=SESSION_ID, - memory_id=memory_id, - feedback="reject", - comment="Integration test cleanup marker; safe to ignore/delete.", - client=client, - ) - steps.append(Step("memory_feedback", bool(feedback.get("ok")), f"/v1/memory/{memory_id}/feedback", feedback.get("status_code"), _result_detail(feedback), feedback.get("data"))) - else: - steps.append(Step("memory_feedback", False, "/v1/memory/{memory_id}/feedback", None, "skipped because memory_upsert did not return memory id")) - - required = {"health", "memory_search_initial", "memory_append_episode", "memory_commit_session", "memory_upsert", "memory_search_after_write", "memory_feedback"} - ok = all(step.ok for step in steps if step.name in required) - return {"ok": ok, "gateway_url": config.gateway_url, "debug_raw": debug_raw_enabled(), "test_identity": {"user_id": USER_ID, "agent_id": AGENT_ID, "workspace_id": WORKSPACE_ID, "session_id": SESSION_ID}, "steps": [s.to_dict() for s in steps]} - - -def main() -> int: - result = run() - print(dumps_safe(result)) - return 0 if result.get("ok") else 1 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/plugins/memory-gateway-agent/scripts/health.py b/plugins/memory-gateway-agent/scripts/health.py deleted file mode 100644 index e195e2d..0000000 --- a/plugins/memory-gateway-agent/scripts/health.py +++ /dev/null @@ -1,32 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import json -import sys -import urllib.error -import urllib.request -from pathlib import Path - -PLUGIN_ROOT = Path(__file__).resolve().parents[1] -if str(PLUGIN_ROOT) not in sys.path: - sys.path.insert(0, str(PLUGIN_ROOT)) - -from memory_gateway_plugin.config import load_config -from memory_gateway_plugin.output import dumps_safe, summarize_data - - -def main() -> None: - config = load_config() - request = urllib.request.Request(config.gateway_url.rstrip("/") + "/health", method="GET") - if config.api_key: - request.add_header("X-API-Key", config.api_key) - try: - with urllib.request.urlopen(request, timeout=config.timeout) as response: - payload = json.loads(response.read().decode("utf-8")) - print(dumps_safe({"ok": True, "endpoint": "/health", "status_code": getattr(response, "status", 200), "data": summarize_data(payload)})) - except urllib.error.URLError as exc: - print(dumps_safe({"ok": False, "endpoint": "/health", "status_code": None, "error": str(exc)[:300]})) - - -if __name__ == "__main__": - main() diff --git a/plugins/memory-gateway-agent/scripts/hermes_hook_probe.py b/plugins/memory-gateway-agent/scripts/hermes_hook_probe.py deleted file mode 100644 index 1a69a7e..0000000 --- a/plugins/memory-gateway-agent/scripts/hermes_hook_probe.py +++ /dev/null @@ -1,174 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import json -import os -import sys -import urllib.error -import urllib.request -from pathlib import Path -from typing import Any - - -USER_ID = "test_user_memory_gateway_plugin" -AGENT_ID = "test_hermes_memory_gateway_plugin" -WORKSPACE_ID = "test_workspace_memory_gateway_plugin" -SESSION_ID = "test_session_memory_gateway_plugin_001" - - -def _ensure_paths() -> None: - plugin_root = Path(__file__).resolve().parents[1] - hermes_repo = Path(os.environ.get("HERMES_REPO", "/home/tom/.hermes/hermes-agent")) - hermes_cli = hermes_repo / "hermes_cli" - for path in [plugin_root, hermes_repo, hermes_cli]: - if str(path) not in sys.path: - sys.path.insert(0, str(path)) - - -_ensure_paths() - -from memory_gateway_plugin.output import dumps_safe, summarize_data - - -def _request(method: str, url: str, payload: dict[str, Any] | None = None, api_key: str = "") -> dict[str, Any]: - headers = {"Content-Type": "application/json"} - if api_key: - headers["X-API-Key"] = api_key - body = None if payload is None else json.dumps(payload, ensure_ascii=False).encode("utf-8") - req = urllib.request.Request(url, data=body, headers=headers, method=method) - try: - with urllib.request.urlopen(req, timeout=10) as response: - raw = response.read().decode("utf-8") - return {"ok": True, "status_code": getattr(response, "status", 200), "data": json.loads(raw) if raw else {}} - except urllib.error.HTTPError as exc: - try: - body_text = exc.read().decode("utf-8") - except Exception: - body_text = str(exc.reason) - return {"ok": False, "status_code": exc.code, "error": body_text[:500]} - except Exception as exc: - return {"ok": False, "status_code": None, "error": str(exc)[:500]} - - -def _ensure_user() -> dict[str, Any]: - gateway_url = os.environ.get("MEMORY_GATEWAY_URL", "http://127.0.0.1:1934").rstrip("/") - api_key = os.environ.get("MEMORY_GATEWAY_API_KEY", "") - return _request( - "POST", - gateway_url + "/v1/users", - {"user_id": USER_ID, "display_name": "Memory Gateway Hook Probe", "preferences": {"purpose": "hook_probe"}}, - api_key=api_key, - ) - - -def _audit_count(action: str) -> int: - gateway_url = os.environ.get("MEMORY_GATEWAY_URL", "http://127.0.0.1:1934").rstrip("/") - api_key = os.environ.get("MEMORY_GATEWAY_API_KEY", "") - result = _request("GET", gateway_url + "/v1/audit?limit=1000", api_key=api_key) - if not result.get("ok"): - return -1 - rows = result.get("data") or [] - return sum( - 1 - for row in rows - if row.get("action") == action - and row.get("actor_user_id") == USER_ID - and row.get("actor_agent_id") == AGENT_ID - ) - - -def _hook_report(manager: Any, hook_name: str, payload: dict[str, Any], audit_action: str = "") -> dict[str, Any]: - registered = hook_name in getattr(manager, "_hooks", {}) and bool(manager._hooks[hook_name]) - before = _audit_count(audit_action) if audit_action else -1 - try: - result = manager.invoke_hook(hook_name, **payload) - after = _audit_count(audit_action) if audit_action else -1 - return { - "registered": registered, - "invoked": True, - "result_type": type(result).__name__, - "result": summarize_data(result), - "audit_action": audit_action, - "audit_delta": (after - before) if before >= 0 and after >= 0 else None, - "error": "", - } - except Exception as exc: - return { - "registered": registered, - "invoked": False, - "result_type": "", - "result": None, - "audit_action": audit_action, - "audit_delta": None, - "error": str(exc)[:500], - } - - -def run(auto_commit: bool = False) -> dict[str, Any]: - os.environ.setdefault("MEMORY_GATEWAY_URL", "http://127.0.0.1:1934") - os.environ["MEMORY_GATEWAY_DEFAULT_USER_ID"] = USER_ID - os.environ["MEMORY_GATEWAY_DEFAULT_AGENT_ID"] = AGENT_ID - os.environ["MEMORY_GATEWAY_DEFAULT_WORKSPACE_ID"] = WORKSPACE_ID - os.environ["MEMORY_GATEWAY_AUTO_COMMIT_SESSION"] = "true" if auto_commit else "false" - - from plugins import PluginManager - - ensure_user = _ensure_user() - manager = PluginManager() - manager.discover_and_load() - - base = { - "user_id": USER_ID, - "agent_id": AGENT_ID, - "workspace_id": WORKSPACE_ID, - "session_id": SESSION_ID, - "task_id": SESSION_ID, - "model": "hook-probe", - "platform": "cli", - } - hooks = { - "on_session_start": dict(base), - "pre_llm_call": { - **base, - "user_message": "Memory Gateway plugin integration test memory preference", - "conversation_history": [], - "is_first_turn": True, - }, - "post_llm_call": { - **base, - "user_message": "请记住：Memory Gateway plugin hook probe 偏好保存简短摘要型 episode。", - "assistant_response": "已记录为候选摘要，后续由 session commit 判断是否提升为长期记忆。", - }, - "on_session_end": dict(base), - } - - audit_actions = { - "pre_llm_call": "memory_search", - "post_llm_call": "append_episode", - "on_session_end": "commit_session", - } - reports = {name: _hook_report(manager, name, payload, audit_actions.get(name, "")) for name, payload in hooks.items()} - plugin = manager._plugins.get("memory-gateway-agent") - return { - "ok": all(item["registered"] and item["invoked"] for item in reports.values()), - "auto_commit": auto_commit, - "ensure_user": {"ok": ensure_user.get("ok"), "status_code": ensure_user.get("status_code"), "data": summarize_data(ensure_user.get("data"))}, - "plugin": { - "enabled": bool(plugin and plugin.enabled), - "tools_registered": sorted(getattr(plugin, "tools_registered", []) if plugin else []), - "hooks_registered": sorted(getattr(plugin, "hooks_registered", []) if plugin else []), - "error": getattr(plugin, "error", None) if plugin else "plugin_not_found", - }, - "hooks": reports, - } - - -def main() -> int: - auto_commit = os.environ.get("MEMORY_GATEWAY_AUTO_COMMIT_SESSION", "").strip().lower() in {"1", "true", "yes", "on"} - result = run(auto_commit=auto_commit) - print(dumps_safe(result)) - return 0 if result.get("ok") else 1 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/plugins/memory-gateway-agent/scripts/hermes_interactive_session_check.py b/plugins/memory-gateway-agent/scripts/hermes_interactive_session_check.py deleted file mode 100644 index 6153112..0000000 --- a/plugins/memory-gateway-agent/scripts/hermes_interactive_session_check.py +++ /dev/null @@ -1,147 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import json -import os -import shutil -import subprocess -import sys -import urllib.error -import urllib.request -from pathlib import Path -from typing import Any - -PLUGIN_ROOT = Path(__file__).resolve().parents[1] -if str(PLUGIN_ROOT) not in sys.path: - sys.path.insert(0, str(PLUGIN_ROOT)) - -from memory_gateway_plugin.output import dumps_safe, short_id, summarize_data - - -USER_ID = "test_user_memory_gateway_plugin" -AGENT_ID = "test_hermes_memory_gateway_plugin" -WORKSPACE_ID = "test_workspace_memory_gateway_plugin" -SESSION_ID = "test_session_memory_gateway_plugin_interactive_002" -PROMPT = "Please remember this integration test preference: Memory Gateway plugin should store summarized episodes, not raw transcripts." - - -def _request(method: str, url: str, payload: dict[str, Any] | None = None, api_key: str = "") -> dict[str, Any]: - headers = {"Content-Type": "application/json"} - if api_key: - headers["X-API-Key"] = api_key - body = None if payload is None else json.dumps(payload, ensure_ascii=False).encode("utf-8") - req = urllib.request.Request(url, data=body, headers=headers, method=method) - try: - with urllib.request.urlopen(req, timeout=10) as response: - raw = response.read().decode("utf-8") - return {"ok": True, "status_code": getattr(response, "status", 200), "data": json.loads(raw) if raw else {}} - except urllib.error.HTTPError as exc: - return {"ok": False, "status_code": exc.code, "error": str(exc.reason)[:300]} - except Exception as exc: - return {"ok": False, "status_code": None, "error": str(exc)[:300]} - - -def _gateway_url() -> str: - return os.environ.get("MEMORY_GATEWAY_URL", "http://127.0.0.1:1934").rstrip("/") - - -def _api_key() -> str: - return os.environ.get("MEMORY_GATEWAY_API_KEY", "") - - -def _audit_counts() -> dict[str, int]: - result = _request("GET", _gateway_url() + "/v1/audit?limit=1000", api_key=_api_key()) - rows = result.get("data") or [] - actions = {"memory_search": 0, "append_episode": 0, "commit_session": 0} - for row in rows: - if row.get("actor_user_id") != USER_ID or row.get("actor_agent_id") != AGENT_ID: - continue - action = row.get("action") - if action in actions: - actions[action] += 1 - return actions - - -def _run_cmd(args: list[str], timeout: int = 20, env: dict[str, str] | None = None) -> dict[str, Any]: - try: - completed = subprocess.run(args, capture_output=True, text=True, timeout=timeout, env=env, check=False) - return {"ok": completed.returncode == 0, "returncode": completed.returncode, "stdout_chars": len(completed.stdout), "stderr_chars": len(completed.stderr)} - except FileNotFoundError: - return {"ok": False, "returncode": None, "error": "command_not_found"} - except subprocess.TimeoutExpired: - return {"ok": False, "returncode": None, "error": "timeout"} - - -def _manual_instructions(reason: str) -> dict[str, Any]: - return { - "mode": "manual", - "reason": reason, - "commands": [ - "hermes plugins list", - "hermes tools list", - "MEMORY_GATEWAY_URL=http://127.0.0.1:1934 MEMORY_GATEWAY_DEFAULT_USER_ID=test_user_memory_gateway_plugin MEMORY_GATEWAY_DEFAULT_AGENT_ID=test_hermes_memory_gateway_plugin MEMORY_GATEWAY_DEFAULT_WORKSPACE_ID=test_workspace_memory_gateway_plugin MEMORY_GATEWAY_AUTO_COMMIT_SESSION=false hermes chat -Q -q 'Please remember this integration test preference: Memory Gateway plugin should store summarized episodes, not raw transcripts.' --source memory-gateway-plugin-test --toolsets memory_gateway", - "python plugins/memory-gateway-agent/scripts/hermes_interactive_session_check.py", - ], - "expected": [ - "Gateway audit memory_search count increases for the test user/agent.", - "Gateway audit append_episode count increases for the test user/agent.", - "commit_session count does not increase while MEMORY_GATEWAY_AUTO_COMMIT_SESSION=false.", - ], - } - - -def run() -> dict[str, Any]: - hermes = shutil.which("hermes") or "/home/tom/.local/bin/hermes" - plugin_list = _run_cmd([hermes, "plugins", "list"], timeout=10) - tools_list = _run_cmd([hermes, "tools", "list"], timeout=10) - health = _request("GET", _gateway_url() + "/health", api_key=_api_key()) - if not health.get("ok"): - return {"ok": False, "mode": "blocked", "plugin_list": plugin_list, "tools_list": tools_list, "gateway_health": {"ok": False, "status_code": health.get("status_code"), "error": health.get("error")}, "manual": _manual_instructions("gateway_unhealthy")} - - before = _audit_counts() - env = os.environ.copy() - env.update( - { - "MEMORY_GATEWAY_URL": _gateway_url(), - "MEMORY_GATEWAY_DEFAULT_USER_ID": USER_ID, - "MEMORY_GATEWAY_DEFAULT_AGENT_ID": AGENT_ID, - "MEMORY_GATEWAY_DEFAULT_WORKSPACE_ID": WORKSPACE_ID, - "MEMORY_GATEWAY_AUTO_SEARCH": "true", - "MEMORY_GATEWAY_AUTO_APPEND_EPISODE": "true", - "MEMORY_GATEWAY_AUTO_COMMIT_SESSION": os.environ.get("MEMORY_GATEWAY_AUTO_COMMIT_SESSION", "false"), - } - ) - chat = _run_cmd( - [hermes, "chat", "-Q", "-q", PROMPT, "--source", "memory-gateway-plugin-test", "--toolsets", "memory_gateway"], - timeout=int(os.environ.get("MEMORY_GATEWAY_PLUGIN_CHAT_TIMEOUT", "180")), - env=env, - ) - after = _audit_counts() - delta = {key: after.get(key, 0) - before.get(key, 0) for key in before} - auto_commit = env["MEMORY_GATEWAY_AUTO_COMMIT_SESSION"].strip().lower() in {"1", "true", "yes", "on"} - expected_commit = delta.get("commit_session", 0) > 0 if auto_commit else delta.get("commit_session", 0) == 0 - passed = chat.get("ok") and delta.get("memory_search", 0) > 0 and delta.get("append_episode", 0) > 0 and expected_commit - return { - "ok": bool(passed), - "mode": "auto" if chat.get("ok") else "manual", - "plugin_list": plugin_list, - "tools_list": tools_list, - "gateway_health": {"ok": True, "status_code": health.get("status_code"), "data": summarize_data(health.get("data"))}, - "chat": chat, - "auto_commit": auto_commit, - "audit_before": before, - "audit_after": after, - "audit_delta": delta, - "test_identity": {"user_id": USER_ID, "agent_id": AGENT_ID, "workspace_id": WORKSPACE_ID, "session_id": short_id(SESSION_ID)}, - "manual": None if chat.get("ok") else _manual_instructions(chat.get("error", "chat_command_failed")), - } - - -def main() -> int: - result = run() - print(dumps_safe(result)) - return 0 if result.get("ok") else 1 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/plugins/memory-gateway-agent/scripts/hermes_smoke_check.py b/plugins/memory-gateway-agent/scripts/hermes_smoke_check.py deleted file mode 100644 index 08189bb..0000000 --- a/plugins/memory-gateway-agent/scripts/hermes_smoke_check.py +++ /dev/null @@ -1,73 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import importlib.util -import json -import sys -import types -from pathlib import Path - - -class FakeHermesContext: - def __init__(self) -> None: - self.registered_tools = [] - self.registered_hooks = [] - - def register_tool(self, name, toolset, schema, handler, **kwargs): - self.registered_tools.append( - { - "name": name, - "toolset": toolset, - "schema": schema, - "handler_callable": callable(handler), - "kwargs": kwargs, - } - ) - - def register_hook(self, hook_name, callback): - self.registered_hooks.append({"name": hook_name, "handler_callable": callable(callback)}) - - -def load_plugin_module(): - plugin_dir = Path(__file__).resolve().parents[1] - init_file = plugin_dir / "__init__.py" - module_name = "hermes_plugins.memory_gateway_agent_smoke" - if "hermes_plugins" not in sys.modules: - parent = types.ModuleType("hermes_plugins") - parent.__path__ = [] - sys.modules["hermes_plugins"] = parent - spec = importlib.util.spec_from_file_location( - module_name, - init_file, - submodule_search_locations=[str(plugin_dir)], - ) - if spec is None or spec.loader is None: - raise RuntimeError("Cannot create plugin module spec") - module = importlib.util.module_from_spec(spec) - module.__package__ = module_name - module.__path__ = [str(plugin_dir)] - sys.modules[module_name] = module - spec.loader.exec_module(module) - return module - - -def main() -> None: - module = load_plugin_module() - ctx = FakeHermesContext() - module.register(ctx) - print( - json.dumps( - { - "ok": True, - "has_register": callable(getattr(module, "register", None)), - "registered_tools": ctx.registered_tools, - "registered_hooks": ctx.registered_hooks, - }, - ensure_ascii=False, - indent=2, - ) - ) - - -if __name__ == "__main__": - main() diff --git a/plugins/memory-gateway-agent/scripts/smoke_test.py b/plugins/memory-gateway-agent/scripts/smoke_test.py deleted file mode 100644 index 3d64a10..0000000 --- a/plugins/memory-gateway-agent/scripts/smoke_test.py +++ /dev/null @@ -1,34 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import json -import sys -import uuid -from pathlib import Path - -PLUGIN_ROOT = Path(__file__).resolve().parents[1] -if str(PLUGIN_ROOT) not in sys.path: - sys.path.insert(0, str(PLUGIN_ROOT)) - -from memory_gateway_plugin.output import dumps_safe, summarize_result -from memory_gateway_plugin.tools import memory_append_episode, memory_commit_session, memory_search - - -def main() -> None: - user_id = "plugin_smoke_user" - agent_id = "plugin_smoke_agent" - session_id = f"plugin_smoke_{uuid.uuid4().hex[:8]}" - episode = memory_append_episode( - user_id=user_id, - agent_id=agent_id, - session_id=session_id, - episode_summary="结论：Memory Gateway Agent Plugin smoke test 写入短期 episode。", - tags=["smoke-test"], - ) - commit = memory_commit_session(user_id=user_id, agent_id=agent_id, session_id=session_id) - search = memory_search(query="Memory Gateway Agent Plugin smoke test", user_id=user_id, agent_id=agent_id, session_id=session_id) - print(dumps_safe({"episode": summarize_result(episode), "commit": summarize_result(commit), "search": summarize_result(search)})) - - -if __name__ == "__main__": - main() diff --git a/plugins/memory-gateway-agent/tests/test_cleanup_requires_test_user.py b/plugins/memory-gateway-agent/tests/test_cleanup_requires_test_user.py deleted file mode 100644 index 56b8000..0000000 --- a/plugins/memory-gateway-agent/tests/test_cleanup_requires_test_user.py +++ /dev/null @@ -1,26 +0,0 @@ -from __future__ import annotations - -import importlib.util -import sys -from pathlib import Path - - -def _load_cleanup(): - path = Path(__file__).resolve().parents[1] / "scripts" / "cleanup_test_memories.py" - spec = importlib.util.spec_from_file_location("cleanup_test_memories_guard_test", path) - assert spec and spec.loader - module = importlib.util.module_from_spec(spec) - sys.modules[spec.name] = module - spec.loader.exec_module(module) - return module - - -def test_cleanup_requires_test_user(): - module = _load_cleanup() - - try: - module.run("real_user") - except ValueError as exc: - assert "cleanup_refuses_non_test_user" in str(exc) - else: - raise AssertionError("cleanup accepted a non-test user") diff --git a/plugins/memory-gateway-agent/tests/test_cleanup_test_memories_imports.py b/plugins/memory-gateway-agent/tests/test_cleanup_test_memories_imports.py deleted file mode 100644 index c0984a9..0000000 --- a/plugins/memory-gateway-agent/tests/test_cleanup_test_memories_imports.py +++ /dev/null @@ -1,17 +0,0 @@ -from __future__ import annotations - -import importlib.util -import sys -from pathlib import Path - - -def test_cleanup_test_memories_imports(): - path = Path(__file__).resolve().parents[1] / "scripts" / "cleanup_test_memories.py" - spec = importlib.util.spec_from_file_location("cleanup_test_memories", path) - assert spec and spec.loader - module = importlib.util.module_from_spec(spec) - sys.modules[spec.name] = module - spec.loader.exec_module(module) - - assert callable(module.run) - assert module.USER_ID.startswith("test_user_") diff --git a/plugins/memory-gateway-agent/tests/test_client.py b/plugins/memory-gateway-agent/tests/test_client.py deleted file mode 100644 index db41c13..0000000 --- a/plugins/memory-gateway-agent/tests/test_client.py +++ /dev/null @@ -1,105 +0,0 @@ -from __future__ import annotations - -import io -import json -import urllib.error - -from memory_gateway_plugin.client import MemoryGatewayClient -from memory_gateway_plugin.config import PluginConfig - - -class FakeResponse: - status = 200 - - def __init__(self, payload): - self.payload = payload - - def __enter__(self): - return self - - def __exit__(self, *args): - return False - - def read(self): - return json.dumps(self.payload).encode("utf-8") - - -def test_client_search_success(monkeypatch): - seen = {} - - def fake_urlopen(request, timeout): - seen["url"] = request.full_url - seen["timeout"] = timeout - return FakeResponse({"results": [], "total": 0}) - - monkeypatch.setattr("urllib.request.urlopen", fake_urlopen) - client = MemoryGatewayClient(PluginConfig(gateway_url="http://gateway", timeout=7)) - result = client.search_memory({"user_id": "u", "query": "demo"}) - - assert result["ok"] is True - assert result["endpoint"] == "/v1/memory/search" - assert seen["url"] == "http://gateway/v1/memory/search" - assert seen["timeout"] == 7 - - -def test_client_network_error(monkeypatch): - def fake_urlopen(request, timeout): - raise urllib.error.URLError("connection refused") - - monkeypatch.setattr("urllib.request.urlopen", fake_urlopen) - client = MemoryGatewayClient(PluginConfig(gateway_url="http://gateway")) - result = client.search_memory({"user_id": "u", "query": "demo"}) - - assert result["ok"] is False - assert result["status_code"] is None - assert "connection refused" in result["error"] - - -def test_commit_session_calls_correct_endpoint(monkeypatch): - seen = {} - - def fake_urlopen(request, timeout): - seen["url"] = request.full_url - return FakeResponse({"session_id": "sess_1"}) - - monkeypatch.setattr("urllib.request.urlopen", fake_urlopen) - client = MemoryGatewayClient(PluginConfig(gateway_url="http://gateway")) - result = client.commit_session("sess_1", {"user_id": "u", "session_id": "sess_1"}) - - assert result["ok"] is True - assert seen["url"] == "http://gateway/v1/sessions/sess_1/commit" - - -def test_feedback_calls_correct_endpoint(monkeypatch): - seen = {} - - def fake_urlopen(request, timeout): - seen["url"] = request.full_url - return FakeResponse({"status": "ok"}) - - monkeypatch.setattr("urllib.request.urlopen", fake_urlopen) - client = MemoryGatewayClient(PluginConfig(gateway_url="http://gateway")) - result = client.send_feedback("mem_1", {"user_id": "u", "feedback": "incorrect"}) - - assert result["ok"] is True - assert seen["url"] == "http://gateway/v1/memory/mem_1/feedback" - - -def test_client_http_error(monkeypatch): - def fake_urlopen(request, timeout): - raise urllib.error.HTTPError( - url=request.full_url, - code=401, - msg="unauthorized", - hdrs=None, - fp=io.BytesIO(b'{"detail":"Invalid or missing API key"}'), - ) - - monkeypatch.setattr("urllib.request.urlopen", fake_urlopen) - client = MemoryGatewayClient(PluginConfig(gateway_url="http://gateway")) - result = client.search_memory({"user_id": "u", "query": "demo"}) - - assert result["ok"] is False - assert result["status_code"] == 401 - assert result["endpoint"] == "/v1/memory/search" - diff --git a/plugins/memory-gateway-agent/tests/test_debug_raw_disabled_by_default.py b/plugins/memory-gateway-agent/tests/test_debug_raw_disabled_by_default.py deleted file mode 100644 index ff4a186..0000000 --- a/plugins/memory-gateway-agent/tests/test_debug_raw_disabled_by_default.py +++ /dev/null @@ -1,16 +0,0 @@ -from __future__ import annotations - -from memory_gateway_plugin.output import debug_raw_enabled, summarize_data - - -def test_debug_raw_disabled_by_default(monkeypatch): - monkeypatch.delenv("MEMORY_GATEWAY_PLUGIN_DEBUG_RAW", raising=False) - - assert debug_raw_enabled() is False - assert summarize_data({"results": [{"memory": {"content": "raw"}}], "total": 1}) == { - "count": 1, - "total": 1, - "local_total": None, - "openviking_total": None, - "searched_namespaces": [], - } diff --git a/plugins/memory-gateway-agent/tests/test_gateway_e2e_script_imports.py b/plugins/memory-gateway-agent/tests/test_gateway_e2e_script_imports.py deleted file mode 100644 index fbd1e83..0000000 --- a/plugins/memory-gateway-agent/tests/test_gateway_e2e_script_imports.py +++ /dev/null @@ -1,17 +0,0 @@ -from __future__ import annotations - -import importlib.util -import sys -from pathlib import Path - - -def test_gateway_e2e_script_imports(): - path = Path(__file__).resolve().parents[1] / "scripts" / "gateway_e2e_check.py" - spec = importlib.util.spec_from_file_location("gateway_e2e_check", path) - assert spec and spec.loader - module = importlib.util.module_from_spec(spec) - sys.modules[spec.name] = module - spec.loader.exec_module(module) - - assert callable(module.run) - assert module.USER_ID == "test_user_memory_gateway_plugin" diff --git a/plugins/memory-gateway-agent/tests/test_hermes_hook_probe_imports.py b/plugins/memory-gateway-agent/tests/test_hermes_hook_probe_imports.py deleted file mode 100644 index 838083e..0000000 --- a/plugins/memory-gateway-agent/tests/test_hermes_hook_probe_imports.py +++ /dev/null @@ -1,15 +0,0 @@ -from __future__ import annotations - -import importlib.util -from pathlib import Path - - -def test_hermes_hook_probe_script_imports(): - path = Path(__file__).resolve().parents[1] / "scripts" / "hermes_hook_probe.py" - spec = importlib.util.spec_from_file_location("hermes_hook_probe", path) - assert spec and spec.loader - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - - assert callable(module.run) - assert module.SESSION_ID == "test_session_memory_gateway_plugin_001" diff --git a/plugins/memory-gateway-agent/tests/test_hermes_register_hooks.py b/plugins/memory-gateway-agent/tests/test_hermes_register_hooks.py deleted file mode 100644 index 4660140..0000000 --- a/plugins/memory-gateway-agent/tests/test_hermes_register_hooks.py +++ /dev/null @@ -1,28 +0,0 @@ -from __future__ import annotations - -from test_hermes_register_tools import FakeHermesContext, load_plugin_module - - -def test_register_registers_expected_hooks(): - module = load_plugin_module() - ctx = FakeHermesContext() - - module.register(ctx) - - assert [item[0] for item in ctx.registered_hooks] == [ - "on_session_start", - "pre_llm_call", - "post_llm_call", - "on_session_end", - ] - assert all(callable(item[1]) for item in ctx.registered_hooks) - - -def test_hook_callbacks_accept_kwargs(): - module = load_plugin_module() - - assert isinstance(module.on_session_start(session_id="s", extra="x"), dict) - assert isinstance(module.pre_llm_call(user_message="", session_id="s", extra="x"), dict) - assert module.post_llm_call(user_message="hi", assistant_response="hello", extra="x") is None - assert module.on_session_end(session_id="s", extra="x") is None - diff --git a/plugins/memory-gateway-agent/tests/test_hermes_register_tools.py b/plugins/memory-gateway-agent/tests/test_hermes_register_tools.py deleted file mode 100644 index 42e9515..0000000 --- a/plugins/memory-gateway-agent/tests/test_hermes_register_tools.py +++ /dev/null @@ -1,63 +0,0 @@ -from __future__ import annotations - -import importlib.util -import sys -import types -from pathlib import Path - - -class FakeHermesContext: - def __init__(self) -> None: - self.registered_tools = [] - self.registered_hooks = [] - - def register_tool(self, name, toolset, schema, handler, **kwargs): - self.registered_tools.append((name, toolset, schema, handler, kwargs)) - - def register_hook(self, hook_name, callback): - self.registered_hooks.append((hook_name, callback)) - - -def load_plugin_module(): - plugin_dir = Path(__file__).resolve().parents[1] - if "hermes_plugins" not in sys.modules: - parent = types.ModuleType("hermes_plugins") - parent.__path__ = [] - sys.modules["hermes_plugins"] = parent - spec = importlib.util.spec_from_file_location( - "hermes_plugins.memory_gateway_agent_test", - plugin_dir / "__init__.py", - submodule_search_locations=[str(plugin_dir)], - ) - module = importlib.util.module_from_spec(spec) - module.__package__ = "hermes_plugins.memory_gateway_agent_test" - module.__path__ = [str(plugin_dir)] - sys.modules["hermes_plugins.memory_gateway_agent_test"] = module - assert spec.loader is not None - spec.loader.exec_module(module) - return module - - -def test_register_registers_five_tools(): - module = load_plugin_module() - ctx = FakeHermesContext() - - module.register(ctx) - - assert [item[0] for item in ctx.registered_tools] == [ - "memory_search", - "memory_append_episode", - "memory_commit_session", - "memory_upsert", - "memory_feedback", - ] - assert all(item[1] == "memory_gateway" for item in ctx.registered_tools) - - -def test_registered_handlers_are_callable(): - module = load_plugin_module() - ctx = FakeHermesContext() - - module.register(ctx) - - assert all(callable(item[3]) for item in ctx.registered_tools) diff --git a/plugins/memory-gateway-agent/tests/test_hermes_schemas.py b/plugins/memory-gateway-agent/tests/test_hermes_schemas.py deleted file mode 100644 index d99c5e0..0000000 --- a/plugins/memory-gateway-agent/tests/test_hermes_schemas.py +++ /dev/null @@ -1,36 +0,0 @@ -from __future__ import annotations - -from test_hermes_register_tools import load_plugin_module - - -def test_tool_schemas_exist_for_all_tools(): - module = load_plugin_module() - schemas = module.schemas.TOOL_SCHEMAS - - assert set(schemas) == { - "memory_search", - "memory_append_episode", - "memory_commit_session", - "memory_upsert", - "memory_feedback", - } - - -def test_tool_schemas_have_required_fields(): - module = load_plugin_module() - schemas = module.schemas.TOOL_SCHEMAS - - assert schemas["memory_search"]["parameters"]["required"] == ["query", "user_id", "agent_id"] - assert schemas["memory_append_episode"]["parameters"]["required"] == ["content", "user_id", "agent_id", "session_id"] - assert schemas["memory_commit_session"]["parameters"]["required"] == ["user_id", "agent_id", "session_id"] - assert schemas["memory_upsert"]["parameters"]["required"] == ["user_id", "agent_id", "content", "memory_type"] - assert schemas["memory_feedback"]["parameters"]["required"] == ["memory_id", "user_id", "agent_id", "feedback"] - - -def test_upsert_schema_warns_high_risk(): - module = load_plugin_module() - - description = module.schemas.TOOL_SCHEMAS["memory_upsert"]["description"].lower() - assert "high-risk" in description - assert "do not call automatically" in description - diff --git a/plugins/memory-gateway-agent/tests/test_hook_auto_commit_disabled_by_default.py b/plugins/memory-gateway-agent/tests/test_hook_auto_commit_disabled_by_default.py deleted file mode 100644 index 0af207e..0000000 --- a/plugins/memory-gateway-agent/tests/test_hook_auto_commit_disabled_by_default.py +++ /dev/null @@ -1,28 +0,0 @@ -from __future__ import annotations - -from memory_gateway_plugin.config import PluginConfig -from memory_gateway_plugin.lifecycle import on_session_end - - -class CountingClient: - def __init__(self) -> None: - self.commit_calls = 0 - - def commit_session(self, session_id, payload): - self.commit_calls += 1 - return {"ok": True, "data": {"session_id": session_id, "payload": payload}} - - -def test_hook_auto_commit_disabled_by_default(): - client = CountingClient() - - result = on_session_end( - {"user_id": "u", "agent_id": "a", "session_id": "s"}, - client=client, - config=PluginConfig(auto_commit_session=False), - ) - - assert result["ok"] is True - assert result["committed"] is False - assert result["reason"] == "auto_commit_disabled" - assert client.commit_calls == 0 diff --git a/plugins/memory-gateway-agent/tests/test_hook_post_llm_does_not_save_raw_transcript.py b/plugins/memory-gateway-agent/tests/test_hook_post_llm_does_not_save_raw_transcript.py deleted file mode 100644 index 0b91f5a..0000000 --- a/plugins/memory-gateway-agent/tests/test_hook_post_llm_does_not_save_raw_transcript.py +++ /dev/null @@ -1,35 +0,0 @@ -from __future__ import annotations - -from memory_gateway_plugin.config import PluginConfig -from memory_gateway_plugin.lifecycle import after_user_message - - -class RecordingClient: - def __init__(self) -> None: - self.payloads = [] - - def append_episode(self, payload): - self.payloads.append(payload) - return {"ok": True, "data": payload} - - -def test_hook_post_llm_does_not_save_raw_transcript(): - client = RecordingClient() - raw_transcript = "user: a\nassistant: b\nuser: c\nassistant: d" - - result = after_user_message( - { - "user_id": "u", - "agent_id": "a", - "session_id": "s", - "user_message": raw_transcript, - "assistant_response": "请记住这个完整原始对话。", - }, - client=client, - config=PluginConfig(auto_append_episode=True), - ) - - assert result["ok"] is True - assert result["appended"] is False - assert result["reason"] == "policy_skip" - assert client.payloads == [] diff --git a/plugins/memory-gateway-agent/tests/test_hook_pre_llm_search_failure_non_blocking.py b/plugins/memory-gateway-agent/tests/test_hook_pre_llm_search_failure_non_blocking.py deleted file mode 100644 index d76228d..0000000 --- a/plugins/memory-gateway-agent/tests/test_hook_pre_llm_search_failure_non_blocking.py +++ /dev/null @@ -1,40 +0,0 @@ -from __future__ import annotations - -import importlib.util -import sys -import types -from pathlib import Path - - -def _load_root_plugin(): - plugin_dir = Path(__file__).resolve().parents[1] - if "hermes_plugins" not in sys.modules: - parent = types.ModuleType("hermes_plugins") - parent.__path__ = [] # type: ignore[attr-defined] - sys.modules["hermes_plugins"] = parent - module_name = "hermes_plugins.memory_gateway_agent_pre_llm_test" - spec = importlib.util.spec_from_file_location( - module_name, - plugin_dir / "__init__.py", - submodule_search_locations=[str(plugin_dir)], - ) - assert spec and spec.loader - module = importlib.util.module_from_spec(spec) - module.__package__ = module_name - module.__path__ = [str(plugin_dir)] # type: ignore[attr-defined] - sys.modules[module_name] = module - spec.loader.exec_module(module) - return module - - -def test_hook_pre_llm_search_failure_non_blocking(monkeypatch): - plugin = _load_root_plugin() - - class FailingLifecycle: - @staticmethod - def on_conversation_start(context): - return {"ok": False, "error": "network_error"} - - monkeypatch.setattr(plugin, "lifecycle", FailingLifecycle) - - assert plugin.pre_llm_call(user_id="u", agent_id="a", session_id="s", user_message="search") == {} diff --git a/plugins/memory-gateway-agent/tests/test_hook_trace_disabled_by_default.py b/plugins/memory-gateway-agent/tests/test_hook_trace_disabled_by_default.py deleted file mode 100644 index 47221c8..0000000 --- a/plugins/memory-gateway-agent/tests/test_hook_trace_disabled_by_default.py +++ /dev/null @@ -1,9 +0,0 @@ -from __future__ import annotations - -from memory_gateway_plugin.trace import trace_enabled - - -def test_hook_trace_disabled_by_default(monkeypatch): - monkeypatch.delenv("MEMORY_GATEWAY_PLUGIN_TRACE_HOOKS", raising=False) - - assert trace_enabled() is False diff --git a/plugins/memory-gateway-agent/tests/test_hook_trace_does_not_log_api_key.py b/plugins/memory-gateway-agent/tests/test_hook_trace_does_not_log_api_key.py deleted file mode 100644 index 1f1471f..0000000 --- a/plugins/memory-gateway-agent/tests/test_hook_trace_does_not_log_api_key.py +++ /dev/null @@ -1,18 +0,0 @@ -from __future__ import annotations - -from memory_gateway_plugin.trace import trace_hook - - -def test_hook_trace_does_not_log_api_key(monkeypatch, tmp_path): - import memory_gateway_plugin.trace as trace_mod - - path = tmp_path / "hook_trace.log" - monkeypatch.setenv("MEMORY_GATEWAY_PLUGIN_TRACE_HOOKS", "true") - monkeypatch.setenv("MEMORY_GATEWAY_API_KEY", "sk-should-not-appear") - monkeypatch.setattr(trace_mod, "trace_path", lambda: path) - - trace_hook("post_llm_call", session_id="s", gateway_action="append_episode", gateway_called=True, ok=True) - - text = path.read_text(encoding="utf-8") - assert "sk-should-not-appear" not in text - assert "api_key" not in text.lower() diff --git a/plugins/memory-gateway-agent/tests/test_hook_trace_redacts_content.py b/plugins/memory-gateway-agent/tests/test_hook_trace_redacts_content.py deleted file mode 100644 index bfd170c..0000000 --- a/plugins/memory-gateway-agent/tests/test_hook_trace_redacts_content.py +++ /dev/null @@ -1,18 +0,0 @@ -from __future__ import annotations - -from memory_gateway_plugin.trace import trace_hook - - -def test_hook_trace_redacts_content(monkeypatch, tmp_path): - import memory_gateway_plugin.trace as trace_mod - - path = tmp_path / "hook_trace.log" - monkeypatch.setenv("MEMORY_GATEWAY_PLUGIN_TRACE_HOOKS", "true") - monkeypatch.setattr(trace_mod, "trace_path", lambda: path) - - trace_hook("pre_llm_call", session_id="test_session_1234567890", gateway_action="memory_search", gateway_called=True, ok=True, reason="password=abc") - - text = path.read_text(encoding="utf-8") - assert "password=abc" not in text - assert "pre_llm_call" in text - assert "test_ses" in text diff --git a/plugins/memory-gateway-agent/tests/test_interactive_session_check_imports.py b/plugins/memory-gateway-agent/tests/test_interactive_session_check_imports.py deleted file mode 100644 index bf14814..0000000 --- a/plugins/memory-gateway-agent/tests/test_interactive_session_check_imports.py +++ /dev/null @@ -1,17 +0,0 @@ -from __future__ import annotations - -import importlib.util -import sys -from pathlib import Path - - -def test_interactive_session_check_imports(): - path = Path(__file__).resolve().parents[1] / "scripts" / "hermes_interactive_session_check.py" - spec = importlib.util.spec_from_file_location("hermes_interactive_session_check", path) - assert spec and spec.loader - module = importlib.util.module_from_spec(spec) - sys.modules[spec.name] = module - spec.loader.exec_module(module) - - assert callable(module.run) - assert module.SESSION_ID == "test_session_memory_gateway_plugin_interactive_002" diff --git a/plugins/memory-gateway-agent/tests/test_lifecycle.py b/plugins/memory-gateway-agent/tests/test_lifecycle.py deleted file mode 100644 index 98ddae7..0000000 --- a/plugins/memory-gateway-agent/tests/test_lifecycle.py +++ /dev/null @@ -1,70 +0,0 @@ -from __future__ import annotations - -from memory_gateway_plugin import register -from memory_gateway_plugin.config import PluginConfig -from memory_gateway_plugin.lifecycle import after_user_message, on_conversation_start, on_session_end - - -class FakeClient: - def search_memory(self, payload): - return { - "ok": True, - "data": { - "results": [ - { - "memory": { - "id": "mem_1", - "namespace": "user/u/long_term", - "summary": "用户偏好中文输出。", - } - } - ] - }, - } - - def append_episode(self, payload): - return {"ok": True, "data": payload} - - def commit_session(self, session_id, payload): - return {"ok": True, "data": {"session_id": session_id}} - - -def test_lifecycle_hooks_do_not_crash_when_ctx_missing_features(): - result = register(object()) - - assert result["ok"] is True - assert result["mode"] == "manual" - - -def test_lifecycle_search_returns_compact_context(): - result = on_conversation_start( - {"user_id": "u", "agent_id": "a", "session_id": "s", "user_message": "之前偏好是什么？"}, - client=FakeClient(), - config=PluginConfig(auto_search=True), - ) - - assert result["ok"] is True - assert "用户偏好中文输出" in result["memory_context"] - - -def test_lifecycle_append_policy_accepts_stable_preference(): - result = after_user_message( - {"user_id": "u", "agent_id": "a", "session_id": "s", "user_message": "请记住：我偏好中文。"}, - client=FakeClient(), - config=PluginConfig(auto_append_episode=True), - ) - - assert result["ok"] is True - assert result["appended"] is True - - -def test_lifecycle_session_end_auto_commit_disabled(): - result = on_session_end( - {"user_id": "u", "agent_id": "a", "session_id": "s"}, - client=FakeClient(), - config=PluginConfig(auto_commit_session=False), - ) - - assert result["ok"] is True - assert result["committed"] is False - diff --git a/plugins/memory-gateway-agent/tests/test_output_redaction.py b/plugins/memory-gateway-agent/tests/test_output_redaction.py deleted file mode 100644 index bfec896..0000000 --- a/plugins/memory-gateway-agent/tests/test_output_redaction.py +++ /dev/null @@ -1,24 +0,0 @@ -from __future__ import annotations - -from memory_gateway_plugin.output import dumps_safe, redact, short_id - - -def test_output_redaction_hides_secret_fields(): - payload = { - "api_key": "sk-test", - "headers": {"Authorization": "Bearer abc"}, - "nested": {"cookie": "sid=abc"}, - "safe": "value", - } - - text = dumps_safe(payload) - - assert "sk-test" not in text - assert "Bearer abc" not in text - assert "sid=abc" not in text - assert "value" in text - assert redact("password=abc") == "" - - -def test_output_redaction_shortens_memory_ids(): - assert short_id("mem_1234567890abcdef") == "mem_1234...cdef" diff --git a/plugins/memory-gateway-agent/tests/test_policy.py b/plugins/memory-gateway-agent/tests/test_policy.py deleted file mode 100644 index 7c916d7..0000000 --- a/plugins/memory-gateway-agent/tests/test_policy.py +++ /dev/null @@ -1,22 +0,0 @@ -from __future__ import annotations - -from memory_gateway_plugin.config import PluginConfig -from memory_gateway_plugin.policy import should_append_episode, should_commit_session, should_search_memory - - -def test_policy_should_append_for_explicit_remember(): - assert should_append_episode("请记住：我偏好中文技术说明。", "", {}, PluginConfig()) - - -def test_policy_should_not_append_for_small_talk(): - assert not should_append_episode("你好", "", {}, PluginConfig()) - - -def test_policy_should_search_when_enabled(): - assert should_search_memory("这个项目之前有什么约束？", {}, PluginConfig(auto_search=True)) - - -def test_policy_should_commit_only_when_enabled_or_forced(): - assert not should_commit_session({}, PluginConfig(auto_commit_session=False)) - assert should_commit_session({"force_commit": True}, PluginConfig(auto_commit_session=False)) - diff --git a/plugins/memory-gateway-agent/tests/test_safety.py b/plugins/memory-gateway-agent/tests/test_safety.py deleted file mode 100644 index 725e100..0000000 --- a/plugins/memory-gateway-agent/tests/test_safety.py +++ /dev/null @@ -1,24 +0,0 @@ -from __future__ import annotations - -from memory_gateway_plugin.safety import detect_large_log, sanitize_memory_content, validate_memory_write - - -def test_safety_rejects_private_key(): - content = "-----BEGIN PRIVATE KEY-----\nabc\n-----END PRIVATE KEY-----" - result = validate_memory_write(content) - - assert result["allowed"] is False - assert result["reason"] == "secret_like_content" - - -def test_safety_rejects_large_log(): - content = "\n".join(f"2026-05-06 10:00:{i:02d} ERROR failure" for i in range(10)) - blocked, reason = detect_large_log(content) - - assert blocked is True - assert reason == "large_or_raw_log" - - -def test_safety_sanitizes_secret_when_called_directly(): - assert "sk-test" not in sanitize_memory_content("api_key=sk-test") - diff --git a/plugins/memory-gateway-agent/tests/test_tools.py b/plugins/memory-gateway-agent/tests/test_tools.py deleted file mode 100644 index 1c7965a..0000000 --- a/plugins/memory-gateway-agent/tests/test_tools.py +++ /dev/null @@ -1,106 +0,0 @@ -from __future__ import annotations - -from memory_gateway_plugin.tools import memory_append_episode, memory_feedback, memory_search, memory_upsert - - -class FakeClient: - def __init__(self): - self.calls = [] - - def search_memory(self, payload): - self.calls.append(("search", payload)) - return {"ok": True, "data": {"results": []}} - - def append_episode(self, payload): - self.calls.append(("append", payload)) - return {"ok": True, "data": payload} - - def upsert_memory(self, payload): - self.calls.append(("upsert", payload)) - return {"ok": True, "data": payload} - - def send_feedback(self, memory_id, payload): - self.calls.append(("feedback", memory_id, payload)) - return {"ok": True, "data": payload} - - -def test_memory_search_empty_query_rejected(): - client = FakeClient() - result = memory_search(query="", user_id="u", agent_id="a", client=client) - - assert result["ok"] is False - assert client.calls == [] - - -def test_append_episode_rejects_api_key(): - result = memory_append_episode( - user_id="u", - agent_id="a", - session_id="s", - episode_summary="api_key=sk-secret", - client=FakeClient(), - ) - - assert result["ok"] is False - assert result["reason"] == "secret_like_content" - - -def test_append_episode_rejects_password(): - result = memory_append_episode( - user_id="u", - agent_id="a", - session_id="s", - episode_summary="password=hunter2", - client=FakeClient(), - ) - - assert result["ok"] is False - assert result["reason"] == "secret_like_content" - - -def test_append_episode_rejects_raw_transcript(): - content = "\n".join(["User: hi", "Assistant: hello", "User: remember this", "Assistant: ok"]) - result = memory_append_episode(user_id="u", agent_id="a", session_id="s", episode_summary=content, client=FakeClient()) - - assert result["ok"] is False - assert result["reason"] == "raw_chat_transcript" - - -def test_append_episode_accepts_stable_preference(): - client = FakeClient() - result = memory_append_episode( - user_id="u", - agent_id="a", - session_id="s", - episode_summary="用户稳定偏好：以后所有技术方案都使用中文输出。", - tags=["preference"], - client=client, - ) - - assert result["ok"] is True - assert client.calls[0][0] == "append" - - -def test_upsert_uses_long_term_namespace_when_provided(): - client = FakeClient() - namespace = "user/u/long_term" - result = memory_upsert( - user_id="u", - agent_id="a", - namespace=namespace, - memory_type="preference", - content="用户稳定偏好：使用中文输出。", - client=client, - ) - - assert result["ok"] is True - assert client.calls[0][1]["namespace"] == namespace - - -def test_feedback_calls_correct_endpoint(): - client = FakeClient() - result = memory_feedback(user_id="u", agent_id="a", memory_id="mem_1", feedback="reject", client=client) - - assert result["ok"] is True - assert client.calls[0] == ("feedback", "mem_1", {"user_id": "u", "agent_id": "a", "feedback": "incorrect", "comment": None}) - diff --git a/plugins/memory-gateway-agent/tools.py b/plugins/memory-gateway-agent/tools.py deleted file mode 100644 index 92e7db4..0000000 --- a/plugins/memory-gateway-agent/tools.py +++ /dev/null @@ -1,42 +0,0 @@ -from __future__ import annotations - -import json -from typing import Any - -try: - from .memory_gateway_plugin import tools as impl -except ImportError: - from memory_gateway_plugin import tools as impl - - -def _json_result(payload: dict[str, Any]) -> str: - return json.dumps(payload, ensure_ascii=False, default=str) - - -def memory_search(args: dict[str, Any], **_: Any) -> str: - return _json_result(impl.memory_search(**args)) - - -def memory_append_episode(args: dict[str, Any], **_: Any) -> str: - return _json_result(impl.memory_append_episode(**args)) - - -def memory_commit_session(args: dict[str, Any], **_: Any) -> str: - return _json_result(impl.memory_commit_session(**args)) - - -def memory_upsert(args: dict[str, Any], **_: Any) -> str: - return _json_result(impl.memory_upsert(**args)) - - -def memory_feedback(args: dict[str, Any], **_: Any) -> str: - return _json_result(impl.memory_feedback(**args)) - - -HANDLERS = { - "memory_search": memory_search, - "memory_append_episode": memory_append_episode, - "memory_commit_session": memory_commit_session, - "memory_upsert": memory_upsert, - "memory_feedback": memory_feedback, -} diff --git a/pyproject.toml b/pyproject.toml index 7ecdbc3..e79c3b0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,25 +1,20 @@ [project] -name = "memory-gateway" +name = "memory-system-api" version = "0.1.0" -description = "Generic Memory Gateway for OpenViking, Obsidian, LLM summarization, and agent memory workflows" +description = "Lightweight Memory System API for OpenViking session memory and EverOS user profiles" readme = "README.md" requires-python = ">=3.10" dependencies = [ "fastapi>=0.109.0", - "sse-starlette>=2.0.0", - "mcp[cli]>=1.1.0", "httpx>=0.26.0", "pydantic>=2.5.0", "pyyaml>=6.0", "uvicorn>=0.27.0", - "markitdown[all]>=0.1.5", - "python-multipart>=0.0.9", ] [project.optional-dependencies] dev = [ "pytest>=8.0.0", - "pytest-asyncio>=0.23.0", "ruff>=0.1.0", ] @@ -27,5 +22,8 @@ dev = [ requires = ["hatchling"] build-backend = "hatchling.build" +[tool.hatch.build.targets.wheel] +packages = ["memory_system_api"] + [tool.ruff] target-version = "py310" diff --git a/tests/integration/test_real_ingest.py b/tests/integration/test_real_ingest.py deleted file mode 100644 index 58edc2a..0000000 --- a/tests/integration/test_real_ingest.py +++ /dev/null @@ -1,102 +0,0 @@ -import asyncio -import os -from uuid import uuid4 - -import pytest -from fastapi import FastAPI -from httpx import ASGITransport, AsyncClient - -import memory_gateway.api_v2 as api_v2 -from memory_gateway.everos_client import EverOSClient -from memory_gateway.openviking_client import OpenVikingClient -from memory_gateway.repositories import InMemoryRepository -from memory_gateway.schemas_v2 import BackendRefStatus, BackendType, IngestRequest, IngestResponse, OperationStatus -from memory_gateway.server_auth import verify_api_key_compat -from memory_gateway.services_v2 import MemoryGatewayV2Service - - -pytestmark = pytest.mark.skipif( - os.environ.get("RUN_REAL_BACKEND_TESTS") != "1", - reason="real backend ingest test is opt-in; set RUN_REAL_BACKEND_TESTS=1", -) - - -def _env(name: str) -> str: - value = os.environ.get(name) - if not value: - pytest.skip(f"{name} is required for real backend ingest test") - return value - - -def test_real_openviking_and_everos_ingest_writes_memory_refs(): - openviking_base_url = _env("OPENVIKING_BASE_URL") - everos_base_url = _env("EVEROS_BASE_URL") - openviking_api_key = os.environ.get("OPENVIKING_API_KEY", "") - everos_api_key = os.environ.get("EVEROS_API_KEY", "") - openviking_ingest_path = os.environ.get("OPENVIKING_INGEST_PATH") - everos_ingest_path = os.environ.get("EVEROS_INGEST_PATH") - - async def openviking_factory(): - return OpenVikingClient( - mode="real", - base_url=openviking_base_url, - api_key=openviking_api_key, - ingest_path=openviking_ingest_path, - ) - - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=openviking_factory, - everos_client=EverOSClient( - mode="real", - base_url=everos_base_url, - api_key=everos_api_key, - ingest_path=everos_ingest_path, - ), - ) - run_id = uuid4().hex[:12] - - response = asyncio.run(post_ingest(service, run_id)) - - refs = repo.list_memory_refs(session_id=f"real_ingest_sess_{run_id}", limit=10) - assert {ref.backend_type for ref in refs} == {BackendType.OPENVIKING, BackendType.EVEROS} - assert all(ref.content_hash for ref in refs) - openviking_ref = next(ref for ref in refs if ref.backend_type == BackendType.OPENVIKING) - everos_ref = next(ref for ref in refs if ref.backend_type == BackendType.EVEROS) - - assert openviking_ref.status == BackendRefStatus.SUCCESS - if everos_ref.status == BackendRefStatus.SUCCESS: - assert response.status == OperationStatus.SUCCESS - assert everos_ref.native_id - assert everos_ref.native_uri - else: - assert everos_ref.status == BackendRefStatus.FAILED - assert response.status == OperationStatus.PARTIAL_SUCCESS - assert everos_ref.error_message - - -async def post_ingest(service: MemoryGatewayV2Service, run_id: str): - api_v2.v2_service = service - app = FastAPI() - app.dependency_overrides[verify_api_key_compat] = lambda: None - app.include_router(api_v2.router) - request = IngestRequest( - workspace_id=os.environ.get("REAL_BACKEND_WORKSPACE_ID", "ws_real_ingest"), - user_id=os.environ.get("REAL_BACKEND_USER_ID", "user_real_ingest"), - agent_id=os.environ.get("REAL_BACKEND_AGENT_ID", "agent_real_ingest"), - session_id=f"real_ingest_sess_{run_id}", - turn_id=f"real_ingest_turn_{run_id}", - request_id=f"real_ingest_req_{run_id}", - idempotency_key=f"real_ingest_idem_{run_id}", - namespace=os.environ.get("REAL_BACKEND_NAMESPACE", "workspace/ws_real_ingest/user/user_real_ingest"), - source_type="integration_test", - source_event_id=f"real_ingest_evt_{run_id}", - role="user", - content=f"Memory Gateway real ingest smoke test {run_id}", - metadata={"source_channel": "integration_test"}, - ) - async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client: - response = await client.post("/v2/conversations/ingest", json=request.model_dump(mode="json")) - response.raise_for_status() - return IngestResponse.model_validate(response.json()) diff --git a/tests/test_memory_system_clients.py b/tests/test_memory_system_clients.py new file mode 100644 index 0000000..a4f2f33 --- /dev/null +++ b/tests/test_memory_system_clients.py @@ -0,0 +1,33 @@ +from memory_system_api.clients import EverOSMemorySystemClient + + +def test_everos_assistant_payload_does_not_use_user_id_as_sender(): + client = EverOSMemorySystemClient() + + payload = client.build_message_payload( + user_id="tom", + session_id="sess-1", + role="assistant", + content="我记住了", + ) + + message = payload["messages"][0] + assert message["role"] == "assistant" + assert message["sender_id"] != "tom" + assert message["sender_name"] != "tom" + + +def test_everos_user_payload_uses_user_id_as_sender(): + client = EverOSMemorySystemClient() + + payload = client.build_message_payload( + user_id="tom", + session_id="sess-1", + role="user", + content="我喜欢拿铁", + ) + + message = payload["messages"][0] + assert message["role"] == "user" + assert message["sender_id"] == "tom" + assert message["sender_name"] == "tom" diff --git a/tests/test_memory_system_server.py b/tests/test_memory_system_server.py new file mode 100644 index 0000000..7e74bf3 --- /dev/null +++ b/tests/test_memory_system_server.py @@ -0,0 +1,7 @@ +def test_memory_system_server_exposes_routes(): + from memory_system_api.server import app + + paths = {route.path for route in app.routes} + assert "/memory-system/messages" in paths + assert "/memory-system/search" in paths + assert "/memory-system/users/{user_id}/profile" in paths diff --git a/tests/test_memory_system_service.py b/tests/test_memory_system_service.py new file mode 100644 index 0000000..ece25b4 --- /dev/null +++ b/tests/test_memory_system_service.py @@ -0,0 +1,138 @@ +import asyncio + +from memory_system_api.schemas import MessageIngestRequest, SearchRequest +from memory_system_api.service import MemorySystemService + + +class FakeOpenViking: + def __init__(self, fail_on_append: bool = False): + self.fail_on_append = fail_on_append + self.calls = [] + + async def ensure_user(self, user_id: str) -> str: + self.calls.append(("ensure_user", user_id)) + return f"key-{user_id}" + + async def ensure_session(self, user_key: str, session_id: str) -> dict: + self.calls.append(("ensure_session", user_key, session_id)) + return {"session_id": session_id} + + async def append_message(self, user_key: str, session_id: str, role: str, content: str) -> dict: + self.calls.append(("append_message", user_key, session_id, role, content)) + if self.fail_on_append: + raise RuntimeError("openviking append failed") + return {"message_count": len([call for call in self.calls if call[0] == "append_message"])} + + async def find(self, user_key: str, user_id: str, query: str, limit: int) -> dict: + self.calls.append(("find", user_key, user_id, query, limit)) + await asyncio.sleep(0.01) + return {"items": [{"source": "openviking-find"}]} + + async def search(self, user_key: str, session_id: str | None, query: str, limit: int) -> dict: + self.calls.append(("search", user_key, session_id, query, limit)) + await asyncio.sleep(0.01) + return {"items": [{"source": "openviking-search"}]} + + +class FakeEverOS: + def __init__(self, fail_on_append: bool = False): + self.fail_on_append = fail_on_append + self.calls = [] + + async def append_message(self, user_id: str, session_id: str, role: str, content: str) -> dict: + self.calls.append(("append_message", user_id, session_id, role, content)) + if self.fail_on_append: + raise RuntimeError("everos append failed") + return {"status": "accumulated"} + + async def search(self, user_id: str, session_id: str | None, query: str, method: str, limit: int) -> dict: + self.calls.append(("search", user_id, session_id, query, method, limit)) + await asyncio.sleep(0.01) + return {"items": [{"source": f"everos-{method}"}]} + + +def test_ingest_splits_user_and_assistant_messages(): + openviking = FakeOpenViking() + everos = FakeEverOS() + service = MemorySystemService(openviking=openviking, everos=everos) + + response = asyncio.run(service.ingest_messages( + MessageIngestRequest( + user_id="tom", + session_id="sess-1", + user_message="我喜欢拿铁", + assistant_message="我记住了", + ) + )) + + assert response.status == "success" + assert response.message_count == 2 + assert openviking.calls == [ + ("ensure_user", "tom"), + ("ensure_session", "key-tom", "sess-1"), + ("append_message", "key-tom", "sess-1", "user", "我喜欢拿铁"), + ("append_message", "key-tom", "sess-1", "assistant", "我记住了"), + ] + assert everos.calls == [ + ("append_message", "tom", "sess-1", "user", "我喜欢拿铁"), + ("append_message", "tom", "sess-1", "assistant", "我记住了"), + ] + + +def test_ingest_requires_at_least_one_message(): + service = MemorySystemService(openviking=FakeOpenViking(), everos=FakeEverOS()) + + try: + asyncio.run(service.ingest_messages(MessageIngestRequest(user_id="tom", session_id="sess-1"))) + except ValueError as exc: + assert "at least one message" in str(exc) + else: + raise AssertionError("expected ValueError") + + +def test_ingest_returns_partial_success_when_one_backend_fails(): + service = MemorySystemService(openviking=FakeOpenViking(fail_on_append=True), everos=FakeEverOS()) + + response = asyncio.run(service.ingest_messages( + MessageIngestRequest(user_id="tom", session_id="sess-1", user_message="hello") + )) + + assert response.status == "partial_success" + assert response.backends["openviking"].status == "failed" + assert response.backends["everos"].status == "success" + + +def test_search_uses_find_and_hybrid_without_llm(): + openviking = FakeOpenViking() + everos = FakeEverOS() + service = MemorySystemService(openviking=openviking, everos=everos) + + response = asyncio.run(service.search( + SearchRequest(user_id="tom", session_id="sess-1", query="咖啡偏好", use_llm=False, limit=5) + )) + + assert response.status == "success" + assert response.items == [ + {"source_backend": "openviking", "source": "openviking-find"}, + {"source_backend": "everos", "source": "everos-hybrid"}, + ] + assert ("find", "key-tom", "tom", "咖啡偏好", 5) in openviking.calls + assert ("search", "tom", "sess-1", "咖啡偏好", "hybrid", 5) in everos.calls + + +def test_search_uses_search_and_agentic_with_llm(): + openviking = FakeOpenViking() + everos = FakeEverOS() + service = MemorySystemService(openviking=openviking, everos=everos) + + response = asyncio.run(service.search( + SearchRequest(user_id="tom", session_id="sess-1", query="咖啡偏好", use_llm=True, limit=5) + )) + + assert response.status == "success" + assert response.items == [ + {"source_backend": "openviking", "source": "openviking-search"}, + {"source_backend": "everos", "source": "everos-agentic"}, + ] + assert ("search", "key-tom", "sess-1", "咖啡偏好", 5) in openviking.calls + assert ("search", "tom", "sess-1", "咖啡偏好", "agentic", 5) in everos.calls diff --git a/tests/test_server.py b/tests/test_server.py deleted file mode 100644 index 470ef32..0000000 --- a/tests/test_server.py +++ /dev/null @@ -1,254 +0,0 @@ -import asyncio -import sys -import types - -import pytest -from fastapi import HTTPException -from fastapi.responses import StreamingResponse - - -def install_test_stubs() -> None: - if "mcp.server" not in sys.modules: - mcp_module = types.ModuleType("mcp") - mcp_server_module = types.ModuleType("mcp.server") - mcp_types_module = types.ModuleType("mcp.types") - - class Server: - def __init__(self, name): - self.name = name - - def list_tools(self): - def decorator(func): - return func - return decorator - - def call_tool(self): - def decorator(func): - return func - return decorator - - class Tool: - def __init__(self, name, description, inputSchema): - self.name = name - self.description = description - self.inputSchema = inputSchema - - class TextContent: - def __init__(self, type, text): - self.type = type - self.text = text - - def model_dump(self): - return {"type": self.type, "text": self.text} - - mcp_server_module.Server = Server - mcp_types_module.Tool = Tool - mcp_types_module.TextContent = TextContent - sys.modules["mcp"] = mcp_module - sys.modules["mcp.server"] = mcp_server_module - sys.modules["mcp.types"] = mcp_types_module - - if "sse_starlette" not in sys.modules: - sse_module = types.ModuleType("sse_starlette") - - class EventSourceResponse(StreamingResponse): - def __init__(self, content, *args, **kwargs): - super().__init__(content, media_type="text/event-stream", *args, **kwargs) - - sse_module.EventSourceResponse = EventSourceResponse - sys.modules["sse_starlette"] = sse_module - - -install_test_stubs() - -import memory_gateway.server as server -from memory_gateway.types import CommitSummaryRequest, Config, ObsidianConfig, SearchRequest, SearchResult, ServerConfig - - -class FakeOVClient: - async def health_check(self): - return {"status": "ok", "backend": "fake"} - - async def search(self, query, namespace=None, limit=None, uri=None): - return SearchResult( - results=[ - { - "uri": "viking://memory-gateway/test", - "abstract": query, - "score": 1.0, - "context_type": "memory", - } - ], - total=1, - ) - - async def add_memory(self, content, namespace=None, memory_type="general"): - return { - "status": "ok", - "content": content, - "namespace": namespace, - "memory_type": memory_type, - } - - async def add_resource(self, uri, content, resource_type="text"): - return { - "status": "ok", - "uri": uri, - "content": content, - "resource_type": resource_type, - } - - async def list_memories(self, namespace=None, memory_type=None, limit=None): - return [] - - async def list_resources(self, namespace=None, limit=None): - return [] - - -async def fake_get_openviking_client(): - return FakeOVClient() - - -async def fake_summarize_with_llm(content, **kwargs): - return { - "title": kwargs.get("title") or "Fake LLM title", - "summary": f"LLM summary: {content[:80]}", - "key_points": ["LLM key point", "Preserve IP 198.51.100.20"], - "tags": kwargs.get("tags") or ["fake"], - "llm": {"provider": "fake", "model": "fake-model"}, - } - - -class FakeUploadFile: - def __init__(self, filename: str, content: bytes) -> None: - self.filename = filename - self._content = content - - async def read(self) -> bytes: - return self._content - -def test_health_requires_api_key(monkeypatch): - monkeypatch.setattr( - "memory_gateway.server.get_config", - lambda: Config(server=ServerConfig(api_key="secret")), - ) - monkeypatch.setattr( - "memory_gateway.server.get_openviking_client", - fake_get_openviking_client, - ) - monkeypatch.setattr("memory_gateway.server.summarize_with_llm", fake_summarize_with_llm) - monkeypatch.setattr("memory_gateway.server.v1_service.everos_health", lambda: {"status": "disabled"}) - - with pytest.raises(HTTPException) as exc_info: - server.verify_api_key() - assert exc_info.value.status_code == 401 - - server.verify_api_key("secret") - payload = asyncio.run(server.health_check()) - assert payload["openviking"]["status"] == "ok" - - -def test_mcp_rpc_lists_tools_with_api_key(monkeypatch): - monkeypatch.setattr( - "memory_gateway.server.get_config", - lambda: Config(server=ServerConfig(api_key="secret")), - ) - monkeypatch.setattr( - "memory_gateway.server.get_openviking_client", - fake_get_openviking_client, - ) - - server.verify_api_key("secret") - tools = asyncio.run(server.list_tools()) - assert len(tools) >= 7 - assert any(tool.name == "commit_summary" for tool in tools) - assert any(tool.name == "memory_search" for tool in tools) - - -def test_search_passes_through_gateway(monkeypatch): - monkeypatch.setattr( - "memory_gateway.server.get_config", - lambda: Config(server=ServerConfig(api_key="")), - ) - monkeypatch.setattr( - "memory_gateway.server.get_openviking_client", - fake_get_openviking_client, - ) - - payload = asyncio.run(server.api_search(SearchRequest(query="phishing"))) - assert payload["total"] == 1 - assert payload["results"][0]["abstract"] == "phishing" - - -def test_summary_endpoint_builds_generic_artifact(monkeypatch): - monkeypatch.setattr( - "memory_gateway.server.get_config", - lambda: Config(server=ServerConfig(api_key="")), - ) - monkeypatch.setattr( - "memory_gateway.server.get_openviking_client", - fake_get_openviking_client, - ) - monkeypatch.setattr("memory_gateway.server.summarize_with_llm", fake_summarize_with_llm) - - payload = asyncio.run( - server.api_commit_summary( - CommitSummaryRequest( - title="Demo investigation summary", - content="结论：这是一次高价值沉淀。\n- 证据：命中历史 case。\n- 建议：后续复用该处置路径。", - namespace="demo", - memory_type="knowledge", - tags=["demo", "summary"], - persist_as="none", - ) - ) - ) - assert payload["status"] == "ok" - assert payload["artifact"]["title"] == "Demo investigation summary" - assert payload["artifact"]["namespace"] == "demo" - assert payload["artifact"]["memory_type"] == "knowledge" - assert payload["artifact"]["summary"].startswith("LLM summary:") - assert payload["artifact"]["llm"]["provider"] == "fake" - assert payload["memory_result"] is None - assert payload["resource_result"] is None - - -def test_knowledge_upload_converts_saves_and_commits(monkeypatch, tmp_path): - monkeypatch.setattr( - "memory_gateway.server.get_config", - lambda: Config( - server=ServerConfig(api_key=""), - obsidian=ObsidianConfig(vault_path=str(tmp_path / "vault"), knowledge_dir="01_Knowledge/Uploaded"), - ), - ) - monkeypatch.setattr("memory_gateway.server.get_openviking_client", fake_get_openviking_client) - monkeypatch.setattr("memory_gateway.server.summarize_with_llm", fake_summarize_with_llm) - monkeypatch.setattr("memory_gateway.server.convert_file_to_markdown", lambda path: "# Uploaded Doc\n\nImportant uploaded knowledge.") - - async def fake_to_thread(func, *args, **kwargs): - return func(*args, **kwargs) - - monkeypatch.setattr("memory_gateway.server.asyncio.to_thread", fake_to_thread) - - upload = FakeUploadFile(filename="sample.txt", content=b"hello") - payload = asyncio.run( - server.api_upload_knowledge( - file=upload, - title="Uploaded Knowledge", - namespace="demo", - knowledge_type="playbook", - tags="demo,upload", - source=None, - obsidian_dir=None, - resource_uri=None, - persist_as="resource", - max_summary_chars=1000, - ) - ) - - assert payload["status"] == "ok" - assert payload["artifact"]["schema_version"] == "memory-gateway.knowledge_upload.v1" - assert payload["artifact"]["knowledge_type"] == "playbook" - assert payload["artifact"]["markdown_content"].startswith("# Uploaded Doc") - assert payload["resource_result"]["status"] == "ok" - assert (tmp_path / "vault" / payload["artifact"]["obsidian_relative_path"]).exists() diff --git a/tests/test_v1_mcp.py b/tests/test_v1_mcp.py deleted file mode 100644 index db735a1..0000000 --- a/tests/test_v1_mcp.py +++ /dev/null @@ -1,29 +0,0 @@ -import asyncio - -from memory_gateway.repositories import InMemoryRepository -from memory_gateway.services import MemoryGatewayService - - -def test_v1_mcp_tools_are_exposed_and_dispatch(monkeypatch): - import memory_gateway.server as server - - service = MemoryGatewayService(InMemoryRepository()) - monkeypatch.setattr(server, "v1_service", service) - - tools = asyncio.run(server.list_tools()) - assert any(tool.name == "memory_search" for tool in tools) - assert any(tool.name == "memory_commit_session" for tool in tools) - - result = asyncio.run( - server.call_v1_memory_tool( - "memory_upsert", - { - "user_id": "user_a", - "content": "MCP 写入的 v1 memory", - "visibility": "private", - }, - ) - ) - - assert result["user_id"] == "user_a" - assert result["namespace"] == "user/user_a/long_term" diff --git a/tests/test_v1_service.py b/tests/test_v1_service.py deleted file mode 100644 index 4b1df12..0000000 --- a/tests/test_v1_service.py +++ /dev/null @@ -1,183 +0,0 @@ -import asyncio - -from memory_gateway.repositories import InMemoryRepository, SQLiteRepository -from memory_gateway.schemas import ( - AccessContext, - CommitSessionRequest, - CreateUserRequest, - EpisodeAppendRequest, - MemorySearchRequest, - MemoryUpsertRequest, - Visibility, -) -from memory_gateway.services import MemoryGatewayService -from memory_gateway.types import Config, EverOSConfig, ObsidianConfig - - -def test_private_memory_is_isolated_by_user(): - service = MemoryGatewayService(InMemoryRepository()) - service.create_user(CreateUserRequest(user_id="user_a", display_name="A")) - service.create_user(CreateUserRequest(user_id="user_b", display_name="B")) - - memory = service.upsert_memory( - MemoryUpsertRequest( - user_id="user_a", - content="用户 A 的私有偏好是中文输出", - visibility=Visibility.PRIVATE, - ) - ) - - own_results = service.search_memory(MemorySearchRequest(user_id="user_a", query="中文")) - other_results = service.search_memory(MemorySearchRequest(user_id="user_b", query="中文")) - - assert own_results["total"] == 1 - assert own_results["results"][0]["memory"].id == memory.id - assert other_results["total"] == 0 - - -def test_workspace_memory_requires_matching_workspace(): - service = MemoryGatewayService(InMemoryRepository()) - memory = service.upsert_memory( - MemoryUpsertRequest( - user_id="user_a", - workspace_id="ws_1", - content="workspace 共享的项目决策", - visibility=Visibility.WORKSPACE_SHARED, - ) - ) - - visible = service.get_memory(memory.id, AccessContext(user_id="user_b", workspace_id="ws_1")) - assert visible.id == memory.id - - hidden = service.search_memory(MemorySearchRequest(user_id="user_b", workspace_id="ws_2", query="项目决策")) - assert hidden["total"] == 0 - - -def test_sqlite_repository_persists_memory(tmp_path): - db_path = tmp_path / "memory_gateway.sqlite3" - repo = SQLiteRepository(db_path) - service = MemoryGatewayService(repo) - - service.create_user(CreateUserRequest(user_id="user_a", display_name="A")) - memory = service.upsert_memory(MemoryUpsertRequest(user_id="user_a", content="持久化 SQLite memory")) - - reloaded_service = MemoryGatewayService(SQLiteRepository(db_path)) - reloaded = reloaded_service.get_memory(memory.id, AccessContext(user_id="user_a")) - - assert reloaded.content == "持久化 SQLite memory" - - -def test_commit_session_disabled_does_not_use_local_fallback(monkeypatch, tmp_path): - monkeypatch.setattr( - "memory_gateway.services.get_config", - lambda: Config(everos=EverOSConfig(enabled=False)), - ) - monkeypatch.setattr( - "memory_gateway.obsidian_review.get_config", - lambda: Config(obsidian=ObsidianConfig(vault_path=str(tmp_path / "vault"), review_dir="Reviews/Queue")), - ) - service = MemoryGatewayService(InMemoryRepository()) - service.append_episode( - EpisodeAppendRequest( - user_id="user_a", - session_id="sess_1", - content="结论：这个项目必须保留用户隔离和 namespace ACL。", - tags=["decision"], - ) - ) - service.append_episode( - EpisodeAppendRequest( - user_id="user_a", - session_id="sess_1", - content="重要：这条高价值记忆需要人工 review 后再进入长期记忆。", - tags=["review", "high-value"], - ) - ) - - result = service.commit_session( - "sess_1", - CommitSessionRequest( - user_id="user_a", - session_id="sess_1", - min_importance=0.6, - ), - ) - - assert result["promoted"] == [] - assert result["everos_backend"] == "disabled" - - -def test_commit_session_uses_external_everos(monkeypatch): - monkeypatch.setattr( - "memory_gateway.services.get_config", - lambda: Config(everos=EverOSConfig(enabled=True)), - ) - - class FakeEverOSClient: - def consolidate_session(self, **kwargs): - return { - "episodes": 1, - "candidates": [], - "promoted": [ - { - "content": "外部 EverOS 总结出的长期记忆", - "summary": "外部 EverOS 长期记忆", - "memory_type": "summary", - "tags": ["external-everos"], - } - ], - "duplicates": [], - "conflicts": [], - "review_drafts": [], - } - - def health(self): - return {"status": "ok"} - - service = MemoryGatewayService(InMemoryRepository(), everos_client=FakeEverOSClient()) - service.append_episode( - EpisodeAppendRequest( - user_id="user_a", - session_id="sess_external", - content="这条 episode 应该交给外部 EverOS。", - ) - ) - result = service.commit_session( - "sess_external", - CommitSessionRequest(user_id="user_a", session_id="sess_external"), - ) - - assert result["everos_backend"] == "external" - assert len(result["promoted"]) == 1 - search = service.search_memory(MemorySearchRequest(user_id="user_a", query="外部 EverOS")) - assert search["total"] == 1 - - -def test_search_fans_out_to_openviking_after_namespace_acl(monkeypatch): - service = MemoryGatewayService(InMemoryRepository()) - - class FakeSearchResult: - results = [{"uri": "viking://user/user_a/long_term/demo", "abstract": "OpenViking result", "score": 0.9}] - - class FakeOpenVikingClient: - async def search(self, query, namespace=None, limit=None, uri=None): - assert namespace == "user/user_a/long_term" - return FakeSearchResult() - - async def fake_get_openviking_client(): - return FakeOpenVikingClient() - - monkeypatch.setattr("memory_gateway.services.get_openviking_client", fake_get_openviking_client) - - result = asyncio.run( - service.search_memory_with_openviking( - MemorySearchRequest( - user_id="user_a", - query="demo", - namespaces=["user/user_a/long_term", "user/user_b/long_term"], - ) - ) - ) - - assert result["openviking_total"] == 1 - assert result["searched_namespaces"] == ["user/user_a/long_term"] diff --git a/tests/test_v2_api.py b/tests/test_v2_api.py deleted file mode 100644 index bb8e0ae..0000000 --- a/tests/test_v2_api.py +++ /dev/null @@ -1,1972 +0,0 @@ -import asyncio -import json -from datetime import datetime, timedelta, timezone -from pathlib import Path -from fastapi import FastAPI -import httpx -from httpx import ASGITransport, AsyncClient - -from memory_gateway.config import load_config -from memory_gateway.backend_adapter_mapping import ( - ADAPTER_MAPPING_SPECS, - DISALLOWED_PAYLOAD_FIELDS, - validate_control_plane_persisted_payload, - get_adapter_mapping_spec, - validate_control_plane_payload, -) -from memory_gateway.backend_normalization import ( - map_backend_error_to_retryable, - normalize_everos_commit_response, - normalize_everos_ingest_response, - normalize_everos_retrieve_response, - normalize_openviking_commit_response, - normalize_openviking_ingest_response, - normalize_openviking_retrieve_response, -) -from memory_gateway.backend_contracts import ( - BackendCommitResult, - BackendOperation, - BackendProducedRef, - BackendRetrieveItem, - BackendResultStatus, - BackendRetrieveResult, - BackendWriteResult, - OutboxEventStatus, -) -from memory_gateway.backend_ref_mapping import map_backend_ref_type -from memory_gateway.everos_client import EverOSClient -from memory_gateway.obsidian_review_client import ObsidianReviewClient -from memory_gateway.openviking_client import OpenVikingClient -from memory_gateway.repositories import InMemoryRepository, SQLiteRepository -from memory_gateway.schemas_v2 import ( - BackendRefStatus, - BackendType, - CommitRequest, - IngestRequest, - MemoryRefType, - OperationStatus, - OutboxProcessResponse, - RetrieveRequest, -) -from memory_gateway.server_auth import verify_api_key_compat -from memory_gateway.services_v2 import MemoryGatewayV2Service - - -DOCS_DIR = Path(__file__).parent.parent / "docs" - - -def backend_response(name: str): - responses = { - "openviking_ingest_success.json": { - "status": "created", - "id": "ov_turn_fixture_1", - "uri": "viking://sessions/sess_fixture/turns/ov_turn_fixture_1", - "metadata": {"schema_version": "openviking.fixture.ingest.v2", "conversation": "SECRET"}, - }, - "openviking_ingest_real_success.json": { - "status": "created", - "id": "ov_real_turn_fixture_1", - "uri": "viking://sessions/ov_real_sess_fixture_1/turns/ov_real_turn_fixture_1", - "metadata": {"backend_request_id": "ov_req_real_1", "content": "SECRET"}, - }, - "openviking_ingest_real_error_401.json": {"status": "failed", "error": "unauthorized", "error_code": "unauthorized"}, - "openviking_ingest_real_error_422.json": {"status": "failed", "error": "validation failed", "error_code": "validation_error"}, - "openviking_ingest_real_error_500.json": {"status": "failed", "error": "server error", "error_code": "server_error"}, - "openviking_commit_success.json": { - "status": "ok", - "session_id": "sess_fixture", - "result": { - "refs": [ - {"type": "session_archive", "id": "ov_archive_fixture_1"}, - {"type": "context_resource", "id": "ov_resource_fixture_1"}, - ] - }, - "metadata": {"schema_version": "openviking.fixture.commit.v2", "messages": ["SECRET"]}, - }, - "openviking_retrieve_success.json": { - "status": "ok", - "result": { - "items": [ - {"text": "Relevant session summary", "id": "ov_archive_fixture_1", "score": 0.91, "type": "session_archive"}, - {"text": "Relevant resource", "id": "ov_resource_fixture_1", "score": 0.84, "type": "context_resource"}, - ] - }, - "metadata": {"schema_version": "openviking.fixture.retrieve.v2", "transcript": "SECRET"}, - }, - "everos_ingest_success.json": { - "status": "success", - "memory_id": "em_memory_fixture_1", - "metadata": {"schema_version": "everos.fixture.ingest.v2", "transcript": "SECRET"}, - }, - "everos_commit_success_multiple_refs.json": { - "status": "success", - "data": { - "produced_refs": [ - {"ref_type": "episodic_memory", "memory_id": "em_episode_fixture_1"}, - {"ref_type": "profile", "profile_id": "em_profile_fixture_1"}, - {"ref_type": "unknown_kind", "id": "em_long_fixture_1"}, - ] - }, - "metadata": {"schema_version": "everos.fixture.commit.v2", "messages": ["SECRET"]}, - }, - "everos_retrieve_success.json": { - "status": "success", - "data": { - "items": [ - {"text": "Relevant episodic memory", "memory_id": "em_episode_fixture_1", "score": 0.88, "memory_type": "episodic_memory"}, - {"text": "Relevant profile", "profile_id": "em_profile_fixture_1", "score": 0.73, "memory_type": "profile"}, - ] - }, - "metadata": {"schema_version": "everos.fixture.retrieve.v2", "conversation": "SECRET"}, - }, - } - return responses[name] - - -def build_ingest_payload(**overrides): - payload = { - "workspace_id": "ws_1", - "user_id": "user_a", - "agent_id": "agent_cli", - "session_id": "sess_1", - "turn_id": "turn_1", - "request_id": "req_1", - "namespace": "workspace/ws_1/user/user_a", - "source_type": "cli", - "source_event_id": "evt_1", - "role": "user", - "content": "Need to remember this conversation turn.", - "metadata": {"channel": "test"}, - } - payload.update(overrides) - return payload - - -class FakeOpenVikingClient: - async def ingest_conversation_turn(self, payload): - return { - "status": "success", - "native_id": f"ov_{payload['turn_id']}", - "native_uri": f"viking://sessions/{payload['session_id']}/{payload['turn_id']}", - } - - async def retrieve_context_v2(self, payload): - return BackendRetrieveResult( - backend_type=BackendType.OPENVIKING, - status=BackendResultStatus.SUCCESS, - items=[ - BackendRetrieveItem( - text="OpenViking context for remember", - source_backend=BackendType.OPENVIKING, - ref_id="ov_ctx_1", - score=0.82, - memory_type="context_resource", - ) - ], - ) - - -async def fake_openviking_factory(): - return FakeOpenVikingClient() - - -class FakeEverOSClient: - def ingest_message(self, payload): - return { - "status": "success", - "native_id": f"em_{payload['turn_id']}", - "native_uri": f"everos://memories/{payload['turn_id']}", - } - - def retrieve_context_v2(self, payload): - return BackendRetrieveResult( - backend_type=BackendType.EVEROS, - status=BackendResultStatus.SUCCESS, - items=[ - BackendRetrieveItem( - text="EverOS memory for remember", - source_backend=BackendType.EVEROS, - ref_id="em_ctx_1", - score=0.91, - memory_type="episodic_memory", - ) - ], - ) - - -class FailingEverOSClient: - def ingest_message(self, payload): - raise RuntimeError("everos unavailable") - - -class FakeCommitOpenVikingClient: - def __init__(self, result: BackendCommitResult) -> None: - self.result = result - - async def commit_session_v2(self, payload): - return self.result - - -def fake_commit_openviking_factory(result: BackendCommitResult): - async def factory(): - return FakeCommitOpenVikingClient(result) - - return factory - - -class FakeCommitEverOSClient: - def __init__(self, result: BackendCommitResult) -> None: - self.result = result - - def extract_profile_long_term_v2(self, payload): - return self.result - - -def commit_result( - backend_type: BackendType, - status: BackendResultStatus, - native_id: str | None = None, - native_uri: str | None = None, - retryable: bool = False, - error_message: str | None = None, -): - return BackendCommitResult( - backend_type=backend_type, - operation=BackendOperation.COMMIT_SESSION, - status=status, - native_id=native_id, - native_uri=native_uri, - retryable=retryable, - error_message=error_message, - ) - - -def test_v2_adapters_return_backend_write_result_contract(): - ov_result = asyncio.run( - OpenVikingClient(mode="offline").ingest_conversation_turn( - { - "workspace_id": "ws_1", - "session_id": "sess_1", - "turn_id": "turn_1", - } - ) - ) - em_result = EverOSClient(mode="offline").ingest_message( - { - "workspace_id": "ws_1", - "session_id": "sess_1", - "turn_id": "turn_1", - } - ) - - assert isinstance(ov_result, BackendWriteResult) - assert isinstance(em_result, BackendWriteResult) - assert ov_result.backend_type == BackendType.OPENVIKING - assert em_result.backend_type == BackendType.EVEROS - assert ov_result.operation == BackendOperation.INGEST_TURN - assert em_result.operation == BackendOperation.INGEST_TURN - assert ov_result.status == BackendResultStatus.SKIPPED - assert em_result.status == BackendResultStatus.SKIPPED - - -def test_backend_env_overrides_enable_real_modes(monkeypatch, tmp_path): - monkeypatch.setenv("OPENVIKING_MODE", "real") - monkeypatch.setenv("OPENVIKING_BASE_URL", "http://openviking.env.test") - monkeypatch.setenv("OPENVIKING_API_KEY", "ov-env-token") - monkeypatch.setenv("OPENVIKING_TIMEOUT_SECONDS", "17") - monkeypatch.setenv("EVEROS_MODE", "real") - monkeypatch.setenv("EVEROS_BASE_URL", "http://everos.env.test") - monkeypatch.setenv("EVEROS_API_KEY", "em-env-token") - monkeypatch.setenv("EVEROS_INGEST_PATH", "/api/v1/memories") - - config = load_config(str(tmp_path / "missing.yaml")) - - assert config.openviking.mode == "real" - assert config.openviking.url == "http://openviking.env.test" - assert config.openviking.api_key == "ov-env-token" - assert config.openviking.timeout == 17 - assert config.everos.mode == "real" - assert config.everos.url == "http://everos.env.test" - assert config.everos.api_key == "em-env-token" - assert config.everos.ingest_path == "/api/v1/memories" - - -def test_openviking_default_ingest_does_not_touch_network(): - def handler(request): - raise AssertionError("offline OpenViking ingest should not perform HTTP") - - client = OpenVikingClient( - mode="offline", - base_url="http://openviking.test", - transport=httpx.MockTransport(handler), - ) - - result = asyncio.run(client.ingest_conversation_turn({"session_id": "sess_offline", "turn_id": "turn_1"})) - - assert result.status == BackendResultStatus.SKIPPED - assert result.native_uri == "viking://sessions/sess_offline" - - -def test_openviking_adapter_config_doc_exists_and_covers_modes_and_security(): - doc = (DOCS_DIR / "openviking_adapter_config.md").read_text() - - assert "offline" in doc - assert "real" in doc - assert "base_url" in doc - assert "api_key" in doc - assert "verify_ssl" in doc - assert "ingest_path" in doc - assert "content" in doc - assert "messages" in doc - assert "transcript" in doc - - -def test_openviking_mode_offline_does_not_touch_network_even_with_base_url(): - def handler(request): - raise AssertionError("offline mode should not perform HTTP") - - client = OpenVikingClient( - mode="offline", - base_url="http://openviking.test", - enabled=False, - transport=httpx.MockTransport(handler), - ) - - result = asyncio.run(client.ingest_conversation_turn({"session_id": "sess_offline_mode", "turn_id": "turn_1"})) - - assert result.status == BackendResultStatus.SKIPPED - - -def test_openviking_mode_skeleton_does_not_touch_network_even_with_base_url(): - def handler(request): - raise AssertionError("skeleton mode should not perform HTTP") - - client = OpenVikingClient( - mode="skeleton", - base_url="http://openviking.test", - enabled=False, - transport=httpx.MockTransport(handler), - ) - - result = asyncio.run(client.ingest_conversation_turn({"session_id": "sess_skeleton_mode", "turn_id": "turn_1"})) - - assert result.status == BackendResultStatus.SKIPPED - - -def test_openviking_mode_real_with_base_url_uses_mock_http(): - calls = {"count": 0} - - def handler(request): - calls["count"] += 1 - return httpx.Response(200, json=backend_response("openviking_ingest_real_success.json")) - - client = OpenVikingClient( - mode="real", - enabled=False, - base_url="http://openviking.test", - transport=httpx.MockTransport(handler), - ) - - result = asyncio.run(client.ingest_conversation_turn({"session_id": "ov_real_sess_fixture_1", "turn_id": "turn_real", "content": "SECRET"})) - - assert calls["count"] == 1 - assert result.status == BackendResultStatus.SUCCESS - - -def test_openviking_enabled_true_without_mode_real_does_not_touch_network(): - seen = {"calls": 0} - - def handler(request): - seen["calls"] += 1 - raise AssertionError("enabled=True must not perform HTTP without mode=real") - - client = OpenVikingClient( - mode="offline", - enabled=True, - base_url="http://openviking.test", - transport=httpx.MockTransport(handler), - ) - - result = asyncio.run(client.ingest_conversation_turn({"session_id": "ov_real_sess_fixture_1", "turn_id": "turn_x", "content": "SECRET"})) - - assert seen["calls"] == 0 - assert result.status == BackendResultStatus.SKIPPED - - -def test_openviking_real_ingest_mode_real_without_base_url_returns_config_error(): - client = OpenVikingClient(mode="real", base_url="") - - result = asyncio.run(client.ingest_conversation_turn({"session_id": "sess_missing_url", "content": "SECRET"})) - - assert result.status == BackendResultStatus.FAILED - assert result.retryable is False - assert result.error_code == "config_error" - assert "SECRET" not in json.dumps(result.model_dump(mode="json"), ensure_ascii=False) - - -def test_openviking_real_ingest_success_uses_mock_http_and_normalization(): - seen_payload = {} - seen_headers = {} - fixture = backend_response("openviking_ingest_real_success.json") - - def handler(request): - seen_payload.update(json.loads(request.content.decode())) - seen_headers.update(dict(request.headers)) - return httpx.Response( - 200, - json=fixture, - ) - - client = OpenVikingClient( - mode="real", - base_url="http://openviking.test", - api_key="token", - transport=httpx.MockTransport(handler), - ) - - result = asyncio.run( - client.ingest_conversation_turn( - { - "workspace_id": "ws_1", - "session_id": "ov_real_sess_fixture_1", - "turn_id": "turn_real", - "content": "SECRET_REAL_CONTENT", - } - ) - ) - expected = normalize_openviking_ingest_response(fixture) - - assert seen_payload["content"] == "SECRET_REAL_CONTENT" - assert seen_headers["x-api-key"] == "token" - assert result == expected - assert result.status == BackendResultStatus.SUCCESS - serialized = json.dumps(result.model_dump(mode="json"), ensure_ascii=False) - assert "SECRET_REAL_CONTENT" not in serialized - assert "content" not in serialized - assert "token" not in serialized - - -def test_openviking_real_ingest_timeout_is_retryable_and_safe(): - def handler(request): - raise httpx.ReadTimeout("timeout while sending SECRET_TIMEOUT_CONTENT") - - client = OpenVikingClient( - mode="real", - base_url="http://openviking.test", - transport=httpx.MockTransport(handler), - ) - - result = asyncio.run(client.ingest_conversation_turn({"session_id": "sess_timeout", "content": "SECRET_TIMEOUT_CONTENT"})) - - assert result.status == BackendResultStatus.FAILED - assert result.retryable is True - assert result.error_code == "timeout" - assert "SECRET_TIMEOUT_CONTENT" not in json.dumps(result.model_dump(mode="json"), ensure_ascii=False) - - -def test_openviking_real_ingest_http_retryable_and_nonretryable_statuses(): - def client_for_fixture(name, status_code): - return OpenVikingClient( - mode="real", - base_url="http://openviking.test", - api_key="super-secret-token", - transport=httpx.MockTransport(lambda request: httpx.Response(status_code, json=backend_response(name))), - ) - - result_429 = asyncio.run(client_for_fixture("openviking_ingest_real_error_500.json", 429).ingest_conversation_turn({"session_id": "sess_http"})) - assert result_429.status == BackendResultStatus.FAILED - assert result_429.retryable is True - assert result_429.error_code == "http_429" - - result_500 = asyncio.run(client_for_fixture("openviking_ingest_real_error_500.json", 500).ingest_conversation_turn({"session_id": "sess_http"})) - assert result_500.status == BackendResultStatus.FAILED - assert result_500.retryable is True - assert result_500.error_code == "http_500" - assert "super-secret-token" not in json.dumps(result_500.model_dump(mode="json"), ensure_ascii=False) - - for name, status_code in ( - ("openviking_ingest_real_error_401.json", 401), - ("openviking_ingest_real_error_401.json", 403), - ("openviking_ingest_real_error_422.json", 422), - ): - result = asyncio.run(client_for_fixture(name, status_code).ingest_conversation_turn({"session_id": "sess_http"})) - assert result.status == BackendResultStatus.FAILED - assert result.retryable is False - assert result.error_code == f"http_{status_code}" - - -def test_openviking_real_ingest_invalid_json_returns_failed_retryable(): - client = OpenVikingClient( - mode="real", - base_url="http://openviking.test", - transport=httpx.MockTransport(lambda request: httpx.Response(200, content=b"not-json")), - ) - - result = asyncio.run(client.ingest_conversation_turn({"session_id": "sess_invalid_json", "content": "SECRET_JSON"})) - - assert result.status == BackendResultStatus.FAILED - assert result.retryable is True - assert result.error_code == "invalid_json" - assert "SECRET_JSON" not in json.dumps(result.model_dump(mode="json"), ensure_ascii=False) - - -def test_everos_default_ingest_does_not_touch_network_even_if_enabled(): - def handler(request): - raise AssertionError("EverOS ingest should not perform HTTP unless mode=real") - - client = EverOSClient( - enabled=True, - mode="offline", - base_url="http://everos.test", - transport=httpx.MockTransport(handler), - ) - - result = client.ingest_message({"session_id": "sess_offline", "turn_id": "turn_1", "content": "SECRET"}) - - assert result.status == BackendResultStatus.SKIPPED - - -def test_everos_real_ingest_mode_real_without_base_url_returns_config_error(): - client = EverOSClient(mode="real", base_url="") - - result = client.ingest_message({"session_id": "sess_missing_url", "content": "SECRET"}) - - assert result.status == BackendResultStatus.FAILED - assert result.retryable is False - assert result.error_code == "config_error" - assert "SECRET" not in json.dumps(result.model_dump(mode="json"), ensure_ascii=False) - - -def test_everos_real_ingest_success_uses_mock_http_and_normalization(): - seen_payload = {} - seen_headers = {} - fixture = backend_response("everos_ingest_success.json") - - def handler(request): - seen_payload.update(json.loads(request.content.decode())) - seen_headers.update(dict(request.headers)) - return httpx.Response(200, json=fixture) - - client = EverOSClient( - mode="real", - base_url="http://everos.test", - api_key="em-token", - transport=httpx.MockTransport(handler), - ) - - result = client.ingest_message( - { - "workspace_id": "ws_1", - "user_id": "user_a", - "session_id": "sess_1", - "turn_id": "turn_1", - "role": "user", - "content": "SECRET_EM_CONTENT", - "source_type": "cli", - "source_event_id": "evt_1", - "metadata": {"channel": "test"}, - } - ) - expected = normalize_everos_ingest_response(fixture) - - assert seen_payload["messages"][0]["content"] == "SECRET_EM_CONTENT" - assert seen_headers["x-api-key"] == "em-token" - assert seen_headers["authorization"] == "Bearer em-token" - assert result == expected - serialized = json.dumps(result.model_dump(mode="json"), ensure_ascii=False) - assert "SECRET_EM_CONTENT" not in serialized - assert "content" not in serialized - assert "em-token" not in serialized - - -def test_everos_real_ingest_errors_are_backend_write_results_and_safe(): - def client_for_response(status_code, body=None, content=None): - return EverOSClient( - mode="real", - base_url="http://everos.test", - api_key="em-super-secret-token", - transport=httpx.MockTransport(lambda request: httpx.Response(status_code, json=body, content=content)), - ) - - result_500 = client_for_response(500, {"error_code": "server_error"}).ingest_message({"content": "SECRET"}) - assert result_500.status == BackendResultStatus.FAILED - assert result_500.retryable is True - assert result_500.error_code == "http_500" - - for status_code in (401, 403, 422): - result = client_for_response(status_code, {"error_code": "auth_or_validation"}).ingest_message({"content": "SECRET"}) - assert result.status == BackendResultStatus.FAILED - assert result.retryable is False - assert result.error_code == f"http_{status_code}" - - invalid = client_for_response(200, content=b"not-json").ingest_message({"content": "SECRET"}) - assert invalid.status == BackendResultStatus.FAILED - assert invalid.retryable is True - assert invalid.error_code == "invalid_json" - - serialized = json.dumps( - [result_500.model_dump(mode="json"), invalid.model_dump(mode="json")], - ensure_ascii=False, - ) - assert "SECRET" not in serialized - assert "em-super-secret-token" not in serialized - - -def test_everos_real_ingest_timeout_is_retryable_and_safe(): - def handler(request): - raise httpx.ReadTimeout("timeout while sending SECRET_TIMEOUT_CONTENT") - - client = EverOSClient( - mode="real", - base_url="http://everos.test", - transport=httpx.MockTransport(handler), - ) - - result = client.ingest_message({"session_id": "sess_timeout", "content": "SECRET_TIMEOUT_CONTENT"}) - - assert result.status == BackendResultStatus.FAILED - assert result.retryable is True - assert result.error_code == "timeout" - assert "SECRET_TIMEOUT_CONTENT" not in json.dumps(result.model_dump(mode="json"), ensure_ascii=False) - - -def test_backend_adapter_mapping_spec_is_contract_first_and_control_plane_only(): - expected = { - (BackendType.OPENVIKING, BackendOperation.INGEST_TURN), - (BackendType.OPENVIKING, BackendOperation.COMMIT_SESSION), - (BackendType.OPENVIKING, BackendOperation.RETRIEVE_CONTEXT), - (BackendType.EVEROS, BackendOperation.INGEST_TURN), - (BackendType.EVEROS, BackendOperation.COMMIT_SESSION), - (BackendType.EVEROS, BackendOperation.RETRIEVE_CONTEXT), - (BackendType.OBSIDIAN, BackendOperation.CREATE_REVIEW_DRAFT), - } - - assert {(spec.backend_type, spec.operation) for spec in ADAPTER_MAPPING_SPECS} == expected - for spec in ADAPTER_MAPPING_SPECS: - assert not DISALLOWED_PAYLOAD_FIELDS.intersection(spec.allowed_payload_fields) - - openviking_commit = get_adapter_mapping_spec(BackendType.OPENVIKING, BackendOperation.COMMIT_SESSION) - everos_ingest = get_adapter_mapping_spec(BackendType.EVEROS, BackendOperation.INGEST_TURN) - - assert openviking_commit.adapter_method == "commit_session_v2" - assert openviking_commit.result_model is BackendCommitResult - assert everos_ingest.adapter_method == "ingest_message" - assert everos_ingest.result_model is BackendWriteResult - - -def test_control_plane_persisted_payload_validator_rejects_content_and_raw_request(): - validate_control_plane_payload({"gateway_id": "gw_1", "session_id": "sess_1", "metadata": {"content_hash": "abc"}}) - validate_control_plane_persisted_payload({"gateway_id": "gw_1", "metadata": {"source_channel": "test"}}) - - for blocked_key in ("content", "raw_request", "messages"): - try: - validate_control_plane_persisted_payload({"gateway_id": "gw_1", blocked_key: "should-not-pass"}) - except ValueError as exc: - assert blocked_key in str(exc) - else: - raise AssertionError(f"{blocked_key} should be rejected") - - -def test_runtime_adapter_request_may_be_transient_but_outbox_payload_is_control_plane_only(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service(repo=repo) - runtime_payload = service._apply_safety_policy(IngestRequest(**build_ingest_payload(content="TRANSIENT_ONLY_CONTENT"))) - - assert runtime_payload["content"] == "TRANSIENT_ONLY_CONTENT" - - response = asyncio.run( - service.commit_session("sess_boundary", CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli")) - ) - event = repo.list_outbox_events_by_job(response.job_id)[0] - outbox_payload = service._outbox_payload(event) - - assert "content" not in outbox_payload - assert "raw_request" not in outbox_payload - validate_control_plane_persisted_payload(outbox_payload) - - -def test_commit_adapter_skeletons_return_unified_contracts(): - payload = {"workspace_id": "ws_1", "session_id": "sess_1", "gateway_id": "gw_1"} - - ov_commit = asyncio.run(OpenVikingClient(mode="skeleton").commit_session_v2(payload)) - em_commit = EverOSClient(mode="skeleton").extract_profile_long_term_v2(payload) - - assert isinstance(ov_commit, BackendCommitResult) - assert isinstance(em_commit, BackendCommitResult) - assert ov_commit.status == BackendResultStatus.SUCCESS - assert em_commit.status == BackendResultStatus.SUCCESS - assert ov_commit.refs[0].ref_type == MemoryRefType.SESSION_ARCHIVE - assert {ref.ref_type for ref in em_commit.refs} == {MemoryRefType.PROFILE, MemoryRefType.LONG_TERM_MEMORY} - - -def test_client_skeletons_use_normalization_contracts_and_safe_metadata(): - payload = { - "workspace_id": "ws_1", - "user_id": "user_a", - "session_id": "sess_contract", - "turn_id": "turn_contract", - "content": "TRANSIENT_CONTENT_ONLY", - "raw_request": {"content": "TRANSIENT_CONTENT_ONLY"}, - } - ov_client = OpenVikingClient(mode="skeleton") - em_client = EverOSClient(mode="skeleton") - - ov_ingest = asyncio.run(ov_client.ingest_conversation_turn(payload)) - ov_commit = asyncio.run(ov_client.commit_session_v2(payload)) - em_ingest = em_client.ingest_message(payload) - em_commit = em_client.extract_profile_long_term_v2(payload) - - assert isinstance(ov_ingest, BackendWriteResult) - assert isinstance(em_ingest, BackendWriteResult) - assert isinstance(ov_commit, BackendCommitResult) - assert isinstance(em_commit, BackendCommitResult) - assert ov_ingest == ov_client._normalize_ingest_response( - { - "status": "skipped", - "session_id": "sess_contract", - "uri": "viking://sessions/sess_contract", - "metadata": { - "reason": "openviking_v2_ingest_adapter_not_configured", - "schema_version": "openviking.fixture.ingest.v2", - }, - } - ) - assert em_ingest == em_client._normalize_ingest_response( - { - "status": "skipped", - "memory_id": "turn_contract", - "metadata": { - "reason": "everos_v2_ingest_adapter_not_configured", - "schema_version": "everos.fixture.ingest.v2", - }, - } - ) - serialized = json.dumps( - { - "ov_ingest": ov_ingest.model_dump(mode="json"), - "ov_commit": ov_commit.model_dump(mode="json"), - "em_ingest": em_ingest.model_dump(mode="json"), - "em_commit": em_commit.model_dump(mode="json"), - }, - ensure_ascii=False, - ) - for blocked in ("TRANSIENT_CONTENT_ONLY", "content", "raw_request", "messages", "conversation", "transcript"): - assert blocked not in serialized - - -def test_openviking_commit_skeleton_ref_type_is_mapped_from_fixture(): - result = asyncio.run(OpenVikingClient(mode="skeleton").commit_session_v2({"session_id": "sess_ov_map"})) - - assert result.refs - assert result.refs[0].ref_type == MemoryRefType.SESSION_ARCHIVE - assert result.refs[0].native_id == "ov_session_summary:sess_ov_map" - - -def test_everos_skeleton_multiple_refs_are_written_by_process_outbox_event(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_commit_openviking_factory( - commit_result(BackendType.OPENVIKING, BackendResultStatus.SKIPPED) - ), - everos_client=EverOSClient(), - ) - response = asyncio.run( - service.commit_session("sess_em_skeleton", CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli")) - ) - event = next(event for event in repo.list_outbox_events_by_job(response.job_id) if event.backend_type == BackendType.EVEROS) - - updated = asyncio.run(service.process_outbox_event(event.id)) - refs = repo.list_memory_refs(session_id="sess_em_skeleton", backend_type=BackendType.EVEROS, status=BackendRefStatus.SUCCESS) - - assert updated.status == OutboxEventStatus.SUCCESS - assert len(refs) == 2 - assert {ref.ref_type for ref in refs} == {MemoryRefType.PROFILE, MemoryRefType.LONG_TERM_MEMORY} - - -def test_obsidian_review_adapter_skeleton_returns_skipped_write_result(): - result = ObsidianReviewClient().create_review_draft_v2({"event_id": "evt_review"}) - - assert isinstance(result, BackendWriteResult) - assert result.backend_type == BackendType.OBSIDIAN - assert result.operation == BackendOperation.CREATE_REVIEW_DRAFT - assert result.status == BackendResultStatus.SKIPPED - - -def test_backend_commit_result_supports_multiple_produced_refs(): - result = BackendCommitResult( - backend_type=BackendType.EVEROS, - status=BackendResultStatus.SUCCESS, - refs=[ - BackendProducedRef(ref_type=MemoryRefType.PROFILE, native_id="profile_1"), - BackendProducedRef(ref_type=MemoryRefType.LONG_TERM_MEMORY, native_uri="everos://memories/long_1"), - ], - ) - - dumped = result.model_dump(mode="json") - assert len(result.refs) == 2 - assert dumped["refs"][0]["ref_type"] == "profile" - assert dumped["refs"][1]["native_uri"] == "everos://memories/long_1" - - -def test_backend_ref_type_mapping_and_unknown_fallback_preserves_original_type(): - mapped, metadata = map_backend_ref_type(BackendType.OPENVIKING, "context_resource") - assert mapped == MemoryRefType.CONTEXT_RESOURCE - assert metadata == {} - - mapped, metadata = map_backend_ref_type(BackendType.OPENVIKING, "session_summary") - assert mapped == MemoryRefType.SESSION_ARCHIVE - assert metadata == {} - - mapped, metadata = map_backend_ref_type(BackendType.EVEROS, "preference") - assert mapped == MemoryRefType.PROFILE - assert metadata == {} - - mapped, metadata = map_backend_ref_type(BackendType.EVEROS, "unknown_signal") - assert mapped == MemoryRefType.LONG_TERM_MEMORY - assert metadata["original_ref_type"] == "unknown_signal" - - -def test_openviking_commit_fixture_normalizes_to_backend_commit_result_without_unsafe_metadata(): - raw = { - "status": "ok", - "session_id": "sess_norm", - "latency_ms": 18, - "metadata": {"backend_request_id": "ov_req_1", "content": "SECRET", "raw_request": {"content": "SECRET"}}, - "result": { - "refs": [ - { - "type": "session_archive", - "id": "ov_archive_1", - "uri": "viking://sessions/sess_norm", - "metadata": {"schema_version": "ov.v1", "messages": ["SECRET"]}, - } - ] - }, - } - - result = normalize_openviking_commit_response(raw) - - assert result.status == BackendResultStatus.SUCCESS - assert result.backend_type == BackendType.OPENVIKING - assert len(result.refs) == 1 - assert result.refs[0].ref_type == MemoryRefType.SESSION_ARCHIVE - assert result.refs[0].native_id == "ov_archive_1" - serialized = json.dumps(result.model_dump(mode="json"), ensure_ascii=False) - assert "SECRET" not in serialized - assert "raw_request" not in serialized - assert "messages" not in serialized - - -def test_openviking_success_fixtures_normalize_without_unsafe_metadata(): - ingest = normalize_openviking_ingest_response(backend_response("openviking_ingest_success.json")) - commit = normalize_openviking_commit_response(backend_response("openviking_commit_success.json")) - retrieve = normalize_openviking_retrieve_response(backend_response("openviking_retrieve_success.json")) - - assert ingest.status == BackendResultStatus.SUCCESS - assert ingest.native_id == "ov_turn_fixture_1" - assert commit.status == BackendResultStatus.SUCCESS - assert {ref.ref_type for ref in commit.refs} == {MemoryRefType.SESSION_ARCHIVE, MemoryRefType.CONTEXT_RESOURCE} - assert retrieve.status == BackendResultStatus.SUCCESS - assert len(retrieve.items) == 2 - assert retrieve.items[0].source_backend == BackendType.OPENVIKING - serialized = json.dumps( - { - "ingest": ingest.model_dump(mode="json"), - "commit": commit.model_dump(mode="json"), - "retrieve": retrieve.model_dump(mode="json"), - }, - ensure_ascii=False, - ) - for blocked in ("content", "raw_request", "messages", "conversation", "transcript"): - assert blocked not in serialized - - -def test_everos_commit_fixture_normalizes_multiple_produced_refs_and_unknown_type(): - raw = { - "status": "success", - "data": { - "produced_refs": [ - {"ref_type": "episodic_memory", "memory_id": "episode_1", "metadata": {"confidence": 0.82}}, - {"ref_type": "profile", "profile_id": "profile_1", "metadata": {"content": "SECRET_PROFILE"}}, - {"ref_type": "unknown_kind", "id": "long_1", "metadata": {"score": 0.9}}, - ] - }, - } - - result = normalize_everos_commit_response(raw) - - assert result.status == BackendResultStatus.SUCCESS - assert len(result.refs) == 3 - assert [ref.ref_type for ref in result.refs] == [ - MemoryRefType.EPISODIC_MEMORY, - MemoryRefType.PROFILE, - MemoryRefType.LONG_TERM_MEMORY, - ] - assert result.refs[2].metadata["original_ref_type"] == "unknown_kind" - assert "SECRET_PROFILE" not in json.dumps(result.model_dump(mode="json"), ensure_ascii=False) - - -def test_everos_success_fixtures_normalize_without_unsafe_metadata(): - ingest = normalize_everos_ingest_response(backend_response("everos_ingest_success.json")) - commit = normalize_everos_commit_response(backend_response("everos_commit_success_multiple_refs.json")) - retrieve = normalize_everos_retrieve_response(backend_response("everos_retrieve_success.json")) - - assert ingest.status == BackendResultStatus.SUCCESS - assert ingest.native_id == "em_memory_fixture_1" - assert commit.status == BackendResultStatus.SUCCESS - assert {ref.ref_type for ref in commit.refs} == { - MemoryRefType.EPISODIC_MEMORY, - MemoryRefType.PROFILE, - MemoryRefType.LONG_TERM_MEMORY, - } - assert retrieve.status == BackendResultStatus.SUCCESS - assert len(retrieve.items) == 2 - assert retrieve.items[0].source_backend == BackendType.EVEROS - assert retrieve.items[0].memory_type == "episodic_memory" - serialized = json.dumps( - { - "ingest": ingest.model_dump(mode="json"), - "commit": commit.model_dump(mode="json"), - "retrieve": retrieve.model_dump(mode="json"), - }, - ensure_ascii=False, - ) - for blocked in ("content", "raw_request", "messages", "conversation", "transcript"): - assert blocked not in serialized - - -def test_malformed_retrieve_response_returns_skipped_empty_result(): - ov = normalize_openviking_retrieve_response({}) - em = normalize_everos_retrieve_response({"data": {"unexpected": "shape"}}) - - assert ov.status == BackendResultStatus.SKIPPED - assert ov.items == [] - assert em.status == BackendResultStatus.SUCCESS - assert em.items == [] - - -def test_ingest_response_normalizers_return_write_results_and_sanitize_metadata(): - ov = normalize_openviking_ingest_response( - { - "status": "created", - "id": "ov_turn_1", - "uri": "viking://sessions/sess/turn", - "metadata": {"backend_request_id": "ov_req", "conversation": "SECRET"}, - } - ) - em = normalize_everos_ingest_response( - { - "status": "success", - "memory_id": "em_turn_1", - "metadata": {"trace_id": "trace_1", "transcript": "SECRET"}, - } - ) - - assert isinstance(ov, BackendWriteResult) - assert isinstance(em, BackendWriteResult) - assert ov.native_id == "ov_turn_1" - assert em.native_id == "em_turn_1" - serialized = json.dumps({"ov": ov.model_dump(mode="json"), "em": em.model_dump(mode="json")}, ensure_ascii=False) - assert "SECRET" not in serialized - assert "conversation" not in serialized - assert "transcript" not in serialized - - -def test_backend_error_retryable_mapping(): - for status_code in (429, 500, 502, 503, 504): - assert map_backend_error_to_retryable(BackendType.OPENVIKING, status_code=status_code) is True - assert map_backend_error_to_retryable(BackendType.EVEROS, error_code="timeout") is True - assert map_backend_error_to_retryable(BackendType.EVEROS, error_message="network_error: reset") is True - assert map_backend_error_to_retryable(BackendType.OPENVIKING, error_code="mystery") is True - - for status_code in (400, 401, 403, 404, 422): - assert map_backend_error_to_retryable(BackendType.EVEROS, status_code=status_code) is False - - -def test_client_map_error_contracts_for_future_http_integration(): - class ResponseLike: - def __init__(self, status_code): - self.status_code = status_code - - def __str__(self): - return f"response {self.status_code}" - - ov_client = OpenVikingClient(mode="skeleton") - em_client = EverOSClient(mode="skeleton") - - for status_code in (429, 500, 502, 503, 504): - assert ov_client._map_error(ResponseLike(status_code)) is True - assert em_client._map_error(ResponseLike(status_code)) is True - for status_code in (400, 401, 403, 404, 422): - assert ov_client._map_error(ResponseLike(status_code)) is False - assert em_client._map_error(ResponseLike(status_code)) is False - - assert ov_client._map_error(TimeoutError("timeout while reading")) is True - assert em_client._map_error(ConnectionError("network_error connection reset")) is True - assert ov_client._map_error(RuntimeError("unknown backend failure")) is True - - -def test_v2_ingest_schema_constructs(): - request = IngestRequest(**build_ingest_payload()) - - assert request.workspace_id == "ws_1" - assert request.request_id == "req_1" - assert request.policy.allow_openviking is True - - -def test_ingest_service_records_two_success_refs(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_openviking_factory, - everos_client=FakeEverOSClient(), - ) - - response = asyncio.run(service.ingest_conversation_turn(IngestRequest(**build_ingest_payload()))) - - assert response.status == "success" - assert len(response.refs) == 2 - assert {ref.backend_type.value for ref in response.refs} == {"openviking", "everos"} - assert {ref.status for ref in repo.list_memory_refs()} == {BackendRefStatus.SUCCESS} - assert len(repo.list_memory_refs(backend_type="openviking", status=BackendRefStatus.SUCCESS)) == 1 - - -def test_v2_ingest_service_openviking_real_mock_success_writes_safe_memory_ref(): - fixture = backend_response("openviking_ingest_real_success.json") - - def handler(request): - payload = json.loads(request.content.decode()) - assert payload["content"] == "SECRET_SERVICE_REAL_CONTENT" - assert request.headers["x-api-key"] == "ov-super-secret-token" - return httpx.Response(200, json=fixture) - - async def real_openviking_factory(): - return OpenVikingClient( - mode="real", - base_url="http://openviking.test", - api_key="ov-super-secret-token", - transport=httpx.MockTransport(handler), - ) - - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=real_openviking_factory, - everos_client=FakeEverOSClient(), - ) - - response = asyncio.run( - service.ingest_conversation_turn( - IngestRequest(**build_ingest_payload(session_id="ov_real_sess_fixture_1", content="SECRET_SERVICE_REAL_CONTENT")) - ) - ) - ov_ref = repo.list_memory_refs(backend_type=BackendType.OPENVIKING, status=BackendRefStatus.SUCCESS)[0] - audit_json = json.dumps([entry.model_dump(mode="json") for entry in repo.list_audit()], ensure_ascii=False) - - assert response.status == OperationStatus.SUCCESS - assert ov_ref.native_id == "ov_real_turn_fixture_1" - assert ov_ref.native_uri == "viking://sessions/ov_real_sess_fixture_1/turns/ov_real_turn_fixture_1" - serialized = json.dumps(ov_ref.model_dump(mode="json"), ensure_ascii=False) - assert "SECRET_SERVICE_REAL_CONTENT" not in serialized - assert "ov-super-secret-token" not in serialized - assert "raw_request" not in serialized - assert "content" not in ov_ref.metadata - assert "ov-super-secret-token" not in audit_json - - -def test_v2_ingest_service_real_mock_success_writes_openviking_and_everos_refs_safely(): - ov_fixture = backend_response("openviking_ingest_real_success.json") - em_fixture = backend_response("everos_ingest_success.json") - seen = {"openviking": 0, "everos": 0} - - def openviking_handler(request): - payload = json.loads(request.content.decode()) - assert payload["content"] == "SECRET_DUAL_REAL_CONTENT" - assert request.headers["x-api-key"] == "ov-dual-token" - seen["openviking"] += 1 - return httpx.Response(200, json=ov_fixture) - - def everos_handler(request): - payload = json.loads(request.content.decode()) - assert payload["messages"][0]["content"] == "SECRET_DUAL_REAL_CONTENT" - assert request.headers["x-api-key"] == "em-dual-token" - assert request.headers["authorization"] == "Bearer em-dual-token" - seen["everos"] += 1 - return httpx.Response(200, json=em_fixture) - - async def real_openviking_factory(): - return OpenVikingClient( - mode="real", - base_url="http://openviking.test", - api_key="ov-dual-token", - transport=httpx.MockTransport(openviking_handler), - ) - - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=real_openviking_factory, - everos_client=EverOSClient( - mode="real", - base_url="http://everos.test", - api_key="em-dual-token", - transport=httpx.MockTransport(everos_handler), - ), - ) - - response = asyncio.run( - service.ingest_conversation_turn( - IngestRequest( - **build_ingest_payload( - session_id="ov_real_sess_fixture_1", - source_type="cli", - content="SECRET_DUAL_REAL_CONTENT", - trace={"trace_id": "trace_dual_real", "request_id": "trace_req_dual"}, - ) - ) - ) - ) - - refs = repo.list_memory_refs() - serialized_refs = json.dumps([ref.model_dump(mode="json") for ref in refs], ensure_ascii=False) - audit_json = json.dumps([entry.model_dump(mode="json") for entry in repo.list_audit()], ensure_ascii=False) - - assert response.status == OperationStatus.SUCCESS - assert seen == {"openviking": 1, "everos": 1} - assert {ref.backend_type for ref in refs} == {BackendType.OPENVIKING, BackendType.EVEROS} - assert {ref.status for ref in refs} == {BackendRefStatus.SUCCESS} - assert {ref.content_hash for ref in refs} - assert "trace_dual_real" in serialized_refs - for blocked in ("SECRET_DUAL_REAL_CONTENT", "ov-dual-token", "em-dual-token", "raw_request", "messages", "conversation", "transcript"): - assert blocked not in serialized_refs - for blocked in ("SECRET_DUAL_REAL_CONTENT", "ov-dual-token", "em-dual-token", "raw_request", "messages", "transcript"): - assert blocked not in audit_json - - -def test_ingest_service_backend_failure_is_partial_success(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_openviking_factory, - everos_client=FailingEverOSClient(), - ) - - response = asyncio.run(service.ingest_conversation_turn(IngestRequest(**build_ingest_payload()))) - - assert response.status == "partial_success" - assert len(response.refs) == 2 - failed = [ref for ref in response.refs if ref.status == BackendRefStatus.FAILED] - assert len(failed) == 1 - assert failed[0].backend_type.value == "everos" - assert "everos unavailable" in failed[0].error_message - - -def test_ingest_service_records_two_skipped_refs_when_policy_disables_backends(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_openviking_factory, - everos_client=FakeEverOSClient(), - ) - - response = asyncio.run( - service.ingest_conversation_turn( - IngestRequest( - **build_ingest_payload( - policy={ - "allow_openviking": False, - "allow_everos": False, - } - ) - ) - ) - ) - - assert response.status == "skipped" - assert len(response.refs) == 2 - assert {ref.status for ref in response.refs} == {BackendRefStatus.SKIPPED} - assert len(repo.list_memory_refs()) == 2 - - -def test_duplicate_idempotency_key_upserts_memory_refs_without_duplicates(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_openviking_factory, - everos_client=FakeEverOSClient(), - ) - - first = asyncio.run( - service.ingest_conversation_turn( - IngestRequest(**build_ingest_payload(idempotency_key="idem_1", request_id="req_1")) - ) - ) - second = asyncio.run( - service.ingest_conversation_turn( - IngestRequest( - **build_ingest_payload( - idempotency_key="idem_1", - request_id="req_2", - source_event_id="evt_changed", - turn_id="turn_changed", - ) - ) - ) - ) - - refs = repo.list_memory_refs() - assert len(refs) == 2 - assert {ref.id for ref in first.refs} == {ref.id for ref in second.refs} - assert first.gateway_id == second.gateway_id - - -def test_memory_ref_metadata_does_not_store_conversation_content_or_raw_request(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_openviking_factory, - everos_client=FakeEverOSClient(), - ) - sensitive_content = "SECRET_CONVERSATION_CONTENT_SHOULD_NOT_BE_STORED" - - asyncio.run( - service.ingest_conversation_turn( - IngestRequest( - **build_ingest_payload( - content=sensitive_content, - metadata={"channel": "cli", "raw_request": {"content": sensitive_content}}, - ) - ) - ) - ) - - for ref in repo.list_memory_refs(): - metadata_json = json.dumps(ref.metadata, ensure_ascii=False) - assert sensitive_content not in metadata_json - assert "raw_request" not in metadata_json - assert ref.content_hash - assert ref.content_hash in metadata_json - - -def test_sqlite_repository_persists_v2_memory_refs(tmp_path): - repo = SQLiteRepository(tmp_path / "memory_gateway.sqlite3") - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_openviking_factory, - everos_client=FakeEverOSClient(), - ) - - asyncio.run(service.ingest_conversation_turn(IngestRequest(**build_ingest_payload(turn_id="turn_sqlite")))) - - reloaded = SQLiteRepository(tmp_path / "memory_gateway.sqlite3") - refs = reloaded.list_memory_refs( - workspace_id="ws_1", - backend_type="openviking", - status=BackendRefStatus.SUCCESS, - ) - assert len(refs) == 1 - assert refs[0].turn_id == "turn_sqlite" - - -def test_commit_session_creates_commit_job_and_outbox_events(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service(repo=repo) - - response = asyncio.run( - service.commit_session( - "sess_commit", - CommitRequest( - workspace_id="ws_1", - user_id="user_a", - agent_id="agent_cli", - namespace="workspace/ws_1/user/user_a", - request_id="commit_req_1", - ), - ) - ) - - job = repo.get_commit_job(response.job_id) - events = repo.list_outbox_events(gateway_id=response.metadata["gateway_id"]) - - assert response.status == "accepted" - assert job is not None - assert job.session_id == "sess_commit" - assert job.status.value == "accepted" - assert len(events) == 2 - assert {event.backend_type for event in events} == {BackendType.OPENVIKING, BackendType.EVEROS} - assert {event.operation for event in events} == {BackendOperation.COMMIT_SESSION} - assert {event.status for event in events} == {OutboxEventStatus.PENDING} - - -def test_sqlite_repository_persists_commit_job_and_outbox_events(tmp_path): - repo = SQLiteRepository(tmp_path / "memory_gateway.sqlite3") - service = MemoryGatewayV2Service(repo=repo) - - response = asyncio.run( - service.commit_session( - "sess_commit_sqlite", - CommitRequest( - workspace_id="ws_1", - user_id="user_a", - agent_id="agent_cli", - namespace="workspace/ws_1/user/user_a", - idempotency_key="commit_idem_1", - ), - ) - ) - - reloaded = SQLiteRepository(tmp_path / "memory_gateway.sqlite3") - job = reloaded.get_commit_job(response.job_id) - events = reloaded.list_outbox_events(gateway_id=response.metadata["gateway_id"]) - - assert job is not None - assert job.session_id == "sess_commit_sqlite" - assert len(events) == 2 - assert {event.payload_ref for event in events} == {f"commit_job:{response.job_id}"} - - -def test_sqlite_repository_claims_due_outbox_with_lease_fields(tmp_path): - repo = SQLiteRepository(tmp_path / "memory_gateway.sqlite3") - service = MemoryGatewayV2Service(repo=repo) - response = asyncio.run( - service.commit_session( - "sess_sqlite_claim", - CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli"), - ) - ) - - claimed = repo.claim_pending_outbox_events(limit=1, worker_id="sqlite_worker", lease_seconds=30) - reloaded = SQLiteRepository(tmp_path / "memory_gateway.sqlite3") - events = reloaded.list_outbox_events_by_job(response.job_id) - - assert len(claimed) == 1 - assert sum(1 for event in events if event.status == OutboxEventStatus.PROCESSING) == 1 - claimed_event = next(event for event in events if event.status == OutboxEventStatus.PROCESSING) - assert claimed_event.locked_by == "sqlite_worker" - assert claimed_event.lease_expires_at is not None - - -def test_outbox_event_does_not_store_conversation_content_or_raw_request(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service(repo=repo) - sensitive_content = "SECRET_COMMIT_CONTENT_SHOULD_NOT_BE_STORED" - - response = asyncio.run( - service.commit_session( - "sess_commit", - CommitRequest( - workspace_id="ws_1", - user_id="user_a", - agent_id="agent_cli", - namespace="workspace/ws_1/user/user_a", - metadata={"raw_request": {"content": sensitive_content}}, - ), - ) - ) - - for event in repo.list_outbox_events(gateway_id=response.metadata["gateway_id"]): - event_json = json.dumps(event.model_dump(mode="json"), ensure_ascii=False) - assert sensitive_content not in event_json - assert "raw_request" not in event_json - assert event.payload_ref == f"commit_job:{response.job_id}" - - -def test_retrieve_response_contract_contains_items_refs_conflicts_trace_id_status(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_openviking_factory, - everos_client=FakeEverOSClient(), - ) - asyncio.run(service.ingest_conversation_turn(IngestRequest(**build_ingest_payload()))) - - response = asyncio.run( - service.retrieve_context( - RetrieveRequest( - workspace_id="ws_1", - user_id="user_a", - agent_id="agent_cli", - session_id="sess_1", - query="remember", - metadata={"trace_id": "trace_1"}, - ) - ) - ) - - dumped = response.model_dump() - assert set(["items", "refs", "conflicts", "trace_id", "status"]).issubset(dumped) - assert response.trace_id == "trace_1" - assert response.status.value == "success" - assert [item.source_backend for item in response.items] == [BackendType.EVEROS, BackendType.OPENVIKING] - assert [item.score for item in response.items] == [0.91, 0.82] - assert len(response.refs) == 2 - assert response.conflicts == [] - assert response.metadata["backend_results"] == [ - {"backend_type": "openviking", "status": "success", "items": 1, "error_code": None, "error_message": None}, - {"backend_type": "everos", "status": "success", "items": 1, "error_code": None, "error_message": None}, - ] - - -def test_process_commit_job_success_updates_job_and_writes_memory_refs(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_commit_openviking_factory( - commit_result(BackendType.OPENVIKING, BackendResultStatus.SUCCESS, native_id="ov_commit_1", native_uri="viking://sessions/sess_commit") - ), - everos_client=FakeCommitEverOSClient( - commit_result(BackendType.EVEROS, BackendResultStatus.SUCCESS, native_id="em_commit_1", native_uri="everos://memories/em_commit_1") - ), - ) - response = asyncio.run( - service.commit_session( - "sess_commit", - CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli", namespace="workspace/ws_1/user/user_a"), - ) - ) - - job = asyncio.run(service.process_commit_job(response.job_id)) - events = repo.list_outbox_events_by_job(response.job_id) - refs = repo.list_memory_refs(session_id="sess_commit", status=BackendRefStatus.SUCCESS) - - assert job.status.value == "success" - assert job.started_at is not None - assert job.finished_at is not None - assert job.created_refs_count == 2 - assert {event.status for event in events} == {OutboxEventStatus.SUCCESS} - assert len(refs) == 2 - assert {ref.backend_type for ref in refs} == {BackendType.OPENVIKING, BackendType.EVEROS} - - -def test_process_outbox_event_writes_multiple_produced_memory_refs(): - repo = InMemoryRepository() - sensitive_content = "SECRET_PRODUCED_REF_CONTENT" - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_commit_openviking_factory( - BackendCommitResult( - backend_type=BackendType.OPENVIKING, - operation=BackendOperation.COMMIT_SESSION, - status=BackendResultStatus.SUCCESS, - refs=[ - BackendProducedRef( - ref_type=MemoryRefType.SESSION_ARCHIVE, - native_id="ov_session_archive_1", - native_uri="viking://sessions/sess_multi", - metadata={"backend_request_id": "req_ov_1", "content": sensitive_content}, - ), - BackendProducedRef( - ref_type=MemoryRefType.PROFILE, - native_id="ov_profile_1", - metadata={"source_channel": "worker", "raw_request": {"content": sensitive_content}}, - ), - ], - metadata={"latency_ms": 12, "messages": [sensitive_content]}, - ) - ), - ) - response = asyncio.run( - service.commit_session( - "sess_multi", - CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli", namespace="workspace/ws_1/user/user_a"), - ) - ) - event = next(event for event in repo.list_outbox_events_by_job(response.job_id) if event.backend_type == BackendType.OPENVIKING) - - updated = asyncio.run(service.process_outbox_event(event.id)) - refs = repo.list_memory_refs(session_id="sess_multi", backend_type=BackendType.OPENVIKING, status=BackendRefStatus.SUCCESS) - - assert updated.status == OutboxEventStatus.SUCCESS - assert len(refs) == 2 - assert {ref.ref_type for ref in refs} == {MemoryRefType.SESSION_ARCHIVE, MemoryRefType.PROFILE} - assert {ref.native_id for ref in refs} == {"ov_session_archive_1", "ov_profile_1"} - for ref in refs: - serialized = json.dumps(ref.model_dump(mode="json"), ensure_ascii=False) - assert sensitive_content not in serialized - assert "raw_request" not in serialized - assert "messages" not in serialized - assert "conversation" not in serialized - assert "transcript" not in serialized - - -def test_process_outbox_event_writes_same_ref_type_with_different_native_ids(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_commit_openviking_factory( - BackendCommitResult( - backend_type=BackendType.OPENVIKING, - operation=BackendOperation.COMMIT_SESSION, - status=BackendResultStatus.SUCCESS, - refs=[ - BackendProducedRef(ref_type=MemoryRefType.CONTEXT_RESOURCE, native_id="resource_1"), - BackendProducedRef(ref_type=MemoryRefType.CONTEXT_RESOURCE, native_id="resource_2"), - ], - ) - ), - ) - response = asyncio.run( - service.commit_session("sess_same_type", CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli")) - ) - event = next(event for event in repo.list_outbox_events_by_job(response.job_id) if event.backend_type == BackendType.OPENVIKING) - - asyncio.run(service.process_outbox_event(event.id)) - refs = repo.list_memory_refs(session_id="sess_same_type", backend_type=BackendType.OPENVIKING, ref_type=MemoryRefType.CONTEXT_RESOURCE) - - assert len(refs) == 2 - assert {ref.native_id for ref in refs} == {"resource_1", "resource_2"} - assert len({ref.id for ref in refs}) == 2 - - -def test_memory_ref_id_uses_stable_fallback_when_native_ref_is_missing(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_commit_openviking_factory( - BackendCommitResult( - backend_type=BackendType.OPENVIKING, - operation=BackendOperation.COMMIT_SESSION, - status=BackendResultStatus.SUCCESS, - refs=[ - BackendProducedRef(ref_type=MemoryRefType.SESSION_ARCHIVE, metadata={"stable_key": "summary_a"}), - BackendProducedRef(ref_type=MemoryRefType.SESSION_ARCHIVE, metadata={"stable_key": "summary_b"}), - ], - ) - ), - ) - response = asyncio.run( - service.commit_session("sess_stable_key", CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli")) - ) - event = next(event for event in repo.list_outbox_events_by_job(response.job_id) if event.backend_type == BackendType.OPENVIKING) - - asyncio.run(service.process_outbox_event(event.id)) - refs = repo.list_memory_refs(session_id="sess_stable_key", backend_type=BackendType.OPENVIKING, ref_type=MemoryRefType.SESSION_ARCHIVE) - - assert len(refs) == 2 - assert len({ref.id for ref in refs}) == 2 - assert {ref.metadata["stable_key"] for ref in refs} == {"summary_a", "summary_b"} - - -def test_process_outbox_event_keeps_single_native_ref_fallback_compatible(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_commit_openviking_factory( - commit_result(BackendType.OPENVIKING, BackendResultStatus.SUCCESS, native_id="ov_single", native_uri="viking://sessions/ov_single") - ), - ) - response = asyncio.run( - service.commit_session("sess_single_fallback", CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli")) - ) - event = next(event for event in repo.list_outbox_events_by_job(response.job_id) if event.backend_type == BackendType.OPENVIKING) - - asyncio.run(service.process_outbox_event(event.id)) - refs = repo.list_memory_refs(session_id="sess_single_fallback", backend_type=BackendType.OPENVIKING, status=BackendRefStatus.SUCCESS) - - assert len(refs) == 1 - assert refs[0].ref_type == MemoryRefType.SESSION_ARCHIVE - assert refs[0].native_id == "ov_single" - - -def test_process_commit_job_one_success_one_failed_is_partial_success(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_commit_openviking_factory( - commit_result(BackendType.OPENVIKING, BackendResultStatus.SUCCESS, native_id="ov_commit_1") - ), - everos_client=FakeCommitEverOSClient( - commit_result(BackendType.EVEROS, BackendResultStatus.FAILED, retryable=False, error_message="everos failed") - ), - ) - response = asyncio.run( - service.commit_session("sess_partial", CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli")) - ) - - job = asyncio.run(service.process_commit_job(response.job_id)) - events = repo.list_outbox_events_by_job(response.job_id) - - assert job.status.value == "partial_success" - assert job.created_refs_count == 1 - assert "everos failed" in job.error_message - assert {event.status for event in events} == {OutboxEventStatus.SUCCESS, OutboxEventStatus.DEAD_LETTER} - - -def test_process_commit_job_two_failed_is_failed(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_commit_openviking_factory( - commit_result(BackendType.OPENVIKING, BackendResultStatus.FAILED, retryable=False, error_message="openviking failed") - ), - everos_client=FakeCommitEverOSClient( - commit_result(BackendType.EVEROS, BackendResultStatus.FAILED, retryable=False, error_message="everos failed") - ), - ) - response = asyncio.run( - service.commit_session("sess_failed", CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli")) - ) - - job = asyncio.run(service.process_commit_job(response.job_id)) - - assert job.status.value == "failed" - assert job.created_refs_count == 0 - assert "openviking failed" in job.error_message - assert "everos failed" in job.error_message - - -def test_retryable_failed_outbox_event_requeues_with_next_retry(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_commit_openviking_factory( - commit_result(BackendType.OPENVIKING, BackendResultStatus.FAILED, retryable=True, error_message="temporary openviking failure") - ), - ) - response = asyncio.run( - service.commit_session("sess_retry", CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli")) - ) - event = next(event for event in repo.list_outbox_events_by_job(response.job_id) if event.backend_type == BackendType.OPENVIKING) - - updated = asyncio.run(service.process_outbox_event(event.id)) - - assert updated.status == OutboxEventStatus.PENDING - assert updated.attempt_count == 1 - assert updated.next_retry_at is not None - assert "temporary openviking failure" in updated.last_error - - -def test_process_pending_outbox_events_processes_pending_batch(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_commit_openviking_factory( - commit_result(BackendType.OPENVIKING, BackendResultStatus.SUCCESS, native_id="ov_commit_1") - ), - everos_client=FakeCommitEverOSClient( - commit_result(BackendType.EVEROS, BackendResultStatus.SUCCESS, native_id="em_commit_1") - ), - ) - asyncio.run( - service.commit_session("sess_batch", CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli")) - ) - - processed = asyncio.run(service.process_pending_outbox_events()) - - assert len(processed) == 2 - assert {event.status for event in processed} == {OutboxEventStatus.SUCCESS} - assert len(repo.list_memory_refs(session_id="sess_batch", status=BackendRefStatus.SUCCESS)) == 2 - - -def test_retryable_failed_outbox_event_exceeding_max_attempts_dead_letters(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_commit_openviking_factory( - commit_result(BackendType.OPENVIKING, BackendResultStatus.FAILED, retryable=True, error_message="still failing") - ), - ) - response = asyncio.run( - service.commit_session("sess_dead", CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli")) - ) - event = next(event for event in repo.list_outbox_events_by_job(response.job_id) if event.backend_type == BackendType.OPENVIKING) - event.max_attempts = 1 - repo.save_outbox_event(event) - - updated = asyncio.run(service.process_outbox_event(event.id)) - - assert updated.status == OutboxEventStatus.DEAD_LETTER - assert updated.attempt_count == 1 - assert updated.next_retry_at is None - - -def test_commit_pipeline_metadata_does_not_store_content_or_raw_request(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_commit_openviking_factory( - commit_result(BackendType.OPENVIKING, BackendResultStatus.SUCCESS, native_id="ov_commit_1") - ), - everos_client=FakeCommitEverOSClient( - commit_result(BackendType.EVEROS, BackendResultStatus.SUCCESS, native_id="em_commit_1") - ), - ) - sensitive_content = "SECRET_COMMIT_PIPELINE_CONTENT_SHOULD_NOT_BE_STORED" - response = asyncio.run( - service.commit_session( - "sess_secure", - CommitRequest( - workspace_id="ws_1", - user_id="user_a", - agent_id="agent_cli", - metadata={"raw_request": {"content": sensitive_content}}, - ), - ) - ) - - asyncio.run(service.process_commit_job(response.job_id)) - - for event in repo.list_outbox_events_by_job(response.job_id): - assert sensitive_content not in json.dumps(event.model_dump(mode="json"), ensure_ascii=False) - assert "raw_request" not in json.dumps(event.model_dump(mode="json"), ensure_ascii=False) - for ref in repo.list_memory_refs(session_id="sess_secure"): - assert sensitive_content not in json.dumps(ref.model_dump(mode="json"), ensure_ascii=False) - assert "raw_request" not in json.dumps(ref.model_dump(mode="json"), ensure_ascii=False) - - -def test_claim_pending_outbox_events_only_claims_due_pending_events(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service(repo=repo) - response = asyncio.run( - service.commit_session("sess_claim", CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli")) - ) - events = repo.list_outbox_events_by_job(response.job_id) - delayed = events[0] - delayed.next_retry_at = datetime.now(timezone.utc) + timedelta(minutes=5) - repo.save_outbox_event(delayed) - - claimed = repo.claim_pending_outbox_events(limit=10, worker_id="worker_claim", lease_seconds=30) - - assert len(claimed) == 1 - assert claimed[0].id != delayed.id - assert claimed[0].status == OutboxEventStatus.PROCESSING - assert claimed[0].locked_by == "worker_claim" - assert claimed[0].lease_expires_at is not None - - -def test_next_retry_not_due_event_is_not_claimed(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service(repo=repo) - response = asyncio.run( - service.commit_session("sess_retry_wait", CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli")) - ) - for event in repo.list_outbox_events_by_job(response.job_id): - event.next_retry_at = datetime.now(timezone.utc) + timedelta(minutes=5) - repo.save_outbox_event(event) - - claimed = repo.claim_pending_outbox_events(limit=10, worker_id="worker_wait", lease_seconds=30) - - assert claimed == [] - assert {event.status for event in repo.list_outbox_events_by_job(response.job_id)} == {OutboxEventStatus.PENDING} - - -def test_expired_processing_event_is_released_to_pending(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service(repo=repo) - response = asyncio.run( - service.commit_session("sess_expired", CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli")) - ) - claimed = repo.claim_pending_outbox_events(limit=1, worker_id="worker_old", lease_seconds=1) - assert len(claimed) == 1 - - released = repo.release_expired_processing_events(datetime.now(timezone.utc) + timedelta(seconds=2)) - - assert len(released) == 1 - assert released[0].status == OutboxEventStatus.PENDING - assert released[0].locked_by is None - assert released[0].lease_expires_at is None - - -def test_process_pending_outbox_events_uses_claim_and_does_not_process_existing_lock(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_commit_openviking_factory( - commit_result(BackendType.OPENVIKING, BackendResultStatus.SUCCESS, native_id="ov_claimed") - ), - everos_client=FakeCommitEverOSClient( - commit_result(BackendType.EVEROS, BackendResultStatus.SUCCESS, native_id="em_claimed") - ), - ) - response = asyncio.run( - service.commit_session("sess_no_double", CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli")) - ) - externally_claimed = repo.claim_pending_outbox_events(limit=1, worker_id="worker_a", lease_seconds=300)[0] - - processed = asyncio.run(service.process_pending_outbox_events(worker_id="worker_b")) - events = repo.list_outbox_events_by_job(response.job_id) - - assert len(processed) == 1 - assert sum(1 for event in events if event.status == OutboxEventStatus.SUCCESS) == 1 - still_locked = next(event for event in events if event.id == externally_claimed.id) - assert still_locked.status == OutboxEventStatus.PROCESSING - assert still_locked.locked_by == "worker_a" - - -def test_terminal_outbox_statuses_clear_lock_fields(): - repo = InMemoryRepository() - success_service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_commit_openviking_factory( - commit_result(BackendType.OPENVIKING, BackendResultStatus.SUCCESS, native_id="ov_lock_clear") - ), - everos_client=FakeCommitEverOSClient( - commit_result(BackendType.EVEROS, BackendResultStatus.SKIPPED) - ), - ) - response = asyncio.run( - success_service.commit_session("sess_lock_clear", CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli")) - ) - - processed = asyncio.run(success_service.process_pending_outbox_events(worker_id="worker_lock")) - - assert {event.status for event in processed} == {OutboxEventStatus.SUCCESS, OutboxEventStatus.SKIPPED} - assert all(event.locked_by is None for event in processed) - assert all(event.lease_expires_at is None for event in processed) - - fail_service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_commit_openviking_factory( - commit_result(BackendType.OPENVIKING, BackendResultStatus.FAILED, retryable=False, error_message="fatal") - ), - ) - failed = asyncio.run( - fail_service.commit_session("sess_dead_lock", CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli")) - ) - event = next(event for event in repo.list_outbox_events_by_job(failed.job_id) if event.backend_type == BackendType.OPENVIKING) - updated = asyncio.run(fail_service.process_outbox_event(event.id)) - - assert updated.status == OutboxEventStatus.DEAD_LETTER - assert updated.locked_by is None - assert updated.lease_expires_at is None - assert repo.list_outbox_events_by_job(response.job_id) - - -def test_retryable_failed_outbox_event_clears_lock_when_requeued(): - repo = InMemoryRepository() - service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_commit_openviking_factory( - commit_result(BackendType.OPENVIKING, BackendResultStatus.FAILED, retryable=True, error_message="temporary") - ), - ) - response = asyncio.run( - service.commit_session("sess_retry_lock", CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli")) - ) - - updated = asyncio.run( - service.process_outbox_event( - next(event.id for event in repo.list_outbox_events_by_job(response.job_id) if event.backend_type == BackendType.OPENVIKING) - ) - ) - - assert updated.status == OutboxEventStatus.PENDING - assert updated.next_retry_at is not None - assert updated.locked_by is None - assert updated.lease_expires_at is None - - -def test_job_query_api_returns_job_status_and_outbox_summary(monkeypatch): - import memory_gateway.api_v2 as api_v2 - - repo = InMemoryRepository() - api_v2.v2_service = MemoryGatewayV2Service(repo=repo) - commit_response = asyncio.run( - api_v2.v2_service.commit_session( - "sess_job_api", - CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli", namespace="workspace/ws_1/user/user_a"), - ) - ) - app = FastAPI() - app.dependency_overrides[verify_api_key_compat] = lambda: None - app.include_router(api_v2.router) - - async def get_request(): - async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client: - return await client.get(f"/v2/jobs/{commit_response.job_id}") - - response = asyncio.run(asyncio.wait_for(get_request(), timeout=2)) - - assert response.status_code == 200 - payload = response.json() - assert payload["job_id"] == commit_response.job_id - assert payload["status"] == "accepted" - assert payload["outbox_summary"]["total_events"] == 2 - assert payload["outbox_summary"]["pending_events"] == 2 - - -def test_admin_process_outbox_endpoint_triggers_pending_processing(monkeypatch): - import memory_gateway.api_v2 as api_v2 - - repo = InMemoryRepository() - api_v2.v2_service = MemoryGatewayV2Service( - repo=repo, - openviking_client_factory=fake_commit_openviking_factory( - commit_result(BackendType.OPENVIKING, BackendResultStatus.SUCCESS, native_id="ov_admin") - ), - everos_client=FakeCommitEverOSClient( - commit_result(BackendType.EVEROS, BackendResultStatus.SUCCESS, native_id="em_admin") - ), - ) - asyncio.run( - api_v2.v2_service.commit_session( - "sess_admin", - CommitRequest(workspace_id="ws_1", user_id="user_a", agent_id="agent_cli"), - ) - ) - app = FastAPI() - app.dependency_overrides[verify_api_key_compat] = lambda: None - app.include_router(api_v2.router) - - async def post_request(): - async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client: - return await client.post("/v2/admin/outbox/process?limit=10&worker_id=test_worker") - - response = asyncio.run(asyncio.wait_for(post_request(), timeout=2)) - - assert response.status_code == 200 - payload = response.json() - assert payload["worker_id"] == "test_worker" - assert payload["processed_count"] == 2 - assert payload["outbox_summary"]["success_events"] == 2 - - -def test_worker_v2_cli_processes_once_and_prints_control_plane_summary(monkeypatch, capsys): - import memory_gateway.worker_v2 as worker_v2 - - class FakeWorkerService: - async def process_pending_outbox_events_summary(self, limit: int, worker_id: str, lease_seconds: int): - assert limit == 7 - assert worker_id == "cli_worker" - assert lease_seconds == 45 - return OutboxProcessResponse( - status=OperationStatus.SUCCESS, - worker_id=worker_id, - processed_count=2, - ) - - monkeypatch.setattr(worker_v2, "v2_service", FakeWorkerService()) - - exit_code = worker_v2.main(["--limit", "7", "--worker-id", "cli_worker", "--lease-seconds", "45"]) - - assert exit_code == 0 - payload = json.loads(capsys.readouterr().out) - assert payload["worker_id"] == "cli_worker" - assert payload["processed_count"] == 2 - assert "content" not in json.dumps(payload) - assert "raw_request" not in json.dumps(payload) - - -def test_v2_ingest_router_accepts_legal_request(monkeypatch): - import memory_gateway.api_v2 as api_v2 - - api_v2.v2_service = MemoryGatewayV2Service( - repo=InMemoryRepository(), - openviking_client_factory=fake_openviking_factory, - everos_client=FakeEverOSClient(), - ) - app = FastAPI() - app.dependency_overrides[verify_api_key_compat] = lambda: None - app.include_router(api_v2.router) - - async def post_request(): - async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client: - return await client.post("/v2/conversations/ingest", json=build_ingest_payload(turn_id="turn_router")) - - response = asyncio.run(asyncio.wait_for(post_request(), timeout=2)) - - assert response.status_code == 200 - payload = response.json() - assert payload["turn_id"] == "turn_router" - assert len(payload["refs"]) == 2 diff --git a/uvicorn.yaml b/uvicorn.yaml deleted file mode 100644 index b8935e2..0000000 --- a/uvicorn.yaml +++ /dev/null @@ -1,4 +0,0 @@ -host: "0.0.0.0" -port: 1934 -reload: true -log_level: "info"