memory-gateway/eval/hermes_memory_eval/README.md

Hermes Memory Evaluation

This is a small LoCoMo-style memory evaluation runner for Hermes Agent. It follows the same shape as openclaw-eval: ingest historical conversations, ask QA questions with the same user id, then use an LLM judge to score the answers.

1. Configure Hermes Memory

Install or copy the memory_system Hermes plugin, then put Memory System settings in /home/tom/.hermes/memory_system.env:

MEMORY_SYSTEM_ENDPOINT=http://127.0.0.1:1934
MEMORY_SYSTEM_USER_ID=default
MEMORY_SYSTEM_SEARCH_USE_LLM=false
MEMORY_SYSTEM_COMMIT_EVERY_TURNS=1
MEMORY_SYSTEM_COMMIT_INTERVAL_SECONDS=0

The eval runner overrides MEMORY_SYSTEM_USER_ID per LoCoMo sample, so one sample maps to one memory user.

2. Prepare Config

Copy and edit:

cp eval/hermes_memory_eval/config.example.yaml eval/hermes_memory_eval/config.yaml

For a stable eval, keep:

memory:
  commit_every_turns: 1
  commit_interval_seconds: 0

3. Ingest Conversations

Before ingest, verify the eval Hermes home can see the plugin:

HERMES_HOME=/home/tom/memory-gateway/eval/hermes_memory_eval/hermes_home hermes memory status

The status must show memory_system as installed and active.

Run a small smoke test first:

python eval/hermes_memory_eval/run_eval.py ingest /path/to/locomo10_small.json \
  --config eval/hermes_memory_eval/config.yaml \
  --sample 0 \
  --sessions 1-2 \
  --output output/hermes_ingest.jsonl

This sends each selected session to:

hermes chat -Q --source memory-eval -q "<formatted session>"

4. Ask QA Questions

Use the same sample and user mapping:

python eval/hermes_memory_eval/run_eval.py qa /path/to/locomo10_small.json \
  --config eval/hermes_memory_eval/config.yaml \
  --sample 0 \
  --count 10 \
  --output output/hermes_qa.jsonl

Each QA runs in a fresh Hermes CLI call, so the answer should come from persistent memory rather than the prior short-term chat context. The default QA prompt explicitly asks Hermes to call memory_system_search before answering. If Memory System API does not log POST /memory-system/search, inspect the session JSON to confirm whether the model made a tool call.

5. Judge Answers

Use the judge section in config.yaml:

judge:
  base_url: "https://api.openai.com/v1"
  api_key_env: "OPENAI_API_KEY"
  model: "gpt-4o-mini"
  parallel: 4
  timeout_seconds: 120

Then run:

OPENAI_API_KEY=sk-... python eval/hermes_memory_eval/judge.py output/hermes_qa.jsonl \
  --config eval/hermes_memory_eval/config.yaml \
  --output output/hermes_grades.json

For Ark/Doubao-style endpoints:

judge:
  base_url: "https://ark.cn-beijing.volces.com/api/v3"
  api_key_env: "ARK_API_KEY"
  model: "doubao-seed-2-0-pro-260215"

ARK_API_KEY=... python eval/hermes_memory_eval/judge.py output/hermes_qa.jsonl \
  --config eval/hermes_memory_eval/config.yaml \
  --output output/hermes_grades.json

Recommended Comparisons

Run the same dataset in these modes:

• no external memory
• MEMORY_SYSTEM_SEARCH_USE_LLM=false
• MEMORY_SYSTEM_SEARCH_USE_LLM=true

Compare final QA score and inspect failed examples. If search recall is high but QA accuracy is low, Hermes is not using retrieved memory well. If search recall is low, the issue is likely write/extract/search quality.

Current Small Dataset Result

On locomo10_small.json sample conv-26, the current smoke test results are:

Mode	Score	Category 1	Category 2	Category 3	Category 4
Memory System enabled	5/35 (14.29%)	0/5 (0.00%)	1/9 (11.11%)	1/2 (50.00%)	3/19 (15.79%)
No external memory	0/35 (0.00%)	0/5 (0.00%)	0/9 (0.00%)	0/2 (0.00%)	0/19 (0.00%)

This means the Memory System path is contributing signal over the no-memory baseline, but the absolute score is still low. The main follow-up is to inspect failed QA examples and separate retrieval failure from answer-use failure:

• If POST /memory-system/search does not appear during QA, Hermes did not call the memory tool.
• If search results do not contain the evidence/gold answer, the write/extract/search path needs improvement.
• If search results contain the evidence but the answer is wrong, Hermes is not using retrieved memory effectively.

For future runs, keep a fresh user_prefix per mode so OpenViking/EverOS memory from prior runs does not contaminate results.