tomtan/EverOS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: initialize EverOS 1.0.0

Browse Source 
md-first memory extraction framework for AI agents.

Markdown is the single source of truth; SQLite holds state and LanceDB
provides the rebuildable vector + BM25 + scalar index. The codebase follows
a single-direction DDD layering (entrypoints -> service -> memory -> infra,
with component / core / config cross-cutting) enforced by import-linter.

Engineering surface:
- Coding conventions in .claude/rules/ (path-scoped) and workflows in
  .claude/skills/ (/commit, /new-branch, /pr).
- GitHub Actions CI runs make lint + test + integration; pre-commit mirrors
  the gates locally (ruff, hygiene hooks, gitlint commit-msg).
- Commit messages follow Conventional Commits, enforced by gitlint.
- make lint also enforces datetime two-zone discipline and OpenAPI drift.
This commit is contained in:
Elliot Chen
2026-06-05 22:35:51 +08:00
commit 518b8eca85
636 changed files with 160553 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

553
docs/engineering.md Normal file
View File

@ -0,0 +1,553 @@
# Engineering & Dev-Efficiency Infrastructure
> Companions: business architecture lives in [architecture.md](architecture.md);
> hard coding constraints live in [../.claude/rules/](../.claude/rules/).
> This document covers the surrounding tooling, configuration, and processes
> — what we adopted, what role each piece plays, and how they fit together.
> CI runs on GitHub Actions; all checks are invoked through the `Makefile`.
---
## 1. Scope
Engineering / dev-efficiency infrastructure does not solve business problems —
it solves **team + code + time** problems:
```
┌──────────────────────────────────────────────────────────┐
│ │
│ Business architecture (docs/architecture.md) │
│ — answers "how to build the system" │
│ │
│ Engineering rules (.claude/rules/) │
│ — answers "how to write the code" │
│ │
│ Engineering / dev-efficiency infrastructure (this doc) │
│ — answers "how the team collaborates, │
│ how code is auto-checked, │
│ how releases are automated, │
│ how tools land in the project" │
│ │
└──────────────────────────────────────────────────────────┘
```
Reasons this is documented separately:
- **Cross-project reusable** — `CLAUDE.md` / rules / `pyproject.toml` are
patterns, not content. The next project can adopt them as-is.
- **Decoupled from business** — business architecture changes do not affect
these; upgrading these does not affect business.
- **Onboarding-oriented** — new contributors read this first to understand
what the tooling looks like.
---
## 2. Infrastructure overview
```
┌─────────────────────────────────────────────────────────────────────┐
│ Team collaboration / Code quality / CI/CD │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─ Claude Code engineering layer ────────────────────────────┐ │
│ │ │ │
│ │ CLAUDE.md ← team-shared context (auto loaded into │ │
│ │ system prompt) │ │
│ │ .claude/ │ │
│ │ ├── CLAUDE.md subdir context (optional) │ │
│ │ ├── rules/ (10) path-scoped hard coding rules │ │
│ │ ├── skills/ (3) slash command workflows │ │
│ │ └── settings.json permissions allowlist │ │
│ │ │ │
│ └────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─ Code quality gates ───────────────────────────────────────┐ │
│ │ │ │
│ │ pre-commit runs locally before commit │ │
│ │ ├ ruff (lint+fmt) │ │
│ │ ├ trailing-whitespace / end-of-file-fixer │ │
│ │ ├ check-yaml / check-toml │ │
│ │ ├ check-added-large-files (≥1MB warn) │ │
│ │ ├ detect-private-key │ │
│ │ └ gitlint (commit-msg stage) │ │
│ │ │ │
│ │ ruff lint + format │ │
│ │ (replaces black / isort / flake8) │ │
│ │ import-linter DDD layer-direction enforcement │ │
│ │ pytest unit / integration │ │
│ │ │ │
│ └────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─ Dependencies & build ─────────────────────────────────────┐ │
│ │ │ │
│ │ uv sole package manager │ │
│ │ (no `pip install`) │ │
│ │ pyproject.toml src layout + extras + groups │ │
│ │ uv.lock checked in; CI uses --frozen │ │
│ │ hatchling wheel build backend │ │
│ │ Makefile unified entry; CI calls it │ │
│ │ src/everos/templates/env.template │ │
│ │ environment variable template │ │
│ │ │ │
│ └────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─ CI/CD (GitHub Actions) ───────────────────────────────────┐ │
│ │ │ │
│ │ CI: .github/workflows/ci.yml lint / test / integ │ │
│ │ Docs: .github/workflows/docs.yml Markdown link check │ │
│ │ Both invoke Makefile targets; the Makefile is the │ │
│ │ single source of truth for commands. │ │
│ │ │ │
│ └────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─ Collaboration workflow ───────────────────────────────────┐ │
│ │ │ │
│ │ Branch model: dev / master (GitFlow Lite) │ │
│ │ PR template: .github/PULL_REQUEST_TEMPLATE.md │ │
│ │ ISSUE_TEMPLATE: bug / feature / use-case / docs / config │ │
│ │ CONTRIBUTING.md: contributor onboarding │ │
│ │ │ │
│ └────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
```
---
## 3. Claude Code engineering layer
### 3.1 Loading mechanism
Claude Code automatically loads the following into the system prompt at
session start (no manual import):
```
┌────────────────────────┬──────────────────────────────────────────┐
│ File │ Purpose │
├────────────────────────┼──────────────────────────────────────────┤
│ CLAUDE.md (repo root) │ Team-shared context: architecture │
│ │ overview, commands, convention index │
│ .claude/rules/*.md │ Hard coding constraints │
│ │ (path-scoped on-demand load) │
│ .claude/settings.json │ Permissions allowlist (not in prompt) │
│ ~/.claude/CLAUDE.md │ User-level (personal preferences) │
│ CLAUDE.local.md │ Project-local personal (gitignored) │
└────────────────────────┴──────────────────────────────────────────┘
```
### 3.2 Rules (10 files, path-scoped)
| File | Paths (auto-load condition) |
|---|---|
| architecture.md | always loaded (no paths) |
| code-style.md | always loaded (no paths) |
| language-policy.md | always loaded (no paths) |
| imports.md | `src/**/*.py`, `tests/**/*.py` |
| init-py-and-reexport.md | `src/**/__init__.py`, `src/**/*.py` |
| module-docstring.md | `src/{infra,memory,service,component,core}/**/*.py` |
| async-programming.md | `src/**/*.py`, `tests/**/*.py` |
| datetime-handling.md | `src/**/*.py`, `tests/**/*.py` |
| logging-observability.md | `src/**/*.py` |
| testing.md | `tests/**/*.py` |
**Why path-scoped**: avoid loading 1000+ lines of rules every session
(~5–8K tokens). At startup only architecture + code-style + language-policy
load (~1.5–2K tokens); the rest load on demand when Claude Code reads a
matching `.py` file.
### 3.3 Skills (3 slash commands)
| Command | Purpose | When to use |
|---|---|---|
| `/commit` | Generate a Conventional Commits message | After a focused change, ready to commit |
| `/new-branch` | Create branch under dev/master strategy | Starting a new feat / fix / hotfix |
| `/pr` | Open a GitHub PR with the repo template | Ready to merge |
Skills and rules use **independent loading mechanisms**: rules auto-load
into the system prompt, skills only trigger when the user types `/<name>`.
### 3.4 settings.json
```json
{
"permissions": {
"allow": ["Bash(uv sync*)", "Bash(make*)", "Bash(uv run pytest*)", ...]
}
}
```
**Purpose**: reduce permission prompts. Team-shared config goes into
`settings.json` (in git); personal preferences go into `settings.local.json`
(gitignored).
---
## 4. Code quality gates
```
┌──────────────────────────────────────────────────────┐
│ Each stage can independently fail the change │
└──────────────────────────────────────────────────────┘
[Local editor]
│
▼
Stage 1: editor real-time feedback
├ ruff (lint + format) on save
└ path-relevant .claude/rules guide Claude Code
│
▼
Stage 2: pre-commit (triggered by `git commit`)
├ ruff fix + format
├ trailing-whitespace, end-of-file-fixer
├ check-yaml, check-toml
├ check-added-large-files (≥1MB)
├ detect-private-key
└ gitlint (commit-msg stage; rejects malformed messages)
│
▼
Stage 3: local `make ci` (manual, before push)
├ make lint (ruff check + ruff format --check + import-linter)
├ make test (pytest tests/unit)
└ make integration (pytest tests/integration)
│
▼
Stage 4: CI (GitHub Actions, push + PR triggered)
└ re-runs the same `make lint / test / integration` targets
│
▼
Stage 5: PR review
├ ≥ 1 approval
└ all threads resolved + all CI green
```
**Key design**: when any stage fails, **never merge** — there is no
`--no-verify` / `--allow-failure` escape hatch.
---
## 5. Dependencies & build
### 5.1 pyproject.toml overview
```toml
[project]
name = "everos"
requires-python = ">=3.12"
dependencies = [...] # runtime deps (minimal set)
[project.optional-dependencies]
multimodal = [...] # extras (install on demand)
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/everos"] # src layout
[project.scripts]
everos = "everos.entrypoints.cli.main:app" # exposes CLI command
[tool.ruff] # code style
[tool.pytest.ini_options] # tests
[tool.coverage.run] # coverage config (gate lives in `make cov`)
[tool.importlinter] # dependency direction
[dependency-groups]
dev = ["ruff", "pytest", "pytest-asyncio", "pytest-cov",
"import-linter", "pre-commit", "ipdb"]
```
**Single-file principle**: configuration that used to live in `pylintrc`,
`pytest.ini`, `.isort.cfg` is **all consolidated into `pyproject.toml`**.
### 5.2 Makefile commands
```
make help list all targets
make install uv sync --frozen
make format ruff fix + format
make lint ruff + import-linter + datetime discipline + openapi drift
make test pytest tests/unit
make integration pytest tests/integration
make cov pytest unit + integration, coverage gate (fail under 80%)
make ci lint + test + integration ← CI invokes these targets
make clean clear caches
```
**Single source of truth**: CI only invokes `make <target>`, so local and CI
run identical commands and cannot drift.
### 5.3 env.template (slimmed down)
The template lives at `src/everos/templates/env.template` (bundled
inside the wheel as package data, copied to `./.env` via `everos init`).
It groups settings by provider, each block sharing the OpenAI-protocol
`MODEL` / `API_KEY` / `BASE_URL` triple:
```
EVEROS_LLM__* # text model (model / api_key / base_url)
EVEROS_MULTIMODAL__* # vision model for image/office inputs
EVEROS_EMBEDDING__* # embedding model (vector index)
EVEROS_RERANK__* # cross-encoder reranker
EVEROS_MEMORY__ROOT # memory-root (md files + .index/{sqlite,lancedb}/)
EVEROS_LOG_LEVEL # DEBUG | INFO | WARNING | ERROR
EVEROS_LOG_FORMAT # json | text
TZ # display timezone (storage is always UTC)
```
Every key has a sensible default except the `API_KEY` fields, which you fill in.
---
## 6. CI/CD (GitHub Actions)
### 6.1 Strategy
```
┌──────────────────────────────────────────────────────────┐
│ │
│ GitHub Actions (.github/workflows/) │
│ ci.yml push (main/dev/master) + PR │
│ ├ make install-deps (uv sync --frozen) │
│ ├ make lint (ruff + import-linter + │
│ │ datetime + openapi drift) │
│ ├ make test (pytest tests/unit) │
│ └ make integration (pytest tests/integration) │
│ docs.yml Markdown link check + issue-template YAML │
│ │
│ Consistency: │
│ ├ astral-sh/setup-uv (cache keyed by uv.lock) │
│ ├ Makefile is the single source of CI commands │
│ └ pre-commit runs locally first to reduce CI churn │
│ │
└──────────────────────────────────────────────────────────┘
```
### 6.2 CI checklist
| Check | Tool | Failure condition |
|---|---|---|
| Lint | `make lint` (ruff check + ruff format --check) | any error |
| Layer direction | `make lint` (lint-imports inside) | layer violation |
| Datetime discipline | `make lint` (check_datetime_discipline.py) | bypasses helper module |
| OpenAPI drift | `make lint` (dump_openapi.py --check) | schema ≠ committed openapi.json |
| Unit | `make test` (pytest tests/unit) | any failure |
| Integration | `make integration` (pytest tests/integration) | any failure |
Integration tests run with a `FakeLLMClient` — no live credentials are needed in CI.
Commit message format is enforced **locally** via `gitlint` in the `commit-msg`
pre-commit stage; it does not run in CI.
### 6.3 Branch protection
| Branch | Rule |
|---|---|
| **master** | branch protection: PR + 1 review + green CI; no direct push |
| **dev** | same as above |
| feat / fix / hotfix | free push; rebase parent before merge |
---
## 7. Collaboration workflow
### 7.1 Branch model (GitFlow Lite)
```
v0.1 v0.2 v1.0
▲ ▲ ▲
│ release PR │ release PR │ release PR
│ (dev→master+tag) │ (dev→master+tag) │ (dev→master+tag)
master ●──────────────────────●─────────────●──────────────────●──────────────────────────────────●────► stable / released
│ ▲ │ │
│ │ merge hotfix │ │
│ │ │ │
│ ●──●──┘ │ │
│ │ hotfix branch │ │
│ │ (cut from master) │ │
│ │ │ │
│ ▼ sync to dev │ │
│ │ │ │
dev ●──●──●──●──●──●──●──●──●─●──●──●─●──●──●──●──●──●──●──●──●─●──●──●──●──●──●──●──●──●──●──●──●─────► integration
▲ ↑ ↑ ↑
│ release point release point release point
feat/A (dev HEAD → (dev HEAD → (dev HEAD →
●──●──● master + v0.1) master + v0.2) master + v1.0)
feat/* : cut from dev → PR → merge into dev
hotfix/* : cut from master → merge into master + sync into dev (double merge)
release : dev → master + tag on master (no separate release branch)
Vertical │ in the diagram = "dev HEAD merged into master via release PR + v0.x tag"
```
Details in [../.claude/skills/new-branch/SKILL.md](../.claude/skills/new-branch/SKILL.md).
### 7.2 PR template
A single PR template at [`.github/PULL_REQUEST_TEMPLATE.md`](../.github/PULL_REQUEST_TEMPLATE.md)
with five sections: **Summary / Area / Verification / Checklist / Notes for
Reviewers**. The `/pr` skill fills it in (see
[../.claude/skills/pr/SKILL.md](../.claude/skills/pr/SKILL.md)).
### 7.3 Commit convention (Conventional Commits)
Format: `<type>[(scope)][!]: <description>` per
[Conventional Commits](https://www.conventionalcommits.org).
```
feat: new feature
fix: bug fix
refactor: restructuring (no behavior change)
test: add / update tests
docs: documentation
style: formatting
perf: performance optimization
chore: configuration / build / tooling
build: build system or dependencies
ci: CI configuration
revert: revert a previous commit
```
`gitlint` enforces the format **locally** via its `contrib-title-conventional-commits`
rule in the commit-msg pre-commit stage. See
[../.claude/skills/commit/SKILL.md](../.claude/skills/commit/SKILL.md).
---
## 8. Issue templates / user support
```
.github/ISSUE_TEMPLATE/
├── bug_report.yml structured bug report (form)
├── feature_request.yml feature proposal (form)
├── use_case.yml share a use case / integration
├── docs.yml documentation issue
└── config.yml disable blank issues + community links
CONTRIBUTING.md contributor onboarding: setup / code style /
branch / commit / PR / testing
```
---
## 9. Infrastructure summary table
```
┌─────────────────────┬──────────────────────────────────────┬─────────────┐
│ Facility │ Location / file │ Failure │
│ │ │ impact │
├─────────────────────┼──────────────────────────────────────┼─────────────┤
│ CLAUDE.md │ /CLAUDE.md │ cc loses │
│ │ │ context │
│ Team rules │ /.claude/rules/ (10) │ cc unaware │
│ │ │ of conv. │
│ Team skills │ /.claude/skills/ (3) │ no slash │
│ │ │ workflows │
│ Permissions │ /.claude/settings.json │ cc prompts │
│ │ │ on each op │
├─────────────────────┼──────────────────────────────────────┼─────────────┤
│ pyproject │ /pyproject.toml │ build fail │
│ Lock file │ /uv.lock │ dep drift │
│ Makefile │ /Makefile │ no unified │
│ │ │ entry │
│ pre-commit │ /.pre-commit-config.yaml │ no local │
│ │ │ gate │
│ env template │ /src/everos/templates/env.template │ newcomers │
│ │ │ lost on env│
├─────────────────────┼──────────────────────────────────────┼─────────────┤
│ CI │ /.github/workflows/ci.yml │ PR cannot │
│ │ │ merge │
│ Docs CI │ /.github/workflows/docs.yml │ broken │
│ │ │ doc links │
│ PR template │ /.github/PULL_REQUEST_TEMPLATE.md │ no PR temp │
│ Issue templates │ /.github/ISSUE_TEMPLATE/ (5) │ scattered │
│ CONTRIBUTING │ /CONTRIBUTING.md │ contrib. │
│ │ │ confused │
└─────────────────────┴──────────────────────────────────────┴─────────────┘
```
---
## 10. Future extensions
```
Near-term
□ /new-module skill: scaffold a subpackage that complies with rules
□ ruff rule sets: add D (docstring), ANN (annotations)
□ Static type checking (pyright or mypy) once hot paths stabilize
Mid-term
□ release-please / Conventional Commits → automated changelog
□ Automated PyPI wheel upload on tag
□ Multi-Python version matrix (3.12 / 3.13)
□ Performance benchmark CI with historical comparison
Long-term
□ Mutation testing (mutmut)
□ Coverage ratchet (raise the 80% gate as the suite matures)
```
---
## 11. On investing in engineering infrastructure
```
┌──────────────────────────────────────────────────────────┐
│ │
│ Plain business code ≠ an engineering project │
│ │
│ Engineering project = business code + │
│ coding rules + │
│ quality gates (pre-commit + CI) + │
│ automation (Makefile + skills) + │
│ collaboration (branch + PR) + │
│ knowledge base (CLAUDE.md + │
│ rules + docs) │
│ │
│ The earlier this infrastructure lands, the faster and │
│ farther the team can run. │
│ │
└──────────────────────────────────────────────────────────┘
```
Old project vs. new project after this rewrite:
| Dimension | Old project | New project |
|---|---|---|
| Lint tools | black + isort + pylint | ruff (single tool) |
| Config files | pyproject + pylintrc + pyrightconfig + pytest.ini | unified pyproject.toml |
| pre-commit | basic | adds gitlint commit-msg + import / yaml / private-key checks |
| Layer direction | not enforced | import-linter enforced in CI |
| Commit format | freeform | gitlint pre-commit hook (Conventional Commits) |
| Claude Code integration | partial rules | rules + skills + settings (full) |
| CI platform | ad hoc | GitHub Actions calling Makefile targets |
| Tests | basic | unit + integration + e2e + coverage report |
These are not perfectionism — they are baseline requirements for
**multi-person collaboration, long-term maintenance, and sustainable
evolution**.
---
## 12. References
- Hard coding rules: [../.claude/rules/](../.claude/rules/) (auto-loaded by Claude Code)
- Slash command workflows: [../.claude/skills/](../.claude/skills/)
- Contributor onboarding: [../CONTRIBUTING.md](../CONTRIBUTING.md)
- Architecture: [architecture.md](architecture.md)
- Claude Code memory mechanism: [code.claude.com/docs/en/memory.md](https://code.claude.com/docs/en/memory.md)
- Claude Code skills: [code.claude.com/docs/en/skills.md](https://code.claude.com/docs/en/skills.md)
- ruff: [docs.astral.sh/ruff](https://docs.astral.sh/ruff/)
- import-linter: [import-linter.readthedocs.io](https://import-linter.readthedocs.io/)
- gitlint: [jorisroovers.com/gitlint](https://jorisroovers.com/gitlint/)
- uv: [docs.astral.sh/uv](https://docs.astral.sh/uv/)
- pre-commit: [pre-commit.com](https://pre-commit.com/)
- Conventional Commits: [conventionalcommits.org](https://www.conventionalcommits.org/)
- GitHub Actions: [docs.github.com/en/actions](https://docs.github.com/en/actions)