clawql-memory (Memory 2.0)

clawql-memory is ClawQL’s durable knowledge layer — the answer to ephemeral agent state. Chat threads, in-process cache keys, and one-shot RAG chunks disappear when a process restarts or a session ends. Memory persists institutional knowledge as Markdown in an Obsidian-style vault, indexes it for hybrid recall, and (optionally) shares it across teammates.

Hands-on MCP walkthrough: Vault memory between chats · Canonical repo docs: memory-obsidian.md · MCP tools § memory · Modularization § Layer 3

What problem memory solves

Agents lose context when:

• The MCP server restarts or Kubernetes reschedules a pod
• You open a new chat days later on the same project
• Multiple teammates run isolated vault directories

Memory turns distilled outcomes — decisions, debugging notes, API quirks, runbooks — into files you can diff, back up, and recall. The pattern is sometimes called an LLM wiki or incremental wiki: the model is an editor of stable pages, not only a stateless answerer. See memory-obsidian.md for the Karpathy / Obsidian framing.

Core design principles

• Markdown as source of truth — Notes live under Memory/ as plain .md with YAML frontmatter. Humans can open the vault in Obsidian or any editor.
• Hybrid retrieval — Keyword scoring, wikilink graph hops, and optional vector similarity run together in memory_recall.
• Derived index — memory.db (SQLite via sql.js) mirrors chunks and link edges; it is rebuilt from Markdown and is not the canonical store.
• Governance where it matters — Optional Presidio redaction at the gateway, vault path allowlists, and team sync that never uploads API secrets. Full ATRClaims filtering on every recall and WORM on every memory mutation are roadmap (see DAOS Layer 3).
• Fail-closed sync — Team sync failures are logged but do not block ingest/recall locally.

Architecture: vault, graph index, PageIndex, vectors

Today’s shipped stack is a practical subset of the full Memory 2.0 triple-store in the DAOS spec:

Roadmap (DAOS): separate encrypted blob vault, typed link ActionTypes, Merkle root on every node at ingest, and recall filtered by signed ATRClaims purpose. Optional Cuckoo / Merkle snapshot env gates exist in memory.db today for chunk membership experiments — see hybrid-memory-backends.md.

MCP tools and enablement

Default on; hide all memory + PageIndex tools with CLAWQL_ENABLE_MEMORY=0. Hide PageIndex only with CLAWQL_ENABLE_PAGEINDEX=0. Requires a writable CLAWQL_OBSIDIAN_VAULT_PATH for real I/O.

Ingestion: from agent summary to vault page

A typical memory_ingest flow:

1. Agent distills — Goal, done, next, constraints, links (not raw megabyte logs unless via toolOutputsFile).
2. Write Markdown — File under Memory/<slug>.md with frontmatter; append: true (default) adds a dated section to an existing title.
3. Wikilinks — Optional wikilinks array becomes [[Related Page]] lines in the note.
4. Index rebuild — On success, ClawQL rescans the recall subtree and refreshes memory.db rows (chunks + wikilink_edge).
5. Index page — Optionally maintains _INDEX_{Provider}.md with navigable wikilinks (CLAWQL_MEMORY_INDEX_PAGE, CLAWQL_MEMORY_INDEX_PROVIDER).
6. Team sync — If CLAWQL_SYNC_AUTO=1, debounced push after ingest.

Gateway redaction (opt-in): Presidio hooks can run before persistence when enabled on the gateway — see security. Mandatory pre-store redaction on every memory write is the target DAOS pipeline, not yet the default for all deployments.

Wiki-style linking and the knowledge graph

Wikilinks are ClawQL’s lightweight knowledge graph:

• In prose or via wikilinks: [[Page Name]] or [[Page Name|alias]].
• Semantics today — Links are untyped: they mean “related page,” like Obsidian. They are not yet contradicts vs depends_on enums from the DAOS spec (roadmap).
• Graph traversal — memory_recall walks forward and backward links up to maxDepth (env default CLAWQL_MEMORY_RECALL_MAX_DEPTH). Results can show reason: "link".
• Persistence — Edges are parsed from Markdown and stored in wikilink_edge so recall can use the graph even when vector search is off.

Why wikilinks matter: Stable title + wikilinks turn isolated session notes into a navigable web agents can traverse without re-embedding entire histories. Operators can inspect backlinks in Obsidian.

Practical pattern:

{
  "title": "Acme API — rate limits",
  "insights": "## Finding\n429 when burst > 40 rps.\n\n## Mitigation\nExponential backoff on execute.",
  "wikilinks": ["Acme API hardening", "Slack MCP"],
  "append": true
}

Hybrid recall: keywords, wikilinks, and vectors

memory_recall always runs a lexical leg: keyword scoring over body, headings, and filenames. It merges wikilink neighbors when maxDepth > 0.

Optional vector leg (all must be configured):

• CLAWQL_VECTOR_BACKEND = sqlite or postgres
• memory.db enabled and embedding API key (OPENAI_API_KEY or CLAWQL_EMBEDDING_API_KEY)
• Query embedded via OpenAI-compatible /embeddings; chunks ranked by similarity

Hits include reason: "keyword", "link", or "vector". Tune limit, maxDepth, minScore, and CLAWQL_MEMORY_RECALL_* env vars.

{
  "query": "Acme API rate limit backoff",
  "limit": 8,
  "maxDepth": 2
}

PageIndex complements vectors: hierarchical traverse + synthesize gives deterministic, token-budgeted context without embeddings — useful when you want stable section addresses. See Phase 1 platform guide § PageIndex.

Roadmap: unified hybrid ranking score across exact PageIndex hits, cosine similarity, and graph proximity with ATRClaims classification filters on each node.

Team sync: shared vaults across agents and teammates

Team sync shares Markdown — not secrets — via object storage:

CLI: clawql sync init, push, pull, status — providers R2 (default), S3, GCS. MCP: memory_sync with direction: auto (pull then push), pull, or push.

Auto sync on MCP runtime:

Full setup: Team vault sync · For teams.

Roadmap (DAOS): governed publish_to_team_memory ActionTypes, role-based visibility on shared nodes, Ouroboros NSV/SGDOP drift signals between agent swarms, and Diversity Dividends for high-value shared knowledge. Today’s sync is file-level consistency with conflict listing — appropriate for engineering teams sharing a vault prefix.

Self-improving loop: memory meets inference

Memory and clawql-inference form a flywheel when both are deployed:

Production inference (OpenAI-compatible gateway)
  → InferenceRecord stored (prompt, response, tier, evaluator verdict, correlation_id)
  → Agent distills outcomes → memory_ingest (wikilinks + stable titles)
  → memory_recall enriches the next session
  → Verdict-filtered export (passed + min score)
  → PII-scrubbed JSONL + WORM dataset manifest
  → Fine-tune → register model on frugal tier
  → Cheaper, better completions → richer ingest material

Shipped inference pieces:

• ObservedInferenceGateway — durable call records
• clawql inference export — filter by --verdict passed, tier, --min-score
• Presidio scrubbing on export by default
• Semantic cache on the gateway (embedding similarity) — separate from vault vectors but same embedding habit

Memory’s role: Vault notes capture curated lessons (not every raw completion). Wikilinks connect new findings to existing topics. Team sync spreads refined notes across agents.

Roadmap: automatic distillation of long session history into Memory during semantic pruning (below); evaluator-linked correlation_id edges from inference records into graph nodes; Ouroboros feedback when shared knowledge drifts.

Semantic pruning and pre-pruning snapshots (roadmap)

Long sessions can exhaust context windows. The DAOS design adds a pruning engine (not yet the default MCP experience):

• Trigger — Context utilization ≥ pruning_context_threshold (from Manifest policy)
• Distillation — Older turns condensed; verbatim tool results and recent pruning_retain_turns kept
• Causal lock — Turns before pending high-impact actions cannot be pruned
• Pre-pruning snapshot — Bit-perfect encrypted snapshot to cold storage before distillation; WORM audit of the event

Practical substitute today: agents memory_ingest checkpoints before context fills up, and use pageindex_traverse / memory_recall instead of carrying full transcripts. See DAOS § Memory pruning and Build plan P2-A.

Governance, security, and observability

Treat vault Markdown as sensitive if notes contain tokens or PII. Prefer redaction in prose and short-lived credentials. For enterprise Onyx citations in ingest, see enterpriseCitations in mcp-tools.md.

Related guides and references

• Vault memory between chats — MCP examples, PVC/K8s, append threading
• Cache handoff between chats — when cache is enough
• Team vault sync — R2/S3/GCS, Helm teamSync, memory_sync
• clawql-inference — export, fine-tune, semantic cache, flywheel
• Memory plugin — env table and tool list
• hybrid-memory-backends.md — sqlite vs postgres vectors
• memory-db-schema.md — wikilink_edge and chunks
• DAOS Unified Architecture § Layer 3 — full Memory 2.0 vision