Canonical repo docs: memory-obsidian.md · MCP tools § memory · Architecture

Shipped vs vision: This page describes what memory_ingest / memory_recall do today, then explains how they fit the broader Memory 2.0 design (typed graph links, ATRClaims on every recall, semantic pruning, full WORM on memory events). Items marked roadmap are specified in DAOS v2.7 and Build plan v2.7.1 but not fully implemented in the MCP path yet.

clawql-memory (Memory 2.0)

clawql-memory is ClawQL's durable knowledge layer. Chat threads, in-process cache keys, and one-shot RAG chunks disappear when a process restarts or a session ends. Memory persists institutional knowledge as Markdown in an Obsidian-style vault, indexes it for hybrid recall, and optionally shares it across teammates.

What Problem Memory Solves

Agents lose context when the MCP server restarts or Kubernetes reschedules a pod, when you open a new chat days later on the same project, or when multiple teammates run isolated vault directories. Memory turns distilled outcomes — decisions, debugging notes, API quirks, runbooks — into files you can diff, back up, and recall. The pattern is sometimes called an LLM wiki or incremental wiki: the model is an editor of stable pages, not only a stateless answerer.

Core Design Principles

• Markdown as source of truth — Notes live under Memory/ as plain .md with YAML frontmatter. Humans can open the vault in Obsidian or any editor. Derived indexes cite back into this canon.
• One job per layer — Wikilinks/memory.db for explicit graph; embeddings for fuzzy semantics; PageIndex for heading trees; codegraph for source structure; Onyx for external enterprise search.
• Hybrid retrieval facade — memory_recall with optional sources returns normalized hits[] and specialist followUps; deep tools stay typed.
• Ontology structured filters — for exact field predicates (schema + filters → ontology.db via clawql-ontology), not semantic similarity. See memory_recall structured filters.
• Derived indexes are disposable — memory.db, embeddings, and PageIndex rebuild from Markdown; they are not the canonical store.
• Governance where it matters — Optional Presidio redaction at the gateway, vault path allowlists, and team sync that never uploads API secrets. Full ATRClaims filtering on every recall and WORM on every memory mutation are roadmap (see DAOS Layer 3).
• Fail-closed sync — Team sync failures are logged but do not block ingest/recall locally.

Memory 2.0 Components

Each piece has one job. The vault is the only canonical store; everything else is a derived index, a code index, an external peer, or the recall facade.

Obsidian / Vault

The single source of truth where knowledge is stored as human-readable Markdown files with YAML frontmatter under Memory/. Every derived index and query path ultimately references or cites back into the vault.

Tools: memory_ingest (write), memory_recall (read), optional team memory_sync.

Wikilinks + memory.db

Parses [[links]] into a queryable SQLite graph (wikilink_edge and vault_chunk tables) during ingest, and supports graph traversal and chunk retrieval during recall. This is the explicit structured graph layer over the vault — reliable link-based navigation without depending on vectors. Rebuilt from Markdown (not a second source of truth).

Embeddings

Creates vector representations of vault chunks for similarity search inside memory_recall when CLAWQL_VECTOR_BACKEND and an embedding API key are configured. The fuzzy semantic retrieval layer that surfaces relevant content when no explicit wikilinks or structural paths exist. Stored in memory.db BLOBs and/or pgvector depending on backend.

PageIndex

Generates a hierarchical heading tree (pageindex.db.json) for section-level addressing via pageindex_build_tree, then supports traversal, synthesis, and content retrieval. The deterministic, vectorless navigation and token-budget synthesis layer built directly on the vault's document structure.

clawql-codegraph

Parses source with the TypeScript compiler (TS/JS) and tree-sitter WASM (30+ languages: Python, Go, Rust, Java, C/C++, C#, Ruby, …). Preferred ingestion: codegraph_sync (index → Louvain → report → vault). TS/JS gets enclosing-scope calls, heritage/exports, cross-file linking, React/Next tags, plus codegraph_explore / codegraph_impact. Optional Graphify graph.json import — no Python.

Onyx

Queried via knowledge_search_onyx (or memory_recall with sources: ["onyx"]) and optionally pulling enterpriseCitations back into the vault during memory_ingest as short citation blocks. The external search peer that supplies organizational knowledge without ClawQL assembling or owning the enterprise corpus itself.

Ontology index (ontology.db)

When vault notes carry machine-readable fields (for example legal CLAWQL_* tags from the legal domain pack), ClawQL syncs them into ontology.db. memory_recall with schema + filters evaluates typed predicates against that index — exact matches, no near-miss ranking. This is what closed OpenBench B-7 enumeration after semantic-only recall scored zero on false positives. Spec: memory_recall structured filters. Essay: Memory Finds. Ontology Decides..

How Agents Should Use Them

Architecture: How the Pieces Connect

Today's shipped stack is a practical subset of the full Memory 2.0 design in the DAOS spec:

Ingest path: memory_ingest writes or appends vault notes → rescans → rebuilds memory.db (chunks + wikilink edges; embeddings when configured). Optional rebuild.pageindex refreshes the heading tree for that note.

Recall path: memory_recall can query sources: vault | vector | codegraph | pageindex | onyx. Returns normalized hits[], specialist followUps, and legacy results / codeGraphHits.

Team sync (optional): pushes and pulls the Memory/ tree via object storage. memory.db is rebuilt locally after pull. Code graph storage is local unless you sync codegraph.db.json yourself.

Roadmap (DAOS): separate encrypted blob vault, typed link ActionTypes, Merkle root on every node at ingest, and recall filtered by signed ATRClaims purpose.

MCP Tools and Enablement

Default on for vault + PageIndex; hide all memory + PageIndex tools with CLAWQL_ENABLE_MEMORY=0. Hide PageIndex only with CLAWQL_ENABLE_PAGEINDEX=0. Register code graph tools with CLAWQL_ENABLE_CODEGRAPH=1. Requires a writable CLAWQL_OBSIDIAN_VAULT_PATH for vault I/O; code graph additionally needs CLAWQL_CODEGRAPH_ROOT and CLAWQL_CODEGRAPH_PATH.

memory_recall Sources

Pass sources: vault | vector | codegraph | pageindex | onyx. Defaults (when omitted): vault + vector, plus hybrid env flags. Response includes normalized hits[], specialist followUps, and legacy results / codeGraphHits.

Hands-On: Memory Between Chats

Use memory_ingest and memory_recall when context should survive MCP server restarts, pod reschedules, or starting a new chat days later.

Storage You Must Provide

Set CLAWQL_OBSIDIAN_VAULT_PATH to a directory that already exists and is writable by the ClawQL process at startup. memory_ingest creates pages under Memory/<slug>.md with YAML frontmatter; memory_recall scans that tree.

Local / VM / bare Docker: choose a folder on disk, ensure permissions match the user running clawql-mcp, export CLAWQL_OBSIDIAN_VAULT_PATH to that path.

Kubernetes: the container filesystem is ephemeral — use a PersistentVolumeClaim mounted at the path you pass as CLAWQL_OBSIDIAN_VAULT_PATH. For more than one replica, use a volume that is shared read-write across pods (RWX) so every pod sees the same Markdown.

Managed platforms (e.g. Cloud Run) treat unattached disk as ephemeral unless you add a mounted volume.

Before You Start

1. Confirm CLAWQL_ENABLE_MEMORY is not set to 0 (tools hidden).
2. Confirm CLAWQL_OBSIDIAN_VAULT_PATH points at your real vault and that Memory/ can be created or updated.
3. ingest and recall must use the same vault root — path mismatches produce empty recall or writes you cannot see.
4. Treat vault Markdown as sensitive if you put secrets inside; prefer redaction and short-lived tokens in prose.

Pattern: Checkpoint with memory_ingest

At the end of a thread (or after a milestone), persist a distilled note: goal, done, next, constraints, links. Use a title you can memory_recall against later.

{
  "title": "Handoff — Acme API hardening",
  "insights": "## Goal\nLock down the Acme public API review workflow.\n\n## Done\n- OpenAPI merge uses CLAWQL_SPEC_PATH + bundled slack\n\n## Next\n1. execute slack::conversations_list\n2. Post summary to #review-bot\n\n## Constraints\n- Branch feat/api-review; no production executes",
  "append": true,
  "wikilinks": ["Slack MCP", "Security review"]
}

Re-using the title with append: true (default) appends a new dated section instead of overwriting the whole note — good for running threads.

Pattern: Hydrate with memory_recall

In a new chat (or after a restart), pull vault context before planning.

{
  "query": "Acme API hardening slack review",
  "limit": 8,
  "maxDepth": 2
}

Responses include results[] with path, snippet, score, reason (keyword, link, or vector when embeddings are configured). Tune limit, maxDepth, and minScore to balance noise vs coverage.

Limits and Pitfalls

• No vault path — tools may be registered but writes/recall require a valid CLAWQL_OBSIDIAN_VAULT_PATH.
• Read-only mount — memory_ingest fails; fix volume permissions or mount options.
• Replicas without shared storage — each pod's vault diverges; use one replica or RWX PVC.
• Large logs — use toolOutputsFile with CLAWQL_MEMORY_INGEST_FILE_ROOTS instead of megabyte toolOutputs in JSON.

Ingestion: From Agent Summary to Vault Page

A typical memory_ingest flow:

1. Agent distills — Goal, done, next, constraints, links (not raw megabyte logs unless via toolOutputsFile).
2. Write Markdown — File under Memory/<slug>.md with frontmatter; append: true (default) adds a dated section to an existing title.
3. Wikilinks — Optional wikilinks array becomes [[Related Page]] lines in the note.
4. Index rebuild — On success, ClawQL rescans the recall subtree and refreshes memory.db rows (chunks + wikilink_edge).
5. Index page — Optionally maintains _INDEX_{Provider}.md with navigable wikilinks (CLAWQL_MEMORY_INDEX_PAGE, CLAWQL_MEMORY_INDEX_PROVIDER).
6. Team sync — If CLAWQL_SYNC_AUTO=1, debounced push after ingest.

Gateway redaction (opt-in): Presidio hooks can run before persistence when enabled on the gateway. Mandatory pre-store redaction on every memory write is the target DAOS pipeline, not yet the default for all deployments.

Wiki-Style Linking and the Knowledge Graph

Wikilinks are ClawQL's lightweight knowledge graph. Use [[Page Name]] or [[Page Name|alias]] in prose or via the wikilinks field.

Semantics today: links are untyped — they mean "related page." They are not yet contradicts vs depends_on enums from the DAOS spec (roadmap).

Graph traversal: memory_recall walks forward and backward links up to maxDepth (env default CLAWQL_MEMORY_RECALL_MAX_DEPTH). Results can show reason: "link".

Persistence: edges are parsed from Markdown and stored in wikilink_edge so recall can use the graph even when vector search is off.

Stable title + wikilinks turn isolated session notes into a navigable web agents can traverse without re-embedding entire histories. Operators can inspect backlinks in Obsidian.

{
  "title": "Acme API — rate limits",
  "insights": "## Finding\n429 when burst > 40 rps.\n\n## Mitigation\nExponential backoff on execute.",
  "wikilinks": ["Acme API hardening", "Slack MCP"],
  "append": true
}

Hybrid Recall: Keywords, Wikilinks, and Vectors

memory_recall always runs a lexical leg: keyword scoring over body, headings, and filenames. It merges wikilink neighbors when maxDepth > 0.

Optional vector leg requires all of: CLAWQL_VECTOR_BACKEND = sqlite or postgres, memory.db enabled, and an embedding API key. Chunks are ranked by similarity to the embedded query.

Hits include reason: "keyword", "link", or "vector". Tune limit, maxDepth, minScore, and CLAWQL_MEMORY_RECALL_* env vars.

PageIndex complements vectors: hierarchical traverse + synthesize gives deterministic, token-budgeted context without embeddings — useful when you want stable section addresses.

Code graph complements vault narrative: AST-derived imports and calls give precise architecture traces without pasting entire repos. Enable hybrid recall (CLAWQL_MEMORY_RECALL_HYBRID_CODEGRAPH=1) to merge vault hits and symbol matches in one memory_recall call.

Roadmap: unified hybrid ranking score across exact PageIndex hits, cosine similarity, graph proximity, and code-graph symbol relevance with ATRClaims classification filters on each node.

Team Sync: Shared Vaults Across Agents and Teammates

Team sync shares Markdown — not secrets — via object storage:

CLI: clawql sync init, push, pull, status — providers R2 (default), S3, GCS. MCP: memory_sync with direction: auto (pull then push), pull, or push.

Full setup: Team vault sync.

Roadmap (DAOS): governed publish_to_team_memory ActionTypes, role-based visibility on shared nodes, Ouroboros NSV/SGDOP drift signals between agent swarms, and Diversity Dividends for high-value shared knowledge. Today's sync is file-level consistency with conflict listing.

Self-Improving Loop: Memory Meets Inference

Memory and clawql-inference form a flywheel when both are deployed:

Production inference (OpenAI-compatible gateway)
  → InferenceRecord stored (prompt, response, tier, evaluator verdict, correlation_id)
  → Agent distills outcomes → memory_ingest (wikilinks + stable titles)
  → memory_recall enriches the next session
  → Verdict-filtered export (passed + min score)
  → PII-scrubbed JSONL + WORM dataset manifest
  → Fine-tune → register model on frugal tier
  → Cheaper, better completions → richer ingest material

Shipped inference pieces: ObservedInferenceGateway (durable call records), clawql inference export (filter by --verdict passed, tier, --min-score), Presidio scrubbing on export by default, semantic cache on the gateway.

Memory's role: vault notes capture curated lessons (not every raw completion). Wikilinks connect new findings to existing topics. Team sync spreads refined notes across agents.

Roadmap: automatic distillation of long session history into Memory during semantic pruning; evaluator-linked correlation_id edges from inference records into graph nodes.

Semantic Pruning and Pre-Pruning Snapshots (Roadmap)

Long sessions can exhaust context windows. The DAOS design adds a pruning engine not yet in the default MCP experience:

• Trigger — Context utilization ≥ pruning_context_threshold (from Manifest policy)
• Distillation — Older turns condensed; verbatim tool results and recent pruning_retain_turns kept
• Causal lock — Turns before pending high-impact actions cannot be pruned
• Pre-pruning snapshot — Bit-perfect encrypted snapshot to cold storage before distillation; WORM audit of the event

Practical substitute today: agents memory_ingest checkpoints before context fills up, and use pageindex_traverse / memory_recall instead of carrying full transcripts. See DAOS § Memory pruning and Build plan P2-A.

Governance, Security, and Observability

Treat vault Markdown as sensitive if notes contain tokens or PII. Prefer redaction in prose and short-lived credentials.

Related Guides and References

• Cache handoff between chats — when cache is enough
• Team vault sync
• clawql-inference — export, fine-tune, semantic cache, flywheel
• Memory plugin — env table and tool list
• Code graph plugin — native sync, 30+ languages, explore/impact, optional Graphify import
• hybrid-memory-backends.md
• DAOS Unified Architecture § Layer 3 — full Memory 2.0 vision

© Copyright 2026. All rights reserved. · ClawQL on GitHub