clawql-inference

Status: Shipped (July 2026)
Package: packages/clawql-inference
Epic: #556

clawql-inference is ClawQL's TypeScript-native inference gateway and model-improvement platform — a LiteLLM-class layer with ClawQL's trust model: WORM-auditable routing, semantic cache, model tier escalation, agent coordination hooks, verdict-filtered export, and fine-tuning flywheel. It is designed as a drop-in OpenAI replacement (OPENAI_BASE_URL=http://127.0.0.1:8080/v1) while closing the production loop that generic proxies leave open.

Related docs: Inference provider plugins · Token efficiency (Layer 5 cache) · Ouroboros library

What it does

Architecture overview

flowchart TB
  subgraph clients [Clients]
    SDK[OpenAI SDK / curl]
    CLI[clawql inference complete]
    OB[Ouroboros EvolutionaryLoop]
  end

  subgraph http [HTTP layer]
    AUTH[Virtual key auth middleware]
    OAI[OpenAI-compat router]
  end

  subgraph gateway [Gateway decorator stack — inner to outer]
    CFG[ConfiguredInferenceGateway]
    FB[FallbackChainGateway]
    CACHE[SemanticCachedGateway]
    ENT[EntitlementEnforcedGateway]
    OBS[ObservedInferenceGateway]
  end

  subgraph backends [Backends]
    PLG[Provider plugins — openai / anthropic / ollama]
    STORE[(Inference store — memory / jsonl / postgres)]
  end

  SDK --> AUTH --> OAI --> OBS
  CLI --> OBS
  OB --> CFG
  OBS --> ENT --> CACHE --> FB --> CFG --> PLG
  OBS --> STORE

Every successful complete() call flows outward through decorators and ends in the call store. HTTP clients additionally pass through virtual-key auth before the OpenAI router invokes the same gateway.

Gateway decorator stack

createInferenceGateway() composes decorators in this order (inner → outer):

import { createInferenceGateway } from 'clawql-inference'

const gateway = createInferenceGateway({ env: process.env })
const result = await gateway.complete({
  model: 'anthropic/claude-sonnet-4',
  messages: [{ role: 'user', content: 'Summarize this spec' }],
  correlationId: 'seed_abc_gen_2',
  team: 'eng',
})

Disable layers via options: \{ semanticCache: false \}, \{ fallback: false \}, \{ store: null \}.

HTTP API layer

Entry points

createInferenceHttpApp() (packages/clawql-inference/src/api/server.ts):

1. express.json() — 2 MB body limit
2. GET /healthz — liveness (no auth)
3. GET /v1 — capability discovery
4. createVirtualKeyAuthMiddleware — Bearer / x-api-key when keys enabled
5. createOpenAiCompatRouter — models list + chat completions

OpenAI-compatible endpoints

Request headers

Drop-in OpenAI client

export OPENAI_BASE_URL=http://127.0.0.1:8080/v1
export OPENAI_API_KEY=<virtual-key-or-upstream-key>
npx clawql-inference
# or: clawql inference serve --port 8080

Provider plugins

Built-ins register via composeDefaultProviderPlugins() unless allow/denylisted:

Model ids use provider/model (e.g. ollama/phi4, anthropic/claude-sonnet-4). Bare public ids like gpt-4o resolve through the registry when a matching provider is configured.

Third-party plugins use the same InferenceProviderPlugin contract — see Inference provider plugins. Subpath export: clawql-inference/plugin.

Model tier escalation

TierEscalationRouter implements AdaptiveRouter with three tiers:

Kill switches:

• Escalation is off by default unless CLAWQL_INFERENCE_ROUTING_ENABLED=1 or CLAWQL_INFERENCE_MODEL_PIN is set
• CLAWQL_INFERENCE_MODEL_PIN=<modelId> bypasses the ladder

Tier map overrides persist at $CLAWQL_HOME/Inference/tier-map.json (written by clawql inference finetune register or escalation set-tier).

clawql inference escalation show
clawql inference escalation set-tier --tier frugal --model ollama/phi4-custom

Ouroboros integration

When EvolutionaryLoop runs with an AdaptiveRouter:

1. router.initialTier() picks starting tier per seed / decomposed-child context
2. On generation failure (AC fail, low eval score, drift exceeded), router.escalate() moves one notch up
3. model_escalation audit events append to the Ouroboros Postgres / in-memory event store
4. Correlation id: \{seedId\}_gen_\{n\}

See Agent coordination for the Hermes tripwire path.

Semantic cache

Layer 5 in token efficiency. Enabled with CLAWQL_INFERENCE_SEMANTIC_CACHE=1.

Cache hits return stored responses with cacheHit: true on inference records. Embedding failures fail open to live inference.

clawql inference cache   # show active config

Fallback chains

When the primary provider/model fails, try configured alternates before surfacing an error.

export CLAWQL_INFERENCE_FALLBACK_ENABLED=1
export CLAWQL_INFERENCE_FALLBACK_FRUGAL=ollama/phi4,openai/gpt-4o-mini
export CLAWQL_INFERENCE_FALLBACK_STANDARD=groq/llama-3.3-70b,anthropic/claude-haiku-4

Chains also persist at $CLAWQL_HOME/Inference/fallback-chains.json (byTier / byModel). Responses include fallback.attempted and fallback.succeeded.

clawql inference fallback

Virtual keys and HTTP auth

Per-team API keys with optional USD budgets and rate limits (100rpm, 10rps).

export CLAWQL_INFERENCE_KEYS_ENABLED=1
clawql inference keys create --team eng --budget-usd 500 --rate-limit 100rpm
clawql inference keys list
clawql inference keys revoke --id vk_abc123

Keys persist at $CLAWQL_HOME/Inference/virtual-keys.json (secrets stored as SHA-256 hashes only). When enforcement is active, /v1/* requires a valid key; /healthz stays open.

team and virtualKeyId are stamped on inference records; spend rollups support --group-by team.

Plan entitlements (clawql-payments)

When CLAWQL_PAYMENTS_ENFORCE_INFERENCE=1, the gateway checks plan limits before each completion and increments inference_calls usage after success. Tenant resolution order:

1. x-clawql-tenant-id header
2. Virtual key team
3. $CLAWQL_HOME/Payments config default tenant

Limit breaches throw EntitlementLimitError (HTTP 402-shaped OpenAI error) and append a WORM payment audit entry. Configure plans via clawql payments — see packages/clawql-payments.

Inference call store

Every successful completion writes an InferenceRecord used by observability and export.

Backends

Postgres: CLAWQL_INFERENCE_DATABASE_URL or split CLAWQL_INFERENCE_DB_* vars. Default JSONL path: $CLAWQL_HOME/Inference/calls.jsonl.

Record fields

Observability CLI

clawql inference logs [--model M] [--tier T] [--since 24h] [--limit 50]
clawql inference spend [--group-by model|tier|team|provider] [--since 7d]
clawql inference trace --correlation-id <id>

Export and fine-tuning flywheel

Production traffic
  → WORM-logged inference (prompt, response, tier, verdict, correlation_id)
  → Evaluator verdicts + quality filters
  → PII-scrubbed dataset export (JSONL)
  → Fine-tuning job (Anthropic / OpenAI)
  → Custom model registered in ModelTierMap
  → Deployed to Frugal tier
  → Better cheap-tier results → better verdicts → better training data

LiteLLM routes inference. ClawQL closes the loop: infer → observe → evaluate → export → fine-tune → redeploy.

Export

clawql inference export \
  --output ./training-data/export.jsonl \
  --verdict passed \
  --tier frugal \
  --format openai-jsonl \
  --min-score 0.8

PII scrubbing (Presidio) is on by default. Every export writes a WORM dataset manifest (sample hashes, filter criteria, Merkle root, policy version).

Fine-tune and register

clawql inference finetune \
  --dataset ./training-data/export.jsonl \
  --base-model gpt-4o-mini \
  --provider openai

clawql inference finetune status --job-id ftjob_abc123
clawql inference finetune register --job-id ftjob_abc123 --tier frugal --alias ollama/phi4-production-v3

Pipeline automation

Scheduled auto-export when sample thresholds are met.

clawql inference pipeline enable \
  --schedule "0 2 * * 0" \
  --min-samples 500 \
  --verdict passed \
  --target-tier frugal \
  --base-model gpt-4o-mini

clawql inference pipeline status
clawql inference pipeline run          # manual trigger
clawql inference pipeline worker       # cron sidecar

Config persists at $CLAWQL_HOME/Inference/pipeline.json. Cron worker:

• CLAWQL_INFERENCE_PIPELINE_WORKER=1 — starts with inference serve
• CLAWQL_INFERENCE_PIPELINE_POLL_MS — poll interval (default 60s)
• Tracks lastRunAt, lastRunStatus, lastRunDetail on each tick

Agent coordination (#562)

TierEscalationRouter.shouldTriggerAgentCoordination() fires when:

• Combined drift exceeds 0.3, or
• Standard-tier exhaustion with active failure signals

When CLAWQL_INFERENCE_AGENT_COORDINATION_ENABLED=1:

1. evaluateAgentCoordination() builds an agent_coordination audit entry
2. Hermes stub runs when HERMES_BASE_URL is unset; live MoA when configured
3. Ouroboros appends the event to the lineage store alongside model_escalation

Policy

clawql inference policy show [--json]

resolveInferencePolicy() aggregates the effective view from environment: escalation, cache, fallback, keys, store backend, export defaults, pipeline worker, and agent coordination. Manifest YAML overrides are planned; today the source is env.

Persistence layout

All operator state under $CLAWQL_HOME/Inference/:

Postgres store uses clawql_inference_calls (separate from CLAWQL_INFERENCE_DATABASE_URL).

CLI wiring

The clawql binary dispatches inference subcommands in src/onboarding/cli.ts → thin wrappers in src/onboarding/inference-cli.ts → packages/clawql-inference/src/cli/*.

clawql inference <subcommand>
├── serve              HTTP gateway (+ optional pipeline worker)
├── complete           One-shot completion
├── logs | trace | spend
├── export | finetune [status|register]
├── escalation [show|set-tier]
├── pipeline [enable|status|disable|run|worker]
├── cache | fallback | policy
└── keys [create|list|revoke]

Environment variables

Package exports

Differentiation vs LiteLLM

Implementation phasing

Planned (not blockers)

• Langfuse / OpenTelemetry full wiring (ADR 0005)
• Manifest YAML inference block overrides for policy show
• Live Hermes MoA HTTP client (stub ships today)
• Postgres advisory locks for multi-instance pipeline dedup

References

• Inference provider plugins
• Token efficiency architecture
• Ouroboros library
• Upstream Q00 sync roadmap
• Issue #556 — inference epic
• Issue #560 — model tier escalation