---

title: clawql-inference

clawql-inference

Status: Shipped (July 2026) Package: packages/clawql-inference

clawql-inference is ClawQL's TypeScript-native inference gateway and model-improvement platform — a LiteLLM-class layer with ClawQL's trust model: WORM-auditable routing, semantic cache, model tier escalation, agent coordination hooks, verdict-filtered export, and fine-tuning flywheel. It is designed as a drop-in OpenAI replacement (OPENAI_BASE_URL=http://127.0.0.1:8080/v1) while closing the production loop that generic proxies leave open.

Related docs: Inference provider plugins · Token efficiency (12 layers) · Ouroboros library

---

What It Does

Capability	Summary
Gateway	OpenAI-compatible REST (/v1/chat/completions, /v1/models, SSE streaming)
Providers	Built-in OpenAI, Anthropic, Ollama; extensible plugin registry
Routing	Frugal → standard → frontier tier escalation with kill switches
Cache	Embedding similarity semantic cache (cosine threshold + TTL)
Efficiency	Layers 3–11 (terse, prompt cache, history/prompt compress, HTTP routing, structured/prefill)
Resilience	Per-tier / per-model fallback chains before hard failure
Auth	Virtual keys (per-team budgets, rate limits)
Entitlements	Optional plan limits via clawql-payments
Observability	Durable call store, logs / trace / spend CLI
Flywheel	Verdict-filtered export → fine-tune → register custom model in tier map
Automation	Scheduled pipeline worker (cron export when sample threshold met)
Ouroboros	model_escalation + agent_coordination audit events in lineage store

---

Quick Start

Example policy + walkthrough: examples/inference/ (policy.yaml + README).

export CLAWQL_HOME="${CLAWQL_HOME:-$HOME/.clawql}"
mkdir -p "$CLAWQL_HOME/Inference"
cp examples/inference/policy.yaml "$CLAWQL_HOME/Inference/policy.yaml"

export OPENAI_API_KEY=sk-...
export OLLAMA_BASE_URL=http://127.0.0.1:11434

clawql inference policy show          # expect policy_source: manifest+env
clawql inference serve --port 8080

Point clients at OPENAI_BASE_URL=http://127.0.0.1:8080/v1. Manifest YAML merges at runtime (resolveInferenceEffectiveEnv); env vars override YAML on conflicts.

One-shot CLI (no HTTP):

clawql inference complete --model ollama/phi4 --message "hello"

---

Architecture Overview

Entry Points

Client	Path into the gateway
OpenAI SDK / curl	Virtual key auth middleware → OpenAI-compat router → gateway stack
clawql inference complete	Calls the gateway stack directly (no HTTP auth layer)
Ouroboros EvolutionaryLoop	Calls ConfiguredInferenceGateway at the innermost layer (bypasses HTTP)

Gateway Decorator Stack (inner → outer)

Each complete() call passes outward through these layers before reaching a provider:

1. ConfiguredInferenceGateway — resolves provider/model, invokes the provider adapter
2. FallbackChainGateway — tries alternate models on primary failure (when fallback enabled)
3. SemanticCachedGateway — embedding-similarity cache lookup (when semantic cache enabled)
4. TokenEfficiencyGateway — Layers 3–11 (terse, compress, route, prompt-cache, structured/prefill markers)
5. EntitlementEnforcedGateway — plan limit checks via clawql-payments (when enforcement enabled)
6. ObservedInferenceGateway — appends an InferenceRecord to the call store
7. TracedInferenceGateway — OTLP spans to infra + Langfuse when configured

Call order from outside in: Traced → Observed → Entitlement → Efficiency → Cache → Fallback → Configured → provider.

Data Flow Summary

HTTP clients → auth middleware → OpenAI router → [Observed → Entitlement → Cache → Fallback → Configured] → provider plugin
CLI / library callers ──────────────────────────► [same decorator stack] ────────────────────────────────► provider plugin
Ouroboros loop ─────────────────────────────────► ConfiguredInferenceGateway (innermost) ───────────────► provider plugin

ObservedInferenceGateway ──writes──► Inference store (jsonl / postgres / memory)

---

Gateway Decorator Stack

createInferenceGateway() composes decorators in this order (inner → outer):

Layer	Module	When active	Behavior
1	ConfiguredInferenceGateway	Always	Resolves provider/model, calls provider adapter
2	FallbackChainGateway	CLAWQL_INFERENCE_FALLBACK_ENABLED=1 or chains file	Tries alternates on primary failure; sets response.fallback
3	SemanticCachedGateway	embeddings configured or CLAWQL_INFERENCE_SEMANTIC_CACHE=1	Read-safe semantic cache; write invalidation
4	TokenEfficiencyGateway	Always (per-layer kill switches)	Terse, compress, route, prompt-cache markers
5	EntitlementEnforcedGateway	CLAWQL_PAYMENTS_ENFORCE_INFERENCE=1	Checks plan limits, records usage
6	ObservedInferenceGateway	Store not off	Appends InferenceRecord to call store

const gateway = createInferenceGateway({ env: process.env })
const result = await gateway.complete({
  model: 'anthropic/claude-sonnet-4',
  messages: [{ role: 'user', content: 'Summarize this spec' }],
  correlationId: 'seed_abc_gen_2',
  team: 'eng',
})

Disable layers via options: \{ semanticCache: false \}, \{ fallback: false \}, \{ store: null \}.

---

HTTP API Layer

Entry Points

Command	Binary	Notes
clawql inference serve	clawql CLI	Wired via src/onboarding/inference-cli.ts
npx clawql-inference	packages/clawql-inference/bin	Standalone sidecar

OpenAI-Compatible Endpoints

Method	Path	Notes
GET	/healthz	Liveness
GET	/v1/models	Tier map + env models + Ollama tags
GET	/v1/models/:id	Single model
POST	/v1/chat/completions	Bare gpt-4o or provider/model; stream: true SSE

Request Headers

Header	Purpose
Authorization: Bearer <key>	Virtual key or upstream-style auth
x-api-key / x-clawql-api-key	Alternative key header
x-correlation-id / x-clawql-correlation-id	WORM lineage; echoed on response
x-clawql-tenant-id	Plan entitlement tenant override

Drop-In OpenAI Client

export OPENAI_BASE_URL=http://127.0.0.1:8080/v1
export OPENAI_API_KEY=<virtual-key-or-upstream-key>
npx clawql-inference

---

Provider Plugins

Built-ins register via composeDefaultProviderPlugins() unless allow/denylisted:

Plugin id	Adapter	Credentials
openai	Chat completions	OPENAI_API_KEY, CLAWQL_OPENAI_BASE_URL
anthropic	Messages API	ANTHROPIC_API_KEY, CLAWQL_ANTHROPIC_BASE_URL
ollama	Local /api/chat	OLLAMA_BASE_URL (default http://127.0.0.1:11434)

Model ids use provider/model (e.g. ollama/phi4, anthropic/claude-sonnet-4). Bare public ids like gpt-4o resolve through the registry when a matching provider is configured.

Third-party plugins use the same InferenceProviderPlugin contract. Subpath export: clawql-inference/plugin.

---

Model Tier Escalation

TierEscalationRouter implements AdaptiveRouter with three tiers.

Tier	Default model (env override)	Role
frugal	CLAWQL_INFERENCE_MODEL_FRUGAL → ollama/phi4	Cheapest / custom fine-tuned
standard	CLAWQL_INFERENCE_MODEL_STANDARD → groq/llama-3.3-70b	Balanced
frontier	CLAWQL_INFERENCE_MODEL_FRONTIER → anthropic/claude-sonnet-4	Highest capability

Escalation is off by default unless CLAWQL_INFERENCE_ROUTING_ENABLED=1 or CLAWQL_INFERENCE_MODEL_PIN is set. CLAWQL_INFERENCE_MODEL_PIN=<modelId> bypasses the ladder.

Tier map overrides persist at $CLAWQL_HOME/Inference/tier-map.json (written by clawql inference finetune register or escalation set-tier).

clawql inference escalation show
clawql inference escalation set-tier --tier frugal --model ollama/phi4-custom

Ouroboros Integration

When EvolutionaryLoop runs with an AdaptiveRouter: router.initialTier() picks starting tier per seed / decomposed-child context. On generation failure (AC fail, low eval score, drift exceeded), router.escalate() moves one notch up. model_escalation audit events append to the Ouroboros Postgres / in-memory event store.

---

Semantic Cache

Layer 5 in token efficiency. Enabled with CLAWQL_INFERENCE_SEMANTIC_CACHE=1.

Setting	Default	Purpose
CLAWQL_INFERENCE_CACHE_THRESHOLD	0.92	Cosine similarity floor
CLAWQL_INFERENCE_CACHE_TTL	24h	Entry TTL
CLAWQL_INFERENCE_CACHE_MAX_ENTRIES	1000	In-memory cap
CLAWQL_EMBEDDING_MODEL	text-embedding-3-small	Embedding model

Cache hits return stored responses with cacheHit: true on inference records. Embedding failures fail open to live inference.

clawql inference cache   # show active config

---

Fallback Chains

When the primary provider/model fails, configured alternates are tried before surfacing an error.

export CLAWQL_INFERENCE_FALLBACK_ENABLED=1
export CLAWQL_INFERENCE_FALLBACK_FRUGAL=ollama/phi4,openai/gpt-4o-mini
export CLAWQL_INFERENCE_FALLBACK_STANDARD=groq/llama-3.3-70b,anthropic/claude-haiku-4

Chains also persist at $CLAWQL_HOME/Inference/fallback-chains.json. Responses include fallback.attempted and fallback.succeeded.

clawql inference fallback

---

Virtual Keys and HTTP Auth

Per-team API keys with optional USD budgets and rate limits.

export CLAWQL_INFERENCE_KEYS_ENABLED=1
clawql inference keys create --team eng --budget-usd 500 --rate-limit 100rpm
clawql inference keys list
clawql inference keys revoke --id vk_abc123

Keys persist at $CLAWQL_HOME/Inference/virtual-keys.json (secrets stored as SHA-256 hashes only). When enforcement is active, /v1/* requires a valid key; /healthz stays open.

team and virtualKeyId are stamped on inference records; spend rollups support --group-by team.

---

Plan Entitlements and Payments

The inference HTTP server integrates with clawql-payments for four independent billing modes:

Mode	Toggle	Behavior
Plan entitlements	CLAWQL_PAYMENTS_ENFORCE_INFERENCE=1	Pre-check monthly caps; 402 insufficient_quota when over limit
Stripe meters	CLAWQL_PAYMENTS_REPORT_STRIPE_METER=1	Post-call meterEvents.create
x402 pay-per-call	CLAWQL_X402_ENFORCE=1	Middleware returns 402 until facilitator verifies PAYMENT-SIGNATURE
MPP sessions	CLAWQL_MPP_ENABLED=1	Dual x402 + MPP 402 challenges; Stripe SPT / optional Tempo via mppx

Do not conflate plan usage (usage.json), inference call-store tokens (clawql inference spend), and virtual-key USD budgets — see Three usage systems.

export CLAWQL_PAYMENTS_ENFORCE_INFERENCE=1
export CLAWQL_PAYMENTS_REPORT_STRIPE_METER=1
export STRIPE_METER_EVENT_NAME=clawql_inference_calls
export CLAWQL_X402_ENFORCE=1

clawql payments plan show
clawql inference serve --port 8080

---

Inference Call Store

Every successful completion writes an InferenceRecord used by observability and export.

Backends

CLAWQL_INFERENCE_STORE	Behavior
off	No persistence
memory	In-process (tests)
jsonl	Append-only file (default when CLAWQL_HOME set)
postgres	clawql_inference_calls table (JSONB records)

Key Record Fields

Field	Purpose
id, correlation_id	Link to WORM / Ouroboros generation
model_id, provider, tier	Model and escalation tier at call time
team, virtual_key_id	Virtual key attribution
messages, response	Fine-tuning message pairs
evaluator_verdict	passed / failed / none — primary export filter
policy_version	Manifest Merkle anchor (when set)

Observability CLI

clawql inference logs [--model M] [--tier T] [--since 24h] [--limit 50]
clawql inference spend [--group-by model|tier|team|provider] [--since 7d]
clawql inference trace --correlation-id <id>

---

Export and Fine-Tuning Flywheel

Production traffic
  → WORM-logged inference (prompt, response, tier, verdict, correlation_id)
  → Evaluator verdicts + quality filters (+ OKF v0.2 verified/status filters)
  → PII-scrubbed dataset export (JSONL or PorTAL bundle)
  → Fine-tuning job (Anthropic / OpenAI) **or** PorTAL task-latent + alignment
  → Custom model / LoRA registered in ModelTierMap
  → Deployed to Frugal tier
  → Better cheap-tier results → better verdicts → better training data

PorTAL (Ramp Labs): ClawQL intends to make Flywheel adapters portable across base models via task-latent + per-base alignment. See PorTAL + Intelligence Flywheel for staged CLI shapes (--format portal-bundle, finetune refit) and trade-offs.

Local / Argo GPU training (SFT → DPO → GRPO → SPIN): When RTP/OBT traces from Harvey LAB (or OpenBench) land in the training bucket, the method-aware pipeline formats datasets, schedules Unsloth/TRL jobs, and promotes domain adapters. Spec + scaffold: Training Pipeline v0.1. GRPO is the strongest fit for Harvey LAB because rubric criteria are verifiable rewards.

Export

clawql inference export \
  --output ./training-data/export.jsonl \
  --verdict passed \
  --tier frugal \
  --format openai-jsonl \
  --min-score 0.8

Format	Target
openai-jsonl	OpenAI fine-tuning
anthropic-jsonl	Anthropic fine-tuning
raw-jsonl	Full inference records
sharegpt	Community tooling interop
portal-bundle	PorTAL task-latent + alignment (staged)

PII scrubbing (Presidio) is on by default. Every export writes a WORM dataset manifest (sample hashes, filter criteria, Merkle root, policy version).

Fine-Tune and Register

clawql inference finetune \
  --dataset ./training-data/export.jsonl \
  --base-model gpt-4o-mini \
  --provider openai

clawql inference finetune status --job-id ftjob_abc123
clawql inference finetune register --job-id ftjob_abc123 --tier frugal --alias ollama/phi4-production-v3

---

Pipeline Automation

Scheduled auto-export when sample thresholds are met.

clawql inference pipeline enable \
  --schedule "0 2 * * 0" \
  --min-samples 500 \
  --verdict passed \
  --target-tier frugal \
  --base-model gpt-4o-mini

clawql inference pipeline status
clawql inference pipeline run          # manual trigger
clawql inference pipeline worker       # cron sidecar

Config persists at $CLAWQL_HOME/Inference/pipeline.json. Cron worker: CLAWQL_INFERENCE_PIPELINE_WORKER=1 starts with inference serve.

---

Agent Coordination

TierEscalationRouter.shouldTriggerAgentCoordination() fires when combined drift exceeds 0.3, or when standard-tier is exhausted with active failure signals.

When CLAWQL_INFERENCE_AGENT_COORDINATION_ENABLED=1: evaluateAgentCoordination() builds an agent_coordination audit entry. Hermes stub runs when HERMES_BASE_URL is unset; live MoA when configured. Ouroboros appends the event to the lineage store alongside model_escalation.

---

Policy

clawql inference policy show [--json]

resolveInferencePolicy() aggregates the effective view from manifest YAML ($CLAWQL_HOME/Inference/policy.yaml or CLAWQL_INFERENCE_POLICY_MANIFEST) merged with environment variables — env wins on conflicts. Source is manifest+env when a manifest is loaded, otherwise env.

Example manifest (full copy: examples/inference/policy.yaml):

policyVersion: '2026.07.01'
inference:
  escalation:
    enabled: true
    tierMap:
      frugal: ollama/phi4
  cache:
    enabled: true
    threshold: 0.92
  pipelineWorker:
    enabled: true
    pollMs: 60000
  observability:
    profile: external

Multi-instance serve workers use Postgres advisory locks (pg_try_advisory_lock) keyed by pipeline schedule + UTC minute when CLAWQL_INFERENCE_DATABASE_URL is set, so only one replica runs each cron tick.

---

Persistence Layout

All operator state under $CLAWQL_HOME/Inference/:

File	Written by
calls.jsonl	Call store (jsonl backend)
tier-map.json	finetune register, escalation set-tier
fallback-chains.json	Fallback config merge
virtual-keys.json	keys create / revoke
pipeline.json	pipeline enable / worker ticks
policy.yaml	Operator inference policy manifest

---

CLI Wiring

clawql inference <subcommand>
├── serve              HTTP gateway (+ optional pipeline worker)
├── complete           One-shot completion
├── logs | trace | spend
├── export | finetune [status|register]
├── escalation [show|set-tier]
├── pipeline [enable|status|disable|run|worker]
├── cache | fallback | policy
└── keys [create|list|revoke]

---

Environment Variables

Variable	Default	Purpose
OPENAI_API_KEY	—	OpenAI provider
ANTHROPIC_API_KEY	—	Anthropic provider
OLLAMA_BASE_URL	http://127.0.0.1:11434	Ollama runtime
CLAWQL_INFERENCE_PORT	8080	HTTP listen port
CLAWQL_INFERENCE_ROUTING_ENABLED	off	Tier escalation
CLAWQL_INFERENCE_MODEL_FRUGAL	ollama/phi4	Frugal tier model
CLAWQL_INFERENCE_MODEL_STANDARD	groq/llama-3.3-70b	Standard tier model
CLAWQL_INFERENCE_MODEL_FRONTIER	anthropic/claude-sonnet-4	Frontier tier model
CLAWQL_INFERENCE_MODEL_PIN	—	Pin single model
CLAWQL_INFERENCE_SEMANTIC_CACHE	auto when embeddings configured	Semantic cache (Layer 5)
CLAWQL_INFERENCE_TERSE	on	Terse output post-processor (Layer 3)
CLAWQL_INFERENCE_PROMPT_CACHE	on	Provider prompt-cache markers (Layer 4)
CLAWQL_INFERENCE_HISTORY_COMPRESS	off	History distillation (Layer 6)
CLAWQL_INFERENCE_PROMPT_COMPRESS	off	Final prompt compression (Layer 7)
CLAWQL_INFERENCE_HTTP_AUTO_ROUTE	on when routing enabled	HTTP clawql/auto aliases (Layer 8)
CLAWQL_INFERENCE_STRUCTURED_OUTPUT	on	Structured output hints (Layer 9)
CLAWQL_INFERENCE_TOKEN_BUDGET	on	Token budget signaling (Layer 10)
CLAWQL_INFERENCE_PREFILL	off	Assistant prefill opener (Layer 11)
CLAWQL_INFERENCE_CACHE_THRESHOLD	0.92	Cache similarity floor
CLAWQL_INFERENCE_CACHE_TTL	24h	Cache TTL
CLAWQL_INFERENCE_FALLBACK_ENABLED	off	Fallback chains
CLAWQL_INFERENCE_KEYS_ENABLED	off	Require virtual keys
CLAWQL_INFERENCE_STORE	jsonl when CLAWQL_HOME	memory / jsonl / postgres / off
CLAWQL_INFERENCE_DATABASE_URL	—	Postgres URL
CLAWQL_INFERENCE_PIPELINE_WORKER	off	Cron worker with serve
CLAWQL_INFERENCE_AGENT_COORDINATION_ENABLED	off	Agent coordination
CLAWQL_PAYMENTS_ENFORCE_INFERENCE	off	Plan entitlement gate
CLAWQL_ENABLE_OTEL_TRACING	off	Infra OTLP spans
CLAWQL_ENABLE_LANGFUSE	on when keys set	Langfuse work-trace OTLP

Full env table: clawql-inference.md.

---

Package Exports

Import	Contents
clawql-inference	Gateway, routing, store, CLI runners, HTTP server, audit builders
clawql-inference/routing	AdaptiveRouter, tier types, config loaders
clawql-inference/plugin	Provider plugin factories, compose helpers
clawql-inference/api/server	createInferenceHttpApp for embedding

---

Differentiation vs LiteLLM

ClawQL	LiteLLM-class proxies
Outcome-driven tier escalation from agent failure signals	Static model routing
correlation_id → WORM / Ouroboros lineage	Generic request logs
Verdict-filtered export + PII scrub + dataset manifest	Ad-hoc dataset dumps
Custom models promoted back into frugal tier	One-off fine-tunes
Manifest / policy-governed tier map and export rules	Env-only config
TypeScript-native, catalog-mirrored adapters	Python proxy dependency

---

References

• Inference provider plugins
• Token efficiency architecture
• Ouroboros library

© Copyright 2026. All rights reserved. · ClawQL on GitHub