NATS IDP pipeline

Document JetStream is how ClawQL runs the IDP path asynchronously: a Nextcloud inbox webhook publishes to NATS, a worker Deployment calls run_idp_pipeline, and optional follow-up (Coneshare viewer, agent bridge, KEDA scale) happens without blocking MCP HTTP. This guide is the dedicated walkthrough — shorter Streams Lab 2b rehearses the same loop in the Streams context.

Canonical references: NATS IDP E2E runbook · NATS KEDA worker · IDP pipeline runner · Document pipeline (Learn) · NATS JetStream (Helm)

Before you start

1. Know the vendor stack — this path orchestrates the seven-vendor IDP chain (Document pipeline); NATS is the async bus, not a replacement for Tika/Gotenberg/Stirling.
2. Pick your consumer model:

3. Have a cluster or local NATS — in-cluster JetStream from charts/clawql-mcp (nats.enabled=true) or external nats.url.

Architecture

Nextcloud Flow / curl
        │
        ▼
POST /idp/nextcloud/webhook  ──publish──►  clawql.document.inbox.arrived
                                                    │
                                                    ▼
                              nats-worker (idpPipeline=true)
                                                    │
                              run_idp_pipeline (dry_run or live)
                                                    │
                              Stirling redact → processed upload → …
                                                    │
ConeShare automation ──► POST /idp/coneshare/webhook
                                                    │
                              clawql.document.coneshare.viewer
                                                    │
                              resume Argo + optional Slack notify
                                                    │
                              pipeline.completed / failed  ──►  agent bridge (Lab / Hermes)

• MCP HTTP publishes when nats.appIntegration.publish=true (#254).
• Worker runs clawql-automation NATS CLI — no HTTP listener on the worker pod.
• Agent bridge (#128) subscribes to completion events and calls MCP tools (memory_ingest, notify, audit).

Path A — Helm overlay (lean)

Apply the bundled example values — enables NATS, document consumers, publish, and agent bridge:

helm upgrade --install clawql ./charts/clawql-mcp \
  -f ./charts/clawql-mcp/values-nats-idp.example.yaml \
  --set envFromSecret=clawql-provider-env \
  -n clawql --create-namespace

Full umbrella: charts/clawql-idp/values-idp-full.yaml per clawql-idp-helm.md.

Secret keys (minimum)

Path B — KEDA autoscaling

When inbox volume spikes, scale worker replicas on JetStream consumer lag instead of sizing MCP HTTP for peak document load (#257).

Prerequisites:

1. KEDA installed in the cluster.
2. In-cluster NATS with JetStream — KEDA reads the NATS monitoring port (8222). External-only nats.url without a monitoring endpoint is not supported for KEDA today.
3. At least one consumer purpose: nats.worker.idpPipeline, nats.worker.coneshareFollowup, and/or enableWorkflow (HITL resume).

Enable on top of Path A:

helm upgrade --install clawql ./charts/clawql-mcp \
  -f ./charts/clawql-mcp/values-nats-idp.example.yaml \
  --set envFromSecret=clawql-provider-env \
  --set nats.keda.enabled=true \
  --set nats.keda.bootstrapConsumer=true \
  -n clawql --create-namespace

When nats.worker.enabled=true, the MCP Deployment does not set CLAWQL_NATS_ENABLE_CONSUMER — only the worker Deployment consumes.

Verify KEDA:

kubectl -n clawql get deploy,scaledobject | rg nats
kubectl -n clawql describe scaledobject clawql-mcp-http-nats-worker   # name may match release
kubectl -n clawql port-forward svc/clawql-mcp-http-nats 8222:8222 &
curl -s http://127.0.0.1:8222/jsz | head -c 400

Post a dry-run inbox webhook (Path C), then watch worker replicas rise when lag exceeds lagThreshold. Add more durables via nats.keda.extraConsumers if you add custom subjects.

Deep reference: nats-keda-worker.md.

Path C — Smoke tests

Offline (Helm render only)

SMOKE_HELM_ONLY=1 bash scripts/dev/smoke-nats-idp-webhooks.sh

Asserts rendered env for IDP + Coneshare consumers and KEDA ScaledObject triggers when nats.keda.enabled=true.

Live webhook smoke

kubectl -n clawql port-forward svc/clawql-mcp-http 8080:8080 &

export CLAWQL_HTTP_BASE=http://127.0.0.1:8080
export CLAWQL_NEXTCLOUD_WEBHOOK_TOKEN=…   # from Secret
export CLAWQL_CONESHARE_WEBHOOK_TOKEN=…

bash scripts/dev/smoke-nats-idp-webhooks.sh

kubectl -n clawql logs deploy/clawql-mcp-http-nats-worker --tail=80

Inbox payloads use dry_run: true by default in the smoke script — worker plans hops without calling vendors. Expect clawql.document.inbox.arrived consumption and terminal pipeline.completed when publish is on.

Live pipeline (non-dry-run)

1. Place a PDF at Nextcloud IDP/inbox/… (or use lending W-2 fixtures).
2. POST webhook without dry_run (or dry_run: false) with document_path + redact_list.
3. Confirm Stirling hop and processed file at processed_path.
4. Optional: Coneshare viewer webhook → clawql.document.coneshare.viewer → resume + Slack.

Path D — Agent bridge (Hermes / Pi)

Terminal agents should not invent a second queue — subscribe to JetStream completion events and call the same MCP HTTP surface:

export CLAWQL_NATS_URL=nats://127.0.0.1:4222
export CLAWQL_NATS_JETSTREAM=1
export CLAWQL_NATS_ENABLE_CONSUMER=1
export CLAWQL_NATS_AGENT_BRIDGE=1
export CLAWQL_MCP_HTTP_URL=http://127.0.0.1:8080/mcp

npm run nats:agent-bridge

Durable consumer: clawql-idp-agent-bridge (override: CLAWQL_NATS_AGENT_BRIDGE_DURABLE). Helm: nats.agentBridge.enabled=true in the IDP overlay.

Sample pack: deployment/samples/idp-nats-agent · Contract: clawql-agent-idp-nats.md.

Subjects and durables

Lifecycle (publish-only): clawql.document.pipeline.hop|completed|failed.

Local / non-Helm

# Terminal 1 — MCP with publish + webhooks
export CLAWQL_NATS_URL=nats://localhost:4222
export CLAWQL_NATS_JETSTREAM=1
export CLAWQL_NATS_ENABLE_PUBLISH=1
export CLAWQL_ENABLE_IDP_PIPELINE=1
export CLAWQL_NEXTCLOUD_WEBHOOK_TOKEN=dev-token
PORT=8080 npx -p clawql-mcp clawql-mcp-http

# Terminal 2 — standalone worker
export CLAWQL_NATS_ENABLE_CONSUMER=1
export CLAWQL_NATS_CONSUMER_IDP_PIPELINE=1
export CLAWQL_NATS_CONSUMER_CONESHARE_FOLLOWUP=1
export CLAWQL_MCP_INTERNAL_URL=http://localhost:8080
export CLAWQL_IDP_PIPELINE_RUN_TOKEN=dev-token
npm run nats:worker

Bootstrap durables before KEDA or first consume:

export CLAWQL_NATS_CONSUMER_IDP_PIPELINE=1
export CLAWQL_NATS_CONSUMER_CONESHARE_FOLLOWUP=1
npm run nats:bootstrap-consumer

Troubleshooting

What to read next

• Streams getting started — Labs 1–4 in the broader Streams narrative
• Document pipeline — vendor composition inside run_idp_pipeline
• NATS JetStream — chart values, subject taxonomy, monitor port
• Schedule & notify — Slack on pipeline failure via notify