Document pipeline: Nextcloud → anydoc / pdf-inspector → Docling → Tika → Gotenberg → Stirling → Paperless → Onyx → Coneshare

ClawQL does not run one hidden daemon that magically pipes bytes through all services. Instead, it loads bundled OpenAPI specs for Nextcloud, Docling (layout parse), Apache Tika, Gotenberg, Stirling-PDF, Paperless-ngx, Onyx, and ConeShare into the same search / execute index as the rest of your providers. You (or an agent) choose which operations to call and in what order—that composition is the document pipeline.

The usual production story is: intake from Nextcloud WebDAV → optional fast convert (anydoc via MCP convert_document, or pdf-inspector via inspect_pdf) → layout OCR (Docling, when scanned/forms) → extract heterogeneous files (Tika) → normalize to stable PDFs (Gotenberg) → redact/remediate (Stirling) → archive with metadata (Paperless self-hosted optional, or ClawQL archive layer — Nextcloud + Postgres + Onyx — default / hosted) → index for enterprise search (Onyx) → sync processed artifacts back to Nextcloud → share via secure links / VDR (Coneshare). Canonical recipe: DEFAULT_IDP_PIPELINE in clawql-documents — see IDP pipeline hub. Product design: IDP Platform · Full-stack Helm: clawql-idp-helm.md · Slack runbook: slack-first-idp-runbook.md.

Intro essays (background): Tika · Gotenberg · Stirling · Paperless · Onyx. Matrix: Bundled specs, providers/README.md. Helm / topology: Helm, Docker Desktop observability.

What the eight-vendor document stack is

MCP surface: ingest_external_knowledge (vault Markdown / URL) and knowledge_search_onyx (when CLAWQL_ENABLE_ONYX=1) sit beside raw execute. Optional IDP tools: run_idp_pipeline (CLAWQL_ENABLE_IDP_PIPELINE=1), convert_document (CLAWQL_ENABLE_ANYDOC=1), inspect_pdf (CLAWQL_ENABLE_PDF_INSPECTOR=1), classify_document (CLAWQL_ENABLE_IDP_CLASSIFIER=1), extract_document (CLAWQL_ENABLE_LANGEXTRACT=1). See External ingest & knowledge lake, Onyx enterprise search, Documents plugin, and mcp-tools.md.

Recommended IDP data flow

[Nextcloud IDP/inbox …]
        │  WebDAV download
        ▼
[PDF / binary / Office / email …]
        │  optional: convert_document (anydoc) — Office/PDF/CSV → GFM
        │            inspect_pdf (pdf-inspector) — TextBased → local Markdown
        │            Scanned / Mixed / OCR needed → Docling OCR
        ▼
[optional: Docling layout parse (forms, W-2, OCR pages)]
        │
        ▼
        │  Tika: detect + extract text/metadata (non-PDF or fallback)
        ▼
[optional: Office/HTML → PDF]
        │  Gotenberg: deterministic PDF artifact
        ▼
[PDF cleanup + PII redaction]
        │  Stirling: remediation profile per tenant policy
        ▼
[archive + tags + correspondent + OCR in DMS]
        │  Paperless: system of record + human workflows
        ▼
[optional: push same text/metadata into enterprise index]
        │  Onyx: ingestion API + knowledge_search_onyx for assistants
        ▼
[sync processed file back to Nextcloud]
        │  WebDAV upload to IDP/processed/…
        ▼
[secure share link / data room]
        │  Coneshare: VDR + viewer webhook → vault/audit
        ▼
Users / agents query with permissions enforced by Onyx

Why not “Onyx before Paperless”? You can call Onyx search at any time, but durable filing, legal retention, and human classification usually belong in Paperless first; Onyx then indexes what you want searchable across the org. The bundled onyx::upsert_ingestion_doc path is documented as post-Paperless for that reason (onyx-knowledge-tool.md § 5).

anydoc convert (optional, in-process)

• Use when inputs are Office, CSV, or text PDFs and you want a fast local GFM Markdown path before Docling (Firecrawl anydoc).
• In ClawQL: MCP convert_document when CLAWQL_ENABLE_ANYDOC=1 — no sidecar; @firecrawl/anydoc runs inside clawql-documents. Returns optional markdown and route: local_markdown | docling_ocr | tika_fallback.
• Paths: path= must fall under CLAWQL_ANYDOC_FILE_ROOTS (falls back to CLAWQL_PDF_INSPECTOR_FILE_ROOTS, then cwd); or pass base64.
• Helm: enableAnydoc: true (env only — no Deployment).
• Guide: anydoc-onboarding.md.

pdf-inspector route (optional, in-process)

• Use when inputs are PDFs and you want a fast local classify + Markdown path before spending Docling CPU on every file (Firecrawl pdf-inspector).
• In ClawQL: MCP inspect_pdf when CLAWQL_ENABLE_PDF_INSPECTOR=1 — no sidecar; @firecrawl/pdf-inspector runs inside clawql-documents. Returns pdf_type, optional markdown, and route: local_markdown | docling_ocr | hybrid_docling.
• Paths: path= must fall under CLAWQL_PDF_INSPECTOR_FILE_ROOTS (default: process cwd); or pass base64.
• Helm: enablePdfInspector: true (env only — no Deployment).
• Guide: pdf-inspector-onboarding.md.

Nextcloud collaboration storage

• Use when operators or upstream systems drop files in a shared folder (IDP/inbox/…) and agents should download, then upload processed output to IDP/processed/….
• In ClawQL: NEXTCLOUD_BASE_URL, NEXTCLOUD_USERNAME, NEXTCLOUD_APP_PASSWORD; search / execute on WebDAV + OCS (nextcloud-onboarding.md).
• Helm: idpCollaboration.enabled + nextcloud.enabled — enabled in values-docker-desktop.yaml for local dev.

Docling layout parse (optional)

• Use when inputs are forms, W-2s, scanned pages, or complex layouts — especially after inspect_pdf returns docling_ocr / hybrid_docling.
• In ClawQL: DOCLING_BASE_URL (+ optional DOCLING_API_KEY); search / execute on bundled Docling Serve OpenAPI (docling-onboarding.md).
• Helm: documentPipeline.docling.enabled: true (large CPU image — opt-in).

Classify and extract (optional MCP tools)

• classify_document (CLAWQL_ENABLE_IDP_CLASSIFIER=1) — document-type label + confidence via CLASSIFIER_BASE_URL or local heuristic. Helm: enableIdpClassifier + documentPipeline.classifier.
• extract_document (CLAWQL_ENABLE_LANGEXTRACT=1) — schema-grounded LangExtract fields with char_interval (+ optional HTML path). Helm: enableLangextract + documentPipeline.langextract. Compose reference: docling-classifier.compose.yml.
• Guides: fine-tuned-classifier.md · langextract-onboarding.md.

Tika text and metadata extraction

• Use when inputs are mixed formats (PDF, DOCX, EML, …) and you need plain text + metadata before routing to conversion or ML steps.
• In ClawQL: bundled spec providers/tika/openapi.yaml; set TIKA_BASE_URL (and CLAWQL_BEARER_TOKEN if your Tika server requires it per Bundled specs). search for operations like “put document”, “detect”, “parse”, then execute with the right operationId and multipart or body fields your spec exposes.
• Caveats: complex PDF layout and scanned pages still need OCR strategy outside vanilla Tika extraction (introducing-clawql-tika.md § limitations) — prefer convert_document / inspect_pdf → Docling for those.

Gotenberg normalize to PDF

• Use when you receive Office or HTML artifacts and want a single PDF representation for Stirling/Paperless.
• In ClawQL: GOTENBERG_BASE_URL + CLAWQL_BEARER_TOKEN as needed; search then execute on Chromium or LibreOffice routes from the bundled Gotenberg spec (introducing-clawql-gotenberg.md).
• Caveats: heavy CPU; fidelity vs desktop Office can differ — plan capacity and spot-check templates.

Stirling PDF remediation

• Use when PDFs are oversized, rotated wrong, merged incorrectly, or need split/compress before archival quality is acceptable (introducing-clawql-stirling.md).
• In ClawQL: STIRLING_BASE_URL + STIRLING_API_KEY as X-API-KEY; execute on the bundled Stirling paths (refresh spec from /v3/api-docs when developing — providers/README.md).
• Caveats: broad tool surface — govern which operations each workflow may call.

Paperless archive and search

• Use when you need long-lived storage, metadata taxonomy, and human-friendly browse/filter UX (introducing-clawql-paperless.md).
• In ClawQL: PAPERLESS_BASE_URL + PAPERLESS_API_TOKEN (Authorization: Token …); search / execute on document consumption, listing, and metadata APIs (paperless-onboarding.md).
• Pairing: after execute returns a new document id, optionally call onyx::upsert_ingestion_doc with a stable paperless-{id} semantic identifier (onyx-knowledge-tool.md § Post-Paperless) or enable the Ouroboros hook CLAWQL_OUROBOROS_ONYX_AFTER_PAPERLESS for automated follow-up when you run spec-first loops (Ouroboros tools).

Onyx enterprise retrieval and ingestion

• Use when the audience is the whole company (Slack, Drive, Confluence, …) and answers must respect ACLs (introducing-clawql-onyx.md).
• In ClawQL: ONYX_BASE_URL + Bearer token; optional knowledge_search_onyx for ergonomic query → search_query; execute("onyx::upsert_ingestion_doc", …) for pushing Paperless-linked text into the index (Onyx enterprise search).
• Flink / connectors: continuous sync into Onyx is a deployment concern (#119, Flink Onyx sync) — ClawQL exposes the API surface, not the connector daemons themselves.

Coneshare secure sharing and VDR

• Use when archived documents need external review under audit — secure share links, virtual data rooms, viewer analytics.
• In ClawQL: CONESHARE_BASE_URL, CONESHARE_API_TOKEN; CLAWQL_ENABLE_CONESHARE=1 registers POST /idp/coneshare/webhook for viewer events → vault/audit (coneshare-onboarding.md).
• Helm: idpCollaboration.coneshare.externalUrl for production coneshare-compose stacks; lab co-deploy is optional.

Orchestrating with search and execute

1. convert_document (when enabled) for Office/PDF/CSV → GFM, or inspect_pdf for PDF-type routing.
2. search with a natural-language query (“split pdf with stirling”, “upload document paperless”) to list operationId candidates.
3. execute with operationId, args, and optional fields to keep responses small (Using search and execute).
4. classify_document / extract_document for typed routing and grounded fields.
5. memory_ingest (or ingest_external_knowledge) for operator notes linking document ids across vendors — use enterpriseCitations when mixing in Onyx hits (clawql-memory).
6. notify for human-visible milestones when a long pipeline step finishes (Schedule & notify workflows).

No single tool replaces thinking through state: pass explicit file buffers or URLs each vendor accepts, and handle errors between steps (e.g. Gotenberg 413 → Stirling never runs).

Documents feature flag and environment

• CLAWQL_ENABLE_DOCUMENTS (default on): when set to 0, ClawQL drops document/IDP vendors from all-providers and hides document MCP tools — the default stack (no spec env) does not include Tika, Paperless, etc. anyway. See Concepts and configuration § Feature tiers. You can still set CLAWQL_BUNDLED_PROVIDERS to list only the document vendors you want.
• IDP specialist flags: CLAWQL_ENABLE_ANYDOC, CLAWQL_ENABLE_PDF_INSPECTOR, CLAWQL_ENABLE_IDP_CLASSIFIER, CLAWQL_ENABLE_LANGEXTRACT, CLAWQL_ENABLE_IDP_PIPELINE — each default off.
• Per-service base URLs and tokens live in .env.example and IDP pipeline hub — refresh committed specs with npm run fetch-provider-specs when your upstream exposes /openapi.json or Paperless /api/schema/ (providers/README.md § refresh).
• Helm: umbrella charts/clawql-idp + values-idp-full.yaml, or base enableDocuments / enableAnydoc / enablePdfInspector / enableIdpClassifier / enableLangextract / documentPipeline + idpCollaboration + onyx — Helm.

Related posts guides and references