Docker Desktop: Istio and the observability stack

When you install ClawQL on Docker Desktop Kubernetes with CLAWQL_LOCAL_K8S_ISTIO=ambient or sidecar, the repo also brings up a local observability lab in the istio-system namespace: Prometheus, Grafana, Jaeger, Kiali, and a small OpenTelemetry Collector that forwards app traces to Jaeger.

Canonical doc (same content, PR-friendly, deeper detail): docs/deployment/docker-desktop-istio-observability.md. Install commands and env toggles: docker/README.md. Kubernetes overview: Kubernetes.

One command (recap)

CLAWQL_LOCAL_K8S_ISTIO=ambient make local-k8s-up

Skip all Istio sample addons: CLAWQL_ISTIO_INSTALL_KIALI=0
Keep Prometheus + Kiali but skip Grafana, Jaeger, and the OTel collector (lighter RAM): CLAWQL_ISTIO_INSTALL_HEAVY_OBSERVABILITY_ADDONS=0

Where everything lives

Tool	Namespace	Reach it
Prometheus	`istio-system`	`kubectl port-forward -n istio-system svc/prometheus 9090:9090` → http://localhost:9090
Grafana	`istio-system`	`kubectl port-forward -n istio-system svc/grafana 3000:3000` → http://localhost:3000
Jaeger	`istio-system`	`kubectl port-forward -n istio-system svc/tracing 16686:80` → http://localhost:16686
Kiali	`istio-system`	`kubectl port-forward -n istio-system svc/kiali 20001:20001` → http://localhost:20001/kiali
OTel Collector (ClawQL)	`istio-system`	In-cluster DNS `clawql-otel-collector.istio-system.svc` ports 4317 / 4318 (no browser UI)
ClawQL MCP metrics	`clawql`	`GET /metrics` on the MCP HTTP port (OpenMetrics), separate from mesh Prometheus

Use kubectl --context docker-desktop if your default context is not Docker Desktop.

Prometheus — getting started (new users)

Idea: a time-series database plus a query UI. You write PromQL to ask questions like “how many requests per second?”

Run the port-forward from the table above.
Open http://localhost:9090.
Run the query up — confirms scrape targets are reachable.
Try istio_requests_total after you generate traffic to the mesh (for example MCP or the docs UI).

Grafana — getting started (new users)

Idea: dashboards on top of metrics. Grafana reads Prometheus (and other sources) and draws charts.

Port-forward 3000 (see table).
Open http://localhost:3000.
Log in with the Istio sample defaults: admin / admin (change this outside a throwaway lab).
Open Dashboards and pick an Istio dashboard folder, then generate traffic and watch panels update.

Jaeger — getting started (new users)

Idea: distributed tracing — one request, many services, one trace made of spans with timings.

Port-forward 16686:80 to svc/tracing (see table).
Open http://localhost:16686.
Use Search → pick a Service → Find Traces.
Open a trace to see the waterfall (span timeline).

ClawQL MCP spans: set CLAWQL_ENABLE_OTEL_TRACING=1 and OTEL_EXPORTER_OTLP_ENDPOINT=http://clawql-otel-collector.istio-system.svc:4318/v1/traces on clawql-mcp-http (Helm extraEnv or equivalent). See commented block in charts/clawql-mcp/values-docker-desktop.yaml.

Kiali — getting started (new users)

Idea: Istio console — who talks to whom, traffic rates, and config health.

Port-forward 20001 (see table).
Open http://localhost:20001/kiali.
Go to Graph, namespace clawql, and explore the service graph.
Click a service for details and links to related signals.

OpenTelemetry Collector — getting started (new users)

Idea: a small relay: apps send OTLP; the collector forwards to Jaeger over OTLP gRPC. You usually do not open a browser for it.

Confirm it exists: kubectl -n istio-system get deploy,svc clawql-otel-collector.
Point the MCP pod at http://clawql-otel-collector.istio-system.svc:4318/v1/traces (HTTP OTLP) as above.
Confirm traces in Jaeger after calling MCP tools.

ClawQL metrics vs mesh metrics

GET /metrics on clawql-mcp-http = application OpenMetrics (native protocol gauges/counters, etc.) — see Deployment.
Prometheus in istio-system = mesh and platform scrapes (Istio sample config). Different scrape targets, same Prometheus query language.