Secrets at Rest: Vault Integration, HSM Backing, and Tamper-Proof Audit Logging

Vault Integration, HSM Backing, and Tamper-Proof Audit Logging

Hello and welcome to Module 8!

Modules 1–7 have given us trusted images, cluster admission, vetted skills, zero-trust networking, a hardened gateway, strict egress controls, and scoped Kubernetes identities. Now we eliminate the single biggest source of long-term risk in any platform: static secrets.

A secret written to a config file, an environment variable, or a Kubernetes Secret object is no longer a secret — it is simply data waiting to be discovered. In an agentic platform where pods run autonomous code for hours or days, static credentials create breach windows measured in months.

In this module we replace every static secret with dynamic, short-lived credentials issued by HashiCorp Vault. We make Vault highly available, tamper-evident, and impossible for agents to access directly. By the end you will have a secrets architecture where credentials exist for minutes, not months, and every access is permanently recorded in a forensic-grade audit trail.

Why Static Secrets Are a Systemic Risk

Static secrets create three unavoidable problems:

Any pod with the right RBAC (or any attacker who reaches etcd) can read Kubernetes Secrets — they are only base64-encoded, not encrypted.
Long-lived credentials accumulate over time: every developer, CI runner, or former team member who ever had access retains the secret until it is rotated.
When a breach occurs, the window of exposure is measured in months for static secrets versus minutes for dynamic ones.

Agentic platforms make this worse: an agent with a static credential in its environment can use it for every tool call across dozens of sessions. We must eliminate the static credential class of risk entirely.

Vault Architecture for ClawQL

Vault becomes the single source of truth for every credential the platform uses.

Core design principles:

Dynamic secrets only: Vault generates a unique credential per request with a short, configurable TTL.
No static credential ever exists in any config file, environment variable, Kubernetes Secret, or agent memory.
Vault runs in high-availability mode using Raft integrated storage — no external database dependency.
Minimum of three replicas in production (Raft requires a quorum of (n/2)+1 for writes).

This architecture ensures that even if an attacker compromises a pod, there is nothing static for them to steal.

HSM-Backed Unsealing

Vault’s master key encrypts the keyring that protects all data. If that master key is compromised, everything is lost.

We solve this with a Hardware Security Module (HSM):

The master key is stored in tamper-resistant hardware (never on disk).
Vault uses Auto Unseal with AWS KMS, GCP Cloud KMS, or Azure Key Vault as the HSM provider.
Vault can now restart automatically without any human key-share holders — critical for automated disaster recovery (Module 28).

For break-glass scenarios we still support Shamir secret sharing, but it requires multiple authorized key holders to be physically present simultaneously. HSM-backed unsealing removes the human as a single point of failure for availability while keeping the highest level of protection.

Dynamic Secrets for ClawQL Components

Every credential the platform needs is generated on demand:

Database credentials: Vault creates a unique username/password per agent session; TTL equals the session duration.
Cloud credentials: Vault’s AWS/GCP secrets engines generate temporary IAM roles or service-account tokens exactly when needed.
PKI certificates: Vault’s PKI engine issues short-lived TLS certificates for mTLS (replacing cert-manager for internal certificates).

Each dynamic secret is tied to a specific Vault lease. Revocation is instant and complete — the moment a session ends or an anomaly is detected, the lease is revoked and the credential becomes useless everywhere.

Token Exchange at the Gateway

Agents never hold Vault tokens directly — that would defeat the purpose.

The secure exchange pattern works as follows:

The agent presents its session JWT to the gateway.
The gateway exchanges the JWT for a short-lived Vault token scoped to that agent’s exact policy.
The Vault token TTL matches the session duration and expires automatically when the session ends.
The token is never exposed to the agent, never logged, and never stored in memory beyond the duration of the specific tool call.

This “gateway-as-exchange-point” design means a compromised agent has no direct path to Vault.

Tamper-Proof Audit Logging

Every interaction with Vault is recorded forever.

Vault’s audit device emits structured JSON logs of every API call (who requested what, when, from where).
Logs are shipped via Fluent Bit to WORM storage (S3 Object Lock in COMPLIANCE mode).
Presidio redaction is applied before shipping — secret values themselves never appear; only the paths are logged.
A Merkle root of the audit log is computed and recorded in Prometheus every 60 seconds.

This audit trail becomes the ground truth for any investigation: “Did this agent access this secret?” The combination of WORM immutability and Merkle integrity means the log cannot be altered or deleted, even by root.

Vault Policy Structure

Policies are minimal and explicit:

One policy per agent role.
Path patterns are tightly scoped: secret/tenants//agents//\* — never broad wildcards across tenants.
Explicit deny rules override any allow rule for sensitive paths.
Any policy change requires the same 4-eyes approval process used for ATR rule changes (Module 30).

Key Takeaways (Memorize These!)

Dynamic secrets with short TTLs are not just a best practice — they eliminate the long-lived credential as an attack category entirely.
HSM-backed unsealing removes the human key holder as a single point of failure for Vault availability.
The Vault audit log to WORM storage is the forensic foundation for every credential-related security investigation.
The gateway-as-exchange-point pattern ensures agents never hold Vault tokens — a compromised agent cannot directly access Vault.

You now have a secrets architecture where credentials are created on demand, live for minutes, and every access is permanently and verifiably recorded. Static secrets are gone. The last major class of long-term credential risk has been closed.