Witness AI inference from 50 microservices without modifying application code. A passive sidecar reads telemetry from a shared volume. Deploy in three commands.
v8.2 Preview • Tail mode (--mode tail), Kustomize sidecar injection, Prometheus alerting rules, and multi-tenant deployment topology are shipping in v8.2. The Go binary, Helm chart, Terraform alarm modules, and all Kustomize overlays described in this guide are available now. pip install swt3-ai or pull ghcr.io/tenova-labs/swt3-infra-witness:8.1.0.
Who this is for: Platform engineers deploying AI governance across multiple teams, SREs responsible for compliance infrastructure, and DevOps leads evaluating SWT3 for production. Also useful for C3PAOs and assessors who need to understand how sidecar-collected evidence compares to SDK-collected evidence.
Prerequisite: Familiarity with Kubernetes Deployments, Kustomize or Helm, and Prometheus. No SWT3 SDK experience required -- that is the point.
Assessor Notice: The infrastructure witness sidecar is a passive observer. It reads telemetry files that applications voluntarily write to a shared volume. It does not intercept, proxy, or modify network traffic. It does not read prompts, completions, or any payload content. The evidence it produces is structurally identical to evidence from SDK-instrumented services. Anchors from the sidecar and anchors from the SDK land in the same ledger, the same Merkle tree, and are verified through the same public endpoint.
Most organizations discover the same problem around month three of an AI governance program. The teams that adopted the SDK early -- the ones who cared about compliance from the start -- have excellent evidence. Every inference is witnessed. Every model change is anchored. Their assessor is happy.
Then someone asks: "What about the other forty-five services?"
Every large AI deployment has this shape. A few teams instrumented early. Many teams have not. The uninstrumented services are not malicious or negligent. They are busy. They have roadmaps. Adding a compliance SDK to a production service requires a code change, a review, a deployment, and a conversation with a team lead who has competing priorities. Multiply that by forty-five services and you have a coverage gap that will take quarters to close.
Meanwhile, every uninstrumented inference is an evidence gap your assessor will ask about.
When reviewing an organization's AI governance posture, ask for the coverage percentage: how many AI-calling services are witnessed versus total AI-calling services? The gap between those numbers is the governance blind spot. The sidecar approach described here closes that gap at the infrastructure layer, without requiring each service team to prioritize SDK adoption.
The sidecar is a 9MB static Go binary that runs as a second container alongside your application in the same Kubernetes pod. It does one thing: it watches a shared directory for JSONL files and witnesses each line as an AI inference event.
+-----------------+ +-----------------------+
| Your App | write | SWT3 Witness Sidecar |
| | -------> | |
| (makes AI calls,| shared | (reads new lines, |
| appends JSONL) | emptyDir | computes fingerprint, |
| | | witnesses via HTTPS) |
+-----------------+ +-----------------------+
|
| HTTPS (port 443)
v
SWT3 Witness Ledger
This is passive observation. The sidecar never intercepts network traffic. It never sits in the request path. It never modifies or delays your application's API calls. It reads a file. If your application does not write to the shared volume, the sidecar has nothing to witness. This is a feature, not a limitation -- the application controls what is observed.
The binary runs in --mode tail, which:
.jsonl and .json files/healthz and /metrics endpoints for liveness probes and Prometheus scrapingThe sidecar runs inside the customer's Kubernetes cluster. Telemetry files exist only in an ephemeral emptyDir volume scoped to the pod's lifetime. The sidecar reads the volume as read-only and transmits only model identifiers, token counts, latency, and status to the witness ledger. No prompt content, no completion text, and no PII reach the ledger. Ask to see the Kustomize patch to confirm the readOnly: true mount and the securityContext.
Your application writes one JSON object per line to any .jsonl file in the shared volume:
{"model":"gpt-4o","tokens_in":1200,"tokens_out":400,"latency_ms":2100,"status":"ok","timestamp":"2026-11-14T15:00:00Z"}
Six fields. Only model is required.
| Field | Required | What It Captures |
|---|---|---|
model | Yes | Model identifier (e.g., gpt-4o, claude-sonnet-4-20250514) |
tokens_in | No | Input/prompt token count |
tokens_out | No | Output/completion token count |
latency_ms | No | Request latency in milliseconds |
status | No | ok, error, or provider error code |
timestamp | No | ISO 8601 timestamp (defaults to current time) |
Two optional fields provide additional context:
agent_id -- service identifier, overrides the pod-level agent ID. Useful when multiple services write to the same volume.request_id -- unique request identifier for deduplication. The sidecar skips lines it has already processed.If your application already emits OpenTelemetry spans with gen_ai.* semantic convention attributes, the sidecar detects and parses them automatically. No format flag needed. The sidecar extracts gen_ai.request.model, gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens, and gen_ai.system from the span attributes. Both the OTLP JSON envelope format and flat span arrays are supported.
This means teams that already have OTel instrumentation for observability purposes may not need to change anything. Their existing telemetry pipeline writes spans to a file; the sidecar reads them.
No account. No API key. No network calls.
# Write a test telemetry line
echo '{"model":"gpt-4o","tokens_in":500,"tokens_out":200,"latency_ms":800,"status":"ok"}' > /tmp/test.jsonl
# Point the sidecar at it
./swt3-infra-witness --mode tail --watch-dir /tmp --dry-run --poll-interval 1
# Output:
# swt3-infra-witness tail mode
# watching: /tmp
# --dry-run: events will be parsed but not sent
#
# + ai.inference:gpt-4o fp: 5bed3cd3f3c6
The --dry-run flag parses and fingerprints without sending anything. No API key required. The sample events directory includes CloudTrail, Azure, GCP, and OCSF files for testing infrastructure witnessing as well.
When you are ready to witness for real, set your API key and remove the dry-run flag. Everything else stays the same.
Three commands. No changes to your existing Deployment manifests.
kubectl create secret generic swt3-api-key \
--from-literal=api-key=axm_live_YOUR_KEY \
-n your-namespace
The overlay adds two things to your Deployment: a sidecar container running in tail mode, and a shared emptyDir volume.
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml # your existing Deployment
patches:
- path: sidecar-patch.yaml
target:
kind: Deployment
name: my-ai-service
The sidecar patch is a standard Kubernetes strategic merge patch. It does not modify your application container. It adds a second container and a shared volume.
# Check the sidecar is running
kubectl exec deploy/my-ai-service -c swt3-witness-sidecar \
-- wget -qO- http://localhost:9091/healthz
# Expected output:
# {"ok":true,"version":"8.1.0","mode":"tail","files_watched":1,"events_processed":47}
Your application container mounts the shared volume and writes telemetry after each AI call. Here is the minimal integration:
Python:
import json, time
def log_ai_call(model, tokens_in, tokens_out, latency_ms, status="ok"):
with open("/shared/swt3/telemetry.jsonl", "a") as f:
f.write(json.dumps({
"model": model, "tokens_in": tokens_in,
"tokens_out": tokens_out, "latency_ms": latency_ms,
"status": status,
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
}) + "\n")
Node.js:
const fs = require('fs');
function logAICall(model, tokensIn, tokensOut, latencyMs, status = 'ok') {
fs.appendFileSync('/shared/swt3/telemetry.jsonl',
JSON.stringify({
model, tokens_in: tokensIn, tokens_out: tokensOut,
latency_ms: latencyMs, status,
timestamp: new Date().toISOString(),
}) + '\n'
);
}
One function. No SDK import. No initialization. No flush. The sidecar handles the rest.
For Helm users: If your chart supports extraContainers and extraVolumes, a ready-made values snippet ships with the witness package. See helm/swt3-witness/examples/sidecar-values.yaml.
The sidecar container runs with the tightest security posture possible in Kubernetes:
| Property | Setting | Why It Matters |
|---|---|---|
| User | runAsNonRoot: true | Cannot escalate to root even if compromised |
| Filesystem | readOnlyRootFilesystem: true | Cannot write malware to disk |
| Capabilities | drop: ["ALL"] | No NET_RAW, no SYS_ADMIN, no anything |
| Privilege escalation | allowPrivilegeEscalation: false | Cannot gain privileges via setuid binaries |
| Shared volume | readOnly: true | Sidecar cannot write to the telemetry volume |
| Inbound ports | None exposed to the network | Metrics are scraped from localhost within the pod |
| Outbound | HTTPS only (port 443) | Restrict with NetworkPolicy to SWT3 endpoint + DNS |
| Resources | 25m CPU / 32Mi request, 100m CPU / 64Mi limit | Smaller than most log shippers |
The sidecar never reads prompts or completions. It processes the JSON telemetry your application writes -- model name, token counts, latency, status. If your application does not write prompt content to the shared volume, the sidecar physically cannot access it. The telemetry format has no field for prompt or completion text.
The sidecar exposes standard Prometheus metrics on the --metrics-port (default: 9091 in the Kustomize patch):
| Metric | Type | What It Tells You |
|---|---|---|
swt3_events_processed_total | Counter | Total AI inference events witnessed |
swt3_events_errors_total | Counter | Parse failures or send errors |
swt3_tail_files_watched | Gauge | Number of JSONL files being tailed |
swt3_tail_bytes_read_total | Counter | Total bytes read from telemetry files |
swt3_batches_sent_total | Counter | Witness batch transmissions |
swt3_last_witnessed_timestamp | Gauge | Unix epoch of last successful witness |
swt3_last_batch_latency_ms | Gauge | Round-trip time for last witness batch |
swt3_uptime_seconds | Gauge | Seconds since sidecar started |
Six Prometheus alerting rules ship with the binary:
| Alert | Severity | Fires When |
|---|---|---|
SWT3EventsStale | Warning | No events witnessed in over 1 hour |
SWT3HighErrorRate | Critical | Error rate exceeds 10% for 5 minutes |
SWT3BatchLatencyHigh | Warning | Batch latency exceeds 5 seconds for 5 minutes |
SWT3RecentRestart | Info | Sidecar restarted within the last minute |
SWT3DeduplicationHigh | Warning | Over 50% of events are duplicates for 15 minutes |
SWT3TailNoFiles | Warning | Zero files in the watch directory for 10 minutes |
Apply the rules standalone with kubectl apply -f prometheus-alerts.yml or enable them in the Helm chart with --set prometheus.rules.enabled=true.
This is the point that changes how you think about the coverage problem.
An anchor produced by the sidecar and an anchor produced by the SDK are structurally identical. Same fingerprint algorithm. Same anchor token format. Same procedure (AI-INF.1). Same Merkle tree. Same verification endpoint. An assessor reviewing the witness ledger sees a table of anchors. There is no column that says "collected via SDK" or "collected via sidecar." The proof is the proof.
What this means in practice: the five services your team instrumented with the SDK and the forty-five services covered by the sidecar produce the same quality of evidence. Your assessor reviews one evidence stream, not two. Your compliance posture is not "five services governed, forty-five services pending." It is "fifty services governed."
The coverage gap closes overnight. Not in the next quarter. Not after forty-five teams prioritize it. Overnight.
If the organization uses both SDK and sidecar collection, verify that both paths produce anchors with the same procedure ID (AI-INF.1) and that both appear in the same tenant ledger. The verification endpoint does not distinguish collection method. You can verify any anchor from either path using the same public /api/v1/verify endpoint. The evidence standard is the protocol, not the collection mechanism.
A single sidecar covers one service. Covering twenty teams requires thinking about isolation, key management, and cost attribution. The Multi-Tenant Deployment Topology guide covers this in detail. Here is the summary:
| Teams | Recommended Topology | Key Management |
|---|---|---|
| 1-5 | Shared tenant, one API key | Manual |
| 5-20 | Hybrid: shared infra tenant + per-team AI tenants | Secret manager + automation |
| 20+ | Tenant-per-team, fully automated provisioning | Programmatic tenant creation via API |
In the hybrid model, each team's namespace gets its own API key, clearing level, and sidecar. Infrastructure witnessing (CloudTrail, hardware attestation) goes into a shared platform tenant. Each assessor sees only the evidence relevant to their scope.
The sidecar derives its agent_id from the Kubernetes namespace via the downward API. This means Prometheus metrics can be aggregated by namespace for per-team chargeback, and the agent ID in each anchor tells you which team produced it.
The sidecar witnesses what your application chooses to write. It does not:
strict=True).The sidecar satisfies evidence collection requirements (AU-2 Audit Events, AI-INF.1 Inference Witnessing) but does not satisfy enforcement requirements (AC-3 Access Enforcement, AI-GRD.1 Guardrail Monitoring). If the organization's system security plan claims both collection and enforcement, verify that enforcement uses the SDK's gatekeeper mode, not the sidecar. The sidecar's role is to prove what happened, not to prevent what should not happen.
RUNBOOK.md in the swt3-infra-witness package covers 11 failure modes including tail-mode sidecar troubleshooting, Lambda timeouts, DLQ accumulation, mock key detection, and Secrets Manager permissions. Share it with your on-call rotation.