v8.2 Preview • Tail mode (--mode tail), Kustomize sidecar injection, Prometheus alerting rules, and multi-tenant deployment topology are shipping in v8.2. The Go binary, Helm chart, Terraform alarm modules, and all Kustomize overlays described in this guide are available now. pip install swt3-ai or pull ghcr.io/tenova-labs/swt3-infra-witness:8.1.0.

Who this is for: Platform engineers deploying AI governance across multiple teams, SREs responsible for compliance infrastructure, and DevOps leads evaluating SWT3 for production. Also useful for C3PAOs and assessors who need to understand how sidecar-collected evidence compares to SDK-collected evidence.

Prerequisite: Familiarity with Kubernetes Deployments, Kustomize or Helm, and Prometheus. No SWT3 SDK experience required -- that is the point.

Deploy Now

Deploy via Terraform (ECS sidecar module) • Pull the binary (swt3-infra-witness)

Assessor Notice: The infrastructure witness sidecar is a passive observer. It reads telemetry files that applications voluntarily write to a shared volume. It does not intercept, proxy, or modify network traffic. It does not read prompts, completions, or any payload content. The evidence it produces is structurally identical to evidence from SDK-instrumented services. Anchors from the sidecar and anchors from the SDK land in the same ledger, the same Merkle tree, and are verified through the same public endpoint.

Contents

1. The Coverage Gap 2. How the Sidecar Works 3. The Telemetry Format 4. Try It in 30 Seconds 5. Deploy to Kubernetes 6. What Your Security Team Will Ask 7. Observability and Alerting 8. The Auditor Cannot Tell the Difference 9. Scaling to 20 Teams 10. Honest Scope 11. Related Guides

1. The Coverage Gap

Most organizations discover the same problem around month three of an AI governance program. The teams that adopted the SDK early -- the ones who cared about compliance from the start -- have excellent evidence. Every inference is witnessed. Every model change is anchored. Their assessor is happy.

Then someone asks: "What about the other forty-five services?"

Every large AI deployment has this shape. A few teams instrumented early. Many teams have not. The uninstrumented services are not malicious or negligent. They are busy. They have roadmaps. Adding a compliance SDK to a production service requires a code change, a review, a deployment, and a conversation with a team lead who has competing priorities. Multiply that by forty-five services and you have a coverage gap that will take quarters to close.

Meanwhile, every uninstrumented inference is an evidence gap your assessor will ask about.

Assessor Tip

When reviewing an organization's AI governance posture, ask for the coverage percentage: how many AI-calling services are witnessed versus total AI-calling services? The gap between those numbers is the governance blind spot. The sidecar approach described here closes that gap at the infrastructure layer, without requiring each service team to prioritize SDK adoption.

2. How the Sidecar Works

The sidecar is a 9MB static Go binary that runs as a second container alongside your application in the same Kubernetes pod. It does one thing: it watches a shared directory for JSONL files and witnesses each line as an AI inference event.

  +-----------------+          +-----------------------+
  |   Your App      |  write   |  SWT3 Witness Sidecar |
  |                 | -------> |                       |
  | (makes AI calls,|  shared  | (reads new lines,     |
  |  appends JSONL) | emptyDir | computes fingerprint, |
  |                 |          |  witnesses via HTTPS)  |
  +-----------------+          +-----------------------+
                                        |
                                        | HTTPS (port 443)
                                        v
                                  SWT3 Witness Ledger
The app writes. The sidecar reads. Raw data stays in the pod.

This is passive observation. The sidecar never intercepts network traffic. It never sits in the request path. It never modifies or delays your application's API calls. It reads a file. If your application does not write to the shared volume, the sidecar has nothing to witness. This is a feature, not a limitation -- the application controls what is observed.

The binary runs in --mode tail, which:

Assessor Tip

The sidecar runs inside the customer's Kubernetes cluster. Telemetry files exist only in an ephemeral emptyDir volume scoped to the pod's lifetime. The sidecar reads the volume as read-only and transmits only model identifiers, token counts, latency, and status to the witness ledger. No prompt content, no completion text, and no PII reach the ledger. Ask to see the Kustomize patch to confirm the readOnly: true mount and the securityContext.

3. The Telemetry Format

Your application writes one JSON object per line to any .jsonl file in the shared volume:

{"model":"gpt-4o","tokens_in":1200,"tokens_out":400,"latency_ms":2100,"status":"ok","timestamp":"2026-11-14T15:00:00Z"}

Six fields. Only model is required.

FieldRequiredWhat It Captures
modelYesModel identifier (e.g., gpt-4o, claude-sonnet-4-20250514)
tokens_inNoInput/prompt token count
tokens_outNoOutput/completion token count
latency_msNoRequest latency in milliseconds
statusNook, error, or provider error code
timestampNoISO 8601 timestamp (defaults to current time)

Two optional fields provide additional context:

OpenTelemetry Support

If your application already emits OpenTelemetry spans with gen_ai.* semantic convention attributes, the sidecar detects and parses them automatically. No format flag needed. The sidecar extracts gen_ai.request.model, gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens, and gen_ai.system from the span attributes. Both the OTLP JSON envelope format and flat span arrays are supported.

This means teams that already have OTel instrumentation for observability purposes may not need to change anything. Their existing telemetry pipeline writes spans to a file; the sidecar reads them.

4. Try It in 30 Seconds

No account. No API key. No network calls.

# Write a test telemetry line
echo '{"model":"gpt-4o","tokens_in":500,"tokens_out":200,"latency_ms":800,"status":"ok"}' > /tmp/test.jsonl

# Point the sidecar at it
./swt3-infra-witness --mode tail --watch-dir /tmp --dry-run --poll-interval 1

# Output:
#   swt3-infra-witness tail mode
#     watching:  /tmp
#     --dry-run: events will be parsed but not sent
#
#     +  ai.inference:gpt-4o  fp: 5bed3cd3f3c6

The --dry-run flag parses and fingerprints without sending anything. No API key required. The sample events directory includes CloudTrail, Azure, GCP, and OCSF files for testing infrastructure witnessing as well.

When you are ready to witness for real, set your API key and remove the dry-run flag. Everything else stays the same.

5. Deploy to Kubernetes

Three commands. No changes to your existing Deployment manifests.

Step 1: Create the API key secret

kubectl create secret generic swt3-api-key \
  --from-literal=api-key=axm_live_YOUR_KEY \
  -n your-namespace

Step 2: Apply the Kustomize sidecar overlay

The overlay adds two things to your Deployment: a sidecar container running in tail mode, and a shared emptyDir volume.

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml  # your existing Deployment

patches:
  - path: sidecar-patch.yaml
    target:
      kind: Deployment
      name: my-ai-service

The sidecar patch is a standard Kubernetes strategic merge patch. It does not modify your application container. It adds a second container and a shared volume.

Step 3: Verify

# Check the sidecar is running
kubectl exec deploy/my-ai-service -c swt3-witness-sidecar \
  -- wget -qO- http://localhost:9091/healthz

# Expected output:
# {"ok":true,"version":"8.1.0","mode":"tail","files_watched":1,"events_processed":47}

What your app needs to do

Your application container mounts the shared volume and writes telemetry after each AI call. Here is the minimal integration:

Python:

import json, time

def log_ai_call(model, tokens_in, tokens_out, latency_ms, status="ok"):
    with open("/shared/swt3/telemetry.jsonl", "a") as f:
        f.write(json.dumps({
            "model": model, "tokens_in": tokens_in,
            "tokens_out": tokens_out, "latency_ms": latency_ms,
            "status": status,
            "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        }) + "\n")

Node.js:

const fs = require('fs');

function logAICall(model, tokensIn, tokensOut, latencyMs, status = 'ok') {
  fs.appendFileSync('/shared/swt3/telemetry.jsonl',
    JSON.stringify({
      model, tokens_in: tokensIn, tokens_out: tokensOut,
      latency_ms: latencyMs, status,
      timestamp: new Date().toISOString(),
    }) + '\n'
  );
}

One function. No SDK import. No initialization. No flush. The sidecar handles the rest.

For Helm users: If your chart supports extraContainers and extraVolumes, a ready-made values snippet ships with the witness package. See helm/swt3-witness/examples/sidecar-values.yaml.

6. What Your Security Team Will Ask

The sidecar container runs with the tightest security posture possible in Kubernetes:

PropertySettingWhy It Matters
UserrunAsNonRoot: trueCannot escalate to root even if compromised
FilesystemreadOnlyRootFilesystem: trueCannot write malware to disk
Capabilitiesdrop: ["ALL"]No NET_RAW, no SYS_ADMIN, no anything
Privilege escalationallowPrivilegeEscalation: falseCannot gain privileges via setuid binaries
Shared volumereadOnly: trueSidecar cannot write to the telemetry volume
Inbound portsNone exposed to the networkMetrics are scraped from localhost within the pod
OutboundHTTPS only (port 443)Restrict with NetworkPolicy to SWT3 endpoint + DNS
Resources25m CPU / 32Mi request, 100m CPU / 64Mi limitSmaller than most log shippers

The sidecar never reads prompts or completions. It processes the JSON telemetry your application writes -- model name, token counts, latency, status. If your application does not write prompt content to the shared volume, the sidecar physically cannot access it. The telemetry format has no field for prompt or completion text.

7. Observability and Alerting

The sidecar exposes standard Prometheus metrics on the --metrics-port (default: 9091 in the Kustomize patch):

MetricTypeWhat It Tells You
swt3_events_processed_totalCounterTotal AI inference events witnessed
swt3_events_errors_totalCounterParse failures or send errors
swt3_tail_files_watchedGaugeNumber of JSONL files being tailed
swt3_tail_bytes_read_totalCounterTotal bytes read from telemetry files
swt3_batches_sent_totalCounterWitness batch transmissions
swt3_last_witnessed_timestampGaugeUnix epoch of last successful witness
swt3_last_batch_latency_msGaugeRound-trip time for last witness batch
swt3_uptime_secondsGaugeSeconds since sidecar started

Six Prometheus alerting rules ship with the binary:

AlertSeverityFires When
SWT3EventsStaleWarningNo events witnessed in over 1 hour
SWT3HighErrorRateCriticalError rate exceeds 10% for 5 minutes
SWT3BatchLatencyHighWarningBatch latency exceeds 5 seconds for 5 minutes
SWT3RecentRestartInfoSidecar restarted within the last minute
SWT3DeduplicationHighWarningOver 50% of events are duplicates for 15 minutes
SWT3TailNoFilesWarningZero files in the watch directory for 10 minutes

Apply the rules standalone with kubectl apply -f prometheus-alerts.yml or enable them in the Helm chart with --set prometheus.rules.enabled=true.

8. The Auditor Cannot Tell the Difference

This is the point that changes how you think about the coverage problem.

An anchor produced by the sidecar and an anchor produced by the SDK are structurally identical. Same fingerprint algorithm. Same anchor token format. Same procedure (AI-INF.1). Same Merkle tree. Same verification endpoint. An assessor reviewing the witness ledger sees a table of anchors. There is no column that says "collected via SDK" or "collected via sidecar." The proof is the proof.

What this means in practice: the five services your team instrumented with the SDK and the forty-five services covered by the sidecar produce the same quality of evidence. Your assessor reviews one evidence stream, not two. Your compliance posture is not "five services governed, forty-five services pending." It is "fifty services governed."

The coverage gap closes overnight. Not in the next quarter. Not after forty-five teams prioritize it. Overnight.

Assessor Tip

If the organization uses both SDK and sidecar collection, verify that both paths produce anchors with the same procedure ID (AI-INF.1) and that both appear in the same tenant ledger. The verification endpoint does not distinguish collection method. You can verify any anchor from either path using the same public /api/v1/verify endpoint. The evidence standard is the protocol, not the collection mechanism.

9. Scaling to 20 Teams

A single sidecar covers one service. Covering twenty teams requires thinking about isolation, key management, and cost attribution. The Multi-Tenant Deployment Topology guide covers this in detail. Here is the summary:

TeamsRecommended TopologyKey Management
1-5Shared tenant, one API keyManual
5-20Hybrid: shared infra tenant + per-team AI tenantsSecret manager + automation
20+Tenant-per-team, fully automated provisioningProgrammatic tenant creation via API

In the hybrid model, each team's namespace gets its own API key, clearing level, and sidecar. Infrastructure witnessing (CloudTrail, hardware attestation) goes into a shared platform tenant. Each assessor sees only the evidence relevant to their scope.

The sidecar derives its agent_id from the Kubernetes namespace via the downward API. This means Prometheus metrics can be aggregated by namespace for per-team chargeback, and the agent ID in each anchor tells you which team produced it.

10. Honest Scope

The sidecar witnesses what your application chooses to write. It does not:

Assessor Tip

The sidecar satisfies evidence collection requirements (AU-2 Audit Events, AI-INF.1 Inference Witnessing) but does not satisfy enforcement requirements (AC-3 Access Enforcement, AI-GRD.1 Guardrail Monitoring). If the organization's system security plan claims both collection and enforcement, verify that enforcement uses the SDK's gatekeeper mode, not the sidecar. The sidecar's role is to prove what happened, not to prevent what should not happen.

11. Related Guides

SRE Reference: The RUNBOOK.md in the swt3-infra-witness package covers 11 failure modes including tail-mode sidecar troubleshooting, Lambda timeouts, DLQ accumulation, mock key detection, and Secrets Manager permissions. Share it with your on-call rotation.