Platform engineering guide for managing 10+ internal teams with SWT3 witness infrastructure. Three topology models, clearing level inheritance, and cost allocation.
Platform engineering teams managing 10+ internal teams running AI workloads. Assumes working knowledge of Kubernetes, Terraform workspaces, and SWT3 API key provisioning. If you are deploying for a single team, start with the SDK quickstart instead.
A typical enterprise AI program has 10-50 teams, 50-200 models, and 3-5 clearing levels. Each team ships independently. Some teams run GPT-4 for customer-facing chatbots (clearing level 2). Others run open-weight models for internal analytics (clearing level 0). The compliance posture must be unified without slowing anyone down.
The wrong approach: one shared tenant with every team's anchors in a single ledger. Problems:
Organization
|
+----------------+----------------+
| | |
Team Alpha Team Beta Team Gamma
(chatbot) (fraud ML) (analytics)
| | |
gpt-4, claude custom-ft-7b mixtral-local
clearing: 2 clearing: 2 clearing: 0
anchors: ~5K/mo anchors: ~50K/mo anchors: ~500/mo
SWT3 supports three topology models. Choose based on your regulatory requirements, team autonomy, and operational complexity budget.
| Model | Isolation | Key Management | Best For |
|---|---|---|---|
| Tenant-per-team | Full -- separate ledger, separate posture score, separate API keys | One API key per team. Platform team holds the master provisioning key. | Regulated orgs where teams have different clearing levels or different assessors |
| Shared tenant | Logical -- one ledger, teams distinguished by agent_id |
One API key shared (or one key per team pointing to same tenant) | Small orgs (<5 teams) with uniform clearing levels |
| Hybrid | Split -- central tenant for infrastructure, per-team tenants for AI models | Infrastructure key managed by platform team. AI keys managed by product teams. | Most enterprises. Infrastructure witnessing is shared; AI model witnessing is team-owned. |
The hybrid model is the most common in practice. Here is how it maps:
Platform Team Tenant (infra) Per-Team Tenants (AI)
+--------------------------+ +----------+ +----------+ +----------+
| CloudTrail witnessing | | Team A | | Team B | | Team C |
| K8s hardware attestation | | gpt-4 | | ft-7b | | mixtral |
| Network config audits | | claude | | bedrock | | local |
| Shared across all teams | | CL: 2 | | CL: 2 | | CL: 0 |
+--------------------------+ +----------+ +----------+ +----------+
| | | |
axm_live_infra_xxx axm_live_a axm_live_b axm_live_c
Assessor note
The hybrid model gives assessors the best experience. Infrastructure anchors (CloudTrail, hardware, network) are in one place for the platform review. Each product team's AI model anchors are isolated for the AI-specific assessment. The assessor never sees a 50,000-anchor ledger -- they see the slice relevant to their scope.
Every SWT3 tenant gets its own API key (prefix axm_live_). In a multi-tenant deployment, you need a key hierarchy:
# Platform team creates a new tenant for Team Delta
curl -X POST https://sovereign.tenova.io/api/auth/signup \
-H "Content-Type: application/json" \
-d '{
"email": "team-delta@corp.example.com",
"password": "GENERATED_SECURE_PASSWORD",
"org_name": "Corp - Team Delta",
"framework": "AI-RMF-100"
}'
# Response includes the API key (shown once):
# { "api_key": "axm_live_delta_xxx", "tenant_id": "uuid-..." }
To rotate a team's API key without anchor gaps:
/healthz on the sidecar)Never pass raw API keys in Terraform variables or environment files checked into git. Always use a secret manager ARN/URI. The SWT3 Terraform modules enforce this with regex validation -- raw keys are rejected at terraform plan time.
Clearing levels control what data survives in the anchor. When an organization sets a clearing level, it applies to all anchors from that tenant:
| Level | Survives | Redacted | Use Case |
|---|---|---|---|
| 0 - Analytics | Everything | Nothing | Internal analytics, dev/test |
| 1 - Standard | Model, tokens, latency, status | Actor identity hashed server-side | Production SaaS, general enterprise |
| 2 - Sensitive | Operation, region, status | Actor, resource, model details | PII-adjacent, healthcare, financial |
| 3 - Classified | Hash only | All fields | National security, defense |
In the tenant-per-team or hybrid model, each team's tenant has its own clearing level. Set it at deploy time:
# Kubernetes: per-namespace clearing level via Helm
# team-alpha namespace (chatbot, PII-adjacent)
kubectl create secret generic swt3-api-key \
--from-literal=api-key=axm_live_alpha_xxx -n team-alpha
helm install swt3-alpha tenova/swt3-witness \
--namespace team-alpha \
--set config.mode=cloud \
--set config.clearingLevel=2 \
--set cloud.apiKey=axm_live_alpha_xxx \
--set cloud.tenantId=YOUR_ALPHA_TENANT_ID
# team-gamma namespace (internal analytics)
kubectl create secret generic swt3-api-key \
--from-literal=api-key=axm_live_gamma_xxx -n team-gamma
helm install swt3-gamma tenova/swt3-witness \
--namespace team-gamma \
--set config.mode=cloud \
--set config.clearingLevel=0 \
--set cloud.apiKey=axm_live_gamma_xxx \
--set cloud.tenantId=YOUR_GAMMA_TENANT_ID
The DaemonSet chart (above) is for hardware attestation (AI-HW.1). For AI model witnessing, use the tail-mode sidecar instead -- see Section 5, Pattern B. The DaemonSet discovers GPUs and accelerators on each node; the sidecar witnesses your app's AI inference calls.
In the shared tenant model, clearing level is set once for the entire tenant. All teams inherit it. This is why the shared model is only recommended for orgs with uniform data sensitivity.
Assessor note
Ask the platform team for their clearing level matrix. Each team should have a documented justification for their clearing level. A team processing healthcare data at level 0 is a finding.
In Kubernetes, the natural isolation boundary is the namespace. SWT3 supports two DaemonSet patterns:
One DaemonSet for hardware attestation (AI-HW.1), shared across the cluster. Each pod derives its agent_id from the Kubernetes downward API:
# values.yaml for cluster-wide hardware attestation
config:
mode: cloud
agentId: "" # auto-generates from pod hostname
# The DaemonSet runs on every node regardless of namespace.
# Hardware attestation is infrastructure-level, not team-level.
For AI model witnessing, deploy the --mode tail sidecar per team namespace. The sidecar watches a shared emptyDir volume where your app writes AI telemetry JSONL. No traffic interception -- the app chooses what to emit. Each namespace gets its own API key and clearing level:
# kustomization.yaml for team-alpha namespace
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: team-alpha
resources:
- ../base # team's own Deployments
patches:
- path: ../../swt3-witness/kustomize/sidecar-overlay/sidecar-patch.yaml
target:
kind: Deployment
labelSelector: "swt3.io/witness=enabled"
Create a per-namespace secret with the team's API key:
kubectl create secret generic swt3-api-key \
--from-literal=api-key=axm_live_alpha_xxx \
-n team-alpha
The sidecar only needs egress to the SWT3 HTTPS endpoint. Lock it down with a NetworkPolicy:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: swt3-sidecar-egress
namespace: team-alpha
spec:
podSelector:
matchLabels:
swt3.io/witness: "enabled"
policyTypes: ["Egress"]
egress:
# DNS resolution (required for sovereign.tenova.io)
- to: []
ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
# SWT3 endpoint (HTTPS only)
- to:
- ipBlock:
cidr: 0.0.0.0/0 # Replace with SWT3 endpoint IP if your policy requires
ports:
- port: 443
protocol: TCP
Assessor note
For environments requiring explicit IP allowlisting, resolve sovereign.tenova.io and pin the CIDR. The sidecar makes outbound HTTPS only -- no inbound ports are required (metrics are scraped from localhost within the pod).
Terraform Modules — Five ready-to-deploy modules cover every cloud target:
AWS Lambda (CloudTrail),
AWS ECS sidecar,
AWS EKS DaemonSet,
Azure AKS,
GCP GKE.
Each module enforces secret-ARN injection (raw keys rejected at terraform plan), CloudWatch alarms, and clearing level validation.
For teams deploying SWT3 via Terraform (Lambda, ECS, Cloud Functions), use workspaces to isolate state:
terraform/
modules/ # shared SWT3 modules (git submodule or registry)
environments/
team-alpha/
main.tf # references modules/ with team-specific vars
terraform.tfvars
backend.tf # team-alpha state bucket
team-beta/
main.tf
terraform.tfvars
backend.tf # team-beta state bucket
infra-shared/
main.tf # CloudTrail Lambda (shared infrastructure)
terraform.tfvars
backend.tf
# team-alpha/terraform.tfvars
name_prefix = "team-alpha"
swt3_api_key_secret_arn = "arn:aws:secretsmanager:us-east-1:123456:secret:team-alpha/swt3-api-key"
clearing_level = 2
agent_id = "team-alpha-ecs"
enable_alarms = true
alarm_email = "team-alpha-oncall@corp.example.com"
tags = {
"team" = "alpha"
"cost-center" = "CC-1234"
"swt3:module" = "aws-ecs-sidecar"
}
# infra-shared/main.tf
module "cloudtrail_witness" {
source = "../../modules/aws-lambda-cloudtrail"
swt3_api_key_secret_arn = var.infra_api_key_secret_arn
cloudtrail_bucket_name = var.cloudtrail_bucket_name
cloudtrail_bucket_arn = var.cloudtrail_bucket_arn
binary_version = "8.1.0"
binary_sha256 = "abc123..."
agent_id = "shared-cloudtrail"
clearing_level = 1
enable_alarms = true
alarm_sns_topic_arn = var.ops_sns_topic_arn
tags = { "team" = "platform", "cost-center" = "CC-0000" }
}
State isolation is non-negotiable. Never share Terraform state across teams. A terraform destroy in one team's workspace must not affect another team's infrastructure. Use separate S3 buckets or separate state keys at minimum.
| Module | Use Case | Mode |
|---|---|---|
aws-lambda-cloudtrail | S3 CloudTrail logs via SQS to Lambda | CLI (batch) |
aws-ecs-sidecar | Container definition fragment for ECS task defs | Server (HTTP webhook) |
aws-eks-daemonset | Helm chart wrapper for EKS clusters | Hardware attestation |
azure-function | Event Grid to Azure Function | CLI (batch) |
gcp-cloud-function | Log Router to Pub/Sub to Cloud Function | CLI (batch) |
All modules support enable_alarms = true for CloudWatch/Azure Monitor alerting and output a deployment_fingerprint for portal verification.
Each SWT3 anchor costs the same regardless of content. Estimate monthly volume per team:
| Workload Type | Anchors/Month (typical) | Formula |
|---|---|---|
| AI inference (chatbot) | 5,000-50,000 | requests/day x 30 |
| AI inference (batch ML) | 500-5,000 | batch_runs x models x 30 |
| Infrastructure (CloudTrail) | 10,000-100,000 | API calls/day x 30 (deduplicated) |
| Hardware attestation | 720 | nodes x 24 (hourly) |
All SWT3 Terraform modules automatically tag resources with swt3:module and swt3:version. Add team-specific tags in your tfvars:
tags = {
"team" = "alpha"
"cost-center" = "CC-1234"
"swt3:module" = "aws-ecs-sidecar"
"swt3:version" = "8.1.0"
}
Use AWS Cost Explorer or your cloud's cost allocation tool to filter by these tags.
For K8s deployments, aggregate witness metrics by namespace (which maps to team):
# Total events witnessed per namespace (team)
sum by (namespace) (swt3_events_processed_total)
# Error rate per namespace
sum by (namespace) (rate(swt3_events_errors_total[5m]))
/ sum by (namespace) (rate(swt3_events_processed_total[5m]))
Feed these into your chargeback system. The SWT3_AGENT_ID (set to namespace via downward API) is included in every anchor's observations, making per-team attribution possible at the ledger level.
SWT3 uses a role-based model with three roles per tenant:
| Role | Can View | Can Attest | Can Export | Can Manage Keys |
|---|---|---|---|---|
| Admin | All | Yes | Yes | Yes |
| Analyst | All | Yes | Yes | No |
| Assessor | All (read-only) | No | Yes | No |
In the hybrid model, a platform engineer might be admin on the infrastructure tenant and analyst on their team's AI tenant. Users can have different roles on different tenants.
Platform Engineer Alice
infra-tenant: admin (manages CloudTrail, hardware attestation)
team-alpha: analyst (attests AI model controls for her team)
ML Engineer Bob
team-beta: analyst (attests AI model controls)
infra-tenant: (none) (no access to infrastructure tenant)
External Assessor Carol
team-alpha: assessor (read-only review of team alpha's AI posture)
team-beta: assessor (read-only review of team beta's AI posture)
API keys are scoped to a single tenant. A key for team-alpha cannot write anchors to team-beta's ledger. This is enforced server-side -- the API key resolves to a tenant ID, and all operations are scoped to that tenant.
Do not share API keys across teams. Even in the shared-tenant model, create one key per service that writes anchors. This makes key rotation and revocation granular -- you can revoke a compromised service's key without disrupting other services.
kubectl create secret generic swt3-api-key --from-literal=api-key=axm_live_xxx -n team-namespacekubectl exec ... wget -qO- http://localhost:9091/healthzUse the portfolio endpoint to monitor drift across all tenants:
# Platform team monitors all tenants
curl -s https://sovereign.tenova.io/api/v1/portfolio \
-H "Authorization: Bearer axm_live_platform_xxx" | jq '.enclaves[] | {name, score, direction}'
Set up alerts for any team whose posture score drops below threshold:
# Prometheus alert (if using the witness Prometheus metrics)
- alert: SWT3TeamPostureDegraded
expr: swt3_events_errors_total > 0 and swt3_events_processed_total == 0
for: 30m
labels:
severity: warning
annotations:
summary: "A team's witness sidecar is producing only errors"
| Teams | Recommended Topology | Key Management | Monitoring |
|---|---|---|---|
| 1-5 | Shared tenant | Manual | Dashboard |
| 5-20 | Hybrid | Secret manager + automation | Dashboard + Prometheus alerts |
| 20+ | Tenant-per-team | Fully automated provisioning | Portfolio API + PagerDuty/Opsgenie |
SRE reference
The RUNBOOK.md in the swt3-infra-witness package covers 10 failure modes including tail-mode sidecar troubleshooting, Lambda timeouts, DLQ accumulation, mock key detection, and Secrets Manager permission errors. Share it with your on-call rotation.