Platform engineering teams managing 10+ internal teams running AI workloads. Assumes working knowledge of Kubernetes, Terraform workspaces, and SWT3 API key provisioning. If you are deploying for a single team, start with the SDK quickstart instead.

Contents

1. The Multi-Tenant Challenge 2. Topology Models 3. API Key Partitioning 4. Clearing Level Inheritance 5. Kubernetes Namespace Isolation 6. Terraform Workspace Patterns 7. Cost Allocation 8. RBAC Model 9. Operational Runbook

1. The Multi-Tenant Challenge

A typical enterprise AI program has 10-50 teams, 50-200 models, and 3-5 clearing levels. Each team ships independently. Some teams run GPT-4 for customer-facing chatbots (clearing level 2). Others run open-weight models for internal analytics (clearing level 0). The compliance posture must be unified without slowing anyone down.

The wrong approach: one shared tenant with every team's anchors in a single ledger. Problems:

                        Organization
                             |
            +----------------+----------------+
            |                |                |
        Team Alpha       Team Beta       Team Gamma
        (chatbot)       (fraud ML)       (analytics)
            |                |                |
      gpt-4, claude     custom-ft-7b     mixtral-local
      clearing: 2       clearing: 2      clearing: 0
      anchors: ~5K/mo   anchors: ~50K/mo anchors: ~500/mo
Three teams, three clearing levels, three anchor volumes. One compliance posture.

2. Topology Models

SWT3 supports three topology models. Choose based on your regulatory requirements, team autonomy, and operational complexity budget.

ModelIsolationKey ManagementBest For
Tenant-per-team Full -- separate ledger, separate posture score, separate API keys One API key per team. Platform team holds the master provisioning key. Regulated orgs where teams have different clearing levels or different assessors
Shared tenant Logical -- one ledger, teams distinguished by agent_id One API key shared (or one key per team pointing to same tenant) Small orgs (<5 teams) with uniform clearing levels
Hybrid Split -- central tenant for infrastructure, per-team tenants for AI models Infrastructure key managed by platform team. AI keys managed by product teams. Most enterprises. Infrastructure witnessing is shared; AI model witnessing is team-owned.

Hybrid Model (Recommended)

The hybrid model is the most common in practice. Here is how it maps:

  Platform Team Tenant (infra)          Per-Team Tenants (AI)
  +--------------------------+     +----------+  +----------+  +----------+
  | CloudTrail witnessing    |     | Team A   |  | Team B   |  | Team C   |
  | K8s hardware attestation |     | gpt-4    |  | ft-7b    |  | mixtral  |
  | Network config audits    |     | claude   |  | bedrock  |  | local    |
  | Shared across all teams  |     | CL: 2    |  | CL: 2    |  | CL: 0   |
  +--------------------------+     +----------+  +----------+  +----------+
           |                              |            |            |
     axm_live_infra_xxx             axm_live_a    axm_live_b   axm_live_c
Hybrid topology: shared infrastructure tenant + per-team AI tenants

Assessor note
The hybrid model gives assessors the best experience. Infrastructure anchors (CloudTrail, hardware, network) are in one place for the platform review. Each product team's AI model anchors are isolated for the AI-specific assessment. The assessor never sees a 50,000-anchor ledger -- they see the slice relevant to their scope.

3. API Key Partitioning

Every SWT3 tenant gets its own API key (prefix axm_live_). In a multi-tenant deployment, you need a key hierarchy:

  1. Provisioning key -- held by the platform team. Used to create new tenants via the self-serve signup API.
  2. Team keys -- one per team, scoped to their tenant. Distributed via secret management (K8s Secrets, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager).
  3. Signing keys -- optional HMAC-SHA256 keys for payload non-repudiation. One per team if used.

Programmatic Tenant Creation

# Platform team creates a new tenant for Team Delta
curl -X POST https://sovereign.tenova.io/api/auth/signup \
  -H "Content-Type: application/json" \
  -d '{
    "email": "team-delta@corp.example.com",
    "password": "GENERATED_SECURE_PASSWORD",
    "org_name": "Corp - Team Delta",
    "framework": "AI-RMF-100"
  }'

# Response includes the API key (shown once):
# { "api_key": "axm_live_delta_xxx", "tenant_id": "uuid-..." }

Key Rotation

To rotate a team's API key without anchor gaps:

  1. Generate new key via dashboard Settings page (admin role)
  2. Update the secret in your secret manager (K8s Secret, Secrets Manager, etc.)
  3. Rolling restart the team's pods/functions -- they pick up the new key
  4. Revoke the old key after confirming the new key is active (check /healthz on the sidecar)

Never pass raw API keys in Terraform variables or environment files checked into git. Always use a secret manager ARN/URI. The SWT3 Terraform modules enforce this with regex validation -- raw keys are rejected at terraform plan time.

4. Clearing Level Inheritance

Clearing levels control what data survives in the anchor. When an organization sets a clearing level, it applies to all anchors from that tenant:

LevelSurvivesRedactedUse Case
0 - AnalyticsEverythingNothingInternal analytics, dev/test
1 - StandardModel, tokens, latency, statusActor identity hashed server-sideProduction SaaS, general enterprise
2 - SensitiveOperation, region, statusActor, resource, model detailsPII-adjacent, healthcare, financial
3 - ClassifiedHash onlyAll fieldsNational security, defense

Setting per-team clearing levels

In the tenant-per-team or hybrid model, each team's tenant has its own clearing level. Set it at deploy time:

# Kubernetes: per-namespace clearing level via Helm
# team-alpha namespace (chatbot, PII-adjacent)
kubectl create secret generic swt3-api-key \
  --from-literal=api-key=axm_live_alpha_xxx -n team-alpha

helm install swt3-alpha tenova/swt3-witness \
  --namespace team-alpha \
  --set config.mode=cloud \
  --set config.clearingLevel=2 \
  --set cloud.apiKey=axm_live_alpha_xxx \
  --set cloud.tenantId=YOUR_ALPHA_TENANT_ID

# team-gamma namespace (internal analytics)
kubectl create secret generic swt3-api-key \
  --from-literal=api-key=axm_live_gamma_xxx -n team-gamma

helm install swt3-gamma tenova/swt3-witness \
  --namespace team-gamma \
  --set config.mode=cloud \
  --set config.clearingLevel=0 \
  --set cloud.apiKey=axm_live_gamma_xxx \
  --set cloud.tenantId=YOUR_GAMMA_TENANT_ID

The DaemonSet chart (above) is for hardware attestation (AI-HW.1). For AI model witnessing, use the tail-mode sidecar instead -- see Section 5, Pattern B. The DaemonSet discovers GPUs and accelerators on each node; the sidecar witnesses your app's AI inference calls.

In the shared tenant model, clearing level is set once for the entire tenant. All teams inherit it. This is why the shared model is only recommended for orgs with uniform data sensitivity.

Assessor note
Ask the platform team for their clearing level matrix. Each team should have a documented justification for their clearing level. A team processing healthcare data at level 0 is a finding.

5. Kubernetes Namespace Isolation

In Kubernetes, the natural isolation boundary is the namespace. SWT3 supports two DaemonSet patterns:

Pattern A: Cluster-wide DaemonSet + per-namespace agent_id

One DaemonSet for hardware attestation (AI-HW.1), shared across the cluster. Each pod derives its agent_id from the Kubernetes downward API:

# values.yaml for cluster-wide hardware attestation
config:
  mode: cloud
  agentId: ""  # auto-generates from pod hostname

# The DaemonSet runs on every node regardless of namespace.
# Hardware attestation is infrastructure-level, not team-level.

Pattern B: Per-namespace sidecar (tail mode)

For AI model witnessing, deploy the --mode tail sidecar per team namespace. The sidecar watches a shared emptyDir volume where your app writes AI telemetry JSONL. No traffic interception -- the app chooses what to emit. Each namespace gets its own API key and clearing level:

# kustomization.yaml for team-alpha namespace
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: team-alpha

resources:
  - ../base  # team's own Deployments

patches:
  - path: ../../swt3-witness/kustomize/sidecar-overlay/sidecar-patch.yaml
    target:
      kind: Deployment
      labelSelector: "swt3.io/witness=enabled"

Create a per-namespace secret with the team's API key:

kubectl create secret generic swt3-api-key \
  --from-literal=api-key=axm_live_alpha_xxx \
  -n team-alpha

NetworkPolicy

The sidecar only needs egress to the SWT3 HTTPS endpoint. Lock it down with a NetworkPolicy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: swt3-sidecar-egress
  namespace: team-alpha
spec:
  podSelector:
    matchLabels:
      swt3.io/witness: "enabled"
  policyTypes: ["Egress"]
  egress:
    # DNS resolution (required for sovereign.tenova.io)
    - to: []
      ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP
    # SWT3 endpoint (HTTPS only)
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0  # Replace with SWT3 endpoint IP if your policy requires
      ports:
        - port: 443
          protocol: TCP

Assessor note
For environments requiring explicit IP allowlisting, resolve sovereign.tenova.io and pin the CIDR. The sidecar makes outbound HTTPS only -- no inbound ports are required (metrics are scraped from localhost within the pod).

6. Terraform Workspace Patterns

Terraform Modules — Five ready-to-deploy modules cover every cloud target: AWS Lambda (CloudTrail), AWS ECS sidecar, AWS EKS DaemonSet, Azure AKS, GCP GKE. Each module enforces secret-ARN injection (raw keys rejected at terraform plan), CloudWatch alarms, and clearing level validation.

For teams deploying SWT3 via Terraform (Lambda, ECS, Cloud Functions), use workspaces to isolate state:

Directory layout

terraform/
  modules/           # shared SWT3 modules (git submodule or registry)
  environments/
    team-alpha/
      main.tf        # references modules/ with team-specific vars
      terraform.tfvars
      backend.tf     # team-alpha state bucket
    team-beta/
      main.tf
      terraform.tfvars
      backend.tf     # team-beta state bucket
    infra-shared/
      main.tf        # CloudTrail Lambda (shared infrastructure)
      terraform.tfvars
      backend.tf

Team-specific tfvars

# team-alpha/terraform.tfvars
name_prefix             = "team-alpha"
swt3_api_key_secret_arn = "arn:aws:secretsmanager:us-east-1:123456:secret:team-alpha/swt3-api-key"
clearing_level          = 2
agent_id                = "team-alpha-ecs"
enable_alarms           = true
alarm_email             = "team-alpha-oncall@corp.example.com"

tags = {
  "team"        = "alpha"
  "cost-center" = "CC-1234"
  "swt3:module" = "aws-ecs-sidecar"
}

Shared infrastructure

# infra-shared/main.tf
module "cloudtrail_witness" {
  source = "../../modules/aws-lambda-cloudtrail"

  swt3_api_key_secret_arn = var.infra_api_key_secret_arn
  cloudtrail_bucket_name  = var.cloudtrail_bucket_name
  cloudtrail_bucket_arn   = var.cloudtrail_bucket_arn
  binary_version          = "8.1.0"
  binary_sha256           = "abc123..."
  agent_id                = "shared-cloudtrail"
  clearing_level          = 1
  enable_alarms           = true
  alarm_sns_topic_arn     = var.ops_sns_topic_arn

  tags = { "team" = "platform", "cost-center" = "CC-0000" }
}

State isolation is non-negotiable. Never share Terraform state across teams. A terraform destroy in one team's workspace must not affect another team's infrastructure. Use separate S3 buckets or separate state keys at minimum.

Available Terraform modules

ModuleUse CaseMode
aws-lambda-cloudtrailS3 CloudTrail logs via SQS to LambdaCLI (batch)
aws-ecs-sidecarContainer definition fragment for ECS task defsServer (HTTP webhook)
aws-eks-daemonsetHelm chart wrapper for EKS clustersHardware attestation
azure-functionEvent Grid to Azure FunctionCLI (batch)
gcp-cloud-functionLog Router to Pub/Sub to Cloud FunctionCLI (batch)

All modules support enable_alarms = true for CloudWatch/Azure Monitor alerting and output a deployment_fingerprint for portal verification.

7. Cost Allocation

Anchor volume estimation

Each SWT3 anchor costs the same regardless of content. Estimate monthly volume per team:

Workload TypeAnchors/Month (typical)Formula
AI inference (chatbot)5,000-50,000requests/day x 30
AI inference (batch ML)500-5,000batch_runs x models x 30
Infrastructure (CloudTrail)10,000-100,000API calls/day x 30 (deduplicated)
Hardware attestation720nodes x 24 (hourly)

Tag-based cost tracking

All SWT3 Terraform modules automatically tag resources with swt3:module and swt3:version. Add team-specific tags in your tfvars:

tags = {
  "team"        = "alpha"
  "cost-center" = "CC-1234"
  "swt3:module" = "aws-ecs-sidecar"
  "swt3:version" = "8.1.0"
}

Use AWS Cost Explorer or your cloud's cost allocation tool to filter by these tags.

Prometheus metric aggregation

For K8s deployments, aggregate witness metrics by namespace (which maps to team):

# Total events witnessed per namespace (team)
sum by (namespace) (swt3_events_processed_total)

# Error rate per namespace
sum by (namespace) (rate(swt3_events_errors_total[5m]))
  / sum by (namespace) (rate(swt3_events_processed_total[5m]))

Feed these into your chargeback system. The SWT3_AGENT_ID (set to namespace via downward API) is included in every anchor's observations, making per-team attribution possible at the ledger level.

8. RBAC Model

SWT3 uses a role-based model with three roles per tenant:

RoleCan ViewCan AttestCan ExportCan Manage Keys
AdminAllYesYesYes
AnalystAllYesYesNo
AssessorAll (read-only)NoYesNo

Multi-tenant RBAC mapping

In the hybrid model, a platform engineer might be admin on the infrastructure tenant and analyst on their team's AI tenant. Users can have different roles on different tenants.

  Platform Engineer Alice
    infra-tenant:    admin     (manages CloudTrail, hardware attestation)
    team-alpha:      analyst   (attests AI model controls for her team)

  ML Engineer Bob
    team-beta:       analyst   (attests AI model controls)
    infra-tenant:    (none)    (no access to infrastructure tenant)

  External Assessor Carol
    team-alpha:      assessor  (read-only review of team alpha's AI posture)
    team-beta:       assessor  (read-only review of team beta's AI posture)

API key scoping

API keys are scoped to a single tenant. A key for team-alpha cannot write anchors to team-beta's ledger. This is enforced server-side -- the API key resolves to a tenant ID, and all operations are scoped to that tenant.

Do not share API keys across teams. Even in the shared-tenant model, create one key per service that writes anchors. This makes key rotation and revocation granular -- you can revoke a compromised service's key without disrupting other services.

9. Operational Runbook

Onboarding a new team

  1. Create tenant -- Use the self-serve signup API or dashboard to create a new tenant for the team. Note the API key.
  2. Provision secret -- Store the API key in your secret manager. For K8s: kubectl create secret generic swt3-api-key --from-literal=api-key=axm_live_xxx -n team-namespace
  3. Deploy sidecar -- Apply the Kustomize overlay or Helm values to the team's namespace. Verify with kubectl exec ... wget -qO- http://localhost:9091/healthz

Offboarding a team

  1. Revoke API key -- Delete the key from the dashboard Settings page. Outstanding anchors remain in the ledger.
  2. Remove sidecar -- Remove the Kustomize overlay or Helm values. The app continues running without the sidecar.
  3. Preserve anchors -- Anchors are immutable. The team's ledger is preserved for audit trail even after offboarding. Do not delete the tenant if audit retention policies require it.

Drift detection across teams

Use the portfolio endpoint to monitor drift across all tenants:

# Platform team monitors all tenants
curl -s https://sovereign.tenova.io/api/v1/portfolio \
  -H "Authorization: Bearer axm_live_platform_xxx" | jq '.enclaves[] | {name, score, direction}'

Set up alerts for any team whose posture score drops below threshold:

# Prometheus alert (if using the witness Prometheus metrics)
- alert: SWT3TeamPostureDegraded
  expr: swt3_events_errors_total > 0 and swt3_events_processed_total == 0
  for: 30m
  labels:
    severity: warning
  annotations:
    summary: "A team's witness sidecar is producing only errors"

Scaling checklist

TeamsRecommended TopologyKey ManagementMonitoring
1-5Shared tenantManualDashboard
5-20HybridSecret manager + automationDashboard + Prometheus alerts
20+Tenant-per-teamFully automated provisioningPortfolio API + PagerDuty/Opsgenie

Related Guides

SRE reference
The RUNBOOK.md in the swt3-infra-witness package covers 10 failure modes including tail-mode sidecar troubleshooting, Lambda timeouts, DLQ accumulation, mock key detection, and Secrets Manager permission errors. Share it with your on-call rotation.