When a single inference splits across multiple hardware substrates, who proves what happened at each phase? And who proves the intermediate state was not corrupted in transit?
Who this is for: AI infrastructure engineers, ML platform teams, and compliance architects deploying disaggregated inference pipelines (Cerebras + AMD, NVIDIA Dynamo, vLLM, SGLang, heterogeneous cloud). Assumes familiarity with GPU/accelerator infrastructure and transformer inference architecture.
Architecture preview: Phase-specific witnessing for disaggregated inference is under active protocol development. This guide describes the problem, the compliance gaps, and the architectural approach. SDK integration will be available in a future release. The conceptual framework described here informs the protocol design.
AI inference is moving from monolithic single-device execution to disaggregated multi-phase pipelines. A single inference request now routinely executes across different hardware from different vendors:
| Architecture | Prefill Phase | Decode Phase | Intermediate State |
|---|---|---|---|
| Cerebras + AMD | Cerebras WSE-3 (wafer-scale) | AMD MI300X GPUs | KV cache transferred between vendors |
| NVIDIA Dynamo | GPU pool A (compute-optimized) | GPU pool B (memory-optimized) | KV cache routed by disaggregated scheduler |
| vLLM / SGLang | Draft model on device A | Verification model on device B | Speculative decode candidates shared |
| Heterogeneous cloud | Dense compute instance | Memory-bandwidth instance | Intermediate state traverses network fabric |
This disaggregation delivers significant performance improvements: faster time-to-first-token, higher throughput, lower cost per inference. But it creates a compliance gap that did not exist when inference was atomic.
Existing compliance attestation systems treat inference as either atomic (one device, one anchor) or sequential (workload moves between devices across separate requests). Neither model captures the reality of disaggregated inference: one prompt, multiple phases, multiple hardware vendors, shared intermediate state, one response.
The compliance consequence is straightforward: when a disaggregated inference produces a biased, inaccurate, or harmful output, no existing system can:
When a disaggregated inference produces a problematic output, which phase is responsible? Did bias originate in prefill (how the prompt was encoded) or decode (how tokens were generated)? The answer depends on which hardware ran which phase, and no attestation currently binds phases to hardware.
Without phase-level witnessing, post-incident investigation cannot distinguish between a model problem (the weights were biased) and an infrastructure problem (the KV cache was corrupted during transfer).
KV cache state contains encoded representations of the original prompt. This may include PII, classified information, trade secrets, or regulated financial data. When this state transfers between hardware vendors via PCIe, CXL, network fabric, NVLink, or InfiniBand, there is currently no standard mechanism to attest the transfer, verify the integrity of the received state, or enforce data classification rules on the intermediate data.
The KV cache is the computational intermediate that connects phases. Understanding and governing its custody is essential for compliance in disaggregated architectures.
Orchestrators (vLLM routers, SGLang schedulers, NVIDIA Dynamo, custom schedulers) make phase-to-hardware assignment decisions that affect compliance posture. If prefill is routed to non-TEE hardware because it offers better throughput, the raw prompt transits unprotected memory. This routing decision is compliance-relevant, but no existing system witnesses it.
The SWT3 approach to disaggregated inference witnessing follows the same principles as all other SWT3 witnessing: observe, record, and make independently verifiable. The protocol extension introduces three concepts:
The critical property: each anchor is independently verifiable, but the chain of anchors proves the complete inference path. An assessor can verify any individual phase, or the entire chain, without access to the hardware or the orchestrator.
While phase-specific witnessing is under development, you can witness disaggregated inference at the aggregate level today using the existing SDK:
from swt3_ai import Witness
w = Witness(tenant_id="YOUR_TENANT", api_key="axm_live_...")
client = w.wrap(your_inference_client)
# This witnesses the complete inference (all phases combined)
result = client.chat.completions.create(model="llama-3.1-70b", messages=[...])
The aggregate anchor proves which model produced the response, when, and with what guardrails. It does not attribute to individual phases, but it establishes a baseline that phase-level witnessing will extend.
| Regulation | Requirement | Disaggregation Impact |
|---|---|---|
| EU AI Act Art. 12 | Automatic recording of events over the AI system's lifetime | When inference is disaggregated, the "event" of a single inference spans multiple hardware substrates. Without phase-level attestation, the record is incomplete. |
| EU AI Act Art. 9(2)(a) | Risk management measures proportionate to risk | KV cache custody is a new risk category. Encoded PII traverses vendor boundaries without attestation. |
| EU AI Act Art. 15 | Accuracy, robustness, cybersecurity | KV cache corruption during transfer between phases is a reliability concern specific to disaggregated architectures. |
| NIST AI RMF MAP 1.5 | Characterize the deployment and operational environment | The "operational environment" now spans multiple hardware substrates within a single inference. |
| NIST 800-53 AU-3 | Audit records include identity of subjects associated with the event | The "subjects" include the hardware substrates that executed each phase. |
The disaggregated inference witness chain follows a linear sequence matching the inference pipeline:
Prompt
|
v
[Prefill Phase] ------> Prefill Anchor (phase=prefill, hw=cerebras-wse3)
|
| KV Cache Transfer
|
v
[Custody Witness] ------> Custody Anchor (from=wse3, to=mi300x, hash=kv_state_hash)
|
v
[Decode Phase] ------> Decode Anchor (phase=decode, hw=amd-mi300x)
|
v
[Composite] ------> Composite Anchor (links all 3, proves complete chain)
|
v
ResponseEach anchor is an independent SWT3 Witness Anchor with a standard fingerprint. The composite anchor's factors include the fingerprints of all phase and custody anchors, creating a Merkle-like chain that is verifiable from any individual anchor up to the complete inference.
For speculative decoding architectures (vLLM, SGLang), additional anchors record draft-model speculation and verification-model acceptance, creating a verifiable record of which candidate tokens were proposed and which were accepted.
Phase-specific witnessing for disaggregated inference is under active protocol development. The following is available today:
If you are deploying disaggregated inference pipelines and need compliance attestation, contact us to discuss your architecture. Early adopters help shape the protocol design.