Regulatory obligations do not scale with parameter count. Quantization witnessing, distillation provenance, edge inference patterns, and resource consumption evidence for models from 500M to 13B parameters.
Audience: AI engineers deploying small language models on edge devices and private infrastructure, teams fine-tuning or distilling models for production, compliance officers responsible for on-device AI systems, and platform engineers managing SLM fleets across mobile, embedded, and constrained environments.
Regulatory obligations do not scale with parameter count. EU AI Act Article 6 high-risk classification is based on use case, not model size. A 3B parameter model making healthcare triage decisions carries the same transparency, record-keeping, and human oversight obligations as a 175B parameter cloud model. Article 53 GPAI obligations have a 10^25 FLOP compute threshold that SLMs fall below, but deployer obligations under Article 26 apply regardless. The evidence gap is wider for SLMs: cloud API providers maintain their own logs, but when you run Phi-3 on a phone, you are the only source of truth.
When you call GPT-4o, OpenAI has logs, audit trails, and compliance infrastructure. When you run Phi-3-mini on an Android phone, nobody has logs except you. The compliance burden shifts entirely to the deployer.
SLMs are deployed precisely in the contexts where evidence matters most: healthcare triage in rural clinics, agricultural advisory on field tablets, financial scoring in mobile banking, content moderation on user devices. These are high-risk use cases running on low-resource models. The model is small; the obligation is not.
The models driving this shift are already in production across millions of devices:
These models are designed for edge deployment and represent the fastest-growing segment of AI deployment. Every one of them, when placed in a high-risk use case, inherits the full weight of deployer obligations.
| Obligation | Trigger | Requirements | SLM Impact |
|---|---|---|---|
| Art. 53 GPAI Provider | >10^25 FLOPs training compute threshold | Systemic risk assessment, model cards, red teaming | Does NOT apply to SLMs |
| Art. 26 Deployer | All AI systems in Annex III use cases | Transparency, record-keeping, human oversight, risk management | APPLIES FULLY |
| Art. 6 + Annex III High-risk | Classification by use case (healthcare, education, employment, law enforcement, critical infrastructure) | Full conformity assessment, technical documentation, CE marking | APPLIES FULLY |
| Art. 15 Accuracy / Robustness | All high-risk AI systems | Accuracy, robustness, and cybersecurity measures | APPLIES FULLY -- quantization changes accuracy |
SLMs avoid GPAI overhead but inherit every deployer obligation. The evidence requirements are identical to a 175B cloud model. The difference is that you, not the cloud provider, must produce the evidence.
The model on the device is not the model that was evaluated. A Llama-3.2-3B evaluated at fp16 performs differently when quantized to int4 GGUF for mobile deployment. Accuracy degrades, behavior changes, edge cases shift. The assessor needs proof of what quantization was applied and when.
AI-MDL.7 creates a cryptographic record of the quantization method, bit width, and group size applied to the deployed model.
from swt3_ai import Witness
witness = Witness(
endpoint="https://sovereign.tenova.io",
api_key="axm_live_...",
tenant_id="YOUR_TENANT_ID",
clearing_level=2,
)
# Witness the quantization method
witness.witness_quantization(
method="gguf",
bits=4,
group_size=128,
)witness.witnessQuantization("int4", { bits: 4, groupSize: 128 });| Method | Code | Typical Use |
|---|---|---|
| fp32 | 0 | Research, baseline evaluation |
| fp16 | 1 | GPU inference, standard training |
| bf16 | 2 | Training on Ampere+ GPUs |
| int8 | 3 | Server-side quantized inference |
| int4 | 4 | Mobile and edge deployment |
| gptq | 5 | Post-training quantization (GPU) |
| awq | 6 | Activation-aware weight quantization |
| gguf | 7 | llama.cpp, mobile, CPU inference |
Records the quantization format, bit width, and group size applied to a model before deployment. Creates a cryptographic anchor linking the deployed binary to a specific compression configuration.
Ask for the quantization anchor. If the model was evaluated at fp16 but deployed at int4, require evidence of both: evaluation results AND quantization method. Without the AI-MDL.7 anchor, there is no proof that the deployed model matches what was tested.
When a team distills GPT-4o into a 3B student model, the compliance chain must connect them. Teacher-to-student lineage is not optional -- it is the foundation of traceability for derived models.
# Witness teacher model integrity
witness.witness_model_weights(
model_id="teacher-gpt4o",
weight_hash="a1b2c3d4e5f6...",
format="safetensors",
)
# Witness student model after distillation
witness.witness_model_weights(
model_id="student-phi3-3b-distilled",
weight_hash="f6e5d4c3b2a1...",
format="gguf",
)
# Record distillation data lineage
witness.witness_data_provenance(
dataset_id="distillation-outputs-v3",
source_count=1,
record_count=50000,
)Records the dataset identity, source count, and record count used in training or distillation. Links derived models to their data origins for auditability.
For distilled models, demand the full chain: teacher model hash (AI-MDL.5), distillation dataset (AI-DATA.1), student model hash (AI-MDL.5), and any adapters applied post-distillation (AI-MDL.6). A student model without teacher lineage is an untraceable derivative.
On-device models face integrity risks that cloud models do not: sideloaded APKs, corrupted downloads, supply chain attacks, user-modified model files. AI-MDL.5 hashes the model weight file and creates an integrity anchor. When the same hash appears at deployment as at evaluation, the assessor has proof the model was not modified.
from swt3_ai import hash_model_file
# Hash the model file
model_hash = hash_model_file("/path/to/phi-3-mini-q4.gguf")
# Witness the integrity
witness.witness_model_weights(
model_id="phi-3-mini-4k-instruct",
weight_hash=model_hash,
format="gguf",
file_size_bytes=2_147_483_648,
)val modelHash = SWT3.hashFile(modelFile.absolutePath)
val result = witness.wrap(
prompt = userInput,
response = modelOutput,
modelId = "phi-3-mini-4k-instruct",
)Hashes the model weight file and anchors the digest alongside the model identifier and format. Establishes a tamper-evident binding between the evaluated model and the deployed binary.
Compare the weight hash at evaluation time with the weight hash at deployment. If they differ, the model was modified after evaluation. For GGUF and ONNX files, the hash covers the entire binary -- any change, including metadata edits, produces a different digest.
Small model plus retrieval beats large model alone. This is the dominant edge architecture. A 3B model with a local vector store -- ChromaDB, FAISS, SQLite-vec -- retrieves context before generating. Three anchors record one user interaction:
# Witness RAG retrieval
witness.witness_rag_context(
chunks=retrieved_chunks,
corpus_id="medical-guidelines-v2",
embedding_model="all-MiniLM-L6-v2",
similarity_threshold=0.7,
)
# Witness SLM inference
result = witness.wrap(client).chat.completions.create(
model="phi-3-mini-4k-instruct",
messages=[
{"role": "system", "content": rag_context},
{"role": "user", "content": user_query},
],
)For Android-specific WAL and buffer patterns, see the Mobile Edge AI Attestation Guide. For batch upload strategies when connectivity is intermittent, see Connectivity-Constrained Compliance.
SLMs at scale consume significant resources. A fleet of 10,000 devices running 100 inferences per day produces 1,000,000 inferences daily. Even at zero API cost, compute time, memory usage, and battery consumption are real operational concerns. AI-COST.1 witnesses: tokens in, tokens out, API calls, estimated cost, and compute seconds.
witness.witness_resource_consumption(
tokens_in=512,
tokens_out=256,
api_calls=1,
cost_cents=0, # on-device, no API cost
provider="on-device",
model_id="phi-3-mini-4k-instruct",
compute_seconds=0.8,
)Individual SLM inferences fall far below the Art. 53 FLOP threshold, but aggregated fleet consumption should be tracked for operational governance and potential future reporting requirements. Regulators are increasingly interested in cumulative compute footprint, not just per-inference cost.
| Level | Name | Size | What's Included | Best For |
|---|---|---|---|---|
| CL0 | Analytics | ~2 KB | Full metadata, model ID, context, hashes | Evaluation and testing phase |
| CL1 | Standard | ~1.5 KB | Model ID, reduced context, core hashes | Production, good connectivity |
| CL2 | Sensitive | ~400 B | Model ID, factors only | Mobile SLM default |
| CL3 | Classified | ~200 B | Factors only, model ID hashed | Healthcare and financial on-device |
Recommendation: Use CL2 as the default for SLM edge deployments. It is bandwidth-friendly and preserves the evidence chain without transmitting sensitive context. Use CL0 during model evaluation to capture full metadata. Use CL3 for healthcare and financial use cases where even model identity is sensitive.
| SWT3 Procedure | What It Proves for SLMs | Regulatory Reference |
|---|---|---|
| AI-INF.1 | Every on-device inference happened and when | Art. 12 (record-keeping), Art. 26 (deployer) |
| AI-MDL.5 | Model file has not been tampered with | Art. 15 (accuracy / robustness) |
| AI-MDL.6 | Adapter configuration is known and tracked | Art. 15 (accuracy / robustness) |
| AI-MDL.7 | Quantization method applied to deployed model | Art. 15, Art. 9 (risk management) |
| AI-DATA.1 | Training and distillation data lineage is traceable | Art. 10 (data governance) |
| AI-RAG.1 / RAG.2 | Retrieval provenance and relevance verified | Art. 12 (record-keeping) |
| AI-COST.1 | Resource consumption tracked across fleet | Art. 53 (FLOP thresholds, fleet aggregation) |
| AI-GRD.1 | On-device guardrails are present and active | Art. 9 (risk management) |
| AI-HITL.1 | Human reviewed SLM output before action | Art. 14 (human oversight) |
| AI-FAIR.1 | Fairness assessed across demographics | Art. 10 (data governance, bias) |
| Assessor Question | Where to Look |
|---|---|
| What quantization was applied to the deployed model? | AI-MDL.7 anchor with method, bits, group size |
| Is the deployed model the same file that was evaluated? | AI-MDL.5 anchor -- compare weight hashes |
| What training data produced this model? | AI-DATA.1 anchor with dataset ID and record count |
| Are LoRA adapters modifying the base model? | AI-MDL.6 anchor with adapter hashes and base model binding |
| What retrieval context influenced the response? | AI-RAG.1 anchor with chunk count and corpus ID |
| Were retrieved chunks relevant enough? | AI-RAG.2 anchor with similarity threshold and scores |
| How many inferences is the fleet running daily? | AI-COST.1 anchors aggregated across fleet |
| Are content guardrails active on-device? | AI-GRD.1 anchor with triggered / action status |
| Did a human review this output before it was acted on? | AI-HITL.1 anchor with review outcome and reviewer hash |
| What clearing level is appropriate for my use case? | CL2 default for mobile, CL3 for healthcare / financial, CL0 for evaluation |
A complete example covering quantization, weight integrity, RAG retrieval, inference, and resource consumption for a mobile SLM deployment:
from swt3_ai import Witness, hash_model_file
# Configure for mobile SLM deployment
witness = Witness(
endpoint="https://sovereign.tenova.io",
api_key="axm_live_...",
tenant_id="YOUR_TENANT_ID",
clearing_level=2, # Bandwidth-friendly default
agent_id="field-tablet-001",
jurisdiction="PE", # Peru
)
# 1. Witness quantization method
witness.witness_quantization(method="gguf", bits=4, group_size=128)
# 2. Witness model file integrity
model_hash = hash_model_file("models/phi-3-mini-q4.gguf")
witness.witness_model_weights(
model_id="phi-3-mini-4k-instruct",
weight_hash=model_hash,
format="gguf",
)
# 3. Witness RAG retrieval
witness.witness_rag_context(
chunks=["Patient presents with...", "Treatment guidelines state..."],
corpus_id="clinical-guidelines-v4",
embedding_model="all-MiniLM-L6-v2",
similarity_threshold=0.75,
)
# 4. Witness inference
result = witness.wrap(client).chat.completions.create(
model="phi-3-mini-4k-instruct",
messages=[{"role": "user", "content": "Recommend treatment for..."}],
)
# 5. Witness resource consumption
witness.witness_resource_consumption(
tokens_in=512, tokens_out=256,
api_calls=1, cost_cents=0,
provider="on-device",
model_id="phi-3-mini-4k-instruct",
compute_seconds=0.8,
)
# WAL persists locally, flushes when connected
await witness.flush()Model references: Microsoft Phi-3, Google Gemma 2, Meta Llama 3.2, Mistral 7B, Alibaba Qwen 2.5.