P083 Inference verification overhead

How much latency does inference verification add, and how much serving throughput does it cost?

The latency, serving throughput, verifier compute and record size that deploying inference verification imposes.

Status

Started The authors' own measurements on Llama-2-13B (Table 6): committing overhead 0.26 ms per token, validation time 81 ms, 10 KB of provider GPU memory per token; a measured per-token latency cost, not serving throughput at production batch sizes or end-to-end latency on a deployed service. Amodo's Stage-1 prototype measures verifier GPU utilisation (about 2% versus about 53% for the prover on one H100 tray, small open-weight models) and lists latency and throughput as future work; Karvonen et al. (DiFR) measure record size (1.5 to 30 bytes per token), which is in the statement, not the headline.

Time and memory overhead of generating and validating Llama-2-13B inference using established verifiable inference compared to our method.
Ong et al. (2025), TOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference (ICML 2025) (2025-05-30 (arXiv v2)), L1167–1168

Raised in: Cankaya (2025), Catching Misreporting About ML Hardware Use by Turning Noise into Signal; Cankaya (2026), Bit-Exact AI Inference Verification Without Performance Tradeoffs, arXiv 2606.00279v2; Cankaya (2026), Compute Verification FAQs; Karvonen et al. (2025), DiFR: Inference Verification Despite Nondeterminism; Amodo (2026), An Inference Verification Prototype — Stage 1; Proofworks, Living Verification Documents.

Sub-questions

IDSub-questionWhat the sources report
P083.1What added latency and verifier compute does verification impose (at a stated request rate; as a ratio vs inference), and how large a trusted cluster is needed?a source suggests an answer (not checked)
P083.2What is the minimum bytes per record that still distinguishes all tested production methods, incl. cross-hardware?a source suggests an answer (not checked)
P083.3Can activations be extracted from an unmodified inference process at a stated low overhead and robustly across versions?partly answered

Possibly relevant work found by automated search

Searched; nothing found for these sub-questions.

Data