P083 Inference verification overhead
How much latency does inference verification add, and how much serving throughput does it cost?
The latency, serving throughput, verifier compute and record size that deploying inference verification imposes.
Status
Started The authors' own measurements on Llama-2-13B (Table 6): committing overhead 0.26 ms per token, validation time 81 ms, 10 KB of provider GPU memory per token; a measured per-token latency cost, not serving throughput at production batch sizes or end-to-end latency on a deployed service. Amodo's Stage-1 prototype measures verifier GPU utilisation (about 2% versus about 53% for the prover on one H100 tray, small open-weight models) and lists latency and throughput as future work; Karvonen et al. (DiFR) measure record size (1.5 to 30 bytes per token), which is in the statement, not the headline.
Time and memory overhead of generating and validating Llama-2-13B inference using established verifiable inference compared to our method.
Raised in: Cankaya (2025), Catching Misreporting About ML Hardware Use by Turning Noise into Signal; Cankaya (2026), Bit-Exact AI Inference Verification Without Performance Tradeoffs, arXiv 2606.00279v2; Cankaya (2026), Compute Verification FAQs; Karvonen et al. (2025), DiFR: Inference Verification Despite Nondeterminism; Amodo (2026), An Inference Verification Prototype — Stage 1; Proofworks, Living Verification Documents.
Sub-questions
| ID | Sub-question | What the sources report |
|---|---|---|
| P083.1 | What added latency and verifier compute does verification impose (at a stated request rate; as a ratio vs inference), and how large a trusted cluster is needed? | a source suggests an answer (not checked) |
| P083.2 | What is the minimum bytes per record that still distinguishes all tested production methods, incl. cross-hardware? | a source suggests an answer (not checked) |
| P083.3 | Can activations be extracted from an unmodified inference process at a stated low overhead and robustly across versions? | partly answered |
Possibly relevant work found by automated search
Searched; nothing found for these sub-questions.
Data
- problems.csv — one row per problem
- problem_status.csv — status, scope and evidence per problem
- problem_sources.csv — the posed-by quote and other documents per problem
- leaves.csv — all sub-questions, including those without a checkable form
- leaf_sources.csv — every source row behind each sub-question