P081 Bit-exact inference replay (same hardware class)
Do inference requests replay bit-exactly on a second machine of the same SKU at production batch sizes, and at what overhead?
Bit-exact replay of inference on same-class hardware.
Status
Started Closest to Done on the list: two independent groups show same-SKU inference replays bit-exactly once batch composition is controlled; what is missing is one run on a second machine at production batch size. Same SKU, same software stack, no floating-point atomics in the kernels, batch size logged (one integer per forward pass); shown for Qwen2.5-7B prefill across physical cards at batch sizes 1, 2 and 4. Corroborated by an independent group with its own method: He & Thinking Machines Lab (2025) report 1,000 identical temperature-0 Qwen3-235B completions with batch-invariant kernels at about 20% matmul throughput loss, on the same server. Not shown: production batch sizes on a second machine; mixture-of-experts under tensor parallelism across machines; INT-quantised models, where Cankaya saw non-determinism in vLLM. Earlier 'bitwise reproducibility is rarely achievable in practice' (Karvonen et al. 2025) describes default stacks and predates both.
The within-SKU zeros (A100-SXM4 ↔ A100-PCIe) rule out physical-card variance: yield-binned chips of the same SKU produce bit-identical outputs.
Posed by
- Harack et al. (2025), Verification for International AI Governance (Oxford AIGI):
Noise-free replications of machine learning computations might be possible, but may also be difficult
(§2.5.2.4 fn 210, p.64, L3514–3515) (source states the problem but not the quantity the question asks for)
Also raised in: Baker, Kulp, Marks, Brundage & Heim (2025), Verifying International Agreements on AI: Six Layers of Verification for Rules on Large-Scale AI Development and Deployment (RAND WR-A4077-1); Cankaya (2025), Catching Misreporting About ML Hardware Use by Turning Noise into Signal; Cankaya (2026), A System Overview for Near-Term, Low-Trust AI Compute Verification (v0.2); Cankaya (2026), Bit-Exact AI Inference Verification Without Performance Tradeoffs, arXiv 2606.00279v2; Cankaya (2026), Catching Misreporting About ML Hardware Use by Turning Noise into Signal, Part II; Canary Institute research; Hausenloy & Li (2026), The Compute Verification Post; Karvonen et al. (2025), DiFR: Inference Verification Despite Nondeterminism; Rinberg et al. (2025), Verifying LLM Inference to Detect Model Weight Exfiltration; AI 2040 (2026), Get Involved in Verification; AI Security Forum, Requests for Discussion; Amodo (2026), AI 2040 Plan A — How You Can Help; Amodo (2026), AI 2040 Plan A — Verification SITREP; Dean, Verification Plan (AI 2040); Proofworks, Living Verification Documents.
Sub-questions
| ID | Sub-question | What the sources report |
|---|---|---|
| P081.1 | On a second machine of the same SKU, with batch composition logged, do replayed inference requests match bit-exactly at production batch sizes and frontier scale (a stated parameter count)? | a source suggests an answer (not checked) |
| P081.2 | Are all kernels in frontier production serving stacks (incl. MoE, INT4 dequant) free of floating-point atomics? | a source suggests an answer (not checked) |
| P081.3 | What throughput overhead does a bit-reproducible serving/training stack (incl. network stack) impose vs default (batch-invariant kernels ~20%)? | a source suggests an answer (not checked) |
| P081.4 | Can decode/prefill batch size, tensor parallelism or covert batch elements be detected from numerics or timing (equivalence classes, batch invariance)? | a source suggests an answer (not checked) |
| P081.5 | Can kernels be made deliberately batch-size-identifiable at a stated low throughput cost? | open |
| P081.6 | Has numeric-fingerprint (floating-point forensics) verification run on a production-scale deployment with a measured false-alarm rate? | open |
Possibly relevant work found by automated search
Found by AI agents searching for this problem's sub-questions, mostly by keyword and citation in the OpenAlex scholarly database and on the web; a second AI pass reviewed each one, and we spot-checked some by hand. They seemed reasonably related, as far as one can tell from outside each paper's field. Some come from neighbouring fields, such as arms control and financial audit, and bear on the problem by analogy. Not reviewed against the status above; a paper listed here does not change that status. 928 of 1052 quotes checked word for word against the paper's text.
For P081.1: On a second machine of the same SKU, with batch composition logged, do replayed inference requests match bit-exactly at production batch sizes and frontier scale (a stated parameter count)?
For P081.3: What throughput overhead does a bit-reproducible serving/training stack (incl. network stack) impose vs default (batch-invariant kernels ~20%)?
For P081.4: Can decode/prefill batch size, tensor parallelism or covert batch elements be detected from numerics or timing (equivalence classes, batch invariance)?
Data
- problems.csv — one row per problem
- problem_status.csv — status, scope and evidence per problem
- problem_sources.csv — the posed-by quote and other documents per problem
- leaves.csv — all sub-questions, including those without a checkable form
- leaf_sources.csv — every source row behind each sub-question