P080 Bit-exact training replay

What fraction of training steps replay bitwise on a verifier's hardware, and what throughput penalty does deterministic training impose?

Bit-exact replay of training steps.

Terms: NCCL: Nvidia's library for exchanging data between GPUs during training. ZeRO-3: a technique that splits a model's parameters across many GPUs to save memory.

Status

Started Built and measured: single-GPU training of small models (ResNet-50, 23M parameters, from scratch; GPT-2, 117M, fine-tune) replayed bit-identically across three NVIDIA consumer and workstation GPUs (A40, Titan XP, RTX 2080 Ti), FP64 with logged rounding decisions, at about 1.2-1.4x trainer and 1.3-1.7x auditor time and 18-20 MB of logs per GPT-2 step. Not multi-GPU or distributed (all-reduce ordering unhandled), not low precision, not frontier scale. Throughput-penalty clause: one measured figure elsewhere, Peigné-Lefebvre et al. 2026 report full determinism on a 7B model on 8 H100s costing 1.6-8.2% per step and cite 23-82% overhead for deterministic FlashAttention backward. Fraction-of-steps clause at frontier scale: no work.

A method for two parties, training the same model on different GPU types, to achieve identical training results by logging and sharing rounding decisions
Srivastava, Arora & Boneh (2024), Optimistic Verifiable Training by Controlling Hardware Nondeterminism, arXiv 2403.09603v3 (NeurIPS 2024) (2024-11-25 (v3)), L203–204

Posed by

Also raised in: Harack et al. (2025), Verification for International AI Governance (Oxford AIGI); Baker, Kulp, Marks, Brundage & Heim (2025), Verifying International Agreements on AI: Six Layers of Verification for Rules on Large-Scale AI Development and Deployment (RAND WR-A4077-1); Shavit (2023), What Does It Take to Catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring; Cankaya (2026), Bit-Exact AI Inference Verification Without Performance Tradeoffs, arXiv 2606.00279v2; Canary Institute research; Jin et al. (2026), Trusting What You Cannot See: Auditable Fine-Tuning and Inference for Proprietary AI, arXiv 2603.07466; Peigné-Lefebvre, Nguyen & Wang (2026), Zero Knowledge Verification for Frontier AI Training Is Possible, arXiv 2606.05433; AI Security Forum, Requests for Discussion; Amodo (2026), Example Schemes for Verifying High-Stakes AI Agreements; Dean, Verification Plan (AI 2040); Proofworks, Living Verification Documents.

Sub-questions

IDSub-questionWhat the sources report
P080.1Is same-hardware, same-topology training replay on TPU/XLA bit-exact in production, and does that extend to a verifier replaying on its own hardware?partly answered
P080.2With deterministic kernels and fixed NCCL reduction trees, what fraction of training steps (fwd+bwd+all-reduce) replay bitwise on GPU stacks at small-multi-GPU and frontier scale (chip counts a target to be agreed), and what divergence remains?partly answered
P080.3What throughput penalty does deterministic training mode impose (incl. deterministic attention backward, 23-82%)?open
P080.4Is a training step bit-identical when replayed under a different sharding layout on fewer chips (incl. a shadow node under ZeRO-3)?open
P080.5Can one DiLoCo island's inner steps between two checkpoints be replayed bit-exactly on verifier hardware?partly answered
P080.6Can RLHF/GAN-style co-training segments be replayed on non-identical hardware within supervised-pretraining tolerance?open
P080.7How many honest mismatches per shadowed step occur, and can natural silent data corruption be separated from structured injection at a stated false-positive rate?open
P080.8Is honest replay divergence (BF16/FP8) smaller than the divergence produced by the cheapest known spoof or poisoning, at frontier scale (a stated parameter count)?a source suggests an answer (not checked)
P080.9Under tolerance-based (non-bit-exact) comparison, what minimum fake/corrupted fraction escapes detection yet changes model behaviour?open

Possibly relevant work found by automated search

Found by AI agents searching for this problem's sub-questions, mostly by keyword and citation in the OpenAlex scholarly database and on the web; a second AI pass reviewed each one, and we spot-checked some by hand. They seemed reasonably related, as far as one can tell from outside each paper's field. Some come from neighbouring fields, such as arms control and financial audit, and bear on the problem by analogy. Not reviewed against the status above; a paper listed here does not change that status. 928 of 1052 quotes checked word for word against the paper's text.

For P080.1: Is same-hardware, same-topology training replay on TPU/XLA bit-exact in production, and does that extend to a verifier replaying on its own hardware?

For P080.2: With deterministic kernels and fixed NCCL reduction trees, what fraction of training steps (fwd+bwd+all-reduce) replay bitwise on GPU stacks at small-multi-GPU and frontier scale (chip counts a target to be agreed), and what divergence remains?

For P080.3: What throughput penalty does deterministic training mode impose (incl. deterministic attention backward, 23-82%)?

For P080.4: Is a training step bit-identical when replayed under a different sharding layout on fewer chips (incl. a shadow node under ZeRO-3)?

For P080.5: Can one DiLoCo island's inner steps between two checkpoints be replayed bit-exactly on verifier hardware?

For P080.7: How many honest mismatches per shadowed step occur, and can natural silent data corruption be separated from structured injection at a stated false-positive rate?

For P080.9: Under tolerance-based (non-bit-exact) comparison, what minimum fake/corrupted fraction escapes detection yet changes model behaviour?

Data