P112 Replay mismatches and dispute settlement
What is the baseline rate of benign replay mismatches, and can a dispute procedure attribute a mismatch to hardware fault versus cheating?
The rate of benign mismatches in replay checks, and whether a dispute procedure can tell a hardware fault from cheating.
Status
Started Delegation setting: when two executions of the same ML program disagree, a referee bisects by checkpoint hashes to one training step and one operator, re-executes it and rules; benign floating-point mismatches are removed by bitwise-reproducible operators, implemented and benchmarked on DistilBERT and Llama-3.1 1B and 8B in FP32. It finds which party is wrong, not whether the cause was a hardware fault or cheating; no baseline rate of benign replay mismatches is measured; Cankaya (2026) claims his own bitwise replay settles statistical false alarms, a self-report at the same stage; GPU-stack training replay at frontier scale not shown.
we design Verde, a dispute arbitration protocol that efficiently handles the large scale and graph-based computational model of modern ML programs. For (2), we build RepOps (Reproducible Operators), a library that eliminates hardware “nondeterminism” by controlling the order of floating point operations performed on all hardware. Our implementation shows that refereed delegation achieves both strong guarantees for clients and practical overheads for compute providers.
Raised in: Cankaya (2026), A System Overview for Near-Term, Low-Trust AI Compute Verification (v0.2); Cankaya (2026), Catching Misreporting About ML Hardware Use by Turning Noise into Signal, Part II; Cankaya (2026), Compute Verification FAQs; Canary Institute research.
Sub-questions
| ID | Sub-question | What the sources report |
|---|---|---|
| P112.1 | What is the baseline rate of benign replay mismatches (bit flips, software faults) per record? | open |
| P112.2 | What is the false-alarm rate of statistical replay per record, and can bitwise replay settle such disputes? | partly answered |
| P112.3 | What is each verification mechanism's measured false-positive rate before deployment? | open |
| P112.4 | Can a written dispute procedure attribute a replay mismatch to hardware fault vs cheating without the prover reproducing it (e.g. on GPU stacks)? | open |
Possibly relevant work found by automated search
Found by AI agents searching for this problem's sub-questions, mostly by keyword and citation in the OpenAlex scholarly database and on the web; a second AI pass reviewed each one, and we spot-checked some by hand. They seemed reasonably related, as far as one can tell from outside each paper's field. Some come from neighbouring fields, such as arms control and financial audit, and bear on the problem by analogy. Not reviewed against the status above; a paper listed here does not change that status. 928 of 1052 quotes checked word for word against the paper's text.
For P112.1: What is the baseline rate of benign replay mismatches (bit flips, software faults) per record?
For P112.2: What is the false-alarm rate of statistical replay per record, and can bitwise replay settle such disputes?
Data
- problems.csv — one row per problem
- problem_status.csv — status, scope and evidence per problem
- problem_sources.csv — the posed-by quote and other documents per problem
- leaves.csv — all sub-questions, including those without a checkable form
- leaf_sources.csv — every source row behind each sub-question