P078 Compute accounting: MFU, HFU and declared operations
Can a model predict achievable hardware utilisation closely enough to detect padded or undeclared-efficiency compute, at a tolerable false-alarm rate?
MFU/HFU modelling and detecting padded or undeclared-efficiency compute.
Terms: MFU / HFU (model-FLOP utilisation / hardware-FLOP utilisation): the fraction of a chip's peak compute a training run actually uses - two accounting conventions that differ in how they count recomputed work.
Status
Analyzed A proposed method with a desk analysis of its error sources: estimate achievable utilisation for a declared workload and cluster, compare with declared operations, and back it with physical signatures when an operator's efficiency exceeds the estimate. No model has been validated against measured runs, no padding or no-op detector has been built, and the false-alarm rate for honest operators is only flagged as a risk ('honest Provers [may be] unable to account for their estimated compute use'). Attacks named without defences shown: secret backend optimisation, speculative decoding (1.5-3x), Pareto-inefficient menu choices, restructuring to minimise logged operations. Cankaya's system overview proposes a resource-occupation simulation cross-compared with throughput, also paper-only.
For a given workload type and cluster, optimal MFU could be coarsely approximated as the MFU known to have been achieved on similar workloads and clusters, on which there is private and public data (Epoch AI, 2024b). Optimal MFU could be more precisely estimated with theoretical modeling (Erdil and Schneider-Joseph, 2024; Erdil, 2025). Still, significant error bars may remain.
Posed by
- Cankaya (2025), Catching Misreporting About ML Hardware Use by Turning Noise into Signal:
But this defense fails if the Prover secretly optimizes their backend beyond what the verifier expects.
(Background, p.4; lines 102–103) (source states the problem but not the quantity the question asks for)
Also raised in: Baker, Kulp, Marks, Brundage & Heim (2025), Verifying International Agreements on AI: Six Layers of Verification for Rules on Large-Scale AI Development and Deployment (RAND WR-A4077-1); Scher & Thiergart (2024), Mechanisms to Verify International Agreements About AI Development (MIRI); Cankaya (2026), A System Overview for Near-Term, Low-Trust AI Compute Verification (v0.2); AI Security Forum, Requests for Discussion; Proofworks, Living Verification Documents.
Sub-questions
| ID | Sub-question | What the sources report |
|---|---|---|
| P078.1 | Given a workload and cluster specs, does a model predict achievable MFU / resource occupation within a stated tolerance of measured runs, including latency-optimised legitimate uses? | open |
| P078.2 | Under a conservative-MFU rule, what false-alarm rate results for honest operators? | open |
| P078.3 | What is the measured HFU/MFU ratio (recomputation) for frontier training, and what throughput does forbidding recomputation cost? | partly answered |
| P078.4 | Can a check detect padded or no-op computation that inflates apparent operations by a stated fraction (a target to be agreed) within a declared power envelope? | open |
| P078.5 | Can a verifier detect undeclared efficiency gains (secret backend optimisation, speculative decoding, Pareto-inefficient menu choices, restructuring to minimise logged FLOP) that free hidden capacity? | partly answered |
Sub-questions not yet stated precisely enough to check
| ID | Sub-question |
|---|---|
| P078.6 | Does counting declared model operations need new R&D? |
Possibly relevant work found by automated search
Found by AI agents searching for this problem's sub-questions, mostly by keyword and citation in the OpenAlex scholarly database and on the web; a second AI pass reviewed each one, and we spot-checked some by hand. They seemed reasonably related, as far as one can tell from outside each paper's field. Some come from neighbouring fields, such as arms control and financial audit, and bear on the problem by analogy. Not reviewed against the status above; a paper listed here does not change that status. 928 of 1052 quotes checked word for word against the paper's text.
For P078.1: Given a workload and cluster specs, does a model predict achievable MFU / resource occupation within a stated tolerance of measured runs, including latency-optimised legitimate uses?
For P078.4: Can a check detect padded or no-op computation that inflates apparent operations by a stated fraction (a target to be agreed) within a declared power envelope?
Data
- problems.csv — one row per problem
- problem_status.csv — status, scope and evidence per problem
- problem_sources.csv — the posed-by quote and other documents per problem
- leaves.csv — all sub-questions, including those without a checkable form
- leaf_sources.csv — every source row behind each sub-question