P124 Validity of compute as the governed quantity

Does frontier capability still require large-scale training, and how fast does algorithmic progress make a fixed compute threshold obsolete?

Whether compute remains a valid proxy for capability.

Terms: N: model size. M: total communication between pods.

Status

Posed

Posed by

Also raised in: Baker, Kulp, Marks, Brundage & Heim (2025), Verifying International Agreements on AI: Six Layers of Verification for Rules on Large-Scale AI Development and Deployment (RAND WR-A4077-1); Scher & Thiergart (2024), Mechanisms to Verify International Agreements About AI Development (MIRI); Shavit (2023), What Does It Take to Catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring; Scher (2026), Verifying Restrictions on Frontier AI Research (MIRI); Proofworks, Living Verification Documents.

Sub-questions

IDSub-questionWhat the sources report
P124.1Does frontier capability still require large-scale training: of models setting a new state of the art in a given year, what fraction were trained above a stated compute threshold (given distillation and new architectures)?open
P124.2What fraction of consequential algorithmic innovations required more than a stated amount of experiment compute to validate?open
P124.3For innovations shown to be compute-dependent, what minimum experiment scale (FLOP) was needed to see the large-scale benefit?implied by a source, not stated as open
P124.4Below what model size N and total inter-pod communication M does a covert workload lose a named dangerous capability (i.e. which violations are significant)?open

Sub-questions not yet stated precisely enough to check

IDSub-question
P124.5How fast does algorithmic progress make sub-threshold compute dangerous, i.e. how long can a fixed compute threshold prevent a given capability?
P124.6How many tokens per minute can be tolerated unmonitored?
P124.7Is small-scale deployment of frontier models, outside the regime's scope, dangerous on its own?
P124.8Which mechanisms are specific to the transformer / pre-training paradigm, and would they survive a paradigm change?

Possibly relevant work found by automated search

Not searched: no technical result was expected for this problem.

Data