When should we pause? The AI stampede, with and without verification

Two states can each Build a superintelligence or Pause. A builder with no rival can take its time; H is the chance that such a project escapes its control on completion (a hazard per project, not an overall p(doom)). When two projects rush each other, θ is the extra danger that brings. Each state rates the catastrophe at Vc times the value of finishing first. The map colours each (H, Vc) by the game the two states are in; the circle is the state in the panel. Click the map to move it, then tick Verification to see what changes.

This state

Values and the laggard

Prisoner's Dilemma: stampede Stag Hunt: cooperate if possible Harmony: unilateral pause Deadlock: nobody wants a pause loss of control certain: θ above 1/H Chicken: whoever commits first builds
borders (the belief in the rival pausing that a pause needs) needs certainty (Stag line) needs 50-50 (risk-dominance line) needs nothing (Harmony line)

At these values

Notes

Reading the panel. Payoffs are in safe first builds (a safe first build = 1, mutual Pause = 0). Outlined cells are the equilibria. The "confidence required" figure under the map is the confidence a state needs that its rival is pausing before pausing is its own best move: 100% on the Stag line, 50% on the risk-dominance line, 0% at the Harmony line.

Vl, the rival building first. How much a state would dislike a world in which its rival completed a superintelligence aligned to the rival's values, safely, before it did. 1 means it minds that exactly as much as it would value building its own first; 0 means indifference; −1 means it would welcome the rival's values as its own.

The second 50-50 line under Verification. The 50-50 line can appear twice. A caught cheat doesn't automatically build first: the other side hurries to build its own, and that threat is what pulls the Stag line left. But hurrying is only worth it while the catastrophe looks tolerable. Past a certain hazard the betrayed side would rather stay paused than add a second rushed project, so a caught cheater builds first unopposed, cheating pays more again, and the confidence a pause needs jumps back up. The second 50-50 line marks where that jump carries it past 50%. The lesson for a treaty: the sooner a cheat is caught, the smaller the head start, and the further right that switch sits.

Impact of verification on confidence. Verification does two jobs, and the "confidence required" figure under the map shows only one of them. First, because a cheat can be caught, the payoffs change and the confidence a pause needs falls (here from certainty to 44%). Second, inspections produce evidence about what the rival is actually doing, so the confidence a state has rises: a clean record is (1 − α)/(1 − ε) to one evidence that the rival is pausing, 9.5 to 1 at the starting sliders, which turns even odds into 90% and a 10% prior into 51%. When the second number clears the first, pausing is the clear move. The second job sits outside the one-move game drawn here: it needs inspections to be running and to keep coming back clean.

Frontier and laggard. Whichever project is completed first is the frontier project; it escapes control with probability H. If the frontier project stays under control, the laggard's, completed second, escapes control with probability θ·H. So the overall loss-of-control hazard from both projects is H + (1 − H)·θ·H: 14.6% at the starting values (H 12%, θ 0.25). H is the quieter number, the hazard a project would carry if nobody were hurrying it; the 15% people quote as P(doom) is the two-project total under today's conditions, and this split of it into an unhurried hazard and a rushing penalty is ours. Any extra danger the stampede adds to either project shows up in θ. θ is named for Greek thrasos, reckless overboldness: the laggard is the project most tempted to cut corners to catch up. θ = 0 says the frontier project settles everything (the first system stops the second); then joining a stampede that is already underway adds no danger, only a chance of finishing first, and pausing on your own is never the best move. Harmony, the region where it is, exists only when θ > 0. Anyone who thinks a stampede makes each project more dangerous than careful building is adding to θ: if a lone project carries hazard H and the two-project total is R, then θ = (R − H) / (H(1 − H)). θ above 1 says the laggard's project is the more dangerous one (the one cutting corners to catch up); θ cannot exceed 1/H, where loss of control becomes certain: the slider stops there for the chosen H, and the map paints black the hazards at which the chosen θ would already pass that point.

Model. Two symmetric states choose Build or Pause, after Papenhausen's The AI Race is Not a Prisoner's Dilemma (LessWrong, 2026), with the laggard's hazard θ from his appendix (his α) and the head-start model of Roussel et al., Are we Doomed to an AI Race? Why Self-Interest Could Drive Countries Towards a Moratorium on Superintelligence (arXiv:2605.01297, 2026). Under verification a cheat is caught with probability ε, after which the other side takes the better of staying paused and hurrying to build its own from behind; a known cheater pays the punishment, in safe first builds. Rush noise is Roussel's σ: a head start L builds first with probability 1/(1 + e−L/σ); The game names are the standard game-theory ones (Prisoner's Dilemma, Stag Hunt, Harmony, Deadlock, Chicken), by which pairs of moves are equilibria; risk dominance follows Harsanyi & Selten.