P102 Low-communication distributed training versus bandwidth

At what inter-site/inter-pod bandwidth does low-communication distributed training become uneconomic relative to a co-located baseline for a frontier run?

Efficiency of multi-site / low-communication training as a function of inter-site or inter-pod bandwidth, and the bandwidth thresholds that would make it uneconomic.

Terms: low-communication distributed training (e.g. DiLoCo): training methods that let machines in different places cooperate while exchanging far less data than usual, cutting the bandwidth needed between sites. n: GPUs per pod. B: the cap on external link bandwidth. T: the time allowed for training.

Status

Analyzed Simulated cost for 10^25 FLOP of local-equivalent compute with DiLoCo-family methods over 100 Mbps links and nodes below 16 H100-equivalents; extrapolated 10-300x beyond tested model size and up to 250x in replica count; calibrated against Covenant-72B but not a measured run. No bandwidth at which training becomes uneconomic is identified; the paper's cost table spans 10 Mbps to 1 Gbps.

At 100 Mbps bandwidth, it costs about 3× more to do training distributed relative to centralized, but it is not a substantial barrier to feasibility.
Rahman (2026), Does Distributed Training Undermine Compute Governance? (MIRI), arXiv 2605.29359 (2026-05-28), L498–499

Posed by

Also raised in: Harack et al. (2025), Verification for International AI Governance (Oxford AIGI); Baker, Kulp, Marks, Brundage & Heim (2025), Verifying International Agreements on AI: Six Layers of Verification for Rules on Large-Scale AI Development and Deployment (RAND WR-A4077-1); Scher & Thiergart (2024), Mechanisms to Verify International Agreements About AI Development (MIRI); Aarne & Petrie (2025), International Security Applications of Flexible Hardware-Enabled Guarantees (flexHEG Part 3); Canary Institute research; Ansari (2026), Hardware-Level Governance of AI Compute: A Feasibility Taxonomy for Regulatory Compliance and Treaty Verification; Kulp et al. (2024), Hardware-Enabled Governance Mechanisms (RAND WR-A3056-1); Moon, Vedula, Geneson & Bar-on (2025), Strategies and Detection Gaps in a Game-Theoretic Model of Compute Governance (RAND RRA3686-1); Rahman (2026), Does Distributed Training Undermine Compute Governance? (MIRI); AI Security Forum, Requests for Discussion; Proofworks, Living Verification Documents.

Sub-questions

IDSub-questionWhat the sources report
P102.1At what inter-site bandwidth does low-communication training (DiLoCo-style) across many ~5,000-chip sites reach a stated fraction of centralised efficiency for a ≥10^25 FLOP run (a run size set by this survey)?open
P102.2At what inter-node bandwidth and node count does decentralised training on consumer GPUs reach a stated fraction of datacentre efficiency for a 10^24 FLOP run?implied by a source, not stated as open
P102.3What is the crossover shard size (kW) between the DiLoCo economic floor and the electricity-metering (feeder-tier) detection floor?partly answered
P102.4At what inter-pod / external bandwidth does a frontier run (e.g. 1e26 FLOP) using current low-communication methods take a stated factor longer (or >1 year) than unrestricted, and how robust is that threshold to algorithmic advances?open
P102.5Has any distributed-training method done useful frontier-style pretraining with <=1 Mb/s per inter-pod link (e.g. between 128-GPU pods) at <10x cost?partly answered
P102.6With pods of n GPUs and external links capped at B, what is the largest model trainable in time T (pod-size threshold; tentatively <65 GPUs)?open
P102.7What minimum external bandwidth do named allowed workloads (hosted inference, image/audio/video generation, legitimately variable usage) need, and do they fit under a proposed cap?open
P102.8Can evaders be prevented from obtaining fast internet connections (broadband bandwidth caps)?a source suggests an answer (not checked)
P102.9What combined communication reduction do DiLoCo-style inner steps plus gradient compression achieve at >=7B parameters with FLOP-matched loss?open
P102.10At equal FLOP, what is the loss gap between DiLoCo-style training over internet links and a co-located baseline at >=1e25 FLOP?open
P102.11What is the cost premium over centralized training for a 1e25 run with 10-30 Mbps uplinks per island?open
P102.12Has any DiLoCo-family run above 10^24 FLOP been demonstrated with at most 16 H100-equivalents per node over <=100 Mbps links?a source suggests an answer (not checked)
P102.13What fraction of single-cluster efficiency has been demonstrated for runs of at least 10^24 FLOP split across sites, and at a stated replica count and model scale (targets to be agreed)?a source suggests an answer (not checked)
P102.14What is the efficiency of distributed RL / fine-tuning over internet links at a stated model scale (a target to be agreed)?implied by a source, not stated as open
P102.15Is there published evidence of <=20 inner steps per sync across >=100 workers (both thresholds set by this survey), and what update-compression ratio holds at every-step syncing?open
P102.16Does a power trace sampled at a stated rate show a feature at exchange time in overlapped-communication (Streaming DiLoCo) training?open
P102.17What throughput and cost premium does training at a small legal island cap (e.g. 16 H100-equivalents, with host-memory offload) incur?partly answered
P102.18From one seized node, can a forensic examiner establish participation in a multi-node run and identify peers?open

Sub-questions not yet stated precisely enough to check

IDSub-question
P102.19Can rearranging context length / batch size, or sparse architectures compressing activations, let large-model inference or pipeline-parallel training evade a cap?
P102.20How should bandwidth caps be adapted as communication requirements fall with algorithmic progress and adversary investment?
P102.21If training becomes dominated by synthetic-data inference, do interconnect limits still bind?
P102.22Does decentralised peer-to-peer training remove any practical off-switch?
P102.23How much does uncertainty in the loss-to-FLOP conversion above 6e23 FLOP affect threshold estimates?
P102.24What is the efficiency of a proof-of-workload protocol for maximally distributed training?

Possibly relevant work found by automated search

Found by AI agents searching for this problem's sub-questions, mostly by keyword and citation in the OpenAlex scholarly database and on the web; a second AI pass reviewed each one, and we spot-checked some by hand. They seemed reasonably related, as far as one can tell from outside each paper's field. Some come from neighbouring fields, such as arms control and financial audit, and bear on the problem by analogy. Not reviewed against the status above; a paper listed here does not change that status. 928 of 1052 quotes checked word for word against the paper's text.

For P102.3: What is the crossover shard size (kW) between the DiLoCo economic floor and the electricity-metering (feeder-tier) detection floor?

For P102.4: At what inter-pod / external bandwidth does a frontier run (e.g. 1e26 FLOP) using current low-communication methods take a stated factor longer (or >1 year) than unrestricted, and how robust is that threshold to algorithmic advances?

For P102.5: Has any distributed-training method done useful frontier-style pretraining with <=1 Mb/s per inter-pod link (e.g. between 128-GPU pods) at <10x cost?

For P102.9: What combined communication reduction do DiLoCo-style inner steps plus gradient compression achieve at >=7B parameters with FLOP-matched loss?

For P102.17: What throughput and cost premium does training at a small legal island cap (e.g. 16 H100-equivalents, with host-memory offload) incur?

For P102.18: From one seized node, can a forensic examiner establish participation in a multi-node run and identify peers?

Data