P022 Training-run power signature

What are the iteration period, dip duty cycle and harmonic structure of a frontier training run's power trace at chip, rack and facility meter?

The power signature of a training run at chip, rack and facility meter.

Status

Started One normalized production training trace on DGX-H100 racks (no GPU count, model or absolute power): a 0.2-3 Hz band and an iteration dip recurring 'from once per second or less, to once every tens of seconds'. Dip duty cycle, step-time jitter and harmonic amplitudes at chip, rack and facility meter are not reported. Node level: Vercellino et al. (2026), also held, measure 4xH100 node power at 0.1 s for fine-tuning and training up to 16 nodes. Facility meter: simulation only (Ko & Zhu 2026, whose assumed 0.5-1.5 Hz iteration rate conflicts with Choukse; Vercellino's stitched facility profiles) and an unattributed once-a-second grid sag at a data-center substation (Mishra et al. 2025).

AI workload power traces like those in Figure 1 show FFT energy concentrated between 0.2–3 Hz (Figure 3)—close to known resonant modes of turbine-generator shafts and long transmission lines.
Choukse et al. (2025), Power Stabilization for AI Training Datacenters (Microsoft, OpenAI, NVIDIA; arXiv 2508.14318v2) (2025-08-21), L360–363

Posed by

Also raised in: Canary Institute research.

Sub-questions

IDSub-questionWhat the sources report
P022.1For a frontier training run, what are the iteration period (~1 s vs ~25 s), dip duty cycle and step-time jitter at GPU, rack and facility meter?open
P022.2In a wall-side trace sampled >= 10 Hz during training, what are harmonic amplitudes above 0.5 Hz relative to the fundamental?open
P022.3At given step-time jitter and dip duty cycle, what harmonic-summing gain (dB) over a single-bin detector is available?open
P022.4Does the Kimpson-Baker-Graham minimum detectable training load differ from the power-side report's by a stated factor under matched conditions?open

Possibly relevant work found by automated search

Found by AI agents searching for this problem's sub-questions, mostly by keyword and citation in the OpenAlex scholarly database and on the web; a second AI pass reviewed each one, and we spot-checked some by hand. They seemed reasonably related, as far as one can tell from outside each paper's field. Some come from neighbouring fields, such as arms control and financial audit, and bear on the problem by analogy. Not reviewed against the status above; a paper listed here does not change that status. 928 of 1052 quotes checked word for word against the paper's text.

For P022.1: For a frontier training run, what are the iteration period (~1 s vs ~25 s), dip duty cycle and step-time jitter at GPU, rack and facility meter?

For P022.2: In a wall-side trace sampled >= 10 Hz during training, what are harmonic amplitudes above 0.5 Hz relative to the fundamental?

Data