P006 Sealed and maintenance-free operation

What spare and redundancy fraction and failure rates allow a sealed cluster to run at near-full capacity for years without physical maintenance?

Running sealed clusters without maintenance: spares, failure rates.

Terms: MTBF (mean time between failures): the average operating time before a component fails, used to size spares for an unattended cluster. N+k redundancy: a node built with k spare units beyond the N it needs to run.

Status

Started Measured failure rates on real field data: 18,688 NVIDIA K20X GPUs on ORNL's Titan (2012–2019), survival analysis, with about 11,000 boards replaced for a resistor-corrosion defect; corroborated by Bode et al. 2021 on Blue Waters (4,228 K20X GPUs, roughly 0.0035 replacements per GPU-year, no wear-out rise). Limits: decade-old GPUs in maintained, air-cooled HPC machines, not a sealed cluster of current accelerators. The spare and redundancy fraction clause has no work: Dean's 2025 hazard-rate note (0.001–0.18 per year) is a desk estimate drawn from these datasets and gives no spares fraction; the flexHEG, Harack and AISF rows are open questions.

We write about the last rework cycle and a reliability analysis of over 100,000 years of GPU lifetimes during Titan’s 6-year-long productive period. Using time between failures analysis and statistical survival analysis techniques, we find that GPU reliability is dependent on heat dissipation to an extent that strongly correlates with detailed nuances of the cooling architecture and job scheduling.
Ostrouchov et al. (2020), GPU Lifetimes on Titan Supercomputer: Survival Analysis and Reliability (ORNL) (2020), L52–58

Posed by

Also raised in: Petrie & Aarne (2025), Technical Options for Flexible Hardware-Enabled Guarantees (flexHEG Part 2); AI Security Forum, Requests for Discussion; Proofworks, Living Verification Documents.

Sub-questions

IDSub-questionWhat the sources report
P006.1What spare/overprovisioning fraction keeps a sealed N-GPU cluster at >= 90% capacity (a threshold set by this survey) over multi-year operation?open
P006.2What is the annual unrecoverable-failure hazard of H100-class GPUs at high utilisation?open
P006.3What is the MTBF of an N+k redundant GPU node (node size set by the researcher) at high utilisation?open
P006.4What is the mean interval between physical maintenance events per accelerator in frontier clusters, and how far can it be reduced?open
P006.5Has a prototype sealed, monitored compute container been built and run under a two-party monitoring protocol?open
P006.6What is the minimum GPU count that serves a current frontier model at production throughput (sets enclosure size)?open

Possibly relevant work found by automated search

Found by AI agents searching for this problem's sub-questions, mostly by keyword and citation in the OpenAlex scholarly database and on the web; a second AI pass reviewed each one, and we spot-checked some by hand. They seemed reasonably related, as far as one can tell from outside each paper's field. Some come from neighbouring fields, such as arms control and financial audit, and bear on the problem by analogy. Not reviewed against the status above; a paper listed here does not change that status. 928 of 1052 quotes checked word for word against the paper's text.

For P006.1: What spare/overprovisioning fraction keeps a sealed N-GPU cluster at >= 90% capacity (a threshold set by this survey) over multi-year operation?

For P006.2: What is the annual unrecoverable-failure hazard of H100-class GPUs at high utilisation?

Data