P006 Sealed and maintenance-free operation
What spare and redundancy fraction and failure rates allow a sealed cluster to run at near-full capacity for years without physical maintenance?
Running sealed clusters without maintenance: spares, failure rates.
Terms: MTBF (mean time between failures): the average operating time before a component fails, used to size spares for an unattended cluster. N+k redundancy: a node built with k spare units beyond the N it needs to run.
Status
Started Measured failure rates on real field data: 18,688 NVIDIA K20X GPUs on ORNL's Titan (2012–2019), survival analysis, with about 11,000 boards replaced for a resistor-corrosion defect; corroborated by Bode et al. 2021 on Blue Waters (4,228 K20X GPUs, roughly 0.0035 replacements per GPU-year, no wear-out rise). Limits: decade-old GPUs in maintained, air-cooled HPC machines, not a sealed cluster of current accelerators. The spare and redundancy fraction clause has no work: Dean's 2025 hazard-rate note (0.001–0.18 per year) is a desk estimate drawn from these datasets and gives no spares fraction; the flexHEG, Harack and AISF rows are open questions.
We write about the last rework cycle and a reliability analysis of over 100,000 years of GPU lifetimes during Titan’s 6-year-long productive period. Using time between failures analysis and statistical survival analysis techniques, we find that GPU reliability is dependent on heat dissipation to an extent that strongly correlates with detailed nuances of the cooling architecture and job scheduling.
Posed by
- Harack et al. (2025), Verification for International AI Governance (Oxford AIGI):
significant inefficiencies might be introduced by the requirement that hardware not be manipulated after it leaves the verification facility
(§2.5.4.2, p.70, L3826–3827) (source states the problem but not the quantity the question asks for)
Also raised in: Petrie & Aarne (2025), Technical Options for Flexible Hardware-Enabled Guarantees (flexHEG Part 2); AI Security Forum, Requests for Discussion; Proofworks, Living Verification Documents.
Sub-questions
| ID | Sub-question | What the sources report |
|---|---|---|
| P006.1 | What spare/overprovisioning fraction keeps a sealed N-GPU cluster at >= 90% capacity (a threshold set by this survey) over multi-year operation? | open |
| P006.2 | What is the annual unrecoverable-failure hazard of H100-class GPUs at high utilisation? | open |
| P006.3 | What is the MTBF of an N+k redundant GPU node (node size set by the researcher) at high utilisation? | open |
| P006.4 | What is the mean interval between physical maintenance events per accelerator in frontier clusters, and how far can it be reduced? | open |
| P006.5 | Has a prototype sealed, monitored compute container been built and run under a two-party monitoring protocol? | open |
| P006.6 | What is the minimum GPU count that serves a current frontier model at production throughput (sets enclosure size)? | open |
Possibly relevant work found by automated search
Found by AI agents searching for this problem's sub-questions, mostly by keyword and citation in the OpenAlex scholarly database and on the web; a second AI pass reviewed each one, and we spot-checked some by hand. They seemed reasonably related, as far as one can tell from outside each paper's field. Some come from neighbouring fields, such as arms control and financial audit, and bear on the problem by analogy. Not reviewed against the status above; a paper listed here does not change that status. 928 of 1052 quotes checked word for word against the paper's text.
For P006.1: What spare/overprovisioning fraction keeps a sealed N-GPU cluster at >= 90% capacity (a threshold set by this survey) over multi-year operation?
For P006.2: What is the annual unrecoverable-failure hazard of H100-class GPUs at high utilisation?
Data
- problems.csv — one row per problem
- problem_status.csv — status, scope and evidence per problem
- problem_sources.csv — the posed-by quote and other documents per problem
- leaves.csv — all sub-questions, including those without a checkable form
- leaf_sources.csv — every source row behind each sub-question