P050 Verifier test secrecy
Do secret verifier tests catch spoofs tuned to pass the disclosed tests, and can that secrecy survive a state adversary?
Whether secret verifier tests resist spoofs tuned to disclosed tests.
Status
Posed
Posed by
- Petrie & Aarne (2025), Technical Options for Flexible Hardware-Enabled Guarantees (flexHEG Part 2):
This difficulty depends on the classification logic being kept secret, which would be challenging to do against state adversaries
(Automated Scanning, p29, L1322–1323)
Also raised in: Shavit (2023), What Does It Take to Catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring; Harack (2026), AI Verification: Infrastructure for Prosperity, Governance, and Peace (Lawfare).
Sub-questions
| ID | Sub-question | What the sources report |
|---|---|---|
| P050.1 | Does a held-out (secret) verification test or classifier catch spoofs tuned to pass the disclosed tests, and can secrecy survive a state adversary? | partly answered |
Possibly relevant work found by automated search
Found by AI agents searching for this problem's sub-questions, mostly by keyword and citation in the OpenAlex scholarly database and on the web; a second AI pass reviewed each one, and we spot-checked some by hand. They seemed reasonably related, as far as one can tell from outside each paper's field. Some come from neighbouring fields, such as arms control and financial audit, and bear on the problem by analogy. Not reviewed against the status above; a paper listed here does not change that status. 928 of 1052 quotes checked word for word against the paper's text.
For P050.1: Does a held-out (secret) verification test or classifier catch spoofs tuned to pass the disclosed tests, and can secrecy survive a state adversary?
Data
- problems.csv — one row per problem
- problem_status.csv — status, scope and evidence per problem
- problem_sources.csv — the posed-by quote and other documents per problem
- leaves.csv — all sub-questions, including those without a checkable form
- leaf_sources.csv — every source row behind each sub-question