P050 Verifier test secrecy

Do secret verifier tests catch spoofs tuned to pass the disclosed tests, and can that secrecy survive a state adversary?

Whether secret verifier tests resist spoofs tuned to disclosed tests.

Status

Posed

Posed by

Also raised in: Shavit (2023), What Does It Take to Catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring; Harack (2026), AI Verification: Infrastructure for Prosperity, Governance, and Peace (Lawfare).

Sub-questions

IDSub-questionWhat the sources report
P050.1Does a held-out (secret) verification test or classifier catch spoofs tuned to pass the disclosed tests, and can secrecy survive a state adversary?partly answered

Possibly relevant work found by automated search

Found by AI agents searching for this problem's sub-questions, mostly by keyword and citation in the OpenAlex scholarly database and on the web; a second AI pass reviewed each one, and we spot-checked some by hand. They seemed reasonably related, as far as one can tell from outside each paper's field. Some come from neighbouring fields, such as arms control and financial audit, and bear on the problem by analogy. Not reviewed against the status above; a paper listed here does not change that status. 928 of 1052 quotes checked word for word against the paper's text.

For P050.1: Does a held-out (secret) verification test or classifier catch spoofs tuned to pass the disclosed tests, and can secrecy survive a state adversary?

Data