P098 AI-enabled verification

When will AI auditors be competent and secure enough to deploy, and can they resist jailbreaks, prompt injection and backdoors by a sophisticated adversary?

AI-enabled verification and auditor robustness.

Status

Started A built LLM-auditor prototype (TEE and attestation emulated on a CPU without SEV; frontier model called remotely) evaluated on 21 real codebases: perfect discrimination of correct from wrong papers on objective questions, less stable on subjective ones; the audit task is software-artifact verification, not compute verification. The 'when' clause has no timeline estimate. Resistance to jailbreaks, prompt injection and backdoors: not tested; the author concedes prompt injection cannot be prevented and designs for it to leave evidence in a signed transcript.

We demonstrate this architecture by automating the artifact evaluation process for 21 peer-reviewed computer science papers with released codebases on GitHub (e.g. Does the codebase implement the system described in the paper?). We verified five high-level properties of these codebases described in the corresponding publications, treating the source code as private.
Rowstron (2026), Agentic Witnessing: Pragmatic and Scalable TEE-Enabled Privacy-Preserving Auditing (ARIA; arXiv 2604.24203) (2026-04-27), L63–66

Posed by

Also raised in: Scher & Thiergart (2024), Mechanisms to Verify International Agreements About AI Development (MIRI); Hausenloy & Li (2026), The Compute Verification Post.

Sub-questions

None with a checkable form.

Sub-questions not yet stated precisely enough to check

IDSub-question
P098.1When will AI auditors (zero-knowledge review, 'Evaluations Bot', agent-written tests) be competent, reliable and secure enough to deploy for verification?
P098.2Can AI-agent auditors resist jailbreaks, prompt injection and backdoors by a sophisticated adversary?

Possibly relevant work found by automated search

Searched; nothing found for these sub-questions.

Data