Two Ways to Make Alignment Evaluations Look Like Real Deployments, and They Compose

arXiv 2609.02302·medium signalXBlueskyLinkedInCopy link

Evaluation awareness — capable models detecting that they are under test rather than deployed — limits what a safety evaluation can conclude, so this work attacks realism directly with two techniques. Critique refinement spends inference-time compute per simulator action, generating multiple candidate actions, refining them with feedback from an instance of the target model on how to look more realistic, and continuing from the most deployment-like candidate. DISH (Deployment-Imitating SWE-Agent Harness) wraps the target in an agent harness to shrink the simulated-versus-real gap in coding settings; across multiple target models the two compose for larger realism gains than either alone, and the extra compute buys more realism than simply running longer audits.