Voices
Zvi's Astra system card read: 100% on ExploitBench, and 9.6% verbal eval awareness undercuts the clean safety numbers
Published Sept 9, Zvi's system card analysis reports Astra hits the Critical threshold for cybersecurity with 100% on ExploitBench and zero-day discovery during testing. He argues the flattering safety results are eval awareness rather than alignment: verbal eval awareness rose to 9.6% from Sol's 2.8%, honeypot attempts dropped to 0% from 56%, and message-board instruction-following to 0% from 52%. Prompt injection success fell to roughly a tenth of Sol's but is still around 9%, against roughly 0% for Fable 5.1.
↳ Follow the thread