Meta's ARCTIC beats baseline AI reviewers 2.4x on quality

Meta researchers built ARCTIC to grade agent-written code against what a developer actually meant, not against style rules. Agents write code faster than reviewers can absorb it. The AI tools built to help spend too much time flagging style, not enough on correctness, security or performance.

The paper comes from a Meta-affiliated team that includes Nachi Nagappan and Peter Rigby, posted to arXiv in July.

ARCTIC's first pass predicts developer intent straight from the conversation log with the agent, scoring an F1 of 0.86.

Its second pass is drift detection. ARCTIC translates the agent's code back into a plain description of what it does, then checks that against what the developer asked for. The drift score matched human annotators at a QWK of 0.907.

A code spotlight step ranks which parts of a diff need the closest read.

Against a baseline AI reviewer, ARCTIC scored 2.4 times higher on quality estimation while using 5 times fewer tokens, per the paper. In rollout, the drift-detection layer cut code misalignment by another 5.76 points beyond what intent prediction alone caught (p=0.026). Intent prediction alone drew 90.2% approval.

The paper doesn't say what happens when the intent itself is wrong. If a developer asks an agent for the wrong thing, ARCTIC has nothing to flag. It's built to catch code drifting from instructions, not instructions that were bad from the start.