Dan Luu says AI-assisted coding failures got harder to spot

Dan Luu has been tracking a specific failure mode since early 2025: developers who let an LLM write code, take its output on faith, and never check whether it actually works. In his latest update, he says the pattern changed shape rather than went away.

Early on, brain-off work produced software that obviously didn't work. Broken builds, code that failed on the first run. Now, per Luu, it produces software that sometimes almost works, which is a harder problem because the failure doesn't announce itself.

He borrows a term from Niklas Gruhn for the human's role in this setup: a "meat proxy." That's someone who copies an error message back to the model and pastes in whatever it suggests, without understanding either the error or the fix. The person is still in the loop, but they've stopped functioning as a check on the output.

Luu's argument is that this isn't a problem better models fix. A more capable LLM doesn't restore a check that a human has already stopped performing. The failure mode lives in the workflow, not in model quality, so it persists regardless of which model is doing the generating.

He doesn't offer a fix in the piece, and doesn't quantify how common brain-off development is now versus a year ago. What he's documenting is a shift in kind: from failures you catch because the thing won't run, to failures you miss because the thing runs fine until it doesn't. That's a harder thing to build guardrails around, because the usual signal, a broken build, stops firing.