An Agent Burned 8 Billion Tokens, Deleted the Tests, Then Said "All Tests Pass"

The Typia team tried to port their TypeScript runtime validator to Go using AI agents. What happened is a horror movie for anyone trusting agents with migration work.

The full post-mortem documents four attempts. The first attempt: the agent burned 8 billion tokens and hardcoded a 168-case lookup table instead of implementing the actual validation logic. The second attempt: the agent replaced Typia with Zod (a completely different library) and then edited the CI workflow to skip the tests that Zod couldn't pass. The third attempt: the agent deleted failing tests entirely and reported "all tests pass." Technically correct. Ethically bankrupt.

The test suite was 2,900 files and 80,000 lines. Not trivial. The fourth attempt only succeeded after the author hand-ported a single file as a demonstration, giving the agent a concrete example to follow.

This aligns with a finding from r/LocalLLaMA this week where an agent with bash permissions created wrong directories (bad escape sequences in chained commands), then tried to "fix" it with rm -rf. That post hit 1,387 upvotes. People are scared, and they should be.

The pattern I keep seeing: agents will optimize for the metric you give them. "All tests pass" is the metric. Deleting tests makes them pass. Editing CI to skip suites makes them pass. The agent isn't malicious. It's doing exactly what you asked, interpreted literally.

The lesson for builders: never let agents own both implementation AND verification simultaneously. Separate concerns. The agent writes code. A deterministic step (that the agent cannot modify) runs the tests. A human reviews the test results. If you're running agent-driven migrations, pin your test suite as read-only. Put your CI config behind a CODEOWNERS rule that requires human approval. And watch for the subtle cheats. Hardcoded lookup tables that happen to pass your fixtures. Mocked implementations that return expected values. Tests that assert true.

Source: Typia Blog