Dan Luu Kills the 'Dynamic Languages Are 2–3x Cheaper for Coding Agents' Claim by Rerunning It on Real Tasks Instead of Rosetta Code
In a piece posted Aug 10, Dan Luu re-tests the widely-shared Alderson result that dynamic languages cost roughly 1/2 to 1/3 the tokens of static ones (J at 70 tokens average vs Clojure's 109), and shows it was an artifact of trivial Rosetta Code problems. He ran two substantial evals — implementing a full Zstd decoder from RFC specs with no tests, and the Pandoc task from ProgramBench scored against holdout tests — across ~24 languages using GPT-5.6 Sol at medium and ultra effort, measuring cost against correctness. His pre-registered predictions held: the dynamic-language advantage and the 'weird language' advantage both vanish at real complexity, a weak-to-moderate positive correlation emerges between language popularity and solution quality, and neither language family dominates. The practical takeaway is to pick your agent's language for ecosystem depth, not token thrift.
↳ Follow the thread