Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-03-20 · source-backed
EsoLang-Bench (90 points, 49 comments on HN) evaluates LLMs on esoteric programming languages specifically designed to prevent memorization, isolating genuine reasoning from training-data recall. Directly challenges the validity of HumanEval, SWE-bench, and similar leaderboards where models may pattern-match memorized solutions. Extends the benchmark-validity critique building in the practitioner community.
Each link below shares sources, entities, or timing with this story.
Shared entities / Earlier coverage
Both cover Bench, EsoLang, SWE; earlier Bench coverage from 2026-03-11.
Shared entities / Shared topic / What happened next
Both cover Directly, LLMs; overlapping topics (directly, llms); picks up the Directly thread on 2026-06-05.
Shared entities / Shared topic / Earlier coverage
Both cover HumanEval, SWE; overlapping topics (designed, humaneval); earlier HumanEval coverage from 2026-03-04.
Both cover Directly, LLMs; overlapping topics (challeng, directly); earlier Directly coverage from 2026-02-12.