ARC Prize is crowdsourcing tasks that beat Astra

Greg Kamradt, president of ARC Prize, posted on r/singularity asking for suggestions on tasks Astra can't solve, in a thread soliciting ideas from the subreddit.

The gap matters for anyone deciding how much weight to put on a saturated benchmark. ARC built its name on generating its own abstract reasoning puzzles, grids where a model has to infer a pattern from a handful of examples. Astra reportedly cleared that benchmark, so the organization that owns it is now outsourcing the next round of test design to whoever shows up in a comment section.

What the crowd is proposing looks nothing like the original puzzles. Top replies name a long-horizon grind like a RuneScape fire cape, and getting backstabbed in a game of Civilization, which tests whether a model can read social betrayal and shifting alliances. Persistence over a long stretch and social deception sit on a different axis than compressed visual reasoning. Probably the harder one to fake.

A model that solves ARC's grid puzzles has proven it's good at ARC's grid puzzles. Whether that generalizes to grinding out a long task or noticing it's being lied to in a game is the open question the thread is chasing, and ARC doesn't have those tasks built yet.

Watch whether ARC turns any of these suggestions into a scored benchmark, or whether the thread just becomes a public research note with no follow-through.