When2Think sizes reasoning depth to difficulty, not length

When2Think sizes a reasoning model's thinking time to a difficulty score computed ahead of time, instead of applying one length penalty to every problem.

Teams that run reinforcement learning on reasoning models pay for this the usual way: a flat penalty trims tokens on easy problems where the model was overthinking, but it also trims tokens on hard problems, where the extra steps were earning accuracy. The paper calls that an efficiency tax, and says a difficulty-aware approach can avoid paying it.

The mechanism, called Instance-level Difficulty-Aware Control, pulls reference statistics on accuracy and token usage that were computed in advance, then uses those to set how much reasoning depth a given problem gets. Rewards come from a verifier rather than a trained reward model, and advantages get standardized within each batch instead of against a live reference model.

That design choice is the part worth sitting with. No learned reward model and no online reference queries means two of the more expensive pieces of a typical RL training run, a separate reward network and repeated calls to a reference policy, don't show up in the loop at all.

The abstract doesn't say how those difficulty statistics get computed for problems outside the training distribution, or what happens to accuracy once the precomputed numbers stop matching the problems the model actually sees.