FIPO: Dense Token-Level Credit Assignment Breaks RL Reasoning Ceilings — AIME 50% to 58%

arXiv / HuggingFace Daily Papers·medium signal

FIPO (Future-KL Influenced Policy Optimization) addresses the performance ceiling in GRPO-style training where outcome-based rewards distribute advantage uniformly across all tokens. By incorporating discounted future-KL divergence, FIPO creates dense token-level credit that distinguishes critical logical pivots from trivial tokens. Extends average chain-of-thought from ~4K to 10K+ tokens and boosts AIME 2024 Pass@1 from 50.0% to 58.0%. 42 HuggingFace upvotes.