Generative Reward Models Underperform in RL Because They Rank, Not Score — RRC Fixes the Mismatch

arXiv 2608.06310·medium signalXBlueskyLinkedInCopy link

RRC (arXiv 2608.06310, Aug 6) diagnoses why generative reward models, strong at response ranking, have failed to translate into RL gains: existing RL algorithms consume scalar scores while generative reward modeling is inherently comparative. RRC derives rewards from relative preference rankings via two strategies — self-competitive ranking over sampled responses, and anchor-guided ranking against a small reference set for scalability. Experiments across open-ended chat and reasoning benchmarks show consistent gains over existing reward construction approaches, with code released at github.com/wangclnlp/RRC.