SteerRM: Inference-Time Reward Model Debiasing via Sparse Autoencoders — +7.3pts on Hard Cases, No Retraining Required

arXiv·medium signalXBlueskyLinkedInCopy link

SteerRM identifies that reward models suffer from representation entanglement where format/style biases (favoring better-structured responses over semantically superior ones) are concentrated in shallow layers and transfer across architectures. Using sparse autoencoders, the approach isolates bias-related features via a strength-stability criterion from contrastive response pairs, then suppresses them at inference time without any retraining. The result is 7.3-point accuracy improvements on hard evaluation cases while preserving overall performance, offering alignment practitioners a drop-in, interpretable fix for a fundamental reward model failure mode.