DGPO: Direction-Aware Groupwise Preference Optimization Beyond Pairwise Comparisons

arXiv·medium signal

DGPO extends preference optimization from pairwise to group-level by organizing forward and reverse question-answer comparisons, explicitly modeling direction-aware alignment through multi-candidate supervision signals. The lightweight framework preserves reasoning diversity while enforcing directional consistency — a known failure mode where current methods like DPO collapse response variety. Practical alternative for teams doing RLHF-style training.