Code Review Comment Datasets Contain Structurally Misaligned Pairs That Cleaning Alone Cannot Fix

arXiv 2607.25851·low signalXBlueskyLinkedInCopy link

A paper posted 2026-07-28 (arXiv 2607.25851) pushes back on the current wave of LLM-based dataset cleaning for automated code review, arguing that problematic training instances are not homogeneous and that some limitations come from the task formulation itself rather than noise. Inspecting a widely used review-comment dataset, the authors identify misaligned training pairs where the relationship between code change and review comment provides no reliable signal for generating actionable feedback from localized inputs, and derive a taxonomy with three recurring sources: semantic ambiguity, lack of actionability, and context dependence. Folding that taxonomy into LLM-based filtering still leaves misalignment detection hard, leading them to call for explicit validity criteria, richer contextual inputs, and evaluation aligned with review intent.