Human-in-the-Loop Pipeline Labels 124,000 GitHub Conversations for Toxicity Using One Small Local LLM Call Plus a Random Forest Validator
A paper posted 2026-07-28 (arXiv 2607.25946) presents a low-cost annotation methodology that makes domain-calibrated toxicity labeling practical at scale, addressing why prior GitHub toxicity studies stayed small: implicit, context-dependent toxicity defeats both fully manual annotation and naive LLM labeling. A single call to a small local LLM produces both a toxicity prediction and interpretable event-category scores; a lightweight Random Forest validator then uses those scores to flag only the conversations most likely mislabeled, routing human review where it matters. The validator beats confidence-based and multi-LLM baselines at low added cost. Applied to over 124,000 GitHub issue and pull-request conversations, the resulting dataset confirms some prior small-scale findings and qualifies others.
↳ Follow the thread