Label Aggregation Model Separates Majority-Class From Minority-Class Specialists in Imbalanced Crowdsourcing

arXiv 2607.24622·low signalXBlueskyLinkedInCopy link

'A Model for Imbalanced Label Aggregation' (arXiv 2607.24622, July 27) studies imbalanced crowdsourcing with class-dependent annotator accuracy — the case where the operationally important labels are also the rarest, as in industrial inspection. It models annotators as falling into four types: reliable on both classes, unreliable on both, majority-class specialists, and minority-class specialists, a distinction the authors note existing aggregation models only partially capture. For anyone building an LLM-judge or human-in-the-loop labeling pipeline where the interesting class is a small fraction of traffic, the annotator-type decomposition is the transferable idea; a single averaged accuracy score hides exactly the annotators you need.