UltraX Refines Pre-Training Data at Scale With Adaptive Programmatic Editing

arXiv·medium signal

With training data approaching its physical limit and scaling-law gains diminishing, UltraX targets higher-quality data utilization over raw expansion. It argues rule-based refinement is trapped by fixed heuristics that miss instance-level variation while pure LLM-based refinement is costly and unreliable, and proposes adaptive programmatic editing to refine corpora at scale. Signals the continued industry pivot from 'more data' to 'better data pipelines' for frontier training.