Demystifying Data Organization for Enhanced LLM Training: Four Guidelines and Two Novel Ordering Methods

arXiv·medium signal

Identifies four key principles for optimizing training data order — Boundary Sharpening, Cyclic Scheduling, Curriculum Continuity, and Local Diversity — and introduces STR and SAW ordering methods. Reuses pre-computed sample-level scores for minimal overhead. Validated across multiple model scales for both pre-training and SFT, showing consistent improvements in training stability and final performance.