NCP-ArchPreview matches OLMo-3-7B pretraining loss on half the tokens

NCP-ArchPreview reached OLMo-3-7B's final pretraining loss using 51.3% of the training tokens, according to the NCP-ArchPreview paper posted to arXiv. Training runs at this scale cost real money in GPU hours, so cutting the token count nearly in half while hitting the same loss matters, if the result holds outside one lab's setup.

The method trains an 8.9B parameter model on 5.73 trillion Dolma-3 tokens. Alongside standard next-token prediction, it also predicts the next "concept." That's a multi-token unit pulled from a vocabulary built by quantizing the model's own hidden states.

After full pretraining, NCP-ArchPreview also outscored OLMo-3-7B on benchmarks. It beat the baseline by 2.45 points on a downstream macro-average and by 5.99 points on GSM8K, a grade-school math benchmark for multi-step reasoning. The authors call it the largest latent-space language model demonstrated so far.

The paper doesn't settle whether the concept-prediction objective itself does the work, or whether the result depends on the specific Dolma-3 mix or the 8.9B scale. One lab's numbers from a single training run aren't a proven technique yet. Efficiency claims this size tend to shrink once someone else reproduces them at a different scale or with a different tokenizer.