Covenant-72B: Largest Permissionless Decentralized LLM Pre-Training Run Completes via Bittensor — 70+ Global Participants, 146x Communication Reduction

Import AI #449·medium signal

Bittensor Subnet 3 (Templar) completed a 72B parameter pre-training run across 70+ global nodes on March 10, using SparseLoCo optimizer to reduce cross-node communication overhead 146x via sparsification, 2-bit quantization, and error feedback. The model scores 67.1 MMLU vs. LLaMA-2-70B's 65.6, trained on ~1.1T tokens, with Apache 2.0 weights on Hugging Face. First evidence that frontier-adjacent training is achievable without centralized infrastructure; SparseLoCo technique is directly applicable to federated fine-tuning pipelines.