Sources
Arcee Trinity Large — Open 400B Sparse MoE 13B Active
400B parameters, 13B active per token, 17 trillion tokens. SMEBU for MoE load balancing. Zero loss spikes with Muon optimizer. Largest open-source sparse MoE.
Source
↳ Follow the thread
400B parameters, 13B active per token, 17 trillion tokens. SMEBU for MoE load balancing. Zero loss spikes with Muon optimizer. Largest open-source sparse MoE.
Source
↳ Follow the thread