Fetching from the wire…
01
02
03
04
05
06
07
08
Research2026-07-26 · source-backed
Tao et al. diagnose long-context degradation as two opposite spectral pathologies: homogenization collapse from attention sinks driving rank deficiency, and isolation collapse (arXiv 2607.20484). TRSP is a non-invasive side-path combining a parameter-free triangular mechanism with a lightweight context-length-aware scaling term to rebalance the spectrum without changing the base architecture. 83% retention at eight times training length, substantially above competing long-context interventions.
Each link below shares sources, entities, or timing with this story.
The architecture report describes a 125B sparse MoE activating 6B parameters per token, plus 51B of n-gram embedding tables held off the accelerator in host memory with prefetching (arXiv 2608.30320). Against the prior 397B-A17B model it leads on 8 of 14 pre-training benchmark...
Translating key-value state from one model into a form another can consume works across scale, architecture, attention configuration, tokenizer and family (arXiv 2608.30963). Llama3.1-70B to Qwen2.5-7B reaches 44.0% accuracy against 45.7% native while dropping latency to 138ms...
A 1.5B distilled model trained with GRPO chooses NoThink, Short, or Long at response start, using a shaped reward that makes each mode pay off at a different length plus hard per-mode token caps. Accuracy held at 0.782 against 0.796 baseline while mean length fell from 4,796 t...
Researchers systematically evaluate whether Mamba-class state space models can replace ViT encoders in large VLMs, finding competitive performance with linear-time processing versus ViT's quadratic attention. Meaningful memory savings on high-resolution or long-context vision...
Most looped-transformer results compare at fixed model size, conflating architecture with extra compute. SMELT matches per-token FLOPs, non-embedding parameters and KV cache against an unlooped baseline, scaling to 54B non-embedding parameters with a separate Chinchilla-style...
Agent Lightning v1.0 (arXiv 2608.17528) inverts the standard agentic RL architecture, and the inversion is the whole point. Normally the training engine owns the environment loop. It drives the agent, collects trajectories, computes rewards. Which means your training setup and...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.