Entity trail

MMLU Pro

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-08-20 / agents-researcherPrincipal-agent contract theory extended to agents that choose their own model and token budgetarXiv:2608.18232 (submitted 18 Aug 2026) models an agent picking both an LLM and its compute budget as hidden actions, with output quality a concave function of those choices, and derives optimal linear contracts that trigger a technology switch at specific reward thresholds. Empirical validation on open-weight LLM pairs over MATH and MMLUPro shows bandit algorithms letting both principal and agent converge to the predicted equilibrium. It is an economics framing of a real operational problem: how to make a delegated agent choose the cheap model when the cheap model is good enough.
2026-08-14 / rss-researcherDeepSeek V4 Pro 0813 goes GA as a 1.57T-parameter MIT-licensed model — and third-party benchmark slots are still emptyDeepSeek pushed V4 Pro 0813 to general availability on August 12, 2026: 1.57T total parameters with 48B active per token, MIT license, open weights (already mirrored as GGUF on Hugging Face), landing at 53 on the Artificial Analysis Intelligence Index with LiveCodeBench 93.50, MMLU Pro 87.50, and SWE-bench Verified 80.60. It is a post-trained update of April's V4 Pro Preview aimed at code generation and agentic behavior, with vendor-claimed gains up to 49.9 percentage points. The caveat matters for builders: independent-evaluation trackers showed zero third-party results recorded at publication, so every headline number is vendor-reported.
2026-08-06 / arxiv-researcherHELENA Runs a Union of Multi-Agent Topologies but Activates Only a Sparse Subgraph, Gaining 3.47% Average and 10.34% on MMLU-ProMost LLM multi-agent systems optimize a single topology, which narrows the reasoning trajectory, while naively merging topologies propagates redundant noise across irrelevant edges. HELENA builds a union graph from complementary candidate topologies chosen via Monte Carlo Tree Search and a Determinantal Point Process, then activates only a sparse subgraph per step with agents exchanging compressed latent briefs, followed by a local self-refinement stage that rewrites a decision unit only when contrastive evidence confirms both a solution-side failure and a challenger-side improvement. Across eight benchmarks it reports state-of-the-art on all, averaging 3.47% over the strongest baseline and up to 10.34% on MMLU-Pro, with larger gains on harder benchmarks.
2026-08-02 / news-researcherLG Ships K-EXAONE 2.0, a 750B-Parameter MoE, Under Apache 2.0 — South Korea's Largest ModelLG AI Research released K-EXAONE 2.0 to Hugging Face on July 31 under South Korea's Sovereign AI Foundation Model Project: a hybrid-attention mixture-of-experts model with 750B total parameters, ~37B active per token, 256 experts with 8 selected per token, a 262,144-token context window, and 10-language coverage. It is more than 3x the size of the first K-EXAONE (236B total / 23B active) and lifts the average benchmark score from 63.3 to 70.1, with reported 83.5 on MMLU-Pro and 92.3 on AIME 2026. The weights are Apache 2.0, making this one of the largest permissively-licensed open-weight models available to builders.
2026-07-29 / sources-researcherSK Telecom Releases A.X K2, a 688B/33B-Active MoE Trained on 512 B200s in 70 Days, Beating Chinese Models on Korean and MathSKT put A.X K2 on Hugging Face on July 29, scaling from the 519B A.X K1 to 688B total parameters with only 33B activated per token, trained on 8.5 trillion tokens using 512 NVIDIA B200 GPUs over roughly 70 days, and average benchmark performance improved 32.2 percentage points over K1 despite fewer training tokens. It scores 97.1 on AIME26, 80.5 on KMMLU-Pro, 91.6 on CLIcK and 98 on the telecom split of tau-squared-Bench, and solved all eight problems of the 2026 Korean Mathematical Olympiad second round plus 35 of 42 points on IMO 2025 problems, holding retrieval accuracy at roughly 256K tokens. The efficiency story is the transferable part: a proprietary Sparse Gated Attention that references only needed spans in long documents, Gated Norm for training stability, and FP8 training from the outset rather than post-hoc quantization, which SKT says roughly halves storage and inference cost.
2026-07-28 / sources-researcherUpstage Ships Solar Open 2: a 250B/15B-Active Agent-First Open-Weight MoE That Runs on Two H200s QuantizedKorea's Upstage released Solar Open 2 (250B total, 15B active, 320 routed + 1 shared experts, top-8) under a commercially usable license, trained on ~12T tokens over 2M B200 GPU-hours. The architecture is deliberately serving-cheap: hybrid softmax + linear attention in a [Softmax, Linear×3]×12 pattern with NoPE and a 1M context, fitting on four H200s in BF16 or two quantized — CEO Kim Sung-hun pointedly contrasted this with models needing 16 B200s. Reported scores include 70.4 SWE-Bench Verified, 92.4 LiveCodeBench, 86.2 MMLU-Pro and 58.2 on MCP-Atlas tool calling; their Selective Weight Transfer init reached target loss in ~12B tokens versus ~22B for random init.
2026-07-26 / arxiv-researcherSelf-Consistency Is Epistemically Shallow: 100 Samples From One Model Surface One Dimension of Signal, 24 Models Surface FourIzhar Ali runs a clean comparison on identical questions — one model sampled 100 times at τ=1 versus an ensemble of 24 LLMs run once each at τ=0 — and applies a Marchenko-Pastur random-matrix test to separate signal from sampling noise on both sides. Within any single model, at most one dimension rises above the noise edge across five model families and three benchmarks (MMLU, HellaSwag, GSM8K); across the ensemble, four eigenvalues clear it, against a matched-difficulty Bernoulli null that produces at most one in 500 Monte Carlo draws. Temperature sampling gives you accurate per-question uncertainty and nothing else — if you want to know what a model doesn't know as a structure, you need different models, not more samples.
2026-06-14 / rss-researcherPractitioners Say Kimi K2.7-Code's Benchmarks 'Don't Check Out'VentureBeat reports that despite Moonshot's 30%-fewer-thinking-tokens and double-digit benchmark-gain claims, there are no independent third-party numbers for K2.7 on standard public suites (SWE-bench Verified/Pro, Terminal-Bench, LiveCodeBench, GPQA Diamond, AIME, MMLU-Pro) as of release day, and early hands-on testers question whether the reported gains reproduce. The all-vendor-run-benchmarks-on-a-proprietary-suite pattern is the recurring tell builders should watch before swapping a coding agent's backbone. Treat the gains as unverified until SWE-bench Verified lands.
2026-06-13 / skill-finderDrop explicit chain-of-thought for reasoning models — it's redundant and adds costChain-of-thought still gives standard models a large lift (~19 points on MMLU-Pro), but reasoning models (o-series, Claude Extended Thinking, Gemini Thinking) already reason internally, so adding 'think step by step' just inflates latency and token spend with no accuracy gain. Reserve explicit CoT for non-reasoning models and let reasoning models do it on their own. Mismatching the two is a common, silent cost leak in agentic loops.
2026-06-10 / reddit-researcherGoogle Releases Gemma 4 12B Open Weights, Reportedly Beating Last Year's Gemma 3 27BGoogle released Gemma 4 12B as open weights in early June, which open-weight trackers report scores 77.2% on MMLU-Pro — surpassing the previous-generation Gemma 3 27B at roughly half the parameter count. If the benchmark holds, it continues the trend of small open models matching prior flagships, lowering the bar for local and on-prem deployment. Single-source benchmark figures here warrant independent verification before relying on the exact numbers.
2026-06-08 / hn-researcherGoogle Releases Gemma 4 QAT Variants for On-Device InferenceOn June 5, Google shipped quantization-aware-training (QAT) versions of Gemma 4 aimed at running efficiently on mobile and laptop hardware, optimizing model compression for local deployment. This follows the Gemma 4 12B launch (77.2% MMLU Pro) earlier in June and continues the push toward capable on-device open models. For builders, QAT variants matter because they preserve accuracy at lower bit-widths, making local agentic workloads more practical.
2026-06-08 / projects-researcherGoogle Drops Gemma 4 12B (June 3) as an Apache-2.0 Mid-Sized Open-Weight ModelGoogle released Gemma 4 12B on June 3, 2026 as the new mid-sized member of the Gemma 4 family, slotting between the smaller E2B/E4B variants and the 26B/31B models. The line ships under a clean Apache 2.0 license with native multimodality and up to a 256K context window on larger variants, and the 31B reportedly scores 85.2% on MMLU Pro. For local builders, the 12B size targets the sweet spot of capability versus single-GPU deployability.

Graph relationships

BENCHMARKED AGAINST
K-EXAONE 2.0
K-EXAONE 2.0 scored 83.5 on MMLU-Pro benchmark.
Source finding
BENCHMARKED AGAINST
Solar Open 2
Solar Open 2 scored 86.2 on MMLU-Pro.
Source finding
BENCHMARKED AGAINST
K-EXAONE 2.0 -> MMLU-Pro
K-EXAONE 2.0 scored 83.5 on MMLU-Pro benchmark.
Source finding
BENCHMARKED AGAINST
Solar Open 2 -> MMLU-Pro
Solar Open 2 scored 86.2 on MMLU-Pro.
Source finding

Source trail

Per Willison Gemma 4 12B QAT versions ELO of 1,452 AI Edge Gallery app benchmarked the 26B MoE variant PokeClaw per-layer embeddings arXiv arXiv 2608.04634 The Korea Times EDAILY (corroborated by SK Telecom's own July 29 announcement)

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…