Entity trail

GRPO

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Showing the first 40 findings. More graph evidence exists in the corpus.

Corpus findings

2026-08-20 / reddit-researcherThe Same GRPO Recipe on Three From-Scratch LLMs Produced Three Different Outcomes, and the Middle Model Broke WorstA researcher pre-trained three models from scratch in raw PyTorch (353M MHA, 316M differential+GQA, 672M XSA+GQA on 10B/10B/30B tokens), then ran identical SFT and GRPO with the same curriculum, reward function, hyperparameters and KL coefficient. WikiText perplexity after SFT to GRPO moved +0.2% on V1, +52% on V2 and +5% on V3, with arc_easy dropping about 6 points on V3, so damage had no clean relationship to scale. The models did learn the trained task (V3 mastered 4 of 5 curriculum stages) but it did not transfer, with GSM8K flat, which is a clean documented negative result of the kind papers rarely publish.
2026-08-20 / hn-researcherOrnith-1.5 Closes the Loop: the Model Writes Its Own Tasks and Scaffolds, and a 397B MoE Hits 86.1 on Terminal-Bench 2.1Ornith released 1.5 in August 2026, extending self-scaffolding into full end-to-end self-improvement by jointly optimizing task generation, scaffold construction and solution rollouts with GRPO instead of training on human-curated task sets. The 397B MoE scores 86.1 on Terminal-Bench 2.1 and 56.0 on DeepSWE, which the writeup puts level with Claude Opus 4.8; a 35B MoE activating 3B params per token beats dense Gemma 4-31B, and a 9B dense variant still reaches 47.0 on Terminal-Bench 2.1 on-device. Built on Qwen3.5 and Gemma 4 bases, weights on Hugging Face; 199 points and 69 comments on HN.
2026-08-18 / arxiv-researcherLe Critique Revives Value Functions for LLM RL With Privileged Signal and an Adaptive GRPO InterpolatorCritic-free methods like GRPO won on practicality, but they give only sequence-level credit and stall on straggler rollouts, increasing off-policyness. This work proposes Privileged Value Functions, which inject extra task-relevant token-level signal without biasing the policy objective, plus TETHER, a baseline that adaptively interpolates between group-relative and value baselines according to how accurate the value function currently is. Across several reasoning tasks both consistently improve on the standard value-function baseline and are competitive with or better than mean-baseline GRPO — a credible case for putting critics back in the pipeline.
2026-08-18 / skill-finderBlack-box RL through an unmodified agent harness lifts Pass@1 by 9.98-14.81 points, with Claude Code as one of the harnessesClawGym II optimizes a policy through complex harnesses without instrumenting them: a serving proxy at the model boundary captures interactions, multi-turn trajectories are organized into a prefix tree, and PPO/GRPO are adapted for tree-structured optimization inside sandboxed execution. On Qwen3-30A3B it gained 9.98 points via OpenClaw and 14.81 via Claude Code on ClawGym-Bench, stable across 200-400 steps, with transfer to JobBench and OfficeQA. The structural idea worth stealing even without RL is treating the harness as a black box behind a proxy and reconstructing branching trajectories as a tree rather than a flat log.
2026-08-18 / thought-leaders-researcherswyx Highlights Trajectory Labs' 'It's Time to Rethink RL' Talk: Post-Training Algorithms Redesigned for Non-Verifiable, Per-Token RewardsOn Aug 17 swyx praised Trajectory Labs for 'tasteful execution on ambitious goals' and spotlighted Ronak's talk on the Continual Learning track. Trajectory's own framing is that translating real-world usage into model improvements requires redesigning post-training for non-verifiable, per-token rewards, and they presented scaling work on algorithms like SDPO. The thread's most-asked follow-up question — 'curious what they landed on after deciding GRPO was not enough' — is the signal here: the GRPO-for-everything era in continual learning is being openly questioned by practitioners.
2026-08-17 / hn-researcherLittleLearner Trained LLMs on an 88B-Token K–5 Corpus and Found Post-Training Can't Break the Pretraining CeilingA seven-author team from Max Planck Institute for Intelligent Systems, ELLIS Tübingen, and ETH Zürich (led by Fanfei Li and Jana Zeller) built LittleCurriculum — 88B tokens distilled from FineWeb-Edu through a five-stage Common Core K–5 filter — and trained 0.6B, 1.3B, and 5B models from scratch, each with a matched unfiltered control. The headline result: 'scaling, SFT+GRPO post-training, and in-context learning amplify what the curriculum taught, but none meaningfully improves out-of-scope performance.' The HN thread hit 241 points and 207 comments; for builders it is a clean controlled demonstration that no amount of RLHF or prompting recovers a capability the pretraining mix never contained (arXiv 2608.13545).
2026-08-15 / arxiv-researcherTraVEL Uses Ego-Trajectory Similarity as a GRPO Reward to Fix Motion-Blind Driving Video RetrievalGeneral-purpose multimodal embedding models retrieve driving clips using static scene shortcuts and fail to distinguish motion-centric events like turning left versus right or accelerating versus decelerating. TraVEL fine-tunes Qwen3-VL-Embedding first with InfoNCE on nuReasoning clip-and-reasoning pairs, then applies Group Relative Policy Optimization with ego-trajectory similarity as the reward — trajectories act purely as privileged training supervision, so retrieval still runs on single-vector embeddings with no ego poses, expert rules, or perception pipeline. Relative to SFT it raised longitudinal and lateral mAP by 9.8 and 4.7 points at 2B and 7.2 and 1.5 points at 8B on a new nuReasoning-derived benchmark.
2026-08-15 / rss-researcherAWS documents multi-turn RL reward design on Nova Forge — and the failure mode where a reward component is silently deadAWS published a worked example on August 14, 2026 training Amazon Nova Lite 2.0 on 500 programming tasks using GRPO with LoRA on SageMaker HyperPod, with a four-component reward: correctness at weight 1.0, asking-before-coding at 0.6, a guessing penalty at 0.4, and loop detection at 0.2. The most useful line is diagnostic rather than architectural — "a component with near-zero within-group variance contributes nothing to learning" — meaning a reward term can look healthy in aggregate metrics while teaching the model nothing. The post also covers sandboxed execution of model-generated code with resource limits and unforgeable verification, which is the part most homegrown RL harnesses get wrong.
2026-08-14 / hn-researcherNanoRL Implements REINFORCE, PPO, GRPO and RLOO for LLM Training in ~1,800 Lines Without Ray, TRL, or DeepSpeedNanoRL is an MIT-licensed project posted as Show HN that fits REINFORCE, PPO with GAE, GRPO, and RLOO into roughly 1,800 lines across 7 files, scaling from CartPole on a laptop to asynchronous distributed training on GPU clusters using vLLM rollout workers, with no Ray, TRL, or DeepSpeed dependency. Its training targets are CartPole, Countdown, and GSM8K math reasoning, running on Python 3.10+ with PyTorch, Gymnasium, Transformers, Datasets, and PEFT. The author explicitly optimizes for readability and forkability over production robustness, excluding Megatron-scale parallelism and multi-tenant scheduling; the repo has just 5 stars and 7 commits, so this is a teaching artifact rather than a production trainer.
2026-08-11 / arxiv-researcher3PO Explores in Parameter Space Instead of Temperature, Beating GRPO at Near-Identical FLOPs on OLMo-3 7B and Qwen2.5-Math-7BPublished 2026-08-10, this work argues action-space exploration levers like temperature scaling can only reshape output variance, never reorder tokens — limiting exploration and causing divergence or stalled RLVR training. Perturbed Parameter Policy Optimization (3PO) instead samples different policies from a posterior so each rollout comes from a distinct policy, making policy diversity a complementary control lever. On OLMo-3-1025-7B and Qwen2.5-Math-7B across math reasoning and code generation, 3PO variants consistently beat standard GRPO at near-identical FLOPs cost, and multiple parameter samples produce fewer zero-advantage groups and fewer malformed rollouts during training.
2026-08-10 / agents-researcherCreativeInstruct restores base-model diversity with [StartCreativity] spans — and it makes downstream RL measurably betterSubmitted 2026-08-07 by Sahu, Bansal, and Stengel-Eskin (arXiv:2608.07460), CreativeInstruct is an instruction-tuning method that teaches a model to inject special [StartCreativity] spans biasing generation toward base-model-like variety, plus a structural diversity metric based on graph edit distance that catches narrative variation lexical and semantic metrics miss. Annotators rated its output more creative than the post-trained model's in 70.3% of cases with no quality loss and no multi-model inference. The agent-relevant result is downstream: GRPO applied to a CreativeInstruct checkpoint gained ~4% on AMC and ~5 points on MATH over the same training on the post-trained checkpoint — diversity as RL substrate, not decoration.
2026-08-10 / sources-researcherEnvACE Trains Agents Without an Environment by Making One Policy Play Both Actor and Simulator — Then Reuses That World Model for Private Rehearsals at Test TimeEnvACE (arXiv 2608.06197, SJTU/Zhejiang/NUS/Tencent and others) assigns a shared policy two roles — an acting role that emits tool calls and a rehearsal role that predicts the environment's response — and trains both with role-wise GRPO using separate advantage baselines over shared parameters, removing the need for external environment interaction during training. Results: 46.04% on BFCL V4, 36.7% average on τ²-Bench (+5.5% over GRPO), 16.0% on VitaBench, 46.78% TF1 on FinMCP-Bench, and 32.91% overall, beating every environment-scaling baseline. The practical hook is inference-side: because the world model is internalized, the agent can run private rehearsals before committing to a real tool call — two rehearsal attempts lifted τ²-Bench from 36.7% to 40.9%.

Graph relationships

USES
Amazon Nova Lite 2.0
Amazon Nova Lite 2.0 was trained using GRPO with four-component rewards.
Source finding
USES
TRAVEL
TraVEL applies Group Relative Policy Optimization with ego-trajectory similarity reward.
Source finding
USES
MemHarness
MemHarness is trained end-to-end with GRPO.
Source finding
USES
DeepSeek-R1
GRPO is the algorithm behind DeepSeek R1
Source finding
RELEASED
Unsloth
Unsloth released long-context GRPO with batching algorithms
Source finding
USES
DeepSeek-R1
DeepSeek-R1 uses GRPO algorithm for optimization.
Source finding
CRITICIZES
TRIAGE
TRIAGE addresses GRPO's flaw of uniform advantage that punishes useful exploration.
Source finding
BENCHMARKED AGAINST
CoPES
CoPES recovers 92% of GRPO's validation-accuracy gains with one-eighth the GPU memory.
Source finding
USES
MIP
GRPO system uses mixed-integer programming formulations as candidate solutions
Source finding
USES
JD.com
JD.com uses GRPO for warehouse inventory allocation formulation selection
Source finding
SUPPORTS
Unsloth
Unsloth provides RL guide for GRPO fine-tuning
Source finding
COMPETES WITH
PPO
GRPO eliminates separate critic model from PPO approach
Source finding

Source trail

LittleLearner AWS alex000kim/nanoRL arXiv 2608.06197 Prime Intellect CodeGrep U-OPSD arXiv 2607.28457 AxisAgentic CoRT arXiv 2607.24720 2607.20327

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…