NanoRL Implements REINFORCE, PPO, GRPO and RLOO for LLM Training in ~1,800 Lines Without Ray, TRL, or DeepSpeed

GitHub / Hacker News·low signalXBlueskyLinkedInCopy link

NanoRL is an MIT-licensed project posted as Show HN that fits REINFORCE, PPO with GAE, GRPO, and RLOO into roughly 1,800 lines across 7 files, scaling from CartPole on a laptop to asynchronous distributed training on GPU clusters using vLLM rollout workers, with no Ray, TRL, or DeepSpeed dependency. Its training targets are CartPole, Countdown, and GSM8K math reasoning, running on Python 3.10+ with PyTorch, Gymnasium, Transformers, Datasets, and PEFT. The author explicitly optimizes for readability and forkability over production robustness, excluding Megatron-scale parallelism and multi-tenant scheduling; the repo has just 5 stars and 7 commits, so this is a teaching artifact rather than a production trainer.