NanoRL fits REINFORCE, PPO, GRPO and RLOO into ~1,800 lines with no Ray, TRL or DeepSpeed.

alex000kim/nanoRL (MIT) scales from CartPole on a laptop to async distributed training on GPU clusters with vLLM rollout workers, across 7 files. The author optimizes explicitly for readability and forkability, excluding Megatron-scale parallelism and multi-tenant scheduling. 5 stars and 7 commits, so it's a teaching artifact, not a production trainer, which is exactly what makes it worth reading.