Dwarkesh Patel x Reiner Pope: GPT-5 Trained on 150T Tokens (100x Chinchilla Optimal), Batch Size of 2,400 Critical for Inference Economics

Dwarkesh Podcast·medium signal

Former Google TPU architect and MatX CEO Reiner Pope did a deep-dive blackboard session on how GPT-5, Claude, and Gemini are actually trained and served. Key revelations: GPT-5's pre-training used ~150 trillion tokens (100x Chinchilla optimal); inference costs are up to 1,000x higher without batching, with optimal batch size ~2,400 independent of model size; MoE architectures are physically constrained by 72-GPU rack boundaries; and Google's TPU scale-up domain advantage enables sparser MoE models.