Hacker News
'Compression Is Prediction': GPT-2 Squeezes Dickens to 10% of Original Size vs 24% for an Order-1 Model
Annie Sexton of ngrok published a walkthrough on 2026-08-11 grounding the equivalence of compression and language modeling in arithmetic coding, showing that a no-context model needs 2.59 bits/symbol and an order-1 model 1.16 bits/symbol on the same text, then demonstrating GPT-2 compressing Dickens to roughly 10% of original size against 24% for the basic order-1 baseline. The argument is that LLMs minimize exactly the cross-entropy that sets the compression limit, making the two the same mathematical object. It hit 596 points and 243 comments — the highest-scoring technical explainer on HN this cycle.
↳ Follow the thread