Sources
kernel.org spends 20% of its fleet rendering git commits for scrapers, and legitimate traffic is 2%
Konstantin Ryabitsev published hard numbers on 2026-08-29: git.kernel.org takes about 6M daily requests for random commits, and of the 90 cores across five geo-distributed nodes, 14 to 16 are permanently occupied rendering commits as HTML for crawlers. Under generous assumptions, real developer traffic is roughly 2% of the total. The escalation ladder is instructive for anyone running a public site: user-agent bans failed, then IP bans, then ASN bans, and now crawlers arrive from millions of residential and mobile IPs via proxy SDK monetization, making 4-5 requests each. Anubis proof-of-work worked for months at difficulty 4, then difficulty 5, and today 33% of requests solve the challenge and get through anyway.
↳ Follow the thread