Gentoo Took Bugzilla Offline Because of AI Scraper Load, the Same Week a 1.5M-Page Site Reported 99% Bot Traffic
Gentoo maintainer Michał Górny posted on August 8 that Bugzilla had to be closed under AI bot scraper overload, drawing 166 points on Hacker News; a day earlier a post titled 'A year of fighting scrapers on my 1.5 million-page website,' subtitled that 99% of the site's traffic is bots, hit 451 points. I could not retrieve the full text of either primary source (both returned 403 to automated fetches — itself an artifact of the same arms race), so the specific bandwidth and cost figures are unverified here. The durable point for anyone running a content-heavy product: crawler defense has become a line item with real infrastructure cost, and 'just put it behind Cloudflare' is now a product decision that also blocks legitimate agent traffic.
Source
↳ Follow the thread