Cloudflare Says Half of Web Traffic Is Already Bots

Half of all HTML requests hitting the web are already non-human, Cloudflare's Stephanie Cohen told SaaStr AI 2026.

That share is tracking toward two-thirds by year-end. For any SaaS company whose top-of-funnel is content, the marketing site is turning into a machine-readable API with a failing referral business attached.

Cloudflare's own crawl data shows why. OpenAI's crawler pulls roughly 1,700 pages for every referral it sends back to a publisher. Anthropic's ratio runs to 73,000 pages per referral. That ratio makes the current business model unworkable.

Cloudflare shut down pay-per-crawl on July 1 and switched to paying publishers when their content shows up in an AI answer. Starting September 15, it will block mixed-use AI crawlers by default on any page carrying ads.

Cohen's advice to vendors: serve agents lightweight formats instead of full HTML. Markdown runs about 80% lighter. Add cryptographic verification for bot-initiated transactions, since an agent might complete a purchase or signup before you know it's authorized.

Most SaaS teams haven't priced this in. If two-thirds of your HTML requests are non-human, your marketing site isn't a funnel anymore. It's an API with a design skin on top. The referral traffic that funded the content budget is drying up at a 73,000-to-1 clip.

Watch whether other CDNs follow Cloudflare away from pay-per-crawl toward pay-per-citation. If Cohen's numbers hold, that's the only version of this deal publishers still get paid under.