Article URL: https://tilion.dev/blog/cloudflare-blocks-agents Comments URL: https://news.ycombinator.com/item?id=48867124 Points: 6 # Comments: 0

As of July 1, 2025, Cloudflare blocks AI crawlers by default on every new domain, and it has since turned away hundreds of billions of bot requests. Most of the coverage was about the big fight, AI labs against publishers over training data. That wasn’t my problem. My problem was an agent that kept coming back confidently wrong. It would read a source, check a price, fetch a page I asked for, and give me an answer that sounded right and wasn’t. It was reading the block page instead of the real page: Cloudflare’s “Just a moment” screen, summarized back in the same confident voice it uses for everything else. Nothing in the response said the fetch had failed. So I measured how often it happens, and we built something to fix it. The reason this is easy to miss is that nothing in the system thinks anything went wrong. When you make a plain fetch(), httpx.get(), or requests.get()to a bot-protected site, you usually don’t get an exception. You get a 403, sometimes even a 200, with a full HTML body attached: Cloudflare’s “Just a moment” screen, DataDome’s “please enable JavaScript,” PerimeterX’s “Press & Hold,” Amazon’s “Continue shopping” wall. To your retry logic, that looks like success; the request finished. To a language model, the body is just text to read, so it reads it. Hand that 27KB block page to a model and ask for the jobs on it, and it will list them for you, none of which exist. The wrong answer is impossible to tell apart from the right one. A loud failure, a timeout or a clean 429 with an empty body, would trigger a retry or a skip and you’d move on. The quiet block page slips through because it looks like content. On July 10 I sent an ordinary browser request to a set of popular targets to see which ones Cloudflare turns away. Twelve of the ones people ask about most came back with a hard 403: