This looks like the new robots.txt arms race. Cloudflare recently said sites blocking GPTBot/ClaudeBot outnumber Googlebot blocks by a lot, and more publishers are turning on AI crawl controls. Curious how much of the “bot traffic” is known crawlers vs disguised scraping.
1 comments