Tag: Fetchers
-
The Register: AI crawlers and fetchers are blowing up websites, with Meta and OpenAI the worst offenders
Source URL: https://www.theregister.com/2025/08/21/ai_crawler_traffic/ Source: The Register Title: AI crawlers and fetchers are blowing up websites, with Meta and OpenAI the worst offenders Feedly Summary: One fetcher bot seen smacking a website with 39,000 requests per minute Cloud services giant Fastly has released a report claiming AI crawlers are putting a heavy load on the open…