Source URL: https://www.theregister.com/2025/08/21/ai_crawler_traffic/
Source: The Register
Title: AI crawlers and fetchers are blowing up websites, with Meta and OpenAI the worst offenders
Feedly Summary: One fetcher bot seen smacking a website with 39,000 requests per minute
Cloud services giant Fastly has released a report claiming AI crawlers are putting a heavy load on the open web, slurping up sites at a rate that accounts for 80 percent of all AI bot traffic, with the remaining 20 percent used by AI fetchers. Bots and fetchers can hit websites hard, demanding data from a single site in thousands of requests per minute.…
AI Summary and Description: Yes
Summary: The text highlights a concerning trend regarding AI crawlers and their intensive data requests, particularly focusing on the impact of bots on web infrastructure. This raises significant implications for infrastructure security and management, as the prevalence of AI-related bot traffic increases.
Detailed Description: The report from Fastly discusses the detrimental effects of AI crawlers on web services, underlining the need for heightened infrastructure security measures. Key insights include:
– **AI Crawler Traffic**: The report indicates that a staggering 80% of bot traffic on the web is attributed to AI crawlers, with fetchers constituting the remaining 20%.
– **Impact on Websites**: Websites are experiencing heavy loads from these bots, exemplified by one instance of a bot issuing 39,000 requests per minute. This level of activity can cause performance degradation and potential service outages.
– **Infrastructure Implications**: The high volume of requests necessitates robust infrastructure security strategies to mitigate risks, including:
– Implementing rate limiting to control incoming requests.
– Using bot detection tools to distinguish between legitimate and malicious actions.
– Employing load balancers and traffic management techniques to ensure website stability.
The growing reliance on AI crawlers necessitates both awareness and proactive measures in the fields of cloud computing and infrastructure security to safeguard against potential threats.