AI crawlers – Experimental News Clipping Site

Simon Willison’s Weblog: Cloudflare Radar: AI Insights

Sep 1, 2025

—

by

Source URL: https://simonwillison.net/2025/Sep/1/cloudflare-radar-ai-insights/ Source: Simon Willison’s Weblog Title: Cloudflare Radar: AI Insights Feedly Summary: Cloudflare Radar: AI Insights Cloudflare launched this dashboard back in February, incorporating traffic analysis from Cloudflare’s network along with insights from their popular 1.1.1.1 DNS service. I found this chart particularly interesting, showing which documented AI crawlers are most active collecting…

Slashdot: Are AI Web Crawlers ‘Destroying Websites’ In Their Hunt for Training Data?

Aug 31, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://tech.slashdot.org/story/25/08/31/1820249/are-ai-web-crawlers-destroying-websites-in-their-hunt-for-training-data Source: Slashdot Title: Are AI Web Crawlers ‘Destroying Websites’ In Their Hunt for Training Data? Feedly Summary: AI Summary and Description: Yes Summary: The text discusses the adverse effects of AI web crawlers on website performance, highlighting the increasing web traffic attributed to these bots. It addresses the challenges website owners face…

The Cloudflare Blog: The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals

Aug 29, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://blog.cloudflare.com/crawlers-click-ai-bots-training/ Source: The Cloudflare Blog Title: The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals Feedly Summary: By mid-2025, training drives nearly 80% of AI crawling, while referrals to publishers (especially from Google) are falling and crawl-to-refer ratios show AI consumes far more than it sends back. AI Summary and Description:…

The Cloudflare Blog: The next step for content creators in working with AI bots: Introducing AI Crawl Control

Aug 28, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://blog.cloudflare.com/introducing-ai-crawl-control/ Source: The Cloudflare Blog Title: The next step for content creators in working with AI bots: Introducing AI Crawl Control Feedly Summary: Cloudflare launches AI Crawl Control (formerly AI Audit) and introduces easily customizable 402 HTTP responses. AI Summary and Description: Yes Summary: The text discusses Cloudflare’s launch of AI Crawl Control,…

The Cloudflare Blog: A deeper look at AI crawlers: breaking down traffic by purpose and industry

Aug 28, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://blog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry/ Source: The Cloudflare Blog Title: A deeper look at AI crawlers: breaking down traffic by purpose and industry Feedly Summary: We are extending AI-related insights on Cloudflare Radar with new industry-focused data and a breakdown of bot traffic by purpose, such as training or user action. AI Summary and Description: Yes Summary:…

The Register: AI crawlers and fetchers are blowing up websites, with Meta and OpenAI the worst offenders

Aug 21, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://www.theregister.com/2025/08/21/ai_crawler_traffic/ Source: The Register Title: AI crawlers and fetchers are blowing up websites, with Meta and OpenAI the worst offenders Feedly Summary: One fetcher bot seen smacking a website with 39,000 requests per minute Cloud services giant Fastly has released a report claiming AI crawlers are putting a heavy load on the open…

The Cloudflare Blog: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives

Aug 4, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/ Source: The Cloudflare Blog Title: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives Feedly Summary: Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites. AI Summary and Description: Yes Summary: The…

Slashdot: Cloudflare Flips AI Scraping Model With Pay-Per-Crawl System For Publishers

Jul 1, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://tech.slashdot.org/story/25/07/01/1745245/cloudflare-flips-ai-scraping-model-with-pay-per-crawl-system-for-publishers?utm_source=rss1.0mainlinkanon&utm_medium=feed Source: Slashdot Title: Cloudflare Flips AI Scraping Model With Pay-Per-Crawl System For Publishers Feedly Summary: AI Summary and Description: Yes Summary: Cloudflare’s new “Pay Per Crawl” program introduces a monetization option for website owners, allowing them to charge AI companies for content access used for model training. This initiative is significant as…

The Cloudflare Blog: From Googlebot to GPTBot: who’s crawling your site in 2025

Jul 1, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/ Source: The Cloudflare Blog Title: From Googlebot to GPTBot: who’s crawling your site in 2025 Feedly Summary: From May 2024 to May 2025, crawler traffic rose 18%, with GPTBot growing 305% and Googlebot 96%. AI Summary and Description: Yes Summary: The text discusses the evolution of web crawlers, particularly focusing on the…

The Cloudflare Blog: Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized content

Jul 1, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://blog.cloudflare.com/control-content-use-for-ai-training/ Source: The Cloudflare Blog Title: Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized content Feedly Summary: Cloudflare is making it easier for publishers and content creators of all sizes to prevent their content from being scraped for AI training by managing robots.txt on their behalf. AI…

Tag: AI crawlers