Source URL: https://simonwillison.net/2025/Sep/1/cloudflare-radar-ai-insights/
Source: Simon Willison’s Weblog
Title: Cloudflare Radar: AI Insights
Feedly Summary: Cloudflare Radar: AI Insights
Cloudflare launched this dashboard back in February, incorporating traffic analysis from Cloudflare’s network along with insights from their popular 1.1.1.1 DNS service.
I found this chart particularly interesting, showing which documented AI crawlers are most active collecting training data – lead by GPTBot, ClaudeBot and Meta-ExternalAgent:
Cloudflare’s DNS data also hints at the popularity of different services. ChatGPT holds the first place, which is unsurprising – but second place is a hotly contested race between Claude and Perplexity and #4/#5/#6 is contested by GitHub Copilot, Perplexity, and Codeium/Windsurf.
Google Gemini comes in 7th, though since this is DNS based I imagine this is undercounting instances of Gemini on google.com as opposed to gemini.google.com.
Via Hacker News
Tags: crawling, dns, ai, cloudflare, generative-ai, llms
AI Summary and Description: Yes
Summary: The text presents insights from Cloudflare Radar, specifically highlighting how various AI crawlers are actively gathering training data. It examines the popularity of AI services based on DNS data, providing actionable insights for professionals in AI and cloud computing security.
Detailed Description:
The provided text discusses the launch of Cloudflare’s AI insights dashboard, which aggregates internet traffic and DNS analysis with a focus on the behavior of AI crawlers collecting training data. This data can be crucial for security professionals interested in identifying trends and potential security implications related to AI systems and their infrastructure.
Key Points:
– **Traffic Analysis**: Cloudflare’s network data reveals the activity levels of documented AI crawlers.
– **Top Crawlers**:
– ***GPTBot*** leads the chart, reflecting its significant usage in data collection for training AI models.
– **ClaudeBot** and **Meta-ExternalAgent** follow, showcasing a competitive landscape among AI services in terms of data gathering.
– **Service Popularity**:
– **ChatGPT** ranks as the most visited service, aligning with its widespread adoption.
– The competition for the second place features **Claude**, **Perplexity**, and others like **GitHub Copilot** and **Codeium/Windsurf**, indicating varied preferences in the developer community.
– **Google Gemini’s Position**: Although it ranks 7th, there are considerations about underreporting due to the nature of DNS data collection.
The insights from this analysis serve as valuable intelligence for AI and cloud computing security professionals, illustrating where data is being collected and who the major players are in the landscape of AI development. Understanding these trends can help in shaping security policies and monitoring tools intended to safeguard data and ensure compliance with regulations concerning AI and machine learning.