Tag: Cloudflare

Source URL: https://blog.cloudflare.com/egress-policies-by-hostname/ Source: The Cloudflare Blog Title: Introducing simple and secure egress policies by hostname in Cloudflare’s SASE platform Feedly Summary: Cloudflare’s SASE platform now offers egress policies by hostname, domain, content category, and application in open beta. AI Summary and Description: Yes **Summary:** Cloudflare’s Secure Access Service Edge (SASE) platform introduces egress policies,…

Simon Willison’s Weblog: TIL: Rate limiting by IP using Cloudflare’s rate limiting rules

Jul 3, 2025

—

by

Source URL: https://simonwillison.net/2025/Jul/3/rate-limiting-by-ip/#atom-everything Source: Simon Willison’s Weblog Title: TIL: Rate limiting by IP using Cloudflare’s rate limiting rules Feedly Summary: TIL: Rate limiting by IP using Cloudflare’s rate limiting rules My blog started timing out on some requests a few days ago, and it turned out there were misbehaving crawlers that were spidering my /search/…

The Register: Cloudflare creates AI crawler tollbooth to pay publishers

—

by

Source URL: https://www.theregister.com/2025/07/01/cloudflare_creates_ai_crawler_toll/ Source: The Register Title: Cloudflare creates AI crawler tollbooth to pay publishers Feedly Summary: The bargain between content makers and crawlers has broken down ai-pocalypse Cloudflare has started blocking AI web crawlers by default in a bid to become the internet’s gatekeeper.… AI Summary and Description: Yes Summary: The text highlights a…

Slashdot: Cloudflare Flips AI Scraping Model With Pay-Per-Crawl System For Publishers

—

by

Source URL: https://tech.slashdot.org/story/25/07/01/1745245/cloudflare-flips-ai-scraping-model-with-pay-per-crawl-system-for-publishers?utm_source=rss1.0mainlinkanon&utm_medium=feed Source: Slashdot Title: Cloudflare Flips AI Scraping Model With Pay-Per-Crawl System For Publishers Feedly Summary: AI Summary and Description: Yes Summary: Cloudflare’s new “Pay Per Crawl” program introduces a monetization option for website owners, allowing them to charge AI companies for content access used for model training. This initiative is significant as…

The Cloudflare Blog: Message Signatures are now part of our Verified Bots Program, simplifying bot authentication

—

by

Source URL: https://blog.cloudflare.com/verified-bots-with-cryptography/ Source: The Cloudflare Blog Title: Message Signatures are now part of our Verified Bots Program, simplifying bot authentication Feedly Summary: Bots can start authenticating to Cloudflare using public key cryptography, preventing them from being spoofed and allowing origins to have confidence in their identity. AI Summary and Description: Yes **Summary:** The text…

The Cloudflare Blog: From Googlebot to GPTBot: who’s crawling your site in 2025

—

by

Source URL: https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/ Source: The Cloudflare Blog Title: From Googlebot to GPTBot: who’s crawling your site in 2025 Feedly Summary: From May 2024 to May 2025, crawler traffic rose 18%, with GPTBot growing 305% and Googlebot 96%. AI Summary and Description: Yes Summary: The text discusses the evolution of web crawlers, particularly focusing on the…

The Cloudflare Blog: Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized content

—

by

Source URL: https://blog.cloudflare.com/control-content-use-for-ai-training/ Source: The Cloudflare Blog Title: Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized content Feedly Summary: Cloudflare is making it easier for publishers and content creators of all sizes to prevent their content from being scraped for AI training by managing robots.txt on their behalf. AI…

The Cloudflare Blog: Introducing pay per crawl: enabling content owners to charge AI crawlers for access

—

by

Source URL: https://blog.cloudflare.com/introducing-pay-per-crawl/ Source: The Cloudflare Blog Title: Introducing pay per crawl: enabling content owners to charge AI crawlers for access Feedly Summary: Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content. AI Summary and Description: Yes **Summary:** This text discusses Cloudflare’s innovative “pay…

The Cloudflare Blog: The crawl before the fall… of referrals: understanding AI’s impact on content providers

—

by