robots.txt – Experimental News Clipping Site

Slashdot: Cloudflare Launches Content Signals Policy To Fight AI Crawlers and Scrapers

Sep 24, 2025

—

by

Source URL: https://tech.slashdot.org/story/25/09/24/1953230/cloudflare-launches-content-signals-policy-to-fight-ai-crawlers-and-scrapers?utm_source=rss1.0mainlinkanon&utm_medium=feed Source: Slashdot Title: Cloudflare Launches Content Signals Policy To Fight AI Crawlers and Scrapers Feedly Summary: AI Summary and Description: Yes Summary: Cloudflare’s new Content Signals Policy enhances the existing robots.txt functionality, allowing website owners to better control how their content is accessed and utilized by AI companies. This initiative is particularly…

The Cloudflare Blog: To build a better Internet in the age of AI, we need responsible AI bot principles. Here’s our proposal.

Sep 24, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://blog.cloudflare.com/building-a-better-internet-with-responsible-ai-bot-principles/ Source: The Cloudflare Blog Title: To build a better Internet in the age of AI, we need responsible AI bot principles. Here’s our proposal. Feedly Summary: We are proposing—as starting points—responsible AI bot principles that emphasize transparency, accountability, and respect for content access and use preferences. AI Summary and Description: Yes **Summary:**…

The Cloudflare Blog: Giving users choice with Cloudflare’s new Content Signals Policy

Sep 24, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://blog.cloudflare.com/content-signals-policy/ Source: The Cloudflare Blog Title: Giving users choice with Cloudflare’s new Content Signals Policy Feedly Summary: Cloudflare’s Content Signals Policy gives creators a new tool to control use of their content. AI Summary and Description: Yes **Summary:** The text details the introduction of the Content Signals Policy by Cloudflare, which enables website…

Slashdot: RSS Co-Creator Launches New Protocol For AI Data Licensing

Sep 11, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://tech.slashdot.org/story/25/09/10/2320207/rss-co-creator-launches-new-protocol-for-ai-data-licensing?utm_source=rss1.0mainlinkanon&utm_medium=feed Source: Slashdot Title: RSS Co-Creator Launches New Protocol For AI Data Licensing Feedly Summary: AI Summary and Description: Yes Summary: The Real Simple Licensing (RSL) initiative seeks to standardize and simplify the licensing of online content for AI training, backed by major publishers such as Reddit and Medium. It aims to create…

Slashdot: Are AI Web Crawlers ‘Destroying Websites’ In Their Hunt for Training Data?

Aug 31, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://tech.slashdot.org/story/25/08/31/1820249/are-ai-web-crawlers-destroying-websites-in-their-hunt-for-training-data Source: Slashdot Title: Are AI Web Crawlers ‘Destroying Websites’ In Their Hunt for Training Data? Feedly Summary: AI Summary and Description: Yes Summary: The text discusses the adverse effects of AI web crawlers on website performance, highlighting the increasing web traffic attributed to these bots. It addresses the challenges website owners face…

The Cloudflare Blog: A deeper look at AI crawlers: breaking down traffic by purpose and industry

Aug 28, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://blog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry/ Source: The Cloudflare Blog Title: A deeper look at AI crawlers: breaking down traffic by purpose and industry Feedly Summary: We are extending AI-related insights on Cloudflare Radar with new industry-focused data and a breakdown of bot traffic by purpose, such as training or user action. AI Summary and Description: Yes Summary:…

The Cloudflare Blog: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives

Aug 4, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/ Source: The Cloudflare Blog Title: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives Feedly Summary: Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites. AI Summary and Description: Yes Summary: The…

Slashdot: Perplexity is Using Stealth, Undeclared Crawlers To Evade Website No-Crawl Directives, Cloudflare Says

Aug 4, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://tech.slashdot.org/story/25/08/04/1459240/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives-cloudflare-says?utm_source=rss1.0mainlinkanon&utm_medium=feed Source: Slashdot Title: Perplexity is Using Stealth, Undeclared Crawlers To Evade Website No-Crawl Directives, Cloudflare Says Feedly Summary: AI Summary and Description: Yes Summary: The report highlights ethical concerns regarding the web crawling practices of the AI startup Perplexity. By using undetected methods to bypass website restrictions on automated access, this behavior…

Simon Willison’s Weblog: TIL: Rate limiting by IP using Cloudflare’s rate limiting rules

Jul 3, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://simonwillison.net/2025/Jul/3/rate-limiting-by-ip/#atom-everything Source: Simon Willison’s Weblog Title: TIL: Rate limiting by IP using Cloudflare’s rate limiting rules Feedly Summary: TIL: Rate limiting by IP using Cloudflare’s rate limiting rules My blog started timing out on some requests a few days ago, and it turned out there were misbehaving crawlers that were spidering my /search/…

The Cloudflare Blog: From Googlebot to GPTBot: who’s crawling your site in 2025

Jul 1, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/ Source: The Cloudflare Blog Title: From Googlebot to GPTBot: who’s crawling your site in 2025 Feedly Summary: From May 2024 to May 2025, crawler traffic rose 18%, with GPTBot growing 305% and Googlebot 96%. AI Summary and Description: Yes Summary: The text discusses the evolution of web crawlers, particularly focusing on the…

Tag: robots.txt