Experimental News Clipping Site

Tag: crawler

  • The Register: Cloudflare builds an AI to lead AI scraper bots into a horrible maze of junk content

    —

    by

    Kurt Seifried
    in

    Source URL: https://www.theregister.com/2025/03/21/cloudflare_ai_labyrinth/ Source: The Register Title: Cloudflare builds an AI to lead AI scraper bots into a horrible maze of junk content Feedly Summary: Slop-making machine will feed unauthorized scrapers what they so richly deserve, hopefully without poisoning the internet Cloudflare has created a bot-busting AI to make life hell for AI crawlers.… AI…

  • Simon Willison’s Weblog: Claude can now search the web

    —

    by

    Kurt Seifried
    in

    Source URL: https://simonwillison.net/2025/Mar/20/claude-can-now-search-the-web/#atom-everything Source: Simon Willison’s Weblog Title: Claude can now search the web Feedly Summary: Claude can now search the web Claude 3.7 Sonnet on the paid plan now has a web search tool that can be turned on as a global setting. This was sorely needed. ChatGPT, Gemini and Grok all had this…

  • Hacker News: FOSS infrastructure is under attack by AI companies

    —

    by

    Kurt Seifried
    in

    Source URL: https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ Source: Hacker News Title: FOSS infrastructure is under attack by AI companies Feedly Summary: Comments AI Summary and Description: Yes Summary: The text discusses recent disruptions faced by open-source projects due to aggressive AI crawlers that disregard robots.txt protocols, leading to significant operations challenges and increased workloads for system administrators. It highlights…

  • Slashdot: AI Crawlers Haven’t Learned To Play Nice With Websites

    —

    by

    Kurt Seifried
    in

    Source URL: https://slashdot.org/story/25/03/19/1027251/ai-crawlers-havent-learned-to-play-nice-with-websites?utm_source=rss1.0mainlinkanon&utm_medium=feed Source: Slashdot Title: AI Crawlers Haven’t Learned To Play Nice With Websites Feedly Summary: AI Summary and Description: Yes Summary: SourceHut is experiencing service disruptions due to aggressive web crawling by AI companies collecting data for training large language models (LLMs). They have implemented mitigations, including blocking certain cloud providers due to…

  • The Cloudflare Blog: Cloudflare for AI: supporting AI adoption at scale with a security-first approach

    —

    by

    Kurt Seifried
    in

    Source URL: https://blog.cloudflare.com/cloudflare-for-ai-supporting-ai-adoption-at-scale-with-a-security-first-approach/ Source: The Cloudflare Blog Title: Cloudflare for AI: supporting AI adoption at scale with a security-first approach Feedly Summary: With Cloudflare for AI, developers, security teams and content creators can leverage Cloudflare’s network and portfolio of tools to secure, observe and make AI applications resilient and safe to use. AI Summary and…

  • Hacker News: AI crawlers haven’t learned to play nice with websites

    —

    by

    Kurt Seifried
    in

    Source URL: https://www.theregister.com/2025/03/18/ai_crawlers_sourcehut/ Source: Hacker News Title: AI crawlers haven’t learned to play nice with websites Feedly Summary: Comments AI Summary and Description: Yes Summary: SourceHut reports that excessive crawling by AI companies’ web crawlers is disrupting its services. These crawlers, primarily for training large language models (LLMs), have compelled SourceHut to implement several mitigations,…

  • Hacker News: Please stop externalizing your costs directly into my face

    —

    by

    Kurt Seifried
    in

    Source URL: https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html Source: Hacker News Title: Please stop externalizing your costs directly into my face Feedly Summary: Comments AI Summary and Description: Yes Summary: The text reflects a sysadmin’s frustration with the disruptive impact of LLM crawlers on operational stability. It discusses ongoing battles against the misuse of computing resources by malicious bots, underscoring…

  • The Register: AI crawlers haven’t learned to play nice with websites

    —

    by

    Kurt Seifried
    in

    Source URL: https://www.theregister.com/2025/03/18/ai_crawlers_sourcehut/ Source: The Register Title: AI crawlers haven’t learned to play nice with websites Feedly Summary: SourceHut says it’s getting DDoSed by LLM bots SourceHut, an open source git-hosting service, says web crawlers for AI companies are slowing down services through their excessive demands for data.… AI Summary and Description: Yes Summary: The…

  • Slashdot: BlueSky Proposes ‘New Standard’ for When Scraping Data for AI Training

    —

    by

    Kurt Seifried
    in

    Source URL: https://tech.slashdot.org/story/25/03/17/0434237/bluesky-proposes-new-standard-for-when-scraping-data-for-ai-training?utm_source=rss1.0mainlinkanon&utm_medium=feed Source: Slashdot Title: BlueSky Proposes ‘New Standard’ for When Scraping Data for AI Training Feedly Summary: AI Summary and Description: Yes Summary: The article discusses Bluesky’s proposal for user data consent regarding scraping for generative AI training and archiving. This initiative signifies a potential shift in how user data privacy is managed…