Experimental News Clipping Site

Tag: bot traffic

  • The Cloudflare Blog: The crawl before the fall… of referrals: understanding AI’s impact on content providers

    —

    by

    Kurt Seifried
    in

    Source URL: https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/ Source: The Cloudflare Blog Title: The crawl before the fall… of referrals: understanding AI’s impact on content providers Feedly Summary: Cloudflare Radar now shows how often a given AI model sends traffic to a site relative to how often it crawls that site. This helps site owners make decisions about which AI…

  • Slashdot: Web-Scraping AI Bots Cause Disruption For Scientific Databases and Journals

    —

    by

    Kurt Seifried
    in

    Source URL: https://science.slashdot.org/story/25/06/02/172202/web-scraping-ai-bots-cause-disruption-for-scientific-databases-and-journals?utm_source=rss1.0mainlinkanon&utm_medium=feed Source: Slashdot Title: Web-Scraping AI Bots Cause Disruption For Scientific Databases and Journals Feedly Summary: AI Summary and Description: Yes Summary: The text highlights the impact of automated web-scraping bots on scientific databases and academic journals, driven by the demand for training data for AI models. This has led to significant service…

  • Slashdot: Wikimedia Drowning in AI Bot Traffic as Crawlers Consume 65% of Resources

    —

    by

    Kurt Seifried
    in

    Source URL: https://news.slashdot.org/story/25/04/04/2357233/wikimedia-drowning-in-ai-bot-traffic-as-crawlers-consume-65-of-resources Source: Slashdot Title: Wikimedia Drowning in AI Bot Traffic as Crawlers Consume 65% of Resources Feedly Summary: AI Summary and Description: Yes Summary: The text highlights an emerging issue faced by the Wikimedia Foundation, where web crawlers are significantly impacting their infrastructure by overwhelming it with automated traffic, particularly for training AI…

  • Slashdot: Open Source Devs Say AI Crawlers Dominate Traffic, Forcing Blocks On Entire Countries

    —

    by

    Kurt Seifried
    in

    Source URL: https://tech.slashdot.org/story/25/03/26/016244/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries Source: Slashdot Title: Open Source Devs Say AI Crawlers Dominate Traffic, Forcing Blocks On Entire Countries Feedly Summary: AI Summary and Description: Yes Summary: The text discusses the challenges faced by software developers, particularly open source maintainers, in managing aggressive AI crawler traffic that overwhelms their repositories. This scenario underscores the urgent…

  • Hacker News: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

    —

    by

    Kurt Seifried
    in

    Source URL: https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/ Source: Hacker News Title: Devs say AI crawlers dominate traffic, forcing blocks on entire countries Feedly Summary: Comments AI Summary and Description: Yes Summary: The text discusses the challenges faced by software developers in managing aggressive AI crawler traffic that negatively affects open-source projects, leading to significant service instability and increased operational…

  • Slashdot: AI Crawlers Haven’t Learned To Play Nice With Websites

    —

    by

    Kurt Seifried
    in

    Source URL: https://slashdot.org/story/25/03/19/1027251/ai-crawlers-havent-learned-to-play-nice-with-websites?utm_source=rss1.0mainlinkanon&utm_medium=feed Source: Slashdot Title: AI Crawlers Haven’t Learned To Play Nice With Websites Feedly Summary: AI Summary and Description: Yes Summary: SourceHut is experiencing service disruptions due to aggressive web crawling by AI companies collecting data for training large language models (LLMs). They have implemented mitigations, including blocking certain cloud providers due to…

  • Hacker News: AI crawlers haven’t learned to play nice with websites

    —

    by

    Kurt Seifried
    in

    Source URL: https://www.theregister.com/2025/03/18/ai_crawlers_sourcehut/ Source: Hacker News Title: AI crawlers haven’t learned to play nice with websites Feedly Summary: Comments AI Summary and Description: Yes Summary: SourceHut reports that excessive crawling by AI companies’ web crawlers is disrupting its services. These crawlers, primarily for training large language models (LLMs), have compelled SourceHut to implement several mitigations,…