Experimental News Clipping Site

Tag: web crawler

  • Slashdot: Wikimedia Drowning in AI Bot Traffic as Crawlers Consume 65% of Resources

    —

    by

    Kurt Seifried
    in

    Source URL: https://news.slashdot.org/story/25/04/04/2357233/wikimedia-drowning-in-ai-bot-traffic-as-crawlers-consume-65-of-resources Source: Slashdot Title: Wikimedia Drowning in AI Bot Traffic as Crawlers Consume 65% of Resources Feedly Summary: AI Summary and Description: Yes Summary: The text highlights an emerging issue faced by the Wikimedia Foundation, where web crawlers are significantly impacting their infrastructure by overwhelming it with automated traffic, particularly for training AI…

  • Hacker News: FOSS infrastructure is under attack by AI companies

    —

    by

    Kurt Seifried
    in

    Source URL: https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ Source: Hacker News Title: FOSS infrastructure is under attack by AI companies Feedly Summary: Comments AI Summary and Description: Yes Summary: The text discusses recent disruptions faced by open-source projects due to aggressive AI crawlers that disregard robots.txt protocols, leading to significant operations challenges and increased workloads for system administrators. It highlights…

  • Slashdot: AI Crawlers Haven’t Learned To Play Nice With Websites

    —

    by

    Kurt Seifried
    in

    Source URL: https://slashdot.org/story/25/03/19/1027251/ai-crawlers-havent-learned-to-play-nice-with-websites?utm_source=rss1.0mainlinkanon&utm_medium=feed Source: Slashdot Title: AI Crawlers Haven’t Learned To Play Nice With Websites Feedly Summary: AI Summary and Description: Yes Summary: SourceHut is experiencing service disruptions due to aggressive web crawling by AI companies collecting data for training large language models (LLMs). They have implemented mitigations, including blocking certain cloud providers due to…

  • Hacker News: AI crawlers haven’t learned to play nice with websites

    —

    by

    Kurt Seifried
    in

    Source URL: https://www.theregister.com/2025/03/18/ai_crawlers_sourcehut/ Source: Hacker News Title: AI crawlers haven’t learned to play nice with websites Feedly Summary: Comments AI Summary and Description: Yes Summary: SourceHut reports that excessive crawling by AI companies’ web crawlers is disrupting its services. These crawlers, primarily for training large language models (LLMs), have compelled SourceHut to implement several mitigations,…

  • The Register: AI crawlers haven’t learned to play nice with websites

    —

    by

    Kurt Seifried
    in

    Source URL: https://www.theregister.com/2025/03/18/ai_crawlers_sourcehut/ Source: The Register Title: AI crawlers haven’t learned to play nice with websites Feedly Summary: SourceHut says it’s getting DDoSed by LLM bots SourceHut, an open source git-hosting service, says web crawlers for AI companies are slowing down services through their excessive demands for data.… AI Summary and Description: Yes Summary: The…

  • Slashdot: BlueSky Proposes ‘New Standard’ for When Scraping Data for AI Training

    —

    by

    Kurt Seifried
    in

    Source URL: https://tech.slashdot.org/story/25/03/17/0434237/bluesky-proposes-new-standard-for-when-scraping-data-for-ai-training?utm_source=rss1.0mainlinkanon&utm_medium=feed Source: Slashdot Title: BlueSky Proposes ‘New Standard’ for When Scraping Data for AI Training Feedly Summary: AI Summary and Description: Yes Summary: The article discusses Bluesky’s proposal for user data consent regarding scraping for generative AI training and archiving. This initiative signifies a potential shift in how user data privacy is managed…

  • Hacker News: Moscow-based global news network has infected Western AI tools

    —

    by

    Kurt Seifried
    in

    Source URL: https://www.newsguardrealitycheck.com/p/a-well-funded-moscow-based-global Source: Hacker News Title: Moscow-based global news network has infected Western AI tools Feedly Summary: Comments AI Summary and Description: Yes Summary: The text discusses a disinformation network, “Pravda,” that is manipulating AI chatbots by flooding them with false narratives and propaganda, resulting in a significant percentage of chatbot outputs containing disinformation.…

  • Hacker News: AI haters build tarpits to trap and trick AI scrapers that ignore robots.txt

    —

    by

    Kurt Seifried
    in

    Source URL: https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/ Source: Hacker News Title: AI haters build tarpits to trap and trick AI scrapers that ignore robots.txt Feedly Summary: Comments AI Summary and Description: Yes Summary: The text discusses the creation of a new malware named Nepenthes, designed by a software developer to combat AI web crawlers that ignore “no scraping” directives…

  • Slashdot: Developer Creates Infinite Maze That Traps AI Training Bots

    —

    by

    Kurt Seifried
    in

    Source URL: https://slashdot.org/story/25/01/23/2135205/developer-creates-infinite-maze-that-traps-ai-training-bots?utm_source=rss1.0mainlinkanon&utm_medium=feed Source: Slashdot Title: Developer Creates Infinite Maze That Traps AI Training Bots Feedly Summary: AI Summary and Description: Yes Summary: The text discusses the development of an open-source program called Nepenthes, designed to trap AI web crawlers in an endless loop of link generation, effectively wasting their resources. This innovative approach provides…