Experimental News Clipping Site

Tag: web crawling

  • Slashdot: AI Crawlers Haven’t Learned To Play Nice With Websites

    —

    by

    Kurt Seifried
    in

    Source URL: https://slashdot.org/story/25/03/19/1027251/ai-crawlers-havent-learned-to-play-nice-with-websites?utm_source=rss1.0mainlinkanon&utm_medium=feed Source: Slashdot Title: AI Crawlers Haven’t Learned To Play Nice With Websites Feedly Summary: AI Summary and Description: Yes Summary: SourceHut is experiencing service disruptions due to aggressive web crawling by AI companies collecting data for training large language models (LLMs). They have implemented mitigations, including blocking certain cloud providers due to…

  • Hacker News: AI haters build tarpits to trap and trick AI scrapers that ignore robots.txt

    —

    by

    Kurt Seifried
    in

    Source URL: https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/ Source: Hacker News Title: AI haters build tarpits to trap and trick AI scrapers that ignore robots.txt Feedly Summary: Comments AI Summary and Description: Yes Summary: The text discusses the creation of a new malware named Nepenthes, designed by a software developer to combat AI web crawlers that ignore “no scraping” directives…

  • Hacker News: OpenAI’s bot crushed this seven-person company’s web site ‘like a DDoS attack’

    —

    by

    Kurt Seifried
    in

    Source URL: https://techcrunch.com/2025/01/10/how-openais-bot-crushed-this-seven-person-companys-web-site-like-a-ddos-attack/ Source: Hacker News Title: OpenAI’s bot crushed this seven-person company’s web site ‘like a DDoS attack’ Feedly Summary: Comments AI Summary and Description: Yes Summary: The text highlights a significant incident involving Triplegangers’ CEO Oleksandr Tomchuk, whose e-commerce site was subjected to aggressive scraping by OpenAI’s bot, leading to operational disruptions and…