Source URL: https://simonwillison.net/2025/Jul/3/rate-limiting-by-ip/#atom-everything
Source: Simon Willison’s Weblog
Title: TIL: Rate limiting by IP using Cloudflare’s rate limiting rules
Feedly Summary: TIL: Rate limiting by IP using Cloudflare’s rate limiting rules
My blog started timing out on some requests a few days ago, and it turned out there were misbehaving crawlers that were spidering my /search/ page even though it’s restricted by robots.txt.
I run this site behind Cloudflare and it turns out Cloudflare’s WAF (Web Application Firewall) has a rate limiting tool that I could use to restrict requests to /search/* by a specific IP to a maximum of 5 every 10 seconds.
Tags: rate-limiting, security, cloudflare, til
AI Summary and Description: Yes
**Summary:** The text discusses an incident involving rate limiting by IP using Cloudflare’s rate limiting rules, specifically to control unwanted automated requests to a website. This is highly relevant for security professionals, especially in the context of cloud security, as it highlights practical applications of protective measures against bot attacks.
**Detailed Description:** The text outlines a practical scenario involving web security, where the author experienced issues due to aggressive web crawlers accessing a restricted section of their website. They utilized Cloudflare’s rate limiting feature as a security measure. Key insights and points from the text include:
– **Context of the Issue:** The author’s blog began to experience timeouts due to excessive requests from misbehaving crawlers, indicating a potential threat to the website’s performance and availability.
– **Use of Robots.txt:** The existence of the robots.txt file was mentioned, which is a standard used to manage web crawler behavior, though it didn’t prevent the issue in this case.
– **Cloudflare’s Role:** The author is utilizing Cloudflare’s services, specifically its Web Application Firewall (WAF), which offers various security features including request rate limiting.
– **Rate Limiting Configuration:** Implementing a rule to limit requests to the /search/ endpoint to five requests per IP every ten seconds effectively mitigated the issue with the unwanted crawler activities. This shows practical application and implementation of security features to enhance application security.
– **Relevance to Professionals:** This case serves as a valuable lesson for web security and infrastructure security professionals who manage online resources, emphasizing the importance of monitoring and controlling traffic to protect applications.
Overall, the content signifies how cloud-based security tools like Cloudflare can provide effective measures against potential attacks, illustrating a crucial operational practice for ensuring web application security.