{"id":36,"date":"2026-11-15T12:57:14","date_gmt":"2026-11-15T12:57:14","guid":{"rendered":"https:\/\/globalcorpora.com\/blog\/web-scraping-rate-limiting-retries-backoff\/"},"modified":"2026-11-15T12:57:14","modified_gmt":"2026-11-15T12:57:14","slug":"web-scraping-rate-limiting-retries-backoff","status":"publish","type":"post","link":"https:\/\/globalcorpora.com\/blog\/web-scraping-rate-limiting-retries-backoff\/","title":{"rendered":"Web Scraping Rate Limiting: Retries, Backoff and 429 Handling"},"content":{"rendered":"<p>A crawler that works at 10 requests per second often falls apart at 1,000: hosts start answering <code>429<\/code> and <code>503<\/code>, retries pile up on top of the original load, and a few slow sites quietly eat your whole worker pool. This guide covers web scraping rate limiting from the protocol up: what the status codes and <code>Retry-After<\/code> actually mean, how to budget requests per host, how to back off without synchronized retry storms, and how to wire it all together in Python with httpx, tenacity or Scrapy.<\/p>\n<h2>TL;DR<\/h2>\n<ul>\n<li><strong>Limit per host, not globally.<\/strong> A global cap of 500 in-flight requests says nothing about how hard you hit any single site. Use a per-host concurrency cap plus a per-host token bucket.<\/li>\n<li><strong>Treat <code>429<\/code> and <code>503<\/code> as \"slow down\", not \"try again now\".<\/strong> Honor <code>Retry-After<\/code> when present (seconds or an HTTP date) and apply it to the whole host, not just the one URL.<\/li>\n<li><strong>Back off exponentially with jitter.<\/strong> Plain exponential backoff synchronizes clients; full jitter (<code>random(0, min(cap, base * 2^n))<\/code>) spreads them out.<\/li>\n<li><strong>Only retry what can succeed later:<\/strong> timeouts, connection errors, <code>408<\/code>, <code>429<\/code>, <code>500<\/code>, <code>502<\/code>, <code>503<\/code>, <code>504<\/code>. Never blindly retry <code>404<\/code> or other 4xx, and be careful with non-idempotent methods.<\/li>\n<li><strong>Add a per-host circuit breaker<\/strong> so one failing site cannot consume your retry budget and workers.<\/li>\n<li><strong>Read robots.txt (RFC 9309).<\/strong> <code>Crawl-delay<\/code> is not part of the standard; treat it as a hint you choose to honor.<\/li>\n<\/ul>\n<h2>What the server is telling you: 429, 503 and Retry-After<\/h2>\n<p><code>429 Too Many Requests<\/code> is defined in <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc6585.txt\">RFC 6585<\/a>: the user \"has sent too many requests in a given amount of time\". The response <em>may<\/em> include a <code>Retry-After<\/code> header. The RFC deliberately does not define how the server counts requests or identifies you; it could be per resource, per server or across a fleet, keyed on IP, credentials or a cookie. So a <code>429<\/code> on one path is evidence about the whole host until proven otherwise.<\/p>\n<p>The same RFC notes that servers are not required to send <code>429<\/code> at all. Many sites drop connections, return <code>503<\/code> or <code>403<\/code>, or serve an error page with status <code>200<\/code>. React to all of these, not just the polite one.<\/p>\n<p><code>503 Service Unavailable<\/code> and the <code>Retry-After<\/code> field are defined in <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9110.html\">RFC 9110 (HTTP Semantics)<\/a>. <code>Retry-After<\/code> takes one of two forms:<\/p>\n<pre><code>Retry-After: 120\nRetry-After: Fri, 13 Nov 2026 23:59:59 GMT<\/code><\/pre>\n<p>The first is a delay in seconds, the second an HTTP date. Parse both. A robust parser treats an unparseable value as absent and falls back to your own backoff. Also put an upper bound on how long you will sleep in-process: a <code>Retry-After: 86400<\/code> means \"come back tomorrow\", which is a scheduling decision (requeue the host), not a <code>sleep()<\/code>.<\/p>\n<h2>Budget requests per host<\/h2>\n<p>Two independent controls, applied per host (scheme plus hostname plus port is a reasonable key):<\/p>\n<table>\n<thead>\n<tr>\n<th>Control<\/th>\n<th>Limits<\/th>\n<th>Protects against<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Concurrency cap (semaphore)<\/td>\n<td>Requests in flight at once<\/td>\n<td>Overloading slow hosts; one host hogging connections<\/td>\n<\/tr>\n<tr>\n<td>Token bucket<\/td>\n<td>Average request rate, with a bounded burst<\/td>\n<td>Exceeding a rate the host tolerates, even when responses are fast<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>You need both. A concurrency cap alone lets a fast host absorb a high request rate: four connections to a server that answers in 20 ms is 4 \/ 0.02 = 200 requests per second. A rate limit alone lets a slow host pile up open connections. The <a href=\"https:\/\/docs.scrapy.org\/en\/latest\/topics\/autothrottle.html\">Scrapy AutoThrottle docs<\/a> point out a related trap: error responses are often returned faster than normal ones, so a fixed concurrency cap makes you send <em>faster<\/em> exactly when the server starts failing.<\/p>\n<p>A token bucket has two parameters: a refill rate <em>r<\/em> (tokens per second) and a capacity <em>b<\/em> (maximum burst). Each request spends one token. Over any window of <em>t<\/em> seconds you send at most <em>b + r \u00d7 t<\/em> requests. With <em>r<\/em> = 2 and <em>b<\/em> = 4, a minute allows at most 4 + 2 \u00d7 60 = 124 requests to that host.<\/p>\n<p>Start conservative and adapt. A common pattern is AIMD (additive increase, multiplicative decrease, the same idea TCP uses): on every success nudge the rate up a little, on every throttle signal cut it in half. That converges on what the host tolerates without you guessing a number.<\/p>\n<h2>Exponential backoff with jitter<\/h2>\n<p>When a request fails with a retryable error, wait before retrying, and wait longer each time. The canonical reference is the AWS Architecture Blog post <a href=\"https:\/\/aws.amazon.com\/blogs\/architecture\/exponential-backoff-and-jitter\/\">Exponential Backoff And Jitter<\/a>, whose simulator code is <a href=\"https:\/\/github.com\/aws-samples\/aws-arch-backoff-simulator\">published on GitHub<\/a>. Its variants, with attempt number <em>n<\/em>, base delay <em>base<\/em> and ceiling <em>cap<\/em>:<\/p>\n<table>\n<thead>\n<tr>\n<th>Strategy<\/th>\n<th>Sleep before retry<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Exponential<\/td>\n<td><code>min(cap, base * 2**n)<\/code><\/td>\n<\/tr>\n<tr>\n<td>Equal jitter<\/td>\n<td><code>v\/2 + uniform(0, v\/2)<\/code>, where <code>v = min(cap, base * 2**n)<\/code><\/td>\n<\/tr>\n<tr>\n<td>Full jitter<\/td>\n<td><code>uniform(0, min(cap, base * 2**n))<\/code><\/td>\n<\/tr>\n<tr>\n<td>Decorrelated jitter<\/td>\n<td><code>sleep = min(cap, uniform(base, sleep * 3))<\/code><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Why jitter matters for crawlers: if 200 workers all hit a <code>503<\/code> at the same moment and all wait exactly 2, 4, 8 seconds, they come back in synchronized waves and re-trigger the overload. Randomizing the wait spreads them across the window. That is the point of the post's simulations, which compare total calls and completion time across the variants. Full jitter is the simplest to implement and the usual default.<\/p>\n<p>Two rules on top of the formula. First, if the server sent <code>Retry-After<\/code>, wait at least that long: <code>max(jittered_backoff, retry_after)<\/code>. Second, cap total attempts (5 to 7 is typical) and total elapsed time, then give up and record the failure for a later pass.<\/p>\n<h2>What to retry, and what not to<\/h2>\n<table>\n<thead>\n<tr>\n<th>Outcome<\/th>\n<th>Retry?<\/th>\n<th>Notes<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Connect timeout, read timeout, connection reset, DNS hiccup<\/td>\n<td>Yes, with backoff<\/td>\n<td>Count toward the host's circuit breaker<\/td>\n<\/tr>\n<tr>\n<td><code>429<\/code><\/td>\n<td>Yes, after <code>Retry-After<\/code> or backoff<\/td>\n<td>Also lower that host's rate<\/td>\n<\/tr>\n<tr>\n<td><code>503<\/code><\/td>\n<td>Yes, after <code>Retry-After<\/code> or backoff<\/td>\n<td>Often maintenance or overload; slow the host down<\/td>\n<\/tr>\n<tr>\n<td><code>500<\/code>, <code>502<\/code>, <code>504<\/code>, <code>408<\/code><\/td>\n<td>Yes, a few times<\/td>\n<td>Persistent <code>500<\/code> on one URL is usually a bug on their side; stop after a few<\/td>\n<\/tr>\n<tr>\n<td><code>404<\/code>, <code>410<\/code><\/td>\n<td>No<\/td>\n<td>Record as gone<\/td>\n<\/tr>\n<tr>\n<td><code>400<\/code>, <code>401<\/code>, <code>405<\/code>, <code>422<\/code><\/td>\n<td>No<\/td>\n<td>Your request is wrong; retrying will not fix it<\/td>\n<\/tr>\n<tr>\n<td><code>403<\/code><\/td>\n<td>Not immediately<\/td>\n<td>Often a block or bot check; treat as a throttle signal for the host and investigate<\/td>\n<\/tr>\n<tr>\n<td><code>200<\/code> with a captcha or \"rate limited\" body<\/td>\n<td>Treat as <code>429<\/code><\/td>\n<td>Detect with a content check; it will poison your dataset otherwise<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Scrapy's default retry list is a useful cross-check: <code>RETRY_HTTP_CODES<\/code> defaults to <code>[408, 429, 500, 502, 503, 504, 522, 524]<\/code> (522 and 524 are Cloudflare-specific codes).<\/p>\n<h3>Idempotency<\/h3>\n<p>RFC 9110 defines <code>PUT<\/code>, <code>DELETE<\/code> and the safe methods (<code>GET<\/code>, <code>HEAD<\/code>, <code>OPTIONS<\/code>, <code>TRACE<\/code>) as idempotent: repeating them has the same intended effect as sending them once. Crawlers are almost entirely <code>GET<\/code>, so retries are safe. The exception is <code>POST<\/code>-driven endpoints (search forms, GraphQL, internal APIs). A timed-out <code>POST<\/code> may have been processed; only retry it automatically if you know the operation is a pure read or the API supports an idempotency key.<\/p>\n<h2>A per-host limiter in Python (httpx + asyncio)<\/h2>\n<p>The following is a complete, dependency-light implementation: per-host semaphore, token bucket with AIMD, <code>Retry-After<\/code> parsing, full-jitter backoff and a circuit breaker. It needs only <code>pip install httpx<\/code>.<\/p>\n<pre><code>import asyncio\nimport random\nimport time\nfrom datetime import datetime, timezone\nfrom email.utils import parsedate_to_datetime\nfrom urllib.parse import urlsplit\n\nimport httpx\n\nRETRYABLE_STATUS = {408, 429, 500, 502, 503, 504}\nTHROTTLE_STATUS = {429, 503}\nRETRYABLE_EXC = (httpx.TimeoutException, httpx.NetworkError, httpx.RemoteProtocolError)\nMAX_INLINE_WAIT = 300.0  # longer Retry-After values: defer the URL instead of sleeping\n\n\nclass GiveUp(Exception):\n    pass\n\n\nclass Deferred(Exception):\n    pass\n\n\ndef parse_retry_after(value):\n    \"\"\"Return seconds to wait, or None. Accepts delay-seconds or an HTTP-date.\"\"\"\n    if not value:\n        return None\n    value = value.strip()\n    if value.isdigit():\n        return float(value)\n    try:\n        dt = parsedate_to_datetime(value)\n    except (TypeError, ValueError):\n        return None\n    if dt.tzinfo is None:\n        dt = dt.replace(tzinfo=timezone.utc)\n    return max(0.0, (dt - datetime.now(timezone.utc)).total_seconds())\n\n\ndef full_jitter(attempt, base=1.0, cap=60.0):\n    return random.uniform(0, min(cap, base * 2 ** attempt))\n\n\nclass TokenBucket:\n    def __init__(self, rate, burst, min_rate=0.1, max_rate=10.0):\n        self.rate, self.capacity = rate, burst\n        self.min_rate, self.max_rate = min_rate, max_rate\n        self.tokens = float(burst)\n        self.updated = time.monotonic()\n        self.lock = asyncio.Lock()\n\n    async def acquire(self):\n        async with self.lock:\n            while True:\n                now = time.monotonic()\n                self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)\n                self.updated = now\n                if self.tokens &gt;= 1:\n                    self.tokens -= 1\n                    return\n                await asyncio.sleep((1 - self.tokens) \/ self.rate)\n\n    def on_success(self):   # additive increase\n        self.rate = min(self.max_rate, self.rate + 0.05)\n\n    def on_throttle(self):  # multiplicative decrease\n        self.rate = max(self.min_rate, self.rate \/ 2)\n\n\nclass CircuitBreaker:\n    def __init__(self, threshold=5, cooldown=60.0):\n        self.threshold, self.cooldown = threshold, cooldown\n        self.failures = 0\n        self.opened_at = None\n        self.probing = False\n\n    def allow(self):\n        if self.opened_at is None:\n            return True\n        if not self.probing and time.monotonic() - self.opened_at &gt;= self.cooldown:\n            self.probing = True  # half-open: let one request through\n            return True\n        return False\n\n    def record_success(self):\n        self.failures, self.opened_at, self.probing = 0, None, False\n\n    def record_failure(self):\n        self.failures += 1\n        if self.probing or self.failures &gt;= self.threshold:\n            self.opened_at, self.probing = time.monotonic(), False\n\n\nclass Host:\n    def __init__(self, concurrency=4, rate=2.0, burst=4):\n        self.sem = asyncio.Semaphore(concurrency)\n        self.bucket = TokenBucket(rate, burst)\n        self.breaker = CircuitBreaker()\n        self.not_before = 0.0  # monotonic timestamp; host-wide pause\n\n\nhosts = {}\n\n\ndef host_for(url):\n    parts = urlsplit(url)\n    key = f\"{parts.scheme}:\/\/{parts.netloc.lower()}\"\n    if key not in hosts:\n        hosts[key] = Host()\n    return hosts[key]\n\n\nasync def fetch(client, url, max_attempts=6):\n    host = host_for(url)\n    last_error = None\n    for attempt in range(max_attempts):\n        if not host.breaker.allow():\n            raise Deferred(f\"circuit open for {url}\")\n        pause = host.not_before - time.monotonic()\n        if pause &gt; 0:\n            await asyncio.sleep(pause)\n\n        async with host.sem:\n            await host.bucket.acquire()\n            try:\n                resp = await client.get(url)\n            except RETRYABLE_EXC as exc:\n                host.breaker.record_failure()\n                last_error = repr(exc)\n                delay = full_jitter(attempt)\n            else:\n                status = resp.status_code\n                if status &lt; 400:\n                    host.breaker.record_success()\n                    host.bucket.on_success()\n                    return resp\n                if status not in RETRYABLE_STATUS:\n                    raise GiveUp(f\"HTTP {status} for {url}\")  # 404, 403, 400 ...\n                host.breaker.record_failure()\n                last_error = f\"HTTP {status}\"\n                delay = full_jitter(attempt)\n                if status in THROTTLE_STATUS:\n                    host.bucket.on_throttle()\n                retry_after = parse_retry_after(resp.headers.get(\"Retry-After\"))\n                if retry_after is not None:\n                    host.not_before = max(host.not_before, time.monotonic() + retry_after)\n                    if retry_after &gt; MAX_INLINE_WAIT:\n                        raise Deferred(f\"Retry-After {retry_after:.0f}s for {url}\")\n                    delay = max(delay, retry_after)\n        await asyncio.sleep(delay)\n    raise GiveUp(f\"{url}: gave up after {max_attempts} attempts ({last_error})\")\n\n\nasync def main(urls):\n    limits = httpx.Limits(max_connections=200, max_keepalive_connections=50)\n    timeout = httpx.Timeout(20.0, connect=10.0)\n    headers = {\"User-Agent\": \"ExampleBot\/0.1 (+https:\/\/www.example.com\/bot.html)\"}\n    async with httpx.AsyncClient(limits=limits, timeout=timeout,\n                                 headers=headers, follow_redirects=True) as client:\n        results = await asyncio.gather(*(fetch(client, u) for u in urls),\n                                       return_exceptions=True)\n    for url, r in zip(urls, results):\n        print(url, r.status_code if isinstance(r, httpx.Response) else repr(r))\n\n\nif __name__ == \"__main__\":\n    asyncio.run(main([\"https:\/\/example.com\/\", \"https:\/\/example.org\/\"]))<\/code><\/pre>\n<p>Notes on the design:<\/p>\n<ul>\n<li><code>httpx.Limits<\/code> caps the connection pool globally (the defaults are 100 connections and 20 keep-alive, per the <a href=\"https:\/\/www.python-httpx.org\/advanced\/resource-limits\/\">httpx resource limits docs<\/a>). It is not a per-host limiter; the semaphore and bucket are.<\/li>\n<li><code>Retry-After<\/code> sets <code>not_before<\/code> for the whole host, so every other task targeting that host pauses too, instead of each one discovering the limit separately.<\/li>\n<li>The sleep before a retry happens <em>outside<\/em> the semaphore, so a backing-off request does not hold one of the host's concurrency slots.<\/li>\n<li>For millions of URLs, replace <code>asyncio.gather<\/code> over everything with a bounded queue and a fixed pool of worker tasks, and persist <code>Deferred<\/code> and <code>GiveUp<\/code> outcomes so a later pass can pick them up. The same structure works for browser-based crawls; see <a href=\"https:\/\/globalcorpora.com\/blog\/playwright-headless-browser-web-scraping-at-scale\/\">Playwright headless browser scraping at scale<\/a>.<\/li>\n<\/ul>\n<h3>The same retry policy with tenacity<\/h3>\n<p>If you prefer a library for the retry loop, <a href=\"https:\/\/github.com\/jd\/tenacity\">tenacity<\/a> supports async functions directly. <code>wait_random_exponential<\/code> is full jitter (\"randomly wait up to 2^x * multiplier seconds\"), and any callable that takes <code>retry_state<\/code> can serve as a wait strategy, which is how you layer <code>Retry-After<\/code> on top:<\/p>\n<pre><code>import httpx\nfrom tenacity import retry, retry_if_exception, stop_after_attempt, wait_random_exponential\n\nbackoff = wait_random_exponential(multiplier=1, max=60)\n\n\ndef is_retryable(exc):\n    if isinstance(exc, httpx.HTTPStatusError):\n        return exc.response.status_code in {408, 429, 500, 502, 503, 504}\n    return isinstance(exc, (httpx.TimeoutException, httpx.NetworkError))\n\n\ndef wait_with_retry_after(retry_state):\n    fallback = backoff(retry_state)\n    exc = retry_state.outcome.exception()\n    if isinstance(exc, httpx.HTTPStatusError):\n        ra = parse_retry_after(exc.response.headers.get(\"Retry-After\"))  # from the example above\n        if ra is not None:\n            return max(fallback, min(ra, 300))\n    return fallback\n\n\n@retry(retry=retry_if_exception(is_retryable), wait=wait_with_retry_after,\n       stop=stop_after_attempt(6), reraise=True)\nasync def get(client, url):\n    resp = await client.get(url)\n    resp.raise_for_status()\n    return resp<\/code><\/pre>\n<p>Tenacity only handles the retry loop; it knows nothing about hosts, so keep the per-host semaphore and bucket around the call.<\/p>\n<h2>Circuit breakers per host<\/h2>\n<p>Backoff handles a single request. A circuit breaker handles a host that is down. Without one, a site that times out on every request can tie up workers for <em>attempts \u00d7 timeout<\/em> per URL, across thousands of queued URLs. The breaker in the code above has three states:<\/p>\n<ul>\n<li><strong>Closed:<\/strong> requests flow normally; consecutive failures are counted.<\/li>\n<li><strong>Open:<\/strong> after <em>threshold<\/em> consecutive failures, all requests to the host are deferred for a cooldown period.<\/li>\n<li><strong>Half-open:<\/strong> after the cooldown, one probe request goes through. Success closes the breaker; failure reopens it.<\/li>\n<\/ul>\n<p>Requeue deferred URLs rather than dropping them. If many breakers open at once, the problem is probably on your side (DNS, proxy, egress).<\/p>\n<h2>robots.txt and Crawl-delay<\/h2>\n<p><a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9309.txt\">RFC 9309<\/a> standardized the Robots Exclusion Protocol in 2022. Points that matter for a crawler's control loop:<\/p>\n<ul>\n<li>The standard defines <code>user-agent<\/code>, <code>allow<\/code> and <code>disallow<\/code>, plus the <code>*<\/code> and <code>$<\/code> special characters. The longest (most specific) match wins.<\/li>\n<li>If robots.txt returns a 4xx (\"unavailable\"), the crawler may access anything. If it is unreachable because of a 5xx or network error, the crawler must assume complete disallow.<\/li>\n<li>Crawlers should not use a cached copy for more than 24 hours, unless robots.txt is unreachable.<\/li>\n<li>The rules \"are not a form of access authorization\". They are a request from the operator.<\/li>\n<\/ul>\n<p><code>Crawl-delay<\/code> is <strong>not<\/strong> in RFC 9309. It is a non-standard extension that some crawlers honor and others ignore; Google states that its crawlers <a href=\"https:\/\/developers.google.com\/crawling\/docs\/robots-txt\/robots-txt-spec\">do not support crawl-delay<\/a>. Even so, it is the site operator telling you what rate they consider acceptable, so a sensible policy is to treat it as a floor on the per-host interval: set the bucket rate to at most <code>1 \/ crawl_delay<\/code>.<\/p>\n<p>Python's standard library exposes it through <a href=\"https:\/\/docs.python.org\/3\/library\/urllib.robotparser.html\"><code>urllib.robotparser<\/code><\/a>, with <code>can_fetch()<\/code>, <code>crawl_delay()<\/code> and <code>request_rate()<\/code>. Fetch the file with your own client so you control status-code handling, then feed the lines to the parser:<\/p>\n<pre><code>from urllib.robotparser import RobotFileParser\n\nimport httpx\n\n\nasync def load_robots(client, origin):\n    rp = RobotFileParser()\n    try:\n        r = await client.get(f\"{origin}\/robots.txt\")\n    except httpx.TransportError:\n        rp.parse([\"User-agent: *\", \"Disallow: \/\"])   # unreachable: assume disallow\n        return rp\n    if r.status_code &gt;= 500:\n        rp.parse([\"User-agent: *\", \"Disallow: \/\"])   # unreachable: assume disallow\n    elif r.status_code &gt;= 400:\n        rp.parse([\"User-agent: *\", \"Disallow:\"])     # unavailable: no restrictions\n    else:\n        rp.parse(r.text.splitlines())\n    return rp\n\n# rp.can_fetch(\"ExampleBot\", url)  -&gt; bool\n# rp.crawl_delay(\"ExampleBot\")     -&gt; seconds or None<\/code><\/pre>\n<p>Check your Python version: according to the <a href=\"https:\/\/github.com\/scrapy\/protego\">Protego README<\/a>, <code>urllib.robotparser<\/code> only gained RFC 9309 wildcards and longest-match precedence during the 3.14 series, so older interpreters can misjudge rules that use <code>*<\/code> or <code>$<\/code>. Protego is Scrapy's default robots parser (<code>ROBOTSTXT_PARSER = \"scrapy.robotstxt.ProtegoRobotParser\"<\/code>) and works standalone via <code>Protego.parse(text)<\/code>; note that its <code>can_fetch(url, user_agent)<\/code> takes its arguments in the opposite order from the stdlib.<\/p>\n<h2>Scrapy: AutoThrottle and friends<\/h2>\n<p>Scrapy already implements most of this. Requests are grouped into download slots (by domain by default), concurrency is capped per slot, and the <a href=\"https:\/\/docs.scrapy.org\/en\/latest\/topics\/autothrottle.html\">AutoThrottle extension<\/a> adjusts each slot's delay from observed latency: the target delay is <code>latency \/ AUTOTHROTTLE_TARGET_CONCURRENCY<\/code>, the next delay is the average of the previous and target delays, and non-200 responses are not allowed to <em>decrease<\/em> the delay. A reasonable starting point:<\/p>\n<pre><code># settings.py\nROBOTSTXT_OBEY = True\nUSER_AGENT = \"ExampleBot\/0.1 (+https:\/\/www.example.com\/bot.html)\"\n\nCONCURRENT_REQUESTS = 64            # global cap (default 16)\nCONCURRENT_REQUESTS_PER_DOMAIN = 4  # per-slot cap (default 8)\nDOWNLOAD_DELAY = 0.5                # floor; AutoThrottle never goes below it\n\nAUTOTHROTTLE_ENABLED = True         # default False\nAUTOTHROTTLE_START_DELAY = 5.0      # default 5.0\nAUTOTHROTTLE_MAX_DELAY = 60.0       # default 60.0\nAUTOTHROTTLE_TARGET_CONCURRENCY = 2.0  # default 1.0; a target, not a hard limit\nAUTOTHROTTLE_DEBUG = False          # True logs per-response throttle stats\n\nRETRY_ENABLED = True\nRETRY_TIMES = 3                     # default 2 (3 requests in total)\nRETRY_HTTP_CODES = [408, 429, 500, 502, 503, 504, 522, 524]<\/code><\/pre>\n<p>The defaults in the comments come from Scrapy's <a href=\"https:\/\/github.com\/scrapy\/scrapy\/blob\/master\/scrapy\/settings\/default_settings.py\"><code>default_settings.py<\/code><\/a>; note that <code>scrapy startproject<\/code> generates a more conservative template with <code>ROBOTSTXT_OBEY = True<\/code>, <code>CONCURRENT_REQUESTS_PER_DOMAIN = 1<\/code> and <code>DOWNLOAD_DELAY = 1<\/code>. Two caveats. The built-in <code>RetryMiddleware<\/code> re-schedules failed requests with lower priority (<code>RETRY_PRIORITY_ADJUST<\/code>, default -1); it is not an exponential-backoff implementation and you should not assume it reads <code>Retry-After<\/code>, so let AutoThrottle and a custom middleware handle pacing on <code>429<\/code>. And for specific sites you can pin a slot's <code>delay<\/code> through <code>DOWNLOAD_SLOTS<\/code>, or set the <code>autothrottle_dont_adjust_delay<\/code> request meta key (Scrapy 2.12+) to keep particular requests from moving a slot's delay.<\/p>\n<h2>Proxies do not replace rate limiting<\/h2>\n<p>Spreading traffic over many exit IPs does not change how much load a host receives. Key your budgets on the target host, not the exit IP, or adding proxies will multiply the rate each site sees. Proxies help on your side: egress capacity, regional access, and not funnelling everything through one IP. The trade-offs between proxy types are covered in <a href=\"https:\/\/globalcorpora.com\/blog\/datacenter-vs-residential-vs-isp-proxies-for-scraping\/\">datacenter vs residential vs ISP proxies<\/a>, and for the YouTube-specific case of rate limiting and bot checks, see <a href=\"https:\/\/globalcorpora.com\/blog\/why-youtube-downloads-are-slow\/\">why YouTube downloads are slow<\/a> and <a href=\"https:\/\/globalcorpora.com\/blog\/yt-dlp-sign-in-to-confirm-not-a-bot\/\">fixing \"Sign in to confirm you're not a bot\"<\/a>.<\/p>\n<h2>Compliance<\/h2>\n<p>Rate limits and robots.txt are operational signals, not legal permission. Site terms of service often restrict automated access (YouTube's Terms of Service, for example, restrict automated access and downloading). Copyright and licensing apply to what you collect and redistribute, privacy law such as GDPR applies to personal data including faces and voices, and you should honor takedown and opt-out requests. Get legal review for your jurisdiction and use case.<\/p>\n<h2>If you would rather not run the egress yourself<\/h2>\n<p>If your crawler is well behaved but your bottleneck is egress capacity, Corpora offers <a href=\"https:\/\/globalcorpora.com\/#pricing\">dedicated 1\u201340 Gbps unmetered proxy capacity<\/a> (HTTP(S) and SOCKS5, rotating IP pools, multi-region) that plugs into custom crawlers, and a separate <a href=\"https:\/\/proxy.globalcorpora.com\/\">residential proxy<\/a> product. Either way, keep the per-host limits above in place.<\/p>\n<h2>FAQ<\/h2>\n<h3>How do I handle HTTP 429 Too Many Requests in a scraper?<\/h3>\n<p>Stop sending to that host, wait for the <code>Retry-After<\/code> value if present (it can be seconds or an HTTP date), otherwise use jittered exponential backoff, and lower that host's request rate for the rest of the crawl. Apply the pause to all workers targeting the host, not just the request that got the <code>429<\/code>.<\/p>\n<h3>What is a good rate limit for web scraping?<\/h3>\n<p>There is no universal number. Start conservatively (one or two concurrent requests per host with a delay of around a second, which is roughly what Scrapy's project template does), honor <code>Crawl-delay<\/code> if present, and let an adaptive scheme such as AIMD or AutoThrottle raise the rate only while latency and error rates stay healthy.<\/p>\n<h3>Should I use full jitter or decorrelated jitter?<\/h3>\n<p>Either beats no jitter. Full jitter is stateless and the common default; decorrelated jitter keeps state between attempts. Cap the per-attempt delay and the number of attempts either way.<\/p>\n<h3>Is Crawl-delay part of the robots.txt standard?<\/h3>\n<p>No. RFC 9309 defines only <code>user-agent<\/code>, <code>allow<\/code> and <code>disallow<\/code> (and allows other records such as <code>Sitemap<\/code> to be interpreted). <code>Crawl-delay<\/code> is a non-standard extension; Google ignores it, other crawlers honor it. Treating it as a minimum interval per host is the cautious choice.<\/p>\n<h3>Which errors should a crawler not retry?<\/h3>\n<p>Permanent client errors: <code>400<\/code>, <code>401<\/code>, <code>404<\/code>, <code>405<\/code>, <code>410<\/code>, <code>422<\/code>. Treat <code>403<\/code> as a signal to slow down and investigate rather than retry immediately, and avoid automatic retries of non-idempotent requests such as <code>POST<\/code> unless you know they are safe to repeat.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Web scraping rate limiting done right: per-host token buckets, 429\/503 Retry-After handling, jittered backoff, circuit breakers and Scrapy AutoThrottle.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[94,96,13,97,95,92],"class_list":["post-36","post","type-post","status-publish","format-standard","hentry","category-guides","tag-exponential-backoff","tag-httpx","tag-rate-limiting","tag-robots-txt","tag-scrapy","tag-web-scraping"],"_links":{"self":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/posts\/36","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/comments?post=36"}],"version-history":[{"count":0,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/posts\/36\/revisions"}],"wp:attachment":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/media?parent=36"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/categories?post=36"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/tags?post=36"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}