Tag: httpx

  • Web Scraping Rate Limiting: Retries, Backoff and 429 Handling

    A crawler that works at 10 requests per second often falls apart at 1,000: hosts start answering 429 and 503, retries pile up on top of the original load, and a few slow sites quietly eat your whole worker pool. This guide covers web scraping rate limiting from the protocol up: what the status codes and Retry-After actually mean, how to budget requests per host, how to back off without synchronized retry storms, and how to wire it all together in Python with httpx, tenacity or Scrapy.

    TL;DR

    • Limit per host, not globally. A global cap of 500 in-flight requests says nothing about how hard you hit any single site. Use a per-host concurrency cap plus a per-host token bucket.
    • Treat 429 and 503 as "slow down", not "try again now". Honor Retry-After when present (seconds or an HTTP date) and apply it to the whole host, not just the one URL.
    • Back off exponentially with jitter. Plain exponential backoff synchronizes clients; full jitter (random(0, min(cap, base * 2^n))) spreads them out.
    • Only retry what can succeed later: timeouts, connection errors, 408, 429, 500, 502, 503, 504. Never blindly retry 404 or other 4xx, and be careful with non-idempotent methods.
    • Add a per-host circuit breaker so one failing site cannot consume your retry budget and workers.
    • Read robots.txt (RFC 9309). Crawl-delay is not part of the standard; treat it as a hint you choose to honor.

    What the server is telling you: 429, 503 and Retry-After

    429 Too Many Requests is defined in RFC 6585: the user "has sent too many requests in a given amount of time". The response may include a Retry-After header. The RFC deliberately does not define how the server counts requests or identifies you; it could be per resource, per server or across a fleet, keyed on IP, credentials or a cookie. So a 429 on one path is evidence about the whole host until proven otherwise.

    The same RFC notes that servers are not required to send 429 at all. Many sites drop connections, return 503 or 403, or serve an error page with status 200. React to all of these, not just the polite one.

    503 Service Unavailable and the Retry-After field are defined in RFC 9110 (HTTP Semantics). Retry-After takes one of two forms:

    Retry-After: 120
    Retry-After: Fri, 13 Nov 2026 23:59:59 GMT

    The first is a delay in seconds, the second an HTTP date. Parse both. A robust parser treats an unparseable value as absent and falls back to your own backoff. Also put an upper bound on how long you will sleep in-process: a Retry-After: 86400 means "come back tomorrow", which is a scheduling decision (requeue the host), not a sleep().

    Budget requests per host

    Two independent controls, applied per host (scheme plus hostname plus port is a reasonable key):

    Control Limits Protects against
    Concurrency cap (semaphore) Requests in flight at once Overloading slow hosts; one host hogging connections
    Token bucket Average request rate, with a bounded burst Exceeding a rate the host tolerates, even when responses are fast

    You need both. A concurrency cap alone lets a fast host absorb a high request rate: four connections to a server that answers in 20 ms is 4 / 0.02 = 200 requests per second. A rate limit alone lets a slow host pile up open connections. The Scrapy AutoThrottle docs point out a related trap: error responses are often returned faster than normal ones, so a fixed concurrency cap makes you send faster exactly when the server starts failing.

    A token bucket has two parameters: a refill rate r (tokens per second) and a capacity b (maximum burst). Each request spends one token. Over any window of t seconds you send at most b + r × t requests. With r = 2 and b = 4, a minute allows at most 4 + 2 × 60 = 124 requests to that host.

    Start conservative and adapt. A common pattern is AIMD (additive increase, multiplicative decrease, the same idea TCP uses): on every success nudge the rate up a little, on every throttle signal cut it in half. That converges on what the host tolerates without you guessing a number.

    Exponential backoff with jitter

    When a request fails with a retryable error, wait before retrying, and wait longer each time. The canonical reference is the AWS Architecture Blog post Exponential Backoff And Jitter, whose simulator code is published on GitHub. Its variants, with attempt number n, base delay base and ceiling cap:

    Strategy Sleep before retry
    Exponential min(cap, base * 2**n)
    Equal jitter v/2 + uniform(0, v/2), where v = min(cap, base * 2**n)
    Full jitter uniform(0, min(cap, base * 2**n))
    Decorrelated jitter sleep = min(cap, uniform(base, sleep * 3))

    Why jitter matters for crawlers: if 200 workers all hit a 503 at the same moment and all wait exactly 2, 4, 8 seconds, they come back in synchronized waves and re-trigger the overload. Randomizing the wait spreads them across the window. That is the point of the post's simulations, which compare total calls and completion time across the variants. Full jitter is the simplest to implement and the usual default.

    Two rules on top of the formula. First, if the server sent Retry-After, wait at least that long: max(jittered_backoff, retry_after). Second, cap total attempts (5 to 7 is typical) and total elapsed time, then give up and record the failure for a later pass.

    What to retry, and what not to

    Outcome Retry? Notes
    Connect timeout, read timeout, connection reset, DNS hiccup Yes, with backoff Count toward the host's circuit breaker
    429 Yes, after Retry-After or backoff Also lower that host's rate
    503 Yes, after Retry-After or backoff Often maintenance or overload; slow the host down
    500, 502, 504, 408 Yes, a few times Persistent 500 on one URL is usually a bug on their side; stop after a few
    404, 410 No Record as gone
    400, 401, 405, 422 No Your request is wrong; retrying will not fix it
    403 Not immediately Often a block or bot check; treat as a throttle signal for the host and investigate
    200 with a captcha or "rate limited" body Treat as 429 Detect with a content check; it will poison your dataset otherwise

    Scrapy's default retry list is a useful cross-check: RETRY_HTTP_CODES defaults to [408, 429, 500, 502, 503, 504, 522, 524] (522 and 524 are Cloudflare-specific codes).

    Idempotency

    RFC 9110 defines PUT, DELETE and the safe methods (GET, HEAD, OPTIONS, TRACE) as idempotent: repeating them has the same intended effect as sending them once. Crawlers are almost entirely GET, so retries are safe. The exception is POST-driven endpoints (search forms, GraphQL, internal APIs). A timed-out POST may have been processed; only retry it automatically if you know the operation is a pure read or the API supports an idempotency key.

    A per-host limiter in Python (httpx + asyncio)

    The following is a complete, dependency-light implementation: per-host semaphore, token bucket with AIMD, Retry-After parsing, full-jitter backoff and a circuit breaker. It needs only pip install httpx.

    import asyncio
    import random
    import time
    from datetime import datetime, timezone
    from email.utils import parsedate_to_datetime
    from urllib.parse import urlsplit
    
    import httpx
    
    RETRYABLE_STATUS = {408, 429, 500, 502, 503, 504}
    THROTTLE_STATUS = {429, 503}
    RETRYABLE_EXC = (httpx.TimeoutException, httpx.NetworkError, httpx.RemoteProtocolError)
    MAX_INLINE_WAIT = 300.0  # longer Retry-After values: defer the URL instead of sleeping
    
    
    class GiveUp(Exception):
        pass
    
    
    class Deferred(Exception):
        pass
    
    
    def parse_retry_after(value):
        """Return seconds to wait, or None. Accepts delay-seconds or an HTTP-date."""
        if not value:
            return None
        value = value.strip()
        if value.isdigit():
            return float(value)
        try:
            dt = parsedate_to_datetime(value)
        except (TypeError, ValueError):
            return None
        if dt.tzinfo is None:
            dt = dt.replace(tzinfo=timezone.utc)
        return max(0.0, (dt - datetime.now(timezone.utc)).total_seconds())
    
    
    def full_jitter(attempt, base=1.0, cap=60.0):
        return random.uniform(0, min(cap, base * 2 ** attempt))
    
    
    class TokenBucket:
        def __init__(self, rate, burst, min_rate=0.1, max_rate=10.0):
            self.rate, self.capacity = rate, burst
            self.min_rate, self.max_rate = min_rate, max_rate
            self.tokens = float(burst)
            self.updated = time.monotonic()
            self.lock = asyncio.Lock()
    
        async def acquire(self):
            async with self.lock:
                while True:
                    now = time.monotonic()
                    self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
                    self.updated = now
                    if self.tokens >= 1:
                        self.tokens -= 1
                        return
                    await asyncio.sleep((1 - self.tokens) / self.rate)
    
        def on_success(self):   # additive increase
            self.rate = min(self.max_rate, self.rate + 0.05)
    
        def on_throttle(self):  # multiplicative decrease
            self.rate = max(self.min_rate, self.rate / 2)
    
    
    class CircuitBreaker:
        def __init__(self, threshold=5, cooldown=60.0):
            self.threshold, self.cooldown = threshold, cooldown
            self.failures = 0
            self.opened_at = None
            self.probing = False
    
        def allow(self):
            if self.opened_at is None:
                return True
            if not self.probing and time.monotonic() - self.opened_at >= self.cooldown:
                self.probing = True  # half-open: let one request through
                return True
            return False
    
        def record_success(self):
            self.failures, self.opened_at, self.probing = 0, None, False
    
        def record_failure(self):
            self.failures += 1
            if self.probing or self.failures >= self.threshold:
                self.opened_at, self.probing = time.monotonic(), False
    
    
    class Host:
        def __init__(self, concurrency=4, rate=2.0, burst=4):
            self.sem = asyncio.Semaphore(concurrency)
            self.bucket = TokenBucket(rate, burst)
            self.breaker = CircuitBreaker()
            self.not_before = 0.0  # monotonic timestamp; host-wide pause
    
    
    hosts = {}
    
    
    def host_for(url):
        parts = urlsplit(url)
        key = f"{parts.scheme}://{parts.netloc.lower()}"
        if key not in hosts:
            hosts[key] = Host()
        return hosts[key]
    
    
    async def fetch(client, url, max_attempts=6):
        host = host_for(url)
        last_error = None
        for attempt in range(max_attempts):
            if not host.breaker.allow():
                raise Deferred(f"circuit open for {url}")
            pause = host.not_before - time.monotonic()
            if pause > 0:
                await asyncio.sleep(pause)
    
            async with host.sem:
                await host.bucket.acquire()
                try:
                    resp = await client.get(url)
                except RETRYABLE_EXC as exc:
                    host.breaker.record_failure()
                    last_error = repr(exc)
                    delay = full_jitter(attempt)
                else:
                    status = resp.status_code
                    if status < 400:
                        host.breaker.record_success()
                        host.bucket.on_success()
                        return resp
                    if status not in RETRYABLE_STATUS:
                        raise GiveUp(f"HTTP {status} for {url}")  # 404, 403, 400 ...
                    host.breaker.record_failure()
                    last_error = f"HTTP {status}"
                    delay = full_jitter(attempt)
                    if status in THROTTLE_STATUS:
                        host.bucket.on_throttle()
                    retry_after = parse_retry_after(resp.headers.get("Retry-After"))
                    if retry_after is not None:
                        host.not_before = max(host.not_before, time.monotonic() + retry_after)
                        if retry_after > MAX_INLINE_WAIT:
                            raise Deferred(f"Retry-After {retry_after:.0f}s for {url}")
                        delay = max(delay, retry_after)
            await asyncio.sleep(delay)
        raise GiveUp(f"{url}: gave up after {max_attempts} attempts ({last_error})")
    
    
    async def main(urls):
        limits = httpx.Limits(max_connections=200, max_keepalive_connections=50)
        timeout = httpx.Timeout(20.0, connect=10.0)
        headers = {"User-Agent": "ExampleBot/0.1 (+https://www.example.com/bot.html)"}
        async with httpx.AsyncClient(limits=limits, timeout=timeout,
                                     headers=headers, follow_redirects=True) as client:
            results = await asyncio.gather(*(fetch(client, u) for u in urls),
                                           return_exceptions=True)
        for url, r in zip(urls, results):
            print(url, r.status_code if isinstance(r, httpx.Response) else repr(r))
    
    
    if __name__ == "__main__":
        asyncio.run(main(["https://example.com/", "https://example.org/"]))

    Notes on the design:

    • httpx.Limits caps the connection pool globally (the defaults are 100 connections and 20 keep-alive, per the httpx resource limits docs). It is not a per-host limiter; the semaphore and bucket are.
    • Retry-After sets not_before for the whole host, so every other task targeting that host pauses too, instead of each one discovering the limit separately.
    • The sleep before a retry happens outside the semaphore, so a backing-off request does not hold one of the host's concurrency slots.
    • For millions of URLs, replace asyncio.gather over everything with a bounded queue and a fixed pool of worker tasks, and persist Deferred and GiveUp outcomes so a later pass can pick them up. The same structure works for browser-based crawls; see Playwright headless browser scraping at scale.

    The same retry policy with tenacity

    If you prefer a library for the retry loop, tenacity supports async functions directly. wait_random_exponential is full jitter ("randomly wait up to 2^x * multiplier seconds"), and any callable that takes retry_state can serve as a wait strategy, which is how you layer Retry-After on top:

    import httpx
    from tenacity import retry, retry_if_exception, stop_after_attempt, wait_random_exponential
    
    backoff = wait_random_exponential(multiplier=1, max=60)
    
    
    def is_retryable(exc):
        if isinstance(exc, httpx.HTTPStatusError):
            return exc.response.status_code in {408, 429, 500, 502, 503, 504}
        return isinstance(exc, (httpx.TimeoutException, httpx.NetworkError))
    
    
    def wait_with_retry_after(retry_state):
        fallback = backoff(retry_state)
        exc = retry_state.outcome.exception()
        if isinstance(exc, httpx.HTTPStatusError):
            ra = parse_retry_after(exc.response.headers.get("Retry-After"))  # from the example above
            if ra is not None:
                return max(fallback, min(ra, 300))
        return fallback
    
    
    @retry(retry=retry_if_exception(is_retryable), wait=wait_with_retry_after,
           stop=stop_after_attempt(6), reraise=True)
    async def get(client, url):
        resp = await client.get(url)
        resp.raise_for_status()
        return resp

    Tenacity only handles the retry loop; it knows nothing about hosts, so keep the per-host semaphore and bucket around the call.

    Circuit breakers per host

    Backoff handles a single request. A circuit breaker handles a host that is down. Without one, a site that times out on every request can tie up workers for attempts × timeout per URL, across thousands of queued URLs. The breaker in the code above has three states:

    • Closed: requests flow normally; consecutive failures are counted.
    • Open: after threshold consecutive failures, all requests to the host are deferred for a cooldown period.
    • Half-open: after the cooldown, one probe request goes through. Success closes the breaker; failure reopens it.

    Requeue deferred URLs rather than dropping them. If many breakers open at once, the problem is probably on your side (DNS, proxy, egress).

    robots.txt and Crawl-delay

    RFC 9309 standardized the Robots Exclusion Protocol in 2022. Points that matter for a crawler's control loop:

    • The standard defines user-agent, allow and disallow, plus the * and $ special characters. The longest (most specific) match wins.
    • If robots.txt returns a 4xx ("unavailable"), the crawler may access anything. If it is unreachable because of a 5xx or network error, the crawler must assume complete disallow.
    • Crawlers should not use a cached copy for more than 24 hours, unless robots.txt is unreachable.
    • The rules "are not a form of access authorization". They are a request from the operator.

    Crawl-delay is not in RFC 9309. It is a non-standard extension that some crawlers honor and others ignore; Google states that its crawlers do not support crawl-delay. Even so, it is the site operator telling you what rate they consider acceptable, so a sensible policy is to treat it as a floor on the per-host interval: set the bucket rate to at most 1 / crawl_delay.

    Python's standard library exposes it through urllib.robotparser, with can_fetch(), crawl_delay() and request_rate(). Fetch the file with your own client so you control status-code handling, then feed the lines to the parser:

    from urllib.robotparser import RobotFileParser
    
    import httpx
    
    
    async def load_robots(client, origin):
        rp = RobotFileParser()
        try:
            r = await client.get(f"{origin}/robots.txt")
        except httpx.TransportError:
            rp.parse(["User-agent: *", "Disallow: /"])   # unreachable: assume disallow
            return rp
        if r.status_code >= 500:
            rp.parse(["User-agent: *", "Disallow: /"])   # unreachable: assume disallow
        elif r.status_code >= 400:
            rp.parse(["User-agent: *", "Disallow:"])     # unavailable: no restrictions
        else:
            rp.parse(r.text.splitlines())
        return rp
    
    # rp.can_fetch("ExampleBot", url)  -> bool
    # rp.crawl_delay("ExampleBot")     -> seconds or None

    Check your Python version: according to the Protego README, urllib.robotparser only gained RFC 9309 wildcards and longest-match precedence during the 3.14 series, so older interpreters can misjudge rules that use * or $. Protego is Scrapy's default robots parser (ROBOTSTXT_PARSER = "scrapy.robotstxt.ProtegoRobotParser") and works standalone via Protego.parse(text); note that its can_fetch(url, user_agent) takes its arguments in the opposite order from the stdlib.

    Scrapy: AutoThrottle and friends

    Scrapy already implements most of this. Requests are grouped into download slots (by domain by default), concurrency is capped per slot, and the AutoThrottle extension adjusts each slot's delay from observed latency: the target delay is latency / AUTOTHROTTLE_TARGET_CONCURRENCY, the next delay is the average of the previous and target delays, and non-200 responses are not allowed to decrease the delay. A reasonable starting point:

    # settings.py
    ROBOTSTXT_OBEY = True
    USER_AGENT = "ExampleBot/0.1 (+https://www.example.com/bot.html)"
    
    CONCURRENT_REQUESTS = 64            # global cap (default 16)
    CONCURRENT_REQUESTS_PER_DOMAIN = 4  # per-slot cap (default 8)
    DOWNLOAD_DELAY = 0.5                # floor; AutoThrottle never goes below it
    
    AUTOTHROTTLE_ENABLED = True         # default False
    AUTOTHROTTLE_START_DELAY = 5.0      # default 5.0
    AUTOTHROTTLE_MAX_DELAY = 60.0       # default 60.0
    AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0  # default 1.0; a target, not a hard limit
    AUTOTHROTTLE_DEBUG = False          # True logs per-response throttle stats
    
    RETRY_ENABLED = True
    RETRY_TIMES = 3                     # default 2 (3 requests in total)
    RETRY_HTTP_CODES = [408, 429, 500, 502, 503, 504, 522, 524]

    The defaults in the comments come from Scrapy's default_settings.py; note that scrapy startproject generates a more conservative template with ROBOTSTXT_OBEY = True, CONCURRENT_REQUESTS_PER_DOMAIN = 1 and DOWNLOAD_DELAY = 1. Two caveats. The built-in RetryMiddleware re-schedules failed requests with lower priority (RETRY_PRIORITY_ADJUST, default -1); it is not an exponential-backoff implementation and you should not assume it reads Retry-After, so let AutoThrottle and a custom middleware handle pacing on 429. And for specific sites you can pin a slot's delay through DOWNLOAD_SLOTS, or set the autothrottle_dont_adjust_delay request meta key (Scrapy 2.12+) to keep particular requests from moving a slot's delay.

    Proxies do not replace rate limiting

    Spreading traffic over many exit IPs does not change how much load a host receives. Key your budgets on the target host, not the exit IP, or adding proxies will multiply the rate each site sees. Proxies help on your side: egress capacity, regional access, and not funnelling everything through one IP. The trade-offs between proxy types are covered in datacenter vs residential vs ISP proxies, and for the YouTube-specific case of rate limiting and bot checks, see why YouTube downloads are slow and fixing "Sign in to confirm you're not a bot".

    Compliance

    Rate limits and robots.txt are operational signals, not legal permission. Site terms of service often restrict automated access (YouTube's Terms of Service, for example, restrict automated access and downloading). Copyright and licensing apply to what you collect and redistribute, privacy law such as GDPR applies to personal data including faces and voices, and you should honor takedown and opt-out requests. Get legal review for your jurisdiction and use case.

    If you would rather not run the egress yourself

    If your crawler is well behaved but your bottleneck is egress capacity, Corpora offers dedicated 1–40 Gbps unmetered proxy capacity (HTTP(S) and SOCKS5, rotating IP pools, multi-region) that plugs into custom crawlers, and a separate residential proxy product. Either way, keep the per-host limits above in place.

    FAQ

    How do I handle HTTP 429 Too Many Requests in a scraper?

    Stop sending to that host, wait for the Retry-After value if present (it can be seconds or an HTTP date), otherwise use jittered exponential backoff, and lower that host's request rate for the rest of the crawl. Apply the pause to all workers targeting the host, not just the request that got the 429.

    What is a good rate limit for web scraping?

    There is no universal number. Start conservatively (one or two concurrent requests per host with a delay of around a second, which is roughly what Scrapy's project template does), honor Crawl-delay if present, and let an adaptive scheme such as AIMD or AutoThrottle raise the rate only while latency and error rates stay healthy.

    Should I use full jitter or decorrelated jitter?

    Either beats no jitter. Full jitter is stateless and the common default; decorrelated jitter keeps state between attempts. Cap the per-attempt delay and the number of attempts either way.

    Is Crawl-delay part of the robots.txt standard?

    No. RFC 9309 defines only user-agent, allow and disallow (and allows other records such as Sitemap to be interpreted). Crawl-delay is a non-standard extension; Google ignores it, other crawlers honor it. Treating it as a minimum interval per host is the cautious choice.

    Which errors should a crawler not retry?

    Permanent client errors: 400, 401, 404, 405, 410, 422. Treat 403 as a signal to slow down and investigate rather than retry immediately, and avoid automatic retries of non-idempotent requests such as POST unless you know they are safe to repeat.