Tag: Scrapy

  • URL Frontier Design: Dedup, Politeness and Recrawls at Scale

    The fetchers in a large crawler are the easy part. The hard part is the URL frontier: the component that decides what to fetch next, avoids duplicate fetches, stays polite to every host and survives crawler traps. This guide covers a practical URL frontier design: canonicalization, seen-URL sets with real Bloom filter math, Mercator-style per-host queues, sitemaps, conditional recrawls, Scrapy's built-in pieces and trap defenses, with runnable Python.

    TL;DR

    • Canonicalize before you dedupe (RFC 3986 section 6): lowercase scheme and host, drop default ports and fragments, remove dot segments, normalize percent-encoding. Sorting and stripping query parameters are heuristics, not standards.
    • Use the canonical form as a lookup key, and fetch the original URL.
    • Size the seen-set with arithmetic: a Bloom filter needs -ln(p) / (ln 2)^2 bits per URL, about 9.6 bits at 1% false positives. A false positive means a URL is silently skipped.
    • Separate priority from politeness: front queues for priority, one back queue per host, and a heap of next-allowed fetch times (the Mercator design).
    • Recrawl with validators (If-None-Match, If-Modified-Since) and budget every host so traps cannot eat the crawl.

    What a URL frontier does

    The standard reference is the section The URL frontier in chapter 20 of Manning, Raghavan and Schütze's Introduction to Information Retrieval, which describes the frontier of the Mercator crawler (Heydon and Najork, "Mercator: A scalable, extensible Web crawler", World Wide Web 2(4), 1999). It balances two goals:

    • Priority: important pages, and pages that change often, get fetched first and refetched more often.
    • Politeness: most links point to the same host, so a plain priority queue produces bursts against one server.

    In front of the queues sit a canonicalizer and a seen-URL set; behind them, a URL database with fetch history for recrawls.

    URL canonicalization: safe rules and guesses

    RFC 3986 section 6 notes that extensive normalization "is often used by spiders and indexing engines to prune a search space", and that comparison methods are "designed to minimize false negatives while strictly avoiding false positives". Keep that mindset: every rule merges more URLs, and a wrong merge is a page you never fetch.

    Safe by the spec

    • Case: scheme and host are case-insensitive; lowercase them. Uppercase hex digits in percent-encodings (%3a to %3A). Path and query are case-sensitive.
    • Percent-encoding: decode encoded unreserved characters (letters, digits, -._~), so %7Euser equals ~user.
    • Dot segments: resolve . and .. with remove_dot_segments (section 5.2.4).
    • Port and empty path: RFC 9110 section 4.2.3 says the normal form omits the default port (80 or 443) and uses / for an empty path.
    • Fragments: RFC 3986 section 6.1 says fragments should be excluded when comparing URIs to select a network action such as retrieval.

    Heuristics with caveats

    • Sorting query parameters suits most sites but no standard guarantees it. Scrapy's w3lib.url.canonicalize_url sorts by key, then value, and drops fragments by default.
    • Stripping parameters: utm_*, gclid and fbclid are usually safe to drop. Session and view parameters are site-specific; use per-host rules.
    • Trailing slash, www., index.html, http versus https are different URLs. Let the server's redirects tell you, and mark both source and target as seen.
    • Hash routes in single-page apps may matter if you render pages; see Playwright headless browser scraping at scale.

    A stdlib-only canonicalizer with the safe rules plus sorted, de-tracked queries:

    import re
    from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
    
    DEFAULT_PORTS = {"http": 80, "https": 443}
    UNRESERVED = set("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~")
    TRACKING = re.compile(r"^(utm_[a-z]+|gclid|fbclid|msclkid)$", re.IGNORECASE)
    PCT = re.compile(r"%([0-9A-Fa-f]{2})")
    
    
    def normalize_pct(s):
        def fix(m):
            ch = chr(int(m.group(1), 16))
            return ch if ch in UNRESERVED else "%" + m.group(1).upper()
        return PCT.sub(fix, s)
    
    
    def remove_dot_segments(path):  # segment-stack form of RFC 3986 5.2.4
        segments = path.split("/")
        out = []
        for seg in segments:
            if seg == ".":
                continue
            if seg == "..":
                if len(out) > 1:
                    out.pop()
                continue
            out.append(seg)
        if segments[-1] in (".", ".."):
            out.append("")  # "/a/b/.." -> "/a/"
        return "/".join(out)
    
    
    def canonicalize(url, drop=TRACKING, sort_query=True):
        parts = urlsplit(url.strip())
        scheme = parts.scheme.lower()
        if scheme not in DEFAULT_PORTS:
            raise ValueError(f"unsupported scheme: {scheme!r}")
        host = parts.hostname  # lowercased, userinfo removed
        if not host:
            raise ValueError(f"no host in {url!r}")
        if ":" in host:
            host = f"[{host}]"  # IPv6 literal
        port = parts.port  # ValueError on a malformed port
        netloc = host if port in (None, DEFAULT_PORTS[scheme]) else f"{host}:{port}"
        path = remove_dot_segments(normalize_pct(parts.path)) or "/"
        pairs = [(k, v) for k, v in parse_qsl(parts.query, keep_blank_values=True)
                 if not drop.match(k)]
        if sort_query:
            pairs.sort()
        return urlunsplit((scheme, netloc, path, urlencode(pairs), ""))
    
    
    print(canonicalize("HTTP://www.Example.com:80/a/./b/../c/%7euser?utm_source=x&b=2&a=1#top"))
    # http://www.example.com/a/c/~user?a=1&b=2

    Userinfo is dropped (RFC 9110 section 4.2.4 deprecates it). Because urlencode re-encodes the query, fetch the original URL, not the key; w3lib's docstring gives the same warning.

    Seen-URL sets: Bloom filters and exact stores

    A Bloom filter (Bloom, "Space/Time Trade-offs in Hash Coding with Allowable Errors", CACM 1970, which the Mercator paper cites) stores set membership in m bits with k hash functions. With n items inserted:

    false positive rate     p ≈ (1 - e^(-k*n/m))^k
    optimal hash count      k = (m/n) * ln 2
    bits for a target p     m = -n * ln(p) / (ln 2)^2
    Target p Bits per URL Memory for n = 10^9 k
    1% 4.605 / 0.4805 ≈ 9.59 ≈ 1.2 GB 7
    0.1% 6.908 / 0.4805 ≈ 14.38 ≈ 1.8 GB 10

    An exact set of 64-bit URL hashes needs 8 bytes × 10^9 = 8 GB before index overhead, and the birthday bound puts expected colliding pairs at about n² / 2^65 ≈ 0.03. So the Bloom filter saves roughly 4–7× memory for a known loss rate.

    Know which way the error goes: a Bloom filter sometimes says "seen" for a new URL, so about p of new URLs are silently skipped. Size it for the final n, because the rate climbs fast past it, and a plain Bloom filter cannot delete or resize.

    import hashlib
    import math
    
    
    class BloomFilter:
        def __init__(self, n, p):
            self.m = math.ceil(-n * math.log(p) / math.log(2) ** 2)  # bits
            self.k = max(1, round(self.m / n * math.log(2)))          # hash functions
            self.bits = bytearray((self.m + 7) // 8)
    
        def _positions(self, item):
            d = hashlib.blake2b(item.encode(), digest_size=16).digest()
            h1 = int.from_bytes(d[:8], "little")
            h2 = int.from_bytes(d[8:], "little") | 1
            return [(h1 + i * h2) % self.m for i in range(self.k)]  # double hashing
    
        def add(self, item):
            """Set the item's bits; return True if it was (probably) present already."""
            present = True
            for pos in self._positions(item):
                byte, bit = divmod(pos, 8)
                if not (self.bits[byte] >> bit) & 1:
                    present = False
                    self.bits[byte] |= 1 << bit
            return present

    In our check, probing 100,000 new URLs against BloomFilter(100_000, 0.01) filled with 100,000 others gave about 1.0% false positives, as predicted.

    Exact stores

    • RocksDB or another embedded key-value store: key = hash of the canonical URL, value = fetch state. It doubles as the recrawl database; a Bloom filter in RAM in front keeps most lookups off disk.
    • Redis: SADD on a set of hashes is an atomic "add and tell me if it was new" across workers. Redis also has Bloom filters (BF.RESERVE, BF.ADD) and cuckoo filters (CF.*, which support deletion). It is all RAM, so the memory math still applies.
    • Distributed crawls: partition by host so each node owns its hosts' state.

    URL dedup is not content dedup: mirrors and parameter variants serve the same page under different URLs. Catch those after fetching; see MinHash deduplication for LLM training data.

    Per-host queues and politeness, Mercator style

    In the IR book's account of Mercator, a prioritizer assigns each URL to one of F front queues. Each back queue holds URLs from one host, and a heap holds the earliest time each back queue's host may be contacted again. A fetcher pops the heap, waits if needed, fetches one URL and pushes the host back with a new time. Two rules of thumb from the book: wait about ten times the last fetch duration before the next request to that host, and keep about three times as many back queues as crawler threads.

    The sketch below keeps that structure but uses a dictionary of per-host back queues instead of a fixed number of them. Put it in one file with the two blocks above.

    import heapq
    import random
    import time
    from collections import deque
    from urllib.parse import urlsplit
    
    
    class Frontier:
        def __init__(self, priorities=3, min_delay=1.0, delay_factor=10.0,
                     expected_urls=10_000_000, fp_rate=0.001):
            self.front = [deque() for _ in range(priorities)]  # 0 = highest priority
            self.back = {}         # host -> deque of URLs
            self.ready = []        # heap of (earliest_next_fetch, host)
            self.not_before = {}   # host -> earliest next fetch
            self.seen = BloomFilter(expected_urls, fp_rate)
            self.min_delay, self.delay_factor = min_delay, delay_factor
    
        def add(self, url, priority=1):
            if self.seen.add(canonicalize(url)):
                return False
            self.front[priority].append(url)
            return True
    
        def _refill(self, max_moves=1000):
            weights = [2 ** (len(self.front) - i) for i in range(len(self.front))]
            for _ in range(max_moves):
                nonempty = [i for i, q in enumerate(self.front) if q]
                if not nonempty:
                    return
                i = random.choices(nonempty, [weights[j] for j in nonempty])[0]
                url = self.front[i].popleft()
                host = urlsplit(url).hostname
                if host not in self.back:
                    self.back[host] = deque()
                    t = max(time.monotonic(), self.not_before.get(host, 0.0))
                    heapq.heappush(self.ready, (t, host))
                self.back[host].append(url)
    
        def next_url(self):
            if not self.ready:
                self._refill()
            if not self.ready:
                return None
            t, host = heapq.heappop(self.ready)
            return self.back[host].popleft(), host, max(0.0, t - time.monotonic())
    
        def done(self, host, fetch_seconds):
            t = time.monotonic() + max(self.min_delay, self.delay_factor * fetch_seconds)
            self.not_before[host] = t
            if self.back[host]:
                heapq.heappush(self.ready, (t, host))
            else:
                del self.back[host]
            self._refill(max_moves=100)
    
    
    f = Frontier(min_delay=0.2)
    for u in ["https://example.com/", "https://EXAMPLE.com:443/#top",
              "https://example.org/a?utm_source=news", "https://example.org/a",
              "https://example.org/b"]:
        print(u, "->", "queued" if f.add(u) else "duplicate")
    while (item := f.next_url()) is not None:
        url, host, wait = item
        time.sleep(wait)
        print(f"fetch {url} (waited {wait:.2f}s)")
        f.done(host, fetch_seconds=0.01)

    This queues three URLs, flags two variants as duplicates, and delays the second example.org fetch by 0.2 seconds. A host leaves the heap while its fetch is in flight, so there is one request per host at a time. Production versions keep most of each queue on disk (as the IR book notes) and add robots.txt, Retry-After and adaptive rates, covered in web scraping rate limiting, retries and backoff.

    Priority signals and sitemaps

    Useful priority signals: depth from seeds, the host's track record, URL patterns and sitemaps. The sitemaps.org protocol specifies:

    • At most 50,000 URLs and 50MB (52,428,800 bytes) uncompressed per sitemap; gzip is allowed. Index files may list up to 50,000 sitemaps with the same size limit, and carry a lastmod per sitemap so you can skip unchanged ones.
    • <loc> must be under 2,048 characters, and URLs must share the sitemap's host and protocol.
    • <changefreq> is "a hint and not a command"; <priority> (0.0–1.0, default 0.5) only ranks URLs within one site.
    • Sites can advertise sitemaps with a Sitemap: line in robots.txt.

    Use sitemaps for discovery and lastmod to order recrawls within a host.

    Recrawl scheduling with conditional requests

    A long-running frontier also schedules revisits. Store the ETag and Last-Modified from each response and send them back as If-None-Match and If-Modified-Since (RFC 9110 section 13). An unchanged page can then come back as 304 Not Modified with no content. RFC 9110 says a recipient must ignore If-Modified-Since when If-None-Match is present, so the ETag wins. Many servers send neither validator, so also compare your own content hash.

    import hashlib
    
    import httpx
    
    
    def revisit(client: httpx.Client, url, state):
        """Use httpx.Client(follow_redirects=True). state holds etag, last_modified, sha256, interval."""
        headers = {}
        if state.get("etag"):
            headers["If-None-Match"] = state["etag"]
        if state.get("last_modified"):
            headers["If-Modified-Since"] = state["last_modified"]
        r = client.get(url, headers=headers)
        interval = state.get("interval", 86_400)
        if r.status_code == 304:
            changed = False
        else:
            r.raise_for_status()
            digest = hashlib.sha256(r.content).hexdigest()
            changed = digest != state.get("sha256")
            state.update(sha256=digest, etag=r.headers.get("ETag"),
                         last_modified=r.headers.get("Last-Modified"))
        # heuristic: revisit changing pages sooner, stable ones later, within 1 h to 30 days
        state["interval"] = max(3_600, min(30 * 86_400, interval / 2 if changed else interval * 1.5))
        return changed

    The halve-or-grow rule is a heuristic; the IR book warns that spam pages can change on every fetch, so weigh quality too.

    Scrapy's built-in frontier

    Scrapy's scheduler and dupefilter form a small URL frontier. From the settings reference and jobs docs (Scrapy 2.20 at the time of writing):

    • DUPEFILTER_CLASS defaults to scrapy.dupefilters.RFPDupeFilter, which fingerprints the canonicalized URL, method and body and keeps fingerprints in memory. Scrapy 2.20 added scrapy.dupefilters.DiskDupeFilter, which keeps them in SQLite.
    • JOBDIR persists pending requests (requests.queue/), fingerprints (requests.seen) and spider state, so a cleanly stopped crawl resumes. Use one directory per job and the same Scrapy version.
    • SCHEDULER_PRIORITY_QUEUE defaults to DownloaderAwarePriorityQueue, which the docs recommend for crawling many domains in parallel.
    • JOBDIR_SYNC_EVERY (2.20+) also saves state during the crawl; surviving a hard kill needs the SQLite disk queue types (2.19+).
    # settings.py
    JOBDIR = "crawls/frontier-1"   # or: scrapy crawl myspider -s JOBDIR=crawls/frontier-1
    DUPEFILTER_CLASS = "scrapy.dupefilters.DiskDupeFilter"               # 2.20+
    JOBDIR_SYNC_EVERY = 1000                                              # 2.20+, default 0
    SCHEDULER_DISK_QUEUE = "scrapy.squeues.PickleLifoSQLiteQueue"         # 2.19+
    SCHEDULER_START_DISK_QUEUE = "scrapy.squeues.PickleFifoSQLiteQueue"   # 2.19+
    DEPTH_LIMIT = 20               # default 0 = unlimited
    URLLENGTH_LIMIT = 2083         # the default

    Requests with dont_filter=True bypass the dupefilter; unpicklable requests are lost on pause.

    Crawler traps

    Traps generate unique URLs without new content: endless calendars, faceted search, session IDs, relative-link bugs producing /a/b/a/b/a/b/. Dedup cannot catch them. Defenses, cheapest first:

    • Hard limits: depth, URL length, query parameter count.
    • Per-host budgets: cap URLs queued per host, and raise the cap only for hosts that keep yielding distinct, useful documents.
    • Pattern counters: replace digits and IDs with placeholders (/calendar/{n}/{n}) and cap URLs per pattern; cap distinct values per query key.
    • Repetition checks: reject paths whose segment sequence repeats.
    • Content feedback: demote patterns whose new URLs keep hashing to content you already have.

    For the full text pipeline, see the Common Crawl LLM pretraining data pipeline.

    Compliance

    The frontier is the natural place for robots.txt checks and opt-out lists. Site terms of service often restrict automated access (YouTube's Terms of Service, for example, restrict automated access and downloading). Copyright and licensing apply to what you collect and redistribute, privacy law such as GDPR applies to personal data including faces and voices, and you should honor takedown and opt-out requests. Get legal review for your jurisdiction and use case.

    If you would rather not run the egress yourself

    If your frontier is in good shape and egress capacity is the bottleneck, Corpora offers dedicated 1–40 Gbps unmetered proxy capacity (HTTP(S) and SOCKS5, works with custom crawlers) and a separate residential proxy product. Keep politeness keyed on the target host either way.

    FAQ

    What is a URL frontier?

    The structure that holds URLs waiting to be fetched and decides their order, combining deduplication, priority and per-host politeness.

    How big a Bloom filter do I need for a billion URLs?

    About 1.2 GB at 1% false positives (7 hashes) and 1.8 GB at 0.1% (10), from m = -n ln(p) / (ln 2)^2.

    Should I sort query parameters when normalizing URLs?

    For dedup keys, usually yes; Scrapy's canonicalize_url does. No standard guarantees it, so fetch the original URL and add per-host exceptions when order matters.

    Does Scrapy remember seen URLs between runs?

    Only with JOBDIR set; rerunning with the same JOBDIR restores fingerprints and pending requests.

  • Web Scraping Rate Limiting: Retries, Backoff and 429 Handling

    A crawler that works at 10 requests per second often falls apart at 1,000: hosts start answering 429 and 503, retries pile up on top of the original load, and a few slow sites quietly eat your whole worker pool. This guide covers web scraping rate limiting from the protocol up: what the status codes and Retry-After actually mean, how to budget requests per host, how to back off without synchronized retry storms, and how to wire it all together in Python with httpx, tenacity or Scrapy.

    TL;DR

    • Limit per host, not globally. A global cap of 500 in-flight requests says nothing about how hard you hit any single site. Use a per-host concurrency cap plus a per-host token bucket.
    • Treat 429 and 503 as "slow down", not "try again now". Honor Retry-After when present (seconds or an HTTP date) and apply it to the whole host, not just the one URL.
    • Back off exponentially with jitter. Plain exponential backoff synchronizes clients; full jitter (random(0, min(cap, base * 2^n))) spreads them out.
    • Only retry what can succeed later: timeouts, connection errors, 408, 429, 500, 502, 503, 504. Never blindly retry 404 or other 4xx, and be careful with non-idempotent methods.
    • Add a per-host circuit breaker so one failing site cannot consume your retry budget and workers.
    • Read robots.txt (RFC 9309). Crawl-delay is not part of the standard; treat it as a hint you choose to honor.

    What the server is telling you: 429, 503 and Retry-After

    429 Too Many Requests is defined in RFC 6585: the user "has sent too many requests in a given amount of time". The response may include a Retry-After header. The RFC deliberately does not define how the server counts requests or identifies you; it could be per resource, per server or across a fleet, keyed on IP, credentials or a cookie. So a 429 on one path is evidence about the whole host until proven otherwise.

    The same RFC notes that servers are not required to send 429 at all. Many sites drop connections, return 503 or 403, or serve an error page with status 200. React to all of these, not just the polite one.

    503 Service Unavailable and the Retry-After field are defined in RFC 9110 (HTTP Semantics). Retry-After takes one of two forms:

    Retry-After: 120
    Retry-After: Fri, 13 Nov 2026 23:59:59 GMT

    The first is a delay in seconds, the second an HTTP date. Parse both. A robust parser treats an unparseable value as absent and falls back to your own backoff. Also put an upper bound on how long you will sleep in-process: a Retry-After: 86400 means "come back tomorrow", which is a scheduling decision (requeue the host), not a sleep().

    Budget requests per host

    Two independent controls, applied per host (scheme plus hostname plus port is a reasonable key):

    Control Limits Protects against
    Concurrency cap (semaphore) Requests in flight at once Overloading slow hosts; one host hogging connections
    Token bucket Average request rate, with a bounded burst Exceeding a rate the host tolerates, even when responses are fast

    You need both. A concurrency cap alone lets a fast host absorb a high request rate: four connections to a server that answers in 20 ms is 4 / 0.02 = 200 requests per second. A rate limit alone lets a slow host pile up open connections. The Scrapy AutoThrottle docs point out a related trap: error responses are often returned faster than normal ones, so a fixed concurrency cap makes you send faster exactly when the server starts failing.

    A token bucket has two parameters: a refill rate r (tokens per second) and a capacity b (maximum burst). Each request spends one token. Over any window of t seconds you send at most b + r × t requests. With r = 2 and b = 4, a minute allows at most 4 + 2 × 60 = 124 requests to that host.

    Start conservative and adapt. A common pattern is AIMD (additive increase, multiplicative decrease, the same idea TCP uses): on every success nudge the rate up a little, on every throttle signal cut it in half. That converges on what the host tolerates without you guessing a number.

    Exponential backoff with jitter

    When a request fails with a retryable error, wait before retrying, and wait longer each time. The canonical reference is the AWS Architecture Blog post Exponential Backoff And Jitter, whose simulator code is published on GitHub. Its variants, with attempt number n, base delay base and ceiling cap:

    Strategy Sleep before retry
    Exponential min(cap, base * 2**n)
    Equal jitter v/2 + uniform(0, v/2), where v = min(cap, base * 2**n)
    Full jitter uniform(0, min(cap, base * 2**n))
    Decorrelated jitter sleep = min(cap, uniform(base, sleep * 3))

    Why jitter matters for crawlers: if 200 workers all hit a 503 at the same moment and all wait exactly 2, 4, 8 seconds, they come back in synchronized waves and re-trigger the overload. Randomizing the wait spreads them across the window. That is the point of the post's simulations, which compare total calls and completion time across the variants. Full jitter is the simplest to implement and the usual default.

    Two rules on top of the formula. First, if the server sent Retry-After, wait at least that long: max(jittered_backoff, retry_after). Second, cap total attempts (5 to 7 is typical) and total elapsed time, then give up and record the failure for a later pass.

    What to retry, and what not to

    Outcome Retry? Notes
    Connect timeout, read timeout, connection reset, DNS hiccup Yes, with backoff Count toward the host's circuit breaker
    429 Yes, after Retry-After or backoff Also lower that host's rate
    503 Yes, after Retry-After or backoff Often maintenance or overload; slow the host down
    500, 502, 504, 408 Yes, a few times Persistent 500 on one URL is usually a bug on their side; stop after a few
    404, 410 No Record as gone
    400, 401, 405, 422 No Your request is wrong; retrying will not fix it
    403 Not immediately Often a block or bot check; treat as a throttle signal for the host and investigate
    200 with a captcha or "rate limited" body Treat as 429 Detect with a content check; it will poison your dataset otherwise

    Scrapy's default retry list is a useful cross-check: RETRY_HTTP_CODES defaults to [408, 429, 500, 502, 503, 504, 522, 524] (522 and 524 are Cloudflare-specific codes).

    Idempotency

    RFC 9110 defines PUT, DELETE and the safe methods (GET, HEAD, OPTIONS, TRACE) as idempotent: repeating them has the same intended effect as sending them once. Crawlers are almost entirely GET, so retries are safe. The exception is POST-driven endpoints (search forms, GraphQL, internal APIs). A timed-out POST may have been processed; only retry it automatically if you know the operation is a pure read or the API supports an idempotency key.

    A per-host limiter in Python (httpx + asyncio)

    The following is a complete, dependency-light implementation: per-host semaphore, token bucket with AIMD, Retry-After parsing, full-jitter backoff and a circuit breaker. It needs only pip install httpx.

    import asyncio
    import random
    import time
    from datetime import datetime, timezone
    from email.utils import parsedate_to_datetime
    from urllib.parse import urlsplit
    
    import httpx
    
    RETRYABLE_STATUS = {408, 429, 500, 502, 503, 504}
    THROTTLE_STATUS = {429, 503}
    RETRYABLE_EXC = (httpx.TimeoutException, httpx.NetworkError, httpx.RemoteProtocolError)
    MAX_INLINE_WAIT = 300.0  # longer Retry-After values: defer the URL instead of sleeping
    
    
    class GiveUp(Exception):
        pass
    
    
    class Deferred(Exception):
        pass
    
    
    def parse_retry_after(value):
        """Return seconds to wait, or None. Accepts delay-seconds or an HTTP-date."""
        if not value:
            return None
        value = value.strip()
        if value.isdigit():
            return float(value)
        try:
            dt = parsedate_to_datetime(value)
        except (TypeError, ValueError):
            return None
        if dt.tzinfo is None:
            dt = dt.replace(tzinfo=timezone.utc)
        return max(0.0, (dt - datetime.now(timezone.utc)).total_seconds())
    
    
    def full_jitter(attempt, base=1.0, cap=60.0):
        return random.uniform(0, min(cap, base * 2 ** attempt))
    
    
    class TokenBucket:
        def __init__(self, rate, burst, min_rate=0.1, max_rate=10.0):
            self.rate, self.capacity = rate, burst
            self.min_rate, self.max_rate = min_rate, max_rate
            self.tokens = float(burst)
            self.updated = time.monotonic()
            self.lock = asyncio.Lock()
    
        async def acquire(self):
            async with self.lock:
                while True:
                    now = time.monotonic()
                    self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
                    self.updated = now
                    if self.tokens >= 1:
                        self.tokens -= 1
                        return
                    await asyncio.sleep((1 - self.tokens) / self.rate)
    
        def on_success(self):   # additive increase
            self.rate = min(self.max_rate, self.rate + 0.05)
    
        def on_throttle(self):  # multiplicative decrease
            self.rate = max(self.min_rate, self.rate / 2)
    
    
    class CircuitBreaker:
        def __init__(self, threshold=5, cooldown=60.0):
            self.threshold, self.cooldown = threshold, cooldown
            self.failures = 0
            self.opened_at = None
            self.probing = False
    
        def allow(self):
            if self.opened_at is None:
                return True
            if not self.probing and time.monotonic() - self.opened_at >= self.cooldown:
                self.probing = True  # half-open: let one request through
                return True
            return False
    
        def record_success(self):
            self.failures, self.opened_at, self.probing = 0, None, False
    
        def record_failure(self):
            self.failures += 1
            if self.probing or self.failures >= self.threshold:
                self.opened_at, self.probing = time.monotonic(), False
    
    
    class Host:
        def __init__(self, concurrency=4, rate=2.0, burst=4):
            self.sem = asyncio.Semaphore(concurrency)
            self.bucket = TokenBucket(rate, burst)
            self.breaker = CircuitBreaker()
            self.not_before = 0.0  # monotonic timestamp; host-wide pause
    
    
    hosts = {}
    
    
    def host_for(url):
        parts = urlsplit(url)
        key = f"{parts.scheme}://{parts.netloc.lower()}"
        if key not in hosts:
            hosts[key] = Host()
        return hosts[key]
    
    
    async def fetch(client, url, max_attempts=6):
        host = host_for(url)
        last_error = None
        for attempt in range(max_attempts):
            if not host.breaker.allow():
                raise Deferred(f"circuit open for {url}")
            pause = host.not_before - time.monotonic()
            if pause > 0:
                await asyncio.sleep(pause)
    
            async with host.sem:
                await host.bucket.acquire()
                try:
                    resp = await client.get(url)
                except RETRYABLE_EXC as exc:
                    host.breaker.record_failure()
                    last_error = repr(exc)
                    delay = full_jitter(attempt)
                else:
                    status = resp.status_code
                    if status < 400:
                        host.breaker.record_success()
                        host.bucket.on_success()
                        return resp
                    if status not in RETRYABLE_STATUS:
                        raise GiveUp(f"HTTP {status} for {url}")  # 404, 403, 400 ...
                    host.breaker.record_failure()
                    last_error = f"HTTP {status}"
                    delay = full_jitter(attempt)
                    if status in THROTTLE_STATUS:
                        host.bucket.on_throttle()
                    retry_after = parse_retry_after(resp.headers.get("Retry-After"))
                    if retry_after is not None:
                        host.not_before = max(host.not_before, time.monotonic() + retry_after)
                        if retry_after > MAX_INLINE_WAIT:
                            raise Deferred(f"Retry-After {retry_after:.0f}s for {url}")
                        delay = max(delay, retry_after)
            await asyncio.sleep(delay)
        raise GiveUp(f"{url}: gave up after {max_attempts} attempts ({last_error})")
    
    
    async def main(urls):
        limits = httpx.Limits(max_connections=200, max_keepalive_connections=50)
        timeout = httpx.Timeout(20.0, connect=10.0)
        headers = {"User-Agent": "ExampleBot/0.1 (+https://www.example.com/bot.html)"}
        async with httpx.AsyncClient(limits=limits, timeout=timeout,
                                     headers=headers, follow_redirects=True) as client:
            results = await asyncio.gather(*(fetch(client, u) for u in urls),
                                           return_exceptions=True)
        for url, r in zip(urls, results):
            print(url, r.status_code if isinstance(r, httpx.Response) else repr(r))
    
    
    if __name__ == "__main__":
        asyncio.run(main(["https://example.com/", "https://example.org/"]))

    Notes on the design:

    • httpx.Limits caps the connection pool globally (the defaults are 100 connections and 20 keep-alive, per the httpx resource limits docs). It is not a per-host limiter; the semaphore and bucket are.
    • Retry-After sets not_before for the whole host, so every other task targeting that host pauses too, instead of each one discovering the limit separately.
    • The sleep before a retry happens outside the semaphore, so a backing-off request does not hold one of the host's concurrency slots.
    • For millions of URLs, replace asyncio.gather over everything with a bounded queue and a fixed pool of worker tasks, and persist Deferred and GiveUp outcomes so a later pass can pick them up. The same structure works for browser-based crawls; see Playwright headless browser scraping at scale.

    The same retry policy with tenacity

    If you prefer a library for the retry loop, tenacity supports async functions directly. wait_random_exponential is full jitter ("randomly wait up to 2^x * multiplier seconds"), and any callable that takes retry_state can serve as a wait strategy, which is how you layer Retry-After on top:

    import httpx
    from tenacity import retry, retry_if_exception, stop_after_attempt, wait_random_exponential
    
    backoff = wait_random_exponential(multiplier=1, max=60)
    
    
    def is_retryable(exc):
        if isinstance(exc, httpx.HTTPStatusError):
            return exc.response.status_code in {408, 429, 500, 502, 503, 504}
        return isinstance(exc, (httpx.TimeoutException, httpx.NetworkError))
    
    
    def wait_with_retry_after(retry_state):
        fallback = backoff(retry_state)
        exc = retry_state.outcome.exception()
        if isinstance(exc, httpx.HTTPStatusError):
            ra = parse_retry_after(exc.response.headers.get("Retry-After"))  # from the example above
            if ra is not None:
                return max(fallback, min(ra, 300))
        return fallback
    
    
    @retry(retry=retry_if_exception(is_retryable), wait=wait_with_retry_after,
           stop=stop_after_attempt(6), reraise=True)
    async def get(client, url):
        resp = await client.get(url)
        resp.raise_for_status()
        return resp

    Tenacity only handles the retry loop; it knows nothing about hosts, so keep the per-host semaphore and bucket around the call.

    Circuit breakers per host

    Backoff handles a single request. A circuit breaker handles a host that is down. Without one, a site that times out on every request can tie up workers for attempts × timeout per URL, across thousands of queued URLs. The breaker in the code above has three states:

    • Closed: requests flow normally; consecutive failures are counted.
    • Open: after threshold consecutive failures, all requests to the host are deferred for a cooldown period.
    • Half-open: after the cooldown, one probe request goes through. Success closes the breaker; failure reopens it.

    Requeue deferred URLs rather than dropping them. If many breakers open at once, the problem is probably on your side (DNS, proxy, egress).

    robots.txt and Crawl-delay

    RFC 9309 standardized the Robots Exclusion Protocol in 2022. Points that matter for a crawler's control loop:

    • The standard defines user-agent, allow and disallow, plus the * and $ special characters. The longest (most specific) match wins.
    • If robots.txt returns a 4xx ("unavailable"), the crawler may access anything. If it is unreachable because of a 5xx or network error, the crawler must assume complete disallow.
    • Crawlers should not use a cached copy for more than 24 hours, unless robots.txt is unreachable.
    • The rules "are not a form of access authorization". They are a request from the operator.

    Crawl-delay is not in RFC 9309. It is a non-standard extension that some crawlers honor and others ignore; Google states that its crawlers do not support crawl-delay. Even so, it is the site operator telling you what rate they consider acceptable, so a sensible policy is to treat it as a floor on the per-host interval: set the bucket rate to at most 1 / crawl_delay.

    Python's standard library exposes it through urllib.robotparser, with can_fetch(), crawl_delay() and request_rate(). Fetch the file with your own client so you control status-code handling, then feed the lines to the parser:

    from urllib.robotparser import RobotFileParser
    
    import httpx
    
    
    async def load_robots(client, origin):
        rp = RobotFileParser()
        try:
            r = await client.get(f"{origin}/robots.txt")
        except httpx.TransportError:
            rp.parse(["User-agent: *", "Disallow: /"])   # unreachable: assume disallow
            return rp
        if r.status_code >= 500:
            rp.parse(["User-agent: *", "Disallow: /"])   # unreachable: assume disallow
        elif r.status_code >= 400:
            rp.parse(["User-agent: *", "Disallow:"])     # unavailable: no restrictions
        else:
            rp.parse(r.text.splitlines())
        return rp
    
    # rp.can_fetch("ExampleBot", url)  -> bool
    # rp.crawl_delay("ExampleBot")     -> seconds or None

    Check your Python version: according to the Protego README, urllib.robotparser only gained RFC 9309 wildcards and longest-match precedence during the 3.14 series, so older interpreters can misjudge rules that use * or $. Protego is Scrapy's default robots parser (ROBOTSTXT_PARSER = "scrapy.robotstxt.ProtegoRobotParser") and works standalone via Protego.parse(text); note that its can_fetch(url, user_agent) takes its arguments in the opposite order from the stdlib.

    Scrapy: AutoThrottle and friends

    Scrapy already implements most of this. Requests are grouped into download slots (by domain by default), concurrency is capped per slot, and the AutoThrottle extension adjusts each slot's delay from observed latency: the target delay is latency / AUTOTHROTTLE_TARGET_CONCURRENCY, the next delay is the average of the previous and target delays, and non-200 responses are not allowed to decrease the delay. A reasonable starting point:

    # settings.py
    ROBOTSTXT_OBEY = True
    USER_AGENT = "ExampleBot/0.1 (+https://www.example.com/bot.html)"
    
    CONCURRENT_REQUESTS = 64            # global cap (default 16)
    CONCURRENT_REQUESTS_PER_DOMAIN = 4  # per-slot cap (default 8)
    DOWNLOAD_DELAY = 0.5                # floor; AutoThrottle never goes below it
    
    AUTOTHROTTLE_ENABLED = True         # default False
    AUTOTHROTTLE_START_DELAY = 5.0      # default 5.0
    AUTOTHROTTLE_MAX_DELAY = 60.0       # default 60.0
    AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0  # default 1.0; a target, not a hard limit
    AUTOTHROTTLE_DEBUG = False          # True logs per-response throttle stats
    
    RETRY_ENABLED = True
    RETRY_TIMES = 3                     # default 2 (3 requests in total)
    RETRY_HTTP_CODES = [408, 429, 500, 502, 503, 504, 522, 524]

    The defaults in the comments come from Scrapy's default_settings.py; note that scrapy startproject generates a more conservative template with ROBOTSTXT_OBEY = True, CONCURRENT_REQUESTS_PER_DOMAIN = 1 and DOWNLOAD_DELAY = 1. Two caveats. The built-in RetryMiddleware re-schedules failed requests with lower priority (RETRY_PRIORITY_ADJUST, default -1); it is not an exponential-backoff implementation and you should not assume it reads Retry-After, so let AutoThrottle and a custom middleware handle pacing on 429. And for specific sites you can pin a slot's delay through DOWNLOAD_SLOTS, or set the autothrottle_dont_adjust_delay request meta key (Scrapy 2.12+) to keep particular requests from moving a slot's delay.

    Proxies do not replace rate limiting

    Spreading traffic over many exit IPs does not change how much load a host receives. Key your budgets on the target host, not the exit IP, or adding proxies will multiply the rate each site sees. Proxies help on your side: egress capacity, regional access, and not funnelling everything through one IP. The trade-offs between proxy types are covered in datacenter vs residential vs ISP proxies, and for the YouTube-specific case of rate limiting and bot checks, see why YouTube downloads are slow and fixing "Sign in to confirm you're not a bot".

    Compliance

    Rate limits and robots.txt are operational signals, not legal permission. Site terms of service often restrict automated access (YouTube's Terms of Service, for example, restrict automated access and downloading). Copyright and licensing apply to what you collect and redistribute, privacy law such as GDPR applies to personal data including faces and voices, and you should honor takedown and opt-out requests. Get legal review for your jurisdiction and use case.

    If you would rather not run the egress yourself

    If your crawler is well behaved but your bottleneck is egress capacity, Corpora offers dedicated 1–40 Gbps unmetered proxy capacity (HTTP(S) and SOCKS5, rotating IP pools, multi-region) that plugs into custom crawlers, and a separate residential proxy product. Either way, keep the per-host limits above in place.

    FAQ

    How do I handle HTTP 429 Too Many Requests in a scraper?

    Stop sending to that host, wait for the Retry-After value if present (it can be seconds or an HTTP date), otherwise use jittered exponential backoff, and lower that host's request rate for the rest of the crawl. Apply the pause to all workers targeting the host, not just the request that got the 429.

    What is a good rate limit for web scraping?

    There is no universal number. Start conservatively (one or two concurrent requests per host with a delay of around a second, which is roughly what Scrapy's project template does), honor Crawl-delay if present, and let an adaptive scheme such as AIMD or AutoThrottle raise the rate only while latency and error rates stay healthy.

    Should I use full jitter or decorrelated jitter?

    Either beats no jitter. Full jitter is stateless and the common default; decorrelated jitter keeps state between attempts. Cap the per-attempt delay and the number of attempts either way.

    Is Crawl-delay part of the robots.txt standard?

    No. RFC 9309 defines only user-agent, allow and disallow (and allows other records such as Sitemap to be interpreted). Crawl-delay is a non-standard extension; Google ignores it, other crawlers honor it. Treating it as a minimum interval per host is the cautious choice.

    Which errors should a crawler not retry?

    Permanent client errors: 400, 401, 404, 405, 410, 422. Treat 403 as a signal to slow down and investigate rather than retry immediately, and avoid automatic retries of non-idempotent requests such as POST unless you know they are safe to repeat.