Tag: web scraping

  • Residential vs Datacenter Proxies vs ISP and Mobile for Scraping

    Residential vs datacenter proxies is the first choice most teams face when a crawler outgrows one server. ISP (static residential) and mobile proxies are the other two options. All four are billed differently, and which one fits depends on how many bytes you move, how long a session has to hold one IP, and who actually owns the address you exit from. This guide covers how each type works at the network level, how the billing models compare (with the arithmetic), and how to test a proxy against your own targets before you sign anything.

    TL;DR

    • Datacenter IPs are announced by hosting, cloud or vendor networks. They're fast, stable and cheap per IP, and the easiest type to identify as non-residential.
    • Residential IPs belong to consumer ISPs and exit through real subscriber connections. They're usually billed per GB, and throughput varies from peer to peer.
    • ISP / static residential IPs are announced by an ISP-type network but hosted on servers. They're stable and usually billed per IP.
    • Mobile IPs exit through cellular carriers, often behind carrier-grade NAT. They have the lowest throughput and are rarely suited to bulk data.
    • Cost: metered cost = GB × P, and flat cost = Gbps × F. Break-even is F / P GB per Gbps per month. At video volumes, unmetered capacity wins quickly.
    • Test before you buy: check latency, the success rate on your targets, exit-IP geolocation and ASN.

    How each type works: who announces the prefix

    Every public IP sits inside a prefix that an autonomous system (AS) announces over BGP. The origin ASN and its registered holder are the most reliable way to tell what you're really buying. RIPEstat's prefix-overview endpoint returns "whether and by whom" a prefix is announced.

    Datacenter

    The proxy runs on a server and exits from addresses belonging to the hosting company, cloud provider or vendor, and the same organisation usually announces the prefix. Throughput is limited by the port, so it's high and consistent. The ASN openly identifies a hosting network, though, and many sites apply stricter limits to hosting ranges. Shared datacenter IPs also carry whatever reputation earlier users gave them.

    Residential

    The vendor doesn't own these addresses. Traffic enters the vendor's gateway, which forwards it through a participating end-user device or router, so the exit IP belongs to a consumer ISP's AS. Exits appear and disappear, upstream bandwidth is whatever the household has, and latency includes someone's last mile. Country, region and session behaviour are usually selected with parameters in the proxy username, and the syntax differs between vendors.

    ISP (static residential)

    These are servers in a data centre that exit from address space registered to or announced by an ISP-type network. Some vendors have the ISP announce their prefix. Others lease address space and arrange for an ISP to originate it. Verify it yourself: an "ISP proxy" whose prefix is originated by a hosting ASN is a datacenter proxy. You get server-grade throughput and a stable IP, but fewer IPs per dollar, and a flagged IP stays flagged until it's replaced.

    Mobile

    Traffic exits through a modem or handset on a cellular network. Carriers commonly use carrier-grade NAT (CGN), so many subscribers share one public IPv4 address. RFC 6598, which reserved 100.64.0.0/10 for CGN, notes that "geo-location systems identify the location of the CGN server, not the end host." Rotation depends on the device getting a new address, and the radio link limits throughput. Use mobile IPs when you specifically need what carrier users see, not for bulk transfer.

    Type Who announces the prefix Common billing Throughput Sessions
    Datacenter Hosting / cloud / vendor ASN Per IP, per GB, or per Gbps (unmetered) High, consistent Static; rotating pools available
    Residential Consumer ISP ASN Per GB Variable per peer Rotating; sticky with a TTL
    ISP / static residential ISP-type ASN (verify) Per IP per month High Static
    Mobile Carrier ASN, often behind CGN Per device/port per period, or per GB Lowest Rotates on re-attach

    The billing column shows common contract structures, not a survey. This article quotes no market prices. Plug your own quotes into the formulas below.

    Rotating vs sticky sessions

    A rotating endpoint can give you a new exit IP per request or per TCP connection. A sticky session pins one exit for a time window, usually selected with a session ID in the username. Static datacenter and ISP IPs behave like a sticky session that never expires.

    • Rotation suits stateless requests, such as fetching many unrelated pages once each.
    • Stickiness is needed whenever a server ties state to your IP: cookies, signed URLs, multi-step flows and chunked downloads. The yt-dlp FAQ notes that video requests often "must come from the same IP address", which is why our yt-dlp proxy guide uses one sticky session per video.
    • Because of keep-alive, a "per-request" rotating proxy may really rotate per connection. Measure it.
    • Residential sticky sessions can end early when the peer goes offline. Make jobs resumable and requeue them on a fresh session (see rate limiting, retries and backoff).

    Billing models and the arithmetic

    • Per IP (monthly): common for static datacenter and ISP IPs. Bandwidth may be unlimited, capped or "fair use", so read the clause.
    • Per GB (metered): the norm for residential. The meter counts bytes, not outcomes, so failed requests, retries and TLS overhead usually count too.
    • Per Gbps (committed, unmetered): you reserve capacity and pay the same whatever you move through it.
    Metered:   cost_month = GB_month × P      (P = price per GB)
    Flat:      cost_month = Gbps × F          (F = price per Gbps per month)
    Break-even volume per Gbps:  GB = F / P
    
    Ceiling of 1 Gbps, 30-day month, 100% utilisation:
      1e9 bit/s × 2,592,000 s / 8 / 1e9 = 324,000 GB (324 TB)

    A worked example that needs no assumed market price. Take F = $2,000 per Gbps per month and 100 TB/month (100,000 GB):

    Flat:     1 Gbps × $2,000 = $2,000   (100 / 324 ≈ 31% utilisation)
    Metered:  100,000 GB × P
    Metered is cheaper only if 100,000 × P < 2,000  →  P < $0.02/GB

    Compare that threshold with any per-GB quote you have. At 1 PB/month you need about 3.1 Gbps sustained (provision around 5 Gbps), versus 1,000,000 × P metered. The full sizing tables are in bandwidth for large-scale video downloads. To turn media hours into GB, use the bitrate you actually measure (file size ÷ duration on a sample of real downloads):

    GB_per_hour = bitrate_Mbps × 3,600 / 8,000 = bitrate_Mbps × 0.45
    e.g. 5 Mbps average → 2.25 GB per hour

    When per-GB residential makes sense

    • Low byte volume: HTML, JSON APIs and metadata, with many requests of a few KB each.
    • Geographic breadth: many countries or cities, each with a small share of traffic.
    • Bursty or exploratory work: pilots, monthly refreshes, or a long tail of targets that fail from datacenter ranges, where reserved capacity would sit idle.

    When unmetered dedicated capacity makes sense

    • Bandwidth-heavy workloads: video, audio, large files and dataset mirrors, where the meter would dominate the bill.
    • Sustained volume: you can keep the pipe reasonably full, so you clear F / P easily.
    • Throughput-sensitive jobs: long downloads that need consistent speed and stable exits.

    A hybrid is common: residential for discovery and metadata, dedicated capacity for bulk bytes. When a server binds downloads to the requesting IP, though, an item's metadata fetch and its download must share an exit, so split traffic per item, not per request type.

    Ethical sourcing of residential IPs

    A residential exit is someone's home connection. Vendors typically recruit peers through apps or SDKs whose users agree to share bandwidth, sometimes for payment. How clear that consent is varies, and it belongs in your vendor-risk review. Ask:

    • Which apps or SDKs recruit peers, and can we see the consent screen end users see?
    • Is participation opt-in, easy to withdraw from, and compensated?
    • What limits protect peers, for example bandwidth caps or no use of metered mobile data without consent?
    • What customer KYC and abuse handling do you run?
    • Can you put this in writing for our compliance team?

    If a vendor can't answer clearly, treat that as the answer.

    HTTP(S) vs SOCKS5

    This is a protocol choice, independent of IP type.

    • HTTP proxy: for https:// targets the client sends CONNECT host:443 and the proxy opens a TCP tunnel (RFC 9110 §9.3.6). TLS stays end to end. Bad credentials return 407 Proxy Authentication Required. "HTTPS proxy" sometimes means TLS to the proxy itself, which not every client supports.
    • SOCKS5 (RFC 1928) is protocol-agnostic. It defines CONNECT, BIND and UDP ASSOCIATE, and accepts a domain name as the destination, so the proxy can resolve DNS (the socks5h:// scheme in curl and many libraries). Username/password authentication is method X'02'. Ask the vendor whether UDP is relayed.

    Choose HTTP if your toolchain includes HTTP-only downloaders such as aria2c (details in the yt-dlp proxy guide). Choose SOCKS5 for non-HTTP protocols or for DNS resolved at the exit.

    How to test a proxy

    Run tests from the region and machines you'll use in production, at a low request rate, against targets you're permitted to collect from.

    Latency breakdown with curl

    curl's --write-out variables split a request into phases. Through a proxy, time_connect is the TCP connect to the proxy, and time_appconnect adds the tunnel and the TLS handshake with the target.

    FMT='code=%{http_code} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n'
    for i in $(seq 1 20); do
      curl -s -o /dev/null --max-time 30 -x "http://USER:[email protected]:8080" -w "$FMT" "https://example.com/"
    done

    Success rate, exit IPs, ASN and country

    This script measures the success rate and latency on your own URLs, then samples exit IPs and looks each one up in RIPEstat (prefix-overview for origin ASN and holder, maxmind-geo-lite for country). For SOCKS5, install requests[socks] and use a socks5h:// URL.

    import statistics
    import time
    from collections import Counter
    
    import requests
    
    PROXY = "http://USER:[email protected]:8080"
    PROXIES = {"http": PROXY, "https": PROXY}
    TARGETS = ["https://example.com/"]  # the URLs you will actually collect
    N, PAUSE_S = 30, 1.0
    
    
    def ripestat(call, ip):
        r = requests.get(f"https://stat.ripe.net/data/{call}/data.json",
                         params={"resource": ip}, timeout=20)
        r.raise_for_status()
        return r.json()["data"]
    
    
    for url in TARGETS:
        statuses, ok_lat = Counter(), []
        for _ in range(N):
            t0 = time.perf_counter()
            try:
                status = requests.get(url, proxies=PROXIES, timeout=30).status_code
            except requests.RequestException as exc:
                status = type(exc).__name__
            statuses[status] += 1
            if status == 200:
                ok_lat.append(time.perf_counter() - t0)
            time.sleep(PAUSE_S)
        print(url, f"success={len(ok_lat)}/{N}", dict(statuses))
        if len(ok_lat) >= 2:
            p95 = statistics.quantiles(ok_lat, n=20)[18]
            print(f"  p50={statistics.median(ok_lat):.2f}s p95={p95:.2f}s")
    
    ips = set()
    for _ in range(10):
        ips.add(requests.get("https://api.ipify.org", proxies=PROXIES, timeout=30).text.strip())
        time.sleep(PAUSE_S)
    print(f"{len(ips)} distinct exit IPs in 10 requests")
    for ip in sorted(ips):
        asns = [f"AS{a['asn']} {a['holder']}" for a in ripestat("prefix-overview", ip).get("asns", [])]
        geo = ripestat("maxmind-geo-lite", ip).get("located_resources", [])
        countries = {loc["country"] for res in geo for loc in res.get("locations", [])}
        print(ip, asns or "not announced", sorted(countries))

    What to look for

    • Success rate per target. The failure type (403, 429, timeouts, resets) tells you whether the problem is reputation, rate or capacity.
    • p95 latency, not just the median. Size worker timeouts and concurrency for the tail.
    • ASN matches the label. "Residential" and "ISP" exits should originate from access-network ASNs. Read the holder names.
    • Geolocation consistency. Exits for country X should geolocate to X. Databases disagree, and CGN mobile IPs locate coarsely, so check more than one source when it matters.
    • Rotation behaviour. Count distinct IPs over a sticky session's promised lifetime, and for rotating endpoints with and without connection reuse.
    • Throughput for bulk work. Download a few real large objects and compute bytes per second per connection.

    Whatever the IP type, per-IP concurrency and request rate shape how a target treats you. For YouTube specifically, see fixing "Sign in to confirm you're not a bot".

    Compliance

    A proxy changes where requests come from, not what you're allowed to collect. Site terms still apply. YouTube's Terms of Service, for example, restrict automated access and downloading. For training data, copyright and licensing matter (filtering for Creative Commons content is one approach), as do privacy obligations for faces and voices under GDPR and similar laws, and honouring takedown and opt-out requests. Get legal review for your jurisdiction and use case.

    If you'd rather not build the proxy layer yourself

    For bandwidth-heavy collection, Corpora offers dedicated 1–40 Gbps unmetered proxy capacity (HTTP(S) and SOCKS5, compatible with yt-dlp and custom crawlers, rotating IP pools, multi-region) at $2,000 per Gbps per month, with volume discounts from 5 Gbps. See pricing. For lower-volume work that needs residential exits, there's a separate residential proxy product. Test either one the way you'd test any vendor.

    FAQ

    What is the difference between residential and datacenter proxies?

    Ownership and routing. Datacenter proxies exit from prefixes announced by hosting or cloud networks, on servers with high, stable bandwidth. Residential proxies exit through subscriber connections on consumer ISPs. They look like ordinary home users, but they're usually billed per GB and their speed varies.

    Are ISP proxies the same as residential proxies?

    No. ISP (static residential) proxies are data-centre servers using address space announced by an ISP-type network. They have datacenter speed and stability with an ISP ASN. Verify the origin ASN yourself, for example with RIPEstat.

    Is per-GB or per-Gbps pricing cheaper?

    Above F / P GB per month for each Gbps, flat pricing is cheaper. At $2,000 per Gbps, 100 TB/month on metered pricing only wins if P is below $0.02/GB.

    How do I check what kind of IP my proxy really uses?

    Fetch your exit IP through the proxy (for example from https://api.ipify.org), then look up the origin ASN and holder with RIPEstat's prefix-overview and check its location with a geolocation source. A hosting-company holder means a datacenter IP, whatever the product page says.

  • Web Scraping Rate Limiting: Retries, Backoff and 429 Handling

    A crawler that works at 10 requests per second often falls apart at 1,000: hosts start answering 429 and 503, retries pile up on top of the original load, and a few slow sites quietly eat your whole worker pool. This guide covers web scraping rate limiting from the protocol up: what the status codes and Retry-After actually mean, how to budget requests per host, how to back off without synchronized retry storms, and how to wire it all together in Python with httpx, tenacity or Scrapy.

    TL;DR

    • Limit per host, not globally. A global cap of 500 in-flight requests says nothing about how hard you hit any single site. Use a per-host concurrency cap plus a per-host token bucket.
    • Treat 429 and 503 as "slow down", not "try again now". Honor Retry-After when present (seconds or an HTTP date) and apply it to the whole host, not just the one URL.
    • Back off exponentially with jitter. Plain exponential backoff synchronizes clients; full jitter (random(0, min(cap, base * 2^n))) spreads them out.
    • Only retry what can succeed later: timeouts, connection errors, 408, 429, 500, 502, 503, 504. Never blindly retry 404 or other 4xx, and be careful with non-idempotent methods.
    • Add a per-host circuit breaker so one failing site cannot consume your retry budget and workers.
    • Read robots.txt (RFC 9309). Crawl-delay is not part of the standard; treat it as a hint you choose to honor.

    What the server is telling you: 429, 503 and Retry-After

    429 Too Many Requests is defined in RFC 6585: the user "has sent too many requests in a given amount of time". The response may include a Retry-After header. The RFC deliberately does not define how the server counts requests or identifies you; it could be per resource, per server or across a fleet, keyed on IP, credentials or a cookie. So a 429 on one path is evidence about the whole host until proven otherwise.

    The same RFC notes that servers are not required to send 429 at all. Many sites drop connections, return 503 or 403, or serve an error page with status 200. React to all of these, not just the polite one.

    503 Service Unavailable and the Retry-After field are defined in RFC 9110 (HTTP Semantics). Retry-After takes one of two forms:

    Retry-After: 120
    Retry-After: Fri, 13 Nov 2026 23:59:59 GMT

    The first is a delay in seconds, the second an HTTP date. Parse both. A robust parser treats an unparseable value as absent and falls back to your own backoff. Also put an upper bound on how long you will sleep in-process: a Retry-After: 86400 means "come back tomorrow", which is a scheduling decision (requeue the host), not a sleep().

    Budget requests per host

    Two independent controls, applied per host (scheme plus hostname plus port is a reasonable key):

    Control Limits Protects against
    Concurrency cap (semaphore) Requests in flight at once Overloading slow hosts; one host hogging connections
    Token bucket Average request rate, with a bounded burst Exceeding a rate the host tolerates, even when responses are fast

    You need both. A concurrency cap alone lets a fast host absorb a high request rate: four connections to a server that answers in 20 ms is 4 / 0.02 = 200 requests per second. A rate limit alone lets a slow host pile up open connections. The Scrapy AutoThrottle docs point out a related trap: error responses are often returned faster than normal ones, so a fixed concurrency cap makes you send faster exactly when the server starts failing.

    A token bucket has two parameters: a refill rate r (tokens per second) and a capacity b (maximum burst). Each request spends one token. Over any window of t seconds you send at most b + r × t requests. With r = 2 and b = 4, a minute allows at most 4 + 2 × 60 = 124 requests to that host.

    Start conservative and adapt. A common pattern is AIMD (additive increase, multiplicative decrease, the same idea TCP uses): on every success nudge the rate up a little, on every throttle signal cut it in half. That converges on what the host tolerates without you guessing a number.

    Exponential backoff with jitter

    When a request fails with a retryable error, wait before retrying, and wait longer each time. The canonical reference is the AWS Architecture Blog post Exponential Backoff And Jitter, whose simulator code is published on GitHub. Its variants, with attempt number n, base delay base and ceiling cap:

    Strategy Sleep before retry
    Exponential min(cap, base * 2**n)
    Equal jitter v/2 + uniform(0, v/2), where v = min(cap, base * 2**n)
    Full jitter uniform(0, min(cap, base * 2**n))
    Decorrelated jitter sleep = min(cap, uniform(base, sleep * 3))

    Why jitter matters for crawlers: if 200 workers all hit a 503 at the same moment and all wait exactly 2, 4, 8 seconds, they come back in synchronized waves and re-trigger the overload. Randomizing the wait spreads them across the window. That is the point of the post's simulations, which compare total calls and completion time across the variants. Full jitter is the simplest to implement and the usual default.

    Two rules on top of the formula. First, if the server sent Retry-After, wait at least that long: max(jittered_backoff, retry_after). Second, cap total attempts (5 to 7 is typical) and total elapsed time, then give up and record the failure for a later pass.

    What to retry, and what not to

    Outcome Retry? Notes
    Connect timeout, read timeout, connection reset, DNS hiccup Yes, with backoff Count toward the host's circuit breaker
    429 Yes, after Retry-After or backoff Also lower that host's rate
    503 Yes, after Retry-After or backoff Often maintenance or overload; slow the host down
    500, 502, 504, 408 Yes, a few times Persistent 500 on one URL is usually a bug on their side; stop after a few
    404, 410 No Record as gone
    400, 401, 405, 422 No Your request is wrong; retrying will not fix it
    403 Not immediately Often a block or bot check; treat as a throttle signal for the host and investigate
    200 with a captcha or "rate limited" body Treat as 429 Detect with a content check; it will poison your dataset otherwise

    Scrapy's default retry list is a useful cross-check: RETRY_HTTP_CODES defaults to [408, 429, 500, 502, 503, 504, 522, 524] (522 and 524 are Cloudflare-specific codes).

    Idempotency

    RFC 9110 defines PUT, DELETE and the safe methods (GET, HEAD, OPTIONS, TRACE) as idempotent: repeating them has the same intended effect as sending them once. Crawlers are almost entirely GET, so retries are safe. The exception is POST-driven endpoints (search forms, GraphQL, internal APIs). A timed-out POST may have been processed; only retry it automatically if you know the operation is a pure read or the API supports an idempotency key.

    A per-host limiter in Python (httpx + asyncio)

    The following is a complete, dependency-light implementation: per-host semaphore, token bucket with AIMD, Retry-After parsing, full-jitter backoff and a circuit breaker. It needs only pip install httpx.

    import asyncio
    import random
    import time
    from datetime import datetime, timezone
    from email.utils import parsedate_to_datetime
    from urllib.parse import urlsplit
    
    import httpx
    
    RETRYABLE_STATUS = {408, 429, 500, 502, 503, 504}
    THROTTLE_STATUS = {429, 503}
    RETRYABLE_EXC = (httpx.TimeoutException, httpx.NetworkError, httpx.RemoteProtocolError)
    MAX_INLINE_WAIT = 300.0  # longer Retry-After values: defer the URL instead of sleeping
    
    
    class GiveUp(Exception):
        pass
    
    
    class Deferred(Exception):
        pass
    
    
    def parse_retry_after(value):
        """Return seconds to wait, or None. Accepts delay-seconds or an HTTP-date."""
        if not value:
            return None
        value = value.strip()
        if value.isdigit():
            return float(value)
        try:
            dt = parsedate_to_datetime(value)
        except (TypeError, ValueError):
            return None
        if dt.tzinfo is None:
            dt = dt.replace(tzinfo=timezone.utc)
        return max(0.0, (dt - datetime.now(timezone.utc)).total_seconds())
    
    
    def full_jitter(attempt, base=1.0, cap=60.0):
        return random.uniform(0, min(cap, base * 2 ** attempt))
    
    
    class TokenBucket:
        def __init__(self, rate, burst, min_rate=0.1, max_rate=10.0):
            self.rate, self.capacity = rate, burst
            self.min_rate, self.max_rate = min_rate, max_rate
            self.tokens = float(burst)
            self.updated = time.monotonic()
            self.lock = asyncio.Lock()
    
        async def acquire(self):
            async with self.lock:
                while True:
                    now = time.monotonic()
                    self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
                    self.updated = now
                    if self.tokens >= 1:
                        self.tokens -= 1
                        return
                    await asyncio.sleep((1 - self.tokens) / self.rate)
    
        def on_success(self):   # additive increase
            self.rate = min(self.max_rate, self.rate + 0.05)
    
        def on_throttle(self):  # multiplicative decrease
            self.rate = max(self.min_rate, self.rate / 2)
    
    
    class CircuitBreaker:
        def __init__(self, threshold=5, cooldown=60.0):
            self.threshold, self.cooldown = threshold, cooldown
            self.failures = 0
            self.opened_at = None
            self.probing = False
    
        def allow(self):
            if self.opened_at is None:
                return True
            if not self.probing and time.monotonic() - self.opened_at >= self.cooldown:
                self.probing = True  # half-open: let one request through
                return True
            return False
    
        def record_success(self):
            self.failures, self.opened_at, self.probing = 0, None, False
    
        def record_failure(self):
            self.failures += 1
            if self.probing or self.failures >= self.threshold:
                self.opened_at, self.probing = time.monotonic(), False
    
    
    class Host:
        def __init__(self, concurrency=4, rate=2.0, burst=4):
            self.sem = asyncio.Semaphore(concurrency)
            self.bucket = TokenBucket(rate, burst)
            self.breaker = CircuitBreaker()
            self.not_before = 0.0  # monotonic timestamp; host-wide pause
    
    
    hosts = {}
    
    
    def host_for(url):
        parts = urlsplit(url)
        key = f"{parts.scheme}://{parts.netloc.lower()}"
        if key not in hosts:
            hosts[key] = Host()
        return hosts[key]
    
    
    async def fetch(client, url, max_attempts=6):
        host = host_for(url)
        last_error = None
        for attempt in range(max_attempts):
            if not host.breaker.allow():
                raise Deferred(f"circuit open for {url}")
            pause = host.not_before - time.monotonic()
            if pause > 0:
                await asyncio.sleep(pause)
    
            async with host.sem:
                await host.bucket.acquire()
                try:
                    resp = await client.get(url)
                except RETRYABLE_EXC as exc:
                    host.breaker.record_failure()
                    last_error = repr(exc)
                    delay = full_jitter(attempt)
                else:
                    status = resp.status_code
                    if status < 400:
                        host.breaker.record_success()
                        host.bucket.on_success()
                        return resp
                    if status not in RETRYABLE_STATUS:
                        raise GiveUp(f"HTTP {status} for {url}")  # 404, 403, 400 ...
                    host.breaker.record_failure()
                    last_error = f"HTTP {status}"
                    delay = full_jitter(attempt)
                    if status in THROTTLE_STATUS:
                        host.bucket.on_throttle()
                    retry_after = parse_retry_after(resp.headers.get("Retry-After"))
                    if retry_after is not None:
                        host.not_before = max(host.not_before, time.monotonic() + retry_after)
                        if retry_after > MAX_INLINE_WAIT:
                            raise Deferred(f"Retry-After {retry_after:.0f}s for {url}")
                        delay = max(delay, retry_after)
            await asyncio.sleep(delay)
        raise GiveUp(f"{url}: gave up after {max_attempts} attempts ({last_error})")
    
    
    async def main(urls):
        limits = httpx.Limits(max_connections=200, max_keepalive_connections=50)
        timeout = httpx.Timeout(20.0, connect=10.0)
        headers = {"User-Agent": "ExampleBot/0.1 (+https://www.example.com/bot.html)"}
        async with httpx.AsyncClient(limits=limits, timeout=timeout,
                                     headers=headers, follow_redirects=True) as client:
            results = await asyncio.gather(*(fetch(client, u) for u in urls),
                                           return_exceptions=True)
        for url, r in zip(urls, results):
            print(url, r.status_code if isinstance(r, httpx.Response) else repr(r))
    
    
    if __name__ == "__main__":
        asyncio.run(main(["https://example.com/", "https://example.org/"]))

    Notes on the design:

    • httpx.Limits caps the connection pool globally (the defaults are 100 connections and 20 keep-alive, per the httpx resource limits docs). It is not a per-host limiter; the semaphore and bucket are.
    • Retry-After sets not_before for the whole host, so every other task targeting that host pauses too, instead of each one discovering the limit separately.
    • The sleep before a retry happens outside the semaphore, so a backing-off request does not hold one of the host's concurrency slots.
    • For millions of URLs, replace asyncio.gather over everything with a bounded queue and a fixed pool of worker tasks, and persist Deferred and GiveUp outcomes so a later pass can pick them up. The same structure works for browser-based crawls; see Playwright headless browser scraping at scale.

    The same retry policy with tenacity

    If you prefer a library for the retry loop, tenacity supports async functions directly. wait_random_exponential is full jitter ("randomly wait up to 2^x * multiplier seconds"), and any callable that takes retry_state can serve as a wait strategy, which is how you layer Retry-After on top:

    import httpx
    from tenacity import retry, retry_if_exception, stop_after_attempt, wait_random_exponential
    
    backoff = wait_random_exponential(multiplier=1, max=60)
    
    
    def is_retryable(exc):
        if isinstance(exc, httpx.HTTPStatusError):
            return exc.response.status_code in {408, 429, 500, 502, 503, 504}
        return isinstance(exc, (httpx.TimeoutException, httpx.NetworkError))
    
    
    def wait_with_retry_after(retry_state):
        fallback = backoff(retry_state)
        exc = retry_state.outcome.exception()
        if isinstance(exc, httpx.HTTPStatusError):
            ra = parse_retry_after(exc.response.headers.get("Retry-After"))  # from the example above
            if ra is not None:
                return max(fallback, min(ra, 300))
        return fallback
    
    
    @retry(retry=retry_if_exception(is_retryable), wait=wait_with_retry_after,
           stop=stop_after_attempt(6), reraise=True)
    async def get(client, url):
        resp = await client.get(url)
        resp.raise_for_status()
        return resp

    Tenacity only handles the retry loop; it knows nothing about hosts, so keep the per-host semaphore and bucket around the call.

    Circuit breakers per host

    Backoff handles a single request. A circuit breaker handles a host that is down. Without one, a site that times out on every request can tie up workers for attempts × timeout per URL, across thousands of queued URLs. The breaker in the code above has three states:

    • Closed: requests flow normally; consecutive failures are counted.
    • Open: after threshold consecutive failures, all requests to the host are deferred for a cooldown period.
    • Half-open: after the cooldown, one probe request goes through. Success closes the breaker; failure reopens it.

    Requeue deferred URLs rather than dropping them. If many breakers open at once, the problem is probably on your side (DNS, proxy, egress).

    robots.txt and Crawl-delay

    RFC 9309 standardized the Robots Exclusion Protocol in 2022. Points that matter for a crawler's control loop:

    • The standard defines user-agent, allow and disallow, plus the * and $ special characters. The longest (most specific) match wins.
    • If robots.txt returns a 4xx ("unavailable"), the crawler may access anything. If it is unreachable because of a 5xx or network error, the crawler must assume complete disallow.
    • Crawlers should not use a cached copy for more than 24 hours, unless robots.txt is unreachable.
    • The rules "are not a form of access authorization". They are a request from the operator.

    Crawl-delay is not in RFC 9309. It is a non-standard extension that some crawlers honor and others ignore; Google states that its crawlers do not support crawl-delay. Even so, it is the site operator telling you what rate they consider acceptable, so a sensible policy is to treat it as a floor on the per-host interval: set the bucket rate to at most 1 / crawl_delay.

    Python's standard library exposes it through urllib.robotparser, with can_fetch(), crawl_delay() and request_rate(). Fetch the file with your own client so you control status-code handling, then feed the lines to the parser:

    from urllib.robotparser import RobotFileParser
    
    import httpx
    
    
    async def load_robots(client, origin):
        rp = RobotFileParser()
        try:
            r = await client.get(f"{origin}/robots.txt")
        except httpx.TransportError:
            rp.parse(["User-agent: *", "Disallow: /"])   # unreachable: assume disallow
            return rp
        if r.status_code >= 500:
            rp.parse(["User-agent: *", "Disallow: /"])   # unreachable: assume disallow
        elif r.status_code >= 400:
            rp.parse(["User-agent: *", "Disallow:"])     # unavailable: no restrictions
        else:
            rp.parse(r.text.splitlines())
        return rp
    
    # rp.can_fetch("ExampleBot", url)  -> bool
    # rp.crawl_delay("ExampleBot")     -> seconds or None

    Check your Python version: according to the Protego README, urllib.robotparser only gained RFC 9309 wildcards and longest-match precedence during the 3.14 series, so older interpreters can misjudge rules that use * or $. Protego is Scrapy's default robots parser (ROBOTSTXT_PARSER = "scrapy.robotstxt.ProtegoRobotParser") and works standalone via Protego.parse(text); note that its can_fetch(url, user_agent) takes its arguments in the opposite order from the stdlib.

    Scrapy: AutoThrottle and friends

    Scrapy already implements most of this. Requests are grouped into download slots (by domain by default), concurrency is capped per slot, and the AutoThrottle extension adjusts each slot's delay from observed latency: the target delay is latency / AUTOTHROTTLE_TARGET_CONCURRENCY, the next delay is the average of the previous and target delays, and non-200 responses are not allowed to decrease the delay. A reasonable starting point:

    # settings.py
    ROBOTSTXT_OBEY = True
    USER_AGENT = "ExampleBot/0.1 (+https://www.example.com/bot.html)"
    
    CONCURRENT_REQUESTS = 64            # global cap (default 16)
    CONCURRENT_REQUESTS_PER_DOMAIN = 4  # per-slot cap (default 8)
    DOWNLOAD_DELAY = 0.5                # floor; AutoThrottle never goes below it
    
    AUTOTHROTTLE_ENABLED = True         # default False
    AUTOTHROTTLE_START_DELAY = 5.0      # default 5.0
    AUTOTHROTTLE_MAX_DELAY = 60.0       # default 60.0
    AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0  # default 1.0; a target, not a hard limit
    AUTOTHROTTLE_DEBUG = False          # True logs per-response throttle stats
    
    RETRY_ENABLED = True
    RETRY_TIMES = 3                     # default 2 (3 requests in total)
    RETRY_HTTP_CODES = [408, 429, 500, 502, 503, 504, 522, 524]

    The defaults in the comments come from Scrapy's default_settings.py; note that scrapy startproject generates a more conservative template with ROBOTSTXT_OBEY = True, CONCURRENT_REQUESTS_PER_DOMAIN = 1 and DOWNLOAD_DELAY = 1. Two caveats. The built-in RetryMiddleware re-schedules failed requests with lower priority (RETRY_PRIORITY_ADJUST, default -1); it is not an exponential-backoff implementation and you should not assume it reads Retry-After, so let AutoThrottle and a custom middleware handle pacing on 429. And for specific sites you can pin a slot's delay through DOWNLOAD_SLOTS, or set the autothrottle_dont_adjust_delay request meta key (Scrapy 2.12+) to keep particular requests from moving a slot's delay.

    Proxies do not replace rate limiting

    Spreading traffic over many exit IPs does not change how much load a host receives. Key your budgets on the target host, not the exit IP, or adding proxies will multiply the rate each site sees. Proxies help on your side: egress capacity, regional access, and not funnelling everything through one IP. The trade-offs between proxy types are covered in datacenter vs residential vs ISP proxies, and for the YouTube-specific case of rate limiting and bot checks, see why YouTube downloads are slow and fixing "Sign in to confirm you're not a bot".

    Compliance

    Rate limits and robots.txt are operational signals, not legal permission. Site terms of service often restrict automated access (YouTube's Terms of Service, for example, restrict automated access and downloading). Copyright and licensing apply to what you collect and redistribute, privacy law such as GDPR applies to personal data including faces and voices, and you should honor takedown and opt-out requests. Get legal review for your jurisdiction and use case.

    If you would rather not run the egress yourself

    If your crawler is well behaved but your bottleneck is egress capacity, Corpora offers dedicated 1–40 Gbps unmetered proxy capacity (HTTP(S) and SOCKS5, rotating IP pools, multi-region) that plugs into custom crawlers, and a separate residential proxy product. Either way, keep the per-host limits above in place.

    FAQ

    How do I handle HTTP 429 Too Many Requests in a scraper?

    Stop sending to that host, wait for the Retry-After value if present (it can be seconds or an HTTP date), otherwise use jittered exponential backoff, and lower that host's request rate for the rest of the crawl. Apply the pause to all workers targeting the host, not just the request that got the 429.

    What is a good rate limit for web scraping?

    There is no universal number. Start conservatively (one or two concurrent requests per host with a delay of around a second, which is roughly what Scrapy's project template does), honor Crawl-delay if present, and let an adaptive scheme such as AIMD or AutoThrottle raise the rate only while latency and error rates stay healthy.

    Should I use full jitter or decorrelated jitter?

    Either beats no jitter. Full jitter is stateless and the common default; decorrelated jitter keeps state between attempts. Cap the per-attempt delay and the number of attempts either way.

    Is Crawl-delay part of the robots.txt standard?

    No. RFC 9309 defines only user-agent, allow and disallow (and allows other records such as Sitemap to be interpreted). Crawl-delay is a non-standard extension; Google ignores it, other crawlers honor it. Treating it as a minimum interval per host is the cautious choice.

    Which errors should a crawler not retry?

    Permanent client errors: 400, 401, 404, 405, 410, 422. Treat 403 as a signal to slow down and investigate rather than retry immediately, and avoid automatic retries of non-idempotent requests such as POST unless you know they are safe to repeat.

  • Headless Browser Scraping with Playwright: Running It at Scale

    Headless browser scraping is the most expensive way to fetch a web page: each page costs a renderer process, hundreds of megabytes of RAM and every image, font and tracker the site loads. Sometimes it's the only option that works. This guide covers how to tell when you actually need a browser, and how to run Playwright for Python at scale when you do: browser contexts, request blocking, wait strategies, per-context proxies, concurrency, crash recovery and saved sessions.

    All API names below were checked against the current Playwright for Python docs and the Playwright source docs as of November 2026. The examples use the async API.

    TL;DR

    • Try without a browser first. Look for the JSON/XHR endpoint behind the page, or data embedded in the server-rendered HTML. An httpx request downloads one document; a browser downloads and executes everything the page references.
    • One browser, many contexts. A BrowserContext is an isolated, incognito-like session with its own cookies, storage and (optionally) proxy. Create one per job and close it when the job is done.
    • Block what you don't parse. context.route("**/*", handler) combined with request.resource_type lets you abort images, media and fonts before they hit the network.
    • Don't wait for networkidle. Playwright's own docs mark it DISCOURAGED. Navigate with wait_until="domcontentloaded", then wait for the specific element or response you need.
    • Plan for crashes. Recycle browsers after a fixed number of pages, check browser.is_connected(), and requeue failed URLs instead of retrying them inline.

    Do you actually need a headless browser?

    Most sites that "need JavaScript" are fetching their data from an API that you can call directly. Before you write any Playwright code, open DevTools on the target page, go to the Network tab, filter by Fetch/XHR and reload. Then check these three things, in order:

    1. A JSON API. If the listing or detail data arrives as JSON from an endpoint like /api/search?page=2, call that endpoint directly. You get structured data, no HTML parsing, and a fraction of the bytes.
    2. Data embedded in the HTML. Many server-rendered and hydrated sites ship their state inside the initial document: JSON-LD (<script type="application/ld+json">), a framework state blob such as __NEXT_DATA__ on Next.js Pages Router sites, or plain server-rendered markup. Use "View source", not the Elements panel, to see what the server actually sends.
    3. Genuinely client-side rendering. The data is computed in the browser, requires signed requests generated by obfuscated JavaScript, or sits behind interactions (infinite scroll, a "load more" button that posts state). This is where a browser earns its cost.

    For the first two cases, a plain HTTP client is enough:

    import httpx
    
    with httpx.Client(
        proxy="http://USER:[email protected]:8080",
        timeout=20.0,
        headers={"User-Agent": "acme-research-crawler/1.0 (+mailto:[email protected])"},
    ) as client:
        r = client.get("https://example.com/api/items", params={"page": 1})
        r.raise_for_status()
        items = r.json()

    Note that current httpx takes a single proxy= argument. The old proxies= argument has been removed, so older tutorials will fail.

    A hybrid also works well. Use a browser once to log in or obtain cookies, save the state, then do the bulk fetching over HTTP. Playwright's context.request (an APIRequestContext) shares the context's cookies, so you can call JSON endpoints without rendering pages.

    Playwright vs Selenium vs Puppeteer

    Playwright Puppeteer Selenium
    Languages Python, Node.js, Java, .NET JavaScript/TypeScript Java, Python, C#, Ruby, JavaScript
    Browsers Chromium, Firefox, WebKit Chrome and Firefox, over CDP or WebDriver BiDi Chrome, Edge, Firefox, Safari via W3C WebDriver (BiDi being rolled in)
    Isolated sessions in one browser browser.new_context(), with per-context proxy browser.createBrowserContext({proxyServer}); proxy auth via page.authenticate Usually one driver session per browser; proxy set per session in the options
    Request interception Built in (route) Built in (request interception) Via WebDriver BiDi / CDP network features
    Waiting Auto-waiting locators Explicit waits / locators Explicit waits and expected conditions
    Scale-out story Your own workers; browsers can also run as a remote server Your own workers Selenium Grid

    For a Python data team, Playwright is the usual default: a first-class Python API, cheap contexts, per-context proxies and request routing without plugins. Puppeteer fits crawlers already written in Node; Selenium fits teams already running a Grid. See the Puppeteer API and Selenium WebDriver BiDi docs for the others.

    Setup

    pip install playwright
    playwright install --with-deps --only-shell

    In headless mode, Playwright uses a separate Chromium headless shell build by default. --only-shell skips downloading the full Chromium if you only ever run headless. In Docker, the Playwright docs recommend --ipc=host for Chromium ("Without it, Chromium can run out of memory and crash") and --init to avoid zombie processes. For crawling untrusted sites, they recommend running as a non-root user with their seccomp profile, so that the Chromium sandbox stays enabled.

    Browser contexts: the unit of work

    Launching a browser is slow and memory-heavy. Creating a context is cheap. The Browser API describes new_context() as creating a context that "won't share cookies/cache with other browser contexts". That gives you the right shape for a scraper: one long-lived browser per worker process, one short-lived context per job (a URL, or a small group of related URLs).

    Always close contexts explicitly. The docs call this best practice, because browser.close() is "similar to force-quitting the browser". Avoid browser.new_page() in production code too. It's a convenience method that hides context lifetimes.

    Block images, media and fonts with route()

    Requests you abort never reach the network. Register the handler on the context, not the page, so that it covers popups as well. Use request.resource_type, which the Request docs say is one of document, stylesheet, image, media, font, script, texttrack, xhr, fetch, eventsource, websocket, manifest, beacon, ping, cspreport or other.

    BLOCKED_TYPES = {"image", "media", "font"}
    BLOCKED_HOSTS = ("googletagmanager.com", "doubleclick.net")
    
    async def block_heavy(route):
        req = route.request
        if req.resource_type in BLOCKED_TYPES or any(h in req.url for h in BLOCKED_HOSTS):
            await route.abort()
        else:
            await route.continue_()
    
    await context.route("**/*", block_heavy)

    Three caveats:

    • Stylesheets are a judgment call. Blocking CSS saves bytes, but Playwright's visibility checks depend on layout, so elements that CSS normally hides can count as "visible". Some sites also lazy-load content based on scroll position or element size. Start by blocking only images, media and fonts.
    • Service workers bypass routing. The BrowserContext docs state that route() won't intercept requests made by a service worker, and recommend service_workers="block" on new_context().
    • Every routed request round-trips through your Python process. If that shows up in profiling, use a narrower glob such as "**/*.{png,jpg,jpeg,webp,gif,woff,woff2}".

    To estimate how much blocking saves, use this formula: egress per hour = pages/hour × average bytes transferred per page. Record a few representative pages with and without blocking (the response events or a HAR file give you the sizes) and plug in your own numbers. On per-GB-billed proxies, this decides your bill.

    Wait strategies: avoid networkidle

    page.goto() waits for the load event by default. Its wait_until option accepts "load", "domcontentloaded", "networkidle" and "commit". The navigation docs admit that "there is no way to tell that the page is loaded". The API reference marks networkidle (no connections for at least 500 ms) as DISCOURAGED. On real sites it either never fires (analytics beacons, long polling, websockets) or fires before your data arrives.

    Wait for the thing you actually need instead:

    # 1. Navigate cheaply, then wait for the element you'll parse
    await page.goto(url, wait_until="domcontentloaded", timeout=30_000)
    await page.locator("article h1").first.wait_for(timeout=15_000)
    
    # 2. Or capture the JSON the page fetches. Often better than parsing the DOM.
    async with page.expect_response(
        lambda r: "/api/search" in r.url and r.status == 200
    ) as resp_info:
        await page.goto(url, wait_until="domcontentloaded")
    data = await (await resp_info.value).json()

    The second pattern lets the site's own JavaScript build the request, and you keep the clean JSON instead of rendered markup. page.wait_for_selector() still works, but the API reference marks it discouraged in favour of locator.wait_for(). locator.wait_for() defaults to waiting until the element is visible. Pass state="attached" if you only need it in the DOM, which matters when you block CSS.

    Set timeouts with Playwright's own timeout parameters, or with context.set_default_timeout() and context.set_default_navigation_timeout(). Don't wrap Playwright calls in asyncio.wait_for(). The library docs say cancelling a task that's running a Playwright call "is not supported and results in undefined behavior".

    Per-context proxies

    Since contexts are isolated, each one can exit through a different proxy. The proxy option on new_context() takes server, bypass, username and password, and supports HTTP and SOCKS proxies (Network docs):

    context = await browser.new_context(
        proxy={
            "server": "http://proxy.example.com:8080",
            "username": "USER",
            "password": "PASS",
        },
        service_workers="block",
    )

    The docs describe username/password as being for an HTTP proxy that requires authentication. If you need an authenticated SOCKS5 proxy, test it with your browser engine before relying on it, or use an HTTP endpoint or IP allow-listing instead.

    Map proxies to work deliberately. Keep all requests for one logical session (login, listing, detail pages) on the same exit IP, rotate between sessions, and cap concurrency per IP rather than just in total. The trade-offs between proxy types are covered in datacenter vs residential vs ISP proxies for scraping. Pacing and retry policy are covered in rate limiting, retries and backoff.

    Sessions with storage_state

    If a target needs a login, don't log in for every job. Log in once, save the state, and seed new contexts from it (Authentication docs):

    # once
    await context.storage_state(path="state/site-a.json")
    
    # per job
    context = await browser.new_context(storage_state="state/site-a.json", proxy=proxy)

    The snapshot contains cookies and local storage. IndexedDB is only included if you pass indexed_db=True when saving, and session storage isn't saved at all. Treat these files as credentials: the docs warn that they can be used to impersonate the account. Many sites tie a session to an IP or region, so reuse a state file with the same proxy that created it.

    Concurrency and memory

    Three rules hold up in practice:

    • Scale out with processes. Run one Python process per CPU core or so, each with one browser and an asyncio.Semaphore limiting concurrent contexts. The docs state that Playwright's API is not thread-safe; if you use threads, create one Playwright instance per thread.
    • Measure memory, don't guess it. Browser memory depends heavily on the site. A news article and a heavy single-page app differ by an order of magnitude. Run one worker at your target concurrency for 15 minutes, read the peak RSS of the whole process tree (Python, driver and all browser processes), then size with workers per host = floor((host RAM − headroom) / peak RSS per worker).
    • Recycle browsers. Long-lived browsers accumulate memory and occasionally wedge. Relaunching after a fixed number of pages (a few hundred is a reasonable starting point to tune) bounds both problems.

    Crash recovery: a worker you can leave running

    Pages crash when they "allocate too much memory" (from the page.on("crash") docs), and browsers die too. When that happens, browser.on("disconnected") fires and in-flight operations raise playwright.async_api.Error. The worker below processes URLs in batches per browser lifetime, recycles the browser between batches, and requeues failures with a retry cap:

    import asyncio
    from collections import deque
    
    from playwright.async_api import Error as PlaywrightError, async_playwright
    
    CONCURRENCY = 8          # contexts per browser
    RECYCLE_AFTER = 300      # pages per browser lifetime
    MAX_ATTEMPTS = 3
    PROXIES = [
        {"server": "http://proxy.example.com:8080", "username": "USER", "password": "PASS"},
    ]
    
    async def block_heavy(route):
        if route.request.resource_type in {"image", "media", "font"}:
            await route.abort()
        else:
            await route.continue_()
    
    async def scrape_one(browser, url, proxy):
        context = await browser.new_context(proxy=proxy, service_workers="block")
        try:
            context.set_default_timeout(15_000)
            await context.route("**/*", block_heavy)
            page = await context.new_page()
            resp = await page.goto(url, wait_until="domcontentloaded", timeout=30_000)
            if resp is not None and resp.status >= 400:
                raise PlaywrightError(f"HTTP {resp.status}")
            title = page.locator("h1").first
            await title.wait_for(state="attached")
            return {"url": url, "title": (await title.text_content() or "").strip()}
        finally:
            try:
                await context.close()
            except PlaywrightError:
                pass  # browser already gone
    
    async def run(urls):
        queue = deque((u, 1) for u in urls)
        results, dead = [], []
        async with async_playwright() as pw:
            while queue:
                batch = [queue.popleft() for _ in range(min(RECYCLE_AFTER, len(queue)))]
                browser = await pw.chromium.launch()
                sem = asyncio.Semaphore(CONCURRENCY)
    
                async def job(i, url, attempt):
                    async with sem:
                        if not browser.is_connected():
                            return url, attempt, "browser disconnected"
                        try:
                            return url, attempt, await scrape_one(browser, url, PROXIES[i % len(PROXIES)])
                        except PlaywrightError as e:  # includes TimeoutError
                            return url, attempt, str(e).splitlines()[0][:200]
    
                try:
                    outcomes = await asyncio.gather(*(job(i, u, a) for i, (u, a) in enumerate(batch)))
                finally:
                    if browser.is_connected():
                        await browser.close()
    
                for url, attempt, out in outcomes:
                    if isinstance(out, dict):
                        results.append(out)
                    elif attempt < MAX_ATTEMPTS:
                        queue.append((url, attempt + 1))
                    else:
                        dead.append({"url": url, "error": out})
        return results, dead
    
    if __name__ == "__main__":
        ok, failed = asyncio.run(run(["https://example.com/"]))
        print(len(ok), "ok,", len(failed), "failed")

    Design notes:

    • Failed URLs go to the back of the queue and are retried later, usually on a fresh browser and possibly a different proxy.
    • Timeouts are Playwright's own, so a hung page fails cleanly instead of being cancelled from outside.
    • In production, use a durable queue (Redis, SQS, a database table) instead of the deque, write results as you go, and run the process under a supervisor (systemd, Kubernetes) that restarts it if it exits.

    Operational checklist

    Symptom Likely cause Fix
    goto times out but the HTML looks complete wait_until="load" or "networkidle" waiting on trackers or long polling domcontentloaded plus a locator wait
    Memory climbs steadily Unclosed contexts, long-lived browser Close every context in finally; recycle browsers
    Chromium crashes in Docker Container shared-memory (IPC) limits --ipc=host (per Playwright Docker docs)
    Blocked requests still show up Service worker fetching them service_workers="block"
    Locator never "visible" CSS blocked or element off-screen / lazy state="attached", or stop blocking stylesheets
    Session logged out mid-run State reused from a different IP or expired Pin state files to a proxy; refresh periodically
    Many 403/429 responses Too much concurrency per IP or domain Per-domain rate limits and backoff, not more retries

    If the pages you scrape point to images for a vision-language dataset, download those with a dedicated tool rather than the browser. See building an image-text dataset with img2dataset and CLIP filtering. And if the text you need is already in Common Crawl, start with the WARC-to-text pipeline before crawling anything.

    A note on compliance

    A headless browser doesn't change the rules that apply to automated collection. Site terms of service may restrict automated access (YouTube's Terms, for example, restrict automated access and downloading). Respect robots.txt and rate limits. Copyright and licensing apply to what you collect and redistribute. Privacy law such as GDPR applies to personal data, including faces and voices in media. Have a process for honouring takedown and opt-out requests, and get legal review for your jurisdiction and use case. Don't scrape behind logins you aren't authorised to use.

    If you'd rather not run the network layer yourself

    If your Playwright fleet needs stable, high-throughput egress, Corpora offers dedicated 1–40 Gbps unmetered proxy capacity (HTTP(S) and SOCKS5, rotating pools, multi-region) for self-managed crawlers. See pricing. For targets that need residential exits, there are also residential proxies.

    FAQ

    Is Playwright faster than Selenium for web scraping?

    For many scraping workloads, Playwright needs fewer resources per job. Contexts let one browser serve many isolated sessions, and auto-waiting locators cut down on fixed sleeps. Actual throughput depends mostly on the target site and on how much you block, so benchmark both on your own pages if it matters.

    Should I use networkidle in Playwright?

    Generally no. Playwright's docs mark networkidle as discouraged. Use wait_until="domcontentloaded" and then wait for a specific locator, or capture the API response with page.expect_response().

    How do I use a different proxy per page in Playwright?

    Create a separate browser context per proxy: browser.new_context(proxy={"server": ..., "username": ..., "password": ...}). All pages in that context share the proxy, so you get per-session IPs within a single browser process.

    How many Playwright browsers can I run per server?

    There's no fixed number. It depends on the pages you load and what you block. Measure the peak RSS of one worker at your target concurrency, then divide the available RAM (minus headroom) by that figure. Recycle browsers periodically to keep the number stable.

    Can I block images in headless Chrome with Playwright?

    Yes. Register context.route("**/*", handler) and call route.abort() when route.request.resource_type is "image", "media" or "font". Otherwise call route.continue_().