Tag: Playwright

  • Headless Browser Scraping with Playwright: Running It at Scale

    Headless browser scraping is the most expensive way to fetch a web page: each page costs a renderer process, hundreds of megabytes of RAM and every image, font and tracker the site loads. Sometimes it's the only option that works. This guide covers how to tell when you actually need a browser, and how to run Playwright for Python at scale when you do: browser contexts, request blocking, wait strategies, per-context proxies, concurrency, crash recovery and saved sessions.

    All API names below were checked against the current Playwright for Python docs and the Playwright source docs as of November 2026. The examples use the async API.

    TL;DR

    • Try without a browser first. Look for the JSON/XHR endpoint behind the page, or data embedded in the server-rendered HTML. An httpx request downloads one document; a browser downloads and executes everything the page references.
    • One browser, many contexts. A BrowserContext is an isolated, incognito-like session with its own cookies, storage and (optionally) proxy. Create one per job and close it when the job is done.
    • Block what you don't parse. context.route("**/*", handler) combined with request.resource_type lets you abort images, media and fonts before they hit the network.
    • Don't wait for networkidle. Playwright's own docs mark it DISCOURAGED. Navigate with wait_until="domcontentloaded", then wait for the specific element or response you need.
    • Plan for crashes. Recycle browsers after a fixed number of pages, check browser.is_connected(), and requeue failed URLs instead of retrying them inline.

    Do you actually need a headless browser?

    Most sites that "need JavaScript" are fetching their data from an API that you can call directly. Before you write any Playwright code, open DevTools on the target page, go to the Network tab, filter by Fetch/XHR and reload. Then check these three things, in order:

    1. A JSON API. If the listing or detail data arrives as JSON from an endpoint like /api/search?page=2, call that endpoint directly. You get structured data, no HTML parsing, and a fraction of the bytes.
    2. Data embedded in the HTML. Many server-rendered and hydrated sites ship their state inside the initial document: JSON-LD (<script type="application/ld+json">), a framework state blob such as __NEXT_DATA__ on Next.js Pages Router sites, or plain server-rendered markup. Use "View source", not the Elements panel, to see what the server actually sends.
    3. Genuinely client-side rendering. The data is computed in the browser, requires signed requests generated by obfuscated JavaScript, or sits behind interactions (infinite scroll, a "load more" button that posts state). This is where a browser earns its cost.

    For the first two cases, a plain HTTP client is enough:

    import httpx
    
    with httpx.Client(
        proxy="http://USER:[email protected]:8080",
        timeout=20.0,
        headers={"User-Agent": "acme-research-crawler/1.0 (+mailto:[email protected])"},
    ) as client:
        r = client.get("https://example.com/api/items", params={"page": 1})
        r.raise_for_status()
        items = r.json()

    Note that current httpx takes a single proxy= argument. The old proxies= argument has been removed, so older tutorials will fail.

    A hybrid also works well. Use a browser once to log in or obtain cookies, save the state, then do the bulk fetching over HTTP. Playwright's context.request (an APIRequestContext) shares the context's cookies, so you can call JSON endpoints without rendering pages.

    Playwright vs Selenium vs Puppeteer

    Playwright Puppeteer Selenium
    Languages Python, Node.js, Java, .NET JavaScript/TypeScript Java, Python, C#, Ruby, JavaScript
    Browsers Chromium, Firefox, WebKit Chrome and Firefox, over CDP or WebDriver BiDi Chrome, Edge, Firefox, Safari via W3C WebDriver (BiDi being rolled in)
    Isolated sessions in one browser browser.new_context(), with per-context proxy browser.createBrowserContext({proxyServer}); proxy auth via page.authenticate Usually one driver session per browser; proxy set per session in the options
    Request interception Built in (route) Built in (request interception) Via WebDriver BiDi / CDP network features
    Waiting Auto-waiting locators Explicit waits / locators Explicit waits and expected conditions
    Scale-out story Your own workers; browsers can also run as a remote server Your own workers Selenium Grid

    For a Python data team, Playwright is the usual default: a first-class Python API, cheap contexts, per-context proxies and request routing without plugins. Puppeteer fits crawlers already written in Node; Selenium fits teams already running a Grid. See the Puppeteer API and Selenium WebDriver BiDi docs for the others.

    Setup

    pip install playwright
    playwright install --with-deps --only-shell

    In headless mode, Playwright uses a separate Chromium headless shell build by default. --only-shell skips downloading the full Chromium if you only ever run headless. In Docker, the Playwright docs recommend --ipc=host for Chromium ("Without it, Chromium can run out of memory and crash") and --init to avoid zombie processes. For crawling untrusted sites, they recommend running as a non-root user with their seccomp profile, so that the Chromium sandbox stays enabled.

    Browser contexts: the unit of work

    Launching a browser is slow and memory-heavy. Creating a context is cheap. The Browser API describes new_context() as creating a context that "won't share cookies/cache with other browser contexts". That gives you the right shape for a scraper: one long-lived browser per worker process, one short-lived context per job (a URL, or a small group of related URLs).

    Always close contexts explicitly. The docs call this best practice, because browser.close() is "similar to force-quitting the browser". Avoid browser.new_page() in production code too. It's a convenience method that hides context lifetimes.

    Block images, media and fonts with route()

    Requests you abort never reach the network. Register the handler on the context, not the page, so that it covers popups as well. Use request.resource_type, which the Request docs say is one of document, stylesheet, image, media, font, script, texttrack, xhr, fetch, eventsource, websocket, manifest, beacon, ping, cspreport or other.

    BLOCKED_TYPES = {"image", "media", "font"}
    BLOCKED_HOSTS = ("googletagmanager.com", "doubleclick.net")
    
    async def block_heavy(route):
        req = route.request
        if req.resource_type in BLOCKED_TYPES or any(h in req.url for h in BLOCKED_HOSTS):
            await route.abort()
        else:
            await route.continue_()
    
    await context.route("**/*", block_heavy)

    Three caveats:

    • Stylesheets are a judgment call. Blocking CSS saves bytes, but Playwright's visibility checks depend on layout, so elements that CSS normally hides can count as "visible". Some sites also lazy-load content based on scroll position or element size. Start by blocking only images, media and fonts.
    • Service workers bypass routing. The BrowserContext docs state that route() won't intercept requests made by a service worker, and recommend service_workers="block" on new_context().
    • Every routed request round-trips through your Python process. If that shows up in profiling, use a narrower glob such as "**/*.{png,jpg,jpeg,webp,gif,woff,woff2}".

    To estimate how much blocking saves, use this formula: egress per hour = pages/hour × average bytes transferred per page. Record a few representative pages with and without blocking (the response events or a HAR file give you the sizes) and plug in your own numbers. On per-GB-billed proxies, this decides your bill.

    Wait strategies: avoid networkidle

    page.goto() waits for the load event by default. Its wait_until option accepts "load", "domcontentloaded", "networkidle" and "commit". The navigation docs admit that "there is no way to tell that the page is loaded". The API reference marks networkidle (no connections for at least 500 ms) as DISCOURAGED. On real sites it either never fires (analytics beacons, long polling, websockets) or fires before your data arrives.

    Wait for the thing you actually need instead:

    # 1. Navigate cheaply, then wait for the element you'll parse
    await page.goto(url, wait_until="domcontentloaded", timeout=30_000)
    await page.locator("article h1").first.wait_for(timeout=15_000)
    
    # 2. Or capture the JSON the page fetches. Often better than parsing the DOM.
    async with page.expect_response(
        lambda r: "/api/search" in r.url and r.status == 200
    ) as resp_info:
        await page.goto(url, wait_until="domcontentloaded")
    data = await (await resp_info.value).json()

    The second pattern lets the site's own JavaScript build the request, and you keep the clean JSON instead of rendered markup. page.wait_for_selector() still works, but the API reference marks it discouraged in favour of locator.wait_for(). locator.wait_for() defaults to waiting until the element is visible. Pass state="attached" if you only need it in the DOM, which matters when you block CSS.

    Set timeouts with Playwright's own timeout parameters, or with context.set_default_timeout() and context.set_default_navigation_timeout(). Don't wrap Playwright calls in asyncio.wait_for(). The library docs say cancelling a task that's running a Playwright call "is not supported and results in undefined behavior".

    Per-context proxies

    Since contexts are isolated, each one can exit through a different proxy. The proxy option on new_context() takes server, bypass, username and password, and supports HTTP and SOCKS proxies (Network docs):

    context = await browser.new_context(
        proxy={
            "server": "http://proxy.example.com:8080",
            "username": "USER",
            "password": "PASS",
        },
        service_workers="block",
    )

    The docs describe username/password as being for an HTTP proxy that requires authentication. If you need an authenticated SOCKS5 proxy, test it with your browser engine before relying on it, or use an HTTP endpoint or IP allow-listing instead.

    Map proxies to work deliberately. Keep all requests for one logical session (login, listing, detail pages) on the same exit IP, rotate between sessions, and cap concurrency per IP rather than just in total. The trade-offs between proxy types are covered in datacenter vs residential vs ISP proxies for scraping. Pacing and retry policy are covered in rate limiting, retries and backoff.

    Sessions with storage_state

    If a target needs a login, don't log in for every job. Log in once, save the state, and seed new contexts from it (Authentication docs):

    # once
    await context.storage_state(path="state/site-a.json")
    
    # per job
    context = await browser.new_context(storage_state="state/site-a.json", proxy=proxy)

    The snapshot contains cookies and local storage. IndexedDB is only included if you pass indexed_db=True when saving, and session storage isn't saved at all. Treat these files as credentials: the docs warn that they can be used to impersonate the account. Many sites tie a session to an IP or region, so reuse a state file with the same proxy that created it.

    Concurrency and memory

    Three rules hold up in practice:

    • Scale out with processes. Run one Python process per CPU core or so, each with one browser and an asyncio.Semaphore limiting concurrent contexts. The docs state that Playwright's API is not thread-safe; if you use threads, create one Playwright instance per thread.
    • Measure memory, don't guess it. Browser memory depends heavily on the site. A news article and a heavy single-page app differ by an order of magnitude. Run one worker at your target concurrency for 15 minutes, read the peak RSS of the whole process tree (Python, driver and all browser processes), then size with workers per host = floor((host RAM − headroom) / peak RSS per worker).
    • Recycle browsers. Long-lived browsers accumulate memory and occasionally wedge. Relaunching after a fixed number of pages (a few hundred is a reasonable starting point to tune) bounds both problems.

    Crash recovery: a worker you can leave running

    Pages crash when they "allocate too much memory" (from the page.on("crash") docs), and browsers die too. When that happens, browser.on("disconnected") fires and in-flight operations raise playwright.async_api.Error. The worker below processes URLs in batches per browser lifetime, recycles the browser between batches, and requeues failures with a retry cap:

    import asyncio
    from collections import deque
    
    from playwright.async_api import Error as PlaywrightError, async_playwright
    
    CONCURRENCY = 8          # contexts per browser
    RECYCLE_AFTER = 300      # pages per browser lifetime
    MAX_ATTEMPTS = 3
    PROXIES = [
        {"server": "http://proxy.example.com:8080", "username": "USER", "password": "PASS"},
    ]
    
    async def block_heavy(route):
        if route.request.resource_type in {"image", "media", "font"}:
            await route.abort()
        else:
            await route.continue_()
    
    async def scrape_one(browser, url, proxy):
        context = await browser.new_context(proxy=proxy, service_workers="block")
        try:
            context.set_default_timeout(15_000)
            await context.route("**/*", block_heavy)
            page = await context.new_page()
            resp = await page.goto(url, wait_until="domcontentloaded", timeout=30_000)
            if resp is not None and resp.status >= 400:
                raise PlaywrightError(f"HTTP {resp.status}")
            title = page.locator("h1").first
            await title.wait_for(state="attached")
            return {"url": url, "title": (await title.text_content() or "").strip()}
        finally:
            try:
                await context.close()
            except PlaywrightError:
                pass  # browser already gone
    
    async def run(urls):
        queue = deque((u, 1) for u in urls)
        results, dead = [], []
        async with async_playwright() as pw:
            while queue:
                batch = [queue.popleft() for _ in range(min(RECYCLE_AFTER, len(queue)))]
                browser = await pw.chromium.launch()
                sem = asyncio.Semaphore(CONCURRENCY)
    
                async def job(i, url, attempt):
                    async with sem:
                        if not browser.is_connected():
                            return url, attempt, "browser disconnected"
                        try:
                            return url, attempt, await scrape_one(browser, url, PROXIES[i % len(PROXIES)])
                        except PlaywrightError as e:  # includes TimeoutError
                            return url, attempt, str(e).splitlines()[0][:200]
    
                try:
                    outcomes = await asyncio.gather(*(job(i, u, a) for i, (u, a) in enumerate(batch)))
                finally:
                    if browser.is_connected():
                        await browser.close()
    
                for url, attempt, out in outcomes:
                    if isinstance(out, dict):
                        results.append(out)
                    elif attempt < MAX_ATTEMPTS:
                        queue.append((url, attempt + 1))
                    else:
                        dead.append({"url": url, "error": out})
        return results, dead
    
    if __name__ == "__main__":
        ok, failed = asyncio.run(run(["https://example.com/"]))
        print(len(ok), "ok,", len(failed), "failed")

    Design notes:

    • Failed URLs go to the back of the queue and are retried later, usually on a fresh browser and possibly a different proxy.
    • Timeouts are Playwright's own, so a hung page fails cleanly instead of being cancelled from outside.
    • In production, use a durable queue (Redis, SQS, a database table) instead of the deque, write results as you go, and run the process under a supervisor (systemd, Kubernetes) that restarts it if it exits.

    Operational checklist

    Symptom Likely cause Fix
    goto times out but the HTML looks complete wait_until="load" or "networkidle" waiting on trackers or long polling domcontentloaded plus a locator wait
    Memory climbs steadily Unclosed contexts, long-lived browser Close every context in finally; recycle browsers
    Chromium crashes in Docker Container shared-memory (IPC) limits --ipc=host (per Playwright Docker docs)
    Blocked requests still show up Service worker fetching them service_workers="block"
    Locator never "visible" CSS blocked or element off-screen / lazy state="attached", or stop blocking stylesheets
    Session logged out mid-run State reused from a different IP or expired Pin state files to a proxy; refresh periodically
    Many 403/429 responses Too much concurrency per IP or domain Per-domain rate limits and backoff, not more retries

    If the pages you scrape point to images for a vision-language dataset, download those with a dedicated tool rather than the browser. See building an image-text dataset with img2dataset and CLIP filtering. And if the text you need is already in Common Crawl, start with the WARC-to-text pipeline before crawling anything.

    A note on compliance

    A headless browser doesn't change the rules that apply to automated collection. Site terms of service may restrict automated access (YouTube's Terms, for example, restrict automated access and downloading). Respect robots.txt and rate limits. Copyright and licensing apply to what you collect and redistribute. Privacy law such as GDPR applies to personal data, including faces and voices in media. Have a process for honouring takedown and opt-out requests, and get legal review for your jurisdiction and use case. Don't scrape behind logins you aren't authorised to use.

    If you'd rather not run the network layer yourself

    If your Playwright fleet needs stable, high-throughput egress, Corpora offers dedicated 1–40 Gbps unmetered proxy capacity (HTTP(S) and SOCKS5, rotating pools, multi-region) for self-managed crawlers. See pricing. For targets that need residential exits, there are also residential proxies.

    FAQ

    Is Playwright faster than Selenium for web scraping?

    For many scraping workloads, Playwright needs fewer resources per job. Contexts let one browser serve many isolated sessions, and auto-waiting locators cut down on fixed sleeps. Actual throughput depends mostly on the target site and on how much you block, so benchmark both on your own pages if it matters.

    Should I use networkidle in Playwright?

    Generally no. Playwright's docs mark networkidle as discouraged. Use wait_until="domcontentloaded" and then wait for a specific locator, or capture the API response with page.expect_response().

    How do I use a different proxy per page in Playwright?

    Create a separate browser context per proxy: browser.new_context(proxy={"server": ..., "username": ..., "password": ...}). All pages in that context share the proxy, so you get per-session IPs within a single browser process.

    How many Playwright browsers can I run per server?

    There's no fixed number. It depends on the pages you load and what you block. Measure the peak RSS of one worker at your target concurrency, then divide the available RAM (minus headroom) by that figure. Recycle browsers periodically to keep the number stable.

    Can I block images in headless Chrome with Playwright?

    Yes. Register context.route("**/*", handler) and call route.abort() when route.request.resource_type is "image", "media" or "font". Otherwise call route.continue_().