Tag: proxies

  • yt-dlp Proxy Guide: --proxy, SOCKS5, Sessions and Workers

    Pointing yt-dlp at a proxy takes one flag. Getting it to work reliably across thousands of videos and dozens of workers takes more. This guide covers the proxy URL format, SOCKS vs HTTP, environment variables, config files, the Python API, sticky sessions, aria2c, and a troubleshooting table for the errors you'll actually hit.

    Everything below was checked against the yt-dlp README and source for release 2026.10.26, the current stable release as of November 2026. yt-dlp changes quickly, so run yt-dlp -U and check yt-dlp --help if a flag behaves differently for you.

    TL;DR

    • Use --proxy "http://USER:[email protected]:8080" or --proxy "socks5h://USER:[email protected]:1080". If the password has special characters, URL-encode it.
    • In yt-dlp, socks5:// is silently rewritten to socks5h://, so DNS is resolved by the proxy either way. That's not true of every external downloader.
    • If --proxy is not set, yt-dlp reads HTTP_PROXY/HTTPS_PROXY/ALL_PROXY/NO_PROXY from the environment. --proxy "" forces a direct connection.
    • Fetch a video's metadata and download its media from the same exit IP. Use sticky sessions, or one static IP per worker, rather than a new IP on every request.
    • --geo-verification-proxy does nothing for YouTube in the current source. Only --proxy matters.
    • aria2c only supports HTTP proxies, and ffmpeg can't use SOCKS proxies.

    HTTP/HTTPS vs SOCKS5, and socks5 vs socks5h

    The network options describe --proxy URL as "Use the specified HTTP/HTTPS/SOCKS proxy". The scheme you put in the URL decides the proxy type:

    Scheme What it means Notes
    http:// Plain HTTP proxy. HTTPS traffic is tunnelled through it with CONNECT The most widely supported option. A URL with no scheme is treated as http://.
    https:// TLS connection to the proxy itself Needs the requests or curl_cffi request handler. The plain urllib handler doesn't support it. The README lists requests as the dependency "for HTTPS proxy". Standalone binaries include it.
    socks5:// SOCKS5 yt-dlp rewrites it to socks5h (see below).
    socks5h:// SOCKS5 with remote DNS The proxy resolves hostnames.
    socks4://, socks4a:// SOCKS4 / 4a No username/password support in SOCKS4. Rarely useful today.

    Most tools use the h suffix to mean "let the proxy resolve DNS". Without it, the client resolves hostnames locally, which leaks DNS lookups and can pick a CDN node near your server instead of the proxy. yt-dlp handles this differently: in yt_dlp/utils/networking.py, clean_proxies() maps 'socks5': 'socks5h' with the comment "compat: socks5 was treated as socks5h". As a result, both schemes give you remote DNS for yt-dlp's own requests.

    Still write socks5h:// explicitly. It documents your intent, and the same URL may later be passed to curl or another tool where socks5:// really does mean local DNS.

    HTTP or SOCKS5? For yt-dlp alone, either works. Choose HTTP if you plan to use aria2c or ffmpeg as the downloader (see below). Choose SOCKS5 if your provider only offers SOCKS or you're tunnelling through something like ssh -D.

    The --proxy flag and URL format

    The general form is scheme://user:password@host:port:

    # HTTP proxy with credentials
    yt-dlp --proxy "http://USER:[email protected]:8080" "https://www.youtube.com/watch?v=VIDEO_ID"
    
    # SOCKS5 with remote DNS
    yt-dlp --proxy "socks5h://USER:[email protected]:1080" "https://www.youtube.com/watch?v=VIDEO_ID"
    
    # Force a direct connection (overrides env vars and config files)
    yt-dlp --proxy "" "https://www.youtube.com/watch?v=VIDEO_ID"

    URL-encode special characters in passwords

    Characters like @ : / # ? % in a username or password break URL parsing. You'll usually get a confusing connection error or a 407 instead of a clear message. Percent-encode the credentials. yt-dlp and its HTTP backends decode them before sending authentication:

    python3 -c 'import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1], safe=""))' 'p@ss:w/rd#1'
    # p%40ss%3Aw%2Frd%231
    
    yt-dlp --proxy "http://USER:p%40ss%3Aw%2Frd%[email protected]:8080" URL

    Always quote the proxy URL in the shell. A #, & or ! left unquoted will be read by the shell, not passed to yt-dlp.

    --geo-verification-proxy

    The README says this option makes yt-dlp "use this proxy to verify the IP address for some geo-restricted sites", while --proxy is still used for the actual download. Only extractors that call the geo_verification_headers() helper use it. In 2026.10.26, the YouTube extractor doesn't. For YouTube, --geo-verification-proxy has no effect. If you need a YouTube video to appear to come from a given country, the exit IP from --proxy is what counts.

    Environment variables

    When --proxy is absent, yt-dlp calls Python's urllib.request.getproxies() (see the proxies property in YoutubeDL.py). In practice:

    • http_proxy/HTTP_PROXY and https_proxy/HTTPS_PROXY are used for those URL schemes. If only the HTTP one is set, yt-dlp reuses it for HTTPS. Set HTTPS_PROXY=__noproxy__ to undo that.
    • all_proxy/ALL_PROXY is the fallback for any scheme.
    • no_proxy/NO_PROXY is honoured.
    • On macOS and Windows, if no proxy environment variables are set, getproxies() can fall back to the system proxy settings. A desktop VPN or proxy app can therefore change yt-dlp's routing without you noticing.
    • Any --proxy value replaces the environment completely, including --proxy "".
    export HTTPS_PROXY="http://USER:[email protected]:8080"
    export HTTP_PROXY="$HTTPS_PROXY"
    yt-dlp URL

    Config files

    Any CLI option can go into a config file. yt-dlp searches, in order: the file given to --config-locations, a portable yt-dlp.conf next to the binary, yt-dlp.conf in the -P home path (or the current directory), the user config, and the system config. Recommended user locations are ${XDG_CONFIG_HOME}/yt-dlp/config on Linux/macOS and ${APPDATA}/yt-dlp/config on Windows. ~/yt-dlp.conf, ~/.yt-dlp/config and /etc/yt-dlp.conf also work.

    # ~/.config/yt-dlp/config
    --proxy "http://USER:[email protected]:8080"
    --socket-timeout 30
    --retries 10
    --fragment-retries 10

    Two gotchas. Config lines are parsed like a shell, so quote values containing #. And --ignore-config on the command line skips all of these files. That helps when you're debugging which proxy is actually in effect. yt-dlp -v prints which config files were loaded.

    Using a proxy from the Python API

    The option keys match the CLI flags: proxy, socket_timeout, retries, fragment_retries, source_address. One difference matters a lot: according to the docstrings, retries and fragment_retries default to 0 in the API but 10 in the CLI. Set them explicitly.

    import yt_dlp
    
    opts = {
        "proxy": "http://USER:[email protected]:8080",
        "socket_timeout": 30,
        "retries": 10,
        "fragment_retries": 10,
        "outtmpl": "data/%(id)s.%(ext)s",
    }
    
    with yt_dlp.YoutubeDL(opts) as ydl:
        info = ydl.extract_info("https://www.youtube.com/watch?v=VIDEO_ID", download=True)
        print(info["id"], info.get("title"))

    Check your exit IP

    Before blaming YouTube, confirm the proxy works and see which IP you're actually exiting from. With curl:

    curl -x "http://USER:[email protected]:8080" https://api.ipify.org
    curl -x "socks5h://USER:[email protected]:1080" https://api.ipify.org

    A better check goes through yt-dlp's own networking stack, so the proxy parsing, handlers and environment handling are the same ones your downloads use:

    import yt_dlp
    
    with yt_dlp.YoutubeDL({"proxy": "http://USER:[email protected]:8080"}) as ydl:
        print(ydl.urlopen("https://api.ipify.org").read().decode())

    Run it twice in a row. If the IP changes between calls, your endpoint rotates per request or per connection. That matters for the next section.

    Sticky vs rotating sessions: keep one video on one IP

    Downloading a YouTube video takes several requests: the watch page and player API calls (metadata, format list, signed media URLs), then many range requests to googlevideo.com for the media. The yt-dlp FAQ puts it this way: "In many cases, requests for the video (to download/play it) must come from the same IP address and with the same cookies and/or HTTP headers." Google doesn't document YouTube's signed media URLs, so the exact binding rules are an observed behaviour rather than a contract. The practical rule is simple, though: if extraction happens on IP A and the media download on IP B, expect intermittent 403s. The PO Token Guide adds another binding: most PO tokens are tied to a session or a video ID, which is another reason to keep each video's requests together.

    Proxies come in two broad modes:

    • Rotating per request or connection: every new TCP connection may exit from a different IP. That works for stateless page fetches and is risky for yt-dlp, because one download opens several connections.
    • Sticky sessions: the provider pins you to one exit IP for a set time, usually selected by a session ID in the proxy username. The exact syntax is provider-specific, so check your provider's docs. Static or dedicated IPs behave like a permanently sticky session.

    The pattern that works: one sticky session per video (or per worker, for a batch of videos), long enough to cover extraction and the full media download. Then move on to a new session. Rotation still spreads load across the pool; it just happens between videos, not inside one.

    Running many workers, each with its own proxy session

    Run a separate YoutubeDL instance per job, in separate processes, and derive a session ID from the video so retries of the same video get the same IP. The username format below is illustrative; substitute your provider's sticky-session syntax.

    import hashlib
    import urllib.parse
    from concurrent.futures import ProcessPoolExecutor
    
    import yt_dlp
    
    PROXY_USER = "USER"
    PROXY_PASS = urllib.parse.quote("PASS", safe="")
    PROXY_HOST = "proxy.example.com:8080"
    
    
    def proxy_for(url: str) -> str:
        session = hashlib.sha1(url.encode()).hexdigest()[:12]
        # Sticky-session username syntax varies by provider; this format is illustrative.
        return f"http://{PROXY_USER}-session-{session}:{PROXY_PASS}@{PROXY_HOST}"
    
    
    def fetch(url: str) -> tuple[str, int]:
        opts = {
            "proxy": proxy_for(url),
            "socket_timeout": 30,
            "retries": 5,
            "fragment_retries": 5,
            "outtmpl": "data/%(id)s.%(ext)s",
            "download_archive": "archive.txt",
            "quiet": True,
            "noprogress": True,
        }
        try:
            with yt_dlp.YoutubeDL(opts) as ydl:
                return url, ydl.download([url])
        except yt_dlp.utils.DownloadError:
            return url, 1
    
    
    if __name__ == "__main__":
        with open("urls.txt") as f:
            urls = [line.strip() for line in f if line.strip()]
        with ProcessPoolExecutor(max_workers=8) as pool:
            for url, code in pool.map(fetch, urls):
                print(code, url)

    With a list of static proxies instead, assign one proxy per worker. From the shell, a file with one URL PROXY pair per line plus xargs does the same job:

    xargs -P 8 -L 1 sh -c 'yt-dlp --proxy "$1" -o "data/%(id)s.%(ext)s" -- "$0"' < jobs.txt

    The shared archive.txt above is fine for a single machine. Across machines, use a real job queue. Concurrency per IP matters as much as the total: many parallel videos behind one exit IP is how IPs get flagged. Pacing and IP reputation are covered in fixing "Sign in to confirm you're not a bot", and sizing the total pipe in bandwidth for large-scale video downloads.

    aria2c as the external downloader

    --downloader aria2c hands HTTP/FTP downloads to aria2c. In downloader/external.py, yt-dlp passes your --proxy value to aria2c as --all-proxy. It also sets -x16 -s16 --min-split-size 1M by default, then appends whatever you give --downloader-args.

    yt-dlp --proxy "http://USER:[email protected]:8080" \
      --downloader aria2c \
      --downloader-args "aria2c:-x 4 -s 4 --connect-timeout=15 --timeout=30 --max-tries=5" \
      "https://www.youtube.com/watch?v=VIDEO_ID"

    Three caveats:

    • HTTP proxies only. The aria2 manual gives the proxy format as [http://][USER:PASSWORD@]HOST[:PORT]. aria2c doesn't speak SOCKS. With a SOCKS proxy, use the native downloader. (ffmpeg has the same limitation: yt-dlp warns that it "does not support SOCKS proxies".)
    • 16 connections per file. On a per-connection rotating proxy, each segment may come from a different IP, which brings back the 403 problem. Use a sticky session and fewer connections (-x 4), or skip aria2c.
    • aria2c reads proxy environment variables itself. Only an explicit --proxy is forwarded by yt-dlp, so test the exact configuration you deploy.

    Add -v to see the final command line ([debug] aria2c command line: ...) and confirm which arguments took effect.

    Timeouts and retries

    Flag (API key) Default What it does
    --socket-timeout (socket_timeout) 20 s (DEFAULT_TIMEOUT in the networking layer) How long to wait on an unresponsive connection. Raise it to 30–60 s for slow residential exits.
    -R, --retries (retries) 10 CLI / 0 API Retries for the download on expected network errors. Accepts infinite.
    --fragment-retries (fragment_retries) 10 CLI / 0 API Retries per fragment for DASH, hlsnative and ISM.
    --retry-sleep [TYPE:]EXPR none Backoff between retries, e.g. --retry-sleep exp=1:30 or --retry-sleep fragment:linear=1::2.
    --extractor-retries 3 Retries for known extractor errors.

    Retries go through the same proxy URL. If that URL is a sticky session, they reuse the same IP, which is what you want for a transient error. For a hard block (repeated 403 or 429), retrying on the same IP rarely helps. Fail fast, record the video, and requeue it on a fresh session.

    Choosing a proxy type for this workload

    Video collection is unusual among proxy workloads: few requests, lots of bytes. That changes the trade-offs.

    Type Strengths Weaknesses for yt-dlp
    Datacenter (shared) Cheap, fast, stable IPs Cloud and hosting ranges are the most likely to get bot checks and 403s from YouTube. Shared IPs inherit other users' reputation.
    Residential (rotating) IPs look like home users, many locations Usually billed per GB, which gets expensive at video volumes. Bandwidth and latency vary by peer. Sessions can drop mid-download.
    ISP / static residential Residential-registered IPs on datacenter hardware. Stable, so naturally sticky Fewer IPs per dollar. A flagged IP stays flagged until replaced.
    High-bandwidth dedicated (unmetered, per-Gbps) Predictable cost for terabyte-scale transfer, consistent throughput, capacity reserved for you You pay for the capacity whether you use it or not. IP reputation still depends on the ranges and on how hard you push each IP.

    The test that matters is your own: run a few hundred videos through each candidate, then compare 403/bot-check rates and cost per delivered GB.

    Troubleshooting

    Symptom Likely cause Fix
    407 Proxy Authentication Required Wrong credentials, unencoded special characters in the password, or source-IP allow-listing on the provider side URL-encode the credentials. Test with curl -x. Check the provider's allow-list.
    Unable to connect to proxy / Connection refused Wrong host or port, wrong scheme (SOCKS port with http://), or a firewall Match the scheme to the port the provider documents. Check with curl -x.
    SSL: CERTIFICATE_VERIFY_FAILED or WRONG_VERSION_NUMBER Using https:// for a plain HTTP proxy (or the reverse), or an intercepting proxy Usually you want http://. --no-check-certificates hides the problem instead of fixing it; only use it for debugging.
    Timeouts, Read timed out Slow or overloaded exit, peer dropped Raise --socket-timeout, add --retry-sleep, reduce concurrency per IP.
    HTTP Error 403 on media after extraction succeeded Extraction and download from different IPs (rotating proxy, aria2c multi-connection), missing PO token, outdated yt-dlp Sticky session per video, yt-dlp -U, see the PO Token Guide.
    "Sign in to confirm you're not a bot" IP reputation or request rate See our dedicated guide.
    Proxy seems ignored --proxy in a config file, a system or desktop proxy, or NO_PROXY matching yt-dlp -v --ignore-config, then check the exit IP through ydl.urlopen.

    A note on compliance

    YouTube's Terms of Service restrict automated access and downloading, and a proxy doesn't change that. If you're collecting video for training data, copyright and licensing also matter (for example, filtering for Creative Commons–licensed content), as do privacy obligations for faces and voices (GDPR and similar laws) and a process for honouring takedown and opt-out requests. Get legal review for your jurisdiction and use case.

    If you'd rather not run this yourself

    Corpora offers dedicated 1–40 Gbps unmetered proxy capacity (HTTP(S) and SOCKS5, yt-dlp compatible, rotating pools, multi-region) for teams running their own stack. See pricing. If you only need the end result, Corpora can also collect, deduplicate and deliver video, audio, subtitles and metadata to your object storage. Get in touch for a free sample batch. For the pipeline side, see building a YouTube video dataset for multimodal training and downloading Panda-70M, HD-VILA and InternVid.

    FAQ

    How do I use a SOCKS5 proxy with yt-dlp?

    Pass --proxy "socks5h://USER:[email protected]:1080". yt-dlp treats socks5:// as socks5h://, so DNS goes through the proxy either way. Don't combine SOCKS with --downloader aria2c or ffmpeg, which don't support SOCKS proxies.

    Does yt-dlp use the HTTP_PROXY environment variable?

    Yes, when --proxy isn't given. It reads HTTP_PROXY, HTTPS_PROXY, ALL_PROXY and NO_PROXY (upper or lower case) through Python's urllib. Any --proxy value, including --proxy "", overrides them.

    Why do I get 403 Forbidden when downloading through a rotating proxy?

    Usually because the media requests left from a different IP than the one that fetched the video's metadata and signed URLs. Use a sticky session for the whole video. Also update yt-dlp and check PO-token requirements.

    Can I use a different proxy for each video in yt-dlp?

    Not within a single CLI invocation. --proxy applies to the whole run. Launch one process (or one YoutubeDL instance) per video or per worker, each with its own proxy URL, as in the worker example above.

    How do I put a password with special characters in the yt-dlp proxy URL?

    Percent-encode it, e.g. with urllib.parse.quote(password, safe=""), so that @ becomes %40 and : becomes %3A. Then quote the whole URL in the shell.

  • yt-dlp "Sign in to confirm you're not a bot": How to Fix It

    If yt-dlp suddenly fails with ERROR: [youtube] VIDEO_ID: Sign in to confirm you're not a bot, YouTube has decided that your session or your IP address looks automated. It often shows up together with HTTP 403, HTTP 429 or "This content isn't available, try again later". This guide walks through the fixes in order, cheapest first. It keeps two cases apart: one person on a laptop, and a pipeline doing thousands of downloads per hour from servers.

    Everything below was checked against the yt-dlp README, the official wiki and the source code as of early November 2026, when the latest stable release was 2026.10.26. YouTube changes enforcement often, so treat client names and limits as a snapshot.

    TL;DR

    • Update first. Run yt-dlp -U, or switch to the nightly channel with yt-dlp --update-to nightly. Many "bot" and 403 reports turn out to be outdated installs.
    • Install a JavaScript runtime. YouTube extraction now needs an external JS runtime (Deno is the default) plus the yt-dlp-ejs scripts. Without them, some formats go missing.
    • Slow down. The wiki puts the guest rate limit at about 300 videos/hour per session. -t sleep applies sensible delays for you.
    • PO tokens fix 403s on some clients, not the bot check. Use a PO Token Provider plugin such as bgutil-ytdlp-pot-provider.
    • Cookies work, but they put the account at risk. Fine for one person on a laptop. At pipeline scale, expect accounts to be banned.
    • On servers, IP reputation decides most of the outcome. Datacenter ranges get the bot check much more often than residential IPs. Spread load across IPs and sessions instead of pushing harder through one.

    What the error actually means

    The text comes from YouTube's player response, not from yt-dlp. yt-dlp appends a hint ("Use --cookies-from-browser or --cookies for the authentication") and links the wiki's cookie instructions. Don't misread it: the video doesn't need an account. YouTube wants a session it trusts more than an anonymous request from your IP.

    The related errors have different causes, which is why they need different fixes:

    • "This content isn't available, try again later" means your session hit the rate limit. In current yt-dlp versions, the error says the session "has been rate-limited by YouTube for up to an hour" and suggests -t sleep.
    • HTTP Error 403: Forbidden on the actual media download usually means a missing or invalid PO token for the client in use, an outdated yt-dlp, or a JS challenge that failed. Less often, the IP is blocked.
    • HTTP Error 429: Too Many Requests means your IP is being throttled. The yt-dlp FAQ calls this "usually a soft block".

    Error → cause → fix

    Error message Likely cause First fix to try
    Sign in to confirm you're not a bot Low-reputation IP (often datacenter), or too many requests from one IP or session Update, slow down, and change egress IP. Use cookies only as a last resort (account risk).
    This content isn't available, try again later Session rate limit: about 300 videos/hour for guests, about 2,000/hour for accounts (per the wiki) -t sleep or the sleep flags. Wait up to an hour.
    HTTP Error 403: Forbidden on media/fragments Missing PO token (GVS) for the chosen client, outdated yt-dlp, or a JS challenge that wasn't solved Update, install Deno and EJS, add a PO Token Provider plugin
    HTTP Error 429: Too Many Requests IP throttled for overuse Back off. Use fewer parallel workers per IP. Move to another IP.
    No supported JavaScript runtime could be found (warning) No Deno, or another runtime not enabled Install Deno, or pass --js-runtimes node
    n challenge solving failed: Some formats may be missing EJS scripts missing or outdated, or the runtime is too old Update yt-dlp together with yt-dlp-ejs. Check the runtime version.
    YouTube is requiring a captcha challenge before playback IP or session flagged Stop and rotate the IP. Don't retry in a tight loop.

    Step 1: Diagnose with -v

    Before you change anything, run a single failing URL with verbose output:

    yt-dlp -v "https://www.youtube.com/watch?v=VIDEO_ID"

    Check the [debug] header for the version and channel, the JS runtimes: line, and the detected JS Challenge Providers and PO Token Providers. If the runtimes line says none, or a provider you installed is missing, you've found the problem. Add --print-traffic for the raw HTTP exchange.

    Step 2: Update yt-dlp (and pick the right channel)

    YouTube breaks yt-dlp regularly, and fixes ship fast. The README's update section describes three channels. stable is released on a roughly monthly schedule and is "often 'stale' and prone to external breakage". nightly is the "recommended channel for regular users". master publishes after every push.

    # Official binary: update within the current channel
    yt-dlp -U
    
    # Switch a stable binary to nightly
    yt-dlp --update-to nightly
    
    # pip install: re-run the install, including the default extras (pulls in yt-dlp-ejs)
    python3 -m pip install -U "yt-dlp[default]"
    
    # pip nightly (development releases)
    python3 -m pip install -U --pre "yt-dlp[default]"

    If your distro package is outdated, the FAQ says to install the official release instead. In a pipeline, pin a version in your image but rebuild it often.

    Step 3: Install a JavaScript runtime and the EJS scripts

    This is the newest requirement and a common cause of missing formats and 403s. Per the EJS wiki page, yt-dlp now solves YouTube's JavaScript challenges with an external runtime that runs the yt-dlp-ejs solver scripts. The old built-in interpreter is no longer used for YouTube, and the code now calls extraction without a JS runtime "deprecated".

    • Deno (2.3.0 or newer) is recommended and enabled by default.
    • Node (22.0.0 or newer) works with --js-runtimes node. QuickJS works with --js-runtimes quickjs. Bun support is deprecated.
    • The official binaries (yt-dlp_linux, yt-dlp.exe, the zipimport yt-dlp) bundle yt-dlp-ejs. If you install with pip, use yt-dlp[default]. If your distro package doesn't ship the scripts, add --remote-components ejs:github (or ejs:npm with Deno or Bun).
    # Deno somewhere other than PATH
    yt-dlp --js-runtimes "deno:/opt/deno/bin/deno" URL
    
    # Use Node instead of Deno (clear the defaults first if Deno is also installed)
    yt-dlp --no-js-runtimes --js-runtimes node URL

    The wiki stresses that yt-dlp-ejs must be updated together with yt-dlp, because older versions are ignored. In Docker images, install both in the same layer.

    Step 4: Slow down

    The Extractors wiki gives concrete numbers. With default settings, the limit is "~300 videos/hour (~1000 webpage/player requests per hour)" for guest sessions and "~2000 videos/hour (~4000 webpage/player requests per hour)" for accounts. It recommends a delay of about 5–10 seconds between downloads.

    The easiest option is the built-in preset. -t sleep expands to --sleep-subtitles 5 --sleep-requests 0.75 --sleep-interval 10 --max-sleep-interval 20. The individual flags:

    Flag What it does
    --sleep-requests SECONDS Sleep between requests during extraction (webpage, player API calls)
    --sleep-interval SECONDS (alias --min-sleep-interval) Sleep before each download. Acts as the minimum when combined with the max.
    --max-sleep-interval SECONDS Upper bound for a random sleep. Only valid together with the minimum.
    --sleep-subtitles SECONDS Sleep before each subtitle download
    -r, --limit-rate RATE Cap download bandwidth, e.g. 5M. Makes traffic look less bursty but doesn't change the request count.
    -R, --retries / --retry-sleep Retry count (default 10) and backoff, e.g. --retry-sleep http:exp=1:120

    Throughput math: with -t sleep the per-video sleep averages (10 + 20) / 2 = 15 s, so one worker starts at most 3600 / 15 = 240 videos/hour, under the ~300/hour guest figure. Faster pipelines need more sessions, not less sleep.

    Step 5: PO tokens and player clients (for 403s)

    The PO Token Guide describes a Proof of Origin token that some YouTube clients must send. Without it, "requests for the affected clients' format URLs may return HTTP Error 403, or result in your account or IP address being blocked". Tokens are now bound to the video ID, so pasting a token by hand is no longer recommended. The guide's recommended setup is a PO Token Provider plugin that feeds the mweb client. It features two plugins: bgutil-ytdlp-pot-provider and yt-dlp-getpot-wpc.

    # 1) Run the bgutil token server, bound to loopback only
    docker run --name bgutil-provider -d --init \
      -p 127.0.0.1:4416:4416 brainicism/bgutil-ytdlp-pot-provider
    
    # 2) Install the plugin next to a pip-installed yt-dlp
    python3 -m pip install -U bgutil-ytdlp-pot-provider
    
    # 3) Use it with the mweb client
    yt-dlp -v --extractor-args "youtube:player_client=default,mweb" URL

    In the -v output, look for [pot:bgutil:http] Generating a gvs PO Token. The bgutil README warns that leaving 127.0.0.1 out of the port mapping exposes an unauthenticated server.

    About player_client: in 2026.10.26 the README gives the default as tv_samsung,visionos,web, with a different set when logged-in cookies are passed. Use default,-web syntax to add or exclude clients. The wiki's PO token table predates some of these clients, so check the README for your installed version rather than copying old forum advice.

    An important limit, stated in the bgutil README: a PO token does not bypass IP-based login restrictions. If you still see "Sign in to confirm you're not a bot" after a token was generated, which is common on datacenter IPs, the problem is your IP or session, not the token.

    Step 6: Cookies (laptop yes, pipeline carefully)

    For a single user, logged-in cookies are the quickest way past the bot check. The Extractors wiki has a caution box: "By using your account with yt-dlp, you run the risk of it being banned (temporarily or permanently)." It suggests a throwaway account. YouTube rotates cookies on open tabs, so the wiki's method for long-lived cookies is this: log in from a private window, open https://www.youtube.com/robots.txt in that same tab, export the youtube.com cookies with an extension, then close the window.

    # Laptop: read cookies straight from a browser profile
    yt-dlp --cookies-from-browser firefox URL
    
    # Exported Netscape-format file
    yt-dlp --cookies /secure/path/cookies.txt URL

    Two practical points. First, the wiki says not to export with --cookies-from-browser BROWSER --cookies FILE for this purpose, because that exports your regular browser cookies, not the private session. Second, --cookies FILE both reads and writes the file. If you share one file across parallel workers, they share one session, write over each other's changes, and burn through that account's ~2,000 videos/hour budget together.

    At pipeline scale, don't build on accounts. Accounts get a higher rate limit, but every ban costs you one. Use cookies to retry the occasional login-gated video, not as your main path.

    Step 7: IP reputation, proxies and spreading load

    This is the step laptop guides skip. YouTube judges requests partly by source IP, and cloud and hosting ranges carry far more automated traffic than home connections. They hit the bot check much sooner, so a command that works at home can fail immediately on a fresh VM. Google publishes no thresholds, so test on your own ranges.

    yt-dlp's tools for this are --proxy (HTTP, HTTPS or SOCKS) and --source-address, which binds to a specific local IP if your host has several. Use -4 or -6 to pin the address family.

    # HTTP proxy with auth
    yt-dlp --proxy "http://USER:[email protected]:8080" URL
    
    # SOCKS5 (note the scheme)
    yt-dlp --proxy "socks5://USER:[email protected]:1080" URL

    For a full walkthrough of proxy types, authentication and rotation patterns, see our yt-dlp proxy guide.

    A pattern for thousands of downloads per hour

    1. One worker = one egress IP = one guest session. Give each worker its own proxy endpoint or sticky session and its own cookie jar (or none). Don't share state between workers.
    2. Size the pool with arithmetic, not hope. With -t sleep a worker waits 10–20 s (15 s on average) before each download, so it tops out at about 3,600 ÷ 15 = 240 videos/hour before counting the download time itself. Using that ceiling, a target of 5,000 videos/hour needs at least 5000 / 240 ≈ 21 workers, each on separate IPs. Plan for more, since some IPs will be cooling down at any moment. For bandwidth sizing, see our bandwidth guide.
    3. Treat bot and rate-limit errors as signals, not retryable failures. When an IP returns the bot check or a 429, take it out of rotation for at least an hour. Hammering it with retries extends the block.
    4. Keep the toolchain current everywhere. yt-dlp, yt-dlp-ejs, Deno and the PO token plugin should be rebuilt together on a regular schedule.
    5. Make runs resumable. Use --download-archive so a rotated or restarted worker skips finished IDs.
    # yt-dlp.conf for one pipeline worker
    --proxy http://USER:[email protected]:8080
    --sleep-requests 1
    --sleep-interval 10
    --max-sleep-interval 20
    --sleep-subtitles 5
    --retries 5
    --retry-sleep http:exp=1:120
    --extractor-retries 3
    --download-archive /data/archive.txt
    -o /data/%(id)s.%(ext)s

    For the surrounding pipeline (manifests, deduplication, metadata), see how to build a YouTube video dataset for multimodal training and our notes on downloading Panda-70M, HD-VILA and InternVid.

    Troubleshooting order, cheapest first

    1. Run -v on one URL and read the debug header.
    2. Update yt-dlp (-U or --update-to nightly) and yt-dlp-ejs.
    3. Install Deno (or enable Node/QuickJS) and confirm the JS runtimes: line.
    4. Add -t sleep and cut parallelism per IP.
    5. Install a PO Token Provider plugin. Try player_client variations if 403s persist.
    6. Laptop: pass cookies from a throwaway account. Pipeline: move to cleaner IPs and spread load across more of them.
    7. Still stuck on the latest nightly? Check the pinned known-issues thread and the issue tracker before you file a report with full -v output.

    A note on compliance

    YouTube's Terms of Service restrict automated access and downloading. Building a training set also involves copyright and licensing (for example, filtering for Creative Commons content), privacy rules for faces and voices (such as GDPR), and honoring takedown and opt-out requests. Get legal review for your jurisdiction and use case.

    If you'd rather not run this yourself

    Corpora offers dedicated 1–40 Gbps unmetered proxy capacity (HTTP(S) and SOCKS5, rotating pools, multi-region) that drops into the --proxy setup above, at $2,000 per Gbps per month. If you'd rather skip the infrastructure, we can also collect and deliver video, audio, subtitles and metadata straight to your object storage. Get in touch for a free sample batch.

    FAQ

    Why does yt-dlp say "Sign in to confirm you're not a bot" on my server but not on my laptop?

    The message comes from YouTube and depends heavily on the source IP. Datacenter and cloud IPs are flagged far more readily than residential connections. Updating and slowing down help, but on servers the lasting fix is better IPs plus spreading load across them.

    Does a PO token fix "Sign in to confirm you're not a bot"?

    Usually not. PO tokens fix 403s on clients that need them. Per the bgutil README, a token "does not bypass IP-based login restrictions".

    Do I need Deno for yt-dlp now?

    For YouTube you need a supported JavaScript runtime. Deno is the default. Node and QuickJS work with --js-runtimes.

    Is using cookies with yt-dlp safe?

    The wiki warns the account can be banned temporarily or permanently. Use a throwaway account and low request rates.

    How long does the "This content isn't available, try again later" block last?

    yt-dlp's own message says the session is rate-limited "for up to an hour". Wait, then resume with -t sleep or equivalent delays so you stay under the roughly 300 videos/hour guest limit.

  • Bandwidth for Large-Scale Video Downloads: 100 TB to 5 PB/Month

    If you need to collect 100 TB, 1 PB or 5 PB of video or audio per month, the first question is not which tool to use. It is how many gigabits per second you need to keep running all month. This guide gives you the exact formula, sizing tables for 10 TB to 5 PB per month, the reverse table (what 1–40 Gbps can actually move), and the practical limits that decide whether you hit those numbers.

    TL;DR

    • Formula: Gbps = TB/month × 8,000 ÷ seconds per month. With a 30-day month (2,592,000 s), 1 Gbps at 100% utilization moves 324 TB/month, or 10.8 TB/day.
    • 100 TB/month ≈ 0.31 Gbps, 1 PB/month ≈ 3.09 Gbps, 5 PB/month ≈ 15.4 Gbps. That assumes the link is saturated 24/7, and it won't be.
    • Plan for 60–70% sustained utilization. That means provisioning about 1.4–1.7× the theoretical number.
    • 800 TB/month ≈ 2.47 Gbps at 100%, so provision about 4–5 Gbps.
    • Writing data into S3 or GCS from the internet is free. Reading it back out to the internet is not.
    • Per-GB pricing only makes sense at low volume. At $2,000 per Gbps per month, the break-even is $2,000 ÷ (price per GB) GB per month.

    The formula

    Bandwidth is measured in bits per second, but storage is measured in bytes. You convert between them, then divide by the number of seconds in the period:

    Gbps = TB_per_month × 8,000 / seconds_per_month
    
    # 1 TB (decimal) = 10^12 bytes = 8 × 10^12 bits = 8,000 Gb
    # 30-day month    = 30 × 86,400   = 2,592,000 s
    # average month   = 30.44 × 86,400 ≈ 2,630,016 s  (365.25 / 12 days)

    This article uses the 30-day month. It is the more conservative choice, because fewer seconds means a slightly higher Gbps figure. The 30.44-day average lowers every result by about 1.5% (100 TB/month comes out at 0.304 Gbps instead of 0.309). For capacity planning that difference is smaller than your error bars.

    TB vs TiB

    Network speeds are always decimal: 1 Gbps = 10^9 bits per second. Storage tools often report binary units even when they label them "TB". Check whether df -h, your object-store console or your billing export means TB or TiB:

    • 1 TB = 10^12 bytes; 1 TiB = 2^40 bytes ≈ 1.0995 TB (about 10% larger).
    • 1 PB = 10^15 bytes; 1 PiB = 2^50 bytes ≈ 1.126 PB (about 12.6% larger).
    • 100 TiB/month needs 0.339 Gbps, compared with 0.309 Gbps for 100 TB. 1 PiB/month needs 3.47 Gbps, compared with 3.09 Gbps for 1 PB.

    If your target is set in TiB or PiB, convert it to decimal before you use the tables below.

    Monthly volume → required bandwidth

    The "100%" column is pure arithmetic. The provisioned columns divide it by 0.70 and 0.60, which are the sustained utilization levels you can realistically plan around (the next section explains why). All values use a 30-day month and decimal units.

    Volume / month Per day Gbps at 100% Provision at 70% util. Provision at 60% util.
    10 TB 0.33 TB 0.031 0.04 Gbps 0.05 Gbps
    50 TB 1.67 TB 0.154 0.22 Gbps 0.26 Gbps
    100 TB 3.33 TB 0.309 0.44 Gbps 0.51 Gbps
    250 TB 8.33 TB 0.772 1.10 Gbps 1.29 Gbps
    500 TB 16.7 TB 1.543 2.20 Gbps 2.57 Gbps
    800 TB 26.7 TB 2.469 3.53 Gbps 4.12 Gbps
    1 PB 33.3 TB 3.086 4.41 Gbps 5.14 Gbps
    2 PB 66.7 TB 6.173 8.82 Gbps 10.29 Gbps
    5 PB 166.7 TB 15.432 22.05 Gbps 25.72 Gbps

    Round up to whatever increments your provider sells. For example, 1 PB/month lands at 5 Gbps provisioned, 2 PB/month at 10 Gbps, and 5 PB/month at 25 Gbps.

    Why plan for 60–70% utilization, not 100%

    The 100% number assumes every second of the month moves useful bytes at line rate. In practice, several things get in the way:

    • Protocol overhead. Ethernet, IP and TCP headers, TLS records and HTTP framing all use bandwidth on the wire but don't count toward the bytes stored on disk. Proxy hops add their own overhead on top.
    • Retries and failed or partial downloads. A download that dies at 80% and restarts from zero (or re-fetches fragments) still used bandwidth. Expired stream URLs, bot checks (see fixing "Sign in to confirm you're not a bot") and connection resets all show up as wasted bytes.
    • Non-media traffic. Metadata extraction, manifest and player requests, subtitle fetches and format probing all use requests and time without adding much volume.
    • Per-connection throughput limits. A single stream rarely fills a fast link. Unless you keep enough downloads in flight at all times, the pipe sits partly idle.
    • Diurnal and operational gaps. Queue refills, deploys, worker restarts, rate-limit backoff, proxy rotation and the daily traffic patterns of shared upstreams all create troughs. A 24-hour average is always below the peak.

    Sustained 60–70% utilization is a reasonable target for a well-run pipeline. It is a planning assumption, not a measured guarantee. If your own monitoring shows 80% sustained, you can provision closer to the 100% column. If you are running at 40%, fix the pipeline before you buy more bandwidth.

    Worked example: 800 TB per month

    800 × 8,000 / 2,592,000 = 2.469 Gbps   (at 100% utilization, 30-day month)
    800 × 8,000 / 2,630,016 = 2.433 Gbps   (30.44-day average month)
    
    2.469 / 0.70 = 3.53 Gbps
    2.469 / 0.60 = 4.12 Gbps
    → provision ~4–5 Gbps
    
    At 4 Gbps provisioned, 800 TB needs 2.469 / 4 = 62% sustained utilization.
    At 5 Gbps provisioned, it needs 2.469 / 5 = 49%, which leaves room for re-downloads and growth.

    Bandwidth → monthly volume (reverse table)

    This table answers the opposite question: you already have a link, so how much can it move? The formula is TB = Gbps × seconds / 8,000.

    Link Per day at 100% Per 30-day month at 100% Per 30.44-day month at 100% Per 30-day month at 65%
    1 Gbps 10.8 TB 324 TB 329 TB 211 TB
    2 Gbps 21.6 TB 648 TB 658 TB 421 TB
    5 Gbps 54 TB 1.62 PB 1.64 PB 1.05 PB
    10 Gbps 108 TB 3.24 PB 3.29 PB 2.11 PB
    20 Gbps 216 TB 6.48 PB 6.58 PB 4.21 PB
    40 Gbps 432 TB 12.96 PB 13.15 PB 8.42 PB

    A useful rule of thumb: 1 Gbps sustained ≈ 10.8 TB/day ≈ 324 TB per 30 days. Everything else scales linearly from there.

    What that volume means in hours of media

    Training-data budgets are usually set in hours, not terabytes. To convert between them you need an average bitrate:

    GB per hour      = Mbps × 3,600 / 8 / 1,000  = Mbps × 0.45
    hours per PB     = 1,000,000 / (GB per hour)

    Delivered bitrates on YouTube vary a lot with codec (H.264/AVC, VP9, AV1), frame rate and content. Static talking heads compress far better than sports. The figures below are approximate planning assumptions, not published specifications. For reference, YouTube's recommended upload encoding settings list 8 Mbps for 1080p SDR at 24–30 fps and 12 Mbps at 48–60 fps. Those are upload targets; the re-encoded streams YouTube delivers are usually well below them.

    Format (approximate) Assumed bitrate GB per hour Hours per 100 TB Hours per 1 PB
    Audio only, AAC (m4a) ~128 kbps 0.128 Mbps 0.058 ~1.74 M ~17.4 M
    Audio only, Opus (webm) ~160 kbps 0.16 Mbps 0.072 ~1.39 M ~13.9 M
    720p video + audio 1.5 Mbps 0.675 ~148 k ~1.48 M
    1080p, efficient codec (VP9/AV1) 2 Mbps 0.9 ~111 k ~1.11 M
    1080p, H.264 or high-motion 4 Mbps 1.8 ~56 k ~556 k
    2160p (4K) 15 Mbps 6.75 ~15 k ~148 k

    The format choice can change your bandwidth bill by an order of magnitude. Audio-only collection for speech models needs roughly 1/12th to 1/30th of the bandwidth of 1080p video for the same number of hours. Before you size anything, measure your actual mix on a sample with yt-dlp. -O/--print implies simulate mode, so nothing is downloaded:

    yt-dlp -f "bv*[height<=1080]+ba/b[height<=1080]" \
      -O "%(id)s %(duration)s %(tbr)s %(filesize,filesize_approx)s" \
      --batch-file sample_urls.txt > sample_sizes.txt

    tbr is the total average bitrate in kbps, and filesize or filesize_approx is in bytes (it may print NA when unknown). Use -F to see every available format for a single video. Multiply the sample's mean bytes per second of duration by your target hours to get TB, then use the first table. For format selection and proxy flags, see the yt-dlp proxy guide.

    Storage, ingress and egress costs

    • Ingress is free on the big clouds. The Amazon S3 pricing page lists "data transferred in from the internet" among the transfers you don't pay for. Google's Cloud Storage pricing lists inbound data transfer as free. You still pay per-request charges (PUT/POST, multipart parts), so avoid uploading millions of tiny objects.
    • Egress is charged. S3 includes 100 GB/month of internet egress free, aggregated across AWS services, and charges per GB beyond that. For GCS general network usage, internet egress starts at $0.12/GiB for the first 10 TiB to worldwide destinations (excluding Asia and Australia) and steps down at volume. Moving a petabyte out of a cloud to train elsewhere costs a lot, so decide early where training will run.
    • Storage grows linearly. At 1 PB/month you add 12 PB a year before replicas. Multiply by your per-GB-month rate and storage class, and budget for lifecycle rules (cold tiers for raw downloads, hot tiers for curated shards).
    • Downloading on a cloud VM means your proxy or provider traffic arrives as VM ingress, which is free on AWS and GCP. But the traffic between your VM and an external proxy is outbound requests plus inbound responses, so check your provider's NAT gateway and inter-zone charges too.

    Per-GB vs per-Gbps pricing: break-even

    Proxy and transit capacity is sold in one of two ways: metered (per GB transferred) or flat (per Gbps of committed capacity, unmetered). Using $2,000 per Gbps per month as the flat price, the break-even is:

    break_even_GB_per_month = flat_price_per_Gbps / price_per_GB
                            = 2,000 / price_per_GB
    
    1 Gbps can move at most 324,000 GB per 30 days (~210,000 GB at 65% utilization).

    The per-GB prices below are illustrative examples, not quotes from any provider:

    Example per-GB price Break-even volume per Gbps Cost of 100 TB metered Cost of 1 PB metered
    $0.10 / GB 20 TB/month $10,000 $100,000
    $0.50 / GB 4 TB/month $50,000 $500,000
    $1.00 / GB 2 TB/month $100,000 $1,000,000
    $3.00 / GB 0.67 TB/month $300,000 $3,000,000

    Even at $0.10/GB, metered pricing costs more than a flat 1 Gbps line once you move more than 20 TB/month, which is about 6% of that line's capacity. At the 100 TB to 5 PB volumes in this article, per-GB pricing is only reasonable for small, hard-to-reach slices of traffic. It doesn't work for the bulk of the download volume.

    Practical limits that decide whether you hit the number

    Parallelism

    Required concurrency follows directly from per-connection throughput:

    concurrent_downloads = target_Gbps × 1,000 / per_connection_Mbps
    
    e.g. 2.5 Gbps at 20 Mbps per download  → 125 downloads in flight
         15 Gbps  at 20 Mbps per download  → 750 downloads in flight

    The 20 Mbps figure is only an example. Measure your own per-download throughput through your proxies, because server-side pacing, proxy hops and round-trip time all cap a single stream. TCP itself sets a ceiling of window ÷ RTT: a 64 KiB window at 100 ms RTT tops out around 5.2 Mbps. yt-dlp's -N/--concurrent-fragments parallelizes fragments within one DASH/HLS download, but at petabyte scale most of the parallelism comes from running many independent downloads across many workers and IPs.

    Disk write throughput

    Divide Gbps by 8 to get the sustained write rate: 2.5 Gbps ≈ 312 MB/s, 15.4 Gbps ≈ 1.93 GB/s, and that is around the clock. If you download separate video and audio streams and merge them with ffmpeg locally, every byte is written at least twice and read once. Plan for roughly 2–3× the network rate in disk I/O, or stream finished files directly to object storage and keep only scratch space local.

    NIC and host limits

    A 10 GbE NIC can't carry 10 Gbps of useful payload once you subtract overhead. TLS termination, Python processes per download, ffmpeg remuxing and checksum computation all use CPU. File descriptors, ephemeral ports and conntrack tables run out at thousands of concurrent connections, so raise ulimit -n and check kernel limits before blaming the network.

    Why one fast server is not enough

    A single 25 or 40 Gbps box is a single point of failure, a single egress IP set (which gets rate-limited or challenged as one unit), and a single disk and CPU bottleneck. Horizontal scaling works better: many workers of a few Gbps each, pulling from a shared queue, uploading to object storage, and spread across regions and IP pools. If one node dies, you lose a fraction of throughput instead of the whole month's schedule.

    Monitor utilization, not just "is it running"

    • Track bytes stored per hour against the target rate (for 1 PB/month that is about 1.39 TB/hour), not just link utilization. The gap between the two is your waste.
    • Graph link utilization per worker and in aggregate, and alert on sustained drops below your planning assumption.
    • Track the retry rate, failure reasons (HTTP 403/429, bot checks, timeouts) and the bytes discarded from partial downloads.
    • Watch the queue depth so workers never idle waiting for URLs.

    Compliance

    YouTube's Terms of Service restrict automated access and downloading. Copyright and licensing matter: many teams filter for Creative Commons-licensed videos. Privacy also matters, since faces and voices can be personal data under GDPR, and so does honoring takedown and opt-out requests. Get legal review for your jurisdiction and use case before you collect at scale. For dataset design, see building a YouTube video dataset for multimodal training and downloading Panda-70M, HD-VILA and InternVid.

    If you'd rather not run this yourself

    Corpora sells dedicated 1–40 Gbps unmetered proxy capacity (HTTP(S) and SOCKS5, compatible with yt-dlp and custom crawlers) at $2,000 per Gbps per month, with volume discounts from 5 Gbps. See pricing. If you only want the data, the fully managed option delivers deduplicated video, audio, subtitles and metadata straight to your S3, GCS or Azure bucket, priced per delivered PB. Get in touch for a free sample batch.

    FAQ

    How many Gbps do I need for 1 PB per month?

    1 PB × 8,000 ÷ 2,592,000 s ≈ 3.09 Gbps at 100% utilization over a 30-day month. At a realistic 60–70% sustained utilization, provision about 4.4–5.1 Gbps. In practice that means 5 Gbps.

    How much data can a 1 Gbps connection download in a month?

    About 324 TB in 30 days at 100% utilization (10.8 TB/day), or about 329 TB over an average 30.44-day month. At 65% sustained utilization, expect about 210 TB.

    How much bandwidth do I need for 100 TB per month?

    About 0.31 Gbps sustained. At 60–70% utilization, provision 0.44–0.51 Gbps, so a 1 Gbps line has plenty of headroom.

    Does uploading to S3 or Google Cloud Storage cost bandwidth fees?

    No. Data transfer in from the internet is free on both S3 and GCS according to their pricing pages. Requests and storage are billed, and so is data transfer out to the internet.

    Should I size in TB or TiB?

    Use decimal TB to match network units (Gbps is decimal). If your storage reports TiB, multiply by 1.0995 before you use the formula. A 100 TiB target needs about 10% more bandwidth than 100 TB.