Tag: throughput

  • Why Are YouTube Downloads So Slow? A Diagnosis Guide

    You provisioned 10 Gbps, but yt-dlp is crawling along at a few megabytes per second, and you cannot tell whether the problem is YouTube, your proxy, your disk or your pipeline. This guide walks through the usual causes in the order they actually cost you throughput, and gives you a measurement for each one before you buy more bandwidth.

    TL;DR

    • One download is one connection, and one connection rarely fills a fast link. Use -N and run many videos in parallel before you blame your provider.
    • Measure aggregate throughput, not a single file. A 3 MB/s file next to a link at 40% utilization says nothing about your capacity.
    • Most "slow" pipelines are request-bound, not bandwidth-bound. Metadata, format probing and throttling retries cost time that no amount of Gbps fixes.
    • Test direct versus proxied on the same file. That single comparison separates network-path problems from YouTube-side problems.
    • Check the local machine last but seriously. A single spinning disk, a nearly full disk, or CPU-bound merging can cap you below your network.

    Step 0: measure something that means something

    Before changing flags, know what you are measuring. Three numbers matter, and they are easy to confuse:

    Metric How to get it What it tells you
    Per-file speed curl -o /dev/null -w '%{speed_download}\n' URL for one media URL, or watch yt-dlp's progress at the end of one file The ceiling of a single stream. Usually far below your link.
    Aggregate throughput ifstat, nload, sar -n DEV 1, or your provider's usage graph, sampled while the pipeline runs What you are actually using. Compare it to your provisioned capacity.
    Requests per second Count log lines or completed items per minute Whether you are bandwidth-bound or request-bound.

    If aggregate throughput sits near your link size, you are bandwidth-bound and the answer is more capacity. If it sits at 10–20% while your workers look busy, the bottleneck is almost always serial work, per-connection limits or retries, not bandwidth.

    Watch the units. Downloads are reported in bytes, links in bits. 8 MB/s is 64 Mbps, and a link you call "1 Gbps" moves at most about 125 MB/s before protocol overhead. A 10-minute 1080p video is roughly 150–250 MB, so a single such file at 8 MB/s takes around 25–30 seconds.

    Cause 1: one stream does not fill one link

    YouTube serves media over its CDN in chunks, and a single connection tends to settle well below line rate, especially on long-distance paths. This is why a fast pipe can look idle.

    Two fixes, in order of impact:

    # Ask yt-dlp to fetch several fragments of one file concurrently
    yt-dlp -N 8 "https://www.youtube.com/watch?v=VIDEO_ID"
    
    # And run many videos at once instead of one after another
    # (GNU parallel shown; any queue works)
    parallel -j 16 'yt-dlp -N 8 -o "/data/%(id)s.%(ext)s" {}' ::: $(cat ids.txt)

    -N (or --concurrent-fragments) only helps when the format is served in fragments, which is normal for YouTube. Push it to 4–8 per file and compare. Beyond that you mostly add overhead and make rate limiting more likely.

    Parallelism across videos multiplies this. The total you can reach is roughly concurrent downloads × per-stream speed, until you hit a real limit: the proxy, the disk, the CPU or YouTube's tolerance for your source IP.

    Cause 2: your pipeline is serial, not the network

    At scale the wall-clock cost is dominated by the number of round trips per item, not by bytes. Each video means a watch page fetch, player response, format probing, then media requests, then postprocessing. If one worker handles them one at a time, it spends most of its life waiting.

    Checks:

    • How many downloads are actually in flight at any moment? Sample ps or your queue's active count. If the answer is "one per worker" and workers are few, that is your bottleneck.
    • Are you re-running yt-dlp as a fresh process per ID? Process start and import costs are small per file but add up over hundreds of thousands.
    • Are you sleeping more than you intended? -t sleep expands to --sleep-subtitles 5 --sleep-requests 0.75 --sleep-interval 10 --max-sleep-interval 20. With that preset a worker waits 10–20 s between items and tops out around 240 videos per hour before download time.

    For long runs, driving yt-dlp through its Python API in a long-lived worker, or at least batching with --batch-file, avoids repeated startup and lets you keep a steady number of items in flight.

    Cause 3: throttling and bot checks disguised as slowness

    The classic failure mode is a pipeline that "works" but gets slower every hour. That is usually not congestion. It is YouTube pushing your source IP toward stricter paths: fewer formats offered, more retries, occasional 403 on media URLs that then get re-requested from the start.

    Symptoms to look for in the verbose output:

    yt-dlp -v --no-progress "https://www.youtube.com/watch?v=VIDEO_ID" 2>&1 | tail -40
    • Repeated "Unable to download fragment" or 403 lines followed by a restart of the same file.
    • Formats that were available an hour ago now missing, so yt-dlp falls back to a muxed or lower-quality stream.
    • A growing number of "Sign in to confirm you're not a bot" exits. If so, that is a separate problem with its own fixes — see fixing "Sign in to confirm you're not a bot".

    Retries are pure waste: a file that dies at 80% and restarts from zero burned 1.8× the bandwidth for the same output. Adding --retries, --fragment-retries and --continue does not fix the cause, but it stops the waste from compounding.

    Cause 4: the wrong format is expensive in three places

    Format choice affects speed through download size, merge work and decode cost:

    Choice Why it can be slow What to do
    Highest available resolution (4K/8K, AV1 or VP9) Several times the bytes, and a heavy CPU merge or transcode step Cap it: -f "bv*[height<=1080]+ba/b"
    Muxed (progressive) formats Only offered at low resolutions, often served through slower paths Prefer separate video and audio streams, then merge
    Converting everything to MP3 or a fixed resolution at download time Transcoding is CPU-bound and can crush your throughput well below network speed Keep the native stream, convert later on a separate, parallelizable stage

    Use -F on a sample to see what is actually on offer, and -O to print sizes without downloading:

    yt-dlp -O "%(id)s %(format_id)s %(height)s %(tbr)s kbps %(filesize,filesize_approx)s bytes" URL

    If merging is your bottleneck, watch CPU during the run. A worker at 100% CPU on ffmpeg is not downloading anything.

    Cause 5: the proxy path is the bottleneck

    Proxies help you avoid per-IP limits, but they can also silently cap you. Common causes: a provider enforcing a per-account throughput limit, a busy shared exit node, a long network path to the exit, or a proxy that terminates TLS inefficiently.

    Isolate it with one test:

    # Same file, same machine, same time of day
    curl -o /dev/null -w 'direct  %{speed_download} B/s\n' MEDIA_URL
    curl -x "http://USER:[email protected]:8080" -o /dev/null -w 'proxied %{speed_download} B/s\n' MEDIA_URL

    If the proxied number is a fraction of the direct one on a single stream, retest with several concurrent streams through the proxy. A slow single stream with good aggregate throughput means the issue is per-connection latency, not capacity. Slow aggregate means the proxy path is genuinely capped.

    Also check the exit's network family and location. An exit several continents away adds round-trip time to every fragment request, and IPv6 versus IPv4 paths to the CDN can behave differently. For flag-level details, see the yt-dlp proxy guide.

    Cause 6: the local machine

    It is boring and it is frequently the answer:

    • Disk write throughput. Many small concurrent writes on one spinning disk, or on a network volume, cap far below the network. Check with iostat -x 1 and look for high %util and wait time.
    • Disk space. Above roughly 90% utilisation, filesystem behaviour degrades and writes stall.
    • Container and network stack limits. Overlay filesystems and low default buffer sizes cost throughput. Check ss -s for retransmits and socket errors.
    • CPU. Merge, extract, convert and hashing all compete with the network path. Pin the postprocessing stage to separate workers.
    • Ephemeral port and connection limits. Thousands of short connections can exhaust the ephemeral port range. Check net.ipv4.ip_local_port_range and TIME_WAIT counts.

    A symptom-to-cause table

    Symptom Most likely cause First check
    Single file at 5–10 MB/s while the link is idle Per-connection limit Add -N 8, compare
    Aggregate throughput flat regardless of worker count Shared cap: proxy, disk or NIC Raise workers 2× and watch aggregate
    Throughput drops over hours, retries climb Rate limiting / IP reputation Read -v output; rotate or slow down
    Fast network, low completions per minute Serial pipeline, too much sleep Count in-flight downloads
    Worker pegged at 100% CPU Merge or transcode Move postprocessing out of the downloader
    Download finishes then the file write stalls Disk iostat, free space

    What "fast enough" looks like

    Size the pipeline against the number you need, not against a benchmark you saw once. 1 Gbps sustained moves about 324 TB per month, or roughly 10.8 TB per day. If you need 100 TB per month, you need about 0.31 Gbps of sustained throughput, which in practice means provisioning more and keeping enough downloads in flight to use it. The arithmetic, including the utilization assumptions, is in bandwidth for large-scale video downloads.

    A practical target for a well-run collector is a steady 60–70% of provisioned capacity all day. If you cannot get past 20% no matter how many workers you add, fix the pipeline before buying bandwidth: the bottleneck is not the link.

    Compliance

    YouTube's Terms of Service restrict automated access and downloading. Copyright and licensing apply to what you collect: many teams filter for Creative Commons-licensed videos (see finding and filtering Creative Commons videos), and privacy law applies to faces and voices. Get legal review for your jurisdiction and use case; tuning throughput does not change any of that.

    If you would rather not tune it yourself

    Most of the causes above come down to two things: per-connection limits, and a proxy path that is not sized for video. Corpora provides dedicated 1–40 Gbps unmetered proxy capacity (HTTP(S) and SOCKS5, yt-dlp-compatible, rotating pools, multi-region endpoints), or will collect and deliver deduplicated video, audio, subtitle and metadata datasets straight to your S3, GCS or Azure bucket, with a free sample batch. See pricing or talk to sales.

    FAQ

    Why is yt-dlp downloading at only a few MB/s when I have a gigabit link?

    Because one download is one connection, and a single YouTube stream typically settles well below line rate. Use -N 8 and run several videos in parallel. Reaching high aggregate throughput is a parallelism exercise, not a per-file one.

    Does a proxy make downloads slower?

    It adds a hop, so a single stream is often slower. What matters is aggregate throughput with many concurrent streams. If aggregate is capped, the proxy path is limiting you; if only the single-stream figure drops, you are seeing latency rather than lost capacity.

    Why does throughput drop the longer my job runs?

    Usually rate limiting or IP reputation. Check verbose output for repeated 403s on fragments and for the bot check. Retries restart work and inflate the cost, so fix the cause rather than only raising the retry count.

    Is it faster to download video and audio separately?

    Usually yes, because high resolutions are only offered as separate streams, and you avoid muxed formats that are served at low resolutions. Download bv*+ba, but skip transcoding at download time and convert on a separate stage.