If you need to collect 100 TB, 1 PB or 5 PB of video or audio per month, the first question is not which tool to use. It is how many gigabits per second you need to keep running all month. This guide gives you the exact formula, sizing tables for 10 TB to 5 PB per month, the reverse table (what 1–40 Gbps can actually move), and the practical limits that decide whether you hit those numbers.
TL;DR
- Formula: Gbps = TB/month × 8,000 ÷ seconds per month. With a 30-day month (2,592,000 s), 1 Gbps at 100% utilization moves 324 TB/month, or 10.8 TB/day.
- 100 TB/month ≈ 0.31 Gbps, 1 PB/month ≈ 3.09 Gbps, 5 PB/month ≈ 15.4 Gbps. That assumes the link is saturated 24/7, and it won't be.
- Plan for 60–70% sustained utilization. That means provisioning about 1.4–1.7× the theoretical number.
- 800 TB/month ≈ 2.47 Gbps at 100%, so provision about 4–5 Gbps.
- Writing data into S3 or GCS from the internet is free. Reading it back out to the internet is not.
- Per-GB pricing only makes sense at low volume. At $2,000 per Gbps per month, the break-even is $2,000 ÷ (price per GB) GB per month.
The formula
Bandwidth is measured in bits per second, but storage is measured in bytes. You convert between them, then divide by the number of seconds in the period:
Gbps = TB_per_month × 8,000 / seconds_per_month
# 1 TB (decimal) = 10^12 bytes = 8 × 10^12 bits = 8,000 Gb
# 30-day month = 30 × 86,400 = 2,592,000 s
# average month = 30.44 × 86,400 ≈ 2,630,016 s (365.25 / 12 days)This article uses the 30-day month. It is the more conservative choice, because fewer seconds means a slightly higher Gbps figure. The 30.44-day average lowers every result by about 1.5% (100 TB/month comes out at 0.304 Gbps instead of 0.309). For capacity planning that difference is smaller than your error bars.
TB vs TiB
Network speeds are always decimal: 1 Gbps = 10^9 bits per second. Storage tools often report binary units even when they label them "TB". Check whether df -h, your object-store console or your billing export means TB or TiB:
- 1 TB = 10^12 bytes; 1 TiB = 2^40 bytes ≈ 1.0995 TB (about 10% larger).
- 1 PB = 10^15 bytes; 1 PiB = 2^50 bytes ≈ 1.126 PB (about 12.6% larger).
- 100 TiB/month needs 0.339 Gbps, compared with 0.309 Gbps for 100 TB. 1 PiB/month needs 3.47 Gbps, compared with 3.09 Gbps for 1 PB.
If your target is set in TiB or PiB, convert it to decimal before you use the tables below.
Monthly volume → required bandwidth
The "100%" column is pure arithmetic. The provisioned columns divide it by 0.70 and 0.60, which are the sustained utilization levels you can realistically plan around (the next section explains why). All values use a 30-day month and decimal units.
| Volume / month | Per day | Gbps at 100% | Provision at 70% util. | Provision at 60% util. |
|---|---|---|---|---|
| 10 TB | 0.33 TB | 0.031 | 0.04 Gbps | 0.05 Gbps |
| 50 TB | 1.67 TB | 0.154 | 0.22 Gbps | 0.26 Gbps |
| 100 TB | 3.33 TB | 0.309 | 0.44 Gbps | 0.51 Gbps |
| 250 TB | 8.33 TB | 0.772 | 1.10 Gbps | 1.29 Gbps |
| 500 TB | 16.7 TB | 1.543 | 2.20 Gbps | 2.57 Gbps |
| 800 TB | 26.7 TB | 2.469 | 3.53 Gbps | 4.12 Gbps |
| 1 PB | 33.3 TB | 3.086 | 4.41 Gbps | 5.14 Gbps |
| 2 PB | 66.7 TB | 6.173 | 8.82 Gbps | 10.29 Gbps |
| 5 PB | 166.7 TB | 15.432 | 22.05 Gbps | 25.72 Gbps |
Round up to whatever increments your provider sells. For example, 1 PB/month lands at 5 Gbps provisioned, 2 PB/month at 10 Gbps, and 5 PB/month at 25 Gbps.
Why plan for 60–70% utilization, not 100%
The 100% number assumes every second of the month moves useful bytes at line rate. In practice, several things get in the way:
- Protocol overhead. Ethernet, IP and TCP headers, TLS records and HTTP framing all use bandwidth on the wire but don't count toward the bytes stored on disk. Proxy hops add their own overhead on top.
- Retries and failed or partial downloads. A download that dies at 80% and restarts from zero (or re-fetches fragments) still used bandwidth. Expired stream URLs, bot checks (see fixing "Sign in to confirm you're not a bot") and connection resets all show up as wasted bytes.
- Non-media traffic. Metadata extraction, manifest and player requests, subtitle fetches and format probing all use requests and time without adding much volume.
- Per-connection throughput limits. A single stream rarely fills a fast link. Unless you keep enough downloads in flight at all times, the pipe sits partly idle.
- Diurnal and operational gaps. Queue refills, deploys, worker restarts, rate-limit backoff, proxy rotation and the daily traffic patterns of shared upstreams all create troughs. A 24-hour average is always below the peak.
Sustained 60–70% utilization is a reasonable target for a well-run pipeline. It is a planning assumption, not a measured guarantee. If your own monitoring shows 80% sustained, you can provision closer to the 100% column. If you are running at 40%, fix the pipeline before you buy more bandwidth.
Worked example: 800 TB per month
800 × 8,000 / 2,592,000 = 2.469 Gbps (at 100% utilization, 30-day month)
800 × 8,000 / 2,630,016 = 2.433 Gbps (30.44-day average month)
2.469 / 0.70 = 3.53 Gbps
2.469 / 0.60 = 4.12 Gbps
→ provision ~4–5 Gbps
At 4 Gbps provisioned, 800 TB needs 2.469 / 4 = 62% sustained utilization.
At 5 Gbps provisioned, it needs 2.469 / 5 = 49%, which leaves room for re-downloads and growth.Bandwidth → monthly volume (reverse table)
This table answers the opposite question: you already have a link, so how much can it move? The formula is TB = Gbps × seconds / 8,000.
| Link | Per day at 100% | Per 30-day month at 100% | Per 30.44-day month at 100% | Per 30-day month at 65% |
|---|---|---|---|---|
| 1 Gbps | 10.8 TB | 324 TB | 329 TB | 211 TB |
| 2 Gbps | 21.6 TB | 648 TB | 658 TB | 421 TB |
| 5 Gbps | 54 TB | 1.62 PB | 1.64 PB | 1.05 PB |
| 10 Gbps | 108 TB | 3.24 PB | 3.29 PB | 2.11 PB |
| 20 Gbps | 216 TB | 6.48 PB | 6.58 PB | 4.21 PB |
| 40 Gbps | 432 TB | 12.96 PB | 13.15 PB | 8.42 PB |
A useful rule of thumb: 1 Gbps sustained ≈ 10.8 TB/day ≈ 324 TB per 30 days. Everything else scales linearly from there.
What that volume means in hours of media
Training-data budgets are usually set in hours, not terabytes. To convert between them you need an average bitrate:
GB per hour = Mbps × 3,600 / 8 / 1,000 = Mbps × 0.45
hours per PB = 1,000,000 / (GB per hour)Delivered bitrates on YouTube vary a lot with codec (H.264/AVC, VP9, AV1), frame rate and content. Static talking heads compress far better than sports. The figures below are approximate planning assumptions, not published specifications. For reference, YouTube's recommended upload encoding settings list 8 Mbps for 1080p SDR at 24–30 fps and 12 Mbps at 48–60 fps. Those are upload targets; the re-encoded streams YouTube delivers are usually well below them.
| Format (approximate) | Assumed bitrate | GB per hour | Hours per 100 TB | Hours per 1 PB |
|---|---|---|---|---|
| Audio only, AAC (m4a) ~128 kbps | 0.128 Mbps | 0.058 | ~1.74 M | ~17.4 M |
| Audio only, Opus (webm) ~160 kbps | 0.16 Mbps | 0.072 | ~1.39 M | ~13.9 M |
| 720p video + audio | 1.5 Mbps | 0.675 | ~148 k | ~1.48 M |
| 1080p, efficient codec (VP9/AV1) | 2 Mbps | 0.9 | ~111 k | ~1.11 M |
| 1080p, H.264 or high-motion | 4 Mbps | 1.8 | ~56 k | ~556 k |
| 2160p (4K) | 15 Mbps | 6.75 | ~15 k | ~148 k |
The format choice can change your bandwidth bill by an order of magnitude. Audio-only collection for speech models needs roughly 1/12th to 1/30th of the bandwidth of 1080p video for the same number of hours. Before you size anything, measure your actual mix on a sample with yt-dlp. -O/--print implies simulate mode, so nothing is downloaded:
yt-dlp -f "bv*[height<=1080]+ba/b[height<=1080]" \
-O "%(id)s %(duration)s %(tbr)s %(filesize,filesize_approx)s" \
--batch-file sample_urls.txt > sample_sizes.txttbr is the total average bitrate in kbps, and filesize or filesize_approx is in bytes (it may print NA when unknown). Use -F to see every available format for a single video. Multiply the sample's mean bytes per second of duration by your target hours to get TB, then use the first table. For format selection and proxy flags, see the yt-dlp proxy guide.
Storage, ingress and egress costs
- Ingress is free on the big clouds. The Amazon S3 pricing page lists "data transferred in from the internet" among the transfers you don't pay for. Google's Cloud Storage pricing lists inbound data transfer as free. You still pay per-request charges (PUT/POST, multipart parts), so avoid uploading millions of tiny objects.
- Egress is charged. S3 includes 100 GB/month of internet egress free, aggregated across AWS services, and charges per GB beyond that. For GCS general network usage, internet egress starts at $0.12/GiB for the first 10 TiB to worldwide destinations (excluding Asia and Australia) and steps down at volume. Moving a petabyte out of a cloud to train elsewhere costs a lot, so decide early where training will run.
- Storage grows linearly. At 1 PB/month you add 12 PB a year before replicas. Multiply by your per-GB-month rate and storage class, and budget for lifecycle rules (cold tiers for raw downloads, hot tiers for curated shards).
- Downloading on a cloud VM means your proxy or provider traffic arrives as VM ingress, which is free on AWS and GCP. But the traffic between your VM and an external proxy is outbound requests plus inbound responses, so check your provider's NAT gateway and inter-zone charges too.
Per-GB vs per-Gbps pricing: break-even
Proxy and transit capacity is sold in one of two ways: metered (per GB transferred) or flat (per Gbps of committed capacity, unmetered). Using $2,000 per Gbps per month as the flat price, the break-even is:
break_even_GB_per_month = flat_price_per_Gbps / price_per_GB
= 2,000 / price_per_GB
1 Gbps can move at most 324,000 GB per 30 days (~210,000 GB at 65% utilization).The per-GB prices below are illustrative examples, not quotes from any provider:
| Example per-GB price | Break-even volume per Gbps | Cost of 100 TB metered | Cost of 1 PB metered |
|---|---|---|---|
| $0.10 / GB | 20 TB/month | $10,000 | $100,000 |
| $0.50 / GB | 4 TB/month | $50,000 | $500,000 |
| $1.00 / GB | 2 TB/month | $100,000 | $1,000,000 |
| $3.00 / GB | 0.67 TB/month | $300,000 | $3,000,000 |
Even at $0.10/GB, metered pricing costs more than a flat 1 Gbps line once you move more than 20 TB/month, which is about 6% of that line's capacity. At the 100 TB to 5 PB volumes in this article, per-GB pricing is only reasonable for small, hard-to-reach slices of traffic. It doesn't work for the bulk of the download volume.
Practical limits that decide whether you hit the number
Parallelism
Required concurrency follows directly from per-connection throughput:
concurrent_downloads = target_Gbps × 1,000 / per_connection_Mbps
e.g. 2.5 Gbps at 20 Mbps per download → 125 downloads in flight
15 Gbps at 20 Mbps per download → 750 downloads in flightThe 20 Mbps figure is only an example. Measure your own per-download throughput through your proxies, because server-side pacing, proxy hops and round-trip time all cap a single stream. TCP itself sets a ceiling of window ÷ RTT: a 64 KiB window at 100 ms RTT tops out around 5.2 Mbps. yt-dlp's -N/--concurrent-fragments parallelizes fragments within one DASH/HLS download, but at petabyte scale most of the parallelism comes from running many independent downloads across many workers and IPs.
Disk write throughput
Divide Gbps by 8 to get the sustained write rate: 2.5 Gbps ≈ 312 MB/s, 15.4 Gbps ≈ 1.93 GB/s, and that is around the clock. If you download separate video and audio streams and merge them with ffmpeg locally, every byte is written at least twice and read once. Plan for roughly 2–3× the network rate in disk I/O, or stream finished files directly to object storage and keep only scratch space local.
NIC and host limits
A 10 GbE NIC can't carry 10 Gbps of useful payload once you subtract overhead. TLS termination, Python processes per download, ffmpeg remuxing and checksum computation all use CPU. File descriptors, ephemeral ports and conntrack tables run out at thousands of concurrent connections, so raise ulimit -n and check kernel limits before blaming the network.
Why one fast server is not enough
A single 25 or 40 Gbps box is a single point of failure, a single egress IP set (which gets rate-limited or challenged as one unit), and a single disk and CPU bottleneck. Horizontal scaling works better: many workers of a few Gbps each, pulling from a shared queue, uploading to object storage, and spread across regions and IP pools. If one node dies, you lose a fraction of throughput instead of the whole month's schedule.
Monitor utilization, not just "is it running"
- Track bytes stored per hour against the target rate (for 1 PB/month that is about 1.39 TB/hour), not just link utilization. The gap between the two is your waste.
- Graph link utilization per worker and in aggregate, and alert on sustained drops below your planning assumption.
- Track the retry rate, failure reasons (HTTP 403/429, bot checks, timeouts) and the bytes discarded from partial downloads.
- Watch the queue depth so workers never idle waiting for URLs.
Compliance
YouTube's Terms of Service restrict automated access and downloading. Copyright and licensing matter: many teams filter for Creative Commons-licensed videos. Privacy also matters, since faces and voices can be personal data under GDPR, and so does honoring takedown and opt-out requests. Get legal review for your jurisdiction and use case before you collect at scale. For dataset design, see building a YouTube video dataset for multimodal training and downloading Panda-70M, HD-VILA and InternVid.
If you'd rather not run this yourself
Corpora sells dedicated 1–40 Gbps unmetered proxy capacity (HTTP(S) and SOCKS5, compatible with yt-dlp and custom crawlers) at $2,000 per Gbps per month, with volume discounts from 5 Gbps. See pricing. If you only want the data, the fully managed option delivers deduplicated video, audio, subtitles and metadata straight to your S3, GCS or Azure bucket, priced per delivered PB. Get in touch for a free sample batch.
FAQ
How many Gbps do I need for 1 PB per month?
1 PB × 8,000 ÷ 2,592,000 s ≈ 3.09 Gbps at 100% utilization over a 30-day month. At a realistic 60–70% sustained utilization, provision about 4.4–5.1 Gbps. In practice that means 5 Gbps.
How much data can a 1 Gbps connection download in a month?
About 324 TB in 30 days at 100% utilization (10.8 TB/day), or about 329 TB over an average 30.44-day month. At 65% sustained utilization, expect about 210 TB.
How much bandwidth do I need for 100 TB per month?
About 0.31 Gbps sustained. At 60–70% utilization, provision 0.44–0.51 Gbps, so a 1 Gbps line has plenty of headroom.
Does uploading to S3 or Google Cloud Storage cost bandwidth fees?
No. Data transfer in from the internet is free on both S3 and GCS according to their pricing pages. Requests and storage are billed, and so is data transfer out to the internet.
Should I size in TB or TiB?
Use decimal TB to match network units (Gbps is decimal). If your storage reports TiB, multiply by 1.0995 before you use the formula. A 100 TiB target needs about 10% more bandwidth than 100 TB.