Millions of loose MP4 files in a bucket work fine until a training job tries to read them. Then per-object request overhead, listing, and shuffling decide your GPU utilization. This guide compares the main video dataset storage formats (loose files, WebDataset tar shards, Parquet, Hugging Face streaming, and MosaicML MDS), shows how to size shards and shuffle buffers, and gives you a manifest layout with sha256 checksums that you can verify and audit.
Key takeaways
- Don't train from one object per clip. Pack clips into shards of hundreds of MB to a few GB, and read them sequentially.
- Put the bytes in WebDataset tar shards or MDS. Put the metadata (captions, scores, lineage, shard and offset) in Parquet. Bytes belong in Parquet only for small payloads such as short audio clips or thumbnails.
- Shuffle at write time, and again at read time. Streaming shuffle buffers hold encoded video, so size them in bytes, not samples.
- Keep your own sha256 per shard in a manifest. For multipart uploads, S3's stored SHA-256 is a composite value and won't match
sha256sum. - Random access is a separate path. Use a Parquet index with byte offsets, WIDS, or MDS when you need a specific clip. Use sequential streaming for training.
Why many small files fail at training time
Object stores bill and throttle per request, and every request pays a latency cost. AWS's S3 performance guide says you can achieve at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix. It also quotes first-byte latencies of roughly 100–200 ms. Scaling to a new request rate happens gradually, with 503 Slow Down errors while it does. Google publishes equivalent request-rate and ramp-up guidelines for Cloud Storage.
Run the numbers for a job that consumes 2,000 clips per second:
one object per clip:
GET rate = 2,000 req/s (more with .json/.txt sidecars)
in-flight needed = rate x latency = 2,000 x 0.1..0.2 s = 200..400 concurrent GETs
1 GB shards, ~2.5 MB per clip (assumed: 10 s at ~2 Mbps):
clips per shard = 1e9 / 2.5e6 = 400
GET rate = 2,000 / 400 = 5 req/s
Loose files also slow down listing, copying, takedown deletes and checksumming. The Hugging Face Hub storage limits recommend fewer than 100k files per repo and fewer than 10k per folder, and suggest merging data into Parquet or WebDataset. The WebDataset README claims 3x–10x higher I/O rates for sequential versus random reads on local drives.
| Layout | Read pattern | Random access | Best for |
|---|---|---|---|
| Loose files + metadata table | One GET per file | Trivial | Raw archive, inspection, re-processing |
| WebDataset (tar shards) | Sequential stream | Via WIDS or your own offset index | Training on video/audio at any scale |
| Parquet | Column chunks per row group | Row group granularity | Metadata, small binary payloads |
| MDS (MosaicML Streaming) | Shard download + local cache | Yes, dataset[i] |
Deterministic multi-node training, resumption |
| Lance | Columnar + blob reads | Yes, by row | Metadata and blobs in one table |
Keep raw downloads as loose files in their own prefix as the source of truth for re-cutting (see our YouTube video dataset pipeline guide), and train from shards built per release.
WebDataset: tar shards for video and audio
The WebDataset format has two conventions. Files that make up one sample share a basename, and shards are numbered like clips-000000.tar, addressed with brace notation clips-{000000..004095}.tar. Clips stay in their native encoding (.mp4, .opus, .flac), with .json or .txt sidecars. Standard tar can read any shard. The current PyPI release is webdataset 1.0.2.
Writing shards
ShardWriter rolls over to a new shard when it hits maxcount samples or maxsize bytes (defaults: 100,000 and 3e9). It only writes to local paths, but its post callback runs on each finished shard. That's the right place to hash, upload and log it. The script below pre-shuffles the clip list, so clips from one source video don't end up together in a shard, and writes a manifest line per shard:
import hashlib, json, os, random, subprocess, tarfile
import webdataset as wds
DEST = "s3://BUCKET/ytds/v2026-11/shards"
os.makedirs("out", exist_ok=True)
def sha256_file(path, bufsize=8 << 20):
h = hashlib.sha256()
with open(path, "rb") as f:
while chunk := f.read(bufsize):
h.update(chunk)
return h.hexdigest()
def count_samples(path):
with tarfile.open(path) as t:
return len({m.name.split(".", 1)[0] for m in t.getmembers()})
def finish_shard(path):
name = os.path.basename(path)
rec = {"path": f"shards/{name}", "bytes": os.path.getsize(path),
"sha256": sha256_file(path), "nsamples": count_samples(path)}
subprocess.run(["aws", "s3", "cp", path, f"{DEST}/{name}"], check=True)
with open("manifest.jsonl", "a", encoding="utf-8") as m:
m.write(json.dumps(rec) + "\n")
os.remove(path)
with open("clips.jsonl", encoding="utf-8") as f:
rows = [json.loads(line) for line in f]
random.Random(2026).shuffle(rows) # write-time shuffle
with wds.ShardWriter("out/clips-%06d.tar", maxcount=100_000,
maxsize=1e9, post=finish_shard) as sink:
for r in rows: # r: {"clip_id", "path", "caption", ...}
with open(r["path"], "rb") as v:
sink.write({"__key__": r["clip_id"], "mp4": v.read(), "json": r})
Keys must not contain dots, because WebDataset splits the basename at the first dot to find the extension. YouTube IDs plus a clip index (abc123XYZ_-_000012) are safe. A shard can exceed maxsize by one sample. For datasets too large to shuffle in memory, assign a random sort key in DuckDB or Spark.
Reading shards
import json
import webdataset as wds
SHARDS = ("pipe:aws s3 cp --quiet "
"s3://BUCKET/ytds/v2026-11/shards/clips-{000000..004095}.tar -")
def to_pair(sample):
return sample["mp4"], json.loads(sample["json"]) # decode frames downstream
dataset = (
wds.WebDataset(SHARDS, resampled=True, nodesplitter=wds.split_by_node)
.shuffle(64) # buffer of *encoded* clips
.map(to_pair)
.with_epoch(10_000) # samples per DataLoader worker per epoch
)
loader = wds.WebLoader(dataset, batch_size=None, num_workers=8)
The details matter in multi-node runs. In the current source, the default nodesplitter raises an error when it detects more than one node, so pass one explicitly. If you don't use resampled=True, set shardshuffle to a number: the constructor warns when it's left at None. The README notes that exactly balanced epochs across many nodes are hard with an IterableDataset, which is why multi-node setups usually resample shards and define epochs with with_epoch.
For indexed access, the same repo ships WIDS. wids.ShardListDataset takes a JSON descriptor listing each shard's URL and nsamples, and supports dataset[i]. It downloads whole shards into a local cache, so random access works at shard granularity on the network.
Parquet: the metadata layer (and when bytes belong in it)
Parquet is where you filter, join and audit. Use one row per clip with clip ID, source video ID, timestamps, captions, quality scores, license, collection date, and the location of its bytes: shard name, plus byte offset and length if you want direct reads. Python's tarfile exposes each member's data offset as TarInfo.offset_data. Together with an HTTP Range request (which S3 supports), that lets you fetch one clip from the middle of a 1 GB shard:
import pyarrow.parquet as pq
idx = pq.read_table("clips.parquet",
columns=["clip_id", "shard", "offset", "size"],
filters=[("aesthetic", ">=", 5.0), ("license", "=", "cc-by")])
# fetch one clip without downloading the shard
# aws s3api get-object --bucket BUCKET --key ytds/v2026-11/shards/clips-000417.tar \
# --range bytes=OFFSET-$((OFFSET+SIZE-1)) clip.mp4
Row group size controls how much a reader touches. The Parquet docs recommend large row groups (512 MB–1 GB) for sequential analytic scans. Hugging Face datasets uses smaller defaults when it writes Parquet with push_to_hub: a 500 MB maximum shard size and a 100 MB maximum row group size, per its config.py. In pyarrow.parquet.write_table, row_group_size counts rows, not bytes.
When to embed bytes in Parquet: for short audio segments, thumbnails or a few keyframes per clip, a binary column is convenient and the Hub's dataset viewer handles it. Full video clips are a poor fit: row groups fill with hundreds of MB of opaque bytes, and rewriting metadata means rewriting video. Keep video in tar or MDS shards and join on clip ID. If you want metadata and blobs in a single table, Hugging Face's video docs now cover Lance, which stores blobs as bytes with offsets so metadata scans don't load the videos.
For speech corpora built from short segments, Parquet with embedded audio is common. Our speech training data guide covers the audio-side preparation.
Hugging Face datasets streaming
load_dataset(..., streaming=True) returns an IterableDataset that reads as you iterate. For video, the docs recommend WebDataset (they give "1GB of videos per TAR archive" as an example) and load it with the built-in builder:
from datasets import load_dataset
ds = load_dataset("webdataset", data_dir="/path/to/shards", split="train", streaming=True)
ds = ds.shuffle(seed=42, buffer_size=256) # default buffer is 1,000 examples
ds.set_epoch(1) # call each epoch: seed + epoch
According to the streaming docs, IterableDataset.shuffle fills its buffer from up to max_buffer_input_shards shards at a time (default 10) and also shuffles shard order. state_dict() and load_state_dict() resume from the current shard. Examples sitting in the shuffle buffer are lost on resume. Video support in datasets is marked experimental and decodes with torchcodec.
MosaicML Streaming (MDS)
Status as of November 2026: mosaicml/streaming is not archived and still receives commits. The latest tagged release is v0.13.0 (July 2025), the project sits under Databricks, and it installs as pip install mosaicml-streaming. Weigh the slow release cadence for a long-lived pipeline.
from streaming import MDSWriter, StreamingDataset
columns = {"clip_id": "str", "mp4": "bytes", "meta": "json"}
with MDSWriter(out="s3://BUCKET/ytds-mds/v2026-11", columns=columns,
size_limit="128mb", hashes=["sha256"]) as out:
for r in rows:
with open(r["path"], "rb") as v:
out.write({"clip_id": r["clip_id"], "mp4": v.read(), "meta": r})
ds = StreamingDataset(remote="s3://BUCKET/ytds-mds/v2026-11",
local="/nvme/mds-cache", shuffle=True,
batch_size=8, cache_limit="2tb")
clip = ds[12345] # random access; downloads the shard if needed
Per the conversion docs, size_limit applies to uncompressed shards, defaults to 67 MB, and "50-100MB works well in practice". At ~2.5 MB per clip, that's only 20–40 clips per shard, so video teams often go larger. Compression is optional and helps text far more than already-compressed media. The default shuffle, py1e, uses shuffle_block_size (default max(4000000/num_canonical_nodes, 1<<18) samples), which the shuffling docs say should be at least 10x the samples per shard. The README highlights deterministic sample order across GPU counts, mid-epoch resumption and cache_limit eviction.
Shard sizing
| Source | Stated default or guidance |
|---|---|
| HF datasets video docs | Example: ~1 GB of video per TAR, thousands of TARs |
WebDataset ShardWriter |
Defaults maxcount=100000, maxsize=3e9 |
MDS MDSWriter |
Default 67 MB uncompressed; 50–100 MB "works well in practice" |
HF push_to_hub (Parquet) |
500 MB max shard, 100 MB max row group |
| HF Hub repo | <100k files per repo, <10k per folder, <200 GB per file |
For video, a 256 MB–2 GB tar shard is a reasonable starting range. Check it against three constraints:
- Enough shards for every reader:
readers = nodes x GPUs per node x workers per GPU. With 32 x 8 x 8 = 2,048 readers and a 25 TB dataset in 1 GB shards (25,000 shards), each reader sees about 12 shards per epoch. Fewer shards than readers leaves some workers idle or duplicating data. - Enough samples per shard that per-request latency is amortized: aim for hundreds, not tens.
- Small enough to retry cheaply. A failed 10 GB download hurts more than a failed 1 GB one. Bandwidth and cost planning for the dataset as a whole is covered in bandwidth planning for large-scale video downloads.
Shuffling: buffers are measured in bytes
Streaming loaders shuffle in three places: the order of shards written (write-time shuffle), the order of shards read (shard shuffle), and a per-worker sample buffer. With video, the third is the expensive one:
buffer RAM per node = GPUs x workers x buffer_samples x avg_encoded_clip
= 8 x 8 x 1,000 x 2.5 MB = 160 GB
That's why the read example uses a buffer of 64. Shuffle before decoding, so the buffer holds compressed bytes rather than frame tensors. Lean on write-time shuffling and shard shuffling for most of the randomness. Panda-70M's README warns that its shards are unshuffled, with all clips from one video in one shard. Our Panda-70M / video2dataset guide covers that case.
Manifests and sha256 checksums
A release should be verifiable without trusting the storage layer. Keep one line per object:
{"path": "shards/clips-000417.tar", "bytes": 1000213504, "sha256": "9f86d0...0a08", "nsamples": 401}
Verify a full release, or spot-check a shard straight from the bucket:
jq -r '"\(.sha256) \(.path)"' manifest.jsonl > SHA256SUMS
sha256sum -c SHA256SUMS # run from the release root
aws s3 cp s3://BUCKET/ytds/v2026-11/shards/clips-000417.tar - | sha256sum
Don't substitute S3's built-in checksums for this manifest. Per AWS's integrity docs, multipart uploads support full-object checksums only for CRC algorithms. SHA-256 on a multipart object is a composite "checksum of checksums", and a multipart ETag is not an MD5 of the file. Large shards usually go up as multipart uploads. Your own sha256 is what consumers can check with standard tools. Store the manifest, the Parquet index and a takedowns.jsonl next to the shards, and never mutate a published shard. Removals produce a new release. The video training data buying checklist lists the same fields from the buyer's side.
Random access vs sequential streaming
| Job | Access pattern | Use |
|---|---|---|
| Pretraining epochs | Sequential, shuffled | WebDataset, MDS, HF streaming |
| Exact mid-epoch resume | Deterministic order | MDS; HF state_dict (minus buffered samples) |
| Audits, eval sets, takedown checks | Point lookups by ID | Parquet index + byte ranges, WIDS, MDS dataset[i], Lance |
| Filtering for a new mix | Column scans | Parquet + DuckDB/pyarrow, then re-shard |
Sampling clips for human review is a Parquet query plus byte-range reads, not a shard scan (see dataset accuracy audits).
Compliance
If the clips come from YouTube, note that YouTube's Terms of Service restrict automated access and downloading. Copyright and licensing apply to the content: many teams filter for Creative Commons and record the license per clip. Faces and voices are personal data under regimes like GDPR. A storage layout that supports takedowns (a per-clip index, immutable releases, rebuildable shards) makes honoring removal and opt-out requests practical. Get legal review for your jurisdiction and use case.
If you'd rather not run this yourself
Corpora's fully managed option collects and deduplicates YouTube video, audio, subtitles and metadata. It delivers JSONL/Parquet metadata with sha256 checksums and manifests straight to S3, GCS, Azure Blob or private object storage, priced per delivered PB, with a free sample batch. Contact us to scope a delivery layout.
FAQ
What is the best storage format for a video training dataset?
For training, WebDataset tar shards (or MDS for deterministic resumption) plus a Parquet table of per-clip metadata. Keep raw downloads separately for reprocessing.
How big should WebDataset shards be?
For video, start somewhere between 256 MB and 2 GB, with hundreds of clips per shard. Then check that you have many more shards than total DataLoader workers. Hugging Face's docs use ~1 GB per TAR as an example.
Can I store video bytes in Parquet?
You can, but full clips make row groups huge. It works well for short audio segments, thumbnails and keyframes.
Is MosaicML Streaming still maintained?
As of November 2026, the GitHub repo is active and not archived, but the latest tagged release is v0.13.0 from July 2025. Pin a version and test upgrades.
How large should the shuffle buffer be for video?
Size it by memory: GPUs x workers x buffer x average encoded clip size. With multi-MB clips that usually means tens to low hundreds of samples, with write-time and shard-level shuffling doing the rest.