Video Data Loader: Fast Video Decoding for Model Training

Written by

in

Your GPUs sit at 40% utilization, and the profiler blames the video data loader rather than the model. Video training is usually limited by decoding: the work grows with every frame the codec has to reconstruct, not just the frames you keep. This guide covers why that happens, how keyframes and GOP length set the cost of a seek, what PyAV, decord, TorchCodec and NVIDIA DALI offer as of November 2026, and how to estimate the CPU or GPU decode capacity you need before you buy it.

TL;DR

  • Cost is decoded frames, not sampled frames. A seek lands on the previous keyframe, so with long GOPs you pay for many frames you throw away.
  • Re-encode training clips with a short, fixed GOP (-g, -keyint_min, -sc_threshold 0) at the resolution and frame rate you train on. It is usually the cheapest speedup available.
  • TorchCodec is the PyTorch default now. torchvision removed its video decoding API (read_video, VideoReader) in 0.26. decord's last PyPI release was 0.6.0 in June 2021.
  • GPU decoding (NVDEC via TorchCodec or DALI) helps when CPUs run out, mainly at high resolution. Check codec support and CPU fallback.
  • Keep encoded bytes in WebDataset shards, shuffle before decoding, and decode inside the workers.

Why decoding is the bottleneck

Codecs like H.264, HEVC, VP9 and AV1 store a complete picture only at keyframes. Every other frame is predicted from its neighbours. To get frame k, the decoder starts at the previous keyframe and reconstructs every frame up to k. The PyAV container docs say it plainly: seeking goes to the nearest keyframe, "because to properly decode video frames you need to start from the previous keyframe".

Three costs add up per training sample:

  • Entropy decoding and reconstruction of every frame between the keyframe and the target. This scales with resolution and bitrate.
  • Color conversion and resize from YUV 4:2:0 to RGB at the source resolution, before you downscale to 224 or 256 px.
  • Memory traffic. A raw 1080p RGB frame is 1920 × 1080 × 3 = 6,220,800 bytes. A 224 × 224 one is 150,528 bytes.

Images are different: one JPEG decode yields one sample. A 16-frame video clip can need hundreds of frame decodes if sampling ignores where the keyframes are.

Seeking cost: keyframes and GOP length

Let G be the GOP length in frames (keyframe interval), N the frames you sample, and L the clip length in frames. If a target frame lands uniformly within a GOP:

decoded per random seek   ~ (G + 1) / 2
seek per sampled frame    D_seek ~ N x (G + 1) / 2     (targets in different GOPs)
decode straight through   D_seq  ~ index of last sampled frame  (<= L)
cost per sample           D = min(D_seek, D_seq)

example A: G = 250, N = 8, L = 300 -> D_seek ~ 1,004, D_seq ~ 300 -> decode through
example B: G = 16,  N = 8, L = 300 -> D_seek ~ 68                  -> seek

The example values are illustrative. Measure your own files: ffprobe -v error -select_streams v:0 -show_entries packet=pts_time,flags -of csv=p=0 clip.mp4 lists packets, and keyframes carry a K flag. Web video is typically encoded for streaming, so keyframes are often seconds apart. Our guide to splitting videos into clips with PySceneDetect covers the cutting side of the same problem.

Two more effects to know about:

  • Exact vs approximate indexing. TorchCodec's default seek_mode="exact" scans the file when the decoder is created, to build a frame index. "approximate" skips the scan and uses header metadata. Its docs say approximate is faster for long videos when you decode only a few frames. DALI's fn.decoders.video has the same trade-off via build_index.
  • Variable frame rate. DALI's fn.readers.video supports only constant frame rate input. Normalizing frame rate during re-encoding avoids index-vs-timestamp surprises in every library.

The decoder options in late 2026

Library Status (Nov 2026) Device Notes
TorchCodec Active; 0.17.0 on PyPI (Sept 2026) CPU, CUDA (NVDEC) Returns PyTorch tensors; accepts paths, bytes, tensors, file-like objects; built-in clip samplers
PyAV Active; 19.0.2 (Oct 2026) CPU Thin FFmpeg bindings; full control, more code
NVIDIA DALI Active; 2.4.0 (Oct 2026) GPU reader; CPU or mixed decoder Whole pipeline on GPU; WebDataset reader built in
decord Last PyPI release 0.6.0 (June 2021); last push July 2024; not archived CPU wheels; GPU needs a source build Convenient get_batch API; pin carefully or migrate
torchvision.io video Removed in torchvision 0.26 n/a Release notes point users to TorchCodec

On torchvision: the 0.26 release notes say the long-deprecated video decoding and encoding utilities are removed. That covers torchvision.io.video.*, read_video, write_video and the VideoReader class. torchvision 0.29 went further and deprecated its image decoders in favor of TorchCodec. If old training code imports torchvision.io.read_video, it fails on current releases, and TorchCodec has a migration guide.

On decord: its README says only CPU builds ship on PyPI and that GPU decoding needs a source build with -DUSE_CUDA=ON. The 0.6.0 wheels are a manylinux2010 Linux build, a Windows build, and macOS builds for Python 3.6–3.8 only. It still works in many existing codebases, but treat it as unmaintained for new pipelines.

TorchCodec

VideoDecoder takes device, seek_mode, num_ffmpeg_threads (default 1, recommended when many decoders run in parallel), dimension_order and decoder-side transforms. The batch methods (get_frames_at, get_frames_played_at, get_frames_in_range) are faster than repeated single-frame calls, according to its performance tips.

import torch
from torchcodec.decoders import VideoDecoder
from torchcodec.transforms import Resize

def uniform_n_frames(src, n=16):
    """src: path, bytes or tensor. Returns uint8 [n, C, 224, 224]."""
    dec = VideoDecoder(src, seek_mode="approximate",
                       transforms=[Resize(size=(224, 224))])
    total = dec.metadata.num_frames            # from headers in approximate mode
    idx = torch.linspace(0, total - 1, n).round().long().tolist()
    return dec.get_frames_at(indices=idx).data

def fixed_fps(src, fps=4.0, max_frames=32):
    dec = VideoDecoder(src, seek_mode="approximate")
    dur = dec.metadata.duration_seconds
    ts = [i / fps for i in range(min(max_frames, int(dur * fps)))]
    return dec.get_frames_played_at(seconds=ts).data

For GPU decoding, pass device="cuda". Frames are then decoded by NVDEC and stay in GPU memory. Per TorchCodec's docs, decoding can silently fall back to CPU when NVDEC doesn't support the codec or format, so log dec.cpu_fallback once a frame has been decoded. The same page recommends CUDA decoding for high-resolution video or when the CPU is saturated. It advises against it for small-resolution video, for an already busy GPU, or when you need bit-exact parity with CPU decoding.

PyAV

PyAV gives you packets and frames and nothing else, which suits custom sampling logic. Enable frame threading with stream.thread_type = "AUTO". The PyAV cookbook notes the default is slice threading, and frame threading adds decode latency.

import av

def frames_at_times(path, times_s):
    out = []
    with av.open(path) as c:
        s = c.streams.video[0]
        s.thread_type = "AUTO"
        for t in sorted(times_s):
            c.seek(int(t / s.time_base), stream=s)    # lands on keyframe <= t
            for f in c.decode(s):                     # decode forward to t
                if f.time is not None and f.time >= t:
                    out.append(f.to_ndarray(format="rgb24"))
                    break
    return out

Each loop iteration pays the (G + 1) / 2 cost from above. When the targets are dense, decode straight through and keep every k-th frame.

NVIDIA DALI

fn.readers.video reads files and decodes on the GPU, returning batches shaped (N, F, H, W, C). sequence_length sets frames per sequence, stride the distance between frames, and step the interval between sequence starts (defaults to sequence_length).

from nvidia.dali import pipeline_def, fn
from nvidia.dali.plugin.pytorch import DALIGenericIterator

@pipeline_def(batch_size=8, num_threads=4, device_id=0)
def video_pipe(files):
    return fn.readers.video(device="gpu", filenames=files, sequence_length=16,
                            stride=2, step=32, random_shuffle=True,
                            initial_fill=256, name="reader")

pipe = video_pipe(files=["/data/clips/a.mp4", "/data/clips/b.mp4"])
pipe.build()
loader = DALIGenericIterator(pipe, ["frames"], reader_name="reader")

To decode from in-memory bytes (for example, from shards), use fn.decoders.video. Older DALI releases exposed it as fn.experimental.decoders.video. That name no longer appears in the current operator reference, though some examples on the operator's own page still use it. The decoder accepts frames, or start_frame with sequence_length or end_frame and stride, plus a pad_mode. Its docs say the mixed backend handles H.264, HEVC, AV1 and MPEG-4, and the CPU backend handles VP8, VP9 and MJPEG.

Pre-extract, decode on the fly, or re-encode?

Strategy Decode cost at train time Storage Flexibility
Pre-extracted frames (JPEG/PNG or raw arrays) Image decode only Highest; raw 16 × 224 × 224 × 3 = 2,408,448 B per clip Sampling and resolution frozen at extraction
On-the-fly from source files Highest; long GOPs inflate it Lowest Any sampling, any resolution
Re-encoded clips, short GOP, training resolution Low; seeks are cheap Low (still compressed video) Resample per epoch; re-encode to change resolution

Raw-frame storage scales linearly: 1,000,000 clips × 2,408,448 B is about 2.4 TB before compression. Pre-extraction makes sense for small, fixed-sampling datasets or evaluation sets. For large training sets, re-encoding is usually the middle ground:

ffmpeg -i in.mp4 -an -vf "fps=8,scale=-2:256" \
  -c:v libx264 -preset medium -crf 20 -pix_fmt yuv420p \
  -g 16 -keyint_min 16 -sc_threshold 0 \
  -movflags +faststart out.mp4

That produces a constant 8 fps, 256 px tall (width rounded to an even number), with a keyframe every 16 frames (2 s). The FFmpeg codec docs define -g as the GOP size and keyint_min as the minimum IDR interval. sc_threshold maps to x264's scene-cut detection, and setting it to 0 stops extra keyframes at scene cuts. -force_key_frames expr:gte(t,n_forced*2) is a time-based alternative. Drop -an if you train on audio too. Smaller G makes seeks cheaper and files bigger. Pick it so that (G + 1) / 2 is small next to your frame stride.

Sampling strategies

  • Uniform N frames over the clip. Cost per sample is fixed and coverage is good for classification and captioning. With short GOPs, use per-frame seeks. With long GOPs, decode straight through.
  • Fixed fps (e.g. 2–8 fps). The temporal scale stays the same across clips, which matters for motion and for generative models. Normalize fps at re-encode time and the index math stays simple.
  • Random contiguous clip with dilation: TorchCodec's clips_at_random_indices(decoder, num_clips=..., num_frames_per_clip=..., num_indices_between_frames=...), or DALI's stride/step. A contiguous window costs one seek plus span frames.

Whichever you choose, record the sampling parameters in your run config. They change what the model sees.

CPU and GPU throughput math

Size decode capacity with formulas and your own measured rates. Don't use someone else's benchmark.

demand (decoded frames/s)
  F = GPUs x samples_per_s_per_GPU x D          # D from the seek formula

CPU decode
  r_core = decoded frames/s per core at your resolution and codec (measure)
  cores  = F / r_core x 1.3                     # headroom for resize, collate, I/O

GPU decode (NVDEC)
  r_gpu  = decoded frames/s per GPU (measure; varies by GPU, codec, resolution)
  GPU decode is enough if  F_per_GPU <= r_gpu
  PCIe saved per frame     = H x W x 3 bytes    # 6,220,800 B at 1080p RGB

example (assumed): 8 GPUs x 20 samples/s x D = 68  -> F = 10,880 frames/s
                   same with D = 300               -> F = 48,000 frames/s

Measure r_core on one of your clips with ffmpeg -benchmark -threads 1 -i clip.mp4 -f null - and divide frames by the reported time. The example shows why GOP length matters: the same training job needs about 4.4× the decode capacity (300/68) with long GOPs. Count DataLoader workers as well. Each worker that decodes on CPU needs a core, and num_ffmpeg_threads=1 keeps parallel decoders from oversubscribing the cores.

Integrating with WebDataset shards

Store encoded clips as .mp4 members in tar shards with a .json sidecar. Our WebDataset and Parquet storage guide covers shard sizing and manifests. Shuffle the encoded bytes, then decode in the worker. TorchCodec accepts bytes directly, so there is no temp file:

import json
import webdataset as wds
from torchcodec.decoders import VideoDecoder
from torchcodec.samplers import clips_at_random_indices
from torchcodec.transforms import Resize

SHARDS = "pipe:aws s3 cp --quiet s3://BUCKET/clips/clips-{000000..004095}.tar -"

def decode(sample):
    dec = VideoDecoder(sample["mp4"], seek_mode="approximate",
                       transforms=[Resize(size=(224, 224))])
    clip = clips_at_random_indices(dec, num_clips=1, num_frames_per_clip=16,
                                   num_indices_between_frames=2)
    return clip.data[0], json.loads(sample["json"])["caption"]

dataset = (wds.WebDataset(SHARDS, resampled=True, nodesplitter=wds.split_by_node)
           .shuffle(64)            # buffer holds encoded clips, not frames
           .map(decode)
           .with_epoch(10_000))
loader = wds.WebLoader(dataset, batch_size=8, num_workers=8)

Filter out clips shorter than the sampled span (here 31 frames) when you build the shards, so the sampler never pads. For NVDEC decoding, keep DataLoader workers on CPU work: have them yield bytes and decode with device="cuda" in the main process. Alternatively, use DALI's fn.readers.webdataset(paths=..., ext=["mp4"]) feeding fn.decoders.video(..., device="mixed"). That reader takes local tar paths and optional index files from DALI's wds2idx.py. Without the index files it infers the index, which its docs warn "will add considerable startup time for big datasets".

Compliance

If your clips come from YouTube, note that YouTube's Terms of Service restrict automated access and downloading. Copyright and licensing still apply to re-encoded or frame-extracted derivatives, and many teams filter for Creative Commons content (see our Creative Commons guide). Faces and voices are personal data under regimes like GDPR. Keep clip-level IDs through re-encoding so you can honor takedown and opt-out requests, and get legal review for your jurisdiction and use case.

If you'd rather not run the collection side

Decoding is your training stack's job. Getting the source video is a separate one. Corpora's managed option collects, deduplicates and delivers YouTube video, audio, subtitles and metadata (JSONL/Parquet, sha256 checksums, manifests) to S3, GCS, Azure Blob or private object storage, with a free sample batch. Get in touch if that helps.

FAQ

What is the fastest video data loader for PyTorch?

There's no universal answer. It depends on codec, resolution, GOP length and how many CPU cores you have per GPU. TorchCodec (CPU or NVDEC) and DALI are the maintained options. Re-encoding to a short GOP at training resolution usually matters more than which library you pick.

Is torchvision.io.read_video deprecated?

It's gone. torchvision 0.26 removed read_video, write_video, VideoReader and torchvision.io.video.*, and points users to TorchCodec.

Is decord still maintained?

Its last PyPI release is 0.6.0 from June 2021, and the repository's last push was in July 2024. It isn't archived, but for new pipelines TorchCodec or PyAV is the safer choice.

Why is random frame access so slow in video?

Each seek restarts decoding at the previous keyframe, so you decode about (G + 1) / 2 frames per random target. Shorten the GOP with -g, -keyint_min and -sc_threshold 0, or sample contiguous windows.

Should I decode video on the GPU?

Decode on the GPU when CPU decode is the measured bottleneck, especially at high resolution, and when NVDEC supports your codec. Check for silent CPU fallback, and keep GPU decoding out of forked DataLoader workers.