Tag: video datasets

Panda-70M, HD-VILA-100M, InternVid and most other large video-language datasets don't ship any video. You get YouTube IDs, clip timestamps and captions, and you have to collect the videos yourself. This guide covers what each dataset actually releases (checked against the paper and the official repo), which tooling works for it, and how to plan storage, failures and scaling before you start a multi-terabyte download.

TL;DR

• Panda-70M, HD-VILA-100M, InternVid, HowTo100M, AudioSet and VoxCeleb2 all release metadata only (IDs, timestamps, text or labels). Kinetics-700-2020 is the exception: CVDF hosts the video files.
• For Panda-70M, use the video2dataset fork inside the Panda-70M repo. The authors say upstream video2dataset can't read their CSV format.
• Many clips per source video: download the video once and cut it locally with ffmpeg. One short segment per video: use yt-dlp --download-sections.
• Expect missing videos. Kinetics-700-2020's own download log shows about 2% of videos already gone a few months after the 2020 release.
• Storage ≈ hours × 3600 × bitrate ÷ 8. Panda-70M's own figure (~36 TB for 167k hours at 360p) works out to roughly 0.48 Mbps.
• Annotation licenses don't give you any rights to the underlying videos.

Why these datasets ship as IDs and timestamps

The videos belong to their uploaders. Dataset authors can license their annotations (captions, labels, clip boundaries), but they generally can't redistribute the footage. What you download is a table of pointers, so every team that wants the pixels has to collect them again. This has three practical effects: your copy of the dataset is smaller than the paper's, it shrinks over time, and it depends on how well your downloader keeps up with YouTube.

Dataset overview

All numbers below come from each dataset's paper, project page or official repo. Durations appear only where the source states them.

YT-Temporal-180M (6 million videos, 180M extracted frames) and YT-Temporal-1B (MERLOT Reserve, 20 million videos) aren't in the table. The MERLOT repo still lists "information about the videos used in this work" as planned, and I found no public ID list for either. Ask the authors directly.

HD-VILA's numbers are consistent with each other: 100M clips × 13.4 s ≈ 372k hours. That means the 371.5k hours figure counts clip duration, not full source-video duration.

Step 1: get the metadata

• Panda-70M: CSVs are linked from the dataset_dataloading README. Training sets are zipped (full: 2.73 GB; 10M: 504 MB; 2M: 118 MB). The 10M subset keeps at most 3 clips per source video, and every clip has a matching score above 0.43.
• HD-VILA-100M: the README links a URL archive and a metadata archive on Azure Blob, last updated August 2026. Each JSONL record has video_id, url and a clip list of {clip_id, span: [start, end]}.
• InternVid: OpenGVLab/InternVid on Hugging Face is gated. You fill in name, organization and email and accept the terms before the JSONL files become available.
• AudioSet: three CSVs (eval 20,383, balanced train 22,176, unbalanced train 2,042,985 segments).

Step 2: choose your downloader

Panda-70M: the official video2dataset fork

Panda-70M's README says to use the bundled fork, because "the video2dataset from the official repository cannot work with our csv format":

git clone https://github.com/snap-research/Panda-70M.git
cd Panda-70M/dataset_dataloading/video2dataset
pip install -e .
cd ..

video2dataset --url_list="panda70m_training_2m.csv" \
              --url_col="url" \
              --caption_col="caption" \
              --clip_col="timestamp" \
              --output_folder="panda70m_2m" \
              --save_additional_columns="[matching_score,desirable_filtering,shot_boundary_detection]" \
              --config="video2dataset/video2dataset/configs/panda70m.yaml"

The bundled panda70m.yaml sets download_size: 360 and download_audio: True, downloads English subtitles, and writes 100 samples per shard using 32 processes × 32 threads. The README's known-issues table maps HTTP 403 to "your IP got blocked" and HTTP 429 to "reduce processes_count and thread_count". It also notes that several YouTube errors are fixed by upgrading yt-dlp, so run pip install -U "yt-dlp[default]" inside that environment after installing. Current yt-dlp needs Python 3.11+, plus yt-dlp-ejs and a JavaScript runtime such as Deno for full YouTube support (see the yt-dlp README).

Upstream video2dataset

iejMac/video2dataset is still at that address (MIT license, not archived). Its last push was in July 2024, and the latest PyPI release is 1.3.0 from February 2024, so treat it as stable but largely unmaintained. It reads CSV or Parquet (--input_format), writes files, webdataset, parquet or tfrecord (--output_format), and distributes work over multiprocessing, PySpark or Slurm. The README says PySpark mode was used to download 40M videos. It's a reasonable base for InternVid or HD-VILA, but you'll have to convert their timestamp formats to whatever clip_col format your version expects. Test on 1,000 rows before launching a full run.

yt-dlp directly

For custom pipelines, call yt-dlp directly. How you handle clipping depends on how many clips come from each source video:

• Many clips per video (Panda-70M averages ~18.7 clips per source video in the full set; HD-VILA has ~30): download the whole video once, then cut it locally.
• One short segment per video (AudioSet, Kinetics): download just the section with --download-sections. The * prefix marks a time range, and the option needs ffmpeg.

# Full video, capped at 360p, smallest file among the remaining candidates
yt-dlp -S "res:360,+size" --merge-output-format mp4 \
  -o "videos/%(id)s.%(ext)s" \
  "https://www.youtube.com/watch?v=QMi8x8o55Ns"

# One 10-second AudioSet segment, audio only
yt-dlp -f bestaudio -x --audio-format flac \
  --download-sections "*420-430" \
  -o "audio/%(id)s_%(section_start)s.%(ext)s" \
  "https://www.youtube.com/watch?v=-0RWZT-miFs"

Section downloads cut at keyframes by default. Add --force-keyframes-at-cuts for accurate boundaries. The README warns that this is slow because it re-encodes.

Cutting clips with ffmpeg

HD-VILA's cut_videos.py uses ffmpeg -ss … -t … -c copy. Stream copy is fast, but each clip starts at the nearest keyframe, so a 5-second clip can pick up extra frames. If your training needs exact caption alignment, re-encode:

import json
import subprocess
from pathlib import Path

def to_sec(ts):  # "00:04:52.140" to 292.14
    h, m, s = ts.split(":")
    return int(h) * 3600 + int(m) * 60 + float(s)

with open("meta_part0.jsonl") as f:  # one HD-VILA metadata part
    for line in f:
        meta = json.loads(line)
        src = Path("videos") / f"{meta['video_id']}.mp4"
        if not src.exists():
            continue
        out_dir = Path("clips") / meta["video_id"]
        out_dir.mkdir(parents=True, exist_ok=True)
        for c in meta["clip"]:
            start, end = (to_sec(t) for t in c["span"])
            subprocess.run([
                "ffmpeg", "-y", "-loglevel", "error",
                "-ss", f"{start:.3f}", "-i", str(src), "-t", f"{end - start:.3f}",
                "-c:v", "libx264", "-preset", "veryfast", "-crf", "23",
                "-c:a", "aac", "-b:a", "96k",
                str(out_dir / c["clip_id"]),
            ], check=True)

Choosing format and resolution

Resolution has more effect on storage and bandwidth than any other setting. In yt-dlp's format sorting, res:360 prefers the largest format that is no larger than 360p. Adding +size breaks ties toward smaller files. Panda-70M's tooling defaults to 360p for this reason. If you're training generation models that need HD, set a higher download_size and budget for it. Use audio-only formats for AudioSet and VoxCeleb-style work. Also consider whether you need audio at all: Panda's README notes that turning off download_audio speeds up downloads.

Link rot: plan for missing videos

Videos get deleted, set to private, region-locked or taken down by their owners. CVDF's Kinetics-700-2020 readme is one of the few published measurements. It was downloaded in September 2020 with youtube-dl:

• Train: 536,699 of 545,793 videos (98.3%)
• Validation: 33,966 of 34,256 (99.2%)
• Test: 64,535 of 67,858 (95.1%)

About 7,000 videos that were unavailable by September 2020 were recovered from a December 2019 crawl. Expect losses to grow the longer a dataset has been public. Panda-70M's README lists "set to private or removed" as an expected failure and tells you to skip those samples. VoxCeleb2's official site no longer provides the URL list at all. In practice, always report how many clips you actually obtained, and don't assume your numbers are directly comparable to the paper's.

Estimating storage

Formula: bytes = hours × 3600 × bitrate (bits/s) ÷ 8.

Panda-70M gives a calibration point: ~36 TB for 167k hours at 360p with audio. That's 36×10¹² B ÷ (167,000 × 3600 s) ≈ 59,900 B/s ≈ 0.48 Mbps. The table below assumes 0.5 Mbps for 360p and 2.5 Mbps for 720p H.264. These are round assumptions, not measurements.

These estimates cover the clips you store. Transfer volume is higher if you download full source videos and keep only the clipped parts, and the sources don't state total source duration for Panda-70M or HD-VILA. For transfer time: 37.6 TB × 8 ÷ 1 Gbps ≈ 300,800 s ≈ 3.5 days at full line rate, which you won't sustain in practice. Bandwidth for large-scale video downloads goes through the full calculation.

Output formats

• Loose files (video2dataset's default files): Panda-70M's layout is one folder per shard, with .mp4, .txt (caption) and .json (metadata, including status) per sample. This is easy to inspect but slow on object storage once you have millions of files.
• WebDataset tar shards (--output_format webdataset): this is what the video2dataset README recommends "for most use cases". It streams well from S3/GCS into PyTorch, TF or JAX.
• Parquet metadata: video2dataset writes a Parquet file per shard alongside the tars. Merge these into a single manifest (clip ID, video ID, start, end, caption, status, error, bytes, resolution, sha256, download date) so you can filter and audit without opening any shards.

Panda's README also warns that shards are unshuffled, so all clips from one source video end up in the same shard. Shuffle before training.

Resumability and failure tracking

video2dataset's incremental_mode (default incremental) skips finished shards on restart. This only works if you keep the same output folder, the same number_sample_per_shard and the same input list. Failed samples record "status": "failed_to_download" and an error_message in their JSON.

With plain yt-dlp, --download-archive FILE skips IDs that have already been downloaded. You still need to separate permanent failures from transient ones. A minimal status log:

import json
import subprocess
from pathlib import Path

IDS = Path("video_ids.txt")
LOG = Path("status.jsonl")
OUT = Path("videos")
PERMANENT = ("Video unavailable", "Private video", "removed by the uploader")

finished = set()
if LOG.exists():
    with LOG.open() as f:
        for line in f:
            rec = json.loads(line)
            if rec["status"] in ("ok", "gone"):
                finished.add(rec["id"])

OUT.mkdir(exist_ok=True)
with LOG.open("a") as log:
    for vid in IDS.read_text().split():
        if vid in finished:
            continue
        p = subprocess.run(
            ["yt-dlp", "--no-progress", "-S", "res:360,+size",
             "--merge-output-format", "mp4",
             "-o", str(OUT / "%(id)s.%(ext)s"),
             f"https://www.youtube.com/watch?v={vid}"],
            capture_output=True, text=True,
        )
        if p.returncode == 0:
            status = "ok"
        elif any(s in p.stderr for s in PERMANENT):
            status = "gone"
        else:
            status = "retry"  # rate limits, bot checks, network errors
        log.write(json.dumps({"id": vid, "status": status,
                              "error": p.stderr.strip()[-500:]}) + "\n")
        log.flush()

On the next run, IDs marked retry are attempted again, while ok and gone are skipped. YouTube's error strings change over time, so check a sample of retry errors periodically and update the matcher.

Scaling: rate limits and bot checks

At 10k+ videos, throughput is limited by YouTube's limits rather than your code. Typical failures are HTTP 429, 403 and "Sign in to confirm you're not a bot". Things that help:

• Lower concurrency per IP. With video2dataset that means processes_count and thread_count; with yt-dlp, --sleep-requests and --sleep-interval/--max-sleep-interval.
• Spread requests across egress IPs with --proxy http://USER:[email protected]:8080. See the yt-dlp proxy guide.
• Diagnose bot checks with fixing "Sign in to confirm you're not a bot" in yt-dlp.
• Keep yt-dlp current. Several of Panda-70M's known issues were fixed just by upgrading it.

For the end-to-end pipeline after download (dedup, filtering, captioning), see building a YouTube video dataset for multimodal training.

Licensing and compliance

The licenses above (Snap's non-commercial license, R-UDA, CC BY-NC-SA 4.0, CC BY 4.0) cover the annotations. They grant no rights to the underlying videos, which belong to their uploaders. YouTube's Terms of Service restrict automated access and downloading. Copyright and licensing still apply, and some teams filter for Creative Commons-licensed videos. Faces and voices raise privacy issues (e.g., GDPR), and takedown and opt-out requests should be honored. Panda-70M's authors, for example, offer removal on request. Get legal review for your jurisdiction and use case before collecting or redistributing.

If you'd rather not run this yourself

Corpora offers two options. If you run your own stack, dedicated 1–40 Gbps unmetered proxy capacity (HTTP(S)/SOCKS5, compatible with yt-dlp) is $2,000 per Gbps per month. Alternatively, a fully managed collection service delivers video, audio, subtitles and metadata (JSONL/Parquet with sha256 checksums and manifests) to your S3, GCS or Azure bucket, priced per delivered PB, with a free sample batch. Contact us to scope it.

FAQ

Can I download Panda-70M videos directly instead of from YouTube?

Not officially. The release consists of CSVs (URLs, timestamps, captions, scores) and code. The videos have to be fetched from YouTube with the bundled video2dataset fork or your own tooling.

How big is Panda-70M once downloaded?

The README lists ~36 TB for the full training set (70.7M clips, 167k hours), ~8.0 TB for the 10M subset and ~1.6 TB for the 2M subset, at the default 360p with audio.

Can I use HD-VILA-100M or InternVid commercially?

Their annotation licenses don't allow it: R-UDA v1.0 is limited to non-commercial research, and InternVid is CC BY-NC-SA 4.0. Separately, neither license covers rights to the videos themselves.

Why do I get fewer clips than the paper reports?

Link rot. Videos get removed or set to private over time. Kinetics-700-2020's own download already missed about 2% within months of release. Log every failure so you can report what you actually obtained.

Should I use video2dataset or yt-dlp?

Use the official fork for Panda-70M. Elsewhere, video2dataset gives you sharding, WebDataset/Parquet output and resumption. Plain yt-dlp gives you more control, but you write the bookkeeping yourself.