Corpora Blog

Tag: video-language models

  • How to Build a YouTube Video Dataset for Multimodal Training

    Downloading a few thousand YouTube videos is a script. Building a YouTube video dataset that a video-LLM, vision-language or speech model can train on (filtered, deduplicated, captioned, sharded, and traceable back to every source) is a pipeline. This article walks through that pipeline stage by stage, with the yt-dlp flags, open-source tools and published dataset recipes that each stage relies on, plus a runnable metadata script and a sample manifest record.

    TL;DR

    • Metadata first. Enumerate IDs with yt-dlp --flat-playlist, fetch full metadata with --skip-download, and filter on duration, license, language and resolution before you spend any bandwidth on video bytes.
    • The YouTube Data API is for enrichment, not discovery. The default allocation is 100 search.list calls per day plus 10,000 units for other endpoints. videos.list costs 1 unit.
    • Make the download stage idempotent. Use --download-archive, a job queue keyed by video ID, and one proxy session per worker.
    • Processing follows published recipes: shot detection (PySceneDetect), clip stitching or splitting, near-duplicate removal, motion/OCR/aesthetic filters (as in Stable Video Diffusion), recaptioning (as in Panda-70M), and ASR with Whisper.
    • Store samples as WebDataset tar shards, keep metadata in Parquet/JSONL, put a sha256 on every object, and record lineage (video ID, license, collection time) per sample so takedowns can be applied.

    The pipeline at a glance

      seeds (keywords, channels, playlists, existing ID lists)
     |
     v
[1] DISCOVER   yt-dlp --flat-playlist -j  /  Data API videos.list
     |            -> candidate IDs (cheap, no media)
     v
[2] FILTER     full metadata (--skip-download): duration, license,
     |            language, views, date, category, max height
     |            -> selected IDs + metadata.jsonl
     v
[3] DOWNLOAD   job queue -> workers (yt-dlp, 1 proxy session each)
     |            video | audio-only | subs | thumbnail | info.json
     v
[4] PROCESS    shot detection -> clips -> dedup -> quality filters
     |            -> captions / ASR / language ID
     v
[5] STORE      WebDataset tar shards + Parquet metadata + sha256 manifest
     |            -> object storage (S3 / GCS / Azure Blob)
     v
[6] GOVERN     lineage per sample, takedown list, re-release builds

    Stage 7, cost and throughput planning, applies to all of them. Each stage writes its output to durable storage, so a failure in stage 4 never forces a re-download from stage 3.

    Stage 1: Scope and discovery

    Start from what the model needs: target hours, modalities (video+audio, audio only, video+captions), languages, domains and minimum resolution. Then build seed lists:

    • Channels and playlists are the highest-yield seeds. Topically coherent channels produce consistent data.
    • Keyword lists produce search result pages. Each query yields a limited number of results, so you need many queries.
    • Existing ID lists from open datasets. Panda-70M, HD-VILA-100M and InternVid all ship YouTube IDs, covered in our guide to downloading Panda-70M, HD-VILA-100M and InternVid.

    Enumerate without resolving each video. In yt-dlp, --flat-playlist skips extraction of a playlist's entries, and -j/--dump-json prints one JSON object per entry without downloading:

    yt-dlp --flat-playlist -j "https://www.youtube.com/@CHANNEL/videos" > seeds/channel.jsonl
yt-dlp --flat-playlist -j "ytsearch200:woodworking tutorial" > seeds/q_woodworking.jsonl

    Flat entries are thin: the README warns that "some entry metadata may be missing". Typically you get an ID, a title and maybe a duration. Treat this as the candidate list, not as filter input.

    YouTube Data API v3 as an alternative

    Google's quota documentation currently says projects get a default allocation of 100 search.list calls, 100 videos.insert calls, and 10,000 units per day for all other endpoints combined. Quotas reset at midnight Pacific Time. videos.list costs 1 unit and accepts a comma-separated list of IDs. The video resource exposes status.license (creativeCommon vs. youtube) and snippet.defaultAudioLanguage. That makes the API good for authoritative enrichment of IDs you already have, and a poor fit for discovery at scale given the search cap. Older articles quote "10,000 units, search costs 100", so check the current page before you plan around either model.

    Stage 2: Filter on metadata before downloading

    Video bytes are the main cost. Metadata costs a few requests per video. Fetch full metadata for every candidate, then drop what you won't train on:

    Filter yt-dlp field Why
    Duration duration Drop shorts and multi-hour streams that blow up processing.
    Live status live_status, is_live Skip ongoing and upcoming streams.
    License license Creative Commons-only subsets. YouTube CC videos report "Creative Commons Attribution license (reuse allowed)".
    Language subtitles, automatic_captions, per-format language Cheap pre-filter. Confirm later with audio language ID.
    Popularity / date view_count, upload_date Proxy for production quality; time-sliced splits.
    Category categories, tags Domain balance (e.g. cap gaming or music).
    Resolution formats[].height Skip videos whose best format is below your minimum.
    Availability availability, age_limit Skip anything that needs sign-in.

    The CLI route uses --skip-download with --write-info-json and --match-filters:

    yt-dlp -a candidates.txt --skip-download --write-info-json \
  --match-filters "duration>=30 & duration<=1800 & !is_live & license~='Creative Commons'" \
  -o "meta/%(id)s.%(ext)s"

    For a pipeline, the Python API is easier to control. The script below reads IDs, fetches metadata with extract_info(download=False), keeps the fields you filter on, and appends JSONL. It's resumable: IDs that already have a record are skipped, and failures go to a separate file so they get retried on the next run.

    import json
import os
import sys
from datetime import datetime, timezone

import yt_dlp
from yt_dlp.utils import DownloadError

KEEP = ["id", "title", "channel", "channel_id", "duration", "upload_date",
        "license", "categories", "tags", "view_count", "availability",
        "live_status", "age_limit"]


def done_ids(path):
    """IDs that already have a successful record (errors get retried)."""
    done = set()
    if os.path.exists(path):
        with open(path, encoding="utf-8") as f:
            for line in f:
                try:
                    done.add(json.loads(line)["id"])
                except (ValueError, KeyError):
                    continue  # tolerate a truncated last line
    return done


def to_record(info):
    rec = {k: info.get(k) for k in KEEP}
    formats = info.get("formats") or []
    rec["max_height"] = max((f.get("height") or 0 for f in formats), default=0)
    rec["audio_langs"] = sorted({f["language"] for f in formats
                                 if f.get("acodec") not in (None, "none")
                                 and f.get("language")})
    rec["subtitle_langs"] = sorted(info.get("subtitles") or {})
    rec["has_auto_captions"] = bool(info.get("automatic_captions"))
    return rec


def now():
    return datetime.now(timezone.utc).isoformat(timespec="seconds")


def main(ids_path, out_path, proxy=None):
    with open(ids_path, encoding="utf-8") as f:
        ids = list(dict.fromkeys(line.strip() for line in f if line.strip()))
    done = done_ids(out_path)
    todo = [i for i in ids if i not in done]

    opts = {"quiet": True, "no_warnings": True, "skip_download": True,
            "socket_timeout": 30, "extractor_retries": 3}
    if proxy:
        opts["proxy"] = proxy  # e.g. http://USER:[email protected]:8000

    with yt_dlp.YoutubeDL(opts) as ydl, \
            open(out_path, "a", encoding="utf-8") as out, \
            open(out_path + ".errors", "a", encoding="utf-8") as err:
        for vid in todo:
            url = f"https://www.youtube.com/watch?v={vid}"
            try:
                info = ydl.sanitize_info(ydl.extract_info(url, download=False))
            except DownloadError as e:
                err.write(json.dumps({"id": vid, "error": str(e),
                                      "fetched_at": now()}) + "\n")
                continue
            rec = to_record(info)
            rec["fetched_at"] = now()
            out.write(json.dumps(rec, ensure_ascii=False) + "\n")
            out.flush()


if __name__ == "__main__":
    if len(sys.argv) not in (3, 4):
        sys.exit("usage: fetch_meta.py ids.txt metadata.jsonl [proxy_url]")
    main(*sys.argv[1:])

    Run it as python fetch_meta.py ids.txt metadata.jsonl. Then filter metadata.jsonl with pandas or DuckDB to produce selected.txt. sanitize_info is there because the yt-dlp README says extract_info's return value isn't guaranteed to be JSON-serializable. Current yt-dlp lists Python 3.11+ as supported, and it needs a JavaScript runtime plus yt-dlp-ejs for full YouTube support. If metadata requests start failing with bot checks, see fixing "Sign in to confirm you're not a bot".

    Stage 3: Download

    One yt-dlp invocation per job, configured for the modality you need:

    # video + subtitles + thumbnail + metadata, capped at 720p
yt-dlp -a selected.txt \
  -S "res:720,fps" --merge-output-format mp4 \
  --write-subs --write-auto-subs --sub-langs "en.*,ja" --sub-format "vtt/best" \
  --write-thumbnail --write-info-json \
  --download-archive archive.txt \
  -R 10 --fragment-retries 10 --retry-sleep exp=1:60 \
  --sleep-requests 1 --sleep-subtitles 2 \
  --proxy "http://USER:[email protected]:8000" \
  -o "raw/%(id)s/%(id)s.%(ext)s"

# speech corpus: best audio-only stream, no video bytes at all
yt-dlp -a selected.txt -f "ba" --download-archive archive_audio.txt \
  -o "raw_audio/%(id)s.%(ext)s"
    • Format selection. -f chooses formats. -S sorts them, and res:720 prefers the largest resolution not above 720p. Pin a resolution ceiling: the bytes difference between 720p and 4K dominates your bill.
    • Audio-only. -f ba keeps the native stream (usually Opus or AAC). Add -x --audio-format flac only if your ASR stage needs it. Decoding to 16 kHz mono at training time is usually cheaper than storing WAV.
    • Subtitles. --write-subs fetches uploader captions and --write-auto-subs fetches YouTube's ASR. Keep them separate in metadata: they differ in quality, and auto-captions are a weak label.
    • Idempotency. --download-archive records completed IDs, so re-runs skip them. At scale, put IDs in a queue (Redis, SQS, a Postgres table with SELECT ... FOR UPDATE SKIP LOCKED), lease jobs with a timeout, and mark a job done only after the upload to object storage and its checksum are confirmed.
    • Proxies. Give each worker its own proxy session and keep one video on one IP for its whole lifetime (metadata, manifest, fragments). Details, including per-worker configuration, are in the yt-dlp proxy guide.

    Stage 4: Processing

    Shot detection and clip splitting

    Most video-text training uses clips of a few seconds to a minute, not whole videos. PySceneDetect provides ContentDetector, a two-pass AdaptiveDetector that handles fast camera movement better, and ThresholdDetector for fades. Splitting needs ffmpeg (or mkvmerge):

    from scenedetect import detect, AdaptiveDetector, split_video_ffmpeg

scenes = detect("raw/abc123XYZ_-/abc123XYZ_-.mp4", AdaptiveDetector())
keep = [(s, e) for s, e in scenes if 2.0 <= (e - s).get_seconds() <= 60.0]
split_video_ffmpeg("raw/abc123XYZ_-/abc123XYZ_-.mp4", keep)

    Raw shot boundaries over-split. Panda-70M runs shot boundary detection with PySceneDetect first. It then uses ImageBind frame embeddings to stitch adjacent clips that are semantically similar. Its splitting code also cuts the first 5 seconds of any clip longer than 7 seconds, recursively, to catch fades and unedited footage. Store clip boundaries as timestamps in metadata. Whether you materialize cut files or decode ranges at load time is a storage-vs-compute trade-off.

    Deduplication

    • Exact: sha256 of the downloaded file. This only catches byte-identical copies.
    • Perceptual: re-uploads are re-encoded, so hash sampled frames (e.g. 1 fps) with a perceptual hash and compare sequences by Hamming distance.
    • Semantic near-dup: embed clips with an image/video encoder, index them in an approximate-nearest-neighbor index, and drop pairs above a cosine-similarity threshold you tune on a labeled sample.
    • Against evaluation sets: drop any video ID that appears in benchmarks you report on, to avoid contamination.

    Quality filters

    The Stable Video Diffusion paper is one of the clearer public write-ups. It uses cascaded cut detection, dense optical flow computed at 2 FPS to remove near-static clips below a motion threshold, text detection to remove clips with excessive text, and CLIP-based aesthetic and text-image similarity scores. The same paper notes that InternVid-10M is "specifically filtered for high aesthetics". Practical guidance:

    • Compute scores and store them as metadata columns rather than deleting clips. Thresholds change between training runs, and re-scoring is expensive.
    • Calibrate each threshold by viewing a few hundred clips on both sides of it.
    • Text-heavy filters (slides, lyric videos, screen recordings) matter for generation models. For document-understanding models you may want exactly those clips.

    Captioning and recaptioning

    Titles and descriptions are noisy captions, and ASR transcripts describe speech, not visuals. Panda-70M's approach is to caption each clip with multiple cross-modality teacher models (using frames, the video description and subtitles as inputs). A retrieval model, fine-tuned on a small manually-labeled subset, then picks the best caption per clip. A simpler version: run one strong vision-language model per clip, prompted with the video title and the ASR text for that time range. Store the model name and prompt version with every caption.

    ASR and language ID

    Whisper does multilingual transcription, translation and spoken language identification. The turbo model is an optimized large-v3 with faster transcription and minimal accuracy loss. Per the README, it isn't trained for translation. For language ID, model.detect_language(mel) returns per-language probabilities. Compare the result with the metadata language and caption tracks: mismatches often flag dubbed content, music, or mislabeled uploads. Whisper's accuracy varies widely by language (the README publishes per-language WER/CER), so set per-language confidence thresholds.

    Stage 5: Storage layout and formats

    WebDataset stores samples as plain tar files. Files belonging to one sample share a basename, and shards are numbered like name-000000.tar. That gives sequential reads from object storage and trivial parallelism.

    s3://bucket/ytds/v2026-11/
  shards/clips-000000.tar        # abc123XYZ_-_000012.mp4 / .json / .txt / .opus
  shards/clips-000001.tar
  metadata/clips.parquet         # one row per clip: scores, captions, lineage
  metadata/videos.parquet        # one row per source video
  manifest.jsonl                 # one line per object: path, bytes, sha256
  takedowns.jsonl                # removed video_ids, with date and reason
    • Keep raw downloads in a separate prefix from training shards, so you can re-cut or re-filter without re-downloading.
    • Parquet for analytical metadata (filtering, stats, joins). JSONL for append-only logs and human-readable manifests.
    • Shard size: pick a size that streams efficiently from your object store. Don't write millions of small per-clip objects.
    • Version releases (v2026-11) and never mutate a published shard. Takedowns produce a new release.

    Stage 6: Data governance and lineage

    Every sample should answer: where did this come from, when, under what license, and has it been withdrawn? A clip-level record (illustrative values):

    {
  "sample_id": "abc123XYZ_-_000012",
  "video_id": "abc123XYZ_-",
  "source_url": "https://www.youtube.com/watch?v=abc123XYZ_-",
  "channel_id": "UCxxxxxxxxxxxxxxxxxxxxxx",
  "license": "Creative Commons Attribution license (reuse allowed)",
  "upload_date": "20240611",
  "collected_at": "2026-10-28T14:03:11Z",
  "collector_version": "yt-dlp 2026.10.xx / pipeline 1.4.0",
  "clip": {"start": 83.417, "end": 91.758, "height": 720, "fps": 30},
  "audio": {"lang_meta": "en", "lang_whisper": "en", "lang_prob": 0.97},
  "captions": [
    {"source": "vlm", "model": "MODEL_NAME", "prompt_version": "v3", "text": "..."},
    {"source": "asr", "model": "whisper-turbo", "text": "..."}
  ],
  "scores": {"motion": 3.1, "ocr_text_area": 0.02, "aesthetic": 5.4},
  "dedup_cluster": 1882301,
  "shard": "clips-000417.tar",
  "sha256": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"
}

    Takedown handling is a join: keep takedowns.jsonl keyed by video_id or channel_id, rebuild the affected shards without those samples, publish a new version, and delete the raw files. Periodically re-check source availability. A video that became private or was deleted is a strong signal to drop it from future releases.

    Stage 7: Cost and throughput planning

    Size everything from hours and bitrate:

    bytes      = hours x 3600 x avg_bitrate_bps / 8
throughput = bytes x 8 / collection_window_seconds

example (assumed 2 Mbps average at a 720p cap):
  1,000,000 h x 3600 x 2e6 / 8   = 9.0e14 B  = 900 TB
  9.0e14 x 8 / (30 x 86400 s)    = ~2.78 Gbps sustained for 30 days

    Measure your own average bitrate from filesize/filesize_approx and duration in a metadata sample, because it varies widely by resolution, codec and content. Then budget for headroom: retries, metadata requests, re-downloads after failures, and the fraction of IDs that turn out unavailable. GPU stages (embedding, captioning, ASR) usually become the bottleneck after download. Benchmark clips per GPU-hour on your own hardware before you commit to a schedule. Detailed tables are in bandwidth planning for large-scale video downloads.

    Compliance

    YouTube's Terms of Service restrict automated access and downloading. Copyright and licensing still apply to the content: some teams restrict collection to Creative Commons-licensed videos and track the license per sample. Faces and voices are personal data under regimes like GDPR. Honor takedown and opt-out requests, and design the pipeline so you can. Get legal review for your jurisdiction and use case before you collect or redistribute.

    If you'd rather not run this yourself

    Corpora offers dedicated 1–40 Gbps unmetered proxy capacity for teams running their own yt-dlp stack. It also offers a fully managed option that collects, deduplicates and delivers video, audio, subtitles and metadata (JSONL/Parquet with sha256 checksums and manifests) to your S3, GCS or Azure bucket, with a free sample batch. Get in touch to scope it.

    FAQ

    How do I get only metadata from YouTube with yt-dlp?

    Use -j/--dump-json to print JSON without downloading, or --skip-download --write-info-json to write .info.json files. Add --flat-playlist when you only need the entry list of a channel or playlist.

    How many API calls does the YouTube Data API allow per day?

    Google's quota page currently states a default of 100 search.list calls, 100 videos.insert calls and 10,000 units per day for other endpoints, resetting at midnight Pacific Time. You can request more through Google's quota audit process.

    How do I filter for Creative Commons videos?

    In yt-dlp, filter on the license field (e.g. --match-filters "license~='Creative Commons'"). In the Data API, read status.license, which is creativeCommon for CC videos.

    Should I store whole videos or pre-cut clips?

    Keep the raw videos plus clip timestamps as the source of truth. Materialize cut clips into WebDataset shards for each training release. That way, changing the splitting or filtering logic doesn't require a re-download.

    Which ASR model should I use for a YouTube speech dataset?

    Whisper is the common default. turbo is the faster option for transcription, while the multilingual models are needed for translation. Validate WER on a hand-transcribed sample per language before trusting the transcripts as labels.