Tag: video data curation

  • Video Data Curation for Generative Models: Filters That Matter

    Raw video scraped at scale is mostly unusable for training a generative model as-is: hard cuts in the middle of clips, static slides, burned-in subtitles, shaky phone footage, letterboxing, near-duplicates. Published video model papers spend as much effort on curation as on architecture. This guide lays out the filters that matter, the order to run them in so you don't burn GPU hours on clips you'll throw away, and how to store the results so you can change your mind later.

    TL;DR

    • Split before you filter. Almost every downstream score is meaningless on a clip that contains a hard cut.
    • Run filters in cost order: metadata → shot detection → cheap frame statistics → optical flow → OCR → model-based aesthetic and alignment scores → near-duplicate search.
    • Store scores, not booleans. Thresholds change with every training run; recomputing scores over millions of clips does not have to.
    • Watch your keep-rate per filter. Over-aggressive aesthetic filters quietly turn a diverse dataset into stock footage.
    • Curation is an experiment, not a preprocessing step. Validate thresholds with small training runs, not by eyeballing samples.

    Why curation matters as much as scale

    Two widely cited papers make the case well. The Stable Video Diffusion paper (arXiv 2311.15127) describes a systematic curation pipeline for its large video pretraining set: cut detection, optical-flow-based motion filtering, text detection to drop clips dominated by on-screen text, and CLIP-based scoring for aesthetics and text-image similarity. The authors report that models trained on the curated subset were preferred over models trained on uncurated data. Panda-70M (arXiv 2402.19479) took long YouTube videos, split them into semantically coherent clips, and then invested heavily in captioning those clips.

    The common thread: a smaller, cleaner set of well-segmented clips beats a larger pile of raw footage. If you are building your own set, start from the papers' filter list and calibrate it to your target model.

    The filter stack, in cost order

    Stage Signal Relative cost Catches
    Metadata Duration, resolution, frame rate, aspect ratio, license, language Free (before download) Shorts you don't want, 360p uploads, wrong language
    Shot detection Frame-difference / content-change detection Low (CPU) Hard cuts, fades, montage edits
    Frame statistics Brightness, contrast, blur (Laplacian variance), black-border detection Low (CPU, sampled frames) Dark, blurry, letterboxed or pillarboxed clips
    Motion Optical flow magnitude on sampled frames Medium (GPU or fast CPU flow) Static slides and talking-head stills, or extreme camera shake
    Text / OCR Text detection area ratio Medium Burned-in subtitles, slides, screen recordings, heavy watermarks
    Aesthetic / quality Learned aesthetic or quality predictor on frame embeddings Medium–high (GPU) Low production value, compression artifacts
    Text–video alignment Similarity between caption and frames (CLIP-style) High (needs captions first) Captions that don't describe the clip
    Near-duplicates Perceptual hashes, embedding nearest-neighbor search Medium–high at scale Re-uploads, compilations, reaction videos, your eval set
    Safety / privacy NSFW classifiers, face detection, PII in text Medium Content you can't train on or must handle separately

    The ordering principle is simple: every stage should only see clips that passed all cheaper stages. A metadata filter that drops 30% of candidates before download saves 30% of bandwidth, storage and every GPU hour downstream.

    Stage by stage

    1. Metadata, before you download anything

    Filter on what the platform tells you for free: duration, available resolutions, aspect ratio, upload date, category and license. Decide early what you are training on. A text-to-video model rarely benefits from 12-hour livestreams or vertical 9:16 Shorts unless you specifically want them. See building a YouTube video dataset for multimodal training for the metadata-first workflow, and filtering for Creative Commons videos if licensing drives your selection.

    2. Shot detection and clip splitting

    Run this first on downloaded video, because every later score assumes a clip shows one continuous shot. Content-based detectors (for example PySceneDetect's content and adaptive detectors) handle hard cuts well; fades and dissolves need a threshold-based or learned detector. SVD reports using cut detection at multiple frame rates to catch transitions that a single pass misses.

    Splitting purely on cuts can over-fragment: an interview that alternates between two cameras becomes dozens of 2-second clips. Panda-70M addressed this by stitching adjacent shots back together when they were semantically similar. Decide what a "clip" means for your model — a single shot, or a coherent scene — and set minimum and maximum clip lengths accordingly.

    3. Cheap frame statistics

    Sample a few frames per clip and compute brightness, contrast, a blur score and black-border ratio. These are fast, explainable and catch a surprising share of junk: dark concert footage, out-of-focus phone video, 4:3 content pillarboxed inside 16:9. Crop borders rather than discarding clips where the content itself is fine.

    4. Motion

    Optical flow on a low-frame-rate sample (SVD used 2 FPS) gives you a motion score per clip. The bottom of the distribution is slides, still images with music and near-static talking heads — useless for learning motion. The top of the distribution is often camera shake or rapid editing. Most teams drop both tails, and keep the score so they can rebalance motion levels later.

    5. Text and OCR

    Text detection gives you the fraction of the frame covered by text. High values mean slides, tutorials, screen recordings or burned-in subtitles. A generative model trained on those learns to produce garbled text overlays. Small, persistent text in a corner is often a channel watermark — decide whether to drop or crop.

    6. Aesthetic and quality scores

    Learned aesthetic predictors on CLIP-style embeddings are cheap per frame once embeddings exist, and they correlate with production quality. They also carry taste bias: they reward well-lit, centered, stock-footage-looking frames. Use them to drop the bottom of the distribution, not to keep only the top, unless your target is explicitly high-aesthetic output.

    7. Caption alignment

    Once clips have captions, a text–video similarity score tells you when a caption doesn't describe its clip. This matters most when captions come from titles, descriptions or ASR transcripts, which are frequently about the video rather than what is on screen. Better captions usually beat stricter alignment filtering — see recaptioning video datasets.

    8. Near-duplicates and contamination

    Large web video collections are full of re-uploads, compilations and reaction videos that embed other videos. Two layers work well together: perceptual hashes on keyframes for exact and near-exact copies, and embedding nearest-neighbor search for semantic duplicates. Run the same search against your evaluation sets. Benchmark contamination is easy to introduce and hard to detect after training.

    9. Safety and privacy

    NSFW classifiers, face detection and text PII checks belong in the pipeline, not in a later audit. Faces and voices can be personal data under GDPR and similar laws, so record what you detected even if you keep the clip.

    Store scores, not decisions

    The most expensive mistake is writing only the filtered output. Thresholds change with every model, and recomputing optical flow or embeddings over tens of millions of clips is days of GPU time. Keep one row per clip with every score:

    {
      "clip_id": "dQw4w9WgXcQ_000123",
      "video_id": "dQw4w9WgXcQ",
      "start_s": 61.2,
      "end_s": 68.9,
      "width": 1920,
      "height": 1080,
      "fps": 25.0,
      "shot_boundary_confidence": 0.94,
      "brightness_mean": 0.41,
      "blur_laplacian_var": 212.5,
      "black_border_ratio": 0.0,
      "flow_magnitude_p50": 1.83,
      "ocr_area_ratio": 0.012,
      "aesthetic_score": 5.2,
      "nsfw_score": 0.01,
      "face_count_max": 1,
      "phash_cluster": "c_88412",
      "embedding_dup_of": null,
      "caption_alignment": 0.31,
      "pipeline_version": "curate-2026.11"
    }

    Your training set then becomes a query over this table. A new threshold is a new query, not a new pipeline run.

    Calibrating thresholds

    • Report keep-rate per filter. If one filter removes 60% of clips, look at what it removed before trusting it.
    • Look at the boundary, not the extremes. Sample clips just above and just below each threshold; that is where mistakes live.
    • Check distribution shift. Compare category, language, motion and resolution distributions before and after filtering. Curation that silently removes most sports or most outdoor footage changes what your model learns.
    • Validate with training runs. The real test is a small model trained on two subsets with different thresholds, evaluated on the same prompts.

    Compliance

    Curation decides what goes into the model, so it is also where compliance decisions get enforced. YouTube's Terms of Service restrict automated access and downloading, copyright and licensing apply to the underlying videos, and faces and voices can be personal data. Keep per-clip provenance (source ID, collection time, license at collection time), honor takedown and opt-out requests across derived clips, and get legal review for your jurisdiction and use case.

    If you'd rather not build this yourself

    Corpora collects YouTube video, audio, subtitles and structured metadata at petabyte scale and delivers deduplicated datasets with checksums and manifests to your S3, GCS or Azure bucket, with a free sample batch to validate quality first. See pricing or talk to sales.

    FAQ

    What is the first filter to run on raw video?

    Metadata filtering before download, then shot detection right after. Every other score assumes the clip is a single continuous shot.

    How much data do curation filters typically remove?

    It depends heavily on the source and your thresholds, so measure it on a sample. Web video often loses a large share to cuts, static content, text overlays and duplicates, which is why filtering before download is worth so much.

    Should I filter on aesthetic score?

    Use it to drop the worst clips. Keeping only the top of the distribution biases the dataset toward stock-footage style and reduces diversity.

    How do I avoid training on my evaluation set?

    Run near-duplicate search (perceptual hashes plus embedding similarity) of your training clips against every evaluation set, and record matches as a score so you can exclude them.