Tag: training data quality

  • Video Deduplication: Near-Duplicate Detection for Training Data

    Web-scale video collections are full of copies: re-uploads, mirrored and cropped versions, compilations that stitch together other people's clips, and reaction videos with the original running in a corner. Exact duplicates are easy to remove. Near-duplicates decide how much of your training budget goes to the same footage seen ten times. This guide builds a layered video deduplication pipeline (exact hashes, perceptual frame hashes, audio fingerprints, embedding search with FAISS) and covers thresholds, clustering, which copy to keep, and cost at scale.

    TL;DR

    • Layer cheap before expensive: video ID and sha256 → perceptual hashes on sampled frames → audio fingerprints → embedding nearest-neighbor search → video-level verification.
    • Match at frame or segment level, decide at video or clip level. Compilations and reaction videos only share part of their content with the source.
    • Calibrate thresholds on labeled pairs before you delete anything.
    • Cluster with care: similarity edges can chain unrelated videos into one giant cluster.
    • Keep the best copy by explicit rules (license, original uploader, resolution) and record the cluster ID so takedowns propagate to every copy.
    • Search your evaluation sets too, with a looser threshold.

    Why duplicates hurt

    Duplicates do three kinds of damage:

    • Wasted compute and skewed weighting. A viral clip with 40 re-uploads is effectively upweighted 40x.
    • Memorization. In Extracting Training Data from Diffusion Models, Carlini et al. extracted near-copies of training images from Stable Diffusion and Imagen. They focused their search on duplicated training examples because, citing prior work, those are orders of magnitude more likely to be memorized. For text, Lee et al. report that deduplicated training data made models emit memorized text about ten times less often and reach the same or better accuracy in fewer steps.
    • Evaluation contamination. Lee et al. also found train-test overlap affecting over 4% of the validation sets of standard text datasets. Many video benchmarks come from YouTube too, so a re-upload of a benchmark clip silently inflates your scores.

    Removal also tends to be cheap in accuracy terms: SemDeDup reports removing 50% of a LAION subset via embedding-based semantic dedup with minimal performance loss. That is image-text data; measure on your own video set.

    The duplicate taxonomy

    Different copies need different detectors. No single method catches all of them.

    Duplicate type What changes Catches it Misses it
    Same video listed twice Nothing Video ID —
    Byte-identical file Nothing sha256 —
    Re-upload / re-encode Codec, bitrate, resolution, small trims pHash/dHash on sampled frames, audio fingerprint sha256, video ID
    Mirrored copy Horizontal flip pHash of the flipped query, embeddings Plain pHash
    Cropped / letterboxed / watermarked Borders, overlays, aspect ratio Embeddings, crop-resistant hashing, copy-detection descriptors Plain pHash on large crops
    Compilation Segments of many sources concatenated Frame-level matching with per-segment coverage Whole-video embeddings
    Reaction video Source shrunk into a picture-in-picture, new audio on top Copy-detection descriptors, sometimes audio fingerprint pHash, whole-frame embeddings

    Layer 1: exact deduplication

    Start with the free wins. Dedup source lists on the platform video ID before you download anything. Overlapping playlists, search queries and public datasets like Panda-70M and HD-VILA often reference the same YouTube IDs (see downloading Panda-70M, HD-VILA and InternVid). Every ID you drop here saves bandwidth, storage and all downstream compute.

    After download, sha256 each file and keep the digest in your manifest. It catches byte-identical copies and doubles as an integrity check, but misses the same video in two formats, let alone a re-encode.

    Layer 2: perceptual hashes on sampled frames

    A perceptual hash maps visually similar images to similar bit strings, so the Hamming distance between hashes approximates visual difference. The Python imagehash library implements average, perceptual (phash, DCT-based), difference (dhash), wavelet and crop-resistant hashing. With the default hash_size=8, phash and dhash produce 64-bit hashes, and subtracting two hashes returns the Hamming distance.

    Sample frames at a fixed rate rather than extracting codec keyframes. Different encodes of the same video place I-frames at different positions. Fixed-rate sampling keeps the frame timelines of two copies roughly aligned.

    mkdir -p frames/VIDEO_ID
    ffmpeg -hide_banner -loglevel error -i VIDEO_ID.mp4 \
      -vf "fps=1,scale=256:-2" -q:v 3 frames/VIDEO_ID/%06d.jpg

    Then hash every frame, index the hashes in a FAISS binary index (Hamming distance, IndexBinaryFlat), and query with both the original and the mirrored frame:

    from collections import Counter, defaultdict
    from pathlib import Path
    
    import faiss
    import imagehash
    import numpy as np
    from PIL import Image, ImageOps
    
    MAX_HAMMING = 8      # starting point for 64-bit pHash; calibrate on labeled pairs
    MIN_STD = 10.0       # skip near-uniform frames (black, white, solid title cards)
    
    def bits(h):
        return np.packbits(h.hash.flatten())          # 64 bits -> 8 uint8
    
    codes, mirrored, owners = [], [], []
    for p in sorted(Path("frames").glob("*/*.jpg")):
        img = Image.open(p).convert("L")
        if np.asarray(img).std() < MIN_STD:
            continue
        codes.append(bits(imagehash.phash(img)))
        mirrored.append(bits(imagehash.phash(ImageOps.mirror(img))))
        owners.append(p.parent.name)                  # folder name = video ID
    
    codes = np.stack(codes)
    index = faiss.IndexBinaryFlat(64)
    index.add(codes)
    
    frames_per_video = Counter(owners)
    hits = defaultdict(set)                           # (a, b) -> frames of a matched in b
    for queries in (codes, np.stack(mirrored)):
        D, I = index.search(queries, 10)
        for qi in range(len(queries)):
            a = owners[qi]
            for dist, j in zip(D[qi], I[qi]):
                if j >= 0 and dist <= MAX_HAMMING and owners[j] != a:
                    hits[(a, owners[j])].add(qi)
    
    coverage = {pair: len(s) / frames_per_video[pair[0]] for pair, s in hits.items()}
    for (a, b), c in sorted(coverage.items(), key=lambda kv: -kv[1])[:20]:
        print(f"{a} -> {b}: {c:.0%} of {a}'s frames matched")

    Three details matter more than the hash choice:

    • Filter low-information frames. Black frames, fades and solid title cards hash almost identically across unrelated videos and generate false matches.
    • Coverage is asymmetric. If 90% of A's frames match B but only 5% of B's match A, A is likely a clip lifted from B, or B is a compilation containing A. Keep both directions.
    • pHash breaks under heavy crops and picture-in-picture. crop_resistant_hash helps with crops but is much slower. For reaction videos, use embeddings or a copy-detection model.

    Exhaustive binary search is popcount over 8 bytes per frame, so it is fast. At hundreds of millions of frames, switch to IndexBinaryIVF or IndexBinaryHash, documented on the same wiki page.

    Layer 3: audio fingerprints

    Re-uploads often keep the original audio even when the picture is cropped, flipped or overlaid. Chromaprint, the fingerprinting library behind AcoustID, ships a command-line tool, fpcalc. It decodes audio through FFmpeg, so it reads the audio track of a video file directly. According to its source, -length defaults to 120 seconds, and a value of 0 removes the limit. -raw outputs the uncompressed fingerprint as 32-bit integers, and -json makes it machine-readable. The current release on the project page is 1.6.2.

    fpcalc -raw -json -length 0 VIDEO_ID.mp4

    To compare two fingerprints, slide one against the other and measure the fraction of differing bits at the best offset:

    import json
    import subprocess
    
    import numpy as np
    
    def raw_fp(path):
        out = subprocess.run(["fpcalc", "-raw", "-json", "-length", "0", path],
                             capture_output=True, text=True, check=True).stdout
        return np.array(json.loads(out)["fingerprint"], dtype=np.uint32)
    
    def bit_error_rate(a, b, max_offset=300, min_overlap=100):
        best = 1.0
        for off in range(-max_offset, max_offset + 1):
            x, y = a[max(off, 0):], b[max(-off, 0):]
            n = min(len(x), len(y))
            if n < min_overlap:
                continue
            diff = np.unpackbits((x[:n] ^ y[:n]).view(np.uint8)).sum()
            best = min(best, diff / (32 * n))
        return best   # 0.0 = identical; independent random bits would sit near 0.5

    Offsets are in fingerprint items, not seconds. This brute-force alignment verifies candidate pairs; it is not a search method. To find candidates at scale, put the 32-bit items in an inverted index and only align pairs that share many items. Chromaprint was built to identify music, so treat matches on speech-only or ambient audio with more suspicion, and skip near-silent tracks: silence matches silence.

    Layer 4: embedding-based near-duplicate search

    Embeddings catch what hashes miss: crops, letterboxing, color grading and overlays. The standard recipe is to embed sampled frames with a CLIP-family encoder, normalize, and search neighbors by cosine similarity. Carlini et al. found near-duplicates among 160 million training images the same way, comparing CLIP embeddings instead of pixels.

    OpenCLIP loads both CLIP and SigLIP checkpoints (for example ViT-SO400M-14-SigLIP-384). FAISS turns inner product into cosine similarity once vectors are L2-normalized, as described in its metric notes:

    import faiss
    import numpy as np
    import open_clip
    import torch
    from PIL import Image
    
    device = "cuda" if torch.cuda.is_available() else "cpu"
    model, _, preprocess = open_clip.create_model_and_transforms(
        "ViT-B-32", pretrained="laion2b_s34b_b79k")
    model = model.to(device).eval()
    
    @torch.no_grad()
    def embed(paths, batch=256):
        out = []
        for i in range(0, len(paths), batch):
            x = torch.stack([preprocess(Image.open(p).convert("RGB"))
                             for p in paths[i:i + batch]]).to(device)
            out.append(model.encode_image(x).float().cpu().numpy())
        return np.ascontiguousarray(np.concatenate(out), dtype="float32")
    
    X = embed(frame_paths)              # frame_paths: list of sampled frame files
    faiss.normalize_L2(X)
    
    index = faiss.index_factory(X.shape[1], "IVF4096,Flat", faiss.METRIC_INNER_PRODUCT)
    index.train(X)                      # use "Flat" instead for small sets
    index.add(X)
    index.nprobe = 16
    S, I = index.search(X, 20)          # S = cosine similarity, I = neighbor ids

    Practical notes:

    • Search twice. Run a coarse pass on one pooled (mean) embedding per clip to find candidates, then verify at frame level, as in the pHash coverage logic above.
    • Semantic similarity is not duplication. Two news segments from the same studio, or two playthroughs of the same game level, embed very close together. Require temporal consistency (many frames matching at a stable time offset) before calling something a copy.
    • For reaction videos and heavy edits, use a copy-detection descriptor. Meta's SSCD is trained for image copy detection rather than semantics. The 2023 Video Similarity Challenge paper provides a benchmark and baselines for detecting and temporally localizing copied segments between videos.

    Thresholds: calibrate, then audit

    Every threshold here (Hamming distance, bit error rate, cosine similarity, frame coverage) trades precision against recall. The values in the code above are starting points. To set them:

    1. Sample candidate pairs in bands around each threshold, for example cosine 0.85–0.90, 0.90–0.95 and 0.95+.
    2. Have a person label each pair: same footage, same footage with heavy edits, or different.
    3. Pick the threshold where precision is acceptable for deletion. Report it with a confidence interval, not a point estimate. Our post on dataset audits with Wilson intervals covers the sample sizes.

    Use two thresholds: a strict one for deleting within your training set, and a looser one for flagging overlap with evaluation sets, where a miss costs more than a false alarm.

    From pairs to clusters

    Turn verified pairs (for example, coverage of 0.8 or more) into duplicate groups with union-find, also called disjoint-set, over the edges. Then watch for chaining. If A matches B and B matches C, all three land in one cluster even when A and C share nothing. Compilations act as bridges that can link dozens of unrelated sources. If one cluster has thousands of members, inspect it. Common fixes are to keep compilations from creating edges (handle them at segment level instead), to require mutual coverage, or to cluster first and dedup within clusters, which is the approach SemDeDup takes with k-means.

    Choosing which copy to keep

    Within each cluster, rank members with explicit, deterministic rules and keep the top one:

    1. License and provenance: prefer copies with a clear license (see Creative Commons videos for AI training) and the original uploader's channel over re-uploaders.
    2. Earliest upload date as a proxy for the original.
    3. Highest resolution and bitrate, no letterboxing.
    4. Fewest overlays: lower OCR text area, no burned-in watermark.
    5. Tie-break on video ID so reruns produce the same answer.

    For compilations, drop the duplicated segments and keep any unique ones, which requires the temporal localization from the frame-level step. Store dup_cluster_id, dup_role (kept or dropped) and the matched segments as columns, alongside the other curation scores described in video data curation filters. Never delete the evidence. Thresholds will change.

    What it costs at scale

    The arithmetic is dominated by how many vectors you store. Take an example corpus of 1,000,000 videos averaging 8 minutes, sampled at 1 frame per second:

    • Frames: 1,000,000 × 480 s × 1 fps = 480 million.
    • 64-bit pHash: 480M × 8 bytes = 3.84 GB. Fits in RAM on one machine.
    • 512-d float32 embeddings (ViT-B-32): 480M × 512 × 4 bytes = ~983 GB, which is the 4*d bytes per vector listed for flat indexes in the FAISS index overview.
    • One embedding per 5 seconds instead: 96M vectors × 2,048 bytes = ~197 GB flat. With IndexIVFPQ at 64 bytes of code plus 8 bytes of ID, that becomes 96M × 72 bytes = ~6.9 GB, at some loss of accuracy. Rerank the candidates with exact vectors.
    • All-pairs comparison of 96M vectors: 96M² / 2 ≈ 4.6 quadrillion pairs, which is why approximate nearest-neighbor search is required.

    Encoder throughput depends on model, resolution, batch size and decoding, so benchmark it on your hardware. Video decoding is often the bottleneck, so reuse the frames you already extract for shot detection and quality filters. And dedup IDs before download: every duplicate you skip is bandwidth you don't pay for (see bandwidth for large-scale video downloads).

    Compliance

    YouTube's Terms of Service restrict automated access and downloading. Copyright and licensing apply to every copy, and a re-upload's license may not reflect the original creator's. Faces and voices can be personal data under GDPR. Use your duplicate clusters to honor takedown and opt-out requests across every copy and derived clip, not just the one ID named in the request. Get legal review for your jurisdiction and use case.

    If you'd rather not run this yourself

    Corpora's fully managed option collects YouTube video, audio, subtitles and metadata and delivers them deduplicated, with sha256 checksums and manifests, to S3, GCS, Azure Blob or private object storage, with a free sample batch to run your own acceptance tests. For the questions to ask any data vendor about dedup and contamination, see the video training data buying checklist, or contact us.

    FAQ

    What is the difference between exact and near-duplicate video deduplication?

    Exact deduplication matches identical identifiers or bytes (video ID, sha256). Near-duplicate detection matches content that looks or sounds the same after re-encoding, cropping, flipping or editing. It uses perceptual hashes, audio fingerprints or embeddings.

    Is pHash or dHash better for video frames?

    Both tolerate re-encoding and resizing, and both break under heavy crops and picture-in-picture. In imagehash, phash thresholds low-frequency DCT coefficients of a downscaled grayscale frame, while dhash compares adjacent pixels, which is simpler and cheaper. Benchmark both on labeled pairs from your own data.

    What cosine similarity threshold should I use for CLIP-based deduplication?

    There is no universal value. It depends on the encoder, the frame sampling and whether you compare frames or pooled clips. Label pairs in similarity bands, pick the band where precision is acceptable, and require temporal consistency across several frames before treating a pair as a copy.

    How do I check my training set for benchmark contamination?

    Index your training frames, then query with frames from every evaluation video using a looser threshold than you use for training dedup. Also check exact IDs, since many benchmarks are built from YouTube. Record matches as a column so you can exclude them and report the overlap.

  • Recaptioning Video Datasets: Why Caption Quality Matters

    For a text-to-video or video-language model, the caption is half of every training example. Yet the text that comes with web video — titles, descriptions, tags, ASR transcripts — mostly describes what the video is about, not what is on screen. Recaptioning replaces or supplements that text with descriptions generated for training. This article covers why it works, how the published pipelines do it, and how to measure whether your captions are actually good.

    TL;DR

    • Native text is the wrong supervision. Titles are clickbait, descriptions are links, ASR is speech — none of it reliably describes the pixels.
    • Major labs recaption. OpenAI's DALL·E 3 paper and Sora technical report, Stable Video Diffusion and Panda-70M all describe generating their own captions.
    • Dense captions help, but mix lengths. Users type short prompts; train on both long and short captions, or expand prompts at inference.
    • Hallucination is the main failure. Audit a sample by hand and score alignment automatically.
    • Keep every caption with its provenance. Captioner models improve; you will recaption again.

    Why native text fails as training captions

    Source Typical example Problem
    Title "You won't BELIEVE what happened next!!" Describes intent or emotion, not content
    Description Links, sponsor reads, timestamps, hashtags Mostly not about the visuals at all
    Tags Keyword lists for search ranking Unordered, often stuffed
    ASR transcript What the speaker says Misaligned with what is shown; empty for silent or music-only clips
    Per-clip after splitting Same title for 50 clips One caption cannot describe 50 different shots

    The last row matters most at scale. Once you split videos into short clips (see video data curation for generative models), video-level text cannot describe individual clips no matter how good it is.

    What the published pipelines do

    • DALL·E 3 (OpenAI, "Improving Image Generation with Better Captions"): trained a dedicated captioner to produce descriptive synthetic captions and trained the image model mostly on those, reporting better prompt following than with original alt-text.
    • Sora (OpenAI technical report, "Video generation models as world simulators"): applied the same re-captioning technique to video, and used a language model to expand short user prompts into longer detailed captions at generation time, matching the training distribution.
    • Stable Video Diffusion (arXiv 2311.15127): generated multiple synthetic captions per clip with an image captioner on keyframes and a video captioner, and combined them with a language model.
    • Panda-70M (arXiv 2402.19479): captioned each clip with several cross-modality teacher models (using inputs such as frames, subtitles and metadata), then used a fine-tuned retrieval model to pick the caption that best matched the clip.
    • InternVid (arXiv 2307.06942): generated captions at multiple scales for its clips using a pipeline built on image captioning and language models.

    The pattern is consistent: generate candidate descriptions from the visuals, use other signals (transcripts, metadata) as context rather than as the caption, and select or merge the best result.

    What a good training caption contains

    For video generation and video understanding, a useful dense caption usually covers, in roughly this order:

    1. Subjects: who or what, with visible attributes (count, clothing, color, size).
    2. Action: what changes over the clip, in order.
    3. Scene: setting, time of day, weather, indoor or outdoor.
    4. Camera: shot type (close-up, wide), movement (pan, tilt, tracking, static, handheld).
    5. Style: live action, animation, screen recording, lighting, color grade.
    6. On-screen text, if any, quoted exactly or flagged.

    Camera motion and temporal order are where image captioners applied frame by frame fall short. A captioner that only sees one keyframe cannot tell a slow zoom from a static shot. That is why pipelines combine per-frame captions with a model that sees multiple frames, or a video-native captioner.

    A practical recaptioning pipeline

    clip (single shot, 2–20 s)
      │
      ├─ sample frames (e.g. 4–16, evenly spaced)
      ├─ per-frame captions          ← image captioner
      ├─ clip-level caption          ← video-language model on the frame sequence
      ├─ context (optional)          ← ASR snippet for this time range, title, OCR text
      │
      └─ merge / rewrite             ← LLM: produce dense + short caption,
                                        instructed to describe only what is visible
            │
            ├─ alignment score       ← text–video similarity
            ├─ hallucination checks  ← object/attribute spot checks on a sample
            └─ store: caption_dense, caption_short, models, prompt version, scores

    Practical decisions:

    • Treat context as a hint, not a source. Instruct the merging model that transcripts and titles may be unrelated to the visuals. Otherwise "in this video I'll review the new phone" becomes "a person reviews a phone" on a clip of a parking lot.
    • Produce at least two lengths. A dense caption for detail and a short caption that looks like a real user prompt. Train on a mix, or plan to expand prompts at inference as the Sora report describes.
    • Constrain vocabulary for camera motion. A fixed set of terms (static, pan left, tilt up, zoom in, tracking, handheld) is easier to learn from and easier to evaluate than free text.
    • Batch by length. Captioning cost scales with frames processed and tokens generated; sort clips by duration to keep GPU batches efficient.

    Measuring caption quality

    Without measurement, recaptioning can make data worse while looking better. Combine three signals:

    Check How What it catches
    Automatic alignment Text–video similarity per clip; track the distribution Captions unrelated to the clip
    Hallucination audit Human review of a random sample: is every stated object, count and action actually visible? Invented details, which are the most harmful errors
    Coverage audit Same sample: what visible, important elements were missed? Captions that are accurate but too thin
    Distribution checks Caption length, vocabulary, repeated phrases across the dataset Template collapse ("A video showing...") and length bias

    A few hundred hand-reviewed clips per captioner version is enough to compare two pipelines. Re-run the audit whenever you change the captioner model or prompt.

    Cost planning

    Captioning is usually the most GPU-hungry step of dataset preparation. Estimate it before you commit:

    GPU-hours ≈ clips × (frames_per_clip × time_per_frame + tokens_out × time_per_token) / batch_efficiency

    Measure the per-frame and per-token times on your actual hardware with a few thousand clips, then extrapolate. The two levers that matter most are how many frames you feed the video model and how long the dense caption is. Curating first (dropping clips you won't train on) is the cheapest optimization of all.

    Store captions like data, not like a column

    {
      "clip_id": "dQw4w9WgXcQ_000123",
      "captions": [
        {
          "type": "dense",
          "text": "A man in a beige trench coat walks toward the camera along a rain-wet city street at night; neon signs reflect on the pavement; handheld tracking shot, slight push-in.",
          "model": "captioner-v3",
          "prompt_version": "dense-2026.11",
          "alignment": 0.33
        },
        {
          "type": "short",
          "text": "Man in a trench coat walking down a rainy neon street at night",
          "model": "rewriter-v2",
          "prompt_version": "short-2026.11",
          "alignment": 0.31
        },
        {
          "type": "asr",
          "text": "...and that's why I always carry an umbrella...",
          "model": "asr",
          "prompt_version": null,
          "alignment": 0.12
        }
      ]
    }

    Captioner models improve every few months. Keeping every version with its model and prompt lets you recaption selectively, compare versions, and trace a model behavior back to its training text. For the surrounding pipeline see building a YouTube video dataset for multimodal training.

    Compliance

    Recaptioning changes the text, not the rights to the underlying video. YouTube's Terms of Service restrict automated access and downloading, copyright and licensing apply to the source videos, and captions that name or describe identifiable people can carry privacy implications. Keep provenance per clip and get legal review for your use case.

    If you'd rather start from clean source data

    Corpora delivers YouTube video, audio, subtitles and structured metadata at petabyte scale — deduplicated, with checksums and manifests — straight to your cloud bucket, so your team can spend its GPU budget on captioning and training rather than collection. A free sample batch is included. See pricing or talk to sales.

    FAQ

    Why not just use YouTube titles and descriptions as captions?

    They describe what a video is about, not what each clip shows, and after splitting one title would have to cover dozens of different shots. They are useful context for a captioner, not training captions.

    Are auto-generated subtitles good enough as captions?

    For speech models they are useful labels; for video generation they mostly describe speech, not visuals. Use them as optional context when generating visual captions.

    Should training captions be long or short?

    Both. Dense captions teach detail, short ones match how users actually prompt. Train on a mix, or expand short prompts at inference time.

    How do I know if recaptioning helped?

    Audit hallucinations and coverage on a sample, track alignment scores, and ultimately compare small models trained on old versus new captions on the same evaluation prompts.

  • Video Data Curation for Generative Models: Filters That Matter

    Raw video scraped at scale is mostly unusable for training a generative model as-is: hard cuts in the middle of clips, static slides, burned-in subtitles, shaky phone footage, letterboxing, near-duplicates. Published video model papers spend as much effort on curation as on architecture. This guide lays out the filters that matter, the order to run them in so you don't burn GPU hours on clips you'll throw away, and how to store the results so you can change your mind later.

    TL;DR

    • Split before you filter. Almost every downstream score is meaningless on a clip that contains a hard cut.
    • Run filters in cost order: metadata → shot detection → cheap frame statistics → optical flow → OCR → model-based aesthetic and alignment scores → near-duplicate search.
    • Store scores, not booleans. Thresholds change with every training run; recomputing scores over millions of clips does not have to.
    • Watch your keep-rate per filter. Over-aggressive aesthetic filters quietly turn a diverse dataset into stock footage.
    • Curation is an experiment, not a preprocessing step. Validate thresholds with small training runs, not by eyeballing samples.

    Why curation matters as much as scale

    Two widely cited papers make the case well. The Stable Video Diffusion paper (arXiv 2311.15127) describes a systematic curation pipeline for its large video pretraining set: cut detection, optical-flow-based motion filtering, text detection to drop clips dominated by on-screen text, and CLIP-based scoring for aesthetics and text-image similarity. The authors report that models trained on the curated subset were preferred over models trained on uncurated data. Panda-70M (arXiv 2402.19479) took long YouTube videos, split them into semantically coherent clips, and then invested heavily in captioning those clips.

    The common thread: a smaller, cleaner set of well-segmented clips beats a larger pile of raw footage. If you are building your own set, start from the papers' filter list and calibrate it to your target model.

    The filter stack, in cost order

    Stage Signal Relative cost Catches
    Metadata Duration, resolution, frame rate, aspect ratio, license, language Free (before download) Shorts you don't want, 360p uploads, wrong language
    Shot detection Frame-difference / content-change detection Low (CPU) Hard cuts, fades, montage edits
    Frame statistics Brightness, contrast, blur (Laplacian variance), black-border detection Low (CPU, sampled frames) Dark, blurry, letterboxed or pillarboxed clips
    Motion Optical flow magnitude on sampled frames Medium (GPU or fast CPU flow) Static slides and talking-head stills, or extreme camera shake
    Text / OCR Text detection area ratio Medium Burned-in subtitles, slides, screen recordings, heavy watermarks
    Aesthetic / quality Learned aesthetic or quality predictor on frame embeddings Medium–high (GPU) Low production value, compression artifacts
    Text–video alignment Similarity between caption and frames (CLIP-style) High (needs captions first) Captions that don't describe the clip
    Near-duplicates Perceptual hashes, embedding nearest-neighbor search Medium–high at scale Re-uploads, compilations, reaction videos, your eval set
    Safety / privacy NSFW classifiers, face detection, PII in text Medium Content you can't train on or must handle separately

    The ordering principle is simple: every stage should only see clips that passed all cheaper stages. A metadata filter that drops 30% of candidates before download saves 30% of bandwidth, storage and every GPU hour downstream.

    Stage by stage

    1. Metadata, before you download anything

    Filter on what the platform tells you for free: duration, available resolutions, aspect ratio, upload date, category and license. Decide early what you are training on. A text-to-video model rarely benefits from 12-hour livestreams or vertical 9:16 Shorts unless you specifically want them. See building a YouTube video dataset for multimodal training for the metadata-first workflow, and filtering for Creative Commons videos if licensing drives your selection.

    2. Shot detection and clip splitting

    Run this first on downloaded video, because every later score assumes a clip shows one continuous shot. Content-based detectors (for example PySceneDetect's content and adaptive detectors) handle hard cuts well; fades and dissolves need a threshold-based or learned detector. SVD reports using cut detection at multiple frame rates to catch transitions that a single pass misses.

    Splitting purely on cuts can over-fragment: an interview that alternates between two cameras becomes dozens of 2-second clips. Panda-70M addressed this by stitching adjacent shots back together when they were semantically similar. Decide what a "clip" means for your model — a single shot, or a coherent scene — and set minimum and maximum clip lengths accordingly.

    3. Cheap frame statistics

    Sample a few frames per clip and compute brightness, contrast, a blur score and black-border ratio. These are fast, explainable and catch a surprising share of junk: dark concert footage, out-of-focus phone video, 4:3 content pillarboxed inside 16:9. Crop borders rather than discarding clips where the content itself is fine.

    4. Motion

    Optical flow on a low-frame-rate sample (SVD used 2 FPS) gives you a motion score per clip. The bottom of the distribution is slides, still images with music and near-static talking heads — useless for learning motion. The top of the distribution is often camera shake or rapid editing. Most teams drop both tails, and keep the score so they can rebalance motion levels later.

    5. Text and OCR

    Text detection gives you the fraction of the frame covered by text. High values mean slides, tutorials, screen recordings or burned-in subtitles. A generative model trained on those learns to produce garbled text overlays. Small, persistent text in a corner is often a channel watermark — decide whether to drop or crop.

    6. Aesthetic and quality scores

    Learned aesthetic predictors on CLIP-style embeddings are cheap per frame once embeddings exist, and they correlate with production quality. They also carry taste bias: they reward well-lit, centered, stock-footage-looking frames. Use them to drop the bottom of the distribution, not to keep only the top, unless your target is explicitly high-aesthetic output.

    7. Caption alignment

    Once clips have captions, a text–video similarity score tells you when a caption doesn't describe its clip. This matters most when captions come from titles, descriptions or ASR transcripts, which are frequently about the video rather than what is on screen. Better captions usually beat stricter alignment filtering — see recaptioning video datasets.

    8. Near-duplicates and contamination

    Large web video collections are full of re-uploads, compilations and reaction videos that embed other videos. Two layers work well together: perceptual hashes on keyframes for exact and near-exact copies, and embedding nearest-neighbor search for semantic duplicates. Run the same search against your evaluation sets. Benchmark contamination is easy to introduce and hard to detect after training.

    9. Safety and privacy

    NSFW classifiers, face detection and text PII checks belong in the pipeline, not in a later audit. Faces and voices can be personal data under GDPR and similar laws, so record what you detected even if you keep the clip.

    Store scores, not decisions

    The most expensive mistake is writing only the filtered output. Thresholds change with every model, and recomputing optical flow or embeddings over tens of millions of clips is days of GPU time. Keep one row per clip with every score:

    {
      "clip_id": "dQw4w9WgXcQ_000123",
      "video_id": "dQw4w9WgXcQ",
      "start_s": 61.2,
      "end_s": 68.9,
      "width": 1920,
      "height": 1080,
      "fps": 25.0,
      "shot_boundary_confidence": 0.94,
      "brightness_mean": 0.41,
      "blur_laplacian_var": 212.5,
      "black_border_ratio": 0.0,
      "flow_magnitude_p50": 1.83,
      "ocr_area_ratio": 0.012,
      "aesthetic_score": 5.2,
      "nsfw_score": 0.01,
      "face_count_max": 1,
      "phash_cluster": "c_88412",
      "embedding_dup_of": null,
      "caption_alignment": 0.31,
      "pipeline_version": "curate-2026.11"
    }

    Your training set then becomes a query over this table. A new threshold is a new query, not a new pipeline run.

    Calibrating thresholds

    • Report keep-rate per filter. If one filter removes 60% of clips, look at what it removed before trusting it.
    • Look at the boundary, not the extremes. Sample clips just above and just below each threshold; that is where mistakes live.
    • Check distribution shift. Compare category, language, motion and resolution distributions before and after filtering. Curation that silently removes most sports or most outdoor footage changes what your model learns.
    • Validate with training runs. The real test is a small model trained on two subsets with different thresholds, evaluated on the same prompts.

    Compliance

    Curation decides what goes into the model, so it is also where compliance decisions get enforced. YouTube's Terms of Service restrict automated access and downloading, copyright and licensing apply to the underlying videos, and faces and voices can be personal data. Keep per-clip provenance (source ID, collection time, license at collection time), honor takedown and opt-out requests across derived clips, and get legal review for your jurisdiction and use case.

    If you'd rather not build this yourself

    Corpora collects YouTube video, audio, subtitles and structured metadata at petabyte scale and delivers deduplicated datasets with checksums and manifests to your S3, GCS or Azure bucket, with a free sample batch to validate quality first. See pricing or talk to sales.

    FAQ

    What is the first filter to run on raw video?

    Metadata filtering before download, then shot detection right after. Every other score assumes the clip is a single continuous shot.

    How much data do curation filters typically remove?

    It depends heavily on the source and your thresholds, so measure it on a sample. Web video often loses a large share to cuts, static content, text overlays and duplicates, which is why filtering before download is worth so much.

    Should I filter on aesthetic score?

    Use it to drop the worst clips. Keeping only the top of the distribution biases the dataset toward stock-footage style and reduces diversity.

    How do I avoid training on my evaluation set?

    Run near-duplicate search (perceptual hashes plus embedding similarity) of your training clips against every evaluation set, and record matches as a score so you can exclude them.