{"id":23,"date":"2026-11-05T13:23:28","date_gmt":"2026-11-05T13:23:28","guid":{"rendered":"https:\/\/globalcorpora.com\/blog\/video-data-curation-filters-for-generative-models\/"},"modified":"2026-11-05T13:23:28","modified_gmt":"2026-11-05T13:23:28","slug":"video-data-curation-filters-for-generative-models","status":"publish","type":"post","link":"https:\/\/globalcorpora.com\/blog\/video-data-curation-filters-for-generative-models\/","title":{"rendered":"Video Data Curation for Generative Models: Filters That Matter"},"content":{"rendered":"<p>Raw video scraped at scale is mostly unusable for training a generative model as-is: hard cuts in the middle of clips, static slides, burned-in subtitles, shaky phone footage, letterboxing, near-duplicates. Published video model papers spend as much effort on curation as on architecture. This guide lays out the filters that matter, the order to run them in so you don't burn GPU hours on clips you'll throw away, and how to store the results so you can change your mind later.<\/p>\n<h2>TL;DR<\/h2>\n<ul>\n<li><strong>Split before you filter.<\/strong> Almost every downstream score is meaningless on a clip that contains a hard cut.<\/li>\n<li><strong>Run filters in cost order:<\/strong> metadata \u2192 shot detection \u2192 cheap frame statistics \u2192 optical flow \u2192 OCR \u2192 model-based aesthetic and alignment scores \u2192 near-duplicate search.<\/li>\n<li><strong>Store scores, not booleans.<\/strong> Thresholds change with every training run; recomputing scores over millions of clips does not have to.<\/li>\n<li><strong>Watch your keep-rate per filter.<\/strong> Over-aggressive aesthetic filters quietly turn a diverse dataset into stock footage.<\/li>\n<li><strong>Curation is an experiment, not a preprocessing step.<\/strong> Validate thresholds with small training runs, not by eyeballing samples.<\/li>\n<\/ul>\n<h2>Why curation matters as much as scale<\/h2>\n<p>Two widely cited papers make the case well. The Stable Video Diffusion paper (<a href=\"https:\/\/arxiv.org\/abs\/2311.15127\">arXiv 2311.15127<\/a>) describes a systematic curation pipeline for its large video pretraining set: cut detection, optical-flow-based motion filtering, text detection to drop clips dominated by on-screen text, and CLIP-based scoring for aesthetics and text-image similarity. The authors report that models trained on the curated subset were preferred over models trained on uncurated data. Panda-70M (<a href=\"https:\/\/arxiv.org\/abs\/2402.19479\">arXiv 2402.19479<\/a>) took long YouTube videos, split them into semantically coherent clips, and then invested heavily in captioning those clips.<\/p>\n<p>The common thread: a smaller, cleaner set of well-segmented clips beats a larger pile of raw footage. If you are building your own set, start from the papers' filter list and calibrate it to your target model.<\/p>\n<h2>The filter stack, in cost order<\/h2>\n<table>\n<thead>\n<tr>\n<th>Stage<\/th>\n<th>Signal<\/th>\n<th>Relative cost<\/th>\n<th>Catches<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Metadata<\/td>\n<td>Duration, resolution, frame rate, aspect ratio, license, language<\/td>\n<td>Free (before download)<\/td>\n<td>Shorts you don't want, 360p uploads, wrong language<\/td>\n<\/tr>\n<tr>\n<td>Shot detection<\/td>\n<td>Frame-difference \/ content-change detection<\/td>\n<td>Low (CPU)<\/td>\n<td>Hard cuts, fades, montage edits<\/td>\n<\/tr>\n<tr>\n<td>Frame statistics<\/td>\n<td>Brightness, contrast, blur (Laplacian variance), black-border detection<\/td>\n<td>Low (CPU, sampled frames)<\/td>\n<td>Dark, blurry, letterboxed or pillarboxed clips<\/td>\n<\/tr>\n<tr>\n<td>Motion<\/td>\n<td>Optical flow magnitude on sampled frames<\/td>\n<td>Medium (GPU or fast CPU flow)<\/td>\n<td>Static slides and talking-head stills, or extreme camera shake<\/td>\n<\/tr>\n<tr>\n<td>Text \/ OCR<\/td>\n<td>Text detection area ratio<\/td>\n<td>Medium<\/td>\n<td>Burned-in subtitles, slides, screen recordings, heavy watermarks<\/td>\n<\/tr>\n<tr>\n<td>Aesthetic \/ quality<\/td>\n<td>Learned aesthetic or quality predictor on frame embeddings<\/td>\n<td>Medium\u2013high (GPU)<\/td>\n<td>Low production value, compression artifacts<\/td>\n<\/tr>\n<tr>\n<td>Text\u2013video alignment<\/td>\n<td>Similarity between caption and frames (CLIP-style)<\/td>\n<td>High (needs captions first)<\/td>\n<td>Captions that don't describe the clip<\/td>\n<\/tr>\n<tr>\n<td>Near-duplicates<\/td>\n<td>Perceptual hashes, embedding nearest-neighbor search<\/td>\n<td>Medium\u2013high at scale<\/td>\n<td>Re-uploads, compilations, reaction videos, your eval set<\/td>\n<\/tr>\n<tr>\n<td>Safety \/ privacy<\/td>\n<td>NSFW classifiers, face detection, PII in text<\/td>\n<td>Medium<\/td>\n<td>Content you can't train on or must handle separately<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>The ordering principle is simple: every stage should only see clips that passed all cheaper stages. A metadata filter that drops 30% of candidates before download saves 30% of bandwidth, storage and every GPU hour downstream.<\/p>\n<h2>Stage by stage<\/h2>\n<h3>1. Metadata, before you download anything<\/h3>\n<p>Filter on what the platform tells you for free: duration, available resolutions, aspect ratio, upload date, category and license. Decide early what you are training on. A text-to-video model rarely benefits from 12-hour livestreams or vertical 9:16 Shorts unless you specifically want them. See <a href=\"https:\/\/globalcorpora.com\/blog\/build-youtube-video-dataset-for-multimodal-training\/\">building a YouTube video dataset for multimodal training<\/a> for the metadata-first workflow, and <a href=\"https:\/\/globalcorpora.com\/blog\/creative-commons-youtube-videos-for-ai-training\/\">filtering for Creative Commons videos<\/a> if licensing drives your selection.<\/p>\n<h3>2. Shot detection and clip splitting<\/h3>\n<p>Run this first on downloaded video, because every later score assumes a clip shows one continuous shot. Content-based detectors (for example PySceneDetect's content and adaptive detectors) handle hard cuts well; fades and dissolves need a threshold-based or learned detector. SVD reports using cut detection at multiple frame rates to catch transitions that a single pass misses.<\/p>\n<p>Splitting purely on cuts can over-fragment: an interview that alternates between two cameras becomes dozens of 2-second clips. Panda-70M addressed this by stitching adjacent shots back together when they were semantically similar. Decide what a \"clip\" means for your model \u2014 a single shot, or a coherent scene \u2014 and set minimum and maximum clip lengths accordingly.<\/p>\n<h3>3. Cheap frame statistics<\/h3>\n<p>Sample a few frames per clip and compute brightness, contrast, a blur score and black-border ratio. These are fast, explainable and catch a surprising share of junk: dark concert footage, out-of-focus phone video, 4:3 content pillarboxed inside 16:9. Crop borders rather than discarding clips where the content itself is fine.<\/p>\n<h3>4. Motion<\/h3>\n<p>Optical flow on a low-frame-rate sample (SVD used 2 FPS) gives you a motion score per clip. The bottom of the distribution is slides, still images with music and near-static talking heads \u2014 useless for learning motion. The top of the distribution is often camera shake or rapid editing. Most teams drop both tails, and keep the score so they can rebalance motion levels later.<\/p>\n<h3>5. Text and OCR<\/h3>\n<p>Text detection gives you the fraction of the frame covered by text. High values mean slides, tutorials, screen recordings or burned-in subtitles. A generative model trained on those learns to produce garbled text overlays. Small, persistent text in a corner is often a channel watermark \u2014 decide whether to drop or crop.<\/p>\n<h3>6. Aesthetic and quality scores<\/h3>\n<p>Learned aesthetic predictors on CLIP-style embeddings are cheap per frame once embeddings exist, and they correlate with production quality. They also carry taste bias: they reward well-lit, centered, stock-footage-looking frames. Use them to drop the bottom of the distribution, not to keep only the top, unless your target is explicitly high-aesthetic output.<\/p>\n<h3>7. Caption alignment<\/h3>\n<p>Once clips have captions, a text\u2013video similarity score tells you when a caption doesn't describe its clip. This matters most when captions come from titles, descriptions or ASR transcripts, which are frequently about the video rather than what is on screen. Better captions usually beat stricter alignment filtering \u2014 see <a href=\"https:\/\/globalcorpora.com\/blog\/recaptioning-video-datasets-caption-quality\/\">recaptioning video datasets<\/a>.<\/p>\n<h3>8. Near-duplicates and contamination<\/h3>\n<p>Large web video collections are full of re-uploads, compilations and reaction videos that embed other videos. Two layers work well together: perceptual hashes on keyframes for exact and near-exact copies, and embedding nearest-neighbor search for semantic duplicates. Run the same search against your evaluation sets. Benchmark contamination is easy to introduce and hard to detect after training.<\/p>\n<h3>9. Safety and privacy<\/h3>\n<p>NSFW classifiers, face detection and text PII checks belong in the pipeline, not in a later audit. Faces and voices can be personal data under GDPR and similar laws, so record what you detected even if you keep the clip.<\/p>\n<h2>Store scores, not decisions<\/h2>\n<p>The most expensive mistake is writing only the filtered output. Thresholds change with every model, and recomputing optical flow or embeddings over tens of millions of clips is days of GPU time. Keep one row per clip with every score:<\/p>\n<pre><code>{\n  \"clip_id\": \"dQw4w9WgXcQ_000123\",\n  \"video_id\": \"dQw4w9WgXcQ\",\n  \"start_s\": 61.2,\n  \"end_s\": 68.9,\n  \"width\": 1920,\n  \"height\": 1080,\n  \"fps\": 25.0,\n  \"shot_boundary_confidence\": 0.94,\n  \"brightness_mean\": 0.41,\n  \"blur_laplacian_var\": 212.5,\n  \"black_border_ratio\": 0.0,\n  \"flow_magnitude_p50\": 1.83,\n  \"ocr_area_ratio\": 0.012,\n  \"aesthetic_score\": 5.2,\n  \"nsfw_score\": 0.01,\n  \"face_count_max\": 1,\n  \"phash_cluster\": \"c_88412\",\n  \"embedding_dup_of\": null,\n  \"caption_alignment\": 0.31,\n  \"pipeline_version\": \"curate-2026.11\"\n}<\/code><\/pre>\n<p>Your training set then becomes a query over this table. A new threshold is a new query, not a new pipeline run.<\/p>\n<h2>Calibrating thresholds<\/h2>\n<ul>\n<li><strong>Report keep-rate per filter.<\/strong> If one filter removes 60% of clips, look at what it removed before trusting it.<\/li>\n<li><strong>Look at the boundary, not the extremes.<\/strong> Sample clips just above and just below each threshold; that is where mistakes live.<\/li>\n<li><strong>Check distribution shift.<\/strong> Compare category, language, motion and resolution distributions before and after filtering. Curation that silently removes most sports or most outdoor footage changes what your model learns.<\/li>\n<li><strong>Validate with training runs.<\/strong> The real test is a small model trained on two subsets with different thresholds, evaluated on the same prompts.<\/li>\n<\/ul>\n<h2>Compliance<\/h2>\n<p>Curation decides what goes into the model, so it is also where compliance decisions get enforced. YouTube's Terms of Service restrict automated access and downloading, copyright and licensing apply to the underlying videos, and faces and voices can be personal data. Keep per-clip provenance (source ID, collection time, license at collection time), honor takedown and opt-out requests across derived clips, and get legal review for your jurisdiction and use case.<\/p>\n<h2>If you'd rather not build this yourself<\/h2>\n<p>Corpora collects YouTube video, audio, subtitles and structured metadata at petabyte scale and delivers deduplicated datasets with checksums and manifests to your S3, GCS or Azure bucket, with a free sample batch to validate quality first. See <a href=\"https:\/\/globalcorpora.com\/#pricing\">pricing<\/a> or <a href=\"https:\/\/globalcorpora.com\/#contact\">talk to sales<\/a>.<\/p>\n<h2>FAQ<\/h2>\n<h3>What is the first filter to run on raw video?<\/h3>\n<p>Metadata filtering before download, then shot detection right after. Every other score assumes the clip is a single continuous shot.<\/p>\n<h3>How much data do curation filters typically remove?<\/h3>\n<p>It depends heavily on the source and your thresholds, so measure it on a sample. Web video often loses a large share to cuts, static content, text overlays and duplicates, which is why filtering before download is worth so much.<\/p>\n<h3>Should I filter on aesthetic score?<\/h3>\n<p>Use it to drop the worst clips. Keeping only the top of the distribution biases the dataset toward stock-footage style and reduces diversity.<\/p>\n<h3>How do I avoid training on my evaluation set?<\/h3>\n<p>Run near-duplicate search (perceptual hashes plus embedding similarity) of your training clips against every evaluation set, and record matches as a score so you can exclude them.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Video data curation for generative models: shot splitting, motion, OCR, aesthetic, alignment and dedup filters, in cost order, with scores you can re-threshold.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[46,49,48,45,47],"class_list":["post-23","post","type-post","status-publish","format-standard","hentry","category-datasets","tag-data-filtering","tag-deduplication","tag-training-data-quality","tag-video-data-curation","tag-video-generation"],"_links":{"self":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/posts\/23","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/comments?post=23"}],"version-history":[{"count":0,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/posts\/23\/revisions"}],"wp:attachment":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/media?parent=23"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/categories?post=23"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/tags?post=23"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}