{"id":24,"date":"2026-11-05T13:23:53","date_gmt":"2026-11-05T13:23:53","guid":{"rendered":"https:\/\/globalcorpora.com\/blog\/recaptioning-video-datasets-caption-quality\/"},"modified":"2026-11-05T13:23:53","modified_gmt":"2026-11-05T13:23:53","slug":"recaptioning-video-datasets-caption-quality","status":"publish","type":"post","link":"https:\/\/globalcorpora.com\/blog\/recaptioning-video-datasets-caption-quality\/","title":{"rendered":"Recaptioning Video Datasets: Why Caption Quality Matters"},"content":{"rendered":"<p>For a text-to-video or video-language model, the caption is half of every training example. Yet the text that comes with web video \u2014 titles, descriptions, tags, ASR transcripts \u2014 mostly describes what the video is <em>about<\/em>, not what is on screen. Recaptioning replaces or supplements that text with descriptions generated for training. This article covers why it works, how the published pipelines do it, and how to measure whether your captions are actually good.<\/p>\n<h2>TL;DR<\/h2>\n<ul>\n<li><strong>Native text is the wrong supervision.<\/strong> Titles are clickbait, descriptions are links, ASR is speech \u2014 none of it reliably describes the pixels.<\/li>\n<li><strong>Major labs recaption.<\/strong> OpenAI's DALL\u00b7E 3 paper and Sora technical report, Stable Video Diffusion and Panda-70M all describe generating their own captions.<\/li>\n<li><strong>Dense captions help, but mix lengths.<\/strong> Users type short prompts; train on both long and short captions, or expand prompts at inference.<\/li>\n<li><strong>Hallucination is the main failure.<\/strong> Audit a sample by hand and score alignment automatically.<\/li>\n<li><strong>Keep every caption with its provenance.<\/strong> Captioner models improve; you will recaption again.<\/li>\n<\/ul>\n<h2>Why native text fails as training captions<\/h2>\n<table>\n<thead>\n<tr>\n<th>Source<\/th>\n<th>Typical example<\/th>\n<th>Problem<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Title<\/td>\n<td>\"You won't BELIEVE what happened next!!\"<\/td>\n<td>Describes intent or emotion, not content<\/td>\n<\/tr>\n<tr>\n<td>Description<\/td>\n<td>Links, sponsor reads, timestamps, hashtags<\/td>\n<td>Mostly not about the visuals at all<\/td>\n<\/tr>\n<tr>\n<td>Tags<\/td>\n<td>Keyword lists for search ranking<\/td>\n<td>Unordered, often stuffed<\/td>\n<\/tr>\n<tr>\n<td>ASR transcript<\/td>\n<td>What the speaker says<\/td>\n<td>Misaligned with what is shown; empty for silent or music-only clips<\/td>\n<\/tr>\n<tr>\n<td>Per-clip after splitting<\/td>\n<td>Same title for 50 clips<\/td>\n<td>One caption cannot describe 50 different shots<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>The last row matters most at scale. Once you split videos into short clips (see <a href=\"https:\/\/globalcorpora.com\/blog\/video-data-curation-filters-for-generative-models\/\">video data curation for generative models<\/a>), video-level text cannot describe individual clips no matter how good it is.<\/p>\n<h2>What the published pipelines do<\/h2>\n<ul>\n<li><strong>DALL\u00b7E 3<\/strong> (OpenAI, \"Improving Image Generation with Better Captions\"): trained a dedicated captioner to produce descriptive synthetic captions and trained the image model mostly on those, reporting better prompt following than with original alt-text.<\/li>\n<li><strong>Sora<\/strong> (OpenAI technical report, \"Video generation models as world simulators\"): applied the same re-captioning technique to video, and used a language model to expand short user prompts into longer detailed captions at generation time, matching the training distribution.<\/li>\n<li><strong>Stable Video Diffusion<\/strong> (<a href=\"https:\/\/arxiv.org\/abs\/2311.15127\">arXiv 2311.15127<\/a>): generated multiple synthetic captions per clip with an image captioner on keyframes and a video captioner, and combined them with a language model.<\/li>\n<li><strong>Panda-70M<\/strong> (<a href=\"https:\/\/arxiv.org\/abs\/2402.19479\">arXiv 2402.19479<\/a>): captioned each clip with several cross-modality teacher models (using inputs such as frames, subtitles and metadata), then used a fine-tuned retrieval model to pick the caption that best matched the clip.<\/li>\n<li><strong>InternVid<\/strong> (<a href=\"https:\/\/arxiv.org\/abs\/2307.06942\">arXiv 2307.06942<\/a>): generated captions at multiple scales for its clips using a pipeline built on image captioning and language models.<\/li>\n<\/ul>\n<p>The pattern is consistent: generate candidate descriptions from the visuals, use other signals (transcripts, metadata) as context rather than as the caption, and select or merge the best result.<\/p>\n<h2>What a good training caption contains<\/h2>\n<p>For video generation and video understanding, a useful dense caption usually covers, in roughly this order:<\/p>\n<ol>\n<li><strong>Subjects<\/strong>: who or what, with visible attributes (count, clothing, color, size).<\/li>\n<li><strong>Action<\/strong>: what changes over the clip, in order.<\/li>\n<li><strong>Scene<\/strong>: setting, time of day, weather, indoor or outdoor.<\/li>\n<li><strong>Camera<\/strong>: shot type (close-up, wide), movement (pan, tilt, tracking, static, handheld).<\/li>\n<li><strong>Style<\/strong>: live action, animation, screen recording, lighting, color grade.<\/li>\n<li><strong>On-screen text<\/strong>, if any, quoted exactly or flagged.<\/li>\n<\/ol>\n<p>Camera motion and temporal order are where image captioners applied frame by frame fall short. A captioner that only sees one keyframe cannot tell a slow zoom from a static shot. That is why pipelines combine per-frame captions with a model that sees multiple frames, or a video-native captioner.<\/p>\n<h2>A practical recaptioning pipeline<\/h2>\n<pre><code>clip (single shot, 2\u201320 s)\n  \u2502\n  \u251c\u2500 sample frames (e.g. 4\u201316, evenly spaced)\n  \u251c\u2500 per-frame captions          \u2190 image captioner\n  \u251c\u2500 clip-level caption          \u2190 video-language model on the frame sequence\n  \u251c\u2500 context (optional)          \u2190 ASR snippet for this time range, title, OCR text\n  \u2502\n  \u2514\u2500 merge \/ rewrite             \u2190 LLM: produce dense + short caption,\n                                    instructed to describe only what is visible\n        \u2502\n        \u251c\u2500 alignment score       \u2190 text\u2013video similarity\n        \u251c\u2500 hallucination checks  \u2190 object\/attribute spot checks on a sample\n        \u2514\u2500 store: caption_dense, caption_short, models, prompt version, scores<\/code><\/pre>\n<p>Practical decisions:<\/p>\n<ul>\n<li><strong>Treat context as a hint, not a source.<\/strong> Instruct the merging model that transcripts and titles may be unrelated to the visuals. Otherwise \"in this video I'll review the new phone\" becomes \"a person reviews a phone\" on a clip of a parking lot.<\/li>\n<li><strong>Produce at least two lengths.<\/strong> A dense caption for detail and a short caption that looks like a real user prompt. Train on a mix, or plan to expand prompts at inference as the Sora report describes.<\/li>\n<li><strong>Constrain vocabulary for camera motion.<\/strong> A fixed set of terms (static, pan left, tilt up, zoom in, tracking, handheld) is easier to learn from and easier to evaluate than free text.<\/li>\n<li><strong>Batch by length.<\/strong> Captioning cost scales with frames processed and tokens generated; sort clips by duration to keep GPU batches efficient.<\/li>\n<\/ul>\n<h2>Measuring caption quality<\/h2>\n<p>Without measurement, recaptioning can make data worse while looking better. Combine three signals:<\/p>\n<table>\n<thead>\n<tr>\n<th>Check<\/th>\n<th>How<\/th>\n<th>What it catches<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Automatic alignment<\/td>\n<td>Text\u2013video similarity per clip; track the distribution<\/td>\n<td>Captions unrelated to the clip<\/td>\n<\/tr>\n<tr>\n<td>Hallucination audit<\/td>\n<td>Human review of a random sample: is every stated object, count and action actually visible?<\/td>\n<td>Invented details, which are the most harmful errors<\/td>\n<\/tr>\n<tr>\n<td>Coverage audit<\/td>\n<td>Same sample: what visible, important elements were missed?<\/td>\n<td>Captions that are accurate but too thin<\/td>\n<\/tr>\n<tr>\n<td>Distribution checks<\/td>\n<td>Caption length, vocabulary, repeated phrases across the dataset<\/td>\n<td>Template collapse (\"A video showing...\") and length bias<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>A few hundred hand-reviewed clips per captioner version is enough to compare two pipelines. Re-run the audit whenever you change the captioner model or prompt.<\/p>\n<h2>Cost planning<\/h2>\n<p>Captioning is usually the most GPU-hungry step of dataset preparation. Estimate it before you commit:<\/p>\n<pre><code>GPU-hours \u2248 clips \u00d7 (frames_per_clip \u00d7 time_per_frame + tokens_out \u00d7 time_per_token) \/ batch_efficiency<\/code><\/pre>\n<p>Measure the per-frame and per-token times on your actual hardware with a few thousand clips, then extrapolate. The two levers that matter most are how many frames you feed the video model and how long the dense caption is. Curating first (dropping clips you won't train on) is the cheapest optimization of all.<\/p>\n<h2>Store captions like data, not like a column<\/h2>\n<pre><code>{\n  \"clip_id\": \"dQw4w9WgXcQ_000123\",\n  \"captions\": [\n    {\n      \"type\": \"dense\",\n      \"text\": \"A man in a beige trench coat walks toward the camera along a rain-wet city street at night; neon signs reflect on the pavement; handheld tracking shot, slight push-in.\",\n      \"model\": \"captioner-v3\",\n      \"prompt_version\": \"dense-2026.11\",\n      \"alignment\": 0.33\n    },\n    {\n      \"type\": \"short\",\n      \"text\": \"Man in a trench coat walking down a rainy neon street at night\",\n      \"model\": \"rewriter-v2\",\n      \"prompt_version\": \"short-2026.11\",\n      \"alignment\": 0.31\n    },\n    {\n      \"type\": \"asr\",\n      \"text\": \"...and that's why I always carry an umbrella...\",\n      \"model\": \"asr\",\n      \"prompt_version\": null,\n      \"alignment\": 0.12\n    }\n  ]\n}<\/code><\/pre>\n<p>Captioner models improve every few months. Keeping every version with its model and prompt lets you recaption selectively, compare versions, and trace a model behavior back to its training text. For the surrounding pipeline see <a href=\"https:\/\/globalcorpora.com\/blog\/build-youtube-video-dataset-for-multimodal-training\/\">building a YouTube video dataset for multimodal training<\/a>.<\/p>\n<h2>Compliance<\/h2>\n<p>Recaptioning changes the text, not the rights to the underlying video. YouTube's Terms of Service restrict automated access and downloading, copyright and licensing apply to the source videos, and captions that name or describe identifiable people can carry privacy implications. Keep provenance per clip and get legal review for your use case.<\/p>\n<h2>If you'd rather start from clean source data<\/h2>\n<p>Corpora delivers YouTube video, audio, subtitles and structured metadata at petabyte scale \u2014 deduplicated, with checksums and manifests \u2014 straight to your cloud bucket, so your team can spend its GPU budget on captioning and training rather than collection. A free sample batch is included. See <a href=\"https:\/\/globalcorpora.com\/#pricing\">pricing<\/a> or <a href=\"https:\/\/globalcorpora.com\/#contact\">talk to sales<\/a>.<\/p>\n<h2>FAQ<\/h2>\n<h3>Why not just use YouTube titles and descriptions as captions?<\/h3>\n<p>They describe what a video is about, not what each clip shows, and after splitting one title would have to cover dozens of different shots. They are useful context for a captioner, not training captions.<\/p>\n<h3>Are auto-generated subtitles good enough as captions?<\/h3>\n<p>For speech models they are useful labels; for video generation they mostly describe speech, not visuals. Use them as optional context when generating visual captions.<\/p>\n<h3>Should training captions be long or short?<\/h3>\n<p>Both. Dense captions teach detail, short ones match how users actually prompt. Train on a mix, or expand short prompts at inference time.<\/p>\n<h3>How do I know if recaptioning helped?<\/h3>\n<p>Audit hallucinations and coverage on a sample, track alignment scores, and ultimately compare small models trained on old versus new captions on the same evaluation prompts.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Recaptioning video datasets: why titles and ASR make poor training captions, how dense synthetic captions are built, and how to measure caption quality.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[50,52,53,48,51],"class_list":["post-24","post","type-post","status-publish","format-standard","hentry","category-datasets","tag-recaptioning","tag-synthetic-captions","tag-text-to-video","tag-training-data-quality","tag-video-captioning"],"_links":{"self":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/posts\/24","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/comments?post=24"}],"version-history":[{"count":0,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/posts\/24\/revisions"}],"wp:attachment":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/media?parent=24"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/categories?post=24"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/tags?post=24"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}