Tag: video captioning

  • Recaptioning Video Datasets: Why Caption Quality Matters

    For a text-to-video or video-language model, the caption is half of every training example. Yet the text that comes with web video — titles, descriptions, tags, ASR transcripts — mostly describes what the video is about, not what is on screen. Recaptioning replaces or supplements that text with descriptions generated for training. This article covers why it works, how the published pipelines do it, and how to measure whether your captions are actually good.

    TL;DR

    • Native text is the wrong supervision. Titles are clickbait, descriptions are links, ASR is speech — none of it reliably describes the pixels.
    • Major labs recaption. OpenAI's DALL·E 3 paper and Sora technical report, Stable Video Diffusion and Panda-70M all describe generating their own captions.
    • Dense captions help, but mix lengths. Users type short prompts; train on both long and short captions, or expand prompts at inference.
    • Hallucination is the main failure. Audit a sample by hand and score alignment automatically.
    • Keep every caption with its provenance. Captioner models improve; you will recaption again.

    Why native text fails as training captions

    Source Typical example Problem
    Title "You won't BELIEVE what happened next!!" Describes intent or emotion, not content
    Description Links, sponsor reads, timestamps, hashtags Mostly not about the visuals at all
    Tags Keyword lists for search ranking Unordered, often stuffed
    ASR transcript What the speaker says Misaligned with what is shown; empty for silent or music-only clips
    Per-clip after splitting Same title for 50 clips One caption cannot describe 50 different shots

    The last row matters most at scale. Once you split videos into short clips (see video data curation for generative models), video-level text cannot describe individual clips no matter how good it is.

    What the published pipelines do

    • DALL·E 3 (OpenAI, "Improving Image Generation with Better Captions"): trained a dedicated captioner to produce descriptive synthetic captions and trained the image model mostly on those, reporting better prompt following than with original alt-text.
    • Sora (OpenAI technical report, "Video generation models as world simulators"): applied the same re-captioning technique to video, and used a language model to expand short user prompts into longer detailed captions at generation time, matching the training distribution.
    • Stable Video Diffusion (arXiv 2311.15127): generated multiple synthetic captions per clip with an image captioner on keyframes and a video captioner, and combined them with a language model.
    • Panda-70M (arXiv 2402.19479): captioned each clip with several cross-modality teacher models (using inputs such as frames, subtitles and metadata), then used a fine-tuned retrieval model to pick the caption that best matched the clip.
    • InternVid (arXiv 2307.06942): generated captions at multiple scales for its clips using a pipeline built on image captioning and language models.

    The pattern is consistent: generate candidate descriptions from the visuals, use other signals (transcripts, metadata) as context rather than as the caption, and select or merge the best result.

    What a good training caption contains

    For video generation and video understanding, a useful dense caption usually covers, in roughly this order:

    1. Subjects: who or what, with visible attributes (count, clothing, color, size).
    2. Action: what changes over the clip, in order.
    3. Scene: setting, time of day, weather, indoor or outdoor.
    4. Camera: shot type (close-up, wide), movement (pan, tilt, tracking, static, handheld).
    5. Style: live action, animation, screen recording, lighting, color grade.
    6. On-screen text, if any, quoted exactly or flagged.

    Camera motion and temporal order are where image captioners applied frame by frame fall short. A captioner that only sees one keyframe cannot tell a slow zoom from a static shot. That is why pipelines combine per-frame captions with a model that sees multiple frames, or a video-native captioner.

    A practical recaptioning pipeline

    clip (single shot, 2–20 s)
      │
      ├─ sample frames (e.g. 4–16, evenly spaced)
      ├─ per-frame captions          ← image captioner
      ├─ clip-level caption          ← video-language model on the frame sequence
      ├─ context (optional)          ← ASR snippet for this time range, title, OCR text
      │
      └─ merge / rewrite             ← LLM: produce dense + short caption,
                                        instructed to describe only what is visible
            │
            ├─ alignment score       ← text–video similarity
            ├─ hallucination checks  ← object/attribute spot checks on a sample
            └─ store: caption_dense, caption_short, models, prompt version, scores

    Practical decisions:

    • Treat context as a hint, not a source. Instruct the merging model that transcripts and titles may be unrelated to the visuals. Otherwise "in this video I'll review the new phone" becomes "a person reviews a phone" on a clip of a parking lot.
    • Produce at least two lengths. A dense caption for detail and a short caption that looks like a real user prompt. Train on a mix, or plan to expand prompts at inference as the Sora report describes.
    • Constrain vocabulary for camera motion. A fixed set of terms (static, pan left, tilt up, zoom in, tracking, handheld) is easier to learn from and easier to evaluate than free text.
    • Batch by length. Captioning cost scales with frames processed and tokens generated; sort clips by duration to keep GPU batches efficient.

    Measuring caption quality

    Without measurement, recaptioning can make data worse while looking better. Combine three signals:

    Check How What it catches
    Automatic alignment Text–video similarity per clip; track the distribution Captions unrelated to the clip
    Hallucination audit Human review of a random sample: is every stated object, count and action actually visible? Invented details, which are the most harmful errors
    Coverage audit Same sample: what visible, important elements were missed? Captions that are accurate but too thin
    Distribution checks Caption length, vocabulary, repeated phrases across the dataset Template collapse ("A video showing...") and length bias

    A few hundred hand-reviewed clips per captioner version is enough to compare two pipelines. Re-run the audit whenever you change the captioner model or prompt.

    Cost planning

    Captioning is usually the most GPU-hungry step of dataset preparation. Estimate it before you commit:

    GPU-hours ≈ clips × (frames_per_clip × time_per_frame + tokens_out × time_per_token) / batch_efficiency

    Measure the per-frame and per-token times on your actual hardware with a few thousand clips, then extrapolate. The two levers that matter most are how many frames you feed the video model and how long the dense caption is. Curating first (dropping clips you won't train on) is the cheapest optimization of all.

    Store captions like data, not like a column

    {
      "clip_id": "dQw4w9WgXcQ_000123",
      "captions": [
        {
          "type": "dense",
          "text": "A man in a beige trench coat walks toward the camera along a rain-wet city street at night; neon signs reflect on the pavement; handheld tracking shot, slight push-in.",
          "model": "captioner-v3",
          "prompt_version": "dense-2026.11",
          "alignment": 0.33
        },
        {
          "type": "short",
          "text": "Man in a trench coat walking down a rainy neon street at night",
          "model": "rewriter-v2",
          "prompt_version": "short-2026.11",
          "alignment": 0.31
        },
        {
          "type": "asr",
          "text": "...and that's why I always carry an umbrella...",
          "model": "asr",
          "prompt_version": null,
          "alignment": 0.12
        }
      ]
    }

    Captioner models improve every few months. Keeping every version with its model and prompt lets you recaption selectively, compare versions, and trace a model behavior back to its training text. For the surrounding pipeline see building a YouTube video dataset for multimodal training.

    Compliance

    Recaptioning changes the text, not the rights to the underlying video. YouTube's Terms of Service restrict automated access and downloading, copyright and licensing apply to the source videos, and captions that name or describe identifiable people can carry privacy implications. Keep provenance per clip and get legal review for your use case.

    If you'd rather start from clean source data

    Corpora delivers YouTube video, audio, subtitles and structured metadata at petabyte scale — deduplicated, with checksums and manifests — straight to your cloud bucket, so your team can spend its GPU budget on captioning and training rather than collection. A free sample batch is included. See pricing or talk to sales.

    FAQ

    Why not just use YouTube titles and descriptions as captions?

    They describe what a video is about, not what each clip shows, and after splitting one title would have to cover dozens of different shots. They are useful context for a captioner, not training captions.

    Are auto-generated subtitles good enough as captions?

    For speech models they are useful labels; for video generation they mostly describe speech, not visuals. Use them as optional context when generating visual captions.

    Should training captions be long or short?

    Both. Dense captions teach detail, short ones match how users actually prompt. Train on a mix, or expand short prompts at inference time.

    How do I know if recaptioning helped?

    Audit hallucinations and coverage on a sample, track alignment scores, and ultimately compare small models trained on old versus new captions on the same evaluation prompts.