Tag: shot detection

  • Split Video into Clips with PySceneDetect for Training Data

    Long videos are not training examples. Before you caption, filter or train, you have to split video into clips that each show one continuous shot and fit the window your model samples. This guide covers PySceneDetect's detectors (CLI and Python), TransNetV2 as a learned alternative, how Panda-70M split and re-stitched clips, frame-accurate cutting with ffmpeg, and what to record per clip.

    TL;DR

    • Detect, then cut. Run shot detection once, store boundaries as timestamps, and treat file cutting as a separate, repeatable step.
    • Start with AdaptiveDetector, add ThresholdDetector for fades, and tune thresholds from a stats file.
    • Raw shot boundaries over-split. Panda-70M stitched similar neighbors back together and roughly doubled average clip length.
    • Re-encode for frame-accurate cuts. Stream copy (-c copy) can only cut at keyframes.
    • Derive min/max length from your model's sampling window, drop short clips, split long shots, trim edges, and record why every boundary exists.

    Why splitting comes first

    Almost every downstream signal assumes a clip shows one continuous shot. A motion score on a clip containing a hard cut measures the cut; a caption describing two unrelated scenes teaches the model to blend them. The Stable Video Diffusion paper (arXiv 2311.15127) ran PySceneDetect in a cascaded manner at three frame rates to keep cuts and fades out of generated videos, and reports roughly 4x more clips after cut detection, meaning many source clips contained cuts that metadata didn't reveal.

    Splitting also defines your unit of data: every later filter, caption and dedup decision is keyed by clip ID. The broader pipeline is in building a YouTube video dataset for multimodal training, and the filters that run after splitting are in video data curation filters for generative models. This article goes deep on the splitting step.

    PySceneDetect: the detectors that matter

    PySceneDetect (BSD-3-Clause, v0.7.3 at the time of writing) is the tool both SVD and Panda-70M used. Install with pip install scenedetect --upgrade and put ffmpeg (or mkvmerge) on your PATH for splitting. Version 0.7 reworked timestamps to support variable-frame-rate video and moved some imports; if you have v0.6 code, read the migration guide. The docs recommend pinning scenedetect~=0.7.

    The three detectors you'll use most (defaults from the detector reference):

    Detector (CLI command) What it measures Default threshold Use it for
    ContentDetector (detect-content) Weighted frame-to-frame change in hue, saturation and luma (edges optional), 0–255 27.0 Hard cuts, steady cameras
    AdaptiveDetector (detect-adaptive) Same score, compared to a rolling average of neighboring frames 3.0 ratio, min_content_val=15.0 Hard cuts with fast camera motion; the CLI default
    ThresholdDetector (detect-threshold) Average pixel intensity crossing a fixed level 12.0 Fades to and from black

    One trap: min_scene_len defaults to 15 frames in Python but 0.6 s on the CLI, and ThresholdDetector's add_final_scene and min_out_length also differ. If you prototype on the CLI and productionize in Python, set them explicitly. None of these detectors is designed for cross-dissolves or wipes, where change is spread over many frames; that gap is why learned detectors and Panda-70M's extra heuristics exist.

    The CLI: fastest way to look at your data

    Global options go before commands. This detects cuts, drops (rather than merges) scenes shorter than 2 seconds, writes a CSV scene list and splits with ffmpeg:

    scenedetect -i input.mp4 -o clips/ -m 2s --drop-short-scenes \
      detect-adaptive list-scenes split-video

    To tune a threshold, write per-frame metrics first. -s saves a stats CSV with columns such as content_val; list-scenes -n prints scenes without writing a file:

    scenedetect -i input.mp4 -s input.stats.csv detect-content list-scenes -n

    Plot content_val for a few videos per source category. True cuts are isolated spikes; a threshold inside the noise floor of handheld footage over-splits, one above low-contrast cuts misses them. Thresholds that work for studio interviews rarely work for drone footage, so keep them per category.

    split-video re-encodes by default with -map 0:v:0 -map 0:a? -map 0:s? -c:v libx264 -preset veryfast -crf 22 -c:a aac. Options include -hq (CRF 17, preset slow), -crf, -p, -a for custom ffmpeg arguments, -f for filenames, and -c/--copy, which the config reference calls "significantly faster, but can result in inaccurate splits due to keyframe positioning".

    The Python API: what you'll run at scale

    In a pipeline you want boundaries as data. detect(path, AdaptiveDetector()) returns (start, end) FrameTimecode pairs. To run a cut detector and a fade detector in one decoding pass, register both on a SceneManager:

    from scenedetect import open_video, SceneManager, AdaptiveDetector, ThresholdDetector
    
    video = open_video("input.mp4")
    sm = SceneManager()
    sm.add_detector(AdaptiveDetector(min_scene_len="0.6s"))
    sm.add_detector(ThresholdDetector(threshold=12.0, min_scene_len="0.6s"))
    sm.detect_scenes(video, show_progress=True)
    
    for start, end in sm.get_scene_list(start_in_scene=True):
        print(start.get_timecode(), end.get_timecode(), end.seconds - start.seconds)

    start_in_scene=True makes a video with no cuts return one scene instead of an empty list. In v0.7, frame_num, seconds and frame_rate are properties, and video.frame_rate is an exact Fraction such as 30000/1001. Decoding is usually the bottleneck; frames are downscaled automatically, and the docs advise against frame_skip except for very high-frame-rate video.

    TransNetV2: a learned shot boundary detector

    TransNetV2 (arXiv 2008.04838) is a neural network built for shot transition detection, including gradual transitions. In the authors' re-evaluation it scores F1 77.9 on ClipShots, 96.2 on BBC Planet Earth and 93.9 on RAI. Those are their benchmarks; validate on a labeled sample of your own sources before switching.

    The reference inference code pins tensorflow==2.1, old enough that you'll likely want the provided Docker image or the inference-only PyTorch port in inference-pytorch/, which converts the TF weights. Weights are stored with git-lfs. Usage:

    from transnetv2 import TransNetV2
    
    model = TransNetV2()
    frames, single_frame_pred, all_frame_pred = model.predict_video("input.mp4")
    scenes = model.predictions_to_scenes(single_frame_pred)  # threshold defaults to 0.5
    # scenes: array of [start_frame, end_frame], zero-indexed, both inclusive

    The model sees frames resized to 48x27 via ffmpeg, so decoding dominates and a GPU helps. Output is frame indices, so convert with the source frame rate and take care with variable-frame-rate files. Its advantage over thresholded detectors is a per-frame transition probability: store the peak near each boundary as a confidence score and you can re-threshold later without re-running the model. A common compromise is PySceneDetect on everything and TransNetV2 on the subset that later filters flag as suspicious.

    How Panda-70M split and stitched

    Panda-70M split 3.8M long videos from HD-VILA-100M into 70.8M clips averaging 8.5 seconds. Its splitting code is public and encodes several practical lessons:

    1. Shot detection: ContentDetector with threshold=25, min_scene_len=15 frames and start_in_scene=True.
    2. Forced cuts: to handle undetected fades and unedited footage, it "recursively cut[s] the first 5 seconds as a new clip if the video clip is longer than 7 seconds".
    3. Stitching: ImageBind embeddings of frames near each piece's start and end; adjacent pieces with close embeddings are merged. Pieces whose own start and end differ sharply (likely a transition inside) and near-static events are dropped.
    4. Length and trimming: the README says Panda-70M itself was built with a 2.5 s minimum, a 60 s maximum and 10% trimmed from both ends; the repo's default line uses different values.
    5. Cutting: ffmpeg with -ss before -i and -t, re-encoded rather than stream-copied.

    On 1K long videos, the paper reports plain PySceneDetect splitting gave a 4.1 s average clip with max running LPIPS (the largest perceptual jump inside a clip) of 0.247, while its splitting gave 7.9 s at 0.256: nearly double the length for a small loss in consistency. To download Panda-70M itself, see downloading Panda-70M, HD-VILA and InternVid.

    Cutting with ffmpeg: keyframes, stream copy and re-encoding

    Compressed video stores full pictures only at keyframes; other frames are deltas. The FFmpeg seeking wiki spells out the consequences:

    • Re-encoding is frame-accurate. Since FFmpeg 2.1, -ss before -i is both fast and frame-accurate when transcoding.
    • Stream copy is not. With -c copy ffmpeg may only split on keyframes. The wiki's example: requested 157 s, next keyframe at 159 s, so the output starts with about 2 s of audio and no video. In training data that means lead-ins from the previous shot or frozen frames.
    • Keyframes can be far apart. libx264's default GOP of 250 is one keyframe every 10 s at 25 fps. Web uploads vary, so measure.

    To see where keyframes are, list video packets; keyframes have K in the flags field:

    ffprobe -v error -select_streams v:0 \
      -show_entries packet=pts_time,flags -of csv=p=0 input.mp4

    Rule of thumb: re-encode the clips you train on, at a low CRF so compression artifacts don't become something the model learns. Stream copy is acceptable only if your data loader decodes and trims to stored boundaries anyway.

    Clip length targets and short-clip filtering

    Pick lengths from your model, not the detector. If it samples N frames at F fps, a clip shorter than N/F seconds can't fill a window without looping or padding: 64 frames at 8 fps needs 64 / 8 = 8 s. Published reference points:

    Dataset Average clip length Source
    Panda-70M 8.5 s Panda-70M paper, Table 1
    SVD's LVD (before filtering) 11.58 s SVD paper, Table 1
    HD-VILA-100M (subtitle-aligned) 13.4 s Panda-70M paper, Table 1
    • Drop too-short clips. Don't merge them into neighbors without a stitching signal; merging across a cut reintroduces the problem.
    • Split too-long shots into windows at your maximum, and mark those boundaries as forced, not detected.
    • Trim the edges. Boundaries can be a frame off and fades bleed into neighbors; dropping a few frames per side is cheap insurance.

    A complete pipeline: detect, plan, cut, record

    This script detects boundaries, applies the length rules, cuts frame-accurately and writes one JSONL row per clip. The numbers are starting points, not recommendations for your data.

    import hashlib, json, subprocess
    from importlib.metadata import version
    from pathlib import Path
    from scenedetect import open_video, SceneManager, AdaptiveDetector, ThresholdDetector
    
    SRC = Path("raw/abc123XYZ_-.mp4")
    OUT = Path("clips"); OUT.mkdir(exist_ok=True)
    MIN_LEN, MAX_LEN, EDGE_TRIM = 2.0, 20.0, 0.1   # seconds
    
    video = open_video(str(SRC))
    sm = SceneManager()
    sm.add_detector(AdaptiveDetector(min_scene_len="0.6s"))
    sm.add_detector(ThresholdDetector(threshold=12.0, min_scene_len="0.6s"))
    sm.detect_scenes(video)
    scenes = sm.get_scene_list(start_in_scene=True)
    
    def plan(shot_start, shot_end):
        s, e = shot_start + EDGE_TRIM, shot_end - EDGE_TRIM
        reason = "forced_max_len" if (e - s) > MAX_LEN else "shot"
        t = s
        while e - t >= MIN_LEN:
            seg_end = min(t + MAX_LEN, e)
            yield t, seg_end, reason
            t = seg_end
    
    rows = []
    for shot_idx, (start, end) in enumerate(scenes):
        for s, e, reason in plan(start.seconds, end.seconds):
            clip_id = f"{SRC.stem}_{len(rows):05d}"
            out = OUT / f"{clip_id}.mp4"
            subprocess.run([
                "ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
                "-ss", f"{s:.3f}", "-i", str(SRC), "-t", f"{e - s:.3f}",
                "-map", "0:v:0", "-map", "0:a?",
                "-c:v", "libx264", "-preset", "veryfast", "-crf", "18",
                "-c:a", "aac", str(out)], check=True)
            dur = subprocess.run(
                ["ffprobe", "-v", "error", "-show_entries", "format=duration",
                 "-of", "csv=p=0", str(out)],
                capture_output=True, text=True, check=True).stdout.strip()
            rows.append({
                "clip_id": clip_id, "source_video_id": SRC.stem, "shot_index": shot_idx,
                "start_s": round(s, 3), "end_s": round(e, 3),
                "shot_start_frame": start.frame_num, "shot_end_frame": end.frame_num,
                "source_fps": str(video.frame_rate), "boundary_reason": reason,
                "detector": "adaptive(3.0)+threshold(12.0)",
                "scenedetect_version": version("scenedetect"),
                "encode": "libx264 veryfast crf18, aac",
                "probed_duration_s": float(dur),
                "sha256": hashlib.sha256(out.read_bytes()).hexdigest(),
            })
    
    with open(OUT / f"{SRC.stem}.clips.jsonl", "w", encoding="utf-8") as f:
        for r in rows:
            f.write(json.dumps(r) + "\n")

    Two production checks: compare probed_duration_s to end_s - start_s (a large gap means a decode problem), and inspect the first and last frames of a random sample. Measuring dataset accuracy with audits and Wilson intervals shows how to size that sample and report a boundary error rate.

    Metadata to record per clip

    Splitting owns provenance; later stages add scores to the same row, as in the curation filters guide. At minimum:

    • Provenance: clip ID, source video ID, source checksum, collection date, license at collection time.
    • Boundaries: start/end in seconds and source frames, exact source frame rate, whether the source is variable frame rate.
    • Boundary reason: detected cut, fade, forced split or stitched, plus a confidence if your detector provides one.
    • Reproducibility: detector, thresholds, library version, encode settings, pipeline version.
    • Output facts: probed duration, resolution, audio present, sha256.

    Keep boundaries even for clips you drop. Changing a threshold should be a query over stored rows, not a re-run of detection over millions of videos.

    Compliance

    Splitting turns each source into many derived files, so provenance matters for compliance too. YouTube's Terms of Service restrict automated access and downloading; copyright and licensing apply to every clip (filtering for Creative Commons videos is one option); faces and voices can be personal data under GDPR and similar laws; and takedown or opt-out requests must reach every derived clip, which requires each clip to carry its source ID. Get legal review for your jurisdiction and use case.

    If you'd rather not run this yourself

    Corpora collects YouTube video, audio, subtitles and structured metadata and delivers deduplicated datasets as JSONL/Parquet with sha256 checksums and manifests to S3, GCS, Azure Blob or private object storage, with a free sample batch to check quality first. See pricing or get in touch.

    FAQ

    How do I split a video into clips automatically by scene?

    Install PySceneDetect and ffmpeg, then run scenedetect -i input.mp4 detect-adaptive split-video. For training data, add -m 2s --drop-short-scenes and keep the list-scenes CSV as metadata.

    Which PySceneDetect detector should I use?

    AdaptiveDetector is the CLI default and copes with camera motion better than ContentDetector. Add ThresholdDetector if sources fade to black, and tune thresholds per source category with -s stats.csv.

    Why do my ffmpeg clips start early or show black frames?

    You are probably stream-copying (-c copy), which cuts at keyframes. Re-encode, with -ss before -i, for frame-accurate cuts.

    Is TransNetV2 better than PySceneDetect?

    It is a learned model that targets gradual transitions as well as hard cuts and outputs per-frame probabilities, but it needs a GPU at scale and its reference code pins an old TensorFlow. Compare both on a labeled sample of your own videos.

    How long should training clips be?

    At least your model's sampling window (frames divided by sampling fps), with a maximum so long shots don't dominate. Panda-70M averages 8.5 s, built with a 2.5 s minimum and 60 s maximum.