Tag: speech data

  • Introducing Corpora Curation: YouTube Data Filtered to Spec

    Downloading YouTube at scale is a solved problem. Finding the right videos is not. "Natural two-person English conversation, no background music" or "live-action vertical short dramas" are easy to say and slow to find: search results drift off-topic after a few pages, titles lie, and most candidates fail on things you can only hear or see. Corpora Curation is the system we built to do that search continuously and deliver only what matches your spec. It is now open for early-access projects.

    TL;DR

    • You describe what you need in plain language; it becomes a structured, versioned filter spec that we review with you.
    • Discovery runs continuously across search, channels, playlists and channel feeds, and shifts effort toward the sources that actually produce matches.
    • Candidates pass a funnel from cheap to expensive: metadata rules, content classification, then an audio probe that measures speech, language, speaker count and music.
    • Accuracy is measured, not promised: a human-labeled calibration set before the job runs, and an audit sample with a confidence interval in every batch.
    • Delivery is metadata-only by default (video IDs plus structured metadata and scores), or media files when you need them.

    The problem it solves

    Teams building speech, audio and video models usually don't struggle with bandwidth. They struggle with selection:

    • Keyword search returns a few hundred relevant results, then drifts. The good material lives in specific channels and playlists you have to discover.
    • Metadata is unreliable. A video titled "podcast" can be a single narrator over music; a "short drama" can be a recap channel narrating someone else's footage.
    • The requirements that matter most — two or more speakers, little background music, a specific spoken language — are properties of the audio, not the text.
    • Checking everything by downloading it is slow and expensive, and checking a sample by hand doesn't scale past a few thousand videos.

    Curation is a pipeline built around that reality: spend almost nothing on candidates that obviously fail, and save the expensive checks for the ones that might pass. It is the same principle we describe in video data curation for generative models, applied before download instead of after.

    How a job works

    1. Requirement → spec

    You describe the data in plain language, optionally with a few example videos that should and should not qualify. The system drafts a structured spec: seed search queries in the target language, hard rules (duration, Shorts or not, live streams, category), a written definition of positives and negatives for content classification, audio thresholds, the target count and the delivery format. We review and adjust it with you before anything runs. Every change creates a new spec version, and every decision records which version made it.

    A simplified spec for conversational English speech looks like this:

    target: { count: 20000 }
    rules:
      - "duration_s between 300 and 3600"
      - "not is_live and not is_short"
    stages:
      - type: text_cls
        definition: |
          Positive: natural, unscripted conversation between two or more people
          (podcasts, interviews, casual talk, street interviews).
          Negative: single-narrator monologue, audiobooks, news reading, songs,
          music-heavy vlogs, gameplay commentary, ASMR.
      - type: audio_probe
        require:
          lang: en
          speech_ratio: ">= 0.8"
          n_speakers: ">= 2"
          music_ratio: "<= 0.1"
    download:
      mode: metadata

    2. Continuous discovery

    Discovery expands search queries, channels, playlists and channel feeds, and treats each one as a source with its own track record. Sources that produce qualifying videos get more of the crawl budget; sources that don't are paused. When a channel produces matches, its other videos become candidates too, which is usually where the best material is.

    3. The filter funnel

    Layer What it checks Cost per candidate
    List metadata rules Duration, Shorts, live status, upload date — from listing pages, before any per-video request Lowest
    Detail metadata rules Category, license, available caption and audio languages, resolution and aspect ratio Low
    Content classification Title, description, tags and channel judged against your written definition by a language model Low–medium
    Audio probe Short audio segments analysed on CPU: speech ratio, spoken language, estimated speaker count, music and instrumental ratio Highest — only for candidates that passed everything else

    The audio probe is what makes speech requirements enforceable. It samples segments rather than the whole file (several positions for long videos), runs voice-activity detection, language identification, speaker diarization and audio-event classification, and records every value. A video that passes on text but turns out to be one narrator over a music bed is rejected here, before it costs a download.

    If your requirement is purely about topic or text, the audio layer can be skipped entirely and jobs run much faster.

    4. Calibration and audits

    Before a job runs at volume, it runs in calibration on a small number of candidates, and a sample of its decisions is labeled by a person. That gives a measured precision with a 95% confidence interval for the spec, and shows exactly which layer is making which mistakes. The spec is adjusted until the measured precision meets the target you set.

    After that, every delivery batch includes an audit sample and the same interval. If you want to understand the statistics behind it, see measuring curated dataset accuracy.

    5. Delivery

    Results arrive in batches while the job runs, not all at once at the end:

    • Metadata mode (default): video IDs with structured metadata and every score — no media download, so it is fast and you collect the files on your own infrastructure.
    • Media mode: audio or video files in the format your spec requests, for example 16 kHz mono FLAC for speech, with checksums.
    • Every batch includes a manifest.jsonl (one record per video with the scores that qualified it), a batch_report.json (funnel counts per layer, durations, audit results) and the audit sample.
    • No duplicates for you: a video delivered to you once is never delivered to you again in a later batch or another job.

    A manifest record looks like this:

    {
      "video_id": "abc123XYZ00",
      "url": "https://www.youtube.com/watch?v=abc123XYZ00",
      "channel_id": "UC...",
      "title": "...",
      "duration_s": 1843,
      "lang": "en",
      "license": "youtube",
      "scores": { "text_cls": 0.91 },
      "features": { "speech_ratio": 0.93, "n_speakers": 2, "music_ratio": 0.02 },
      "spec_version": 3,
      "accepted_at": "2026-11-06T12:41:09Z"
    }

    What it is good at today

    • Speech and conversation data by language, speaker count and amount of background music — see what makes audio ASR- and TTS-ready for the criteria that matter.
    • Genre and format selection that can be judged from text and audio: interviews, lectures, tutorials, podcasts, music versus speech, specific content categories.
    • License-constrained jobs, for example Creative Commons only, with the license recorded per video (see Creative Commons YouTube videos for AI training).

    Requirements that can only be judged visually (live-action versus animation, on-screen layout, camera style) currently rely on text signals and are less precise; frame-level visual checks are next on the roadmap. Very subjective definitions ("interesting conversations") are hard to hold at a stable accuracy, which is exactly what calibration exposes before you pay for volume.

    Pricing and early access

    Curation is priced per qualified item delivered, with the price depending on duration range and on which layers your spec needs (text only, or text plus audio). We are taking a limited number of early-access projects. Send us the kind of data you need and a rough volume; we will draft the spec, run calibration and show you the measured precision and a sample batch before you commit. Contact sales.

    Compliance

    YouTube's Terms of Service restrict automated access and downloading, and copyright, licensing and privacy law apply to what is collected — voices and faces can be personal data. Curation records the source, license field and collection time for every item, so any removal request can be traced to the batches it appears in, and it does not attempt to identify the people in the content. Customers are responsible for their own legal review of their intended use.

    FAQ

    How is this different from buying a fixed dataset?

    A fixed dataset was selected for someone else's model. Curation selects against your spec, keeps running until it reaches your target count, and measures its own precision on your definition.

    Do I have to receive media files?

    No. The default is metadata only: IDs, metadata and scores. Many teams prefer to download on their own infrastructure.

    How accurate is it?

    It depends on how objective your requirement is, which is why every job is calibrated with human labels first and every batch carries an audit sample with a 95% confidence interval.

    Can the spec change while the job is running?

    Yes. A change creates a new spec version; decisions record the version that made them, so you can see exactly which items qualified under which rules.

  • yt-dlp Subtitles: Download YouTube Auto Captions for ASR Data

    YouTube subtitles look like free ASR labels, but a naive yt-dlp run mixes three different things: captions uploaded by the creator, YouTube's own speech recognition, and machine translations of either. This guide shows which yt-dlp subtitle flags fetch which tracks, how to tell them apart in info.json, and how to turn rolling auto-caption VTT into clean segment-level JSONL. Everything was checked against the yt-dlp README and source at release 2026.10.26.

    TL;DR

    • --write-subs fetches creator-uploaded (manual) tracks. --write-auto-subs fetches YouTube's auto-generated captions. If both are on and a language exists in both, the manual track wins.
    • For training data, request the original-language ASR track as <lang>-orig (e.g. en-orig). A plain en auto-caption on a non-English video is a machine translation.
    • --sub-langs takes comma-separated, case-insensitive regexes that must match the whole code, plus all and - exclusions such as all,-live_chat.
    • Captions-only runs: --skip-download --write-info-json, throttled with --sleep-subtitles. If you use --download-archive, add --force-write-archive or nothing gets recorded.
    • Auto-caption VTT repeats every line across cues. De-duplicate before you use it (a stdlib script is below), or download json3/srv3 if you need word timings.

    Manual subtitles vs auto-generated captions

    yt-dlp's YouTube extractor fills two separate dictionaries in the info dict:

    • subtitles: tracks uploaded or edited by the creator (caption tracks whose kind is not asr).
    • automatic_captions: YouTube's ASR track, plus machine-translated versions of it and, when requested, of the manual tracks.

    The CLI flags map one-to-one onto these: --write-subs reads subtitles and --write-auto-subs (alias --write-automatic-subs) reads automatic_captions. In YoutubeDL.process_subtitles, manual tracks are added first and an automatic caption is only added for a language code that isn't already present. Either way the file is written as <name>.<lang>.<ext>, so the filename alone doesn't tell you which kind you got. Keep the .info.json and check the code against its subtitles keys (example below). Note that the cleaned info.json drops requested_subtitles, so you can't read the selection back from the file.

    Without --sub-langs, yt-dlp downloads exactly one track: en if present, otherwise the first en* code, otherwise the first available language.

    See what exists first: --list-subs

    yt-dlp --list-subs "https://www.youtube.com/watch?v=VIDEO_ID"

    This prints two tables (automatic captions and subtitles) with every language code and the formats on offer, and it only simulates unless you add --no-simulate. The same tables are available as the output-template fields automatic_captions_table and subtitles_table. On a typical video the automatic list is long: that's the translations, covered next.

    Selecting languages with --sub-langs

    The README defines --sub-langs as a comma-separated list of language codes, regexes, or all, where a leading - excludes a code. Two details from the source matter in practice:

    • Each pattern is compiled case-insensitively and must fully match the code. en does not match en-US. en.* matches en, en-US, en-GB, en-orig, and every en-xx translation.
    • Items are processed left to right, so all,-live_chat means "everything, then remove live_chat". Live-stream replays expose chat as a subtitle track named live_chat. You can also use --compat-options no-live-chat.
    # every manual track except chat replay
    yt-dlp --write-subs --sub-langs "all,-live_chat" --skip-download URL
    
    # English and Japanese manual tracks, any region variant
    yt-dlp --write-subs --sub-langs "en.*,ja" --skip-download URL

    Use broad regexes only with --write-subs alone. Combined with --write-auto-subs, they pull in translations.

    Auto-translated tracks: how to spot them and why to drop them

    In the YouTube extractor (extractor/youtube/_video.py), each caption track is expanded against YouTube's list of translation languages. The codes you'll see in automatic_captions:

    Key What it is Use for training?
    en-orig (any <lang>-orig) The ASR track in the video's spoken language. yt-dlp adds the -orig label, with the name "… (Original)", so it can be told apart. Yes, as a weak label
    en when the audio is English The same original ASR track, kept under the plain code for compatibility. Yes (duplicate of en-orig)
    de, fr, … when the audio is English Machine translation of the ASR track (URL carries tlang=). No
    de-en (target-source) Machine translation of the manual en track. Only built when --write-auto-subs or --list-subs is used. No

    Translated tracks are text produced by a translation model, time-aligned to someone else's speech. As ASR labels they're simply wrong for the audio, and as translation pairs they're synthetic. Rules that hold up:

    • Request <lang>-orig explicitly for ASR captions. The -orig key is added when the ASR track is translatable. If it is missing, fall back to the plain code only after confirming the URL has no tlang parameter.
    • Treat any requested track whose URL contains tlang= as a translation and discard it.
    • --extractor-args "youtube:skip=translated_subs" stops yt-dlp from building translations of manual tracks (the xx-yy keys) and saves extraction time. In the current source it does not remove the plain-code translations of the ASR track, so the two rules above still apply.

    Subtitle formats: --sub-format, --convert-subs, --embed-subs

    For YouTube, the extractor currently offers every track in seven formats: json3, srv1, srv2, srv3, ttml, srt and vtt. --sub-format takes a /-separated preference list. best means the last format the extractor lists, which for YouTube is vtt. If nothing matches, yt-dlp warns and falls back to best.

    • vtt: the default and the easiest to parse. Auto-captions include per-word start times as inline tags.
    • json3, srv3: YouTube's own timed-text formats, useful for word-level timing (below). YouTube doesn't document them.
    • --convert-subs FORMAT converts to ass, lrc, srt or vtt through ffmpeg. TTML is converted to SRT first, with a warning that styling is lost. There's no converter for json3 or srv*, so keep those raw and parse them yourself.
    • --embed-subs muxes tracks into mp4, webm or mkv output. Used alone, it deletes the sidecar files after embedding. Add --write-subs to keep them. It does nothing on a --skip-download run, and for datasets sidecar files are easier to process than embedded streams.

    Captions-only runs at scale

    Caption files are kilobytes, so a captions-plus-metadata pass is a cheap way to decide which videos are worth downloading audio for:

    yt-dlp -a ids.txt --skip-download \
      --write-subs --write-auto-subs --sub-langs "en,en-orig" --sub-format vtt \
      --extractor-args "youtube:skip=translated_subs" \
      --write-info-json \
      --sleep-requests 1 --sleep-subtitles 2 \
      --download-archive subs_archive.txt --force-write-archive \
      -o "meta/%(id)s.%(ext)s" \
      -o "subtitle:subs/%(id)s.%(ext)s"
    • Output templates. The subtitle: prefix sets a separate template for subtitle files, and yt-dlp appends the language and extension: subs/VIDEO_ID.en-orig.vtt. -P "subtitle:subs" sets only the directory. The .info.json lands in meta/ (there's an infojson: prefix too).
    • Archive gotcha. With --skip-download, yt-dlp only writes archive entries if --force-write-archive is set. Without it, every re-run starts from zero.
    • Rate limits. --sleep-subtitles sleeps before each subtitle download and --sleep-requests between extraction requests. --sleep-interval applies to media downloads, not subtitle files. The -t sleep preset expands to --sleep-subtitles 5 --sleep-requests 0.75 --sleep-interval 10 --max-sleep-interval 20.
    • Missing tracks. If YouTube requires a PO token for subtitle requests and none is available, the extractor skips subtitles for that client and warns "There are missing subtitles languages because a PO token was not provided." The README documents a subs context for the po_token extractor argument. Current yt-dlp also needs a JavaScript runtime and yt-dlp-ejs for full YouTube support.
    • Bot checks and IPs. At volume, subtitle and metadata requests hit the same anti-bot systems as video. See fixing "Sign in to confirm you're not a bot" and the yt-dlp proxy guide.

    Python API equivalents

    The CLI flags map to YoutubeDL params (the README points to devscripts/cli_to_api.py for translating any command line):

    CLI Python param
    --write-subs writesubtitles=True
    --write-auto-subs writeautomaticsub=True
    --sub-langs "en,en-orig" subtitleslangs=["en", "en-orig"]
    --sub-format vtt subtitlesformat="vtt"
    --list-subs listsubtitles=True
    --skip-download skip_download=True
    --sleep-subtitles 2 sleep_interval_subtitles=2
    --sleep-requests 1 sleep_interval_requests=1
    --force-write-archive force_write_download_archive=True
    --convert-subs srt postprocessors=[{"key": "FFmpegSubtitlesConvertor", "format": "srt", "when": "before_dl"}]
    --embed-subs postprocessors=[{"key": "FFmpegEmbedSubtitle"}]

    This script fetches captions only and writes a manifest line per track labeled manual, auto or auto-translated, using the precedence and tlang rules above:

    import json
    import sys
    
    import yt_dlp
    
    LANG = "en"
    OPTS = {
        "skip_download": True,                       # --skip-download
        "writesubtitles": True,                      # --write-subs
        "writeautomaticsub": True,                   # --write-auto-subs
        "subtitleslangs": [LANG, f"{LANG}-orig"],    # --sub-langs "en,en-orig"
        "subtitlesformat": "vtt",                    # --sub-format vtt
        "writeinfojson": True,                       # --write-info-json
        "outtmpl": {"default": "subs/%(id)s/%(id)s.%(ext)s"},
        "extractor_args": {"youtube": {"skip": ["translated_subs"]}},
        "sleep_interval_requests": 1,                # --sleep-requests 1
        "sleep_interval_subtitles": 2,               # --sleep-subtitles 2
        "ignoreerrors": True,                        # log failures, keep going
        # "proxy": "http://USER:[email protected]:8000",
    }
    
    
    def track_kind(info, lang, sub):
        if lang in (info.get("subtitles") or {}):
            return "manual"            # manual wins when both flags are set
        if "tlang=" in (sub.get("url") or ""):
            return "auto-translated"   # machine translation: drop for training
        return "auto"
    
    
    def main(ids_path, manifest_path):
        with open(ids_path, encoding="utf-8") as f:
            ids = [line.strip() for line in f if line.strip()]
        with yt_dlp.YoutubeDL(OPTS) as ydl, \
                open(manifest_path, "a", encoding="utf-8") as out:
            for vid in ids:
                info = ydl.extract_info(f"https://www.youtube.com/watch?v={vid}")
                if not info:
                    continue  # extraction failed; already reported
                for lang, sub in (info.get("requested_subtitles") or {}).items():
                    out.write(json.dumps({
                        "id": vid,
                        "lang": lang,
                        "kind": track_kind(info, lang, sub),
                        "file": f"subs/{vid}/{vid}.{lang}.{sub['ext']}",
                    }) + "\n")
    
    
    if __name__ == "__main__":
        main(*sys.argv[1:3])

    Run it as python fetch_captions.py ids.txt captions_manifest.jsonl. On an English video with uploaded captions you get en (manual) and en-orig (auto). On a Spanish video without English captions, en is labeled auto-translated and en-orig doesn't exist.

    Cleaning rolling auto-caption VTT

    YouTube's auto-caption VTT is built for a two-line karaoke display, not for parsing. A typical stretch looks like this (abridged):

    00:00:00.160 --> 00:00:02.389 align:start position:0%
     
    in<00:00:00.480><c> this</c><00:00:00.800><c> video</c>
    
    00:00:02.389 --> 00:00:02.399 align:start position:0%
    in this video
     
    
    00:00:02.399 --> 00:00:04.950 align:start position:0%
    in this video
    going<00:00:02.720><c> to</c><00:00:03.040><c> talk</c>

    Each cue repeats the previous line on top, and 10 ms "hold" cues repeat it again. (The seemingly blank line after the first timing line is a single space, so it doesn't end the cue.) Naively concatenating the cue text triples most words. The fix: strip tags, then drop the longest run of lines at the start of each cue that equals the end of the previous cue. Manual tracks don't have this layout, so the script detects auto-captions by their inline word timestamps and only merges identical back-to-back cues for manual tracks.

    #!/usr/bin/env python3
    """YouTube WebVTT -> segment-level JSONL ({"start", "end", "text"}), stdlib only.
    
    Auto-generated tracks use a "rolling" layout: each cue repeats the previous
    line above the new words, and ~10 ms cues re-show text that was already
    emitted. Those repeats are removed. Manual tracks are passed through, with
    identical back-to-back cues merged.
    
    usage: python vtt_to_jsonl.py VIDEO.en-orig.vtt > VIDEO.en-orig.jsonl
    """
    import html
    import json
    import re
    import sys
    
    TS = r"(?:(\d+):)?(\d{2}):(\d{2})\.(\d{3})"
    TIMING = re.compile(rf"^{TS}\s+-->\s+{TS}")
    WORD_TS = re.compile(r"<(?:\d+:)?\d{2}:\d{2}\.\d{3}>")  # only in auto captions
    TAG = re.compile(r"<[^>]*>")             # <c>, </c>, <00:00:01.234>, <i>, ...
    NON_SPEECH = re.compile(r"\[[^\]]*\]")   # [Music], [Applause]; None to keep them
    
    
    def seconds(h, m, s, ms):
        return int(h or 0) * 3600 + int(m) * 60 + int(s) + int(ms) / 1000
    
    
    def clean(line):
        text = html.unescape(TAG.sub("", line))
        if NON_SPEECH:
            text = NON_SPEECH.sub("", text)
        return " ".join(text.split())
    
    
    def parse_cues(lines):
        """Yield (start, end, [text lines]). A cue ends at an empty line."""
        block = []
        for line in lines + [""]:
            if line:
                block.append(line)
                continue
            for i, row in enumerate(block):  # header/NOTE/STYLE blocks never match
                m = TIMING.match(row)
                if m:
                    g = m.groups()
                    text = [clean(t) for t in block[i + 1:]]
                    yield seconds(*g[:4]), seconds(*g[4:]), [t for t in text if t]
                    break
            block = []
    
    
    def overlap(prev, cur):
        """Length of the longest suffix of prev that is also a prefix of cur."""
        for k in range(min(len(prev), len(cur)), 0, -1):
            if prev[-k:] == cur[:k]:
                return k
        return 0
    
    
    def segments(path):
        with open(path, encoding="utf-8-sig") as f:
            raw = f.read()
        rolling = bool(WORD_TS.search(raw))
        out, prev = [], []
        for start, end, lines in parse_cues(raw.splitlines()):
            if not lines:
                continue
            new = lines[overlap(prev, lines):] if rolling else lines
            prev = lines
            if not new:
                continue  # cue only repeats text that was already emitted
            text = " ".join(new)
            last = out[-1] if out else None
            if last and last["text"] == text and start - last["end"] < 0.05:
                last["end"] = round(max(last["end"], end), 3)  # same caption held
                continue
            out.append({"start": round(start, 3), "end": round(end, 3), "text": text})
        return out
    
    
    if __name__ == "__main__":
        if len(sys.argv) != 2:
            sys.exit("usage: vtt_to_jsonl.py FILE.vtt > FILE.jsonl")
        for seg in segments(sys.argv[1]):
            print(json.dumps(seg, ensure_ascii=False))

    On the excerpt above it emits one segment per spoken line, with no repeats:

    {"start": 0.16, "end": 2.389, "text": "in this video"}
    {"start": 2.399, "end": 4.95, "text": "going to talk"}

    Notes: NON_SPEECH strips bracketed tags like [Music]. Set it to None if you want them as non-speech labels. Auto-captions may also contain >> speaker-change markers, which you can split on or strip. Segment boundaries follow YouTube's display lines, not sentences, so re-segment by pauses or punctuation if your model needs utterance-level samples.

    Word-level timing: json3 and srv3

    For forced-alignment checks or word-level supervision, request the native formats with --sub-format json3 or srv3 (they can't be converted, so request them directly). YouTube doesn't publish either schema, so treat what follows as observed structure and validate it on a sample:

    • json3: a top-level events list. Each event has tStartMs, dDurationMs and a segs list of text pieces in utf8. On auto-caption tracks, words are separate segs carrying an offset from the event start (tOffsetMs). Open-source parsers such as this one read the same fields.
    • srv3: XML with <p t="…" d="…"> lines (milliseconds) and <s> spans. See this reverse-engineered format description.
    import json
    
    def json3_words(path):
        with open(path, encoding="utf-8") as f:
            events = json.load(f).get("events", [])
        for ev in events:
            t0 = ev.get("tStartMs", 0)
            for seg in ev.get("segs") or []:
                word = seg.get("utf8", "").strip()
                if word:
                    yield (t0 + seg.get("tOffsetMs", 0)) / 1000, word

    These are word start times only. Derive ends from the next word's start, capped by the event's end, and expect them to be approximate.

    Which flags for which need

    Need Flags
    See what tracks exist --list-subs
    Captions only, no media --skip-download + subtitle flags
    Creator-uploaded English, any region --write-subs --sub-langs "en.*"
    Original-language ASR only --write-auto-subs --sub-langs "en-orig"
    Manual if present, plus original ASR --write-subs --write-auto-subs --sub-langs "en,en-orig", then check info.json
    All manual tracks except chat replay --write-subs --sub-langs "all,-live_chat"
    Skip translations of manual tracks --extractor-args "youtube:skip=translated_subs"
    Word-level timing --sub-format json3 (or srv3)
    SRT files --sub-format srt or --sub-format vtt --convert-subs srt
    Subtitles inside the video file --embed-subs (add --write-subs to keep sidecars)
    Separate subtitle folder -o "subtitle:subs/%(id)s.%(ext)s" or -P "subtitle:subs"
    Throttle subtitle requests --sleep-subtitles N --sleep-requests N or -t sleep
    Resumable captions-only runs --download-archive FILE --force-write-archive

    Captions as ASR labels vs re-transcribing with Whisper

    Neither caption type is a clean transcript. Manual captions are often edited for reading: they condense speech, drop disfluencies, add speaker names and sound tags, and can be offset in time. Auto-captions are another ASR system's output, so training on them distills its errors. They can also lack punctuation and casing. A practical split:

    • Use caption availability and language as cheap filters before you download any audio (pair with the audio-only download guide).
    • Use manual captions as labels only after alignment and filtering, e.g. drop segments where your own ASR disagrees beyond a WER threshold you calibrate per language.
    • Where label quality matters, re-transcribe with Whisper and keep the captions as a cross-check. Disagreement between the two is a useful quality signal.

    The Whisper side (model choice, language ID, per-language thresholds) is covered in building a YouTube video dataset for multimodal training. Whatever you choose, store the track kind (manual/auto) and language with every segment so you can re-weight or drop sources later.

    Compliance

    YouTube's Terms of Service restrict automated access and downloading, and captions are content like any other. Copyright and licensing apply: some teams restrict collection to Creative Commons videos (see our guide to Creative Commons YouTube data) and record the license per sample. Voices and transcribed speech can be personal data under regimes like GDPR. Honor takedown and opt-out requests, and keep the lineage you need to do so. Get legal review for your jurisdiction and use case.

    If you'd rather not run this yourself

    Corpora offers dedicated 1–40 Gbps unmetered proxy capacity for teams running their own yt-dlp stack. It also offers a fully managed option that delivers video, audio, subtitles and metadata as JSONL/Parquet with sha256 checksums and manifests to your S3, GCS or Azure bucket, starting with a free sample batch. Contact us to scope it.

    FAQ

    How do I download only subtitles with yt-dlp, without the video?

    Add --skip-download to your subtitle flags, e.g. yt-dlp --skip-download --write-subs --write-auto-subs --sub-langs "en,en-orig" URL. Add --write-info-json if you need to tell manual and auto tracks apart later.

    What is the difference between --write-subs and --write-auto-subs?

    --write-subs downloads tracks the creator uploaded (the subtitles key). --write-auto-subs downloads YouTube's automatic captions and their machine translations (the automatic_captions key). With both, a manual track takes precedence for the same language code.

    What does en-orig mean in yt-dlp?

    It's yt-dlp's label for the auto-generated caption track in the video's original spoken language. Other plain codes in the automatic list are usually machine translations of that track.

    Why does my auto-generated VTT repeat every line?

    YouTube's auto-caption VTT is a rolling two-line display: each cue repeats the previous line and short hold cues repeat it again. Strip the inline tags and remove the overlap between consecutive cues, as the script above does.

    Can yt-dlp give word-level timestamps from YouTube captions?

    For auto-captions, yes, indirectly. Download json3 or srv3 (or read the inline timestamps in VTT) and parse the per-word offsets yourself. Manual tracks are usually timed per line only.