Tag: ASR datasets

  • yt-dlp Subtitles: Download YouTube Auto Captions for ASR Data

    YouTube subtitles look like free ASR labels, but a naive yt-dlp run mixes three different things: captions uploaded by the creator, YouTube's own speech recognition, and machine translations of either. This guide shows which yt-dlp subtitle flags fetch which tracks, how to tell them apart in info.json, and how to turn rolling auto-caption VTT into clean segment-level JSONL. Everything was checked against the yt-dlp README and source at release 2026.10.26.

    TL;DR

    • --write-subs fetches creator-uploaded (manual) tracks. --write-auto-subs fetches YouTube's auto-generated captions. If both are on and a language exists in both, the manual track wins.
    • For training data, request the original-language ASR track as <lang>-orig (e.g. en-orig). A plain en auto-caption on a non-English video is a machine translation.
    • --sub-langs takes comma-separated, case-insensitive regexes that must match the whole code, plus all and - exclusions such as all,-live_chat.
    • Captions-only runs: --skip-download --write-info-json, throttled with --sleep-subtitles. If you use --download-archive, add --force-write-archive or nothing gets recorded.
    • Auto-caption VTT repeats every line across cues. De-duplicate before you use it (a stdlib script is below), or download json3/srv3 if you need word timings.

    Manual subtitles vs auto-generated captions

    yt-dlp's YouTube extractor fills two separate dictionaries in the info dict:

    • subtitles: tracks uploaded or edited by the creator (caption tracks whose kind is not asr).
    • automatic_captions: YouTube's ASR track, plus machine-translated versions of it and, when requested, of the manual tracks.

    The CLI flags map one-to-one onto these: --write-subs reads subtitles and --write-auto-subs (alias --write-automatic-subs) reads automatic_captions. In YoutubeDL.process_subtitles, manual tracks are added first and an automatic caption is only added for a language code that isn't already present. Either way the file is written as <name>.<lang>.<ext>, so the filename alone doesn't tell you which kind you got. Keep the .info.json and check the code against its subtitles keys (example below). Note that the cleaned info.json drops requested_subtitles, so you can't read the selection back from the file.

    Without --sub-langs, yt-dlp downloads exactly one track: en if present, otherwise the first en* code, otherwise the first available language.

    See what exists first: --list-subs

    yt-dlp --list-subs "https://www.youtube.com/watch?v=VIDEO_ID"

    This prints two tables (automatic captions and subtitles) with every language code and the formats on offer, and it only simulates unless you add --no-simulate. The same tables are available as the output-template fields automatic_captions_table and subtitles_table. On a typical video the automatic list is long: that's the translations, covered next.

    Selecting languages with --sub-langs

    The README defines --sub-langs as a comma-separated list of language codes, regexes, or all, where a leading - excludes a code. Two details from the source matter in practice:

    • Each pattern is compiled case-insensitively and must fully match the code. en does not match en-US. en.* matches en, en-US, en-GB, en-orig, and every en-xx translation.
    • Items are processed left to right, so all,-live_chat means "everything, then remove live_chat". Live-stream replays expose chat as a subtitle track named live_chat. You can also use --compat-options no-live-chat.
    # every manual track except chat replay
    yt-dlp --write-subs --sub-langs "all,-live_chat" --skip-download URL
    
    # English and Japanese manual tracks, any region variant
    yt-dlp --write-subs --sub-langs "en.*,ja" --skip-download URL

    Use broad regexes only with --write-subs alone. Combined with --write-auto-subs, they pull in translations.

    Auto-translated tracks: how to spot them and why to drop them

    In the YouTube extractor (extractor/youtube/_video.py), each caption track is expanded against YouTube's list of translation languages. The codes you'll see in automatic_captions:

    Key What it is Use for training?
    en-orig (any <lang>-orig) The ASR track in the video's spoken language. yt-dlp adds the -orig label, with the name "… (Original)", so it can be told apart. Yes, as a weak label
    en when the audio is English The same original ASR track, kept under the plain code for compatibility. Yes (duplicate of en-orig)
    de, fr, … when the audio is English Machine translation of the ASR track (URL carries tlang=). No
    de-en (target-source) Machine translation of the manual en track. Only built when --write-auto-subs or --list-subs is used. No

    Translated tracks are text produced by a translation model, time-aligned to someone else's speech. As ASR labels they're simply wrong for the audio, and as translation pairs they're synthetic. Rules that hold up:

    • Request <lang>-orig explicitly for ASR captions. The -orig key is added when the ASR track is translatable. If it is missing, fall back to the plain code only after confirming the URL has no tlang parameter.
    • Treat any requested track whose URL contains tlang= as a translation and discard it.
    • --extractor-args "youtube:skip=translated_subs" stops yt-dlp from building translations of manual tracks (the xx-yy keys) and saves extraction time. In the current source it does not remove the plain-code translations of the ASR track, so the two rules above still apply.

    Subtitle formats: --sub-format, --convert-subs, --embed-subs

    For YouTube, the extractor currently offers every track in seven formats: json3, srv1, srv2, srv3, ttml, srt and vtt. --sub-format takes a /-separated preference list. best means the last format the extractor lists, which for YouTube is vtt. If nothing matches, yt-dlp warns and falls back to best.

    • vtt: the default and the easiest to parse. Auto-captions include per-word start times as inline tags.
    • json3, srv3: YouTube's own timed-text formats, useful for word-level timing (below). YouTube doesn't document them.
    • --convert-subs FORMAT converts to ass, lrc, srt or vtt through ffmpeg. TTML is converted to SRT first, with a warning that styling is lost. There's no converter for json3 or srv*, so keep those raw and parse them yourself.
    • --embed-subs muxes tracks into mp4, webm or mkv output. Used alone, it deletes the sidecar files after embedding. Add --write-subs to keep them. It does nothing on a --skip-download run, and for datasets sidecar files are easier to process than embedded streams.

    Captions-only runs at scale

    Caption files are kilobytes, so a captions-plus-metadata pass is a cheap way to decide which videos are worth downloading audio for:

    yt-dlp -a ids.txt --skip-download \
      --write-subs --write-auto-subs --sub-langs "en,en-orig" --sub-format vtt \
      --extractor-args "youtube:skip=translated_subs" \
      --write-info-json \
      --sleep-requests 1 --sleep-subtitles 2 \
      --download-archive subs_archive.txt --force-write-archive \
      -o "meta/%(id)s.%(ext)s" \
      -o "subtitle:subs/%(id)s.%(ext)s"
    • Output templates. The subtitle: prefix sets a separate template for subtitle files, and yt-dlp appends the language and extension: subs/VIDEO_ID.en-orig.vtt. -P "subtitle:subs" sets only the directory. The .info.json lands in meta/ (there's an infojson: prefix too).
    • Archive gotcha. With --skip-download, yt-dlp only writes archive entries if --force-write-archive is set. Without it, every re-run starts from zero.
    • Rate limits. --sleep-subtitles sleeps before each subtitle download and --sleep-requests between extraction requests. --sleep-interval applies to media downloads, not subtitle files. The -t sleep preset expands to --sleep-subtitles 5 --sleep-requests 0.75 --sleep-interval 10 --max-sleep-interval 20.
    • Missing tracks. If YouTube requires a PO token for subtitle requests and none is available, the extractor skips subtitles for that client and warns "There are missing subtitles languages because a PO token was not provided." The README documents a subs context for the po_token extractor argument. Current yt-dlp also needs a JavaScript runtime and yt-dlp-ejs for full YouTube support.
    • Bot checks and IPs. At volume, subtitle and metadata requests hit the same anti-bot systems as video. See fixing "Sign in to confirm you're not a bot" and the yt-dlp proxy guide.

    Python API equivalents

    The CLI flags map to YoutubeDL params (the README points to devscripts/cli_to_api.py for translating any command line):

    CLI Python param
    --write-subs writesubtitles=True
    --write-auto-subs writeautomaticsub=True
    --sub-langs "en,en-orig" subtitleslangs=["en", "en-orig"]
    --sub-format vtt subtitlesformat="vtt"
    --list-subs listsubtitles=True
    --skip-download skip_download=True
    --sleep-subtitles 2 sleep_interval_subtitles=2
    --sleep-requests 1 sleep_interval_requests=1
    --force-write-archive force_write_download_archive=True
    --convert-subs srt postprocessors=[{"key": "FFmpegSubtitlesConvertor", "format": "srt", "when": "before_dl"}]
    --embed-subs postprocessors=[{"key": "FFmpegEmbedSubtitle"}]

    This script fetches captions only and writes a manifest line per track labeled manual, auto or auto-translated, using the precedence and tlang rules above:

    import json
    import sys
    
    import yt_dlp
    
    LANG = "en"
    OPTS = {
        "skip_download": True,                       # --skip-download
        "writesubtitles": True,                      # --write-subs
        "writeautomaticsub": True,                   # --write-auto-subs
        "subtitleslangs": [LANG, f"{LANG}-orig"],    # --sub-langs "en,en-orig"
        "subtitlesformat": "vtt",                    # --sub-format vtt
        "writeinfojson": True,                       # --write-info-json
        "outtmpl": {"default": "subs/%(id)s/%(id)s.%(ext)s"},
        "extractor_args": {"youtube": {"skip": ["translated_subs"]}},
        "sleep_interval_requests": 1,                # --sleep-requests 1
        "sleep_interval_subtitles": 2,               # --sleep-subtitles 2
        "ignoreerrors": True,                        # log failures, keep going
        # "proxy": "http://USER:[email protected]:8000",
    }
    
    
    def track_kind(info, lang, sub):
        if lang in (info.get("subtitles") or {}):
            return "manual"            # manual wins when both flags are set
        if "tlang=" in (sub.get("url") or ""):
            return "auto-translated"   # machine translation: drop for training
        return "auto"
    
    
    def main(ids_path, manifest_path):
        with open(ids_path, encoding="utf-8") as f:
            ids = [line.strip() for line in f if line.strip()]
        with yt_dlp.YoutubeDL(OPTS) as ydl, \
                open(manifest_path, "a", encoding="utf-8") as out:
            for vid in ids:
                info = ydl.extract_info(f"https://www.youtube.com/watch?v={vid}")
                if not info:
                    continue  # extraction failed; already reported
                for lang, sub in (info.get("requested_subtitles") or {}).items():
                    out.write(json.dumps({
                        "id": vid,
                        "lang": lang,
                        "kind": track_kind(info, lang, sub),
                        "file": f"subs/{vid}/{vid}.{lang}.{sub['ext']}",
                    }) + "\n")
    
    
    if __name__ == "__main__":
        main(*sys.argv[1:3])

    Run it as python fetch_captions.py ids.txt captions_manifest.jsonl. On an English video with uploaded captions you get en (manual) and en-orig (auto). On a Spanish video without English captions, en is labeled auto-translated and en-orig doesn't exist.

    Cleaning rolling auto-caption VTT

    YouTube's auto-caption VTT is built for a two-line karaoke display, not for parsing. A typical stretch looks like this (abridged):

    00:00:00.160 --> 00:00:02.389 align:start position:0%
     
    in<00:00:00.480><c> this</c><00:00:00.800><c> video</c>
    
    00:00:02.389 --> 00:00:02.399 align:start position:0%
    in this video
     
    
    00:00:02.399 --> 00:00:04.950 align:start position:0%
    in this video
    going<00:00:02.720><c> to</c><00:00:03.040><c> talk</c>

    Each cue repeats the previous line on top, and 10 ms "hold" cues repeat it again. (The seemingly blank line after the first timing line is a single space, so it doesn't end the cue.) Naively concatenating the cue text triples most words. The fix: strip tags, then drop the longest run of lines at the start of each cue that equals the end of the previous cue. Manual tracks don't have this layout, so the script detects auto-captions by their inline word timestamps and only merges identical back-to-back cues for manual tracks.

    #!/usr/bin/env python3
    """YouTube WebVTT -> segment-level JSONL ({"start", "end", "text"}), stdlib only.
    
    Auto-generated tracks use a "rolling" layout: each cue repeats the previous
    line above the new words, and ~10 ms cues re-show text that was already
    emitted. Those repeats are removed. Manual tracks are passed through, with
    identical back-to-back cues merged.
    
    usage: python vtt_to_jsonl.py VIDEO.en-orig.vtt > VIDEO.en-orig.jsonl
    """
    import html
    import json
    import re
    import sys
    
    TS = r"(?:(\d+):)?(\d{2}):(\d{2})\.(\d{3})"
    TIMING = re.compile(rf"^{TS}\s+-->\s+{TS}")
    WORD_TS = re.compile(r"<(?:\d+:)?\d{2}:\d{2}\.\d{3}>")  # only in auto captions
    TAG = re.compile(r"<[^>]*>")             # <c>, </c>, <00:00:01.234>, <i>, ...
    NON_SPEECH = re.compile(r"\[[^\]]*\]")   # [Music], [Applause]; None to keep them
    
    
    def seconds(h, m, s, ms):
        return int(h or 0) * 3600 + int(m) * 60 + int(s) + int(ms) / 1000
    
    
    def clean(line):
        text = html.unescape(TAG.sub("", line))
        if NON_SPEECH:
            text = NON_SPEECH.sub("", text)
        return " ".join(text.split())
    
    
    def parse_cues(lines):
        """Yield (start, end, [text lines]). A cue ends at an empty line."""
        block = []
        for line in lines + [""]:
            if line:
                block.append(line)
                continue
            for i, row in enumerate(block):  # header/NOTE/STYLE blocks never match
                m = TIMING.match(row)
                if m:
                    g = m.groups()
                    text = [clean(t) for t in block[i + 1:]]
                    yield seconds(*g[:4]), seconds(*g[4:]), [t for t in text if t]
                    break
            block = []
    
    
    def overlap(prev, cur):
        """Length of the longest suffix of prev that is also a prefix of cur."""
        for k in range(min(len(prev), len(cur)), 0, -1):
            if prev[-k:] == cur[:k]:
                return k
        return 0
    
    
    def segments(path):
        with open(path, encoding="utf-8-sig") as f:
            raw = f.read()
        rolling = bool(WORD_TS.search(raw))
        out, prev = [], []
        for start, end, lines in parse_cues(raw.splitlines()):
            if not lines:
                continue
            new = lines[overlap(prev, lines):] if rolling else lines
            prev = lines
            if not new:
                continue  # cue only repeats text that was already emitted
            text = " ".join(new)
            last = out[-1] if out else None
            if last and last["text"] == text and start - last["end"] < 0.05:
                last["end"] = round(max(last["end"], end), 3)  # same caption held
                continue
            out.append({"start": round(start, 3), "end": round(end, 3), "text": text})
        return out
    
    
    if __name__ == "__main__":
        if len(sys.argv) != 2:
            sys.exit("usage: vtt_to_jsonl.py FILE.vtt > FILE.jsonl")
        for seg in segments(sys.argv[1]):
            print(json.dumps(seg, ensure_ascii=False))

    On the excerpt above it emits one segment per spoken line, with no repeats:

    {"start": 0.16, "end": 2.389, "text": "in this video"}
    {"start": 2.399, "end": 4.95, "text": "going to talk"}

    Notes: NON_SPEECH strips bracketed tags like [Music]. Set it to None if you want them as non-speech labels. Auto-captions may also contain >> speaker-change markers, which you can split on or strip. Segment boundaries follow YouTube's display lines, not sentences, so re-segment by pauses or punctuation if your model needs utterance-level samples.

    Word-level timing: json3 and srv3

    For forced-alignment checks or word-level supervision, request the native formats with --sub-format json3 or srv3 (they can't be converted, so request them directly). YouTube doesn't publish either schema, so treat what follows as observed structure and validate it on a sample:

    • json3: a top-level events list. Each event has tStartMs, dDurationMs and a segs list of text pieces in utf8. On auto-caption tracks, words are separate segs carrying an offset from the event start (tOffsetMs). Open-source parsers such as this one read the same fields.
    • srv3: XML with <p t="…" d="…"> lines (milliseconds) and <s> spans. See this reverse-engineered format description.
    import json
    
    def json3_words(path):
        with open(path, encoding="utf-8") as f:
            events = json.load(f).get("events", [])
        for ev in events:
            t0 = ev.get("tStartMs", 0)
            for seg in ev.get("segs") or []:
                word = seg.get("utf8", "").strip()
                if word:
                    yield (t0 + seg.get("tOffsetMs", 0)) / 1000, word

    These are word start times only. Derive ends from the next word's start, capped by the event's end, and expect them to be approximate.

    Which flags for which need

    Need Flags
    See what tracks exist --list-subs
    Captions only, no media --skip-download + subtitle flags
    Creator-uploaded English, any region --write-subs --sub-langs "en.*"
    Original-language ASR only --write-auto-subs --sub-langs "en-orig"
    Manual if present, plus original ASR --write-subs --write-auto-subs --sub-langs "en,en-orig", then check info.json
    All manual tracks except chat replay --write-subs --sub-langs "all,-live_chat"
    Skip translations of manual tracks --extractor-args "youtube:skip=translated_subs"
    Word-level timing --sub-format json3 (or srv3)
    SRT files --sub-format srt or --sub-format vtt --convert-subs srt
    Subtitles inside the video file --embed-subs (add --write-subs to keep sidecars)
    Separate subtitle folder -o "subtitle:subs/%(id)s.%(ext)s" or -P "subtitle:subs"
    Throttle subtitle requests --sleep-subtitles N --sleep-requests N or -t sleep
    Resumable captions-only runs --download-archive FILE --force-write-archive

    Captions as ASR labels vs re-transcribing with Whisper

    Neither caption type is a clean transcript. Manual captions are often edited for reading: they condense speech, drop disfluencies, add speaker names and sound tags, and can be offset in time. Auto-captions are another ASR system's output, so training on them distills its errors. They can also lack punctuation and casing. A practical split:

    • Use caption availability and language as cheap filters before you download any audio (pair with the audio-only download guide).
    • Use manual captions as labels only after alignment and filtering, e.g. drop segments where your own ASR disagrees beyond a WER threshold you calibrate per language.
    • Where label quality matters, re-transcribe with Whisper and keep the captions as a cross-check. Disagreement between the two is a useful quality signal.

    The Whisper side (model choice, language ID, per-language thresholds) is covered in building a YouTube video dataset for multimodal training. Whatever you choose, store the track kind (manual/auto) and language with every segment so you can re-weight or drop sources later.

    Compliance

    YouTube's Terms of Service restrict automated access and downloading, and captions are content like any other. Copyright and licensing apply: some teams restrict collection to Creative Commons videos (see our guide to Creative Commons YouTube data) and record the license per sample. Voices and transcribed speech can be personal data under regimes like GDPR. Honor takedown and opt-out requests, and keep the lineage you need to do so. Get legal review for your jurisdiction and use case.

    If you'd rather not run this yourself

    Corpora offers dedicated 1–40 Gbps unmetered proxy capacity for teams running their own yt-dlp stack. It also offers a fully managed option that delivers video, audio, subtitles and metadata as JSONL/Parquet with sha256 checksums and manifests to your S3, GCS or Azure bucket, starting with a free sample batch. Contact us to scope it.

    FAQ

    How do I download only subtitles with yt-dlp, without the video?

    Add --skip-download to your subtitle flags, e.g. yt-dlp --skip-download --write-subs --write-auto-subs --sub-langs "en,en-orig" URL. Add --write-info-json if you need to tell manual and auto tracks apart later.

    What is the difference between --write-subs and --write-auto-subs?

    --write-subs downloads tracks the creator uploaded (the subtitles key). --write-auto-subs downloads YouTube's automatic captions and their machine translations (the automatic_captions key). With both, a manual track takes precedence for the same language code.

    What does en-orig mean in yt-dlp?

    It's yt-dlp's label for the auto-generated caption track in the video's original spoken language. Other plain codes in the automatic list are usually machine translations of that track.

    Why does my auto-generated VTT repeat every line?

    YouTube's auto-caption VTT is a rolling two-line display: each cue repeats the previous line and short hold cues repeat it again. Strip the inline tags and remove the overlap between consecutive cues, as the script above does.

    Can yt-dlp give word-level timestamps from YouTube captions?

    For auto-captions, yes, indirectly. Download json3 or srv3 (or read the inline timestamps in VTT) and parse the per-word offsets yourself. Manual tracks are usually timed per line only.