Tag: ASR

  • Whisper Transcription at Scale for Speech Datasets

    Running Whisper on one podcast is a one-liner. Running it over 50,000 hours of mixed YouTube audio is a different job: silence and music make the model invent text, long files fall into repetition loops, language labels are wrong, and nobody knows how many GPU-hours the run will take. This guide covers Whisper transcription at dataset scale: model choice, faster-whisper and WhisperX, VAD, failure modes, throughput math, output formats and quality checks, with APIs checked against the current repositories and model cards.

    TL;DR

    • Model: large-v3 for best accuracy, large-v3-turbo (809M parameters, 4 decoder layers instead of 32) when throughput matters more. Measure the WER difference on your own audio.
    • Runtime: faster-whisper's BatchedInferencePipeline for bulk transcription; WhisperX when you need forced-aligned word timestamps or speaker labels.
    • Always run VAD first. Hallucinations cluster in non-speech audio, and VAD also tells you how many hours are actually speech.
    • Keep every segment's avg_logprob, compression_ratio and no_speech_prob and filter afterwards, especially in batched mode, which does not re-decode failed chunks.
    • Plan capacity with a measured RTFx (audio seconds per wall-clock second) from a pilot, not with someone else's benchmark.
    • Validate against manual YouTube subtitles with jiwer, treating the score as an agreement signal, not true WER.

    Choosing a Whisper model

    The openai/whisper README lists six sizes. VRAM and relative speed below are OpenAI's own approximate figures (English, A100); real-world speed varies with language, speaking rate and hardware.

    Size Parameters English-only Required VRAM (approx.) Relative speed (approx.)
    tiny 39 M tiny.en ~1 GB ~10x
    base 74 M base.en ~1 GB ~7x
    small 244 M small.en ~2 GB ~4x
    medium 769 M medium.en ~5 GB ~2x
    large (v3) 1550 M none ~10 GB 1x
    turbo (large-v3-turbo) 809 M none ~6 GB ~8x

    Details that matter for a dataset run:

    • large now means large-v3. Both openai-whisper and faster-whisper map the alias large to the large-v3 checkpoint. Pin the explicit name in your manifests so a rerun is reproducible.
    • large-v3 uses 128 Mel bins instead of 80 and was trained on 1 million hours of weakly labeled audio plus 4 million hours pseudo-labeled with large-v2, according to its model card.
    • large-v3-turbo is, per its model card, a fine-tuned, pruned large-v3 with the decoder reduced from 32 to 4 layers, "at the expense of a minor quality degradation". The README notes it is not trained for translation, which doesn't matter for same-language transcription.
    • Accuracy varies widely by language. OpenAI's per-language WER/CER breakdown is in the README and the Whisper paper. For low-resource languages, run a pilot before committing GPU budget.

    Choosing a runtime

    Runtime What it adds Use it for
    openai-whisper Reference implementation, CLI with txt/vtt/srt/tsv/json output Small jobs, reproducing reference behavior
    faster-whisper CTranslate2 backend, int8/fp16, built-in Silero VAD, batched pipeline Bulk transcription
    WhisperX faster-whisper backend plus wav2vec2 forced alignment and pyannote diarization Precise word timing, speaker labels

    The faster-whisper README claims up to 4x the speed of openai/whisper at the same accuracy with less memory, and publishes a benchmark: 13 minutes of audio with large-v2 on an RTX 3070 Ti took 1m03s unbatched and 17s with batch_size=8 (fp16). That's roughly 46x real time for that one file on that GPU. Treat it as an order of magnitude, not a plan.

    faster-whisper basics

    The model constructor takes device ("cuda", "cpu" or "auto") and compute_type ("float16", "int8_float16", "int8" and others). transcribe() returns a generator of segments plus a TranscriptionInfo; nothing is decoded until you iterate:

    from faster_whisper import WhisperModel
    
    model = WhisperModel("large-v3", device="cuda", compute_type="float16")
    segments, info = model.transcribe(
        "audio.opus",
        vad_filter=True,
        word_timestamps=True,
        condition_on_previous_text=False,
    )
    for s in segments:  # decoding happens here
        print(f"[{s.start:.2f} -> {s.end:.2f}] {s.text}")

    Audio is decoded with PyAV, so faster-whisper doesn't need a system ffmpeg. Feed it the native Opus or M4A you downloaded; see yt-dlp audio-only downloads for speech datasets for getting those files without re-encoding.

    VAD first: fewer hallucinations, honest hours

    Whisper's model card states the problem directly: because it is trained on large-scale noisy data, "the predictions may include texts that are not actually spoken in the audio input (i.e. hallucination)". The Careless Whisper study found that roughly 1% of the transcriptions it examined contained entire hallucinated phrases or sentences, and that hallucinations occurred disproportionately for speakers with longer non-vocal stretches. Silence, music beds and applause are exactly where this happens.

    Voice-activity detection removes those stretches before the decoder sees them. The WhisperX paper also reports that VAD-based pre-segmentation improves long-form transcription quality and enables batching. In faster-whisper:

    • Sequential WhisperModel.transcribe: vad_filter=False by default. When enabled, the default is conservative and only removes silence longer than 2 seconds (min_silence_duration_ms=2000). Tune it with vad_parameters=dict(min_silence_duration_ms=500).
    • BatchedInferencePipeline.transcribe: vad_filter=True by default, with 160 ms minimum silence and speech chunks capped at the 30-second window. Disable VAD on a file longer than 30 seconds without passing clip_timestamps and it raises an error.

    A bonus: info.duration_after_vad is a free speech-duration measurement. Store it next to info.duration; it's the number behind "hours" discussed in what makes speech training data ASR- and TTS-ready. VAD reduces hallucination on non-speech; it does not remove it, and it won't help with speech over loud music.

    Known failure modes and the knobs that address them

    Failure What it looks like Mitigation
    Hallucination on non-speech Fluent sentences over silence, music or noise VAD; no_speech_prob filtering; hallucination_silence_threshold (requires word timestamps, sequential mode)
    Repetition loops The same phrase repeated dozens of times condition_on_previous_text=False; compression-ratio checks; n-gram repetition filter
    Drift and bad timestamps Segment times seconds off on long files VAD segmentation; WhisperX forced alignment
    Wrong language English "transcript" of Spanish audio, or translated text Language-probability threshold, then re-run with a fixed language

    The model card notes that the sequence-to-sequence architecture "makes it prone to generating repetitive texts, which can be mitigated to some degree by beam search and temperature scheduling but not perfectly". faster-whisper's docstring for condition_on_previous_text spells out the trade-off: disabling it may make text less consistent across windows, "but the model becomes less prone to getting stuck in a failure loop, such as repetition looping or timestamps going out of sync". WhisperX sets it to False by default.

    The sequential decoder uses temperature fallback: if a window's gzip compression_ratio exceeds 2.4 or its avg_logprob is below -1.0, it is re-decoded at a higher temperature. A window counts as silent when no_speech_prob exceeds 0.6 and the log-probability is below the threshold. These are the same defaults in openai-whisper and faster-whisper.

    One thing that is easy to miss when reading the source: the batched pipeline decodes each chunk once, at the first temperature only, so the fallback never re-decodes a bad chunk. The metrics are still returned per segment. That makes post-hoc filtering mandatory, not optional, for batched runs.

    A cheap extra guard: count word trigrams per segment and flag any segment where one trigram repeats several times. It catches loops that a compression-ratio threshold alone misses on short segments.

    Language detection

    Without a language argument, Whisper detects the language from the first 30-second window (faster-whisper exposes language_detection_segments to use more windows, and runs detection on VAD-filtered audio in batched mode). info.language, info.language_probability and info.all_language_probs come back with the result.

    A workable pattern for large collections:

    1. Run detection, keep the probability in the manifest.
    2. Accept files above a confidence threshold you calibrate on a hand-checked sample.
    3. Re-run low-confidence files with an explicit language, or route them to review. For genuinely mixed-language audio, multilingual=True runs detection on every segment.

    Don't trust video metadata for this; title language and spoken language often differ.

    A batched worker that writes JSONL

    This worker reads a list of audio paths, writes one segments file per input, appends a per-file row to a manifest, and skips finished files on restart. Run one process per GPU (set CUDA_VISIBLE_DEVICES) and shard the file list across processes.

    # transcribe_worker.py
    # usage: python transcribe_worker.py files.txt out_dir
    import json
    import os
    import sys
    import time
    from pathlib import Path
    
    from faster_whisper import BatchedInferencePipeline, WhisperModel
    
    MODEL = "large-v3"           # or "large-v3-turbo"
    COMPUTE_TYPE = "float16"     # "int8_float16" to save VRAM
    BATCH_SIZE = 16              # lower it if you hit out-of-memory
    
    model = WhisperModel(MODEL, device="cuda", compute_type=COMPUTE_TYPE)
    pipe = BatchedInferencePipeline(model=model)
    
    
    def r(x):
        return round(float(x), 3)
    
    
    def transcribe(audio_path: Path, out_dir: Path) -> dict:
        out_path = out_dir / f"{audio_path.stem}.segments.jsonl"
        tmp_path = out_dir / f"{audio_path.stem}.segments.tmp"
        t0 = time.monotonic()
        segments, info = pipe.transcribe(
            str(audio_path),
            batch_size=BATCH_SIZE,
            vad_filter=True,
            word_timestamps=True,
        )
        n = 0
        with tmp_path.open("w", encoding="utf-8") as f:
            for s in segments:
                row = {
                    "audio_id": audio_path.stem,
                    "seg": n,
                    "start": r(s.start),
                    "end": r(s.end),
                    "text": s.text.strip(),
                    "avg_logprob": r(s.avg_logprob),
                    "compression_ratio": r(s.compression_ratio),
                    "no_speech_prob": r(s.no_speech_prob),
                    "words": [
                        {"w": w.word, "start": r(w.start), "end": r(w.end), "p": r(w.probability)}
                        for w in (s.words or [])
                    ],
                }
                f.write(json.dumps(row, ensure_ascii=False) + "\n")
                n += 1
        os.replace(tmp_path, out_path)  # only complete files get the final name
        wall = time.monotonic() - t0
        return {
            "audio_id": audio_path.stem,
            "duration_s": r(info.duration),
            "speech_s": r(info.duration_after_vad),
            "language": info.language,
            "language_probability": r(info.language_probability),
            "n_segments": n,
            "model": MODEL,
            "compute_type": COMPUTE_TYPE,
            "wall_s": r(wall),
            "rtfx": r(info.duration / wall) if wall > 0 else None,
        }
    
    
    if __name__ == "__main__":
        file_list, out_dir = Path(sys.argv[1]), Path(sys.argv[2])
        out_dir.mkdir(parents=True, exist_ok=True)
        with (out_dir / "manifest.jsonl").open("a", encoding="utf-8") as manifest:
            for line in file_list.read_text(encoding="utf-8").splitlines():
                if not line.strip():
                    continue
                audio = Path(line.strip())
                if (out_dir / f"{audio.stem}.segments.jsonl").exists():
                    continue
                try:
                    row = transcribe(audio, out_dir)
                except Exception as e:  # log and keep the worker alive
                    row = {"audio_id": audio.stem, "error": repr(e)}
                manifest.write(json.dumps(row, ensure_ascii=False) + "\n")
                manifest.flush()

    JSONL per file is easy to append, resume and grep. For training, compact it into Parquet or WebDataset shards later, as described in storing datasets as WebDataset and Parquet shards. Record the model name, compute type and library versions in the manifest; a transcript without that provenance can't be regenerated or compared.

    WhisperX: aligned words and speaker labels

    Whisper's own word timestamps come from cross-attention and dynamic time warping. WhisperX adds a second pass that force-aligns the transcript to a language-specific wav2vec2 phoneme model, then optionally assigns speakers using pyannote. From the WhisperX README:

    import whisperx
    from whisperx.diarize import DiarizationPipeline
    
    device = "cuda"
    audio = whisperx.load_audio("audio.opus")
    
    model = whisperx.load_model("large-v3", device, compute_type="float16")
    result = model.transcribe(audio, batch_size=16)
    
    model_a, metadata = whisperx.load_align_model(language_code=result["language"], device=device)
    result = whisperx.align(result["segments"], model_a, metadata, audio, device,
                            return_char_alignments=False)
    
    diarize_model = DiarizationPipeline(token="HF_TOKEN", device=device)
    diarize_segments = diarize_model(audio)
    result = whisperx.assign_word_speakers(diarize_segments, result)

    Caveats from the README itself: diarization needs a Hugging Face token and acceptance of the pyannote speaker-diarization-community-1 user agreement; words without characters in the alignment model's dictionary (for example "2014." or "£13.60") get no timing; overlapping speech is handled poorly; "diarization is far from perfect"; and alignment needs a wav2vec2 model for each language. If you only need segment-level text, the alignment and diarization passes are extra GPU time you can skip.

    Throughput math

    Use RTFx = audio seconds processed / wall-clock seconds, measured per GPU on your audio, model, batch size and compute type. Then:

    • GPU-hours = audio_hours / RTFx
    • wall-clock hours = audio_hours / (RTFx × number_of_GPUs)

    Which "audio hours" you plug in depends on whether RTFx was measured against file duration or post-VAD duration. The worker above measures against file duration, so VAD savings are already included.

    Worked example with assumed numbers, not measurements: 100,000 file hours, a pilot RTFx of 40 per GPU.

    GPUs GPU-hours Wall-clock
    8 100,000 / 40 = 2,500 2,500 / 8 = 312.5 h (~13 days)
    32 2,500 2,500 / 32 ≈ 78 h (~3.3 days)

    If a turbo pilot measures RTFx 100 on the same data, GPU-hours drop to 1,000. Whether that's worth the accuracy cost is decided by the WER check below, not by the speed table.

    For a trustworthy RTFx, run a pilot of a few dozen hours stratified like the full collection (lengths, languages, speech ratio). Throughput depends on speech density and words spoken, so a pilot of clean audiobooks overstates speed on talk shows. Watch non-GPU bottlenecks too: CPU audio decoding, network storage reads, and per-file overhead on very short files.

    Quality checks: WER against manual subtitles

    When a video has creator-uploaded subtitles, you have a free reference. Download them with yt-dlp's --write-subs and keep the info.json so you can tell manual from automatic tracks; our subtitles guide covers the details. Don't use YouTube's auto-captions as a reference: they are another ASR system's output.

    Normalize both sides the same way, then score with jiwer. The normalizers ship with openai-whisper (pip install openai-whisper jiwer):

    import jiwer
    from whisper.normalizers import EnglishTextNormalizer  # BasicTextNormalizer for other languages
    
    normalize = EnglishTextNormalizer()
    
    def agreement(reference_text: str, hypothesis_text: str) -> dict:
        ref = normalize(reference_text)
        hyp = normalize(hypothesis_text)
        out = jiwer.process_words(ref, hyp)
        return {
            "wer": out.wer,
            "substitutions": out.substitutions,
            "deletions": out.deletions,
            "insertions": out.insertions,
        }

    Concatenate per file rather than matching subtitle cues to segments one by one; cue boundaries and segment boundaries don't line up. Since jiwer 4.0, an empty reference is allowed and every hypothesis word counts as an insertion, which is useful for testing hallucination on known-silent clips.

    Interpret the number carefully. Manual subtitles are frequently not verbatim: fillers and false starts are dropped, sentences are condensed, and some tracks are translations. A high score can mean bad ASR or a loose reference. Use it to:

    • Compare models and settings on the same files (large-v3 vs turbo, int8 vs fp16, VAD settings).
    • Find outliers: files with very high disagreement usually have wrong language, music, or a hallucination loop.
    • Calibrate filters for files that have no subtitles, using the segment metrics.

    Combine it with cheap per-segment checks that need no reference: compression_ratio above 2.4, no_speech_prob high with text present, words per second outside a plausible range, repeated n-grams, and segments whose word timestamps all collapse to one point. Then hand-audit a random sample of what passes, not only what fails.

    Compliance

    If the audio comes from YouTube, its Terms of Service restrict automated access and downloading. Copyright and licensing apply to the recordings and arguably to transcripts derived from them; some teams restrict collection to Creative Commons material. Voices and transcribed speech can be personal data under GDPR and similar laws, and diarization creates per-speaker records. Keep provenance per file, honor takedown and opt-out requests, and get legal review for your jurisdiction and use case.

    If you'd rather not run the collection side yourself

    Corpora's managed service delivers YouTube audio, subtitles (including manual tracks, where they exist, for use as references) and metadata as JSONL/Parquet with sha256 checksums and manifests to S3, GCS or Azure Blob, starting with a free sample batch. Teams that run their own yt-dlp stack can use dedicated unmetered proxy capacity instead. Transcription stays on your GPUs either way.

    FAQ

    Is faster-whisper as accurate as openai-whisper?

    It runs the same Whisper weights converted to CTranslate2, and its README claims the same accuracy at higher speed. Defaults differ (faster-whisper uses beam size 5, openai-whisper's transcribe uses 1), and int8 quantization or batched mode can shift results, so compare on a sample with identical settings.

    Should I use large-v3 or large-v3-turbo?

    Turbo has 809M parameters and a 4-layer decoder, and OpenAI describes the quality loss as minor. For high-resource languages and clean speech it is often the better trade; for low-resource languages, measure both on a labeled sample before deciding.

    How do I stop Whisper from hallucinating on silence?

    Run VAD before decoding (vad_filter=True in faster-whisper), disable condition_on_previous_text, and filter segments on no_speech_prob, compression_ratio and repetition afterwards. With word timestamps in sequential mode, hallucination_silence_threshold skips long silences when a hallucination is suspected. None of these remove the problem completely.

    How long will it take to transcribe 10,000 hours?

    Wall-clock hours = 10,000 / (RTFx × GPUs). Measure RTFx on a representative pilot with your exact model, batch size and compute type; at an assumed RTFx of 40, that's 250 GPU-hours.

  • Speech Training Data: What Makes Audio ASR- and TTS-Ready

    "Ten thousand hours of English speech" can mean ten thousand hours of clean two-person conversation, or ten thousand hours of video files in which speech fills half the time under a music bed. For ASR, TTS and speech-LLM training those are different datasets. This article lists the properties that decide whether audio is actually usable, how each one is measured automatically, and which thresholds matter for which kind of model.

    TL;DR

    • Count speech hours, not file hours. Voice-activity detection turns "hours of video" into the number that matters.
    • Verify language from the audio, not from titles or channel country.
    • ASR wants diversity; TTS wants cleanliness. The same source can be excellent for one and useless for the other.
    • Background music is the most common silent killer — and the easiest to miss by reading metadata.
    • Store measurements per file, so each model can apply its own thresholds.

    The properties that matter

    Property How it is measured Why it matters
    Speech ratio Voice-activity detection (VAD): fraction of time with speech Converts file hours into speech hours; low values mean music, silence or ambience
    Spoken language Audio language identification on speech segments, with a confidence Titles and channel metadata are often wrong or mixed-language
    Speaker count Speaker diarization or speaker-embedding clustering Conversation models need ≥2 speakers; TTS usually needs exactly one per segment
    Music and noise Audio-event classifiers (AudioSet-style labels) per window Background music under speech hurts TTS badly and ASR noticeably
    Audio quality Non-intrusive quality estimators (MOS predictors), signal-to-noise estimates, bandwidth/clipping checks Low-bitrate, clipped or band-limited audio caps achievable quality
    Overlap Diarization overlap regions Overlapping speech is realistic for ASR, but must be removed for TTS
    Transcript availability Manual vs auto-generated captions; or your own ASR Decides whether you need a transcription pass and how trustworthy labels are

    Speech ratio: the number behind "hours"

    A podcast may be 95% speech. A vlog may be 40%. A music video with a spoken intro may be 5%. If you buy or collect by file duration, your effective speech hours can be half of what you paid for. Run a VAD (silero-vad and WebRTC VAD are common, lightweight choices that run fine on CPU) and report both numbers in every manifest: duration_s and speech_s.

    Speech ratio is also a cheap first filter. Requiring ≥0.7–0.8 removes most music, gameplay and ambience content before any expensive analysis.

    Language: trust the audio

    Video metadata language is optional, frequently wrong, and describes the title rather than the speech. Bilingual channels, dubbed audio tracks and code-switching speakers are common. Run language identification on the detected speech segments and keep the confidence. A threshold such as ≥0.9 on the dominant language is a reasonable starting point; mixed-language content can be kept separately if your model benefits from code-switching data.

    If your source offers several audio tracks per video, select the original-language track explicitly rather than whatever is default.

    Speakers: count them, don't identify them

    You rarely need to know who is speaking, only how many and when:

    • Conversational ASR and speech-LLMs benefit from two or more speakers with natural turn-taking.
    • TTS and voice cloning need segments with exactly one speaker, consistent voice and no overlap.
    • Speaker diversity across the dataset matters for generalization: cap the hours per channel so a few prolific creators don't dominate.

    Full diarization pipelines (segmentation plus speaker embeddings plus clustering) give the best counts. Lighter approaches that cluster speaker embeddings on short windows are cheaper but tend to undercount, merging similar voices into one speaker. Sample several positions of long recordings instead of one window, and count only speakers with a meaningful share of speech so a single interjection doesn't turn a monologue into a "conversation".

    Background music: the silent killer

    Commentary, vlogs, tutorials and drama content very often run music under speech. It is invisible in metadata and only partly visible in auto-generated captions. Audio-event classifiers trained on AudioSet-style labels give a per-window music probability; two derived ratios are useful:

    • Music ratio: share of windows with music present, including music under speech.
    • Pure-music ratio: share of windows with music and no speech — intros, interludes, songs.

    For TTS, require music ratio near zero. For ASR, a low level of background music can be acceptable or even useful for robustness, but keep it measured and controlled rather than accidental.

    Quality: what you can't fix later

    Platform audio is lossy (typically Opus or AAC at moderate bitrates), which is fine for ASR and acceptable for many TTS uses, but source quality varies far more than codec quality. Check for:

    • Effective bandwidth (audio upsampled from a phone call sounds full-band in the container but isn't).
    • Clipping and heavy compression, common in livestream recordings.
    • Reverberant rooms and distant microphones.
    • Predicted MOS from a non-intrusive quality model, as a single sortable score.

    Keep native audio as delivered, and convert to your training format (for example 16 kHz mono) as a separate step. Re-encoding to another lossy format at collection time only loses quality.

    ASR versus TTS: different thresholds

    Property ASR / speech-LLM TTS / voice
    Speech ratio ≥0.7–0.8 ≥0.8, then segment to speech only
    Speakers Diverse; multi-speaker welcome One per segment, no overlap
    Background music Low, controlled None
    Noise and reverberation Some variety helps robustness As clean as possible
    Speaking style Spontaneous and read, many accents Consistent, expressive, well articulated
    Transcripts Needed; manual captions or high-quality ASR Needed and must be exact, including punctuation

    Scale expectations differ too. OpenAI's Whisper was trained on about 680,000 hours of weakly supervised audio, which shows how far quantity and diversity can carry ASR. TTS systems are usually trained on far fewer, far cleaner hours, and quality filters matter more than volume.

    Labels: captions, ASR and segmentation

    Manual captions are the best free labels, but cover only a minority of videos; auto-generated captions are useful but contain recognition errors and rolling duplicated lines. Many pipelines re-transcribe with a strong ASR model and keep platform captions as a cross-check: large disagreement flags bad audio or wrong language. Segment long recordings at VAD boundaries into utterances (typically a few seconds to ~30 seconds) and keep segment timestamps in the manifest. For the caption download side see downloading subtitles and auto-captions.

    Measure on a sample before you collect everything

    All of the measurements above can run on short segments of audio rather than whole files. Probing a minute or so at several positions is enough to reject most unsuitable videos before downloading them, which is the approach behind Corpora Curation's audio probe. Store every measurement per file:

    {
      "video_id": "abc123XYZ00",
      "duration_s": 1843,
      "speech_ratio": 0.93,
      "lang": "en",
      "lang_prob": 0.98,
      "n_speakers": 2,
      "music_ratio": 0.02,
      "pure_music_ratio": 0.0,
      "probe_positions": [0.15, 0.45, 0.75],
      "probe_version": 3
    }

    An ASR team and a TTS team can then select different subsets from the same collection without re-analysing anything.

    Compliance

    Voices are personal data under GDPR and similar laws, and voice cloning raises additional consent and misuse issues. YouTube's Terms of Service restrict automated access and downloading, and copyright applies to the recordings. Measure speaker counts without identifying speakers, keep provenance per file, honor removal requests, and get legal review for your jurisdiction and use case.

    If you'd rather receive data already filtered

    Corpora delivers YouTube speech audio filtered by language, speech ratio, speaker count and music, with every measurement in the manifest and an audited precision per batch. See pricing or talk to sales.

    FAQ

    How many hours of speech do I need to train ASR?

    It depends on the language, domain and whether you fine-tune a pretrained model. Large general-purpose models were trained on hundreds of thousands of hours; fine-tuning for a domain can work with far less. Measure speech hours, not file hours, whatever the target.

    Is YouTube audio good enough for TTS?

    Some of it is. Single-speaker, music-free, well-recorded content such as lectures, audiobook-style readings and studio podcasts can work; most vlogs and commentary cannot. Filter on music ratio, speaker count and quality first.

    Can I detect background music automatically?

    Yes. Audio-event classifiers trained on AudioSet-style labels give per-window music probabilities; aggregate them into a music ratio and a pure-music ratio per file.

    Should I trust the language in video metadata?

    No. Run audio language identification on the speech segments and keep the confidence score.

  • yt-dlp Audio Only: Download YouTube Audio for Speech Datasets

    For ASR, TTS and speech-LLM training you rarely need the video, yet many pipelines still download it, or download audio and then re-encode it to MP3 "for compatibility". This guide covers the yt-dlp audio-only path: how to select the right stream, keep it bit-for-bit, pick the original-language track on videos with dubbed audio, filter before downloading, and do the 16 kHz conversion once, at training time. Flags were checked against the yt-dlp README and source at release 2026.10.26.

    TL;DR

    • Use -f ba (bestaudio). It selects audio-only formats. Don't use ba*, which can return a muxed video file.
    • -x with the default --audio-format best doesn't re-encode. YouTube's Opus-in-WebM gets stream-copied to .opus, and M4A stays as it is. Only an explicit target like mp3 triggers a lossy re-encode.
    • On videos with several audio tracks, add -S lang. yt-dlp marks the original track with the highest language preference, but YouTube's default sort only uses it as a tiebreaker after quality and codec.
    • Store the native Opus/M4A. Convert to 16 kHz mono with ffmpeg -ac 1 -ar 16000 when you build training shards, not when you download.
    • Audio is cheap on bandwidth: 100k hours at 160 kbps is 7.2 TB. At that size, request rate limits cost you more time than the link does.

    Select the audio stream, not the video

    YouTube serves audio and video as separate DASH streams, so you can download just the audio. The yt-dlp format selection docs define ba/bestaudio as "the best quality audio-only format", equivalent to best*[vcodec=none]. ba* means "best format that contains audio" and may include video. The README marks it "Do not use!".

    Before you script anything, look at what a video actually offers:

    yt-dlp -F 'https://www.youtube.com/watch?v=VIDEO_ID'

    The audio-only rows show the codec, bitrate, sample rate (asr) and, on videos that have it, a language in brackets plus track notes such as (original) or (default).

    YouTube's audio-only formats

    There are two families: Opus in a WebM container and AAC in an M4A (MP4) container. yt-dlp no longer ships a hardcoded itag table. It now reads codec, bitrate and sample rate from YouTube's player response. The last version that had the table (2024.12.23) listed these nominal values:

    itag Container Codec Nominal bitrate (old yt-dlp table)
    249 webm Opus 50 kbps
    250 webm Opus 70 kbps
    251 webm Opus 160 kbps
    139 m4a AAC 48 kbps
    140 m4a AAC 128 kbps

    Treat these as labels, not measurements. Opus here is variable-bitrate, so the real average depends on the content and is often below the nominal figure. Which itags exist also varies by video and player client. The current extractor's own test suite marks itag 141 (256 kbps AAC) as "not served anymore". Check -F for real numbers.

    Some videos also offer DRC (dynamic range compression) variants, which show up as format IDs like 251-drc with "DRC" in the note. The YouTube extractor ranks them slightly below the regular stream. If you want uncompressed dynamics, exclude them explicitly with -f 'ba[format_id!*=drc]/ba'.

    Format sorting for audio (-S)

    -S changes what counts as "best". The sorting docs list these audio-relevant fields:

    • acodec: flac/alac > wav/aiff > opus > vorbis > aac > mp4a > mp3 > …
    • abr (average audio bitrate, kbps), asr (sample rate, Hz), channels
    • size (exact or approximate file size)
    • lang: "the language preference as determined by the extractor (e.g. original language preferred over audio description)"

    Fields sort descending; a + prefix reverses that. Your -S fields are placed ahead of the extractor's order. That matters on YouTube, whose extractor sets its own order (quality, res, fps, hdr:12, source, vcodec, channels, acodec, lang, proto), so lang normally comes near the end. Useful combinations:

    • -f ba -S lang: original track first, then the usual quality order.
    • -f 'ba[asr>=16000]' -S lang,+size: the smallest audio-only file that still has at least a 16 kHz sample rate. This makes sense if you're storage-bound and only train at 16 kHz. Measure WER on a sample before committing to it.
    • -f 'ba[ext=m4a]/ba': prefer AAC/M4A, for example if your decoder stack handles MP4 more easily than WebM/Ogg.

    Dubbed and multi-language audio tracks

    YouTube can attach several audio tracks to one video: the original, AI or human dubs, and audio description. For a speech dataset, a dub labeled with the original video's metadata is a data-quality bug. From the extractor source, this is how yt-dlp exposes them:

    • Each audio format's language is the part of YouTube's audio track ID before the dot.
    • A track whose display name contains "original" gets language_preference 10 and an (original) note. The default track gets 5, audio-description tracks get −10 with a language like en-desc, and all others get −1.
    • On single-track videos, yt-dlp fills language on audio formats from the original-language auto-caption track when one exists. Without one, language can be empty.

    So -S lang picks the original track whenever YouTube labels it. To require a specific language, use a format filter. String filters support ^=, $=, *=, ~= and negation with !:

    # English audio, but never the audio-description track
    yt-dlp -f 'ba[language^=en][language!$=desc]' -S lang URL

    If no format matches, that video fails with "Requested format is not available". yt-dlp then moves on to the next URL, because --no-abort-on-error is the default. Append /ba only if you'd rather get some audio than none. A format filter also excludes formats whose language is unknown, unless you write it as language^=?en.

    -x, --audio-format and why not MP3

    -x/--extract-audio needs ffmpeg and ffprobe. --audio-format accepts best (default), aac, alac, flac, m4a, mp3, opus, vorbis and wav. Here is what the FFmpegExtractAudioPP source actually does:

    • best + .m4a: does nothing ("already in a common audio format").
    • best + .webm: ffprobe reports Opus, so ffmpeg runs with -acodec copy into a .opus file. It's lossless and quick.
    • A different target (e.g. mp3): a full decode and re-encode with libmp3lame.

    Re-encoding Opus or AAC to MP3 stacks one lossy codec on top of another. Quality can only go down, and you pay CPU for every hour of audio. --audio-quality (0 best to 10 worst for VBR, or a bitrate like 128K; default 5) only matters when a re-encode happens. The built-in preset -t mp3 expands to -f 'ba[acodec^=mp3]/ba/b' -x --audio-format mp3. That suits a music player, not a corpus.

    Convert to 16 kHz mono at training time

    Many ASR models, including Whisper and wav2vec 2.0, take 16 kHz mono input. Do the conversion when you build shards, not at download:

    # 16 kHz, mono, 16-bit PCM WAV
    ffmpeg -nostdin -i abc123.opus -vn -ac 1 -ar 16000 -c:a pcm_s16le abc123.wav
    
    # Same, as FLAC (lossless, smaller than WAV)
    ffmpeg -nostdin -i abc123.opus -vn -ac 1 -ar 16000 -sample_fmt s16 -c:a flac abc123.flac

    -ac sets the channel count and -ar the sample rate (ffmpeg docs). Keeping the native file means you can later produce 24 kHz for TTS, keep stereo, or try another resampler without downloading again.

    If you really only want WAV on disk, you can do the conversion inside yt-dlp with --postprocessor-args (alias --ppa). The ExtractAudio+ffmpeg_o key passes the arguments to ffmpeg just before the output file, and only when the ExtractAudio step runs it:

    yt-dlp -f ba -S lang -x --audio-format wav \
      --ppa 'ExtractAudio+ffmpeg_o:-ac 1 -ar 16000' \
      -o '%(id)s.%(ext)s' URL

    Cut segments with --download-sections

    --download-sections takes a chapter-title regex or, with a * prefix, a time range. Negative timestamps count from the end, *from-url uses the URL's start/end time, and the flag can be repeated. It needs ffmpeg.

    # Minutes 10–20 only, plus any chapter whose title matches "interview"
    yt-dlp -f ba -S lang -x \
      --download-sections '*10:00-20:00' --download-sections 'interview' \
      -o '%(id)s_%(section_start)s.%(ext)s' URL

    Use this when you already have timestamps, for example from a dataset manifest. Treat the cut edges as approximate, and trim to exact sample boundaries in your own preprocessing if your labels need it.

    Filter before you download

    --match-filters accepts any output-template field, compared with the same operators as format filtering. & joins conditions, !field means "field absent or false", and repeating the flag ORs the filters:

    yt-dlp -f ba -S lang -x \
      --match-filters '!is_live & duration>=60 & duration<=7200 & language^=en' \
      URL_OR_PLAYLIST

    A detail that matters for language: YouTube doesn't set a video-level language, but language is a format field. yt-dlp evaluates match filters once before format selection, where missing format fields pass, and again in process_info after selection, before any media is downloaded. In practice language^=en tests the audio track you're about to download. Together with -S lang, that means "the original audio is English". Videos with no language info are rejected unless you write language^=?en.

    Filters on duration and is_live can drop playlist entries early, before a full extraction, when the playlist listing already includes those fields. For license filtering (Creative Commons), see our Creative Commons guide.

    Provenance: info.json, embedded metadata, archive

    • --write-info-json writes the full metadata next to the file, including the selected format_id, language, asr, abr, the license and the channel. That's your per-sample lineage.
    • --embed-metadata writes title, uploader, date and similar fields into the audio file's tags, so they stay with the file if it gets separated from its JSON. (The infojson attachment only applies to mkv/mka.)
    • --download-archive archive.txt records IDs so reruns skip completed videos.
    • Name files by %(id)s, not by title.

    Transcripts are a separate topic: our guide to yt-dlp subtitles and auto captions covers manual vs ASR tracks, the -orig track and VTT cleanup.

    Throttling: audio is request-bound

    An audio file is a small fraction of the size of its video, so per-video request overhead and YouTube's rate limits set your throughput, not bandwidth. The built-in -t sleep preset expands to --sleep-subtitles 5 --sleep-requests 0.75 --sleep-interval 10 --max-sleep-interval 20. For "Sign in to confirm you're not a bot", PO tokens, cookies and spreading load across IPs, see the troubleshooting guide.

    Storage math for 100k hours

    Formula: bytes = bitrate (bits/s) ÷ 8 × seconds. 100,000 h × 3,600 s = 360,000,000 s. The bitrates below are assumptions; your real Opus/AAC averages will differ.

    Storage format (assumed bitrate) Per hour 100k hours
    Native AAC/Opus at 128 kbps 57.6 MB 5.76 TB
    Native Opus at 160 kbps 72 MB 7.2 TB
    16 kHz × 16-bit × mono WAV = 256 kbps 115.2 MB 11.52 TB
    48 kHz × 16-bit × stereo WAV = 1,536 kbps 691.2 MB 69.12 TB

    For example, 160,000 ÷ 8 × 360,000,000 = 7,200,000,000,000 bytes = 7.2 TB. FLAC at 16 kHz mono comes in below the WAV figure, but the ratio depends on the content, so measure it on a sample. Transfer time: 7.2 TB is 57.6 Tbit, or 57,600 s ≈ 16 h at a sustained 1 Gbps.

    The request side looks different. Assume an average video length of 10 minutes: 100k hours is then 600,000 videos. With -t sleep the per-video sleep averages 15 s, so one worker starts at most 240 videos/hour. That's 2,500 worker-hours, about 104 hours of wall-clock time with 24 workers. For link and egress sizing, see bandwidth for large-scale video downloads.

    Goal → command

    Goal Command (add URL)
    Best native audio, original track, no re-encode yt-dlp -f ba -S lang -x
    Same, excluding DRC variants yt-dlp -f 'ba[format_id!*=drc]/ba' -S lang -x
    AAC/M4A only yt-dlp -f 'ba[ext=m4a]' -S lang
    Smallest file, still ≥16 kHz yt-dlp -f 'ba[asr>=16000]' -S lang,+size -x
    English original audio only yt-dlp -f ba -S lang -x --match-filters 'language^=en'
    16 kHz mono WAV straight away yt-dlp -f ba -S lang -x --audio-format wav --ppa 'ExtractAudio+ffmpeg_o:-ac 1 -ar 16000'
    Only a time range yt-dlp -f ba -S lang -x --download-sections '*10:00-20:00'
    Corpus run with provenance and pacing yt-dlp -f ba -S lang -x --embed-metadata --write-info-json --download-archive archive.txt -o '%(id)s.%(ext)s' -t sleep

    Python API

    The same setup through yt_dlp.YoutubeDL. The repo's devscripts/cli_to_api.py translates any CLI flags into these parameters.

    import yt_dlp
    from yt_dlp.utils import match_filter_func
    
    URLS = ['https://www.youtube.com/watch?v=VIDEO_ID']
    
    ydl_opts = {
        'format': 'ba[format_id!*=drc]/ba',
        'format_sort': ['lang'],                      # original audio track first
        'match_filter': match_filter_func('!is_live & duration>=60 & duration<=7200'),
        'paths': {'home': 'audio_raw'},
        'outtmpl': {'default': '%(id)s.%(ext)s'},
        'writeinfojson': True,
        'download_archive': 'archive.txt',
        'sleep_interval_requests': 0.75,
        'sleep_interval': 10,
        'max_sleep_interval': 20,
        'postprocessors': [
            # 'best' = stream-copy WebM/Opus to .opus, leave .m4a untouched
            {'key': 'FFmpegExtractAudio', 'preferredcodec': 'best'},
            {'key': 'FFmpegMetadata', 'add_metadata': True},
        ],
        # For 16 kHz mono WAV instead: set preferredcodec to 'wav' and add
        # 'postprocessor_args': {'extractaudio+ffmpeg_o': ['-ac', '1', '-ar', '16000']},
    }
    
    with yt_dlp.YoutubeDL(ydl_opts) as ydl:
        retcode = ydl.download(URLS)

    Postprocessors run in list order, so audio extraction happens before metadata embedding. Keys in postprocessor_args are lowercase.

    Compliance

    YouTube's Terms of Service restrict automated access and downloading. Copyright and licensing apply to audio just as to video; some teams collect only Creative Commons material and record the license per sample. Voices are personal data under regimes like GDPR, and a voice can identify a speaker even without a name attached. That matters especially for TTS and voice-cloning use. Honor takedown and opt-out requests, and keep the lineage (video ID, channel, info.json) you need to act on them. Get legal review for your jurisdiction and use case.

    If you'd rather not run this yourself

    Corpora offers dedicated 1–40 Gbps unmetered proxy capacity (HTTP(S) and SOCKS5, compatible with yt-dlp) for teams running their own stack. There is also a fully managed option that delivers audio, subtitles and metadata as JSONL/Parquet with sha256 checksums and manifests to S3, GCS or Azure Blob, starting with a free sample batch. Contact us to scope it.

    FAQ

    How do I download only the audio from YouTube with yt-dlp?

    yt-dlp -f ba -x URL. -f ba selects the best audio-only stream. -x with the default best format turns the WebM into .opus by stream copy and leaves M4A untouched, with no re-encoding. Add -S lang to prefer the original track on dubbed videos.

    Is --audio-format mp3 lower quality?

    Yes. YouTube's audio-only streams are Opus or AAC, so yt-dlp has to decode the Opus or AAC and re-encode it, adding a second lossy generation. Keep the native format and convert to WAV/FLAC when you need PCM.

    Why did yt-dlp download a dubbed audio track?

    On YouTube, the extractor's default sort puts quality and codec ahead of lang, so the original-track preference only breaks ties. Pass -S lang to make it the first criterion, and check -F for the (original) note.

    What sample rate should I store for ASR training?

    Store whatever YouTube served (check asr in -F). Resample to the model's rate, typically 16 kHz mono for ASR, with ffmpeg -ac 1 -ar 16000 when building training shards.

    Does --download-sections still download the whole file?

    No. yt-dlp passes each range to ffmpeg as input-side -ss/-t, so ffmpeg seeks into the stream instead of reading it from the start. You can pull labeled spans from long videos without storing the rest.