Tag: Whisper

  • Speaker Diarization with pyannote for Speech Datasets

    A speech dataset scraped from podcasts, interviews and talk shows is useless for TTS until you know who is speaking when, and only half useful for ASR. Speaker diarization answers that question, and pyannote.audio is the open-source default for it. This guide covers what diarization outputs, which pyannote pipeline to use as of pyannote.audio 4.1, how to measure DER, and how to turn the output into single-speaker clips and speaker-attributed Whisper transcripts.

    TL;DR

    • Pipeline: pyannote/speaker-diarization-community-1 on pyannote.audio 4.x. speaker-diarization-3.1 is now labelled legacy. Both are gated on Hugging Face: accept the conditions, then pass a token.
    • Output: (start, end, speaker) turns, saved as RTTM. Labels like SPEAKER_00 mean nothing outside their file.
    • Two outputs since 4.0: speaker_diarization keeps overlapping speech; exclusive_speaker_diarization allows one speaker at a time, which is what you want for aligning words.
    • For TTS, keep only turns with no overlap and no other speaker within a safety margin.
    • Measure DER on hand-labelled files from your own sources. pyannote's published numbers range from under 10% to nearly 50% depending on the domain.
    • Plan GPU time with a formula and a speed you measured yourself.

    What speaker diarization outputs

    Diarization splits a recording into speech turns and gives each turn an anonymous speaker label. It does not identify people. pyannote writes the result as RTTM:

    SPEAKER ep0412 1 6.665 4.120 <NA> <NA> SPEAKER_00 <NA> <NA>
    SPEAKER ep0412 1 10.580 2.310 <NA> <NA> SPEAKER_01 <NA> <NA>
    SPEAKER ep0412 1 12.400 0.910 <NA> <NA> SPEAKER_00 <NA> <NA>

    Field 2 is the file URI, field 4 the start in seconds, field 5 the duration (not the end time) and field 8 the speaker. Annotation.write_rttm() in pyannote.core rejects URIs and labels that contain spaces, because the format is space-separated. Two things to remember: turns can overlap when people talk at once, and labels are per file. SPEAKER_00 in one episode has nothing to do with SPEAKER_00 in the next.

    Which pyannote pipeline to use

    pyannote.audio 4.1.0 was released on PyPI on 17 November 2026 and requires Python 3.11 or newer. Version 4.0.0 (September 2025) introduced community-1, which switched from agglomerative clustering to VBx clustering and added exclusive diarization. The pyannote-audio README lists three pipelines:

    Pipeline Runs License Notes
    speaker-diarization-community-1 Locally CC-BY-4.0 Regular and exclusive diarization, plus per-speaker embeddings
    speaker-diarization-3.1 Locally MIT Legacy; mono 16 kHz input, downmixed and resampled automatically
    speaker-diarization-precision-2 pyannoteAI servers Commercial pyannoteAI API key; audio leaves your infrastructure

    A selection from the README's benchmark (DER %, updated 2025-09). These are the maintainers' own numbers:

    Dataset 3.1 (legacy) community-1
    VoxConverse v0.3 11.2 11.2
    AMI (IHM) 18.8 17.0
    DIHARD 3 (full) 21.4 20.2
    AVA-AVD 49.7 44.6
    Ego4D (dev.) 51.2 46.8

    VoxConverse is built from multi-speaker clips extracted from YouTube videos and scores far better than movies (AVA-AVD) or egocentric video (Ego4D). Your error rate will depend more on your source mix than on the pipeline version.

    Setup and gated access

    Install ffmpeg (since 4.0, audio is decoded by torchcodec; the sox and soundfile backends are gone), then pip install pyannote.audio. On Hugging Face, accept the user conditions on the community-1 page: the repository is public, but its files are gated. Create an access token and keep it in an environment variable. Older tutorials stop working for two reasons: 4.0 renamed use_auth_token= to token= and replaced name@revision with a revision= argument. For air-gapped nodes, the changelog shows how to git clone the pipeline repo with git-lfs and pass the local path to Pipeline.from_pretrained(). Optional usage telemetry can be turned off with PYANNOTE_METRICS_ENABLED=0.

    The Python API

    import os
    import torch
    from pyannote.audio import Pipeline
    
    pipeline = Pipeline.from_pretrained(
        "pyannote/speaker-diarization-community-1",
        token=os.environ["HF_TOKEN"],
    )
    pipeline.to(torch.device("cuda"))
    
    output = pipeline("ep0412.wav")                      # speaker count estimated
    output = pipeline("ep0412.wav", num_speakers=2)      # exactly 2
    output = pipeline("ep0412.wav", min_speakers=1, max_speakers=6)
    
    for turn, speaker in output.speaker_diarization:
        print(f"{turn.start:.2f} {turn.end:.2f} {speaker}")

    Details from SpeakerDiarization.apply in the source:

    • min_speakers/max_speakers are ignored when num_speakers is set. If the detected count falls outside the bounds, the pipeline warns instead of failing.
    • It returns a DiarizeOutput with speaker_diarization, exclusive_speaker_diarization and speaker_embeddings (one row per speaker, in labels() order). The legacy 3.1 pipeline returns a plain Annotation.
    • Unknown keyword arguments only produce a warning, so a typo like max_speaker is silently ignored.
    • Audio you have already decoded can be passed as {"waveform": tensor, "sample_rate": sr}.

    Only fix a speaker count you can trust. A podcast with "2 hosts" in its description also has guests, ad reads and clips, so loose bounds usually beat a confident wrong number.

    A worker that writes RTTM and a manifest

    # diarize_worker.py
    # usage: HF_TOKEN=... python diarize_worker.py files.txt out_dir
    import json
    import os
    import sys
    import time
    from pathlib import Path
    
    import torch
    from pyannote.audio import Pipeline
    
    PIPELINE = "pyannote/speaker-diarization-community-1"
    pipeline = Pipeline.from_pretrained(PIPELINE, token=os.environ["HF_TOKEN"])
    pipeline.to(torch.device("cuda"))
    
    
    def diarize(audio: Path, out_dir: Path) -> dict:
        uri = audio.stem  # no spaces allowed in RTTM
        t0 = time.monotonic()
        out = pipeline(str(audio), min_speakers=1, max_speakers=8)
        wall = time.monotonic() - t0
        diar, excl = out.speaker_diarization, out.exclusive_speaker_diarization
        diar.uri = excl.uri = uri
        with (out_dir / f"{uri}.rttm").open("w") as f:
            diar.write_rttm(f)
        with (out_dir / f"{uri}.exclusive.rttm").open("w") as f:
            excl.write_rttm(f)
        return {
            "audio_id": uri,
            "pipeline": PIPELINE,
            "n_speakers": len(diar.labels()),
            "speech_s": round(diar.get_timeline().support().duration(), 3),
            "overlap_s": round(diar.get_overlap().duration(), 3),
            "speaker_s": {spk: round(d, 3) for spk, d in diar.chart()},
            "wall_s": round(wall, 3),
        }
    
    
    if __name__ == "__main__":
        files, out_dir = Path(sys.argv[1]), Path(sys.argv[2])
        out_dir.mkdir(parents=True, exist_ok=True)
        with (out_dir / "manifest.jsonl").open("a", encoding="utf-8") as m:
            for line in files.read_text(encoding="utf-8").splitlines():
                audio = Path(line.strip())
                if not line.strip() or (out_dir / f"{audio.stem}.rttm").exists():
                    continue
                try:
                    row = diarize(audio, out_dir)
                except Exception as e:
                    row = {"audio_id": audio.stem, "error": repr(e)}
                m.write(json.dumps(row) + "\n")
                m.flush()

    Overlap ratio, dominant-speaker share and speaker count become cheap filter columns, the same properties discussed in what makes speech data ASR- and TTS-ready. Record library versions alongside the pipeline name, as the provenance guide recommends.

    Overlapped speech

    pyannote's segmentation model handles overlap directly (the powerset formulation from Plaquet and Bredin, Interspeech 2023). Version 4.0 dropped the separate overlapped-speech-detection pipeline because overlap is part of the diarization output; diar.get_overlap() returns it as a Timeline. What to do with it depends on what you're training:

    • TTS / voice cloning: drop overlapped regions and any clip that touches them.
    • ASR: a transcript holds one word stream, so overlapped words get lost or mixed. Keep overlap only if your labels can represent it.
    • Conversational models: overlap is signal. Keep the regular output.

    Selecting single-speaker segments for TTS

    A turn with a single label isn't automatically clean. A backchannel ("mm-hm") a few hundred milliseconds away, or a boundary that is slightly off, puts a second voice into the clip. This function keeps a span only if it is long enough once overlap is removed and no other speaker is active within a margin around it:

    from pyannote.core import Annotation, Segment
    
    
    def single_speaker_clips(diar: Annotation, min_dur=2.0, max_dur=20.0,
                             margin=0.5, merge_gap=0.3):
        """Yield (speaker, start, end, needs_split) for clean single-speaker spans."""
        if not diar:
            return
        clean = diar.extrude(diar.get_overlap())       # remove overlapped time
        for spk in clean.labels():
            others = diar.subset([spk], invert=True).get_timeline().support()
            for seg in clean.label_timeline(spk).support(merge_gap):
                if seg.duration < min_dur:
                    continue
                guard = Segment(max(0.0, seg.start - margin), seg.end + margin)
                if others.crop(guard).duration() > 0:
                    continue                           # another speaker too close
                yield spk, round(seg.start, 3), round(seg.end, 3), seg.duration > max_dur
    • margin trades yield for purity. Listen to 100 random clips at two settings and count contaminated ones.
    • merge_gap joins short pauses. The margin check runs after merging, so an interjection inside a merged span still rejects it.
    • Long spans are flagged, not cut. Split them at Whisper word boundaries so no clip ends mid-word.

    Diarization only tells you who spoke. Music beds, noise and clipping need their own filters.

    Combining diarization with Whisper or WhisperX timestamps

    To attribute a transcript, assign each word to the speaker whose turn overlaps it most. Use the exclusive diarization here: pyannote added it specifically to make it easier to reconcile diarization with "sometimes not so precise" transcription timestamps. With the per-word JSONL from our Whisper at scale guide (fields w, start, end):

    from pyannote.core import Annotation, Segment
    
    
    def label_words(words: list[dict], excl: Annotation) -> list[dict]:
        """Attach the speaker with the longest overlap to each word."""
        for w in words:
            start, end = w["start"], w["end"]
            if end <= start:                       # zero-length word timestamps
                start, end = start - 0.05, end + 0.05
            w["speaker"] = excl.argmax(Segment(max(0.0, start), end))  # None if no overlap
        return words

    Then group consecutive same-speaker words into utterances. Words left with None fall between turns. They are often hallucinations or speech the diarizer missed, so check them before dropping.

    WhisperX builds this in. Its DiarizationPipeline(token=..., device=...) wraps pyannote (community-1, per its README), and whisperx.assign_word_speakers() labels aligned words. Its README states the limits plainly: overlapping speech isn't handled well, "diarization is far from perfect", and words that can't be aligned, like "2014." or "£13.60", get no timing.

    Measuring DER with pyannote.metrics

    DER = (false alarm + missed detection + speaker confusion) / total reference speech

    Confusion is counted after the optimal one-to-one mapping between hypothesis and reference labels. False alarm is unbounded, so DER can exceed 100%. Always report collar (seconds ignored around reference boundaries) and skip_overlap with the number. Both default to 0.0 and False in pyannote.metrics.

    from pyannote.core import Annotation
    from pyannote.database.util import load_rttm
    from pyannote.metrics.diarization import DiarizationErrorRate
    
    refs = load_rttm("reference.rttm")      # {uri: Annotation}
    hyps = load_rttm("hypothesis.rttm")
    
    metric = DiarizationErrorRate(collar=0.0, skip_overlap=False)
    for uri, ref in refs.items():
        d = metric(ref, hyps.get(uri, Annotation(uri=uri)), detailed=True)
        print(uri, round(d["diarization error rate"], 3), "miss", round(d["missed detection"], 1),
              "fa", round(d["false alarm"], 1), "conf", round(d["confusion"], 1))
    
    print("overall DER:", round(abs(metric), 3))

    If references cover only part of each file, pass a UEM (uem=, loadable with load_uem). The components tell you more than the total. High confusion means speakers were merged or split, which hurts TTS. High miss or false alarm means speech-detection errors, which distort hour counts. Build the reference set from your own sources, stratified by source type, and check clip purity with audit sampling and confidence intervals.

    GPU throughput: reason with a formula

    Measure s/h, the wall-clock seconds per hour of audio on one GPU, on your own files. Then:

    • GPU-hours = audio_hours × s/h / 3600
    • wall-clock hours = GPU-hours / number_of_GPUs

    The README publishes one reference point: community-1 self-hosted on an NVIDIA H100 80GB took 31 s per hour of audio on AMI (~1-hour files) and 37 s on DIHARD 3 (~5-minute files). If 31 s/h held for your data, 100,000 hours would take 100,000 × 31 / 3600 ≈ 861 H100-hours. Use that to size a pilot, not to promise a deadline.

    • Sliding windows: by the constructor default, segmentation windows overlap by 90%, and one embedding is extracted per (window, local speaker slot). Cost grows with duration, and clustering grows with the embedding count, so time very long files separately.
    • Batching: segmentation_batch_size and embedding_batch_size control GPU batches. Raise them while watching VRAM.
    • Decoding: CPU-side ffmpeg decoding can starve the GPU. Decode once and share the waveform with Whisper, or run several workers per GPU.
    • Stacked passes: Whisper, alignment and diarization each add their own s/h.

    Common failure modes

    Failure What you see Mitigation
    Music and singing Speech detected over music beds; a singer becomes a "speaker" Music/speech classifier first; drop low speech-ratio files or regions
    Laughter, applause, crowd noise Short false-alarm turns, sometimes a new speaker Drop very short turns; ignore speakers with tiny total time
    Short turns, backchannels "Yeah", "right" missed or absorbed by the main speaker Margin check above; clip purity audits
    Similar voices Two hosts merged into one label Compare speaker-count distribution with metadata; set min_speakers only when known
    One voice split in two Same person labelled twice after a mic change, phone segment or shouting Compare speaker_embeddings within a file; merge only after checking a sample
    Inserted clips, voice-overs Extra speakers from ads, trailers, quoted videos Keep only dominant speakers for TTS

    A quick sanity check: plot n_speakers per source. If a solo-commentary channel shows four speakers in many files, the cause is usually music or noise, not guests.

    Compliance

    If the audio comes from YouTube, its Terms of Service restrict automated access and downloading, and copyright and licensing apply to the recordings. Some teams restrict collection to Creative Commons material. Voices are personal data under GDPR and similar laws. Speaker embeddings are designed to tell people apart, so some jurisdictions may treat them as biometric data, and hosted pipelines send the audio to a third party. Keep provenance per file, honor takedown and opt-out requests, and get legal review for your jurisdiction and use case.

    If you'd rather not run the collection side yourself

    Corpora's managed service delivers YouTube audio, subtitles and metadata as JSONL/Parquet with sha256 checksums and manifests to S3, GCS or Azure Blob, starting with a free sample batch. Diarization and transcription stay on your GPUs. Teams running their own yt-dlp stack (see audio-only downloads for speech datasets) can use dedicated unmetered proxy capacity instead.

    FAQ

    Is pyannote speaker-diarization-3.1 still usable?

    Yes. It's MIT-licensed, but the README labels it legacy. In the published benchmark, community-1 matches or beats it everywhere except REPERE, and adds exclusive diarization. When upgrading code to 4.x, change use_auth_token to token and handle the DiarizeOutput return type.

    Why do I get a 401 or "gated repo" error?

    The files are gated. Accept the user conditions on the model page with the same Hugging Face account that owns the token, then pass it with token=. Accepting with a personal account and using a CI account's token produces the same error.

    How do I keep speaker labels consistent across files?

    pyannote doesn't; labels are per file. Cluster output.speaker_embeddings across files or compare them with enrolled reference voices, and validate on a labelled sample. A false merge across files does more damage to TTS than a missed one.

    What is a good DER?

    It depends on the domain and the collar and overlap settings. pyannote's community-1 numbers run from about 9% (REPERE) to about 47% (Ego4D). Set a target from the downstream use: confusion matters most for TTS purity, miss and false alarm matter most for hour counts.

  • Whisper Transcription at Scale for Speech Datasets

    Running Whisper on one podcast is a one-liner. Running it over 50,000 hours of mixed YouTube audio is a different job: silence and music make the model invent text, long files fall into repetition loops, language labels are wrong, and nobody knows how many GPU-hours the run will take. This guide covers Whisper transcription at dataset scale: model choice, faster-whisper and WhisperX, VAD, failure modes, throughput math, output formats and quality checks, with APIs checked against the current repositories and model cards.

    TL;DR

    • Model: large-v3 for best accuracy, large-v3-turbo (809M parameters, 4 decoder layers instead of 32) when throughput matters more. Measure the WER difference on your own audio.
    • Runtime: faster-whisper's BatchedInferencePipeline for bulk transcription; WhisperX when you need forced-aligned word timestamps or speaker labels.
    • Always run VAD first. Hallucinations cluster in non-speech audio, and VAD also tells you how many hours are actually speech.
    • Keep every segment's avg_logprob, compression_ratio and no_speech_prob and filter afterwards, especially in batched mode, which does not re-decode failed chunks.
    • Plan capacity with a measured RTFx (audio seconds per wall-clock second) from a pilot, not with someone else's benchmark.
    • Validate against manual YouTube subtitles with jiwer, treating the score as an agreement signal, not true WER.

    Choosing a Whisper model

    The openai/whisper README lists six sizes. VRAM and relative speed below are OpenAI's own approximate figures (English, A100); real-world speed varies with language, speaking rate and hardware.

    Size Parameters English-only Required VRAM (approx.) Relative speed (approx.)
    tiny 39 M tiny.en ~1 GB ~10x
    base 74 M base.en ~1 GB ~7x
    small 244 M small.en ~2 GB ~4x
    medium 769 M medium.en ~5 GB ~2x
    large (v3) 1550 M none ~10 GB 1x
    turbo (large-v3-turbo) 809 M none ~6 GB ~8x

    Details that matter for a dataset run:

    • large now means large-v3. Both openai-whisper and faster-whisper map the alias large to the large-v3 checkpoint. Pin the explicit name in your manifests so a rerun is reproducible.
    • large-v3 uses 128 Mel bins instead of 80 and was trained on 1 million hours of weakly labeled audio plus 4 million hours pseudo-labeled with large-v2, according to its model card.
    • large-v3-turbo is, per its model card, a fine-tuned, pruned large-v3 with the decoder reduced from 32 to 4 layers, "at the expense of a minor quality degradation". The README notes it is not trained for translation, which doesn't matter for same-language transcription.
    • Accuracy varies widely by language. OpenAI's per-language WER/CER breakdown is in the README and the Whisper paper. For low-resource languages, run a pilot before committing GPU budget.

    Choosing a runtime

    Runtime What it adds Use it for
    openai-whisper Reference implementation, CLI with txt/vtt/srt/tsv/json output Small jobs, reproducing reference behavior
    faster-whisper CTranslate2 backend, int8/fp16, built-in Silero VAD, batched pipeline Bulk transcription
    WhisperX faster-whisper backend plus wav2vec2 forced alignment and pyannote diarization Precise word timing, speaker labels

    The faster-whisper README claims up to 4x the speed of openai/whisper at the same accuracy with less memory, and publishes a benchmark: 13 minutes of audio with large-v2 on an RTX 3070 Ti took 1m03s unbatched and 17s with batch_size=8 (fp16). That's roughly 46x real time for that one file on that GPU. Treat it as an order of magnitude, not a plan.

    faster-whisper basics

    The model constructor takes device ("cuda", "cpu" or "auto") and compute_type ("float16", "int8_float16", "int8" and others). transcribe() returns a generator of segments plus a TranscriptionInfo; nothing is decoded until you iterate:

    from faster_whisper import WhisperModel
    
    model = WhisperModel("large-v3", device="cuda", compute_type="float16")
    segments, info = model.transcribe(
        "audio.opus",
        vad_filter=True,
        word_timestamps=True,
        condition_on_previous_text=False,
    )
    for s in segments:  # decoding happens here
        print(f"[{s.start:.2f} -> {s.end:.2f}] {s.text}")

    Audio is decoded with PyAV, so faster-whisper doesn't need a system ffmpeg. Feed it the native Opus or M4A you downloaded; see yt-dlp audio-only downloads for speech datasets for getting those files without re-encoding.

    VAD first: fewer hallucinations, honest hours

    Whisper's model card states the problem directly: because it is trained on large-scale noisy data, "the predictions may include texts that are not actually spoken in the audio input (i.e. hallucination)". The Careless Whisper study found that roughly 1% of the transcriptions it examined contained entire hallucinated phrases or sentences, and that hallucinations occurred disproportionately for speakers with longer non-vocal stretches. Silence, music beds and applause are exactly where this happens.

    Voice-activity detection removes those stretches before the decoder sees them. The WhisperX paper also reports that VAD-based pre-segmentation improves long-form transcription quality and enables batching. In faster-whisper:

    • Sequential WhisperModel.transcribe: vad_filter=False by default. When enabled, the default is conservative and only removes silence longer than 2 seconds (min_silence_duration_ms=2000). Tune it with vad_parameters=dict(min_silence_duration_ms=500).
    • BatchedInferencePipeline.transcribe: vad_filter=True by default, with 160 ms minimum silence and speech chunks capped at the 30-second window. Disable VAD on a file longer than 30 seconds without passing clip_timestamps and it raises an error.

    A bonus: info.duration_after_vad is a free speech-duration measurement. Store it next to info.duration; it's the number behind "hours" discussed in what makes speech training data ASR- and TTS-ready. VAD reduces hallucination on non-speech; it does not remove it, and it won't help with speech over loud music.

    Known failure modes and the knobs that address them

    Failure What it looks like Mitigation
    Hallucination on non-speech Fluent sentences over silence, music or noise VAD; no_speech_prob filtering; hallucination_silence_threshold (requires word timestamps, sequential mode)
    Repetition loops The same phrase repeated dozens of times condition_on_previous_text=False; compression-ratio checks; n-gram repetition filter
    Drift and bad timestamps Segment times seconds off on long files VAD segmentation; WhisperX forced alignment
    Wrong language English "transcript" of Spanish audio, or translated text Language-probability threshold, then re-run with a fixed language

    The model card notes that the sequence-to-sequence architecture "makes it prone to generating repetitive texts, which can be mitigated to some degree by beam search and temperature scheduling but not perfectly". faster-whisper's docstring for condition_on_previous_text spells out the trade-off: disabling it may make text less consistent across windows, "but the model becomes less prone to getting stuck in a failure loop, such as repetition looping or timestamps going out of sync". WhisperX sets it to False by default.

    The sequential decoder uses temperature fallback: if a window's gzip compression_ratio exceeds 2.4 or its avg_logprob is below -1.0, it is re-decoded at a higher temperature. A window counts as silent when no_speech_prob exceeds 0.6 and the log-probability is below the threshold. These are the same defaults in openai-whisper and faster-whisper.

    One thing that is easy to miss when reading the source: the batched pipeline decodes each chunk once, at the first temperature only, so the fallback never re-decodes a bad chunk. The metrics are still returned per segment. That makes post-hoc filtering mandatory, not optional, for batched runs.

    A cheap extra guard: count word trigrams per segment and flag any segment where one trigram repeats several times. It catches loops that a compression-ratio threshold alone misses on short segments.

    Language detection

    Without a language argument, Whisper detects the language from the first 30-second window (faster-whisper exposes language_detection_segments to use more windows, and runs detection on VAD-filtered audio in batched mode). info.language, info.language_probability and info.all_language_probs come back with the result.

    A workable pattern for large collections:

    1. Run detection, keep the probability in the manifest.
    2. Accept files above a confidence threshold you calibrate on a hand-checked sample.
    3. Re-run low-confidence files with an explicit language, or route them to review. For genuinely mixed-language audio, multilingual=True runs detection on every segment.

    Don't trust video metadata for this; title language and spoken language often differ.

    A batched worker that writes JSONL

    This worker reads a list of audio paths, writes one segments file per input, appends a per-file row to a manifest, and skips finished files on restart. Run one process per GPU (set CUDA_VISIBLE_DEVICES) and shard the file list across processes.

    # transcribe_worker.py
    # usage: python transcribe_worker.py files.txt out_dir
    import json
    import os
    import sys
    import time
    from pathlib import Path
    
    from faster_whisper import BatchedInferencePipeline, WhisperModel
    
    MODEL = "large-v3"           # or "large-v3-turbo"
    COMPUTE_TYPE = "float16"     # "int8_float16" to save VRAM
    BATCH_SIZE = 16              # lower it if you hit out-of-memory
    
    model = WhisperModel(MODEL, device="cuda", compute_type=COMPUTE_TYPE)
    pipe = BatchedInferencePipeline(model=model)
    
    
    def r(x):
        return round(float(x), 3)
    
    
    def transcribe(audio_path: Path, out_dir: Path) -> dict:
        out_path = out_dir / f"{audio_path.stem}.segments.jsonl"
        tmp_path = out_dir / f"{audio_path.stem}.segments.tmp"
        t0 = time.monotonic()
        segments, info = pipe.transcribe(
            str(audio_path),
            batch_size=BATCH_SIZE,
            vad_filter=True,
            word_timestamps=True,
        )
        n = 0
        with tmp_path.open("w", encoding="utf-8") as f:
            for s in segments:
                row = {
                    "audio_id": audio_path.stem,
                    "seg": n,
                    "start": r(s.start),
                    "end": r(s.end),
                    "text": s.text.strip(),
                    "avg_logprob": r(s.avg_logprob),
                    "compression_ratio": r(s.compression_ratio),
                    "no_speech_prob": r(s.no_speech_prob),
                    "words": [
                        {"w": w.word, "start": r(w.start), "end": r(w.end), "p": r(w.probability)}
                        for w in (s.words or [])
                    ],
                }
                f.write(json.dumps(row, ensure_ascii=False) + "\n")
                n += 1
        os.replace(tmp_path, out_path)  # only complete files get the final name
        wall = time.monotonic() - t0
        return {
            "audio_id": audio_path.stem,
            "duration_s": r(info.duration),
            "speech_s": r(info.duration_after_vad),
            "language": info.language,
            "language_probability": r(info.language_probability),
            "n_segments": n,
            "model": MODEL,
            "compute_type": COMPUTE_TYPE,
            "wall_s": r(wall),
            "rtfx": r(info.duration / wall) if wall > 0 else None,
        }
    
    
    if __name__ == "__main__":
        file_list, out_dir = Path(sys.argv[1]), Path(sys.argv[2])
        out_dir.mkdir(parents=True, exist_ok=True)
        with (out_dir / "manifest.jsonl").open("a", encoding="utf-8") as manifest:
            for line in file_list.read_text(encoding="utf-8").splitlines():
                if not line.strip():
                    continue
                audio = Path(line.strip())
                if (out_dir / f"{audio.stem}.segments.jsonl").exists():
                    continue
                try:
                    row = transcribe(audio, out_dir)
                except Exception as e:  # log and keep the worker alive
                    row = {"audio_id": audio.stem, "error": repr(e)}
                manifest.write(json.dumps(row, ensure_ascii=False) + "\n")
                manifest.flush()

    JSONL per file is easy to append, resume and grep. For training, compact it into Parquet or WebDataset shards later, as described in storing datasets as WebDataset and Parquet shards. Record the model name, compute type and library versions in the manifest; a transcript without that provenance can't be regenerated or compared.

    WhisperX: aligned words and speaker labels

    Whisper's own word timestamps come from cross-attention and dynamic time warping. WhisperX adds a second pass that force-aligns the transcript to a language-specific wav2vec2 phoneme model, then optionally assigns speakers using pyannote. From the WhisperX README:

    import whisperx
    from whisperx.diarize import DiarizationPipeline
    
    device = "cuda"
    audio = whisperx.load_audio("audio.opus")
    
    model = whisperx.load_model("large-v3", device, compute_type="float16")
    result = model.transcribe(audio, batch_size=16)
    
    model_a, metadata = whisperx.load_align_model(language_code=result["language"], device=device)
    result = whisperx.align(result["segments"], model_a, metadata, audio, device,
                            return_char_alignments=False)
    
    diarize_model = DiarizationPipeline(token="HF_TOKEN", device=device)
    diarize_segments = diarize_model(audio)
    result = whisperx.assign_word_speakers(diarize_segments, result)

    Caveats from the README itself: diarization needs a Hugging Face token and acceptance of the pyannote speaker-diarization-community-1 user agreement; words without characters in the alignment model's dictionary (for example "2014." or "£13.60") get no timing; overlapping speech is handled poorly; "diarization is far from perfect"; and alignment needs a wav2vec2 model for each language. If you only need segment-level text, the alignment and diarization passes are extra GPU time you can skip.

    Throughput math

    Use RTFx = audio seconds processed / wall-clock seconds, measured per GPU on your audio, model, batch size and compute type. Then:

    • GPU-hours = audio_hours / RTFx
    • wall-clock hours = audio_hours / (RTFx × number_of_GPUs)

    Which "audio hours" you plug in depends on whether RTFx was measured against file duration or post-VAD duration. The worker above measures against file duration, so VAD savings are already included.

    Worked example with assumed numbers, not measurements: 100,000 file hours, a pilot RTFx of 40 per GPU.

    GPUs GPU-hours Wall-clock
    8 100,000 / 40 = 2,500 2,500 / 8 = 312.5 h (~13 days)
    32 2,500 2,500 / 32 ≈ 78 h (~3.3 days)

    If a turbo pilot measures RTFx 100 on the same data, GPU-hours drop to 1,000. Whether that's worth the accuracy cost is decided by the WER check below, not by the speed table.

    For a trustworthy RTFx, run a pilot of a few dozen hours stratified like the full collection (lengths, languages, speech ratio). Throughput depends on speech density and words spoken, so a pilot of clean audiobooks overstates speed on talk shows. Watch non-GPU bottlenecks too: CPU audio decoding, network storage reads, and per-file overhead on very short files.

    Quality checks: WER against manual subtitles

    When a video has creator-uploaded subtitles, you have a free reference. Download them with yt-dlp's --write-subs and keep the info.json so you can tell manual from automatic tracks; our subtitles guide covers the details. Don't use YouTube's auto-captions as a reference: they are another ASR system's output.

    Normalize both sides the same way, then score with jiwer. The normalizers ship with openai-whisper (pip install openai-whisper jiwer):

    import jiwer
    from whisper.normalizers import EnglishTextNormalizer  # BasicTextNormalizer for other languages
    
    normalize = EnglishTextNormalizer()
    
    def agreement(reference_text: str, hypothesis_text: str) -> dict:
        ref = normalize(reference_text)
        hyp = normalize(hypothesis_text)
        out = jiwer.process_words(ref, hyp)
        return {
            "wer": out.wer,
            "substitutions": out.substitutions,
            "deletions": out.deletions,
            "insertions": out.insertions,
        }

    Concatenate per file rather than matching subtitle cues to segments one by one; cue boundaries and segment boundaries don't line up. Since jiwer 4.0, an empty reference is allowed and every hypothesis word counts as an insertion, which is useful for testing hallucination on known-silent clips.

    Interpret the number carefully. Manual subtitles are frequently not verbatim: fillers and false starts are dropped, sentences are condensed, and some tracks are translations. A high score can mean bad ASR or a loose reference. Use it to:

    • Compare models and settings on the same files (large-v3 vs turbo, int8 vs fp16, VAD settings).
    • Find outliers: files with very high disagreement usually have wrong language, music, or a hallucination loop.
    • Calibrate filters for files that have no subtitles, using the segment metrics.

    Combine it with cheap per-segment checks that need no reference: compression_ratio above 2.4, no_speech_prob high with text present, words per second outside a plausible range, repeated n-grams, and segments whose word timestamps all collapse to one point. Then hand-audit a random sample of what passes, not only what fails.

    Compliance

    If the audio comes from YouTube, its Terms of Service restrict automated access and downloading. Copyright and licensing apply to the recordings and arguably to transcripts derived from them; some teams restrict collection to Creative Commons material. Voices and transcribed speech can be personal data under GDPR and similar laws, and diarization creates per-speaker records. Keep provenance per file, honor takedown and opt-out requests, and get legal review for your jurisdiction and use case.

    If you'd rather not run the collection side yourself

    Corpora's managed service delivers YouTube audio, subtitles (including manual tracks, where they exist, for use as references) and metadata as JSONL/Parquet with sha256 checksums and manifests to S3, GCS or Azure Blob, starting with a free sample batch. Teams that run their own yt-dlp stack can use dedicated unmetered proxy capacity instead. Transcription stays on your GPUs either way.

    FAQ

    Is faster-whisper as accurate as openai-whisper?

    It runs the same Whisper weights converted to CTranslate2, and its README claims the same accuracy at higher speed. Defaults differ (faster-whisper uses beam size 5, openai-whisper's transcribe uses 1), and int8 quantization or batched mode can shift results, so compare on a sample with identical settings.

    Should I use large-v3 or large-v3-turbo?

    Turbo has 809M parameters and a 4-layer decoder, and OpenAI describes the quality loss as minor. For high-resource languages and clean speech it is often the better trade; for low-resource languages, measure both on a labeled sample before deciding.

    How do I stop Whisper from hallucinating on silence?

    Run VAD before decoding (vad_filter=True in faster-whisper), disable condition_on_previous_text, and filter segments on no_speech_prob, compression_ratio and repetition afterwards. With word timestamps in sequential mode, hallucination_silence_threshold skips long silences when a hallucination is suspected. None of these remove the problem completely.

    How long will it take to transcribe 10,000 hours?

    Wall-clock hours = 10,000 / (RTFx × GPUs). Measure RTFx on a representative pilot with your exact model, batch size and compute type; at an assumed RTFx of 40, that's 250 GPU-hours.