Tag: speaker diarization

  • Speaker Diarization with pyannote for Speech Datasets

    A speech dataset scraped from podcasts, interviews and talk shows is useless for TTS until you know who is speaking when, and only half useful for ASR. Speaker diarization answers that question, and pyannote.audio is the open-source default for it. This guide covers what diarization outputs, which pyannote pipeline to use as of pyannote.audio 4.1, how to measure DER, and how to turn the output into single-speaker clips and speaker-attributed Whisper transcripts.

    TL;DR

    • Pipeline: pyannote/speaker-diarization-community-1 on pyannote.audio 4.x. speaker-diarization-3.1 is now labelled legacy. Both are gated on Hugging Face: accept the conditions, then pass a token.
    • Output: (start, end, speaker) turns, saved as RTTM. Labels like SPEAKER_00 mean nothing outside their file.
    • Two outputs since 4.0: speaker_diarization keeps overlapping speech; exclusive_speaker_diarization allows one speaker at a time, which is what you want for aligning words.
    • For TTS, keep only turns with no overlap and no other speaker within a safety margin.
    • Measure DER on hand-labelled files from your own sources. pyannote's published numbers range from under 10% to nearly 50% depending on the domain.
    • Plan GPU time with a formula and a speed you measured yourself.

    What speaker diarization outputs

    Diarization splits a recording into speech turns and gives each turn an anonymous speaker label. It does not identify people. pyannote writes the result as RTTM:

    SPEAKER ep0412 1 6.665 4.120 <NA> <NA> SPEAKER_00 <NA> <NA>
    SPEAKER ep0412 1 10.580 2.310 <NA> <NA> SPEAKER_01 <NA> <NA>
    SPEAKER ep0412 1 12.400 0.910 <NA> <NA> SPEAKER_00 <NA> <NA>

    Field 2 is the file URI, field 4 the start in seconds, field 5 the duration (not the end time) and field 8 the speaker. Annotation.write_rttm() in pyannote.core rejects URIs and labels that contain spaces, because the format is space-separated. Two things to remember: turns can overlap when people talk at once, and labels are per file. SPEAKER_00 in one episode has nothing to do with SPEAKER_00 in the next.

    Which pyannote pipeline to use

    pyannote.audio 4.1.0 was released on PyPI on 17 November 2026 and requires Python 3.11 or newer. Version 4.0.0 (September 2025) introduced community-1, which switched from agglomerative clustering to VBx clustering and added exclusive diarization. The pyannote-audio README lists three pipelines:

    Pipeline Runs License Notes
    speaker-diarization-community-1 Locally CC-BY-4.0 Regular and exclusive diarization, plus per-speaker embeddings
    speaker-diarization-3.1 Locally MIT Legacy; mono 16 kHz input, downmixed and resampled automatically
    speaker-diarization-precision-2 pyannoteAI servers Commercial pyannoteAI API key; audio leaves your infrastructure

    A selection from the README's benchmark (DER %, updated 2025-09). These are the maintainers' own numbers:

    Dataset 3.1 (legacy) community-1
    VoxConverse v0.3 11.2 11.2
    AMI (IHM) 18.8 17.0
    DIHARD 3 (full) 21.4 20.2
    AVA-AVD 49.7 44.6
    Ego4D (dev.) 51.2 46.8

    VoxConverse is built from multi-speaker clips extracted from YouTube videos and scores far better than movies (AVA-AVD) or egocentric video (Ego4D). Your error rate will depend more on your source mix than on the pipeline version.

    Setup and gated access

    Install ffmpeg (since 4.0, audio is decoded by torchcodec; the sox and soundfile backends are gone), then pip install pyannote.audio. On Hugging Face, accept the user conditions on the community-1 page: the repository is public, but its files are gated. Create an access token and keep it in an environment variable. Older tutorials stop working for two reasons: 4.0 renamed use_auth_token= to token= and replaced name@revision with a revision= argument. For air-gapped nodes, the changelog shows how to git clone the pipeline repo with git-lfs and pass the local path to Pipeline.from_pretrained(). Optional usage telemetry can be turned off with PYANNOTE_METRICS_ENABLED=0.

    The Python API

    import os
    import torch
    from pyannote.audio import Pipeline
    
    pipeline = Pipeline.from_pretrained(
        "pyannote/speaker-diarization-community-1",
        token=os.environ["HF_TOKEN"],
    )
    pipeline.to(torch.device("cuda"))
    
    output = pipeline("ep0412.wav")                      # speaker count estimated
    output = pipeline("ep0412.wav", num_speakers=2)      # exactly 2
    output = pipeline("ep0412.wav", min_speakers=1, max_speakers=6)
    
    for turn, speaker in output.speaker_diarization:
        print(f"{turn.start:.2f} {turn.end:.2f} {speaker}")

    Details from SpeakerDiarization.apply in the source:

    • min_speakers/max_speakers are ignored when num_speakers is set. If the detected count falls outside the bounds, the pipeline warns instead of failing.
    • It returns a DiarizeOutput with speaker_diarization, exclusive_speaker_diarization and speaker_embeddings (one row per speaker, in labels() order). The legacy 3.1 pipeline returns a plain Annotation.
    • Unknown keyword arguments only produce a warning, so a typo like max_speaker is silently ignored.
    • Audio you have already decoded can be passed as {"waveform": tensor, "sample_rate": sr}.

    Only fix a speaker count you can trust. A podcast with "2 hosts" in its description also has guests, ad reads and clips, so loose bounds usually beat a confident wrong number.

    A worker that writes RTTM and a manifest

    # diarize_worker.py
    # usage: HF_TOKEN=... python diarize_worker.py files.txt out_dir
    import json
    import os
    import sys
    import time
    from pathlib import Path
    
    import torch
    from pyannote.audio import Pipeline
    
    PIPELINE = "pyannote/speaker-diarization-community-1"
    pipeline = Pipeline.from_pretrained(PIPELINE, token=os.environ["HF_TOKEN"])
    pipeline.to(torch.device("cuda"))
    
    
    def diarize(audio: Path, out_dir: Path) -> dict:
        uri = audio.stem  # no spaces allowed in RTTM
        t0 = time.monotonic()
        out = pipeline(str(audio), min_speakers=1, max_speakers=8)
        wall = time.monotonic() - t0
        diar, excl = out.speaker_diarization, out.exclusive_speaker_diarization
        diar.uri = excl.uri = uri
        with (out_dir / f"{uri}.rttm").open("w") as f:
            diar.write_rttm(f)
        with (out_dir / f"{uri}.exclusive.rttm").open("w") as f:
            excl.write_rttm(f)
        return {
            "audio_id": uri,
            "pipeline": PIPELINE,
            "n_speakers": len(diar.labels()),
            "speech_s": round(diar.get_timeline().support().duration(), 3),
            "overlap_s": round(diar.get_overlap().duration(), 3),
            "speaker_s": {spk: round(d, 3) for spk, d in diar.chart()},
            "wall_s": round(wall, 3),
        }
    
    
    if __name__ == "__main__":
        files, out_dir = Path(sys.argv[1]), Path(sys.argv[2])
        out_dir.mkdir(parents=True, exist_ok=True)
        with (out_dir / "manifest.jsonl").open("a", encoding="utf-8") as m:
            for line in files.read_text(encoding="utf-8").splitlines():
                audio = Path(line.strip())
                if not line.strip() or (out_dir / f"{audio.stem}.rttm").exists():
                    continue
                try:
                    row = diarize(audio, out_dir)
                except Exception as e:
                    row = {"audio_id": audio.stem, "error": repr(e)}
                m.write(json.dumps(row) + "\n")
                m.flush()

    Overlap ratio, dominant-speaker share and speaker count become cheap filter columns, the same properties discussed in what makes speech data ASR- and TTS-ready. Record library versions alongside the pipeline name, as the provenance guide recommends.

    Overlapped speech

    pyannote's segmentation model handles overlap directly (the powerset formulation from Plaquet and Bredin, Interspeech 2023). Version 4.0 dropped the separate overlapped-speech-detection pipeline because overlap is part of the diarization output; diar.get_overlap() returns it as a Timeline. What to do with it depends on what you're training:

    • TTS / voice cloning: drop overlapped regions and any clip that touches them.
    • ASR: a transcript holds one word stream, so overlapped words get lost or mixed. Keep overlap only if your labels can represent it.
    • Conversational models: overlap is signal. Keep the regular output.

    Selecting single-speaker segments for TTS

    A turn with a single label isn't automatically clean. A backchannel ("mm-hm") a few hundred milliseconds away, or a boundary that is slightly off, puts a second voice into the clip. This function keeps a span only if it is long enough once overlap is removed and no other speaker is active within a margin around it:

    from pyannote.core import Annotation, Segment
    
    
    def single_speaker_clips(diar: Annotation, min_dur=2.0, max_dur=20.0,
                             margin=0.5, merge_gap=0.3):
        """Yield (speaker, start, end, needs_split) for clean single-speaker spans."""
        if not diar:
            return
        clean = diar.extrude(diar.get_overlap())       # remove overlapped time
        for spk in clean.labels():
            others = diar.subset([spk], invert=True).get_timeline().support()
            for seg in clean.label_timeline(spk).support(merge_gap):
                if seg.duration < min_dur:
                    continue
                guard = Segment(max(0.0, seg.start - margin), seg.end + margin)
                if others.crop(guard).duration() > 0:
                    continue                           # another speaker too close
                yield spk, round(seg.start, 3), round(seg.end, 3), seg.duration > max_dur
    • margin trades yield for purity. Listen to 100 random clips at two settings and count contaminated ones.
    • merge_gap joins short pauses. The margin check runs after merging, so an interjection inside a merged span still rejects it.
    • Long spans are flagged, not cut. Split them at Whisper word boundaries so no clip ends mid-word.

    Diarization only tells you who spoke. Music beds, noise and clipping need their own filters.

    Combining diarization with Whisper or WhisperX timestamps

    To attribute a transcript, assign each word to the speaker whose turn overlaps it most. Use the exclusive diarization here: pyannote added it specifically to make it easier to reconcile diarization with "sometimes not so precise" transcription timestamps. With the per-word JSONL from our Whisper at scale guide (fields w, start, end):

    from pyannote.core import Annotation, Segment
    
    
    def label_words(words: list[dict], excl: Annotation) -> list[dict]:
        """Attach the speaker with the longest overlap to each word."""
        for w in words:
            start, end = w["start"], w["end"]
            if end <= start:                       # zero-length word timestamps
                start, end = start - 0.05, end + 0.05
            w["speaker"] = excl.argmax(Segment(max(0.0, start), end))  # None if no overlap
        return words

    Then group consecutive same-speaker words into utterances. Words left with None fall between turns. They are often hallucinations or speech the diarizer missed, so check them before dropping.

    WhisperX builds this in. Its DiarizationPipeline(token=..., device=...) wraps pyannote (community-1, per its README), and whisperx.assign_word_speakers() labels aligned words. Its README states the limits plainly: overlapping speech isn't handled well, "diarization is far from perfect", and words that can't be aligned, like "2014." or "£13.60", get no timing.

    Measuring DER with pyannote.metrics

    DER = (false alarm + missed detection + speaker confusion) / total reference speech

    Confusion is counted after the optimal one-to-one mapping between hypothesis and reference labels. False alarm is unbounded, so DER can exceed 100%. Always report collar (seconds ignored around reference boundaries) and skip_overlap with the number. Both default to 0.0 and False in pyannote.metrics.

    from pyannote.core import Annotation
    from pyannote.database.util import load_rttm
    from pyannote.metrics.diarization import DiarizationErrorRate
    
    refs = load_rttm("reference.rttm")      # {uri: Annotation}
    hyps = load_rttm("hypothesis.rttm")
    
    metric = DiarizationErrorRate(collar=0.0, skip_overlap=False)
    for uri, ref in refs.items():
        d = metric(ref, hyps.get(uri, Annotation(uri=uri)), detailed=True)
        print(uri, round(d["diarization error rate"], 3), "miss", round(d["missed detection"], 1),
              "fa", round(d["false alarm"], 1), "conf", round(d["confusion"], 1))
    
    print("overall DER:", round(abs(metric), 3))

    If references cover only part of each file, pass a UEM (uem=, loadable with load_uem). The components tell you more than the total. High confusion means speakers were merged or split, which hurts TTS. High miss or false alarm means speech-detection errors, which distort hour counts. Build the reference set from your own sources, stratified by source type, and check clip purity with audit sampling and confidence intervals.

    GPU throughput: reason with a formula

    Measure s/h, the wall-clock seconds per hour of audio on one GPU, on your own files. Then:

    • GPU-hours = audio_hours × s/h / 3600
    • wall-clock hours = GPU-hours / number_of_GPUs

    The README publishes one reference point: community-1 self-hosted on an NVIDIA H100 80GB took 31 s per hour of audio on AMI (~1-hour files) and 37 s on DIHARD 3 (~5-minute files). If 31 s/h held for your data, 100,000 hours would take 100,000 × 31 / 3600 ≈ 861 H100-hours. Use that to size a pilot, not to promise a deadline.

    • Sliding windows: by the constructor default, segmentation windows overlap by 90%, and one embedding is extracted per (window, local speaker slot). Cost grows with duration, and clustering grows with the embedding count, so time very long files separately.
    • Batching: segmentation_batch_size and embedding_batch_size control GPU batches. Raise them while watching VRAM.
    • Decoding: CPU-side ffmpeg decoding can starve the GPU. Decode once and share the waveform with Whisper, or run several workers per GPU.
    • Stacked passes: Whisper, alignment and diarization each add their own s/h.

    Common failure modes

    Failure What you see Mitigation
    Music and singing Speech detected over music beds; a singer becomes a "speaker" Music/speech classifier first; drop low speech-ratio files or regions
    Laughter, applause, crowd noise Short false-alarm turns, sometimes a new speaker Drop very short turns; ignore speakers with tiny total time
    Short turns, backchannels "Yeah", "right" missed or absorbed by the main speaker Margin check above; clip purity audits
    Similar voices Two hosts merged into one label Compare speaker-count distribution with metadata; set min_speakers only when known
    One voice split in two Same person labelled twice after a mic change, phone segment or shouting Compare speaker_embeddings within a file; merge only after checking a sample
    Inserted clips, voice-overs Extra speakers from ads, trailers, quoted videos Keep only dominant speakers for TTS

    A quick sanity check: plot n_speakers per source. If a solo-commentary channel shows four speakers in many files, the cause is usually music or noise, not guests.

    Compliance

    If the audio comes from YouTube, its Terms of Service restrict automated access and downloading, and copyright and licensing apply to the recordings. Some teams restrict collection to Creative Commons material. Voices are personal data under GDPR and similar laws. Speaker embeddings are designed to tell people apart, so some jurisdictions may treat them as biometric data, and hosted pipelines send the audio to a third party. Keep provenance per file, honor takedown and opt-out requests, and get legal review for your jurisdiction and use case.

    If you'd rather not run the collection side yourself

    Corpora's managed service delivers YouTube audio, subtitles and metadata as JSONL/Parquet with sha256 checksums and manifests to S3, GCS or Azure Blob, starting with a free sample batch. Diarization and transcription stay on your GPUs. Teams running their own yt-dlp stack (see audio-only downloads for speech datasets) can use dedicated unmetered proxy capacity instead.

    FAQ

    Is pyannote speaker-diarization-3.1 still usable?

    Yes. It's MIT-licensed, but the README labels it legacy. In the published benchmark, community-1 matches or beats it everywhere except REPERE, and adds exclusive diarization. When upgrading code to 4.x, change use_auth_token to token and handle the DiarizeOutput return type.

    Why do I get a 401 or "gated repo" error?

    The files are gated. Accept the user conditions on the model page with the same Hugging Face account that owns the token, then pass it with token=. Accepting with a personal account and using a CI account's token produces the same error.

    How do I keep speaker labels consistent across files?

    pyannote doesn't; labels are per file. Cluster output.speaker_embeddings across files or compare them with enrolled reference voices, and validate on a labelled sample. A false merge across files does more damage to TTS than a missed one.

    What is a good DER?

    It depends on the domain and the collar and overlap settings. pyannote's community-1 numbers run from about 9% (REPERE) to about 47% (Ego4D). Set a target from the downstream use: confusion matters most for TTS purity, miss and false alarm matter most for hour counts.

  • Speech Training Data: What Makes Audio ASR- and TTS-Ready

    "Ten thousand hours of English speech" can mean ten thousand hours of clean two-person conversation, or ten thousand hours of video files in which speech fills half the time under a music bed. For ASR, TTS and speech-LLM training those are different datasets. This article lists the properties that decide whether audio is actually usable, how each one is measured automatically, and which thresholds matter for which kind of model.

    TL;DR

    • Count speech hours, not file hours. Voice-activity detection turns "hours of video" into the number that matters.
    • Verify language from the audio, not from titles or channel country.
    • ASR wants diversity; TTS wants cleanliness. The same source can be excellent for one and useless for the other.
    • Background music is the most common silent killer — and the easiest to miss by reading metadata.
    • Store measurements per file, so each model can apply its own thresholds.

    The properties that matter

    Property How it is measured Why it matters
    Speech ratio Voice-activity detection (VAD): fraction of time with speech Converts file hours into speech hours; low values mean music, silence or ambience
    Spoken language Audio language identification on speech segments, with a confidence Titles and channel metadata are often wrong or mixed-language
    Speaker count Speaker diarization or speaker-embedding clustering Conversation models need ≥2 speakers; TTS usually needs exactly one per segment
    Music and noise Audio-event classifiers (AudioSet-style labels) per window Background music under speech hurts TTS badly and ASR noticeably
    Audio quality Non-intrusive quality estimators (MOS predictors), signal-to-noise estimates, bandwidth/clipping checks Low-bitrate, clipped or band-limited audio caps achievable quality
    Overlap Diarization overlap regions Overlapping speech is realistic for ASR, but must be removed for TTS
    Transcript availability Manual vs auto-generated captions; or your own ASR Decides whether you need a transcription pass and how trustworthy labels are

    Speech ratio: the number behind "hours"

    A podcast may be 95% speech. A vlog may be 40%. A music video with a spoken intro may be 5%. If you buy or collect by file duration, your effective speech hours can be half of what you paid for. Run a VAD (silero-vad and WebRTC VAD are common, lightweight choices that run fine on CPU) and report both numbers in every manifest: duration_s and speech_s.

    Speech ratio is also a cheap first filter. Requiring ≥0.7–0.8 removes most music, gameplay and ambience content before any expensive analysis.

    Language: trust the audio

    Video metadata language is optional, frequently wrong, and describes the title rather than the speech. Bilingual channels, dubbed audio tracks and code-switching speakers are common. Run language identification on the detected speech segments and keep the confidence. A threshold such as ≥0.9 on the dominant language is a reasonable starting point; mixed-language content can be kept separately if your model benefits from code-switching data.

    If your source offers several audio tracks per video, select the original-language track explicitly rather than whatever is default.

    Speakers: count them, don't identify them

    You rarely need to know who is speaking, only how many and when:

    • Conversational ASR and speech-LLMs benefit from two or more speakers with natural turn-taking.
    • TTS and voice cloning need segments with exactly one speaker, consistent voice and no overlap.
    • Speaker diversity across the dataset matters for generalization: cap the hours per channel so a few prolific creators don't dominate.

    Full diarization pipelines (segmentation plus speaker embeddings plus clustering) give the best counts. Lighter approaches that cluster speaker embeddings on short windows are cheaper but tend to undercount, merging similar voices into one speaker. Sample several positions of long recordings instead of one window, and count only speakers with a meaningful share of speech so a single interjection doesn't turn a monologue into a "conversation".

    Background music: the silent killer

    Commentary, vlogs, tutorials and drama content very often run music under speech. It is invisible in metadata and only partly visible in auto-generated captions. Audio-event classifiers trained on AudioSet-style labels give a per-window music probability; two derived ratios are useful:

    • Music ratio: share of windows with music present, including music under speech.
    • Pure-music ratio: share of windows with music and no speech — intros, interludes, songs.

    For TTS, require music ratio near zero. For ASR, a low level of background music can be acceptable or even useful for robustness, but keep it measured and controlled rather than accidental.

    Quality: what you can't fix later

    Platform audio is lossy (typically Opus or AAC at moderate bitrates), which is fine for ASR and acceptable for many TTS uses, but source quality varies far more than codec quality. Check for:

    • Effective bandwidth (audio upsampled from a phone call sounds full-band in the container but isn't).
    • Clipping and heavy compression, common in livestream recordings.
    • Reverberant rooms and distant microphones.
    • Predicted MOS from a non-intrusive quality model, as a single sortable score.

    Keep native audio as delivered, and convert to your training format (for example 16 kHz mono) as a separate step. Re-encoding to another lossy format at collection time only loses quality.

    ASR versus TTS: different thresholds

    Property ASR / speech-LLM TTS / voice
    Speech ratio ≥0.7–0.8 ≥0.8, then segment to speech only
    Speakers Diverse; multi-speaker welcome One per segment, no overlap
    Background music Low, controlled None
    Noise and reverberation Some variety helps robustness As clean as possible
    Speaking style Spontaneous and read, many accents Consistent, expressive, well articulated
    Transcripts Needed; manual captions or high-quality ASR Needed and must be exact, including punctuation

    Scale expectations differ too. OpenAI's Whisper was trained on about 680,000 hours of weakly supervised audio, which shows how far quantity and diversity can carry ASR. TTS systems are usually trained on far fewer, far cleaner hours, and quality filters matter more than volume.

    Labels: captions, ASR and segmentation

    Manual captions are the best free labels, but cover only a minority of videos; auto-generated captions are useful but contain recognition errors and rolling duplicated lines. Many pipelines re-transcribe with a strong ASR model and keep platform captions as a cross-check: large disagreement flags bad audio or wrong language. Segment long recordings at VAD boundaries into utterances (typically a few seconds to ~30 seconds) and keep segment timestamps in the manifest. For the caption download side see downloading subtitles and auto-captions.

    Measure on a sample before you collect everything

    All of the measurements above can run on short segments of audio rather than whole files. Probing a minute or so at several positions is enough to reject most unsuitable videos before downloading them, which is the approach behind Corpora Curation's audio probe. Store every measurement per file:

    {
      "video_id": "abc123XYZ00",
      "duration_s": 1843,
      "speech_ratio": 0.93,
      "lang": "en",
      "lang_prob": 0.98,
      "n_speakers": 2,
      "music_ratio": 0.02,
      "pure_music_ratio": 0.0,
      "probe_positions": [0.15, 0.45, 0.75],
      "probe_version": 3
    }

    An ASR team and a TTS team can then select different subsets from the same collection without re-analysing anything.

    Compliance

    Voices are personal data under GDPR and similar laws, and voice cloning raises additional consent and misuse issues. YouTube's Terms of Service restrict automated access and downloading, and copyright applies to the recordings. Measure speaker counts without identifying speakers, keep provenance per file, honor removal requests, and get legal review for your jurisdiction and use case.

    If you'd rather receive data already filtered

    Corpora delivers YouTube speech audio filtered by language, speech ratio, speaker count and music, with every measurement in the manifest and an audited precision per batch. See pricing or talk to sales.

    FAQ

    How many hours of speech do I need to train ASR?

    It depends on the language, domain and whether you fine-tune a pretrained model. Large general-purpose models were trained on hundreds of thousands of hours; fine-tuning for a domain can work with far less. Measure speech hours, not file hours, whatever the target.

    Is YouTube audio good enough for TTS?

    Some of it is. Single-speaker, music-free, well-recorded content such as lectures, audiobook-style readings and studio podcasts can work; most vlogs and commentary cannot. Filter on music ratio, speaker count and quality first.

    Can I detect background music automatically?

    Yes. Audio-event classifiers trained on AudioSet-style labels give per-window music probabilities; aggregate them into a music ratio and a pure-music ratio per file.

    Should I trust the language in video metadata?

    No. Run audio language identification on the speech segments and keep the confidence score.