Speaker Diarization with pyannote for Speech Datasets

Written by

in

A speech dataset scraped from podcasts, interviews and talk shows is useless for TTS until you know who is speaking when, and only half useful for ASR. Speaker diarization answers that question, and pyannote.audio is the open-source default for it. This guide covers what diarization outputs, which pyannote pipeline to use as of pyannote.audio 4.1, how to measure DER, and how to turn the output into single-speaker clips and speaker-attributed Whisper transcripts.

TL;DR

  • Pipeline: pyannote/speaker-diarization-community-1 on pyannote.audio 4.x. speaker-diarization-3.1 is now labelled legacy. Both are gated on Hugging Face: accept the conditions, then pass a token.
  • Output: (start, end, speaker) turns, saved as RTTM. Labels like SPEAKER_00 mean nothing outside their file.
  • Two outputs since 4.0: speaker_diarization keeps overlapping speech; exclusive_speaker_diarization allows one speaker at a time, which is what you want for aligning words.
  • For TTS, keep only turns with no overlap and no other speaker within a safety margin.
  • Measure DER on hand-labelled files from your own sources. pyannote's published numbers range from under 10% to nearly 50% depending on the domain.
  • Plan GPU time with a formula and a speed you measured yourself.

What speaker diarization outputs

Diarization splits a recording into speech turns and gives each turn an anonymous speaker label. It does not identify people. pyannote writes the result as RTTM:

SPEAKER ep0412 1 6.665 4.120 <NA> <NA> SPEAKER_00 <NA> <NA>
SPEAKER ep0412 1 10.580 2.310 <NA> <NA> SPEAKER_01 <NA> <NA>
SPEAKER ep0412 1 12.400 0.910 <NA> <NA> SPEAKER_00 <NA> <NA>

Field 2 is the file URI, field 4 the start in seconds, field 5 the duration (not the end time) and field 8 the speaker. Annotation.write_rttm() in pyannote.core rejects URIs and labels that contain spaces, because the format is space-separated. Two things to remember: turns can overlap when people talk at once, and labels are per file. SPEAKER_00 in one episode has nothing to do with SPEAKER_00 in the next.

Which pyannote pipeline to use

pyannote.audio 4.1.0 was released on PyPI on 17 November 2026 and requires Python 3.11 or newer. Version 4.0.0 (September 2025) introduced community-1, which switched from agglomerative clustering to VBx clustering and added exclusive diarization. The pyannote-audio README lists three pipelines:

Pipeline Runs License Notes
speaker-diarization-community-1 Locally CC-BY-4.0 Regular and exclusive diarization, plus per-speaker embeddings
speaker-diarization-3.1 Locally MIT Legacy; mono 16 kHz input, downmixed and resampled automatically
speaker-diarization-precision-2 pyannoteAI servers Commercial pyannoteAI API key; audio leaves your infrastructure

A selection from the README's benchmark (DER %, updated 2025-09). These are the maintainers' own numbers:

Dataset 3.1 (legacy) community-1
VoxConverse v0.3 11.2 11.2
AMI (IHM) 18.8 17.0
DIHARD 3 (full) 21.4 20.2
AVA-AVD 49.7 44.6
Ego4D (dev.) 51.2 46.8

VoxConverse is built from multi-speaker clips extracted from YouTube videos and scores far better than movies (AVA-AVD) or egocentric video (Ego4D). Your error rate will depend more on your source mix than on the pipeline version.

Setup and gated access

Install ffmpeg (since 4.0, audio is decoded by torchcodec; the sox and soundfile backends are gone), then pip install pyannote.audio. On Hugging Face, accept the user conditions on the community-1 page: the repository is public, but its files are gated. Create an access token and keep it in an environment variable. Older tutorials stop working for two reasons: 4.0 renamed use_auth_token= to token= and replaced name@revision with a revision= argument. For air-gapped nodes, the changelog shows how to git clone the pipeline repo with git-lfs and pass the local path to Pipeline.from_pretrained(). Optional usage telemetry can be turned off with PYANNOTE_METRICS_ENABLED=0.

The Python API

import os
import torch
from pyannote.audio import Pipeline

pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-community-1",
    token=os.environ["HF_TOKEN"],
)
pipeline.to(torch.device("cuda"))

output = pipeline("ep0412.wav")                      # speaker count estimated
output = pipeline("ep0412.wav", num_speakers=2)      # exactly 2
output = pipeline("ep0412.wav", min_speakers=1, max_speakers=6)

for turn, speaker in output.speaker_diarization:
    print(f"{turn.start:.2f} {turn.end:.2f} {speaker}")

Details from SpeakerDiarization.apply in the source:

  • min_speakers/max_speakers are ignored when num_speakers is set. If the detected count falls outside the bounds, the pipeline warns instead of failing.
  • It returns a DiarizeOutput with speaker_diarization, exclusive_speaker_diarization and speaker_embeddings (one row per speaker, in labels() order). The legacy 3.1 pipeline returns a plain Annotation.
  • Unknown keyword arguments only produce a warning, so a typo like max_speaker is silently ignored.
  • Audio you have already decoded can be passed as {"waveform": tensor, "sample_rate": sr}.

Only fix a speaker count you can trust. A podcast with "2 hosts" in its description also has guests, ad reads and clips, so loose bounds usually beat a confident wrong number.

A worker that writes RTTM and a manifest

# diarize_worker.py
# usage: HF_TOKEN=... python diarize_worker.py files.txt out_dir
import json
import os
import sys
import time
from pathlib import Path

import torch
from pyannote.audio import Pipeline

PIPELINE = "pyannote/speaker-diarization-community-1"
pipeline = Pipeline.from_pretrained(PIPELINE, token=os.environ["HF_TOKEN"])
pipeline.to(torch.device("cuda"))


def diarize(audio: Path, out_dir: Path) -> dict:
    uri = audio.stem  # no spaces allowed in RTTM
    t0 = time.monotonic()
    out = pipeline(str(audio), min_speakers=1, max_speakers=8)
    wall = time.monotonic() - t0
    diar, excl = out.speaker_diarization, out.exclusive_speaker_diarization
    diar.uri = excl.uri = uri
    with (out_dir / f"{uri}.rttm").open("w") as f:
        diar.write_rttm(f)
    with (out_dir / f"{uri}.exclusive.rttm").open("w") as f:
        excl.write_rttm(f)
    return {
        "audio_id": uri,
        "pipeline": PIPELINE,
        "n_speakers": len(diar.labels()),
        "speech_s": round(diar.get_timeline().support().duration(), 3),
        "overlap_s": round(diar.get_overlap().duration(), 3),
        "speaker_s": {spk: round(d, 3) for spk, d in diar.chart()},
        "wall_s": round(wall, 3),
    }


if __name__ == "__main__":
    files, out_dir = Path(sys.argv[1]), Path(sys.argv[2])
    out_dir.mkdir(parents=True, exist_ok=True)
    with (out_dir / "manifest.jsonl").open("a", encoding="utf-8") as m:
        for line in files.read_text(encoding="utf-8").splitlines():
            audio = Path(line.strip())
            if not line.strip() or (out_dir / f"{audio.stem}.rttm").exists():
                continue
            try:
                row = diarize(audio, out_dir)
            except Exception as e:
                row = {"audio_id": audio.stem, "error": repr(e)}
            m.write(json.dumps(row) + "\n")
            m.flush()

Overlap ratio, dominant-speaker share and speaker count become cheap filter columns, the same properties discussed in what makes speech data ASR- and TTS-ready. Record library versions alongside the pipeline name, as the provenance guide recommends.

Overlapped speech

pyannote's segmentation model handles overlap directly (the powerset formulation from Plaquet and Bredin, Interspeech 2023). Version 4.0 dropped the separate overlapped-speech-detection pipeline because overlap is part of the diarization output; diar.get_overlap() returns it as a Timeline. What to do with it depends on what you're training:

  • TTS / voice cloning: drop overlapped regions and any clip that touches them.
  • ASR: a transcript holds one word stream, so overlapped words get lost or mixed. Keep overlap only if your labels can represent it.
  • Conversational models: overlap is signal. Keep the regular output.

Selecting single-speaker segments for TTS

A turn with a single label isn't automatically clean. A backchannel ("mm-hm") a few hundred milliseconds away, or a boundary that is slightly off, puts a second voice into the clip. This function keeps a span only if it is long enough once overlap is removed and no other speaker is active within a margin around it:

from pyannote.core import Annotation, Segment


def single_speaker_clips(diar: Annotation, min_dur=2.0, max_dur=20.0,
                         margin=0.5, merge_gap=0.3):
    """Yield (speaker, start, end, needs_split) for clean single-speaker spans."""
    if not diar:
        return
    clean = diar.extrude(diar.get_overlap())       # remove overlapped time
    for spk in clean.labels():
        others = diar.subset([spk], invert=True).get_timeline().support()
        for seg in clean.label_timeline(spk).support(merge_gap):
            if seg.duration < min_dur:
                continue
            guard = Segment(max(0.0, seg.start - margin), seg.end + margin)
            if others.crop(guard).duration() > 0:
                continue                           # another speaker too close
            yield spk, round(seg.start, 3), round(seg.end, 3), seg.duration > max_dur
  • margin trades yield for purity. Listen to 100 random clips at two settings and count contaminated ones.
  • merge_gap joins short pauses. The margin check runs after merging, so an interjection inside a merged span still rejects it.
  • Long spans are flagged, not cut. Split them at Whisper word boundaries so no clip ends mid-word.

Diarization only tells you who spoke. Music beds, noise and clipping need their own filters.

Combining diarization with Whisper or WhisperX timestamps

To attribute a transcript, assign each word to the speaker whose turn overlaps it most. Use the exclusive diarization here: pyannote added it specifically to make it easier to reconcile diarization with "sometimes not so precise" transcription timestamps. With the per-word JSONL from our Whisper at scale guide (fields w, start, end):

from pyannote.core import Annotation, Segment


def label_words(words: list[dict], excl: Annotation) -> list[dict]:
    """Attach the speaker with the longest overlap to each word."""
    for w in words:
        start, end = w["start"], w["end"]
        if end <= start:                       # zero-length word timestamps
            start, end = start - 0.05, end + 0.05
        w["speaker"] = excl.argmax(Segment(max(0.0, start), end))  # None if no overlap
    return words

Then group consecutive same-speaker words into utterances. Words left with None fall between turns. They are often hallucinations or speech the diarizer missed, so check them before dropping.

WhisperX builds this in. Its DiarizationPipeline(token=..., device=...) wraps pyannote (community-1, per its README), and whisperx.assign_word_speakers() labels aligned words. Its README states the limits plainly: overlapping speech isn't handled well, "diarization is far from perfect", and words that can't be aligned, like "2014." or "£13.60", get no timing.

Measuring DER with pyannote.metrics

DER = (false alarm + missed detection + speaker confusion) / total reference speech

Confusion is counted after the optimal one-to-one mapping between hypothesis and reference labels. False alarm is unbounded, so DER can exceed 100%. Always report collar (seconds ignored around reference boundaries) and skip_overlap with the number. Both default to 0.0 and False in pyannote.metrics.

from pyannote.core import Annotation
from pyannote.database.util import load_rttm
from pyannote.metrics.diarization import DiarizationErrorRate

refs = load_rttm("reference.rttm")      # {uri: Annotation}
hyps = load_rttm("hypothesis.rttm")

metric = DiarizationErrorRate(collar=0.0, skip_overlap=False)
for uri, ref in refs.items():
    d = metric(ref, hyps.get(uri, Annotation(uri=uri)), detailed=True)
    print(uri, round(d["diarization error rate"], 3), "miss", round(d["missed detection"], 1),
          "fa", round(d["false alarm"], 1), "conf", round(d["confusion"], 1))

print("overall DER:", round(abs(metric), 3))

If references cover only part of each file, pass a UEM (uem=, loadable with load_uem). The components tell you more than the total. High confusion means speakers were merged or split, which hurts TTS. High miss or false alarm means speech-detection errors, which distort hour counts. Build the reference set from your own sources, stratified by source type, and check clip purity with audit sampling and confidence intervals.

GPU throughput: reason with a formula

Measure s/h, the wall-clock seconds per hour of audio on one GPU, on your own files. Then:

  • GPU-hours = audio_hours × s/h / 3600
  • wall-clock hours = GPU-hours / number_of_GPUs

The README publishes one reference point: community-1 self-hosted on an NVIDIA H100 80GB took 31 s per hour of audio on AMI (~1-hour files) and 37 s on DIHARD 3 (~5-minute files). If 31 s/h held for your data, 100,000 hours would take 100,000 × 31 / 3600 ≈ 861 H100-hours. Use that to size a pilot, not to promise a deadline.

  • Sliding windows: by the constructor default, segmentation windows overlap by 90%, and one embedding is extracted per (window, local speaker slot). Cost grows with duration, and clustering grows with the embedding count, so time very long files separately.
  • Batching: segmentation_batch_size and embedding_batch_size control GPU batches. Raise them while watching VRAM.
  • Decoding: CPU-side ffmpeg decoding can starve the GPU. Decode once and share the waveform with Whisper, or run several workers per GPU.
  • Stacked passes: Whisper, alignment and diarization each add their own s/h.

Common failure modes

Failure What you see Mitigation
Music and singing Speech detected over music beds; a singer becomes a "speaker" Music/speech classifier first; drop low speech-ratio files or regions
Laughter, applause, crowd noise Short false-alarm turns, sometimes a new speaker Drop very short turns; ignore speakers with tiny total time
Short turns, backchannels "Yeah", "right" missed or absorbed by the main speaker Margin check above; clip purity audits
Similar voices Two hosts merged into one label Compare speaker-count distribution with metadata; set min_speakers only when known
One voice split in two Same person labelled twice after a mic change, phone segment or shouting Compare speaker_embeddings within a file; merge only after checking a sample
Inserted clips, voice-overs Extra speakers from ads, trailers, quoted videos Keep only dominant speakers for TTS

A quick sanity check: plot n_speakers per source. If a solo-commentary channel shows four speakers in many files, the cause is usually music or noise, not guests.

Compliance

If the audio comes from YouTube, its Terms of Service restrict automated access and downloading, and copyright and licensing apply to the recordings. Some teams restrict collection to Creative Commons material. Voices are personal data under GDPR and similar laws. Speaker embeddings are designed to tell people apart, so some jurisdictions may treat them as biometric data, and hosted pipelines send the audio to a third party. Keep provenance per file, honor takedown and opt-out requests, and get legal review for your jurisdiction and use case.

If you'd rather not run the collection side yourself

Corpora's managed service delivers YouTube audio, subtitles and metadata as JSONL/Parquet with sha256 checksums and manifests to S3, GCS or Azure Blob, starting with a free sample batch. Diarization and transcription stay on your GPUs. Teams running their own yt-dlp stack (see audio-only downloads for speech datasets) can use dedicated unmetered proxy capacity instead.

FAQ

Is pyannote speaker-diarization-3.1 still usable?

Yes. It's MIT-licensed, but the README labels it legacy. In the published benchmark, community-1 matches or beats it everywhere except REPERE, and adds exclusive diarization. When upgrading code to 4.x, change use_auth_token to token and handle the DiarizeOutput return type.

Why do I get a 401 or "gated repo" error?

The files are gated. Accept the user conditions on the model page with the same Hugging Face account that owns the token, then pass it with token=. Accepting with a personal account and using a CI account's token produces the same error.

How do I keep speaker labels consistent across files?

pyannote doesn't; labels are per file. Cluster output.speaker_embeddings across files or compare them with enrolled reference voices, and validate on a labelled sample. A false merge across files does more damage to TTS than a missed one.

What is a good DER?

It depends on the domain and the collar and overlap settings. pyannote's community-1 numbers run from about 9% (REPERE) to about 47% (Ego4D). Set a target from the downstream use: confusion matters most for TTS purity, miss and false alarm matter most for hour counts.