Speech Training Data: What Makes Audio ASR- and TTS-Ready

Written by

in

"Ten thousand hours of English speech" can mean ten thousand hours of clean two-person conversation, or ten thousand hours of video files in which speech fills half the time under a music bed. For ASR, TTS and speech-LLM training those are different datasets. This article lists the properties that decide whether audio is actually usable, how each one is measured automatically, and which thresholds matter for which kind of model.

TL;DR

  • Count speech hours, not file hours. Voice-activity detection turns "hours of video" into the number that matters.
  • Verify language from the audio, not from titles or channel country.
  • ASR wants diversity; TTS wants cleanliness. The same source can be excellent for one and useless for the other.
  • Background music is the most common silent killer — and the easiest to miss by reading metadata.
  • Store measurements per file, so each model can apply its own thresholds.

The properties that matter

Property How it is measured Why it matters
Speech ratio Voice-activity detection (VAD): fraction of time with speech Converts file hours into speech hours; low values mean music, silence or ambience
Spoken language Audio language identification on speech segments, with a confidence Titles and channel metadata are often wrong or mixed-language
Speaker count Speaker diarization or speaker-embedding clustering Conversation models need ≥2 speakers; TTS usually needs exactly one per segment
Music and noise Audio-event classifiers (AudioSet-style labels) per window Background music under speech hurts TTS badly and ASR noticeably
Audio quality Non-intrusive quality estimators (MOS predictors), signal-to-noise estimates, bandwidth/clipping checks Low-bitrate, clipped or band-limited audio caps achievable quality
Overlap Diarization overlap regions Overlapping speech is realistic for ASR, but must be removed for TTS
Transcript availability Manual vs auto-generated captions; or your own ASR Decides whether you need a transcription pass and how trustworthy labels are

Speech ratio: the number behind "hours"

A podcast may be 95% speech. A vlog may be 40%. A music video with a spoken intro may be 5%. If you buy or collect by file duration, your effective speech hours can be half of what you paid for. Run a VAD (silero-vad and WebRTC VAD are common, lightweight choices that run fine on CPU) and report both numbers in every manifest: duration_s and speech_s.

Speech ratio is also a cheap first filter. Requiring ≥0.7–0.8 removes most music, gameplay and ambience content before any expensive analysis.

Language: trust the audio

Video metadata language is optional, frequently wrong, and describes the title rather than the speech. Bilingual channels, dubbed audio tracks and code-switching speakers are common. Run language identification on the detected speech segments and keep the confidence. A threshold such as ≥0.9 on the dominant language is a reasonable starting point; mixed-language content can be kept separately if your model benefits from code-switching data.

If your source offers several audio tracks per video, select the original-language track explicitly rather than whatever is default.

Speakers: count them, don't identify them

You rarely need to know who is speaking, only how many and when:

  • Conversational ASR and speech-LLMs benefit from two or more speakers with natural turn-taking.
  • TTS and voice cloning need segments with exactly one speaker, consistent voice and no overlap.
  • Speaker diversity across the dataset matters for generalization: cap the hours per channel so a few prolific creators don't dominate.

Full diarization pipelines (segmentation plus speaker embeddings plus clustering) give the best counts. Lighter approaches that cluster speaker embeddings on short windows are cheaper but tend to undercount, merging similar voices into one speaker. Sample several positions of long recordings instead of one window, and count only speakers with a meaningful share of speech so a single interjection doesn't turn a monologue into a "conversation".

Background music: the silent killer

Commentary, vlogs, tutorials and drama content very often run music under speech. It is invisible in metadata and only partly visible in auto-generated captions. Audio-event classifiers trained on AudioSet-style labels give a per-window music probability; two derived ratios are useful:

  • Music ratio: share of windows with music present, including music under speech.
  • Pure-music ratio: share of windows with music and no speech — intros, interludes, songs.

For TTS, require music ratio near zero. For ASR, a low level of background music can be acceptable or even useful for robustness, but keep it measured and controlled rather than accidental.

Quality: what you can't fix later

Platform audio is lossy (typically Opus or AAC at moderate bitrates), which is fine for ASR and acceptable for many TTS uses, but source quality varies far more than codec quality. Check for:

  • Effective bandwidth (audio upsampled from a phone call sounds full-band in the container but isn't).
  • Clipping and heavy compression, common in livestream recordings.
  • Reverberant rooms and distant microphones.
  • Predicted MOS from a non-intrusive quality model, as a single sortable score.

Keep native audio as delivered, and convert to your training format (for example 16 kHz mono) as a separate step. Re-encoding to another lossy format at collection time only loses quality.

ASR versus TTS: different thresholds

Property ASR / speech-LLM TTS / voice
Speech ratio ≥0.7–0.8 ≥0.8, then segment to speech only
Speakers Diverse; multi-speaker welcome One per segment, no overlap
Background music Low, controlled None
Noise and reverberation Some variety helps robustness As clean as possible
Speaking style Spontaneous and read, many accents Consistent, expressive, well articulated
Transcripts Needed; manual captions or high-quality ASR Needed and must be exact, including punctuation

Scale expectations differ too. OpenAI's Whisper was trained on about 680,000 hours of weakly supervised audio, which shows how far quantity and diversity can carry ASR. TTS systems are usually trained on far fewer, far cleaner hours, and quality filters matter more than volume.

Labels: captions, ASR and segmentation

Manual captions are the best free labels, but cover only a minority of videos; auto-generated captions are useful but contain recognition errors and rolling duplicated lines. Many pipelines re-transcribe with a strong ASR model and keep platform captions as a cross-check: large disagreement flags bad audio or wrong language. Segment long recordings at VAD boundaries into utterances (typically a few seconds to ~30 seconds) and keep segment timestamps in the manifest. For the caption download side see downloading subtitles and auto-captions.

Measure on a sample before you collect everything

All of the measurements above can run on short segments of audio rather than whole files. Probing a minute or so at several positions is enough to reject most unsuitable videos before downloading them, which is the approach behind Corpora Curation's audio probe. Store every measurement per file:

{
  "video_id": "abc123XYZ00",
  "duration_s": 1843,
  "speech_ratio": 0.93,
  "lang": "en",
  "lang_prob": 0.98,
  "n_speakers": 2,
  "music_ratio": 0.02,
  "pure_music_ratio": 0.0,
  "probe_positions": [0.15, 0.45, 0.75],
  "probe_version": 3
}

An ASR team and a TTS team can then select different subsets from the same collection without re-analysing anything.

Compliance

Voices are personal data under GDPR and similar laws, and voice cloning raises additional consent and misuse issues. YouTube's Terms of Service restrict automated access and downloading, and copyright applies to the recordings. Measure speaker counts without identifying speakers, keep provenance per file, honor removal requests, and get legal review for your jurisdiction and use case.

If you'd rather receive data already filtered

Corpora delivers YouTube speech audio filtered by language, speech ratio, speaker count and music, with every measurement in the manifest and an audited precision per batch. See pricing or talk to sales.

FAQ

How many hours of speech do I need to train ASR?

It depends on the language, domain and whether you fine-tune a pretrained model. Large general-purpose models were trained on hundreds of thousands of hours; fine-tuning for a domain can work with far less. Measure speech hours, not file hours, whatever the target.

Is YouTube audio good enough for TTS?

Some of it is. Single-speaker, music-free, well-recorded content such as lectures, audiobook-style readings and studio podcasts can work; most vlogs and commentary cannot. Filter on music ratio, speaker count and quality first.

Can I detect background music automatically?

Yes. Audio-event classifiers trained on AudioSet-style labels give per-window music probabilities; aggregate them into a music ratio and a pure-music ratio per file.

Should I trust the language in video metadata?

No. Run audio language identification on the speech segments and keep the confidence score.