Tag: training data curation

  • Measuring Curated Dataset Accuracy: Audits and Wilson Intervals

    "95% accurate" is the most common claim about a curated dataset and the least checked. Whether you filter data yourself or buy it, precision — the share of delivered items that truly meet the spec — should be a measured number with an error bar, not an adjective. This article shows how to audit a dataset with a small human-labeled sample, compute an honest confidence interval, and turn it into acceptance rules for each batch.

    TL;DR

    • Audit a random sample, label it by hand, report precision with a confidence interval.
    • Use the Wilson interval, not the textbook normal approximation; it behaves correctly near 100% and with small samples.
    • 50 items gives a rough estimate; 200 gives a usable one. Halving the error bar takes about four times as many labels.
    • Accept a batch on the lower bound, not the point estimate.
    • Calibrate with a stratified sample (passed, borderline, rejected) to tune filters; audit deliveries with a plain random sample.

    What exactly are you measuring?

    For a filtered dataset, two numbers matter and they are easy to confuse:

    • Precision: of the items delivered, what share actually meet the spec? This is what the buyer of a dataset cares about.
    • Recall: of the items that meet the spec among all candidates, what share did the filter keep? This decides cost and speed, because low recall throws away good data you paid to discover.

    Precision is measured by labeling a sample of accepted items. Recall needs labels on rejected items too, which is why calibration samples look different from delivery audits.

    Write the label definition first

    An audit is only as good as its definition. "Natural conversation" is ambiguous; "two or more people talking unscripted for most of the clip, not a single narrator, not mostly music" is labelable. Write the definition down, include borderline examples with the intended answer, and give labelers a third option — unsure — rather than forcing a guess. Track unsure items separately; if they exceed a few percent, the definition needs work, not the filter.

    If two labelers are available, have them overlap on part of the sample. Disagreement between people is the ceiling on how accurate any filter can be measured to be.

    The confidence interval: use Wilson

    If you audit n items and k pass, the point estimate is p = k / n. The interval most people learn — p ± 1.96 × √(p(1−p)/n) — breaks down exactly where data quality numbers live: near 100%, and with small samples. Audit 50 items, find 50 correct, and it claims an interval of [1.00, 1.00], which is absurd.

    The Wilson score interval fixes this:

    z      = 1.96                       # 95% confidence
    center = (p + z²/(2n)) / (1 + z²/n)
    half   = z · √( p(1−p)/n + z²/(4n²) ) / (1 + z²/n)
    interval = [center − half, center + half]

    In Python:

    import math
    
    def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
        if n == 0:
            return (0.0, 1.0)
        p = k / n
        d = 1 + z * z / n
        c = (p + z * z / (2 * n)) / d
        h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
        return (max(0.0, c - h), min(1.0, c + h))
    
    print(wilson(45, 50))   # (0.786..., 0.956...)

    How big should the sample be?

    Same observed precision of 90%, different sample sizes:

    Audited Correct Point estimate 95% Wilson interval
    50 45 0.900 0.786 – 0.957
    100 90 0.900 0.826 – 0.945
    200 180 0.900 0.851 – 0.934
    500 450 0.900 0.871 – 0.923

    And at the top end: 50 out of 50 correct gives an interval of 0.929 – 1.000. A perfect small audit still only tells you precision is probably above 93%.

    Two practical consequences:

    • Error bars shrink with the square root of sample size. Going from 50 to 200 labels roughly halves the width; going to 800 halves it again.
    • The sample size you need doesn't depend on dataset size. 200 random items measure a 10,000-item batch and a 1,000,000-item batch about equally well. That is what makes auditing large datasets affordable.

    A common rule for per-batch audits is max(50, 1% of the batch), with a larger one-off sample (around 200) when a new spec is first calibrated.

    Calibration samples vs delivery audits

    They answer different questions, so they are sampled differently:

    Calibration (before running at volume) Delivery audit (every batch)
    Goal Tune thresholds, find which filter layer makes which mistakes Estimate precision of what was delivered
    Sample Stratified: accepted, borderline rejected, clearly rejected Simple random sample of delivered items
    Typical size ~200 max(50, 1%)
    Output Precision and recall per layer, error examples for the definition Precision with a 95% interval, pass/fail against target

    A stratified calibration sample oversamples borderline cases on purpose, so its raw pass rate is not an estimate of delivered precision. Report precision from the accepted stratum, and use the rejected strata to estimate how much good data the filters discard.

    Turning it into acceptance rules

    Decide before you see the numbers:

    1. Set a precision target, for example 0.90, as part of the spec.
    2. Accept a batch if the lower bound of the 95% interval is at or above an agreed floor — for example the target minus a tolerance, since even a filter that is exactly at target will produce a lower bound below it on a modest sample.
    3. If it fails: pause the job, inspect the failed audit items, fix the definition or thresholds, and either top up the batch with replacement items or adjust billing by the measured precision. Agree in advance which.
    4. Watch for drift. Track the interval batch by batch. Precision usually decays slowly as discovery reaches less typical sources; a falling trend is a signal before a batch actually fails.

    Common pitfalls

    • Hand-picked "samples". If the vendor or the pipeline chooses which items are audited, the number is meaningless. Sample randomly, with a recorded seed.
    • Labeling with the model's output visible. Showing labelers the classifier's score biases them toward agreeing. Hide it.
    • Auditing on metadata only. If the spec is about audio (speakers, music), labelers must listen. See what makes audio ASR- and TTS-ready.
    • Reusing calibration labels as audit labels. They were used to tune the filters, so they overstate precision.
    • Reporting one number for a mixed dataset. If a delivery mixes languages or categories, audit and report per segment that matters to you.

    How Corpora uses this

    Corpora Curation calibrates every job with human labels before it runs at volume, and every delivery batch carries a random audit sample and a 95% Wilson interval in its batch report, so you can check precision against your target yourself. The same method works for any vendor's data — see buying video training data: a checklist — and for your own pipelines. Talk to sales if you want a calibrated sample for your spec.

    FAQ

    How many samples do I need to measure dataset accuracy?

    About 200 randomly sampled items give a 95% interval roughly ±4 percentage points around a 90% precision; 50 give a rough check (about ±8 points). The size of the dataset itself barely matters.

    Why not use the normal-approximation interval?

    It is wrong near 0% and 100% and with small samples: a perfect audit produces a zero-width interval. The Wilson interval stays meaningful in those cases.

    Should I accept a batch on the point estimate?

    No. Use the lower bound of the interval against an agreed floor, so a lucky small sample can't hide a weak batch.

    What is the difference between precision and recall for a filtered dataset?

    Precision is the share of delivered items that meet the spec. Recall is the share of qualifying candidates the filter kept. Buyers mostly care about precision; recall drives cost.

  • Introducing Corpora Curation: YouTube Data Filtered to Spec

    Downloading YouTube at scale is a solved problem. Finding the right videos is not. "Natural two-person English conversation, no background music" or "live-action vertical short dramas" are easy to say and slow to find: search results drift off-topic after a few pages, titles lie, and most candidates fail on things you can only hear or see. Corpora Curation is the system we built to do that search continuously and deliver only what matches your spec. It is now open for early-access projects.

    TL;DR

    • You describe what you need in plain language; it becomes a structured, versioned filter spec that we review with you.
    • Discovery runs continuously across search, channels, playlists and channel feeds, and shifts effort toward the sources that actually produce matches.
    • Candidates pass a funnel from cheap to expensive: metadata rules, content classification, then an audio probe that measures speech, language, speaker count and music.
    • Accuracy is measured, not promised: a human-labeled calibration set before the job runs, and an audit sample with a confidence interval in every batch.
    • Delivery is metadata-only by default (video IDs plus structured metadata and scores), or media files when you need them.

    The problem it solves

    Teams building speech, audio and video models usually don't struggle with bandwidth. They struggle with selection:

    • Keyword search returns a few hundred relevant results, then drifts. The good material lives in specific channels and playlists you have to discover.
    • Metadata is unreliable. A video titled "podcast" can be a single narrator over music; a "short drama" can be a recap channel narrating someone else's footage.
    • The requirements that matter most — two or more speakers, little background music, a specific spoken language — are properties of the audio, not the text.
    • Checking everything by downloading it is slow and expensive, and checking a sample by hand doesn't scale past a few thousand videos.

    Curation is a pipeline built around that reality: spend almost nothing on candidates that obviously fail, and save the expensive checks for the ones that might pass. It is the same principle we describe in video data curation for generative models, applied before download instead of after.

    How a job works

    1. Requirement → spec

    You describe the data in plain language, optionally with a few example videos that should and should not qualify. The system drafts a structured spec: seed search queries in the target language, hard rules (duration, Shorts or not, live streams, category), a written definition of positives and negatives for content classification, audio thresholds, the target count and the delivery format. We review and adjust it with you before anything runs. Every change creates a new spec version, and every decision records which version made it.

    A simplified spec for conversational English speech looks like this:

    target: { count: 20000 }
    rules:
      - "duration_s between 300 and 3600"
      - "not is_live and not is_short"
    stages:
      - type: text_cls
        definition: |
          Positive: natural, unscripted conversation between two or more people
          (podcasts, interviews, casual talk, street interviews).
          Negative: single-narrator monologue, audiobooks, news reading, songs,
          music-heavy vlogs, gameplay commentary, ASMR.
      - type: audio_probe
        require:
          lang: en
          speech_ratio: ">= 0.8"
          n_speakers: ">= 2"
          music_ratio: "<= 0.1"
    download:
      mode: metadata

    2. Continuous discovery

    Discovery expands search queries, channels, playlists and channel feeds, and treats each one as a source with its own track record. Sources that produce qualifying videos get more of the crawl budget; sources that don't are paused. When a channel produces matches, its other videos become candidates too, which is usually where the best material is.

    3. The filter funnel

    Layer What it checks Cost per candidate
    List metadata rules Duration, Shorts, live status, upload date — from listing pages, before any per-video request Lowest
    Detail metadata rules Category, license, available caption and audio languages, resolution and aspect ratio Low
    Content classification Title, description, tags and channel judged against your written definition by a language model Low–medium
    Audio probe Short audio segments analysed on CPU: speech ratio, spoken language, estimated speaker count, music and instrumental ratio Highest — only for candidates that passed everything else

    The audio probe is what makes speech requirements enforceable. It samples segments rather than the whole file (several positions for long videos), runs voice-activity detection, language identification, speaker diarization and audio-event classification, and records every value. A video that passes on text but turns out to be one narrator over a music bed is rejected here, before it costs a download.

    If your requirement is purely about topic or text, the audio layer can be skipped entirely and jobs run much faster.

    4. Calibration and audits

    Before a job runs at volume, it runs in calibration on a small number of candidates, and a sample of its decisions is labeled by a person. That gives a measured precision with a 95% confidence interval for the spec, and shows exactly which layer is making which mistakes. The spec is adjusted until the measured precision meets the target you set.

    After that, every delivery batch includes an audit sample and the same interval. If you want to understand the statistics behind it, see measuring curated dataset accuracy.

    5. Delivery

    Results arrive in batches while the job runs, not all at once at the end:

    • Metadata mode (default): video IDs with structured metadata and every score — no media download, so it is fast and you collect the files on your own infrastructure.
    • Media mode: audio or video files in the format your spec requests, for example 16 kHz mono FLAC for speech, with checksums.
    • Every batch includes a manifest.jsonl (one record per video with the scores that qualified it), a batch_report.json (funnel counts per layer, durations, audit results) and the audit sample.
    • No duplicates for you: a video delivered to you once is never delivered to you again in a later batch or another job.

    A manifest record looks like this:

    {
      "video_id": "abc123XYZ00",
      "url": "https://www.youtube.com/watch?v=abc123XYZ00",
      "channel_id": "UC...",
      "title": "...",
      "duration_s": 1843,
      "lang": "en",
      "license": "youtube",
      "scores": { "text_cls": 0.91 },
      "features": { "speech_ratio": 0.93, "n_speakers": 2, "music_ratio": 0.02 },
      "spec_version": 3,
      "accepted_at": "2026-11-06T12:41:09Z"
    }

    What it is good at today

    • Speech and conversation data by language, speaker count and amount of background music — see what makes audio ASR- and TTS-ready for the criteria that matter.
    • Genre and format selection that can be judged from text and audio: interviews, lectures, tutorials, podcasts, music versus speech, specific content categories.
    • License-constrained jobs, for example Creative Commons only, with the license recorded per video (see Creative Commons YouTube videos for AI training).

    Requirements that can only be judged visually (live-action versus animation, on-screen layout, camera style) currently rely on text signals and are less precise; frame-level visual checks are next on the roadmap. Very subjective definitions ("interesting conversations") are hard to hold at a stable accuracy, which is exactly what calibration exposes before you pay for volume.

    Pricing and early access

    Curation is priced per qualified item delivered, with the price depending on duration range and on which layers your spec needs (text only, or text plus audio). We are taking a limited number of early-access projects. Send us the kind of data you need and a rough volume; we will draft the spec, run calibration and show you the measured precision and a sample batch before you commit. Contact sales.

    Compliance

    YouTube's Terms of Service restrict automated access and downloading, and copyright, licensing and privacy law apply to what is collected — voices and faces can be personal data. Curation records the source, license field and collection time for every item, so any removal request can be traced to the batches it appears in, and it does not attempt to identify the people in the content. Customers are responsible for their own legal review of their intended use.

    FAQ

    How is this different from buying a fixed dataset?

    A fixed dataset was selected for someone else's model. Curation selects against your spec, keeps running until it reaches your target count, and measures its own precision on your definition.

    Do I have to receive media files?

    No. The default is metadata only: IDs, metadata and scores. Many teams prefer to download on their own infrastructure.

    How accurate is it?

    It depends on how objective your requirement is, which is why every job is calibrated with human labels first and every batch carries an audit sample with a 95% confidence interval.

    Can the spec change while the job is running?

    Yes. A change creates a new spec version; decisions record the version that made them, so you can see exactly which items qualified under which rules.