Tag: data quality

  • Measuring Curated Dataset Accuracy: Audits and Wilson Intervals

    "95% accurate" is the most common claim about a curated dataset and the least checked. Whether you filter data yourself or buy it, precision — the share of delivered items that truly meet the spec — should be a measured number with an error bar, not an adjective. This article shows how to audit a dataset with a small human-labeled sample, compute an honest confidence interval, and turn it into acceptance rules for each batch.

    TL;DR

    • Audit a random sample, label it by hand, report precision with a confidence interval.
    • Use the Wilson interval, not the textbook normal approximation; it behaves correctly near 100% and with small samples.
    • 50 items gives a rough estimate; 200 gives a usable one. Halving the error bar takes about four times as many labels.
    • Accept a batch on the lower bound, not the point estimate.
    • Calibrate with a stratified sample (passed, borderline, rejected) to tune filters; audit deliveries with a plain random sample.

    What exactly are you measuring?

    For a filtered dataset, two numbers matter and they are easy to confuse:

    • Precision: of the items delivered, what share actually meet the spec? This is what the buyer of a dataset cares about.
    • Recall: of the items that meet the spec among all candidates, what share did the filter keep? This decides cost and speed, because low recall throws away good data you paid to discover.

    Precision is measured by labeling a sample of accepted items. Recall needs labels on rejected items too, which is why calibration samples look different from delivery audits.

    Write the label definition first

    An audit is only as good as its definition. "Natural conversation" is ambiguous; "two or more people talking unscripted for most of the clip, not a single narrator, not mostly music" is labelable. Write the definition down, include borderline examples with the intended answer, and give labelers a third option — unsure — rather than forcing a guess. Track unsure items separately; if they exceed a few percent, the definition needs work, not the filter.

    If two labelers are available, have them overlap on part of the sample. Disagreement between people is the ceiling on how accurate any filter can be measured to be.

    The confidence interval: use Wilson

    If you audit n items and k pass, the point estimate is p = k / n. The interval most people learn — p ± 1.96 × √(p(1−p)/n) — breaks down exactly where data quality numbers live: near 100%, and with small samples. Audit 50 items, find 50 correct, and it claims an interval of [1.00, 1.00], which is absurd.

    The Wilson score interval fixes this:

    z      = 1.96                       # 95% confidence
    center = (p + z²/(2n)) / (1 + z²/n)
    half   = z · √( p(1−p)/n + z²/(4n²) ) / (1 + z²/n)
    interval = [center − half, center + half]

    In Python:

    import math
    
    def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
        if n == 0:
            return (0.0, 1.0)
        p = k / n
        d = 1 + z * z / n
        c = (p + z * z / (2 * n)) / d
        h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
        return (max(0.0, c - h), min(1.0, c + h))
    
    print(wilson(45, 50))   # (0.786..., 0.956...)

    How big should the sample be?

    Same observed precision of 90%, different sample sizes:

    Audited Correct Point estimate 95% Wilson interval
    50 45 0.900 0.786 – 0.957
    100 90 0.900 0.826 – 0.945
    200 180 0.900 0.851 – 0.934
    500 450 0.900 0.871 – 0.923

    And at the top end: 50 out of 50 correct gives an interval of 0.929 – 1.000. A perfect small audit still only tells you precision is probably above 93%.

    Two practical consequences:

    • Error bars shrink with the square root of sample size. Going from 50 to 200 labels roughly halves the width; going to 800 halves it again.
    • The sample size you need doesn't depend on dataset size. 200 random items measure a 10,000-item batch and a 1,000,000-item batch about equally well. That is what makes auditing large datasets affordable.

    A common rule for per-batch audits is max(50, 1% of the batch), with a larger one-off sample (around 200) when a new spec is first calibrated.

    Calibration samples vs delivery audits

    They answer different questions, so they are sampled differently:

    Calibration (before running at volume) Delivery audit (every batch)
    Goal Tune thresholds, find which filter layer makes which mistakes Estimate precision of what was delivered
    Sample Stratified: accepted, borderline rejected, clearly rejected Simple random sample of delivered items
    Typical size ~200 max(50, 1%)
    Output Precision and recall per layer, error examples for the definition Precision with a 95% interval, pass/fail against target

    A stratified calibration sample oversamples borderline cases on purpose, so its raw pass rate is not an estimate of delivered precision. Report precision from the accepted stratum, and use the rejected strata to estimate how much good data the filters discard.

    Turning it into acceptance rules

    Decide before you see the numbers:

    1. Set a precision target, for example 0.90, as part of the spec.
    2. Accept a batch if the lower bound of the 95% interval is at or above an agreed floor — for example the target minus a tolerance, since even a filter that is exactly at target will produce a lower bound below it on a modest sample.
    3. If it fails: pause the job, inspect the failed audit items, fix the definition or thresholds, and either top up the batch with replacement items or adjust billing by the measured precision. Agree in advance which.
    4. Watch for drift. Track the interval batch by batch. Precision usually decays slowly as discovery reaches less typical sources; a falling trend is a signal before a batch actually fails.

    Common pitfalls

    • Hand-picked "samples". If the vendor or the pipeline chooses which items are audited, the number is meaningless. Sample randomly, with a recorded seed.
    • Labeling with the model's output visible. Showing labelers the classifier's score biases them toward agreeing. Hide it.
    • Auditing on metadata only. If the spec is about audio (speakers, music), labelers must listen. See what makes audio ASR- and TTS-ready.
    • Reusing calibration labels as audit labels. They were used to tune the filters, so they overstate precision.
    • Reporting one number for a mixed dataset. If a delivery mixes languages or categories, audit and report per segment that matters to you.

    How Corpora uses this

    Corpora Curation calibrates every job with human labels before it runs at volume, and every delivery batch carries a random audit sample and a 95% Wilson interval in its batch report, so you can check precision against your target yourself. The same method works for any vendor's data — see buying video training data: a checklist — and for your own pipelines. Talk to sales if you want a calibrated sample for your spec.

    FAQ

    How many samples do I need to measure dataset accuracy?

    About 200 randomly sampled items give a 95% interval roughly ±4 percentage points around a 90% precision; 50 give a rough check (about ±8 points). The size of the dataset itself barely matters.

    Why not use the normal-approximation interval?

    It is wrong near 0% and 100% and with small samples: a perfect audit produces a zero-width interval. The Wilson interval stays meaningful in those cases.

    Should I accept a batch on the point estimate?

    No. Use the lower bound of the interval against an agreed floor, so a lucky small sample can't hide a weak batch.

    What is the difference between precision and recall for a filtered dataset?

    Precision is the share of delivered items that meet the spec. Recall is the share of qualifying candidates the filter kept. Buyers mostly care about precision; recall drives cost.