Tag: dataset documentation

  • Dataset Documentation: Datasheets, Cards and Provenance

    Most training sets ship with a README written the week before release. A year later nobody can say which license each sample had, when it was fetched or whether its source has since opted out. This guide is a working approach to dataset documentation for ML training data. It covers the established frameworks (Datasheets, Data Cards, Hugging Face dataset cards, Croissant), the per-sample provenance fields that make them possible to fill in, the manifests that tie documentation to actual bytes, and how all of this feeds the EU AI Act's training-content summary.

    TL;DR

    • Documentation has two layers. A human-readable card (Datasheet, Data Card or dataset card) answers why, how and with what caveats. Machine-readable records (per-sample provenance, manifests, Croissant) back up every claim the card makes.
    • Record provenance at collection time. Source URL, retrieval time, license as observed, content hash and opt-out status can't be reconstructed reliably afterwards.
    • Don't trust upstream license labels. The Data Provenance Initiative audit found license omission rates above 70% and error rates above 50% on widely used dataset hosting sites.
    • Pin documentation to bytes with a sha256 manifest per release, so "this card describes that data" can be checked.
    • The EU AI Act training-content summary for general-purpose AI models must follow the Commission's mandatory template (published 24 July 2025). It's much easier to write from good per-sample records.

    The frameworks, and what each is for

    Datasheets for Datasets

    Datasheets for Datasets (Gebru et al., first posted 2018, published in CACM in December 2021) borrows the idea of an electronics datasheet: every dataset should come with a document covering its motivation, composition, collection process, preprocessing/cleaning/labeling, uses, distribution and maintenance. The paper's value is its question list. It forces creators to write down things that are obvious to them and invisible to everyone else, such as who funded the collection, what was filtered out, and whether the data relates to people.

    A datasheet is prose: the right place for judgments and caveats, the wrong place for facts that should be computed (counts, license distributions, collection windows). Generate those from your records.

    Data Cards

    Data Cards (Pushkarna, Zaldivar and Kjartansson, Google Research, FAccT 2022) are structured summaries aimed at different stakeholders across a dataset's lifecycle: upstream sources, collection and annotation methods, intended uses, and decisions that affect model performance. The paper reports lessons from deploying over 20 Data Cards, and Google publishes a Data Cards Playbook with templates and activities. The lesson: documentation is a product with distinct readers (legal, researchers, downstream teams), so write for each of them.

    Hugging Face dataset cards

    On the Hugging Face Hub, a dataset card is the repository's README.md: YAML front matter for machine-readable metadata, followed by a Markdown body. The metadata reference lists fields including license (plus license_name and license_link when the license is other), language, pretty_name, tags, annotations_creators, size_categories, source_datasets, task_categories, configs and dataset_info. There are also gating fields (extra_gated_fields, extra_gated_prompt). The body follows the dataset card template.

    ---
    pretty_name: Example Lecture Video Clips
    license: cc-by-4.0
    language:
    - en
    - de
    tags:
    - video
    - audio
    annotations_creators:
    - machine-generated
    size_categories:
    - 100K<n<1M
    source_datasets:
    - original
    task_categories:
    - automatic-speech-recognition
    configs:
    - config_name: default
      data_files:
      - split: train
        path: data/train-*.parquet
    ---

    The YAML license is one value for the whole repository. If your samples carry different licenses, say so in the card and put the per-sample license in a column. A single dataset-level tag that hides a mixed collection is exactly the kind of mislabeling discussed below.

    Croissant and the RAI extension

    Croissant is an MLCommons metadata format built on schema.org's Dataset vocabulary and encoded as JSON-LD. It describes four layers: dataset metadata, resources (FileObject and FileSet, each file with a sha256), structure (RecordSet and Field), and ML semantics. The Croissant 1.1 specification is dated 2026-01-29. Its introduction says it adds machine-readable representations of data provenance and usage conditions, built on the W3C PROV-O and ODRL standards. Hugging Face, Kaggle and OpenML expose Croissant files for hosted datasets; the Hub serves them at https://huggingface.co/api/datasets/<owner>/<name>/croissant.

    The Croissant RAI specification (version 1.0, published 2024-03-06) adds responsible-AI properties under the rai: prefix. They include rai:dataCollection, rai:dataCollectionType (recommended values include "Web Scraping" and "Web API"), rai:dataCollectionTimeframe, rai:dataPreprocessingProtocol, rai:dataLimitations, rai:dataBiases, rai:personalSensitiveInformation, rai:dataUseCases and rai:dataReleaseMaintenancePlan. A trimmed example:

    {
      "@context": {
        "@language": "en",
        "@vocab": "http://schema.org/",
        "sc": "http://schema.org/",
        "cr": "http://mlcommons.org/croissant/",
        "rai": "http://mlcommons.org/croissant/RAI/",
        "dct": "http://purl.org/dc/terms/"
      },
      "@type": "sc:Dataset",
      "name": "example-lecture-clips",
      "dct:conformsTo": ["http://mlcommons.org/croissant/1.1",
                         "http://mlcommons.org/croissant/RAI/1.0"],
      "description": "Speech-heavy lecture clips with captions.",
      "license": "https://creativecommons.org/licenses/by/4.0/",
      "url": "https://example.com/datasets/lecture-clips",
      "version": "1.2.0",
      "rai:dataCollectionType": ["Web Scraping"],
      "rai:dataCollection": "Candidate IDs from channel lists; license checked at discovery and again at download.",
      "rai:dataPreprocessingProtocol": ["sha256 exact dedup", "near-duplicate removal", "scene splitting"],
      "rai:personalSensitiveInformation": ["Faces and voices of speakers"],
      "rai:dataLimitations": ["Mostly English; captions partly machine-generated."],
      "distribution": [{
        "@type": "cr:FileObject",
        "@id": "train-00000.parquet",
        "contentUrl": "data/train-00000.parquet",
        "encodingFormat": "application/x-parquet",
        "sha256": "<sha256 of the file>"
      }]
    }

    This example is illustrative and incomplete: a valid file also needs a recordSet, plus creator and datePublished (required in 1.1). Start from the repository's examples and validate with mlcroissant rather than hand-writing it.

    Which one should you use?

    Format Audience Machine-readable Use it for
    Datasheet People evaluating or auditing the data No Motivation, process, caveats, maintenance
    Data Card Multiple stakeholder groups Mostly no Structured summaries per audience
    HF dataset card Hub users, loaders YAML header Publishing on the Hub; config and split definitions
    Croissant + RAI Tools, search engines, loaders Yes (JSON-LD) File checksums, record structure, RAI fields

    They aren't competitors. A practical stack: datasheet answers in the card body, HF YAML if you publish on the Hub, Croissant generated from your manifest, and per-sample records underneath.

    Why upstream labels can't be trusted

    The Data Provenance Initiative (Longpre et al., 2023) audited more than 1,800 text fine-tuning datasets and traced each one's sources, creators and license conditions. It reports frequent license miscategorization on widely used dataset hosting sites, with license omission of 70%+ and error rates of 50%+. Those figures describe text datasets audited in 2023, not video or speech data today. The lesson still carries over: a license tag on an aggregated dataset tells you what the uploader chose, not what the underlying sources permit.

    So record the license from the original source, per sample, with evidence, and trace upstream sources of third-party data yourself. See also our checklist for buying video training data.

    Per-sample provenance: the fields to record

    Everything in a card should be derivable from per-sample records. These are the fields worth capturing at collection time:

    Field Why
    sample_id Stable key across releases; derive it from the source ID, not the row order
    source_url, source_platform, source_id Where the bytes came from; needed for attribution and takedowns
    retrieved_at (UTC, ISO 8601) Licenses, availability and opt-outs change; time scopes every other field
    license + evidence The value as observed, where it was read, and when
    sha256, bytes Ties the record to exact content; enables dedup and integrity checks
    opt_out_status + checked_at Result of whatever opt-out signals you honor (robots.txt, machine-readable reservations, takedown list)
    collector Crawler name/version and user agent; the EU template asks about crawlers
    parent_id, transforms Lineage for clips, transcodes and captions derived from a source sample
    pipeline_version Which filters and thresholds produced this release

    One JSONL line per sample:

    {"sample_id": "yt:abc123XYZ_-:clip0007",
     "source_platform": "youtube", "source_id": "abc123XYZ_-",
     "source_url": "https://www.youtube.com/watch?v=abc123XYZ_-",
     "retrieved_at": "2026-11-02T08:14:55Z",
     "license": {"value": "creativeCommon", "source": "videos.list status.license",
                 "checked_at": "2026-11-02T08:14:51Z"},
     "opt_out_status": {"value": "none_found", "checked_at": "2026-11-02T08:14:51Z"},
     "collector": "collector/3.4.1",
     "parent_id": "yt:abc123XYZ_-", "transforms": ["clip 83.4-91.8s", "transcode av1"],
     "pipeline_version": "2026.11-r2",
     "bytes": 2841122,
     "sha256": "9f2c...e1"}

    For YouTube specifically, how to read and double-check the license field is covered in finding and filtering Creative Commons YouTube videos. Append new checks instead of overwriting old ones, so you can answer "what did you know, and when?"

    Manifests: pinning documentation to bytes

    A card says "release 1.2.0 contains 412 shards". A manifest proves it. Write one per release, listing every file with its size and sha256, and store it with the card. Consumers verify on download, and you can show later that the documented release is the one you trained on. The script below writes and verifies a JSONL manifest using only the standard library (Python 3.8+):

    #!/usr/bin/env python3
    """Write or verify a sha256 manifest for a directory of dataset files."""
    import argparse
    import hashlib
    import json
    import sys
    from pathlib import Path
    
    
    def sha256_file(path, chunk=8 * 1024 * 1024):
        h = hashlib.sha256()
        with open(path, "rb") as f:
            while block := f.read(chunk):
                h.update(block)
        return h.hexdigest()
    
    
    def write_manifest(root, out):
        root, out = Path(root), Path(out).resolve()
        with open(out, "w", encoding="utf-8") as f:
            for p in sorted(root.rglob("*")):
                if p.is_file() and p.resolve() != out:
                    rec = {"path": p.relative_to(root).as_posix(),
                           "bytes": p.stat().st_size,
                           "sha256": sha256_file(p)}
                    f.write(json.dumps(rec) + "\n")
    
    
    def verify_manifest(root, manifest):
        root, bad = Path(root), 0
        with open(manifest, encoding="utf-8") as f:
            for line in f:
                rec = json.loads(line)
                p = root / rec["path"]
                if not p.is_file():
                    print("MISSING ", rec["path"])
                    bad += 1
                elif p.stat().st_size != rec["bytes"] or sha256_file(p) != rec["sha256"]:
                    print("MISMATCH", rec["path"])
                    bad += 1
        return bad
    
    
    if __name__ == "__main__":
        ap = argparse.ArgumentParser()
        ap.add_argument("mode", choices=["write", "verify"])
        ap.add_argument("root")
        ap.add_argument("manifest")
        a = ap.parse_args()
        if a.mode == "write":
            write_manifest(a.root, a.manifest)
        else:
            sys.exit(1 if verify_manifest(a.root, a.manifest) else 0)

    Usage: python manifest.py write release-1.2.0/ manifest-1.2.0.jsonl, then python manifest.py verify release-1.2.0/ manifest-1.2.0.jsonl. Hash the manifest itself and put that one hash in the card and the Croissant file. The shard-level layout this assumes is covered in storing video datasets as WebDataset and Parquet shards. If you exclude benchmark data, record the exclusion list's hash too (see benchmark decontamination).

    The EU AI Act training-content summary

    Article 53(1)(d) of the AI Act requires providers of general-purpose AI models to publish a sufficiently detailed summary of the content used for training, following a template from the AI Office. The Commission published the Explanatory Notice and Template on 24 July 2025. According to the Commission's Q&A (last updated 26 March 2026):

    • Using the template is mandatory. The obligation applies from 2 August 2025, including to models released under free and open-source licenses.
    • Models placed on the market before that date have until 2 August 2027. Information that is unavailable, or disproportionately burdensome to retrieve, must be stated and justified as a gap.
    • The template has three sections: general information (including modalities and size per modality within broad ranges), a list of data sources (public datasets, private datasets, scraped data, user data, synthetic data), and relevant data processing aspects (such as copyright-related measures and removal of illegal content).
    • For scraped data, it asks for crawler names, the collection period, a description of the content, and the top 10% of scraped domains (for SMEs, the top 5% or 1,000 domains, whichever is lower).
    • Summaries are updated every six months, or sooner after a materially significant change in training data. The AI Office can enforce from 2 August 2026.

    Whether you count as a provider, how fine-tuning changes that, and how your model relates to these dates are legal questions. The Commission has separate guidelines on GPAI providers on that. Have counsel read the template itself; this list summarizes the Q&A, not the full Explanatory Notice.

    From an engineering view, with the per-sample records above the summary becomes a query rather than an archaeology project:

    import json
    import math
    import sys
    from collections import Counter
    from urllib.parse import urlsplit
    
    REQUIRED = ["source_url", "retrieved_at", "license", "sha256", "opt_out_status"]
    licenses, bytes_by_domain, missing = Counter(), Counter(), Counter()
    first = last = None
    
    with open(sys.argv[1], encoding="utf-8") as f:
        for line in f:
            r = json.loads(line)
            missing.update(k for k in REQUIRED if not r.get(k))
            lic = r.get("license") or {}
            licenses[lic.get("value", "UNKNOWN") if isinstance(lic, dict) else lic] += 1
            host = urlsplit(r.get("source_url", "")).hostname or "UNKNOWN"
            bytes_by_domain[host] += r.get("bytes", 0)
            t = r.get("retrieved_at")
            if t:  # ISO 8601 UTC strings sort chronologically
                first = t if first is None or t < first else first
                last = t if last is None or t > last else last
    
    k = max(1, math.ceil(len(bytes_by_domain) * 0.10))
    print("collection window:", first, "to", last)
    print("licenses:", dict(licenses))
    print("records missing fields:", dict(missing))
    print(f"top {k} of {len(bytes_by_domain)} domains by bytes:")
    for host, b in bytes_by_domain.most_common(k):
        print(f"  {host}\t{b}")

    Ranking domains by bytes is an assumption here. Check the Explanatory Notice for the measure it actually specifies. For a single-platform video set the domain list is trivially short. The parts that take work are the collection period, the crawler description and the processing measures, and those come straight from retrieved_at, collector and pipeline_version.

    A release checklist

    1. Freeze shards plus the per-sample table.
    2. Fail the release if any record lacks a required provenance field.
    3. Apply the current takedown/opt-out list; record its hash.
    4. Write and hash the manifest.
    5. Generate the card's numbers from the table; update the prose by hand.
    6. Emit and validate HF YAML and/or Croissant; add a changelog entry.

    Compliance note

    Documentation records what you did; it doesn't make it permissible. If your data comes from YouTube, its Terms of Service restrict automated access and downloading. Copyright and licensing apply, and filtering for Creative Commons narrows the problem rather than settling it. Faces and voices can be personal data under GDPR. Honoring takedown and opt-out requests requires the per-sample lineage described above. None of this is legal advice: get legal review for your jurisdiction and use case, including whether and how the AI Act applies to you.

    If you'd rather not build the collection side

    Corpora's managed service collects YouTube video, audio, subtitles and metadata and delivers them as JSONL or Parquet with sha256 checksums and manifests to your S3, GCS or Azure bucket. You still write the card, but the provenance and integrity inputs arrive with the data. A free sample batch is available through our contact page.

    FAQ

    What is dataset documentation in machine learning?

    A human-readable description of why and how a dataset was built, what it contains and what it shouldn't be used for, backed by machine-readable metadata: per-sample provenance, file manifests and formats such as Croissant or Hugging Face card YAML.

    Does Croissant replace a dataset card?

    No. Croissant makes metadata, file checksums and record structure machine-readable, and its RAI extension adds fields for collection, limitations and sensitive information. The narrative still belongs in a card. Hugging Face exposes both for Hub datasets.

    What provenance should I store for each training sample?

    At minimum: source URL and ID, retrieval time in UTC, license as observed with where and when it was read, sha256 and size, opt-out check result with timestamp, the collector, and lineage for derived clips.

    When does the EU AI Act training data summary apply?

    Per the Commission's Q&A, from 2 August 2025 for general-purpose AI models placed on the EU market, with a deadline of 2 August 2027 for models already on the market before then, and AI Office enforcement from 2 August 2026. Confirm how it applies to you with counsel.