Tag: AI training data

  • Buying Video Training Data: A Checklist for AI Teams

    Buying video or audio training data looks simple on a price sheet — dollars per hour, per terabyte or per clip — and gets complicated the moment the first delivery lands in your bucket. This checklist is what an AI team should agree with a data vendor before paying for volume: what to specify, what to ask for in writing, how to test a sample, and the red flags that predict a painful project.

    TL;DR

    • Write the spec as acceptance tests, not adjectives. "High quality" is not a spec; "≥1080p, single shot, ≤5% OCR-heavy clips" is.
    • Get provenance per sample, not per delivery: source ID, collection time, license status at collection.
    • Insist on a sample batch run through your own pipeline before committing volume.
    • Compare price per usable hour, after your own filtering and deduplication, not price per delivered terabyte.
    • Agree on takedown handling before you need it.

    1. Provenance and rights

    Ask first, because everything else is wasted if the answers are unacceptable to your legal team.

    • Where does the content come from, and how was it collected?
    • Is there a per-sample record of source (platform and ID or URL), collection timestamp, and license status at collection time?
    • Can you filter by license (for example, Creative Commons only)? See finding and filtering Creative Commons YouTube videos for what that does and doesn't settle.
    • How are takedown and opt-out requests handled after delivery? Will the vendor notify you so you can remove affected samples and derived clips?
    • What does the contract say about your permitted uses, and who carries which risks?

    Platform terms, copyright and privacy law (faces and voices can be personal data) all apply to video. Your legal team should review the answers for your jurisdiction and use case; a vendor's assurances don't replace that.

    2. Content specification

    Turn what you want into numbers the vendor can measure and you can verify:

    Dimension Example spec
    Volume 50,000 hours, delivered in monthly tranches of ~8,000 hours
    Resolution / frame rate ≥1080p source, ≥24 fps, native (not upscaled)
    Duration Source videos 1–30 min; or clips 4–20 s after splitting
    Language (audio) ≥70% English, ≥10% each of Spanish, German, Japanese
    Content mix Category distribution, with caps per channel to avoid a few uploaders dominating
    Exclusions No livestream VODs, no Shorts, no screen recordings, no compilations
    Recency Uploaded after a given date, if freshness matters
    Modalities Video + original audio + manual subtitles where available + metadata JSON

    Per-uploader caps are easy to forget and matter a lot: without them, a handful of prolific channels can make up a large share of a dataset and skew style, voice and content.

    3. Quality and filtering

    • What filtering has the vendor already applied (shot detection, motion, OCR/text, blur, NSFW, faces)? Ask for the filter list and per-filter keep-rates.
    • Will they deliver the raw scores per sample, so you can apply your own thresholds? This is worth more than their thresholds. See video data curation for generative models.
    • Are captions or transcripts included? Which model produced them, and is that recorded per sample? See recaptioning video datasets.

    4. Deduplication and contamination

    • Is the delivery deduplicated within itself — exact (hash) and near-duplicate (perceptual or embedding)?
    • Can it be deduplicated against data you already own, and across tranches, so you never pay twice for the same video?
    • Can the vendor exclude a list of IDs you provide, such as your evaluation sets or previous purchases?

    Duplicates are the most common hidden cost: they inflate volume metrics and quietly overweight some content in training.

    5. Metadata schema

    Agree on the schema before the first delivery, with types and units. A minimal per-sample record:

    {
      "sample_id": "c7f1e2...",
      "source": "youtube",
      "source_id": "dQw4w9WgXcQ",
      "collected_at": "2026-11-02T08:14:55Z",
      "license_at_collection": "creativeCommon",
      "duration_s": 212.4,
      "width": 1920,
      "height": 1080,
      "fps": 25.0,
      "video_codec": "vp9",
      "audio_codec": "opus",
      "language": "en",
      "title": "...",
      "channel_id": "UC...",
      "upload_date": "2025-06-18",
      "subtitles": ["en"],
      "file_path": "s3://bucket/tranche-03/shard-000412.tar",
      "sha256": "9f2c..."
    }

    Ask for JSONL or Parquet, one schema version per delivery, and a changelog when fields change.

    6. Delivery

    • Where: directly into your S3, GCS or Azure bucket. Give the vendor write-only credentials scoped to one prefix.
    • Layout: sharded files (for example WebDataset-style tar shards) rather than millions of small objects, plus a metadata table.
    • Integrity: a manifest per tranche listing every file with size and sha256, so you can verify completeness automatically.
    • Cadence: tranche size and schedule, and what happens if a tranche fails acceptance.
    • Incremental updates: if you need ongoing collection, how new data is identified and delivered without re-sending old samples.

    Ingress into major cloud object stores is generally free, but reading data back out is not. Plan where you will process it. The capacity arithmetic is in bandwidth for large-scale video downloads.

    7. The sample batch and acceptance tests

    Never commit volume without running a sample through your own pipeline. A useful acceptance test, written down and agreed with the vendor:

    Test Pass criterion (example)
    Integrity 100% of files match manifest size and sha256
    Schema 100% of records validate against the agreed schema
    Spec compliance ≥98% of samples meet resolution, duration and language specs
    Duplicates ≤1% near-duplicates within tranche; 0 overlap with your exclusion list
    Usable yield After your own curation filters, ≥X% of hours remain
    Provenance 100% of samples have source ID, collection time and license field

    The "usable yield" line is the one that decides cost. Run your real curation pipeline on the sample and compute it; that number turns a price per delivered hour into a price per hour you will actually train on.

    8. Comparing prices

    Vendors quote in different units: per hour, per clip, per terabyte, per petabyte, or as bandwidth if you collect yourself. Normalize to one number:

    effective cost per usable hour = total price ÷ (delivered hours × usable yield)

    A cheaper per-terabyte price with 40% usable yield costs more than a higher price with 80%. If you collect yourself instead, add engineering time, proxy or bandwidth costs, storage, and the cost of retries and failures.

    Red flags

    • No per-sample provenance, or "we can't tell you where it came from".
    • No sample batch, or a sample that is clearly hand-picked rather than random.
    • Volume quoted only in terabytes with no hours, resolution or duplicate rate.
    • No answer on takedown handling.
    • No checksums or manifests — you will be debugging missing files for months.
    • Refusal to dedupe against your existing data or evaluation sets.

    How Corpora works

    Corpora collects YouTube video, audio, subtitles and structured metadata and delivers it to your S3, GCS or Azure bucket as deduplicated data with sha256 checksums and manifests, in your preferred layout, priced per delivered petabyte. Every project starts with a free sample batch so you can run your own acceptance tests. Teams that prefer to collect themselves can use dedicated 1–40 Gbps unmetered proxy capacity instead. See pricing or talk to sales.

    FAQ

    What should a video training data contract specify?

    Content spec as measurable criteria, per-sample provenance, metadata schema, delivery layout and integrity checks, acceptance tests on a sample, deduplication commitments and takedown handling.

    How do I compare vendors who price differently?

    Run each vendor's sample through your own curation pipeline, measure usable yield, and compare effective cost per usable hour.

    Should I buy curated clips or raw videos?

    Raw videos with per-sample scores give you the most flexibility, because your thresholds will change. Curated clips save processing time if the vendor's filtering matches your model's needs.

    Is buying data cheaper than collecting it ourselves?

    It depends on volume and your team's time. Compare the vendor's effective cost per usable hour with your own bandwidth, proxy, storage, engineering and failure costs at the same volume.

  • Creative Commons YouTube Videos for AI Training: Find & Filter

    Filtering for Creative Commons is the most common way data teams try to narrow a YouTube collection to content they're allowed to reuse. The filter is easy to apply but often misunderstood. This guide covers what YouTube's CC option grants, how to find and verify CC videos with the search filter, the YouTube Data API and yt-dlp, and how to record the license for each sample so you can show your work later. It also covers what a CC BY label doesn't settle.

    TL;DR

    • YouTube offers one CC option: CC BY (Attribution). Its Help page currently links CC BY 4.0. Archived 2023–2024 copies linked the CC BY 3.0 legal code, and neither the API nor yt-dlp reports a version for an individual video.
    • The label is the uploader's claim. YouTube says CC BY can only be applied to content the uploader can license, but it can't check every clip, song or person in the video.
    • Discovery: use the "Creative Commons" search filter for manual exploration, or search.list with type=video&videoLicense=creativeCommon (default limit: 100 calls per day). Run enrichment through videos.list (status.license, 1 unit per call).
    • yt-dlp reports CC videos as license: "Creative Commons Attribution license (reuse allowed)". That value is scraped page text, so treat a missing value as unknown, not as "not CC".
    • Check the license twice (at discovery and at download), store both results with timestamps, re-check periodically, and make takedowns a routine operation.

    What YouTube's Creative Commons option is

    According to YouTube Help's License types on YouTube page, uploaders can choose between two licenses: the Standard YouTube license (the default, governed by the Terms of Service) and the Creative Commons Attribution license. YouTube offers no NC, SA or ND variants. If a video shows CC on YouTube, it's CC BY.

    Which version?

    The current Help page links "CC BY license" to CC BY 4.0. That wasn't always the case. Archived copies from 2023 and mid-2024 link to the CC BY 3.0 legal code, while a September 2025 copy already links 4.0. The API's status.license is just creativeCommon or youtube, and yt-dlp's string carries no version either. In practice:

    • You can't read the license version from a video's metadata. Record the date you checked, and note which version YouTube's Help page referenced at that time.
    • Expect older uploads to have been licensed when YouTube referenced 3.0. The two versions differ in details such as attribution and change-marking (see below), so ask counsel which terms to meet. Many teams just satisfy both.

    What attribution requires

    The CC BY 4.0 deed requires you to give appropriate credit, link to the license, and indicate if changes were made, "in any reasonable manner, but not in any way that suggests the licensor endorses you or your use." Appropriate credit means, if supplied: the creator's name, a copyright notice, a license notice, a disclaimer notice and a link to the material. Licenses before 4.0 also require the title, and 3.0 only requires marking changes when you create a derivative. YouTube's own guidance for crediting a CC creator lists title, author, source URL and license info.

    For a dataset, this translates into fields you must capture at collection time, because they may be gone later: video title, channel name, channel URL, video URL, license, and a description of what you changed (transcoded, clipped, captioned).

    What the label doesn't guarantee

    YouTube Help says you can add CC BY "if the content is licenseable under the CC BY license": your own original content, content already under CC BY, or public domain material. It also says CC BY can't be added while the video has a Content ID claim. That rule depends on the uploader telling the truth and on Content ID catching what it can. A CC-labeled video can still contain:

    • third-party clips, TV footage or re-uploads of someone else's work;
    • commercial music that isn't in Content ID, or wasn't detected;
    • people who never agreed to appear (a separate privacy issue, covered below).

    The same Help page states that YouTube "can't grant you rights to use someone else's content." The CC deed also says the license "may not give you all of the permissions necessary for your intended use" and names publicity, privacy and moral rights as possible limits. Treat the CC flag as a strong filter, not as clearance.

    Finding Creative Commons videos

    1. YouTube search filter (manual)

    YouTube Help's FAQ still points to Creative Commons in the list of advanced search filters. Use it to scope a domain by hand, find channels that license consistently under CC BY, and seed channel lists. It's not a collection mechanism.

    2. YouTube Data API v3

    search.list takes a videoLicense parameter with the values any, creativeCommon or youtube. When you set it, you must also set type=video:

    curl -s "https://www.googleapis.com/youtube/v3/search?part=snippet&type=video&videoLicense=creativeCommon&q=woodworking&maxResults=50&key=API_KEY"

    Quota is the constraint. Google's quota cost page currently says projects get a default of 100 search.list calls, 100 videos.insert calls, and 10,000 units per day for all other endpoints combined, resetting at midnight Pacific Time. Each search.list call (including each extra page) costs 1 from the search bucket, and maxResults tops out at 50. That gives a ceiling of 100 × 50 = 5,000 search results per project per day, and the docs warn that pages can come back with fewer items. Searches restricted to one channelId are capped at 500 videos. Older posts describe a "10,000 units, search costs 100" model, so check the current page before you plan around either one.

    The authoritative per-video check is videos.list with part=status,snippet. It costs 1 unit per call and takes comma-separated IDs. The video resource exposes status.license with valid values creativeCommon and youtube:

    curl -s "https://www.googleapis.com/youtube/v3/videos?part=status,snippet&id=VIDEO_ID_1,VIDEO_ID_2&key=API_KEY"

    The same resource includes contentDetails.licensedContent, which says whether the video was uploaded to a channel linked to a content partner and then claimed by that partner. It's worth storing as a secondary signal.

    3. yt-dlp metadata

    yt-dlp documents license as an output-template field ("License name the video is licensed under"). In the current YouTube extractor (yt_dlp/extractor/youtube/_video.py, release 2026.10.26), the value is read from the watch page's metadata rows: when a row's title is License, its text becomes info['license']. The extractor's own test cases expect:

    • "Creative Commons Attribution license (reuse allowed)" for a CC video;
    • "Standard YouTube License" for a standard-licensed one.

    Because this is display text and not a structured field, it can be missing when YouTube doesn't render that row. A missing value means "unknown". Send those IDs to videos.list rather than counting them as standard-licensed. (We couldn't run a live check for this article: our test request hit YouTube's bot check. Validate on a small sample of known-CC IDs before a large run.)

    For a metadata-only CLI pass, -j prints JSON without downloading, and --match-filters drops non-matching videos before anything is printed. The README lists *= (contains) and ~= (regex) among the string operators:

    yt-dlp -a candidates.txt -j \
      --match-filters "license*='Creative Commons' & !is_live" \
      > cc_metadata.jsonl

    Without a ? on the comparison (the README's like_count>?100 example passes videos whose field is missing), a missing license fails the filter. That's the conservative default you want. Also note that passing --match-filters more than once ORs the conditions together. Put all conditions in one expression joined with &. Current yt-dlp supports Python 3.11+ and needs yt-dlp-ejs plus a JavaScript runtime for full YouTube support, per the README. If metadata requests start failing, see fixing "Sign in to confirm you're not a bot".

    Python: keep only CC videos from a list of IDs (no downloads)

    This script fetches metadata for each ID and writes CC videos to one JSONL file. Everything else (not CC, unknown, error) goes to a second file with the reason, so unknowns can be re-checked through the API.

    """cc_filter.py: keep Creative Commons YouTube videos from an ID list (metadata only)."""
    import json
    import sys
    from datetime import datetime, timezone
    
    import yt_dlp
    from yt_dlp.utils import DownloadError
    from yt_dlp.version import __version__ as YTDLP_VERSION
    
    FIELDS = ["id", "title", "channel", "channel_id", "channel_url",
              "upload_date", "license", "availability"]
    
    
    def now():
        return datetime.now(timezone.utc).isoformat(timespec="seconds")
    
    
    def classify(license_value):
        if not license_value:
            return "unknown"  # field missing: verify with videos.list, don't assume
        return "cc" if "Creative Commons" in license_value else "not_cc"
    
    
    def main(ids_path, cc_path, rest_path):
        with open(ids_path, encoding="utf-8") as f:
            ids = list(dict.fromkeys(line.strip() for line in f if line.strip()))
    
        opts = {"quiet": True, "no_warnings": True, "skip_download": True,
                "ignore_no_formats_error": True}
        # opts["proxy"] = "http://USER:[email protected]:8000"
    
        with yt_dlp.YoutubeDL(opts) as ydl, \
                open(cc_path, "a", encoding="utf-8") as cc_out, \
                open(rest_path, "a", encoding="utf-8") as rest_out:
            for vid in ids:
                url = f"https://www.youtube.com/watch?v={vid}"
                try:
                    info = ydl.extract_info(url, download=False)
                    rec = {k: info.get(k) for k in FIELDS}
                    rec["status"] = classify(rec["license"])
                except DownloadError as e:
                    rec = {"id": vid, "status": "error", "error": str(e)}
                rec["source_url"] = url
                rec["checked_at"] = now()
                rec["checker"] = f"yt-dlp {YTDLP_VERSION}"
                out = cc_out if rec["status"] == "cc" else rest_out
                out.write(json.dumps(rec, ensure_ascii=False) + "\n")
    
    
    if __name__ == "__main__":
        if len(sys.argv) != 4:
            sys.exit("usage: python cc_filter.py ids.txt cc.jsonl rest.jsonl")
        main(*sys.argv[1:])

    A resumable, proxy-aware version of the metadata loop, along with the other filter fields (duration, language, resolution), is in our guide to building a YouTube video dataset for multimodal training. To route these requests through your own proxy pool, see the yt-dlp proxy guide.

    Re-check the license at download time and store it per sample

    Uploaders can change a video's license in YouTube Studio at any time, and videos get deleted or made private. The CC deed says the licensor "cannot revoke these freedoms as long as you follow the license terms." Whether that protects material you collected before a switch is a legal question for counsel. The engineering answer is to keep evidence of what the license was, and when, for every sample:

    • Check at discovery, then again immediately before or after download. If the two disagree, drop the video or quarantine it.
    • Store each check as an entry with its source (API or yt-dlp version), raw value and UTC timestamp. Keep the raw API response or info.json and its sha256 as evidence.
    • Record the attribution string you'll publish, plus the list of modifications.

    A per-sample record (illustrative values):

    {
      "sample_id": "abc123XYZ_-_000012",
      "video_id": "abc123XYZ_-",
      "source_url": "https://www.youtube.com/watch?v=abc123XYZ_-",
      "title": "Example video title",
      "channel": "Example Channel",
      "channel_id": "UCxxxxxxxxxxxxxxxxxxxxxx",
      "channel_url": "https://www.youtube.com/channel/UCxxxxxxxxxxxxxxxxxxxxxx",
      "upload_date": "20240611",
      "license": {
        "family": "CC BY",
        "version_note": "YouTube Help linked CC BY 4.0 at check time; video version not exposed",
        "checks": [
          {"stage": "discovery", "source": "data_api videos.list",
           "value": "creativeCommon", "checked_at": "2026-10-02T09:14:55Z"},
          {"stage": "download", "source": "yt-dlp 2026.10.26",
           "value": "Creative Commons Attribution license (reuse allowed)",
           "checked_at": "2026-10-28T14:03:11Z"}
        ],
        "licensed_content_claim": false,
        "evidence_sha256": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"
      },
      "attribution": "\"Example video title\" by Example Channel, https://www.youtube.com/watch?v=abc123XYZ_-, licensed under CC BY. Changes: transcoded, clipped 83.4-91.8 s, captions added.",
      "modifications": ["transcode_720p", "clip", "generated_caption"],
      "review_flags": {"third_party_media_suspected": false, "identifiable_people": true},
      "status": "active",
      "last_rechecked_at": "2026-11-01T03:00:00Z",
      "takedown": null
    }

    This sits alongside the general lineage fields (shard, sha256, clip boundaries, captions) described in the dataset pipeline article.

    Reference corpora built on YouTube CC content

    YouTube-Commons by PleIAs is the best-known public example, and it's text only: transcripts, no audio or video. The dataset card gives two counts. The headline says transcripts of 2,063,066 videos. The content section says 22,709,724 original and automatically translated transcripts from 3,156,703 videos across 721,136 channels, nearly 45 billion words, 71% English among original languages. The dataset is tagged cc-by-4.0. The card states that every video was shared under CC BY and that the dataset provides title, link, channel name and upload date for each one. It also notes an ongoing debate over using free-licensed content for pre-training and recommends preferring open research use. Even if you never use the data, it's a useful model for how to ship attribution at scale.

    The large research video datasets (Panda-70M, HD-VILA-100M, InternVid) are not CC filters. Their licenses cover annotations, not the videos. See downloading Panda-70M, HD-VILA-100M and InternVid. If you derive a CC subset from their ID lists, run the same license checks described here.

    What CC BY does and doesn't settle for AI training

    Question What CC BY gives you What remains open
    Copying and adapting the uploader's content Permission for any purpose, including commercial, subject to attribution Only covers what the uploader actually had the right to license
    Attribution "Any reasonable manner" What's reasonable for millions of clips inside model weights isn't settled. Most teams publish a dataset card plus a per-sample source list
    Endorsement Explicitly no implied endorsement Don't market a model as "made with" named creators
    People in the video Nothing: the deed names privacy and publicity rights as possible limits Faces and voices are personal data under GDPR, and you need your own lawful basis
    Whether training needs a license at all Exceptions such as fair use and fair dealing aren't affected by CC licenses Rules differ by jurisdiction (e.g. EU text-and-data-mining exceptions with opt-outs, US fair use cases in litigation)

    In practice, recording attribution at scale means publishing a dataset card that names the license family and the collection window. Alongside it, ship a machine-readable source list (video URL, title, channel name, channel URL, license, changes made), as YouTube-Commons does with its provenance columns. Keep internal-only fields (review flags, evidence hashes) separate from what you publish.

    A practical pipeline

    [1] DISCOVER   search filter (manual seeds) / search.list videoLicense=creativeCommon
         |          / channel + playlist enumeration (yt-dlp --flat-playlist)
         v
    [2] LICENSE CHECK (metadata only)
         |          videos.list status.license  +  yt-dlp license field
         |          unknown -> API re-check; disagreement -> quarantine
         v
    [3] FILTER     duration, language, resolution, live status,
         |          licensedContent, review flags (music-heavy, re-upload channels)
         v
    [4] DOWNLOAD   re-check license just before fetch; store check #2
         v
    [5] RECORD     per-sample license record + attribution string + evidence sha256
         v
    [6] MAINTAIN   periodic re-check (license changed / private / deleted)
                    takedowns.jsonl -> rebuild shards -> new release

    Notes on the stages that are specific to CC collection:

    • Channel-level priors. Channels that license everything under CC BY (lectures, public institutions, open-education projects) are higher-yield and lower-risk than random CC search hits. Re-upload and compilation channels deserve manual review.
    • Music and third-party content. Flag videos with music-heavy audio, or titles and descriptions mentioning other creators, for review or exclusion. For speech data, see audio-only downloads for speech datasets. For transcripts, see downloading subtitles and auto-captions.
    • Re-checks. A videos.list call costs 1 unit per call, so periodic license and availability sweeps are cheap compared with search. Log every change, and don't overwrite earlier checks.
    • Takedowns. Keep a takedown list keyed by video_id and channel_id, rebuild affected shards, publish a new release, and delete raw files. Add a contact route in the dataset card.

    Compliance

    YouTube's Terms of Service restrict automated access and downloading, and a CC BY label doesn't change those terms. Copyright and licensing still apply. Filtering for Creative Commons narrows the problem but doesn't clear third-party material inside a video. Faces and voices are personal data under regimes like GDPR, regardless of the uploader's license choice. Honor takedown and opt-out requests, and design the pipeline so you can. Get legal review for your jurisdiction and use case before collecting, training on or redistributing this data.

    If you'd rather not run this yourself

    Corpora's fully managed collection delivers YouTube video, audio, subtitles and structured metadata (JSONL/Parquet with sha256 checksums and manifests) to S3, GCS, Azure Blob or private object storage, with a free sample batch. Teams running their own yt-dlp stack can use dedicated 1–40 Gbps unmetered proxy capacity (pricing).

    FAQ

    Can I use Creative Commons YouTube videos to train AI models?

    CC BY permits reuse for any purpose with attribution, but it only covers rights the uploader had. It doesn't address privacy or publicity rights of people in the video, and the legal treatment of training differs by jurisdiction. Filter for CC, keep per-sample records, and get legal review.

    Which Creative Commons license does YouTube use?

    Only CC BY (Attribution). YouTube Help currently links CC BY 4.0. Archived 2023–2024 versions of the page linked the CC BY 3.0 legal code, and per-video metadata doesn't expose a version.

    How do I search only Creative Commons videos with the YouTube API?

    Call search.list with type=video and videoLicense=creativeCommon, then confirm each ID with videos.list and status.license == "creativeCommon". The default quota is 100 search.list calls per day.

    What does yt-dlp's license field show for Creative Commons videos?

    "Creative Commons Attribution license (reuse allowed)". Standard videos show "Standard YouTube License". Filter with --match-filters "license*='Creative Commons'", and treat a missing field as unknown.

    How should I attribute thousands of CC BY videos in a dataset?

    Publish a dataset card plus a source list with title, channel name, channel URL, video URL, license and the changes you made, as YouTube-Commons does. Never imply that creators endorse the model.