Corpora Blog
Engineering guides on YouTube video and audio datasets, yt-dlp at scale, proxies and bandwidth planning for multimodal AI training data.
-
MinHash Deduplication for LLM Training Data: Math, Tools, Costs
MinHash deduplication for LLM training data: shingles, Jaccard, LSH banding math, FineWeb's 14x8 settings, datasketch and datatrove code, and costs.
-
Common Crawl to LLM Pretraining Data: WARC-to-Text Pipeline
Turn Common Crawl into LLM pretraining text: WARC vs WET, trafilatura extraction, lid.176/GlotLID, FineWeb filters, MinHash dedup and PII scrubbing.
-
Video Dataset Storage Format: WebDataset, Parquet, MDS Shards
Choosing a video dataset storage format: WebDataset tar shards, Parquet metadata, HF streaming, MDS, shard sizing, sha256 manifests and shuffle buffers.
-
Split Video into Clips with PySceneDetect for Training Data
Split video into clips for training data: PySceneDetect detectors, TransNetV2, Panda-70M stitching, frame-accurate ffmpeg cuts, length rules, clip metadata.
-
Video Deduplication: Near-Duplicate Detection for Training Data
Video deduplication for training sets: video IDs and sha256, pHash frames, Chromaprint audio, CLIP/SigLIP embeddings with FAISS, thresholds and costs.
-
Measuring Curated Dataset Accuracy: Audits and Wilson Intervals
Measure curated dataset accuracy with human audits: sample sizes, Wilson confidence intervals, stratified calibration and acceptance rules for data batches.
-
Speech Training Data: What Makes Audio ASR- and TTS-Ready
What makes speech training data usable for ASR and TTS: speech ratio, language ID, speaker count, background music, audio quality, segmentation and labels.
-
Introducing Corpora Curation: YouTube Data Filtered to Spec
Corpora Curation finds YouTube videos that match your training data spec: plain-language requirements, a filter funnel, audio probes and audited batches.
-
Buying Video Training Data: A Checklist for AI Teams
Buying video training data? A checklist for AI teams: provenance, content spec, dedup, metadata schema, delivery format, sample acceptance tests and red flags.