Build an Image-Text Dataset: img2dataset + CLIP Score Filtering

Written by

in

Training a CLIP-style model or the vision tower of a VLM starts with hundreds of millions of image-text pairs, and almost all public pipelines build them the same way: alt-text from Common Crawl, a mass download with img2dataset, then aggressive filtering. This guide walks through that pipeline end to end — where the URLs come from, the img2dataset flags that matter, how to pick CLIP score thresholds, and the dedup, safety and recaptioning steps that separate a usable image-text dataset from a pile of noisy links.

TL;DR

  • Start from an existing URL index unless you have a reason not to: DataComp CommonPool (12.8B pairs) or Re-LAION-5B (5.5B pairs). Both are lists of URLs and captions, not images.
  • Download with img2dataset into WebDataset shards: --input_format parquet --url_col --caption_col --output_format webdataset, then tune --processes_count and --thread_count. It skips images served with X-Robots-Tag: noai and similar directives by default.
  • CLIP score filtering is the baseline that works. LAION kept English pairs with OpenAI ViT-B/32 cosine similarity ≥ 0.28; DataComp found that keeping the top 30% by ViT-L/14 score beat it.
  • Thresholds are model-specific. Use percentiles computed on your own pool, not a number copied from a paper that used a different model.
  • Filter for safety with more than a classifier, dedup exact and near-duplicates, decontaminate against your eval sets, and consider recaptioning.
  • Expect link rot. Every month you wait, more URLs die or change content.

Where image-text pairs come from

The dominant source is HTML alt-text. The LAION-5B paper describes parsing <img> tags from Common Crawl's WAT metadata files, keeping images that have alt-text, detecting language with CLD3, and then downloading and filtering. Roughly 50 billion candidate pairs went in; CLIP filtering removed about 90%, leaving 5.85B pairs (2.32B English). If you want to parse Common Crawl yourself, the mechanics of WARC/WAT handling are covered in our Common Crawl pipeline guide.

In practice most teams start from a published index:

Index Size What you get Notes
DataComp CommonPool 12.8B pairs (also 12.8M, 128M, 1.28B pools) Parquet of URLs, captions, uids, precomputed OpenAI CLIP similarities; optional .npz CLIP features URL index released under CC-BY-4.0; NSFW removed, faces blurred, deduplicated against the benchmark's eval sets
DataComp-1B 1.4B pairs The best filtered subset of CommonPool Downloadable with the same script (--scale datacomp_1b)
Re-LAION-5B 5,526,641,167 pairs Parquet of URLs, captions and scores; "research" and "research-safe" versions Gated access on Hugging Face; Apache-2.0; LAION advises against industrial use and end products

The original LAION-5B metadata was taken down in December 2023 after the Stanford Internet Observatory reported links to suspected CSAM. LAION released Re-LAION-5B in August 2024 after matching against hash lists from IWF and C3P, removing 2,236 links plus privacy-related links reported by Human Rights Watch. If you still hold an old LAION-5B derivative, LAION recommends diffing it against Re-LAION metadata and removing everything that no longer appears.

The DataComp repo's README lists disk requirements per scale: 450 GB of image tars for the 12.8M "small" pool, 45 TB for "large" (1.28B) and 450 TB for "xlarge" (12.8B). Budget storage before you start.

Downloading at scale with img2dataset

img2dataset is the downloader behind LAION and DataComp. It reads a list of URLs (and optionally captions), downloads with a thread pool per process, resizes, and writes shards. A typical run over Parquet metadata:

pip install img2dataset

img2dataset \
  --url_list=metadata/ \
  --input_format=parquet \
  --url_col=URL \
  --caption_col=TEXT \
  --output_format=webdataset \
  --output_folder=s3://my-bucket/imgtext/shards \
  --processes_count=16 \
  --thread_count=64 \
  --image_size=384 \
  --resize_mode=keep_ratio \
  --resize_only_if_bigger=True \
  --number_sample_per_shard=10000 \
  --save_additional_columns='["similarity","punsafe"]'

Column names depend on the metadata you start from (LAION files use URL and TEXT; check your Parquet schema). The flags that matter most:

Flag Default What it does
--input_format txt Also csv, tsv, json, jsonl, parquet and gzipped variants
--url_col / --caption_col url / none Column names for URL and caption
--output_format files webdataset (tar shards), parquet, tfrecord, dummy for benchmarking
--processes_count 1 README advice: set to the number of CPU cores
--thread_count 256 Download threads per process; raise while CPU and bandwidth have headroom
--image_size 256 Target size in pixels
--resize_mode border no, border, keep_ratio, keep_ratio_largest, center_crop
--timeout / --retries 10 s / 0 Per-image timeout and retry count
--compute_hash sha256 Hash of the raw bytes stored in metadata — useful for dedup
--disallowed_header_directives ["noai","noimageai","noindex","noimageindex"] Skip images whose response carries these X-Robots-Tag directives

Two choices you cannot easily undo: resolution and resize mode. The default (256 px, border) pads every image to a square, which suits small CLIP runs but wastes pixels for a VLM that wants native aspect ratios. If you may train at higher resolution later, store larger images with keep_ratio and --resize_only_if_bigger=True; downscaling at train time is cheap, re-downloading is not.

Opt-out handling

By default img2dataset drops any image whose HTTP response includes X-Robots-Tag: noai, noimageai, noindex or noimageindex, and records the reason in the per-sample metadata. Directives can also be scoped to a user agent token (for example X-Robots-Tag: img2dataset: noai); set --user_agent_token to identify your crawler so scoped directives apply to you. The README documents --disallowed_header_directives '[]' to turn the check off; for a dataset you plan to train on, leaving it on is the defensible default. Note that the check is on the image response headers — the downloader does not fetch or evaluate robots.txt.

Throughput and bandwidth

The README's architecture notes estimate that downloading 1,000 average images per second needs about 130 MB/s, and report around 1,350 images/s on a single machine in their benchmark. Converted to link capacity:

Gbps ≈ images_per_second × avg_bytes_per_image × 8 / 1e9
     ≈ 1,000 × 130,000 × 8 / 1e9 ≈ 1.04 Gbps

So each gigabit of sustained capacity buys roughly 1,000 images/s at that average size — about 86M images per day. Your average will differ with source mix and whether you keep originals. Beyond bandwidth, the README flags DNS as a common bottleneck: millions of URLs mean millions of domains, and it recommends running a local caching resolver (knot-resolver or bind9). Use --output_format dummy --disable_all_reencoding True for a short run to find your network ceiling before committing storage. For general retry and politeness patterns, see rate limiting, retries and backoff.

img2dataset has no proxy flag. It downloads with Python's urllib, which honors the standard HTTP_PROXY/HTTPS_PROXY environment variables (HTTP proxies; urllib does not speak SOCKS):

export HTTPS_PROXY=http://USER:[email protected]:8080
export HTTP_PROXY=http://USER:[email protected]:8080
img2dataset --url_list=metadata/ --input_format=parquet ...

For more than one machine, img2dataset supports --distributor pyspark or ray. Its README reports downloading 5.8B LAION-5B images in 7 days on 10 machines (240 TB).

Link rot is real

Every pair in an index is a pointer to someone else's server. Images disappear, move, or get replaced with different content under the same URL. The img2dataset LAION-5B example notes that verifying against MD5 hashes from a May 2022 download dropped about 15% of the dataset as of January 2023 — some of those images had only been slightly modified, but the direction is clear. Practical consequences:

  • Measure your own success rate. img2dataset writes a Parquet file per shard with status and error_message for every sample, plus _stats.json files. Track success rate by domain.
  • Verify hashes where the index provides them (--compute_hash "md5" --verify_hash '["md5","md5"]'). Re-LAION's post notes the precomputed hashes also defend against content being swapped behind a URL, the attack studied by Carlini et al.
  • Treat your downloaded snapshot as the dataset. A URL list from 2022 will not reproduce the same training set today.

CLIP score filtering

CLIP score is the cosine similarity between the image embedding and the caption embedding from a pretrained CLIP model. Low scores catch the classic alt-text failures: file names, "image1.jpg", SEO keyword lists, captions that describe the page rather than the picture.

What the papers actually did:

  • LAION-5B: OpenAI ViT-B/32 for English, removing pairs below 0.28; the multilingual CLIP ViT-B/32 for other languages, removing pairs below 0.26 (section 3.1 of the paper).
  • DataComp: the baselines include a B/32 threshold of 0.25, a reproduction of the LAION-2B filter (B/32 ≥ 0.28 plus English detection), and keeping the top 30% by ViT-L/14 score. The best baseline at the medium, large and xlarge scales intersected the L/14 top-30% filter with an image-based filter that keeps images whose embeddings cluster near ImageNet-1k training images; that produced DataComp-1B, and a ViT-L/14 trained on it reached 79.2% zero-shot ImageNet accuracy, 3.7 points above OpenAI's CLIP ViT-L/14 at the same compute.
  • DataComp's broader finding: smaller, more stringently filtered datasets can produce models that generalize better than larger ones from the same pool. At the 12.8B scale the best filter improved ImageNet zero-shot accuracy by 6.9 points over no filtering.

Scores from different CLIP models are not on the same scale, so "0.28" only means something for OpenAI ViT-B/32. The robust approach is DataComp's: score your pool with one fixed model, then keep a fraction. A minimal scorer over img2dataset shards with OpenCLIP and WebDataset:

import torch, open_clip, webdataset as wds
import pyarrow as pa, pyarrow.parquet as pq

model, _, preprocess = open_clip.create_model_and_transforms("ViT-B-32", pretrained="openai")
model = model.cuda().eval()
tokenizer = open_clip.get_tokenizer("ViT-B-32")

ds = (
    wds.WebDataset("shards/{00000..00099}.tar", shardshuffle=False)
    .decode("pil")
    .to_tuple("__key__", "jpg", "txt")
    .map_tuple(lambda k: k, preprocess, lambda t: t)
    .batched(256)
)
loader = torch.utils.data.DataLoader(ds, batch_size=None, num_workers=8)

keys, scores = [], []
with torch.no_grad(), torch.autocast("cuda"):
    for k, img, txt in loader:
        i = model.encode_image(img.cuda())
        t = model.encode_text(tokenizer(list(txt)).cuda())
        i = i / i.norm(dim=-1, keepdim=True)
        t = t / t.norm(dim=-1, keepdim=True)
        scores += (i * t).sum(dim=-1).float().cpu().tolist()
        keys += list(k)

pq.write_table(pa.table({"key": keys, "clip_b32": scores}), "clip_scores.parquet")

Then choose the cut on the distribution, not from memory:

import pandas as pd
df = pd.read_parquet("clip_scores.parquet")
cut = df["clip_b32"].quantile(0.70)          # keep top 30%
keep = df[df["clip_b32"] >= cut]["key"]
print(f"threshold={cut:.3f} kept={len(keep):,} of {len(df):,}")

If you start from CommonPool, you do not need to run this at all: the metadata Parquet already contains OpenAI CLIP similarities, and the repo's baselines.py implements --name clip_score --arch l14 --fraction 0.3 directly. Before trusting any cut, look at a few hundred samples just above and just below it — see our note on auditing curated data with confidence intervals.

One caveat: CLIP filtering inherits CLIP's biases (the LAION-5B paper has a section on this). If your model needs text-heavy images, charts or non-English captions, check that the filter is not quietly removing them.

Deduplication and decontamination

  • Exact duplicates: img2dataset stores a SHA-256 of the raw bytes by default. Group by it and keep one copy (but consider keeping distinct captions).
  • Near duplicates: resized, re-encoded or watermarked copies of the same image. SemDeDup clusters CLIP embeddings and removes semantic duplicates within clusters; on a LAION subset it removed 50% of the data with minimal performance loss. You already computed embeddings for scoring — save them.
  • Caption duplicates: boilerplate alt-text ("click to enlarge", product templates) repeats millions of times. Text-level near-dup methods from MinHash deduplication apply.
  • Eval decontamination: DataComp deduplicated CommonPool against its evaluation sets. Do the same against every benchmark you plan to report, or your zero-shot numbers are partly memorization.

NSFW and safety filtering

LAION ships per-sample scores (punsafe from a CLIP-embedding NSFW classifier, plus a watermark score), and the LAION-5B paper reports about 3% of images detected as NSFW. Re-LAION-5B documents concrete thresholds: the research version removes samples with p_unsafe > 0.95 when keyword-based text filters also flag them, and the research-safe version additionally removes p_unsafe > 0.45. DataComp removed NSFW content and blurred detected faces in CommonPool; img2dataset can blur regions at download time if you supply boxes via --bbox_col.

LAION's own conclusion from the Re-LAION revision is worth repeating: classifiers alone are not reliable enough to guarantee removal of illegal content, and organizations building datasets from the web should match against hash lists from expert bodies such as IWF and C3P. Treat NSFW classifiers as one layer, not the safety system.

Recaptioning

Even after filtering, alt-text is short and often describes what the publisher wanted indexed, not what is in the image. Recaptioning with a vision-language model is now a standard step. Recap-DataComp-1B fine-tuned a LLaVA-1.5 model with a LLaMA-3-8B language backbone and recaptioned 1.3B DataComp-1B images; the authors report better cross-modal retrieval for CLIP and better prompt following for text-to-image diffusion transformers. Keep both the original alt-text and the synthetic caption: alt-text carries proper nouns and world knowledge a captioner will not invent, and the mix is a training-time choice. For workflow and audits, see recaptioning datasets for caption quality.

A pipeline checklist

  1. Pick a source index and record its version.
  2. Pre-filter metadata before downloading (language, caption length, image size, existing scores) — the img2dataset README recommends this whenever feasible.
  3. Benchmark with --output_format dummy, run a local DNS resolver, download to WebDataset shards with hashes saved.
  4. Score with one fixed CLIP model, keep a percentile, audit around the cut.
  5. Dedup by hash and embedding; decontaminate against eval sets.
  6. Layer safety filters, including hash-list matching.
  7. Recaption with provenance and reshard (see WebDataset and Parquet shards).

Compliance

Index datasets like CommonPool and Re-LAION-5B distribute links and captions; downloading the images means copying third-party content that remains subject to copyright, licensing and site terms. Images contain faces and personal data, so privacy law such as GDPR applies. Honor opt-out signals (img2dataset's X-Robots-Tag handling is a start) and removal requests, keep per-sample provenance so you can act on takedowns, and note that LAION releases Re-LAION for research and advises against industrial use. Get legal review for your jurisdiction and use case.

If you'd rather not run the network layer yourself

A billion-image download is mostly a bandwidth and IP-diversity problem. Corpora offers dedicated 1–40 Gbps unmetered proxy capacity (HTTP(S) and SOCKS5, rotating pools, multi-region) that plugs into crawlers like img2dataset via standard proxy environment variables, and separate residential proxies for sources that block datacenter ranges — the tradeoffs are covered in datacenter vs residential vs ISP proxies.

FAQ

What CLIP score threshold should I use for an image-text dataset?

There is no universal number. LAION used 0.28 with OpenAI ViT-B/32 for English (0.26 for multilingual), but scores are model-specific. DataComp's stronger baseline kept the top 30% by ViT-L/14 score. Compute scores with one model on your pool and choose a percentile, then audit samples near the cut.

Is LAION-5B still available?

The original metadata was withdrawn in December 2023. Re-LAION-5B, released in August 2024, replaces it in "research" and "research-safe" versions with gated access on Hugging Face, and LAION advises against using it in industrial settings or end products.

Does img2dataset respect opt-outs?

By default it skips images whose responses carry X-Robots-Tag noai, noimageai, noindex or noimageindex. It does not evaluate robots.txt, and it cannot see opt-outs expressed elsewhere, so it is a floor, not a full opt-out process.

Why is my download success rate low?

Link rot, DNS failures and timeouts are the usual causes. Check the per-shard Parquet error_message column, run a local caching DNS resolver, consider --retries, and expect older indexes to have more dead links.