{"id":37,"date":"2026-11-15T12:57:37","date_gmt":"2026-11-15T12:57:37","guid":{"rendered":"https:\/\/globalcorpora.com\/blog\/build-image-text-dataset-img2dataset-clip-filtering\/"},"modified":"2026-11-15T12:57:37","modified_gmt":"2026-11-15T12:57:37","slug":"build-image-text-dataset-img2dataset-clip-filtering","status":"publish","type":"post","link":"https:\/\/globalcorpora.com\/blog\/build-image-text-dataset-img2dataset-clip-filtering\/","title":{"rendered":"Build an Image-Text Dataset: img2dataset + CLIP Score Filtering"},"content":{"rendered":"<p>Training a CLIP-style model or the vision tower of a VLM starts with hundreds of millions of image-text pairs, and almost all public pipelines build them the same way: alt-text from Common Crawl, a mass download with img2dataset, then aggressive filtering. This guide walks through that pipeline end to end \u2014 where the URLs come from, the img2dataset flags that matter, how to pick CLIP score thresholds, and the dedup, safety and recaptioning steps that separate a usable image-text dataset from a pile of noisy links.<\/p>\n<h2>TL;DR<\/h2>\n<ul>\n<li><strong>Start from an existing URL index<\/strong> unless you have a reason not to: DataComp CommonPool (12.8B pairs) or Re-LAION-5B (5.5B pairs). Both are lists of URLs and captions, not images.<\/li>\n<li><strong>Download with img2dataset<\/strong> into WebDataset shards: <code>--input_format parquet --url_col --caption_col --output_format webdataset<\/code>, then tune <code>--processes_count<\/code> and <code>--thread_count<\/code>. It skips images served with <code>X-Robots-Tag: noai<\/code> and similar directives by default.<\/li>\n<li><strong>CLIP score filtering is the baseline that works.<\/strong> LAION kept English pairs with OpenAI ViT-B\/32 cosine similarity \u2265 0.28; DataComp found that keeping the top 30% by ViT-L\/14 score beat it.<\/li>\n<li><strong>Thresholds are model-specific.<\/strong> Use percentiles computed on your own pool, not a number copied from a paper that used a different model.<\/li>\n<li><strong>Filter for safety with more than a classifier<\/strong>, dedup exact and near-duplicates, decontaminate against your eval sets, and consider recaptioning.<\/li>\n<li><strong>Expect link rot.<\/strong> Every month you wait, more URLs die or change content.<\/li>\n<\/ul>\n<h2>Where image-text pairs come from<\/h2>\n<p>The dominant source is HTML alt-text. The <a href=\"https:\/\/arxiv.org\/abs\/2210.08402\">LAION-5B paper<\/a> describes parsing <code>&lt;img&gt;<\/code> tags from Common Crawl's WAT metadata files, keeping images that have alt-text, detecting language with CLD3, and then downloading and filtering. Roughly 50 billion candidate pairs went in; CLIP filtering removed about 90%, leaving 5.85B pairs (2.32B English). If you want to parse Common Crawl yourself, the mechanics of WARC\/WAT handling are covered in our <a href=\"https:\/\/globalcorpora.com\/blog\/common-crawl-llm-pretraining-data-pipeline\/\">Common Crawl pipeline guide<\/a>.<\/p>\n<p>In practice most teams start from a published index:<\/p>\n<table>\n<thead>\n<tr>\n<th>Index<\/th>\n<th>Size<\/th>\n<th>What you get<\/th>\n<th>Notes<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><a href=\"https:\/\/arxiv.org\/abs\/2304.14108\">DataComp CommonPool<\/a><\/td>\n<td>12.8B pairs (also 12.8M, 128M, 1.28B pools)<\/td>\n<td>Parquet of URLs, captions, uids, precomputed OpenAI CLIP similarities; optional <code>.npz<\/code> CLIP features<\/td>\n<td>URL index released under CC-BY-4.0; NSFW removed, faces blurred, deduplicated against the benchmark's eval sets<\/td>\n<\/tr>\n<tr>\n<td>DataComp-1B<\/td>\n<td>1.4B pairs<\/td>\n<td>The best filtered subset of CommonPool<\/td>\n<td>Downloadable with the same script (<code>--scale datacomp_1b<\/code>)<\/td>\n<\/tr>\n<tr>\n<td><a href=\"https:\/\/laion.ai\/blog\/relaion-5b\/\">Re-LAION-5B<\/a><\/td>\n<td>5,526,641,167 pairs<\/td>\n<td>Parquet of URLs, captions and scores; \"research\" and \"research-safe\" versions<\/td>\n<td>Gated access on Hugging Face; Apache-2.0; LAION advises against industrial use and end products<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>The original LAION-5B metadata was taken down in December 2023 after the Stanford Internet Observatory reported links to suspected CSAM. LAION released Re-LAION-5B in August 2024 after matching against hash lists from IWF and C3P, removing 2,236 links plus privacy-related links reported by Human Rights Watch. If you still hold an old LAION-5B derivative, LAION recommends diffing it against Re-LAION metadata and removing everything that no longer appears.<\/p>\n<p>The DataComp repo's <a href=\"https:\/\/github.com\/mlfoundations\/datacomp\">README<\/a> lists disk requirements per scale: 450 GB of image tars for the 12.8M \"small\" pool, 45 TB for \"large\" (1.28B) and 450 TB for \"xlarge\" (12.8B). Budget storage before you start.<\/p>\n<h2>Downloading at scale with img2dataset<\/h2>\n<p><a href=\"https:\/\/github.com\/rom1504\/img2dataset\">img2dataset<\/a> is the downloader behind LAION and DataComp. It reads a list of URLs (and optionally captions), downloads with a thread pool per process, resizes, and writes shards. A typical run over Parquet metadata:<\/p>\n<pre><code>pip install img2dataset\n\nimg2dataset \\\n  --url_list=metadata\/ \\\n  --input_format=parquet \\\n  --url_col=URL \\\n  --caption_col=TEXT \\\n  --output_format=webdataset \\\n  --output_folder=s3:\/\/my-bucket\/imgtext\/shards \\\n  --processes_count=16 \\\n  --thread_count=64 \\\n  --image_size=384 \\\n  --resize_mode=keep_ratio \\\n  --resize_only_if_bigger=True \\\n  --number_sample_per_shard=10000 \\\n  --save_additional_columns='[\"similarity\",\"punsafe\"]'<\/code><\/pre>\n<p>Column names depend on the metadata you start from (LAION files use <code>URL<\/code> and <code>TEXT<\/code>; check your Parquet schema). The flags that matter most:<\/p>\n<table>\n<thead>\n<tr>\n<th>Flag<\/th>\n<th>Default<\/th>\n<th>What it does<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><code>--input_format<\/code><\/td>\n<td><code>txt<\/code><\/td>\n<td>Also <code>csv<\/code>, <code>tsv<\/code>, <code>json<\/code>, <code>jsonl<\/code>, <code>parquet<\/code> and gzipped variants<\/td>\n<\/tr>\n<tr>\n<td><code>--url_col<\/code> \/ <code>--caption_col<\/code><\/td>\n<td><code>url<\/code> \/ none<\/td>\n<td>Column names for URL and caption<\/td>\n<\/tr>\n<tr>\n<td><code>--output_format<\/code><\/td>\n<td><code>files<\/code><\/td>\n<td><code>webdataset<\/code> (tar shards), <code>parquet<\/code>, <code>tfrecord<\/code>, <code>dummy<\/code> for benchmarking<\/td>\n<\/tr>\n<tr>\n<td><code>--processes_count<\/code><\/td>\n<td>1<\/td>\n<td>README advice: set to the number of CPU cores<\/td>\n<\/tr>\n<tr>\n<td><code>--thread_count<\/code><\/td>\n<td>256<\/td>\n<td>Download threads per process; raise while CPU and bandwidth have headroom<\/td>\n<\/tr>\n<tr>\n<td><code>--image_size<\/code><\/td>\n<td>256<\/td>\n<td>Target size in pixels<\/td>\n<\/tr>\n<tr>\n<td><code>--resize_mode<\/code><\/td>\n<td><code>border<\/code><\/td>\n<td><code>no<\/code>, <code>border<\/code>, <code>keep_ratio<\/code>, <code>keep_ratio_largest<\/code>, <code>center_crop<\/code><\/td>\n<\/tr>\n<tr>\n<td><code>--timeout<\/code> \/ <code>--retries<\/code><\/td>\n<td>10 s \/ 0<\/td>\n<td>Per-image timeout and retry count<\/td>\n<\/tr>\n<tr>\n<td><code>--compute_hash<\/code><\/td>\n<td><code>sha256<\/code><\/td>\n<td>Hash of the raw bytes stored in metadata \u2014 useful for dedup<\/td>\n<\/tr>\n<tr>\n<td><code>--disallowed_header_directives<\/code><\/td>\n<td><code>[\"noai\",\"noimageai\",\"noindex\",\"noimageindex\"]<\/code><\/td>\n<td>Skip images whose response carries these <code>X-Robots-Tag<\/code> directives<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Two choices you cannot easily undo: <strong>resolution<\/strong> and <strong>resize mode<\/strong>. The default (256 px, <code>border<\/code>) pads every image to a square, which suits small CLIP runs but wastes pixels for a VLM that wants native aspect ratios. If you may train at higher resolution later, store larger images with <code>keep_ratio<\/code> and <code>--resize_only_if_bigger=True<\/code>; downscaling at train time is cheap, re-downloading is not.<\/p>\n<h3>Opt-out handling<\/h3>\n<p>By default img2dataset drops any image whose HTTP response includes <code>X-Robots-Tag: noai<\/code>, <code>noimageai<\/code>, <code>noindex<\/code> or <code>noimageindex<\/code>, and records the reason in the per-sample metadata. Directives can also be scoped to a user agent token (for example <code>X-Robots-Tag: img2dataset: noai<\/code>); set <code>--user_agent_token<\/code> to identify your crawler so scoped directives apply to you. The README documents <code>--disallowed_header_directives '[]'<\/code> to turn the check off; for a dataset you plan to train on, leaving it on is the defensible default. Note that the check is on the image response headers \u2014 the downloader does not fetch or evaluate <code>robots.txt<\/code>.<\/p>\n<h3>Throughput and bandwidth<\/h3>\n<p>The README's architecture notes estimate that downloading 1,000 average images per second needs about 130 MB\/s, and report around 1,350 images\/s on a single machine in their benchmark. Converted to link capacity:<\/p>\n<pre><code>Gbps \u2248 images_per_second \u00d7 avg_bytes_per_image \u00d7 8 \/ 1e9\n     \u2248 1,000 \u00d7 130,000 \u00d7 8 \/ 1e9 \u2248 1.04 Gbps<\/code><\/pre>\n<p>So each gigabit of sustained capacity buys roughly 1,000 images\/s at that average size \u2014 about 86M images per day. Your average will differ with source mix and whether you keep originals. Beyond bandwidth, the README flags DNS as a common bottleneck: millions of URLs mean millions of domains, and it recommends running a local caching resolver (knot-resolver or bind9). Use <code>--output_format dummy --disable_all_reencoding True<\/code> for a short run to find your network ceiling before committing storage. For general retry and politeness patterns, see <a href=\"https:\/\/globalcorpora.com\/blog\/web-scraping-rate-limiting-retries-backoff\/\">rate limiting, retries and backoff<\/a>.<\/p>\n<p>img2dataset has no proxy flag. It downloads with Python's <code>urllib<\/code>, which honors the standard <code>HTTP_PROXY<\/code>\/<code>HTTPS_PROXY<\/code> environment variables (HTTP proxies; <code>urllib<\/code> does not speak SOCKS):<\/p>\n<pre><code>export HTTPS_PROXY=http:\/\/USER:PASS@proxy.example.com:8080\nexport HTTP_PROXY=http:\/\/USER:PASS@proxy.example.com:8080\nimg2dataset --url_list=metadata\/ --input_format=parquet ...<\/code><\/pre>\n<p>For more than one machine, img2dataset supports <code>--distributor pyspark<\/code> or <code>ray<\/code>. Its README reports downloading 5.8B LAION-5B images in 7 days on 10 machines (240 TB).<\/p>\n<h3>Link rot is real<\/h3>\n<p>Every pair in an index is a pointer to someone else's server. Images disappear, move, or get replaced with different content under the same URL. The img2dataset LAION-5B example notes that verifying against MD5 hashes from a May 2022 download dropped about 15% of the dataset as of January 2023 \u2014 some of those images had only been slightly modified, but the direction is clear. Practical consequences:<\/p>\n<ul>\n<li><strong>Measure your own success rate.<\/strong> img2dataset writes a Parquet file per shard with <code>status<\/code> and <code>error_message<\/code> for every sample, plus <code>_stats.json<\/code> files. Track success rate by domain.<\/li>\n<li><strong>Verify hashes where the index provides them<\/strong> (<code>--compute_hash \"md5\" --verify_hash '[\"md5\",\"md5\"]'<\/code>). Re-LAION's post notes the precomputed hashes also defend against content being swapped behind a URL, the attack studied by <a href=\"https:\/\/arxiv.org\/abs\/2302.10149\">Carlini et al.<\/a><\/li>\n<li><strong>Treat your downloaded snapshot as the dataset.<\/strong> A URL list from 2022 will not reproduce the same training set today.<\/li>\n<\/ul>\n<h2>CLIP score filtering<\/h2>\n<p>CLIP score is the cosine similarity between the image embedding and the caption embedding from a pretrained CLIP model. Low scores catch the classic alt-text failures: file names, \"image1.jpg\", SEO keyword lists, captions that describe the page rather than the picture.<\/p>\n<p>What the papers actually did:<\/p>\n<ul>\n<li><strong>LAION-5B:<\/strong> OpenAI ViT-B\/32 for English, removing pairs below <strong>0.28<\/strong>; the multilingual CLIP ViT-B\/32 for other languages, removing pairs below <strong>0.26<\/strong> (section 3.1 of the paper).<\/li>\n<li><strong>DataComp:<\/strong> the baselines include a B\/32 threshold of 0.25, a reproduction of the LAION-2B filter (B\/32 \u2265 0.28 plus English detection), and keeping the <strong>top 30% by ViT-L\/14 score<\/strong>. The best baseline at the medium, large and xlarge scales intersected the L\/14 top-30% filter with an image-based filter that keeps images whose embeddings cluster near ImageNet-1k training images; that produced DataComp-1B, and a ViT-L\/14 trained on it reached 79.2% zero-shot ImageNet accuracy, 3.7 points above OpenAI's CLIP ViT-L\/14 at the same compute.<\/li>\n<li>DataComp's broader finding: <strong>smaller, more stringently filtered datasets can produce models that generalize better<\/strong> than larger ones from the same pool. At the 12.8B scale the best filter improved ImageNet zero-shot accuracy by 6.9 points over no filtering.<\/li>\n<\/ul>\n<p>Scores from different CLIP models are not on the same scale, so \"0.28\" only means something for OpenAI ViT-B\/32. The robust approach is DataComp's: score your pool with one fixed model, then keep a fraction. A minimal scorer over img2dataset shards with <a href=\"https:\/\/github.com\/mlfoundations\/open_clip\">OpenCLIP<\/a> and WebDataset:<\/p>\n<pre><code>import torch, open_clip, webdataset as wds\nimport pyarrow as pa, pyarrow.parquet as pq\n\nmodel, _, preprocess = open_clip.create_model_and_transforms(\"ViT-B-32\", pretrained=\"openai\")\nmodel = model.cuda().eval()\ntokenizer = open_clip.get_tokenizer(\"ViT-B-32\")\n\nds = (\n    wds.WebDataset(\"shards\/{00000..00099}.tar\", shardshuffle=False)\n    .decode(\"pil\")\n    .to_tuple(\"__key__\", \"jpg\", \"txt\")\n    .map_tuple(lambda k: k, preprocess, lambda t: t)\n    .batched(256)\n)\nloader = torch.utils.data.DataLoader(ds, batch_size=None, num_workers=8)\n\nkeys, scores = [], []\nwith torch.no_grad(), torch.autocast(\"cuda\"):\n    for k, img, txt in loader:\n        i = model.encode_image(img.cuda())\n        t = model.encode_text(tokenizer(list(txt)).cuda())\n        i = i \/ i.norm(dim=-1, keepdim=True)\n        t = t \/ t.norm(dim=-1, keepdim=True)\n        scores += (i * t).sum(dim=-1).float().cpu().tolist()\n        keys += list(k)\n\npq.write_table(pa.table({\"key\": keys, \"clip_b32\": scores}), \"clip_scores.parquet\")<\/code><\/pre>\n<p>Then choose the cut on the distribution, not from memory:<\/p>\n<pre><code>import pandas as pd\ndf = pd.read_parquet(\"clip_scores.parquet\")\ncut = df[\"clip_b32\"].quantile(0.70)          # keep top 30%\nkeep = df[df[\"clip_b32\"] &gt;= cut][\"key\"]\nprint(f\"threshold={cut:.3f} kept={len(keep):,} of {len(df):,}\")<\/code><\/pre>\n<p>If you start from CommonPool, you do not need to run this at all: the metadata Parquet already contains OpenAI CLIP similarities, and the repo's <code>baselines.py<\/code> implements <code>--name clip_score --arch l14 --fraction 0.3<\/code> directly. Before trusting any cut, look at a few hundred samples just above and just below it \u2014 see our note on <a href=\"https:\/\/globalcorpora.com\/blog\/measuring-curated-dataset-accuracy-audits-wilson-intervals\/\">auditing curated data with confidence intervals<\/a>.<\/p>\n<p>One caveat: CLIP filtering inherits CLIP's biases (the LAION-5B paper has a section on this). If your model needs text-heavy images, charts or non-English captions, check that the filter is not quietly removing them.<\/p>\n<h2>Deduplication and decontamination<\/h2>\n<ul>\n<li><strong>Exact duplicates:<\/strong> img2dataset stores a SHA-256 of the raw bytes by default. Group by it and keep one copy (but consider keeping distinct captions).<\/li>\n<li><strong>Near duplicates:<\/strong> resized, re-encoded or watermarked copies of the same image. <a href=\"https:\/\/arxiv.org\/abs\/2303.09540\">SemDeDup<\/a> clusters CLIP embeddings and removes semantic duplicates within clusters; on a LAION subset it removed 50% of the data with minimal performance loss. You already computed embeddings for scoring \u2014 save them.<\/li>\n<li><strong>Caption duplicates:<\/strong> boilerplate alt-text (\"click to enlarge\", product templates) repeats millions of times. Text-level near-dup methods from <a href=\"https:\/\/globalcorpora.com\/blog\/minhash-deduplication-llm-training-data\/\">MinHash deduplication<\/a> apply.<\/li>\n<li><strong>Eval decontamination:<\/strong> DataComp deduplicated CommonPool against its evaluation sets. Do the same against every benchmark you plan to report, or your zero-shot numbers are partly memorization.<\/li>\n<\/ul>\n<h2>NSFW and safety filtering<\/h2>\n<p>LAION ships per-sample scores (<code>punsafe<\/code> from a CLIP-embedding NSFW classifier, plus a watermark score), and the LAION-5B paper reports about 3% of images detected as NSFW. Re-LAION-5B documents concrete thresholds: the research version removes samples with <code>p_unsafe &gt; 0.95<\/code> when keyword-based text filters also flag them, and the research-safe version additionally removes <code>p_unsafe &gt; 0.45<\/code>. DataComp removed NSFW content and blurred detected faces in CommonPool; img2dataset can blur regions at download time if you supply boxes via <code>--bbox_col<\/code>.<\/p>\n<p>LAION's own conclusion from the Re-LAION revision is worth repeating: classifiers alone are not reliable enough to guarantee removal of illegal content, and organizations building datasets from the web should match against hash lists from expert bodies such as IWF and C3P. Treat NSFW classifiers as one layer, not the safety system.<\/p>\n<h2>Recaptioning<\/h2>\n<p>Even after filtering, alt-text is short and often describes what the publisher wanted indexed, not what is in the image. Recaptioning with a vision-language model is now a standard step. <a href=\"https:\/\/arxiv.org\/abs\/2406.08478\">Recap-DataComp-1B<\/a> fine-tuned a LLaVA-1.5 model with a LLaMA-3-8B language backbone and recaptioned 1.3B DataComp-1B images; the authors report better cross-modal retrieval for CLIP and better prompt following for text-to-image diffusion transformers. Keep both the original alt-text and the synthetic caption: alt-text carries proper nouns and world knowledge a captioner will not invent, and the mix is a training-time choice. For workflow and audits, see <a href=\"https:\/\/globalcorpora.com\/blog\/recaptioning-video-datasets-caption-quality\/\">recaptioning datasets for caption quality<\/a>.<\/p>\n<h2>A pipeline checklist<\/h2>\n<ol>\n<li>Pick a source index and record its version.<\/li>\n<li>Pre-filter metadata before downloading (language, caption length, image size, existing scores) \u2014 the img2dataset README recommends this whenever feasible.<\/li>\n<li>Benchmark with <code>--output_format dummy<\/code>, run a local DNS resolver, download to WebDataset shards with hashes saved.<\/li>\n<li>Score with one fixed CLIP model, keep a percentile, audit around the cut.<\/li>\n<li>Dedup by hash and embedding; decontaminate against eval sets.<\/li>\n<li>Layer safety filters, including hash-list matching.<\/li>\n<li>Recaption with provenance and reshard (see <a href=\"https:\/\/globalcorpora.com\/blog\/storing-video-datasets-webdataset-parquet-shards\/\">WebDataset and Parquet shards<\/a>).<\/li>\n<\/ol>\n<h2>Compliance<\/h2>\n<p>Index datasets like CommonPool and Re-LAION-5B distribute links and captions; downloading the images means copying third-party content that remains subject to copyright, licensing and site terms. Images contain faces and personal data, so privacy law such as GDPR applies. Honor opt-out signals (img2dataset's <code>X-Robots-Tag<\/code> handling is a start) and removal requests, keep per-sample provenance so you can act on takedowns, and note that LAION releases Re-LAION for research and advises against industrial use. Get legal review for your jurisdiction and use case.<\/p>\n<h2>If you'd rather not run the network layer yourself<\/h2>\n<p>A billion-image download is mostly a bandwidth and IP-diversity problem. Corpora offers <a href=\"https:\/\/globalcorpora.com\/#pricing\">dedicated 1\u201340 Gbps unmetered proxy capacity<\/a> (HTTP(S) and SOCKS5, rotating pools, multi-region) that plugs into crawlers like img2dataset via standard proxy environment variables, and separate <a href=\"https:\/\/proxy.globalcorpora.com\/\">residential proxies<\/a> for sources that block datacenter ranges \u2014 the tradeoffs are covered in <a href=\"https:\/\/globalcorpora.com\/blog\/datacenter-vs-residential-vs-isp-proxies-for-scraping\/\">datacenter vs residential vs ISP proxies<\/a>.<\/p>\n<h2>FAQ<\/h2>\n<h3>What CLIP score threshold should I use for an image-text dataset?<\/h3>\n<p>There is no universal number. LAION used 0.28 with OpenAI ViT-B\/32 for English (0.26 for multilingual), but scores are model-specific. DataComp's stronger baseline kept the top 30% by ViT-L\/14 score. Compute scores with one model on your pool and choose a percentile, then audit samples near the cut.<\/p>\n<h3>Is LAION-5B still available?<\/h3>\n<p>The original metadata was withdrawn in December 2023. Re-LAION-5B, released in August 2024, replaces it in \"research\" and \"research-safe\" versions with gated access on Hugging Face, and LAION advises against using it in industrial settings or end products.<\/p>\n<h3>Does img2dataset respect opt-outs?<\/h3>\n<p>By default it skips images whose responses carry <code>X-Robots-Tag<\/code> <code>noai<\/code>, <code>noimageai<\/code>, <code>noindex<\/code> or <code>noimageindex<\/code>. It does not evaluate <code>robots.txt<\/code>, and it cannot see opt-outs expressed elsewhere, so it is a floor, not a full opt-out process.<\/p>\n<h3>Why is my download success rate low?<\/h3>\n<p>Link rot, DNS failures and timeouts are the usual causes. Check the per-shard Parquet <code>error_message<\/code> column, run a local caching DNS resolver, consider <code>--retries<\/code>, and expect older indexes to have more dead links.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Build an image-text dataset for CLIP\/VLM training: LAION and DataComp sources, img2dataset flags, CLIP score thresholds, dedup, safety and recaptioning.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[100,103,101,98,99,102],"class_list":["post-37","post","type-post","status-publish","format-standard","hentry","category-datasets","tag-clip-filtering","tag-data-curation","tag-datacomp","tag-image-text-dataset","tag-img2dataset","tag-laion"],"_links":{"self":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/posts\/37","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/comments?post=37"}],"version-history":[{"count":0,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/posts\/37\/revisions"}],"wp:attachment":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/media?parent=37"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/categories?post=37"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/tags?post=37"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}