Common Crawl to LLM Pretraining Data: WARC-to-Text Pipeline

Written by

in

Common Crawl is the default starting point for open LLM pretraining corpora, but the raw archive is mostly navigation menus, spam, duplicates and pages in languages you don't want. This guide walks through the pipeline that recent open datasets such as FineWeb use to turn Common Crawl into training text: which file format to read, how to extract text, language ID, heuristic and model-based quality filters, deduplication and PII. It uses Hugging Face's datatrove library as the reference implementation and cites published numbers, not guesses.

TL;DR

  • Read WARC, not WET, if quality matters. FineWeb's ablations showed that text extracted from WARC with trafilatura trained better models than Common Crawl's WET text, which keeps too much boilerplate.
  • Order the cheap steps first: URL blocklist, then text extraction, language ID, heuristic filters (Gopher, C4, FineWeb rules), MinHash dedup, PII scrubbing, and an optional model-based quality classifier.
  • Language ID: FineWeb used fastText lid.176 with English score ≥ 0.65. FineWeb2 switched to GlotLID for 1,000+ languages.
  • Dedup per snapshot, not globally. FineWeb found that global MinHash across 96 snapshots hurt quality. Per-snapshot dedup matched RefinedWeb.
  • Start from datatrove's fineweb.py, then change the parts you disagree with.

WARC vs WAT vs WET: what Common Crawl actually ships

Each monthly Common Crawl snapshot (named like CC-MAIN-2026-43) comes in three main formats, described on the Get Started page:

Format Contents CC-MAIN-2026-43 size (compressed) Use it for
WARC Raw HTTP request/response records, including full HTML and headers 86.19 TiB, 100,000 files Your own text extraction (recommended)
WAT JSON metadata computed from WARC: headers, title, meta tags, outlinks 14.06 TiB, 100,000 files Link graphs, URL discovery, image-text candidates
WET Plain text extracted by Common Crawl, one conversion record per page 5.85 TiB, 100,000 files Quick prototypes, cheap language stats

Sizes are from the CC-MAIN-2026-43 index page, which reports 2.16 billion pages for that crawl. WARC is about 15x larger than WET, and that ratio is the main cost of re-extracting.

Why pipelines re-extract from WARC

The FineWeb paper (Penedo et al., 2024) is direct about it: WET files "retained too much boilerplate and menu text." The authors trained 1.71B-parameter ablation models on 28B tokens of trafilatura-extracted WARC text and of WET text, applying only fastText English filtering. The WARC version "clearly results in a more performant model." Common Crawl's own WET sample of a Wikipedia page starts with "Jump to content / Main menu / Navigation...", which shows the problem.

Two extractors dominate:

  • trafilatura was used by RefinedWeb and FineWeb. extract(html, favor_precision=True) prunes more boilerplate. It is accurate but CPU-heavy.
  • Resiliparse HTML2Text is a fast, rule-based extractor. extract_plain_text(html, main_content=True) drops navigation, sidebars and footers. It pairs with FastWARC, a Rust WARC parser that the Resiliparse docs benchmark at 1.5x to 5x faster than WARCIO on typical setups.

If extraction CPU is your bottleneck, compare both on a few thousand pages from your target domains before committing a full snapshot to either.

Accessing the data: S3, HTTPS and the indexes

The data sits in s3://commoncrawl/ in us-east-1, and anyone can download it over HTTPS from https://data.commoncrawl.org/, no AWS account needed. Common Crawl says to run AWS compute in us-east-1 to avoid inter-region charges. Its Get Started page currently shows aws s3 with --no-sign-request, but an older section on the same page says S3 API access requires authenticated users. If anonymous S3 calls fail, use HTTPS or signed requests.

CRAWL=CC-MAIN-2026-43
# list WARC paths (100,000 lines for this crawl)
curl -s https://data.commoncrawl.org/crawl-data/$CRAWL/warc.paths.gz | gunzip > warc.paths
head -n 2 warc.paths

# download one file over HTTPS
curl -O "https://data.commoncrawl.org/$(head -n 1 warc.paths)"

# or, from EC2 in us-east-1
aws s3 cp "s3://commoncrawl/$(head -n 1 warc.paths)" .

A full WARC set is big. At a sustained 10 Gbps: 86.19 TiB × 8.796e12 bits/TiB ÷ 1e10 bit/s ≈ 75,800 s, or about 21 hours, before any processing. Our bandwidth sizing guide covers the same math for other workloads. Download failures happen, so wrap fetches in retries with backoff (see rate limiting, retries and backoff). Check the infra status page before blaming your own code.

Use the index when you don't need everything

Common Crawl publishes two indexes. The CDXJ index (served at index.commoncrawl.org) is built for per-URL lookups. The URL Index, previously called the columnar index, is Parquet under s3://commoncrawl/cc-index/table/cc-main/warc/. You can query it with Athena, Spark or DuckDB. Each row includes warc_filename, warc_record_offset, warc_record_length, content_languages (from CLD2) and content_truncated, so you can select pages by domain or language and fetch only those records:

SELECT url, warc_filename, warc_record_offset, warc_record_length
FROM ccindex.ccindex
WHERE crawl = 'CC-MAIN-2026-43'
  AND subset = 'warc'
  AND url_host_registered_domain = 'example.org'
  AND content_languages LIKE 'eng%'
LIMIT 100;
# fetch a single record with an HTTP range request
curl -s -r$OFFSET-$(($OFFSET+$LENGTH-1)) \
  "https://data.commoncrawl.org/$FILENAME" > record.warc.gz

Range requests work well for targeted subsets such as a domain list or a language. To process a whole snapshot, stream entire WARC files, because millions of tiny requests cost more than sequential reads. Also note the truncation erratum: captures were cut at 1 MiB before CC-MAIN-2025-13 and at 5 MiB from then on. Drop or flag truncated records, because their HTML is often incomplete.

Step 1: extraction you can run on one machine

This minimal extractor uses FastWARC and Resiliparse, written against the Resiliparse 2.0 docs. It reads only response records, keeps HTML, and writes JSONL with the URL and date. Keep those two fields, because you will need them later for URL filters, takedowns and per-domain stats.

# pip install fastwarc resiliparse
import json, sys
from fastwarc.warc import ArchiveIterator, WarcRecordType, WarcHeader
from resiliparse.parse.encoding import detect_encoding, bytes_to_str
from resiliparse.extract.html2text import extract_plain_text

def extract(warc_path, out_path):
    with open(warc_path, "rb") as f, open(out_path, "w", encoding="utf-8") as out:
        for rec in ArchiveIterator(f, record_types=WarcRecordType.response):
            if rec.http_content_type != "text/html":
                continue
            body = rec.reader.read()
            html = bytes_to_str(body, detect_encoding(body))
            text = extract_plain_text(html, main_content=True)
            if len(text.split()) < 50:
                continue
            out.write(json.dumps({
                "url": rec.headers.get(WarcHeader.WARC_TARGET_URI),
                "date": rec.headers.get(WarcHeader.WARC_DATE),
                "text": text,
            }, ensure_ascii=False) + "\n")

if __name__ == "__main__":
    extract(sys.argv[1], sys.argv[2])

To use trafilatura instead, replace the extraction line with trafilatura.extract(html, favor_precision=True) and handle a None return.

Step 2: the full filtering pipeline with datatrove

datatrove is the library Hugging Face built to produce FineWeb. Its fineweb.py example runs the exact production pipeline on Slurm. The version below follows the same order on a single machine, reading WARC files you have already downloaded:

# pip install "datatrove[io,processing]"
from datatrove.executor import LocalPipelineExecutor
from datatrove.pipeline.readers import WarcReader
from datatrove.pipeline.extractors import Trafilatura
from datatrove.pipeline.filters import (
    URLFilter, LanguageFilter, GopherRepetitionFilter,
    GopherQualityFilter, C4QualityFilter, FineWebQualityFilter,
)
from datatrove.pipeline.writers.jsonl import JsonlWriter

CRAWL = "CC-MAIN-2026-43"

LocalPipelineExecutor(
    pipeline=[
        WarcReader("data/warc/", glob_pattern="*.warc.gz",
                   default_metadata={"dump": CRAWL}),
        URLFilter(exclusion_writer=JsonlWriter("out/removed/url")),
        Trafilatura(favour_precision=True),
        LanguageFilter(languages=["en"], language_threshold=0.65),
        GopherRepetitionFilter(),
        GopherQualityFilter(),
        C4QualityFilter(filter_no_terminal_punct=False),
        FineWebQualityFilter(),
        JsonlWriter("out/filtered"),
    ],
    tasks=8, workers=8, logging_dir="logs/cc_filter",
).run()

Every filter accepts an exclusion_writer. Use it while tuning: reading what each rule removed is the fastest way to catch a filter that hurts your domain. datatrove records finished tasks under logging_dir/completions, so rerunning the same executor resumes only failed tasks. Don't change tasks between runs.

URL filtering

FineWeb's base filtering started with an adult-content URL blocklist (FineWeb2 links the UT1 blacklists), applied before extraction because that step costs almost nothing. Add your own lists here too: domains from opt-out requests, known spam farms, and anything your legal team has excluded.

Language identification: lid.176 or GlotLID?

FineWeb used the fastText lid.176 model (176 languages, 126 MB .bin, licensed CC BY-SA 3.0) and kept documents whose top English score was ≥ 0.65. In datatrove this is LanguageFilter(backend="ft176"), the default. According to the FineWeb2 paper, about 40% of extracted documents passed that English threshold.

For multilingual work, FineWeb2 switched to GlotLID, which the paper describes as covering 1,880 languages, separating scripts (e.g. arb_Arab vs arb_Latn) and including "noise" labels. In ablations, GlotLID beat FT176 on higher-resource languages and trailed slightly on lower-resource ones. Instead of a single threshold, FineWeb2 sets per-language cutoffs at max(0.3, min(0.9, median − σ)) of each language's confidence distribution. datatrove exposes this as backend="glotlid".

Heuristic quality filters

Filter set Representative rules Notes from FineWeb
Gopher quality (MassiveText) 50–100,000 words; mean word length 3–10; ≤ 10% hash/ellipsis symbols per word; ≤ 90% bullet lines; ≥ 80% words with a letter; ≥ 2 stop words Used with original thresholds in base filtering
Gopher repetition Duplicate line/paragraph fractions and repeated n-gram ratios Base filtering
C4 (Raffel et al.) Drop lines mentioning javascript or cookie/terms policies; drop docs with "lorem ipsum" or {; terminal-punctuation line filter Terminal-punctuation filter removed ~30% of tokens, so FineWeb disabled it
FineWeb custom Line-ending punctuation ratio ≤ 0.12; chars in duplicated lines ≥ 0.1; lines under 30 chars ≥ 0.67 Together removed ~22% of tokens and raised the aggregate score by about 1%

One detail to check: the paper states the duplicated-line threshold as 0.1, but the current datatrove FineWebQualityFilter source defaults char_duplicates_ratio to 0.01. Set it explicitly to the value you intend. All of these rules were tuned on English. FineWeb2 re-derived thresholds per language from Wikipedia and Common Crawl statistics, so don't reuse English cutoffs for Thai or Chinese.

Step 3: deduplication

FineWeb's MinHash setup used word 5-grams and 112 hash functions in 14 buckets of 8, targeting documents that are at least 75% similar, then clustered duplicates transitively and kept one document per cluster. The surprising result: deduplicating all 96 snapshots globally left 4T tokens and barely improved over no dedup, because it disproportionately kept low-quality pages in older crawls. Deduplicating each snapshot independently left 20T tokens and matched RefinedWeb. The datatrove example runs this as four stages (signatures, buckets, clustering, filter). Only the clustering stage runs as a single task. We cover parameter choices and the false-positive math in MinHash deduplication for LLM training data.

FineWeb2 takes this further with rehydration: it records each kept document's cluster size and uses it to upsample, which improved results even in languages where dedup alone helped little.

Step 4: PII

For its public release, FineWeb anonymized email addresses and public IP addresses. datatrove's PIIFormatter does this with regexes, replacing matches with placeholder emails and IPs and leaving private IP ranges alone. That is a minimal baseline. It does not touch names, phone numbers, street addresses or government IDs. If your use case or jurisdiction requires more, add a dedicated detector and audit a sample by hand. Run PII after dedup so you only scrub what you keep.

Step 5 (optional): model-based quality filtering

FineWeb-Edu is the best-documented recipe here. Llama-3-70B-Instruct scored 460,000 FineWeb pages for educational value on a 0–5 additive scale. A linear regression head on frozen Snowflake-arctic-embed-m was then trained on 410,000 of those annotations. Keeping documents with an integer score ≥ 3 produced a 1.3T-token subset, with an F1 of 82% for that binary cut on held-out data. The paper reports MMLU going from 33% to 37% and ARC from 46% to 57% in its 350B-token ablations. Classifying all 15T tokens of FineWeb took 6,000 H100 GPU hours.

The released fineweb-edu-classifier (Apache-2.0) is ready to use. Its model card warns that it targets grade-school level educational content and may not transfer to other domains. The paper also shows it shifts topic mix away from business, entertainment and travel. If you need a different notion of "quality", repeat the recipe with your own rubric: LLM labels on a few hundred thousand samples, then a cheap classifier on embeddings.

Licensing, privacy and opt-outs

Common Crawl's Terms of Use say crawled content may be subject to the original sites' own terms, require you to respect third-party copyright, and include an indemnification clause that explicitly covers using crawled content to train AI systems. Common Crawl points publishers to robots.txt for controlling its CCBot crawler and publishes an Opt-Out Ledger of legal removal requests it has received. Apply that ledger, and your own takedown list, to past snapshots too, because older crawls predate those requests. Web text contains personal data, so GDPR and similar laws can apply to your corpus even after regex scrubbing. Keep URL and crawl provenance for every document so you can honor deletion requests. Get legal review for your jurisdiction and intended use before training on or redistributing the result.

When Common Crawl isn't enough

Coverage gaps are common: some domains are crawled shallowly, pages that render their content with JavaScript can come out nearly empty from raw HTML, and image-text pairs need their own pipeline (see building an image-text dataset with img2dataset and CLIP filtering). Teams then run a targeted crawler, often headless (Playwright at scale), writing WARC so the same datatrove pipeline applies. If you run that crawler yourself, Corpora sells dedicated unmetered proxy capacity (1–40 Gbps, HTTP(S) and SOCKS5) for your own stack, plus a separate residential proxy product. The same robots.txt, terms-of-service and rate-limit obligations apply either way.

FAQ

Should I use Common Crawl WET files or re-extract text from WARC?

For a production pretraining corpus, re-extract from WARC. FineWeb's ablation showed trafilatura-on-WARC beating WET at equal token budgets. WET is about 15x smaller, so it's still useful for quick language or domain statistics.

Which language identification model did FineWeb use?

fastText lid.176, keeping documents with an English score of at least 0.65. FineWeb2, the multilingual follow-up, uses GlotLID with per-language thresholds.

How large is one Common Crawl snapshot?

For CC-MAIN-2026-43: 2.16 billion pages, with 86.19 TiB of WARC, 14.06 TiB of WAT and 5.85 TiB of WET (compressed), each split into 100,000 files. Other snapshots differ, so check each crawl's index page.

Do I need an AWS account to download Common Crawl?

No. Everything is available over HTTPS at https://data.commoncrawl.org/. AWS mainly helps if you process inside us-east-1, next to the data.

Can I just use FineWeb instead of processing Common Crawl myself?

Often, yes. FineWeb (15T tokens, 96 snapshots) and FineWeb-Edu are released under ODC-By. Process Common Crawl yourself when you need newer snapshots, other languages, different filters, or full control over provenance and opt-outs.