Tag: deduplication

  • URL Frontier Design: Dedup, Politeness and Recrawls at Scale

    The fetchers in a large crawler are the easy part. The hard part is the URL frontier: the component that decides what to fetch next, avoids duplicate fetches, stays polite to every host and survives crawler traps. This guide covers a practical URL frontier design: canonicalization, seen-URL sets with real Bloom filter math, Mercator-style per-host queues, sitemaps, conditional recrawls, Scrapy's built-in pieces and trap defenses, with runnable Python.

    TL;DR

    • Canonicalize before you dedupe (RFC 3986 section 6): lowercase scheme and host, drop default ports and fragments, remove dot segments, normalize percent-encoding. Sorting and stripping query parameters are heuristics, not standards.
    • Use the canonical form as a lookup key, and fetch the original URL.
    • Size the seen-set with arithmetic: a Bloom filter needs -ln(p) / (ln 2)^2 bits per URL, about 9.6 bits at 1% false positives. A false positive means a URL is silently skipped.
    • Separate priority from politeness: front queues for priority, one back queue per host, and a heap of next-allowed fetch times (the Mercator design).
    • Recrawl with validators (If-None-Match, If-Modified-Since) and budget every host so traps cannot eat the crawl.

    What a URL frontier does

    The standard reference is the section The URL frontier in chapter 20 of Manning, Raghavan and Schütze's Introduction to Information Retrieval, which describes the frontier of the Mercator crawler (Heydon and Najork, "Mercator: A scalable, extensible Web crawler", World Wide Web 2(4), 1999). It balances two goals:

    • Priority: important pages, and pages that change often, get fetched first and refetched more often.
    • Politeness: most links point to the same host, so a plain priority queue produces bursts against one server.

    In front of the queues sit a canonicalizer and a seen-URL set; behind them, a URL database with fetch history for recrawls.

    URL canonicalization: safe rules and guesses

    RFC 3986 section 6 notes that extensive normalization "is often used by spiders and indexing engines to prune a search space", and that comparison methods are "designed to minimize false negatives while strictly avoiding false positives". Keep that mindset: every rule merges more URLs, and a wrong merge is a page you never fetch.

    Safe by the spec

    • Case: scheme and host are case-insensitive; lowercase them. Uppercase hex digits in percent-encodings (%3a to %3A). Path and query are case-sensitive.
    • Percent-encoding: decode encoded unreserved characters (letters, digits, -._~), so %7Euser equals ~user.
    • Dot segments: resolve . and .. with remove_dot_segments (section 5.2.4).
    • Port and empty path: RFC 9110 section 4.2.3 says the normal form omits the default port (80 or 443) and uses / for an empty path.
    • Fragments: RFC 3986 section 6.1 says fragments should be excluded when comparing URIs to select a network action such as retrieval.

    Heuristics with caveats

    • Sorting query parameters suits most sites but no standard guarantees it. Scrapy's w3lib.url.canonicalize_url sorts by key, then value, and drops fragments by default.
    • Stripping parameters: utm_*, gclid and fbclid are usually safe to drop. Session and view parameters are site-specific; use per-host rules.
    • Trailing slash, www., index.html, http versus https are different URLs. Let the server's redirects tell you, and mark both source and target as seen.
    • Hash routes in single-page apps may matter if you render pages; see Playwright headless browser scraping at scale.

    A stdlib-only canonicalizer with the safe rules plus sorted, de-tracked queries:

    import re
    from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
    
    DEFAULT_PORTS = {"http": 80, "https": 443}
    UNRESERVED = set("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~")
    TRACKING = re.compile(r"^(utm_[a-z]+|gclid|fbclid|msclkid)$", re.IGNORECASE)
    PCT = re.compile(r"%([0-9A-Fa-f]{2})")
    
    
    def normalize_pct(s):
        def fix(m):
            ch = chr(int(m.group(1), 16))
            return ch if ch in UNRESERVED else "%" + m.group(1).upper()
        return PCT.sub(fix, s)
    
    
    def remove_dot_segments(path):  # segment-stack form of RFC 3986 5.2.4
        segments = path.split("/")
        out = []
        for seg in segments:
            if seg == ".":
                continue
            if seg == "..":
                if len(out) > 1:
                    out.pop()
                continue
            out.append(seg)
        if segments[-1] in (".", ".."):
            out.append("")  # "/a/b/.." -> "/a/"
        return "/".join(out)
    
    
    def canonicalize(url, drop=TRACKING, sort_query=True):
        parts = urlsplit(url.strip())
        scheme = parts.scheme.lower()
        if scheme not in DEFAULT_PORTS:
            raise ValueError(f"unsupported scheme: {scheme!r}")
        host = parts.hostname  # lowercased, userinfo removed
        if not host:
            raise ValueError(f"no host in {url!r}")
        if ":" in host:
            host = f"[{host}]"  # IPv6 literal
        port = parts.port  # ValueError on a malformed port
        netloc = host if port in (None, DEFAULT_PORTS[scheme]) else f"{host}:{port}"
        path = remove_dot_segments(normalize_pct(parts.path)) or "/"
        pairs = [(k, v) for k, v in parse_qsl(parts.query, keep_blank_values=True)
                 if not drop.match(k)]
        if sort_query:
            pairs.sort()
        return urlunsplit((scheme, netloc, path, urlencode(pairs), ""))
    
    
    print(canonicalize("HTTP://www.Example.com:80/a/./b/../c/%7euser?utm_source=x&b=2&a=1#top"))
    # http://www.example.com/a/c/~user?a=1&b=2

    Userinfo is dropped (RFC 9110 section 4.2.4 deprecates it). Because urlencode re-encodes the query, fetch the original URL, not the key; w3lib's docstring gives the same warning.

    Seen-URL sets: Bloom filters and exact stores

    A Bloom filter (Bloom, "Space/Time Trade-offs in Hash Coding with Allowable Errors", CACM 1970, which the Mercator paper cites) stores set membership in m bits with k hash functions. With n items inserted:

    false positive rate     p ≈ (1 - e^(-k*n/m))^k
    optimal hash count      k = (m/n) * ln 2
    bits for a target p     m = -n * ln(p) / (ln 2)^2
    Target p Bits per URL Memory for n = 10^9 k
    1% 4.605 / 0.4805 ≈ 9.59 ≈ 1.2 GB 7
    0.1% 6.908 / 0.4805 ≈ 14.38 ≈ 1.8 GB 10

    An exact set of 64-bit URL hashes needs 8 bytes × 10^9 = 8 GB before index overhead, and the birthday bound puts expected colliding pairs at about n² / 2^65 ≈ 0.03. So the Bloom filter saves roughly 4–7× memory for a known loss rate.

    Know which way the error goes: a Bloom filter sometimes says "seen" for a new URL, so about p of new URLs are silently skipped. Size it for the final n, because the rate climbs fast past it, and a plain Bloom filter cannot delete or resize.

    import hashlib
    import math
    
    
    class BloomFilter:
        def __init__(self, n, p):
            self.m = math.ceil(-n * math.log(p) / math.log(2) ** 2)  # bits
            self.k = max(1, round(self.m / n * math.log(2)))          # hash functions
            self.bits = bytearray((self.m + 7) // 8)
    
        def _positions(self, item):
            d = hashlib.blake2b(item.encode(), digest_size=16).digest()
            h1 = int.from_bytes(d[:8], "little")
            h2 = int.from_bytes(d[8:], "little") | 1
            return [(h1 + i * h2) % self.m for i in range(self.k)]  # double hashing
    
        def add(self, item):
            """Set the item's bits; return True if it was (probably) present already."""
            present = True
            for pos in self._positions(item):
                byte, bit = divmod(pos, 8)
                if not (self.bits[byte] >> bit) & 1:
                    present = False
                    self.bits[byte] |= 1 << bit
            return present

    In our check, probing 100,000 new URLs against BloomFilter(100_000, 0.01) filled with 100,000 others gave about 1.0% false positives, as predicted.

    Exact stores

    • RocksDB or another embedded key-value store: key = hash of the canonical URL, value = fetch state. It doubles as the recrawl database; a Bloom filter in RAM in front keeps most lookups off disk.
    • Redis: SADD on a set of hashes is an atomic "add and tell me if it was new" across workers. Redis also has Bloom filters (BF.RESERVE, BF.ADD) and cuckoo filters (CF.*, which support deletion). It is all RAM, so the memory math still applies.
    • Distributed crawls: partition by host so each node owns its hosts' state.

    URL dedup is not content dedup: mirrors and parameter variants serve the same page under different URLs. Catch those after fetching; see MinHash deduplication for LLM training data.

    Per-host queues and politeness, Mercator style

    In the IR book's account of Mercator, a prioritizer assigns each URL to one of F front queues. Each back queue holds URLs from one host, and a heap holds the earliest time each back queue's host may be contacted again. A fetcher pops the heap, waits if needed, fetches one URL and pushes the host back with a new time. Two rules of thumb from the book: wait about ten times the last fetch duration before the next request to that host, and keep about three times as many back queues as crawler threads.

    The sketch below keeps that structure but uses a dictionary of per-host back queues instead of a fixed number of them. Put it in one file with the two blocks above.

    import heapq
    import random
    import time
    from collections import deque
    from urllib.parse import urlsplit
    
    
    class Frontier:
        def __init__(self, priorities=3, min_delay=1.0, delay_factor=10.0,
                     expected_urls=10_000_000, fp_rate=0.001):
            self.front = [deque() for _ in range(priorities)]  # 0 = highest priority
            self.back = {}         # host -> deque of URLs
            self.ready = []        # heap of (earliest_next_fetch, host)
            self.not_before = {}   # host -> earliest next fetch
            self.seen = BloomFilter(expected_urls, fp_rate)
            self.min_delay, self.delay_factor = min_delay, delay_factor
    
        def add(self, url, priority=1):
            if self.seen.add(canonicalize(url)):
                return False
            self.front[priority].append(url)
            return True
    
        def _refill(self, max_moves=1000):
            weights = [2 ** (len(self.front) - i) for i in range(len(self.front))]
            for _ in range(max_moves):
                nonempty = [i for i, q in enumerate(self.front) if q]
                if not nonempty:
                    return
                i = random.choices(nonempty, [weights[j] for j in nonempty])[0]
                url = self.front[i].popleft()
                host = urlsplit(url).hostname
                if host not in self.back:
                    self.back[host] = deque()
                    t = max(time.monotonic(), self.not_before.get(host, 0.0))
                    heapq.heappush(self.ready, (t, host))
                self.back[host].append(url)
    
        def next_url(self):
            if not self.ready:
                self._refill()
            if not self.ready:
                return None
            t, host = heapq.heappop(self.ready)
            return self.back[host].popleft(), host, max(0.0, t - time.monotonic())
    
        def done(self, host, fetch_seconds):
            t = time.monotonic() + max(self.min_delay, self.delay_factor * fetch_seconds)
            self.not_before[host] = t
            if self.back[host]:
                heapq.heappush(self.ready, (t, host))
            else:
                del self.back[host]
            self._refill(max_moves=100)
    
    
    f = Frontier(min_delay=0.2)
    for u in ["https://example.com/", "https://EXAMPLE.com:443/#top",
              "https://example.org/a?utm_source=news", "https://example.org/a",
              "https://example.org/b"]:
        print(u, "->", "queued" if f.add(u) else "duplicate")
    while (item := f.next_url()) is not None:
        url, host, wait = item
        time.sleep(wait)
        print(f"fetch {url} (waited {wait:.2f}s)")
        f.done(host, fetch_seconds=0.01)

    This queues three URLs, flags two variants as duplicates, and delays the second example.org fetch by 0.2 seconds. A host leaves the heap while its fetch is in flight, so there is one request per host at a time. Production versions keep most of each queue on disk (as the IR book notes) and add robots.txt, Retry-After and adaptive rates, covered in web scraping rate limiting, retries and backoff.

    Priority signals and sitemaps

    Useful priority signals: depth from seeds, the host's track record, URL patterns and sitemaps. The sitemaps.org protocol specifies:

    • At most 50,000 URLs and 50MB (52,428,800 bytes) uncompressed per sitemap; gzip is allowed. Index files may list up to 50,000 sitemaps with the same size limit, and carry a lastmod per sitemap so you can skip unchanged ones.
    • <loc> must be under 2,048 characters, and URLs must share the sitemap's host and protocol.
    • <changefreq> is "a hint and not a command"; <priority> (0.0–1.0, default 0.5) only ranks URLs within one site.
    • Sites can advertise sitemaps with a Sitemap: line in robots.txt.

    Use sitemaps for discovery and lastmod to order recrawls within a host.

    Recrawl scheduling with conditional requests

    A long-running frontier also schedules revisits. Store the ETag and Last-Modified from each response and send them back as If-None-Match and If-Modified-Since (RFC 9110 section 13). An unchanged page can then come back as 304 Not Modified with no content. RFC 9110 says a recipient must ignore If-Modified-Since when If-None-Match is present, so the ETag wins. Many servers send neither validator, so also compare your own content hash.

    import hashlib
    
    import httpx
    
    
    def revisit(client: httpx.Client, url, state):
        """Use httpx.Client(follow_redirects=True). state holds etag, last_modified, sha256, interval."""
        headers = {}
        if state.get("etag"):
            headers["If-None-Match"] = state["etag"]
        if state.get("last_modified"):
            headers["If-Modified-Since"] = state["last_modified"]
        r = client.get(url, headers=headers)
        interval = state.get("interval", 86_400)
        if r.status_code == 304:
            changed = False
        else:
            r.raise_for_status()
            digest = hashlib.sha256(r.content).hexdigest()
            changed = digest != state.get("sha256")
            state.update(sha256=digest, etag=r.headers.get("ETag"),
                         last_modified=r.headers.get("Last-Modified"))
        # heuristic: revisit changing pages sooner, stable ones later, within 1 h to 30 days
        state["interval"] = max(3_600, min(30 * 86_400, interval / 2 if changed else interval * 1.5))
        return changed

    The halve-or-grow rule is a heuristic; the IR book warns that spam pages can change on every fetch, so weigh quality too.

    Scrapy's built-in frontier

    Scrapy's scheduler and dupefilter form a small URL frontier. From the settings reference and jobs docs (Scrapy 2.20 at the time of writing):

    • DUPEFILTER_CLASS defaults to scrapy.dupefilters.RFPDupeFilter, which fingerprints the canonicalized URL, method and body and keeps fingerprints in memory. Scrapy 2.20 added scrapy.dupefilters.DiskDupeFilter, which keeps them in SQLite.
    • JOBDIR persists pending requests (requests.queue/), fingerprints (requests.seen) and spider state, so a cleanly stopped crawl resumes. Use one directory per job and the same Scrapy version.
    • SCHEDULER_PRIORITY_QUEUE defaults to DownloaderAwarePriorityQueue, which the docs recommend for crawling many domains in parallel.
    • JOBDIR_SYNC_EVERY (2.20+) also saves state during the crawl; surviving a hard kill needs the SQLite disk queue types (2.19+).
    # settings.py
    JOBDIR = "crawls/frontier-1"   # or: scrapy crawl myspider -s JOBDIR=crawls/frontier-1
    DUPEFILTER_CLASS = "scrapy.dupefilters.DiskDupeFilter"               # 2.20+
    JOBDIR_SYNC_EVERY = 1000                                              # 2.20+, default 0
    SCHEDULER_DISK_QUEUE = "scrapy.squeues.PickleLifoSQLiteQueue"         # 2.19+
    SCHEDULER_START_DISK_QUEUE = "scrapy.squeues.PickleFifoSQLiteQueue"   # 2.19+
    DEPTH_LIMIT = 20               # default 0 = unlimited
    URLLENGTH_LIMIT = 2083         # the default

    Requests with dont_filter=True bypass the dupefilter; unpicklable requests are lost on pause.

    Crawler traps

    Traps generate unique URLs without new content: endless calendars, faceted search, session IDs, relative-link bugs producing /a/b/a/b/a/b/. Dedup cannot catch them. Defenses, cheapest first:

    • Hard limits: depth, URL length, query parameter count.
    • Per-host budgets: cap URLs queued per host, and raise the cap only for hosts that keep yielding distinct, useful documents.
    • Pattern counters: replace digits and IDs with placeholders (/calendar/{n}/{n}) and cap URLs per pattern; cap distinct values per query key.
    • Repetition checks: reject paths whose segment sequence repeats.
    • Content feedback: demote patterns whose new URLs keep hashing to content you already have.

    For the full text pipeline, see the Common Crawl LLM pretraining data pipeline.

    Compliance

    The frontier is the natural place for robots.txt checks and opt-out lists. Site terms of service often restrict automated access (YouTube's Terms of Service, for example, restrict automated access and downloading). Copyright and licensing apply to what you collect and redistribute, privacy law such as GDPR applies to personal data including faces and voices, and you should honor takedown and opt-out requests. Get legal review for your jurisdiction and use case.

    If you would rather not run the egress yourself

    If your frontier is in good shape and egress capacity is the bottleneck, Corpora offers dedicated 1–40 Gbps unmetered proxy capacity (HTTP(S) and SOCKS5, works with custom crawlers) and a separate residential proxy product. Keep politeness keyed on the target host either way.

    FAQ

    What is a URL frontier?

    The structure that holds URLs waiting to be fetched and decides their order, combining deduplication, priority and per-host politeness.

    How big a Bloom filter do I need for a billion URLs?

    About 1.2 GB at 1% false positives (7 hashes) and 1.8 GB at 0.1% (10), from m = -n ln(p) / (ln 2)^2.

    Should I sort query parameters when normalizing URLs?

    For dedup keys, usually yes; Scrapy's canonicalize_url does. No standard guarantees it, so fetch the original URL and add per-host exceptions when order matters.

    Does Scrapy remember seen URLs between runs?

    Only with JOBDIR set; rerunning with the same JOBDIR restores fingerprints and pending requests.

  • MinHash Deduplication for LLM Training Data: Math, Tools, Costs

    Web text is full of copies: mirrors, syndicated news, templated pages and the same article scraped by ten aggregators. Exact hashing catches only identical copies; MinHash deduplication catches the rest at a cost that scales to billions of documents. This guide covers the math (shingles, Jaccard, MinHash, LSH banding), the exact-dedup layers in front of it, runnable datasketch and datatrove code, FineWeb's parameters and the cost at scale.

    TL;DR

    • Layer it: URL and exact-hash dedup first (cheap), then MinHash LSH for near-duplicate documents, then optionally suffix-array exact-substring dedup for repeated passages.
    • Banding sets the threshold. With b bands of r rows, a pair with Jaccard similarity s becomes a candidate with probability 1 − (1 − s^r)^b, and the threshold sits near (1/b)^(1/r).
    • FineWeb's settings: word 5-grams, 112 hashes as 14 bands × 8 rows, targeting roughly 75% similarity. That gives about 0.72 as the S-curve midpoint, 77% recall at s = 0.75 and 92% at s = 0.8.
    • Per-dump beat global for FineWeb. Deduplicating each Common Crawl snapshot independently scored better than one global pass, which kept only 4T tokens (vs 20T) and left lower-quality data behind.
    • The bottleneck is clustering, not hashing. Signatures parallelize; turning pair matches into duplicate clusters is the single-machine, memory-heavy step.

    Why deduplicate LLM training data

    The clearest evidence is still Lee et al., "Deduplicating Training Data Makes Language Models Better" (ACL 2022). They found a single 61-word English sentence repeated over 60,000 times in C4. Models trained on their deduplicated data emitted memorized text ten times less often and needed fewer training steps to reach the same or better accuracy. They also found train-test overlap affecting over 4% of the validation sets of standard datasets.

    The structure mirrors our guide to near-duplicate video detection: cheap exact checks, approximate matching, then clustering and a keep rule. For text, the signal is set similarity over n-grams rather than perceptual hashes or embeddings.

    Layer 1: exact deduplication

    Start with what is free:

    • URL dedup. Normalize URLs and drop repeats before you fetch or extract anything (see our Common Crawl pipeline guide).
    • Document hashing. Hash normalized text (lowercased, whitespace collapsed, punctuation stripped) and keep one document per hash. A Bloom filter trades a small false-positive rate for memory. text-dedup ships a Bloom-filter mode; datatrove has ExactDedupSignature/ExactFindDedups/ExactDedupFilter.
    • Line or paragraph hashing. Per the FineWeb paper, C4 deduplicates over three-line windows and CC-100 at paragraph level. This strips repeated navigation and footers but can leave holes in documents.

    Exact substring dedup with suffix arrays

    Lee et al. introduced ExactSubstr: concatenate the whole dataset into one sequence, build a suffix array, and scan adjacent suffixes for shared prefixes of at least k tokens. They chose k = 50 BPE tokens, and removed every repeated span rather than whole documents. On C4 this removed 7.18% of training tokens, while their MinHash-based NearDup flagged 3.04% of C4 examples. The two overlap heavily: 77% of examples NearDup removed from C4 had at least one verbatim 50-token match.

    The released Rust implementation, google-research/deduplicate-text-datasets, is honest about cost: for a C4-sized dataset (~300 GB) the README asks for as many cores as possible (they used 96), more than 600 GB of RAM and over 1 TB of disk. The core commands are self-similar to find repeated spans and collect to merge them into byte ranges:

    cargo run self-similar --data-file data/wiki40b.test --length-threshold 100 \
      --cache-dir /tmp/cache --num-threads 8
    cargo run collect --data-file data/wiki40b.test --cache-dir /tmp/cache \
      --length-threshold 100 > /tmp/wiki40b.test.remove.byterange

    The threshold is in bytes; the README equates the paper's 50 tokens with 100 bytes, so check how your data is encoded before copying the number. Removing spans can leave sentences stitched together mid-word. The authors report this did not hurt their models, and they ran the deduplicator twice because removals can create new duplicates.

    Layer 2: MinHash near-duplicate detection, with the math

    Shingles and Jaccard similarity

    Represent each document as a set of shingles: overlapping word n-grams after normalization. "the cat sat on the mat" with n = 3 gives {the cat sat, cat sat on, sat on the, on the mat}. Two documents A and B are compared with Jaccard similarity:

    J(A, B) = |A ∩ B| / |A ∪ B|

    The n-gram size controls sensitivity. Changing one word in a document invalidates up to n shingles, so with n = 5 a single edit costs up to five shingles out of the set. Short n makes same-topic documents look similar; long n makes small template variations look different. FineWeb, Lee et al. and text-dedup's example config all use 5-grams.

    MinHash signatures

    Comparing shingle sets directly is too expensive at web scale. MinHash, from Broder's 1997 work on document resemblance, compresses each set into a fixed-length signature. Apply a random hash function h to every shingle and keep the minimum. For a random permutation, the probability that two sets share the same minimum equals their Jaccard similarity:

    P[ min h(A) = min h(B) ] = J(A, B)

    Repeat with k independent hash functions to get a signature of k integers. The fraction of positions where two signatures agree is an unbiased estimate of J, with standard error sqrt(J(1 − J)/k). At J = 0.75 and k = 112 that is about 0.04, good enough for a dedup decision near a threshold but not for fine distinctions.

    LSH banding and the S-curve

    Signatures still need pairwise comparison, which is quadratic. Locality-sensitive hashing avoids it. Split each k-value signature into b bands of r rows (b × r = k), and hash each band to a bucket. Two documents become a candidate pair if they collide on all r rows of at least one band. For a pair with Jaccard similarity s:

    P(band matches)      = s^r
    P(no band matches)   = (1 - s^r)^b
    P(candidate)         = 1 - (1 - s^r)^b

    Plotted against s, this is an S-curve. Its steepest point is approximately at

    t ≈ (1/b)^(1/r)

    which is the approximation given in Chapter 3 of Mining of Massive Datasets (Leskovec, Rajaraman, Ullman). More rows per band push the threshold up and make it stricter; more bands push it down and raise recall. Here is the curve for three configurations, computed from the formula above:

    Config (b × r) Hashes t ≈ (1/b)^(1/r) s = 0.5 s = 0.6 s = 0.7 s = 0.75 s = 0.8 s = 0.9
    20 × 5 100 0.55 0.470 0.802 0.975 0.996 ~1.000 ~1.000
    14 × 8 (FineWeb) 112 0.72 0.053 0.211 0.565 0.772 0.924 ~1.000
    9 × 13 117 0.84 0.001 0.012 0.084 0.195 0.399 0.929

    Read the FineWeb row carefully. A pair at exactly 75% similarity is caught only 77% of the time, and pairs at 60% still slip in as candidates 21% of the time. LSH is a soft threshold. If false positives matter, verify each candidate pair with the true Jaccard on shingles (or edit similarity, as Lee et al. did) before deleting anything. Lee et al. used 9,000 hashes in 450 bands of 20 rows (threshold ≈ 0.74), then kept only pairs with edit similarity above 0.8.

    Doing it in Python with datasketch

    datasketch (documentation at version 2.0.0 as of this writing) provides MinHash and MinHashLSH. The LSH docs explain that MinHashLSH(threshold=..., num_perm=...) searches for the (b, r) pair that minimizes weighted false-positive and false-negative area, that weights=(fp, fn) skews that choice, and that params=(b, r) pins the bands yourself. The script below runs exact dedup, then MinHash LSH with FineWeb's 14 × 8 layout, and verifies candidates against true Jaccard:

    import hashlib
    import re
    
    from datasketch import MinHash, MinHashLSH
    
    NUM_PERM = 112          # 14 bands x 8 rows, as in FineWeb
    NGRAM = 5
    
    def normalize(text):
        text = text.lower()
        text = re.sub(r"[^\w\s]", " ", text)
        return text.split()
    
    def shingles(text, n=NGRAM):
        words = normalize(text)
        if len(words) < n:
            return {" ".join(words)} if words else set()
        return {" ".join(words[i:i + n]) for i in range(len(words) - n + 1)}
    
    def exact_key(text):
        return hashlib.sha256(" ".join(normalize(text)).encode("utf-8")).hexdigest()
    
    def jaccard(a, b):
        return len(a & b) / len(a | b) if a | b else 0.0
    
    docs = {
        "a": "Affordable and convenient holiday flights take off from your departure country, Canada. "
             "From May 2019 to October 2019, Condor flights to your dream destination will be roughly 6 a week! "
             "Book your Halifax flight now, and look forward to your Switzerland destination!",
        "b": "Affordable and convenient holiday flights take off from your departure country, Canada. "
             "From May 2019 to October 2019, Condor flights to your dream destination will be roughly 7 a week! "
             "Book your Halifax flight now, and look forward to your Switzerland destination!",
        "c": "affordable and convenient holiday flights take off from your departure country canada "
             "from may 2019 to october 2019 condor flights to your dream destination will be roughly 6 a week "
             "book your halifax flight now and look forward to your switzerland destination",
        "d": "MinHash estimates the Jaccard similarity between two sets by comparing the minimum "
             "hash values of their elements under many random hash functions.",
    }
    
    # 1) exact dedup on normalized text
    seen, unique = set(), {}
    for key, text in docs.items():
        h = exact_key(text)
        if h not in seen:
            seen.add(h)
            unique[key] = text
    print("after exact dedup:", list(unique))
    
    # 2) near-dup with MinHash LSH
    lsh = MinHashLSH(num_perm=NUM_PERM, params=(14, 8))
    sets, sigs, keep = {}, {}, []
    for key, text in unique.items():
        s = shingles(text)
        m = MinHash(num_perm=NUM_PERM)
        m.update_batch([g.encode("utf-8") for g in s])
        sets[key], sigs[key] = s, m
        candidates = lsh.query(m)
        # verify candidates against the true Jaccard before dropping anything
        if any(jaccard(s, sets[c]) >= 0.75 for c in candidates):
            continue
        lsh.insert(key, m)
        keep.append(key)
    
    print("kept:", keep)
    print("true J(a,b) = %.3f, MinHash estimate = %.3f"
          % (jaccard(sets["a"], sets["b"]), sigs["a"].jaccard(sigs["b"])))

    With datasketch 2.0.0 this prints after exact dedup: ['a', 'b', 'd'] (c is a normalized copy of a), then kept: ['a', 'd'], with a true Jaccard of 0.767 between a and b and a MinHash estimate within about 0.01 of it. Changing one digit in a 42-word page dropped similarity to 0.77, which is why thresholds around 0.7–0.8 are common.

    A version note from the API docs: since 2.0.0 the default permutation scheme is "affine32", which produces different hash values than earlier versions, so persisted LSH indexes must be rebuilt with sketches of a single scheme. datasketch also supports Redis and Cassandra storage, but at web scale a sort-based pipeline avoids the index entirely.

    Doing it at scale with datatrove

    datatrove is the library Hugging Face used to build FineWeb. Its MinHash dedup runs in four stages that never hold an index in memory: signatures are written to disk per bucket, sorted, and merged.

    1. MinhashDedupSignature: shingle and hash each document, write one sorted file per bucket per task.
    2. MinhashDedupBuckets: merge-sort signatures within each bucket and emit matching pairs. Task count must be a multiple of num_buckets.
    3. MinhashDedupCluster: union-find over all pairs to form duplicate clusters. Runs as a single task.
    4. MinhashDedupFilter: re-read the input and drop all but one document per cluster. The input and task count must match stage 1.

    Adapted from the repo's minhash_deduplication.py example to run locally:

    from datatrove.executor.local import LocalPipelineExecutor
    from datatrove.pipeline.dedup import MinhashDedupSignature
    from datatrove.pipeline.dedup.minhash import (
        MinhashConfig,
        MinhashDedupBuckets,
        MinhashDedupCluster,
        MinhashDedupFilter,
    )
    from datatrove.pipeline.readers import JsonlReader
    from datatrove.pipeline.writers.jsonl import JsonlWriter
    from datatrove.utils.hashing import HashConfig
    from datatrove.utils.typeshelper import Languages
    
    config = MinhashConfig(
        hash_config=HashConfig(precision=64),
        n_grams=5,
        num_buckets=14,
        hashes_per_bucket=8,
    )
    BASE = "work/minhash"
    TASKS = 8
    reader = JsonlReader("data/input/")   # *.jsonl with a "text" field
    
    stage1 = LocalPipelineExecutor(
        pipeline=[reader, MinhashDedupSignature(output_folder=f"{BASE}/signatures",
                                                config=config, language=Languages.english)],
        tasks=TASKS, logging_dir="logs/signatures")
    
    stage2 = LocalPipelineExecutor(
        pipeline=[MinhashDedupBuckets(input_folder=f"{BASE}/signatures",
                                      output_folder=f"{BASE}/buckets", config=config)],
        tasks=config.num_buckets, logging_dir="logs/buckets", depends=stage1)
    
    stage3 = LocalPipelineExecutor(
        pipeline=[MinhashDedupCluster(input_folder=f"{BASE}/buckets",
                                      output_folder=f"{BASE}/remove_ids", config=config)],
        tasks=1, logging_dir="logs/clusters", depends=stage2)
    
    stage4 = LocalPipelineExecutor(
        pipeline=[reader,
                  MinhashDedupFilter(input_folder=f"{BASE}/remove_ids",
                                     exclusion_writer=JsonlWriter(f"{BASE}/removed")),
                  JsonlWriter(output_folder=f"{BASE}/deduplicated")],
        tasks=TASKS, logging_dir="logs/filter", depends=stage3)
    
    stage4.run()   # launches the dependency chain

    The defaults in MinhashConfig are already 5-grams, 14 buckets and 8 hashes per bucket, and the source comments the threshold as (1/14)^(1/8) ≈ 0.72 with 0.924 inclusion probability at s = 0.8. Two behaviors are worth knowing before you trust the output, both documented in the MinhashDedupCluster docstring:

    • The surviving copy is not quality-ranked. The union-find root is kept, with no quality or source preference. To choose yourself, save cluster IDs with save_cluster_id=True and apply your own rule.
    • Filter before you dedup. Quality filters applied afterwards can remove the kept copy even when a discarded copy would have passed.

    Clustering is transitive: if A and B both match C, all three share a cluster even if A and B never collided. Inspect giant clusters manually.

    For a config-driven alternative, text-dedup bundles MinHash + LSH, SimHash, suffix-array substring dedup (wrapping the Google repo) and Bloom-filter exact dedup behind TOML configs (python -m text_dedup.minhash). Its example MinHash config uses num_perm = 240, threshold = 0.7, ngram_size = 5 and check_false_positive = true.

    Choosing parameters

    Setting FineWeb What changing it does
    Shingle Word 5-grams (English word tokenizer, normalized text) Smaller n: more topical false positives. Larger n: misses templated variants.
    Total hashes 112 More hashes: sharper S-curve, linearly more compute and storage.
    Bands × rows 14 × 8 Sets the threshold via (1/b)^(1/r); see the table above.
    Target similarity ~0.75 Lower catches more boilerplate-heavy variants and more false positives.
    Keep rule One random document per cluster Rank by source, length or quality score if that matters to you.

    Whatever you pick, calibrate it. Sample candidate pairs in similarity bands (0.6–0.7, 0.7–0.8, 0.8+), label them as duplicate or not, and report precision with confidence intervals. Our post on dataset audits with Wilson intervals covers sample sizes.

    Global vs per-dump dedup: FineWeb's result

    The FineWeb paper found that more dedup is not always better. The team first ran MinHash globally across all 96 Common Crawl snapshots, starting from the most recent. On the oldest snapshots this removed as much as 90% of the base-filtered data, leaving 4 trillion tokens. Models trained on a 350B-token sample barely beat non-deduplicated data and scored far below RefinedWeb.

    To see why, they took the 2013-48 snapshot and compared the ~31B tokens global dedup kept against 171B tokens from deduplicating, on its own, the ~460B tokens it had removed. The removed data trained the better model; the kept 10% held more ads, keyword lists and badly formatted text.

    Deduplicating each snapshot independently, with the same parameters, kept 20 trillion tokens and matched RefinedWeb. The authors' hypothesis: the gain comes from removing huge clusters (hundreds of thousands of documents) present in every crawl, while deduplicating small clusters (fewer than ~100 copies, roughly the number of crawls) can hurt. None of their "lighter" global variants beat per-snapshot dedup. That is a result for their data and benchmarks, not a law, but it is a good reason to run ablations before deduplicating globally.

    What it costs at scale

    Some arithmetic for a hypothetical corpus of 1 billion documents averaging 1,000 words:

    • Hashing: ~1,000 shingles × 112 hash functions ≈ 112,000 multiply-mod operations per document, ~1.1 × 10^14 in total. Embarrassingly parallel CPU work, plus tokenization.
    • Signature storage (datatrove, 64-bit): each bucket record is 8 hashes × 8 bytes + a 4-byte document ID = 68 bytes; × 14 buckets = 952 bytes per document, or ~952 GB for 1B documents. With HashConfig(precision=32) records are 36 bytes, ~504 GB total, at the cost of more hash collisions.
    • Bucket matching: a sort-merge per bucket, parallel across buckets and hash ranges, mostly I/O-bound.
    • Clustering: one task that holds the union-find over every matched document in memory. The datatrove example reserves mem_per_cpu_gb=70 with 2 CPUs for this stage. Per-dump dedup keeps this step smaller.

    By comparison, suffix-array ExactSubstr is memory-hungry: the Google README asks for more than 600 GB of RAM and over 1 TB of disk for C4. Measure throughput on your own hardware before sizing a cluster.

    Two last notes. For image-text pairs, MinHash on captions catches templated alt text but not re-hosted images; pair it with image-level dedup as in the img2dataset and CLIP filtering guide. And search your evaluation sets against the training data with a looser threshold than you use for dedup.

    Compliance

    Dedup doesn't change your obligations for the data you keep. Respect site terms of service and robots.txt when crawling. Copyright and licensing apply to each document, and a mirror's license may not match the original's. Web text contains personal data subject to GDPR and similar laws; FineWeb, for example, anonymized email and public IP addresses before release. Store duplicate cluster IDs so opt-out and takedown requests propagate to every copy, not just the URL named in the request. Get legal review for your jurisdiction and use case.

    If you run your own crawler

    URL-level dedup before fetching is the cheapest bandwidth saving there is. For what remains, Corpora offers dedicated unmetered proxy capacity for your own collection stack (HTTP(S) and SOCKS5, compatible with custom crawlers), and separately residential proxies. Pair either with sensible rate limiting and backoff.

    FAQ

    What Jaccard threshold should I use for MinHash deduplication?

    Most published pipelines sit between 0.7 and 0.8 on word 5-grams: FineWeb targeted about 0.75, text-dedup's example uses 0.7, and Lee et al. combined MinHash at about 0.74 with an edit-similarity check at 0.8. Label sample pairs from your own data before you commit.

    How many permutations (num_perm) do I need?

    Enough to make the S-curve sharp at your threshold. FineWeb used 112 (14 × 8) and datasketch defaults to 128. More permutations reduce estimation error (standard error sqrt(J(1 − J)/k)) but cost linearly more compute and storage. If precision matters, verify candidates with exact Jaccard instead of adding many more hashes.

    MinHash or SimHash?

    MinHash estimates Jaccard similarity over shingle sets; SimHash compares single fingerprints by Hamming distance. On the text-dedup README benchmarks MinHash scored higher and ran faster. The Pile, SlimPajama and RefinedWeb all used MinHash, per the FineWeb paper's survey.

    Should I deduplicate globally or per Common Crawl snapshot?

    In FineWeb's ablations, per-snapshot dedup won. If you combine many crawls, test both on your own data.

    Is MinHash enough, or do I also need exact substring dedup?

    MinHash removes whole near-duplicate documents. It does not remove a boilerplate paragraph repeated inside otherwise unique documents. Suffix-array ExactSubstr does that, at much higher memory cost. Line-level hashing is a cheaper middle ground.

  • Common Crawl to LLM Pretraining Data: WARC-to-Text Pipeline

    Common Crawl is the default starting point for open LLM pretraining corpora, but the raw archive is mostly navigation menus, spam, duplicates and pages in languages you don't want. This guide walks through the pipeline that recent open datasets such as FineWeb use to turn Common Crawl into training text: which file format to read, how to extract text, language ID, heuristic and model-based quality filters, deduplication and PII. It uses Hugging Face's datatrove library as the reference implementation and cites published numbers, not guesses.

    TL;DR

    • Read WARC, not WET, if quality matters. FineWeb's ablations showed that text extracted from WARC with trafilatura trained better models than Common Crawl's WET text, which keeps too much boilerplate.
    • Order the cheap steps first: URL blocklist, then text extraction, language ID, heuristic filters (Gopher, C4, FineWeb rules), MinHash dedup, PII scrubbing, and an optional model-based quality classifier.
    • Language ID: FineWeb used fastText lid.176 with English score ≥ 0.65. FineWeb2 switched to GlotLID for 1,000+ languages.
    • Dedup per snapshot, not globally. FineWeb found that global MinHash across 96 snapshots hurt quality. Per-snapshot dedup matched RefinedWeb.
    • Start from datatrove's fineweb.py, then change the parts you disagree with.

    WARC vs WAT vs WET: what Common Crawl actually ships

    Each monthly Common Crawl snapshot (named like CC-MAIN-2026-43) comes in three main formats, described on the Get Started page:

    Format Contents CC-MAIN-2026-43 size (compressed) Use it for
    WARC Raw HTTP request/response records, including full HTML and headers 86.19 TiB, 100,000 files Your own text extraction (recommended)
    WAT JSON metadata computed from WARC: headers, title, meta tags, outlinks 14.06 TiB, 100,000 files Link graphs, URL discovery, image-text candidates
    WET Plain text extracted by Common Crawl, one conversion record per page 5.85 TiB, 100,000 files Quick prototypes, cheap language stats

    Sizes are from the CC-MAIN-2026-43 index page, which reports 2.16 billion pages for that crawl. WARC is about 15x larger than WET, and that ratio is the main cost of re-extracting.

    Why pipelines re-extract from WARC

    The FineWeb paper (Penedo et al., 2024) is direct about it: WET files "retained too much boilerplate and menu text." The authors trained 1.71B-parameter ablation models on 28B tokens of trafilatura-extracted WARC text and of WET text, applying only fastText English filtering. The WARC version "clearly results in a more performant model." Common Crawl's own WET sample of a Wikipedia page starts with "Jump to content / Main menu / Navigation...", which shows the problem.

    Two extractors dominate:

    • trafilatura was used by RefinedWeb and FineWeb. extract(html, favor_precision=True) prunes more boilerplate. It is accurate but CPU-heavy.
    • Resiliparse HTML2Text is a fast, rule-based extractor. extract_plain_text(html, main_content=True) drops navigation, sidebars and footers. It pairs with FastWARC, a Rust WARC parser that the Resiliparse docs benchmark at 1.5x to 5x faster than WARCIO on typical setups.

    If extraction CPU is your bottleneck, compare both on a few thousand pages from your target domains before committing a full snapshot to either.

    Accessing the data: S3, HTTPS and the indexes

    The data sits in s3://commoncrawl/ in us-east-1, and anyone can download it over HTTPS from https://data.commoncrawl.org/, no AWS account needed. Common Crawl says to run AWS compute in us-east-1 to avoid inter-region charges. Its Get Started page currently shows aws s3 with --no-sign-request, but an older section on the same page says S3 API access requires authenticated users. If anonymous S3 calls fail, use HTTPS or signed requests.

    CRAWL=CC-MAIN-2026-43
    # list WARC paths (100,000 lines for this crawl)
    curl -s https://data.commoncrawl.org/crawl-data/$CRAWL/warc.paths.gz | gunzip > warc.paths
    head -n 2 warc.paths
    
    # download one file over HTTPS
    curl -O "https://data.commoncrawl.org/$(head -n 1 warc.paths)"
    
    # or, from EC2 in us-east-1
    aws s3 cp "s3://commoncrawl/$(head -n 1 warc.paths)" .

    A full WARC set is big. At a sustained 10 Gbps: 86.19 TiB × 8.796e12 bits/TiB ÷ 1e10 bit/s ≈ 75,800 s, or about 21 hours, before any processing. Our bandwidth sizing guide covers the same math for other workloads. Download failures happen, so wrap fetches in retries with backoff (see rate limiting, retries and backoff). Check the infra status page before blaming your own code.

    Use the index when you don't need everything

    Common Crawl publishes two indexes. The CDXJ index (served at index.commoncrawl.org) is built for per-URL lookups. The URL Index, previously called the columnar index, is Parquet under s3://commoncrawl/cc-index/table/cc-main/warc/. You can query it with Athena, Spark or DuckDB. Each row includes warc_filename, warc_record_offset, warc_record_length, content_languages (from CLD2) and content_truncated, so you can select pages by domain or language and fetch only those records:

    SELECT url, warc_filename, warc_record_offset, warc_record_length
    FROM ccindex.ccindex
    WHERE crawl = 'CC-MAIN-2026-43'
      AND subset = 'warc'
      AND url_host_registered_domain = 'example.org'
      AND content_languages LIKE 'eng%'
    LIMIT 100;
    # fetch a single record with an HTTP range request
    curl -s -r$OFFSET-$(($OFFSET+$LENGTH-1)) \
      "https://data.commoncrawl.org/$FILENAME" > record.warc.gz

    Range requests work well for targeted subsets such as a domain list or a language. To process a whole snapshot, stream entire WARC files, because millions of tiny requests cost more than sequential reads. Also note the truncation erratum: captures were cut at 1 MiB before CC-MAIN-2025-13 and at 5 MiB from then on. Drop or flag truncated records, because their HTML is often incomplete.

    Step 1: extraction you can run on one machine

    This minimal extractor uses FastWARC and Resiliparse, written against the Resiliparse 2.0 docs. It reads only response records, keeps HTML, and writes JSONL with the URL and date. Keep those two fields, because you will need them later for URL filters, takedowns and per-domain stats.

    # pip install fastwarc resiliparse
    import json, sys
    from fastwarc.warc import ArchiveIterator, WarcRecordType, WarcHeader
    from resiliparse.parse.encoding import detect_encoding, bytes_to_str
    from resiliparse.extract.html2text import extract_plain_text
    
    def extract(warc_path, out_path):
        with open(warc_path, "rb") as f, open(out_path, "w", encoding="utf-8") as out:
            for rec in ArchiveIterator(f, record_types=WarcRecordType.response):
                if rec.http_content_type != "text/html":
                    continue
                body = rec.reader.read()
                html = bytes_to_str(body, detect_encoding(body))
                text = extract_plain_text(html, main_content=True)
                if len(text.split()) < 50:
                    continue
                out.write(json.dumps({
                    "url": rec.headers.get(WarcHeader.WARC_TARGET_URI),
                    "date": rec.headers.get(WarcHeader.WARC_DATE),
                    "text": text,
                }, ensure_ascii=False) + "\n")
    
    if __name__ == "__main__":
        extract(sys.argv[1], sys.argv[2])

    To use trafilatura instead, replace the extraction line with trafilatura.extract(html, favor_precision=True) and handle a None return.

    Step 2: the full filtering pipeline with datatrove

    datatrove is the library Hugging Face built to produce FineWeb. Its fineweb.py example runs the exact production pipeline on Slurm. The version below follows the same order on a single machine, reading WARC files you have already downloaded:

    # pip install "datatrove[io,processing]"
    from datatrove.executor import LocalPipelineExecutor
    from datatrove.pipeline.readers import WarcReader
    from datatrove.pipeline.extractors import Trafilatura
    from datatrove.pipeline.filters import (
        URLFilter, LanguageFilter, GopherRepetitionFilter,
        GopherQualityFilter, C4QualityFilter, FineWebQualityFilter,
    )
    from datatrove.pipeline.writers.jsonl import JsonlWriter
    
    CRAWL = "CC-MAIN-2026-43"
    
    LocalPipelineExecutor(
        pipeline=[
            WarcReader("data/warc/", glob_pattern="*.warc.gz",
                       default_metadata={"dump": CRAWL}),
            URLFilter(exclusion_writer=JsonlWriter("out/removed/url")),
            Trafilatura(favour_precision=True),
            LanguageFilter(languages=["en"], language_threshold=0.65),
            GopherRepetitionFilter(),
            GopherQualityFilter(),
            C4QualityFilter(filter_no_terminal_punct=False),
            FineWebQualityFilter(),
            JsonlWriter("out/filtered"),
        ],
        tasks=8, workers=8, logging_dir="logs/cc_filter",
    ).run()

    Every filter accepts an exclusion_writer. Use it while tuning: reading what each rule removed is the fastest way to catch a filter that hurts your domain. datatrove records finished tasks under logging_dir/completions, so rerunning the same executor resumes only failed tasks. Don't change tasks between runs.

    URL filtering

    FineWeb's base filtering started with an adult-content URL blocklist (FineWeb2 links the UT1 blacklists), applied before extraction because that step costs almost nothing. Add your own lists here too: domains from opt-out requests, known spam farms, and anything your legal team has excluded.

    Language identification: lid.176 or GlotLID?

    FineWeb used the fastText lid.176 model (176 languages, 126 MB .bin, licensed CC BY-SA 3.0) and kept documents whose top English score was ≥ 0.65. In datatrove this is LanguageFilter(backend="ft176"), the default. According to the FineWeb2 paper, about 40% of extracted documents passed that English threshold.

    For multilingual work, FineWeb2 switched to GlotLID, which the paper describes as covering 1,880 languages, separating scripts (e.g. arb_Arab vs arb_Latn) and including "noise" labels. In ablations, GlotLID beat FT176 on higher-resource languages and trailed slightly on lower-resource ones. Instead of a single threshold, FineWeb2 sets per-language cutoffs at max(0.3, min(0.9, median − σ)) of each language's confidence distribution. datatrove exposes this as backend="glotlid".

    Heuristic quality filters

    Filter set Representative rules Notes from FineWeb
    Gopher quality (MassiveText) 50–100,000 words; mean word length 3–10; ≤ 10% hash/ellipsis symbols per word; ≤ 90% bullet lines; ≥ 80% words with a letter; ≥ 2 stop words Used with original thresholds in base filtering
    Gopher repetition Duplicate line/paragraph fractions and repeated n-gram ratios Base filtering
    C4 (Raffel et al.) Drop lines mentioning javascript or cookie/terms policies; drop docs with "lorem ipsum" or {; terminal-punctuation line filter Terminal-punctuation filter removed ~30% of tokens, so FineWeb disabled it
    FineWeb custom Line-ending punctuation ratio ≤ 0.12; chars in duplicated lines ≥ 0.1; lines under 30 chars ≥ 0.67 Together removed ~22% of tokens and raised the aggregate score by about 1%

    One detail to check: the paper states the duplicated-line threshold as 0.1, but the current datatrove FineWebQualityFilter source defaults char_duplicates_ratio to 0.01. Set it explicitly to the value you intend. All of these rules were tuned on English. FineWeb2 re-derived thresholds per language from Wikipedia and Common Crawl statistics, so don't reuse English cutoffs for Thai or Chinese.

    Step 3: deduplication

    FineWeb's MinHash setup used word 5-grams and 112 hash functions in 14 buckets of 8, targeting documents that are at least 75% similar, then clustered duplicates transitively and kept one document per cluster. The surprising result: deduplicating all 96 snapshots globally left 4T tokens and barely improved over no dedup, because it disproportionately kept low-quality pages in older crawls. Deduplicating each snapshot independently left 20T tokens and matched RefinedWeb. The datatrove example runs this as four stages (signatures, buckets, clustering, filter). Only the clustering stage runs as a single task. We cover parameter choices and the false-positive math in MinHash deduplication for LLM training data.

    FineWeb2 takes this further with rehydration: it records each kept document's cluster size and uses it to upsample, which improved results even in languages where dedup alone helped little.

    Step 4: PII

    For its public release, FineWeb anonymized email addresses and public IP addresses. datatrove's PIIFormatter does this with regexes, replacing matches with placeholder emails and IPs and leaving private IP ranges alone. That is a minimal baseline. It does not touch names, phone numbers, street addresses or government IDs. If your use case or jurisdiction requires more, add a dedicated detector and audit a sample by hand. Run PII after dedup so you only scrub what you keep.

    Step 5 (optional): model-based quality filtering

    FineWeb-Edu is the best-documented recipe here. Llama-3-70B-Instruct scored 460,000 FineWeb pages for educational value on a 0–5 additive scale. A linear regression head on frozen Snowflake-arctic-embed-m was then trained on 410,000 of those annotations. Keeping documents with an integer score ≥ 3 produced a 1.3T-token subset, with an F1 of 82% for that binary cut on held-out data. The paper reports MMLU going from 33% to 37% and ARC from 46% to 57% in its 350B-token ablations. Classifying all 15T tokens of FineWeb took 6,000 H100 GPU hours.

    The released fineweb-edu-classifier (Apache-2.0) is ready to use. Its model card warns that it targets grade-school level educational content and may not transfer to other domains. The paper also shows it shifts topic mix away from business, entertainment and travel. If you need a different notion of "quality", repeat the recipe with your own rubric: LLM labels on a few hundred thousand samples, then a cheap classifier on embeddings.

    Licensing, privacy and opt-outs

    Common Crawl's Terms of Use say crawled content may be subject to the original sites' own terms, require you to respect third-party copyright, and include an indemnification clause that explicitly covers using crawled content to train AI systems. Common Crawl points publishers to robots.txt for controlling its CCBot crawler and publishes an Opt-Out Ledger of legal removal requests it has received. Apply that ledger, and your own takedown list, to past snapshots too, because older crawls predate those requests. Web text contains personal data, so GDPR and similar laws can apply to your corpus even after regex scrubbing. Keep URL and crawl provenance for every document so you can honor deletion requests. Get legal review for your jurisdiction and intended use before training on or redistributing the result.

    When Common Crawl isn't enough

    Coverage gaps are common: some domains are crawled shallowly, pages that render their content with JavaScript can come out nearly empty from raw HTML, and image-text pairs need their own pipeline (see building an image-text dataset with img2dataset and CLIP filtering). Teams then run a targeted crawler, often headless (Playwright at scale), writing WARC so the same datatrove pipeline applies. If you run that crawler yourself, Corpora sells dedicated unmetered proxy capacity (1–40 Gbps, HTTP(S) and SOCKS5) for your own stack, plus a separate residential proxy product. The same robots.txt, terms-of-service and rate-limit obligations apply either way.

    FAQ

    Should I use Common Crawl WET files or re-extract text from WARC?

    For a production pretraining corpus, re-extract from WARC. FineWeb's ablation showed trafilatura-on-WARC beating WET at equal token budgets. WET is about 15x smaller, so it's still useful for quick language or domain statistics.

    Which language identification model did FineWeb use?

    fastText lid.176, keeping documents with an English score of at least 0.65. FineWeb2, the multilingual follow-up, uses GlotLID with per-language thresholds.

    How large is one Common Crawl snapshot?

    For CC-MAIN-2026-43: 2.16 billion pages, with 86.19 TiB of WARC, 14.06 TiB of WAT and 5.85 TiB of WET (compressed), each split into 100,000 files. Other snapshots differ, so check each crawl's index page.

    Do I need an AWS account to download Common Crawl?

    No. Everything is available over HTTPS at https://data.commoncrawl.org/. AWS mainly helps if you process inside us-east-1, next to the data.

    Can I just use FineWeb instead of processing Common Crawl myself?

    Often, yes. FineWeb (15T tokens, 96 snapshots) and FineWeb-Edu are released under ODC-By. Process Common Crawl yourself when you need newer snapshots, other languages, different filters, or full control over provenance and opt-outs.

  • Video Data Curation for Generative Models: Filters That Matter

    Raw video scraped at scale is mostly unusable for training a generative model as-is: hard cuts in the middle of clips, static slides, burned-in subtitles, shaky phone footage, letterboxing, near-duplicates. Published video model papers spend as much effort on curation as on architecture. This guide lays out the filters that matter, the order to run them in so you don't burn GPU hours on clips you'll throw away, and how to store the results so you can change your mind later.

    TL;DR

    • Split before you filter. Almost every downstream score is meaningless on a clip that contains a hard cut.
    • Run filters in cost order: metadata → shot detection → cheap frame statistics → optical flow → OCR → model-based aesthetic and alignment scores → near-duplicate search.
    • Store scores, not booleans. Thresholds change with every training run; recomputing scores over millions of clips does not have to.
    • Watch your keep-rate per filter. Over-aggressive aesthetic filters quietly turn a diverse dataset into stock footage.
    • Curation is an experiment, not a preprocessing step. Validate thresholds with small training runs, not by eyeballing samples.

    Why curation matters as much as scale

    Two widely cited papers make the case well. The Stable Video Diffusion paper (arXiv 2311.15127) describes a systematic curation pipeline for its large video pretraining set: cut detection, optical-flow-based motion filtering, text detection to drop clips dominated by on-screen text, and CLIP-based scoring for aesthetics and text-image similarity. The authors report that models trained on the curated subset were preferred over models trained on uncurated data. Panda-70M (arXiv 2402.19479) took long YouTube videos, split them into semantically coherent clips, and then invested heavily in captioning those clips.

    The common thread: a smaller, cleaner set of well-segmented clips beats a larger pile of raw footage. If you are building your own set, start from the papers' filter list and calibrate it to your target model.

    The filter stack, in cost order

    Stage Signal Relative cost Catches
    Metadata Duration, resolution, frame rate, aspect ratio, license, language Free (before download) Shorts you don't want, 360p uploads, wrong language
    Shot detection Frame-difference / content-change detection Low (CPU) Hard cuts, fades, montage edits
    Frame statistics Brightness, contrast, blur (Laplacian variance), black-border detection Low (CPU, sampled frames) Dark, blurry, letterboxed or pillarboxed clips
    Motion Optical flow magnitude on sampled frames Medium (GPU or fast CPU flow) Static slides and talking-head stills, or extreme camera shake
    Text / OCR Text detection area ratio Medium Burned-in subtitles, slides, screen recordings, heavy watermarks
    Aesthetic / quality Learned aesthetic or quality predictor on frame embeddings Medium–high (GPU) Low production value, compression artifacts
    Text–video alignment Similarity between caption and frames (CLIP-style) High (needs captions first) Captions that don't describe the clip
    Near-duplicates Perceptual hashes, embedding nearest-neighbor search Medium–high at scale Re-uploads, compilations, reaction videos, your eval set
    Safety / privacy NSFW classifiers, face detection, PII in text Medium Content you can't train on or must handle separately

    The ordering principle is simple: every stage should only see clips that passed all cheaper stages. A metadata filter that drops 30% of candidates before download saves 30% of bandwidth, storage and every GPU hour downstream.

    Stage by stage

    1. Metadata, before you download anything

    Filter on what the platform tells you for free: duration, available resolutions, aspect ratio, upload date, category and license. Decide early what you are training on. A text-to-video model rarely benefits from 12-hour livestreams or vertical 9:16 Shorts unless you specifically want them. See building a YouTube video dataset for multimodal training for the metadata-first workflow, and filtering for Creative Commons videos if licensing drives your selection.

    2. Shot detection and clip splitting

    Run this first on downloaded video, because every later score assumes a clip shows one continuous shot. Content-based detectors (for example PySceneDetect's content and adaptive detectors) handle hard cuts well; fades and dissolves need a threshold-based or learned detector. SVD reports using cut detection at multiple frame rates to catch transitions that a single pass misses.

    Splitting purely on cuts can over-fragment: an interview that alternates between two cameras becomes dozens of 2-second clips. Panda-70M addressed this by stitching adjacent shots back together when they were semantically similar. Decide what a "clip" means for your model — a single shot, or a coherent scene — and set minimum and maximum clip lengths accordingly.

    3. Cheap frame statistics

    Sample a few frames per clip and compute brightness, contrast, a blur score and black-border ratio. These are fast, explainable and catch a surprising share of junk: dark concert footage, out-of-focus phone video, 4:3 content pillarboxed inside 16:9. Crop borders rather than discarding clips where the content itself is fine.

    4. Motion

    Optical flow on a low-frame-rate sample (SVD used 2 FPS) gives you a motion score per clip. The bottom of the distribution is slides, still images with music and near-static talking heads — useless for learning motion. The top of the distribution is often camera shake or rapid editing. Most teams drop both tails, and keep the score so they can rebalance motion levels later.

    5. Text and OCR

    Text detection gives you the fraction of the frame covered by text. High values mean slides, tutorials, screen recordings or burned-in subtitles. A generative model trained on those learns to produce garbled text overlays. Small, persistent text in a corner is often a channel watermark — decide whether to drop or crop.

    6. Aesthetic and quality scores

    Learned aesthetic predictors on CLIP-style embeddings are cheap per frame once embeddings exist, and they correlate with production quality. They also carry taste bias: they reward well-lit, centered, stock-footage-looking frames. Use them to drop the bottom of the distribution, not to keep only the top, unless your target is explicitly high-aesthetic output.

    7. Caption alignment

    Once clips have captions, a text–video similarity score tells you when a caption doesn't describe its clip. This matters most when captions come from titles, descriptions or ASR transcripts, which are frequently about the video rather than what is on screen. Better captions usually beat stricter alignment filtering — see recaptioning video datasets.

    8. Near-duplicates and contamination

    Large web video collections are full of re-uploads, compilations and reaction videos that embed other videos. Two layers work well together: perceptual hashes on keyframes for exact and near-exact copies, and embedding nearest-neighbor search for semantic duplicates. Run the same search against your evaluation sets. Benchmark contamination is easy to introduce and hard to detect after training.

    9. Safety and privacy

    NSFW classifiers, face detection and text PII checks belong in the pipeline, not in a later audit. Faces and voices can be personal data under GDPR and similar laws, so record what you detected even if you keep the clip.

    Store scores, not decisions

    The most expensive mistake is writing only the filtered output. Thresholds change with every model, and recomputing optical flow or embeddings over tens of millions of clips is days of GPU time. Keep one row per clip with every score:

    {
      "clip_id": "dQw4w9WgXcQ_000123",
      "video_id": "dQw4w9WgXcQ",
      "start_s": 61.2,
      "end_s": 68.9,
      "width": 1920,
      "height": 1080,
      "fps": 25.0,
      "shot_boundary_confidence": 0.94,
      "brightness_mean": 0.41,
      "blur_laplacian_var": 212.5,
      "black_border_ratio": 0.0,
      "flow_magnitude_p50": 1.83,
      "ocr_area_ratio": 0.012,
      "aesthetic_score": 5.2,
      "nsfw_score": 0.01,
      "face_count_max": 1,
      "phash_cluster": "c_88412",
      "embedding_dup_of": null,
      "caption_alignment": 0.31,
      "pipeline_version": "curate-2026.11"
    }

    Your training set then becomes a query over this table. A new threshold is a new query, not a new pipeline run.

    Calibrating thresholds

    • Report keep-rate per filter. If one filter removes 60% of clips, look at what it removed before trusting it.
    • Look at the boundary, not the extremes. Sample clips just above and just below each threshold; that is where mistakes live.
    • Check distribution shift. Compare category, language, motion and resolution distributions before and after filtering. Curation that silently removes most sports or most outdoor footage changes what your model learns.
    • Validate with training runs. The real test is a small model trained on two subsets with different thresholds, evaluated on the same prompts.

    Compliance

    Curation decides what goes into the model, so it is also where compliance decisions get enforced. YouTube's Terms of Service restrict automated access and downloading, copyright and licensing apply to the underlying videos, and faces and voices can be personal data. Keep per-clip provenance (source ID, collection time, license at collection time), honor takedown and opt-out requests across derived clips, and get legal review for your jurisdiction and use case.

    If you'd rather not build this yourself

    Corpora collects YouTube video, audio, subtitles and structured metadata at petabyte scale and delivers deduplicated datasets with checksums and manifests to your S3, GCS or Azure bucket, with a free sample batch to validate quality first. See pricing or talk to sales.

    FAQ

    What is the first filter to run on raw video?

    Metadata filtering before download, then shot detection right after. Every other score assumes the clip is a single continuous shot.

    How much data do curation filters typically remove?

    It depends heavily on the source and your thresholds, so measure it on a sample. Web video often loses a large share to cuts, static content, text overlays and duplicates, which is why filtering before download is worth so much.

    Should I filter on aesthetic score?

    Use it to drop the worst clips. Keeping only the top of the distribution biases the dataset toward stock-footage style and reduces diversity.

    How do I avoid training on my evaluation set?

    Run near-duplicate search (perceptual hashes plus embedding similarity) of your training clips against every evaluation set, and record matches as a score so you can exclude them.