{"id":42,"date":"2026-11-19T03:39:04","date_gmt":"2026-11-19T03:39:04","guid":{"rendered":"https:\/\/globalcorpora.com\/blog\/crawler-url-frontier-deduplication-design\/"},"modified":"2026-11-19T03:39:04","modified_gmt":"2026-11-19T03:39:04","slug":"crawler-url-frontier-deduplication-design","status":"publish","type":"post","link":"https:\/\/globalcorpora.com\/blog\/crawler-url-frontier-deduplication-design\/","title":{"rendered":"URL Frontier Design: Dedup, Politeness and Recrawls at Scale"},"content":{"rendered":"<p>The fetchers in a large crawler are the easy part. The hard part is the URL frontier: the component that decides what to fetch next, avoids duplicate fetches, stays polite to every host and survives crawler traps. This guide covers a practical URL frontier design: canonicalization, seen-URL sets with real Bloom filter math, Mercator-style per-host queues, sitemaps, conditional recrawls, Scrapy's built-in pieces and trap defenses, with runnable Python.<\/p>\n<h2>TL;DR<\/h2>\n<ul>\n<li><strong>Canonicalize before you dedupe<\/strong> (RFC 3986 section 6): lowercase scheme and host, drop default ports and fragments, remove dot segments, normalize percent-encoding. Sorting and stripping query parameters are heuristics, not standards.<\/li>\n<li><strong>Use the canonical form as a lookup key<\/strong>, and fetch the original URL.<\/li>\n<li><strong>Size the seen-set with arithmetic:<\/strong> a Bloom filter needs <code>-ln(p) \/ (ln 2)^2<\/code> bits per URL, about 9.6 bits at 1% false positives. A false positive means a URL is silently skipped.<\/li>\n<li><strong>Separate priority from politeness:<\/strong> front queues for priority, one back queue per host, and a heap of next-allowed fetch times (the Mercator design).<\/li>\n<li><strong>Recrawl with validators<\/strong> (<code>If-None-Match<\/code>, <code>If-Modified-Since<\/code>) and <strong>budget every host<\/strong> so traps cannot eat the crawl.<\/li>\n<\/ul>\n<h2>What a URL frontier does<\/h2>\n<p>The standard reference is the section <a href=\"https:\/\/nlp.stanford.edu\/IR-book\/html\/htmledition\/the-url-frontier-1.html\">The URL frontier<\/a> in chapter 20 of Manning, Raghavan and Sch\u00fctze's <em>Introduction to Information Retrieval<\/em>, which describes the frontier of the Mercator crawler (Heydon and Najork, <a href=\"https:\/\/doi.org\/10.1023\/A:1019213109274\">\"Mercator: A scalable, extensible Web crawler\"<\/a>, <em>World Wide Web<\/em> 2(4), 1999). It balances two goals:<\/p>\n<ul>\n<li><strong>Priority:<\/strong> important pages, and pages that change often, get fetched first and refetched more often.<\/li>\n<li><strong>Politeness:<\/strong> most links point to the same host, so a plain priority queue produces bursts against one server.<\/li>\n<\/ul>\n<p>In front of the queues sit a canonicalizer and a seen-URL set; behind them, a URL database with fetch history for recrawls.<\/p>\n<h2>URL canonicalization: safe rules and guesses<\/h2>\n<p><a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc3986.txt\">RFC 3986<\/a> section 6 notes that extensive normalization \"is often used by spiders and indexing engines to prune a search space\", and that comparison methods are \"designed to minimize false negatives while strictly avoiding false positives\". Keep that mindset: every rule merges more URLs, and a wrong merge is a page you never fetch.<\/p>\n<h3>Safe by the spec<\/h3>\n<ul>\n<li><strong>Case:<\/strong> scheme and host are case-insensitive; lowercase them. Uppercase hex digits in percent-encodings (<code>%3a<\/code> to <code>%3A<\/code>). Path and query are case-sensitive.<\/li>\n<li><strong>Percent-encoding:<\/strong> decode encoded unreserved characters (letters, digits, <code>-._~<\/code>), so <code>%7Euser<\/code> equals <code>~user<\/code>.<\/li>\n<li><strong>Dot segments:<\/strong> resolve <code>.<\/code> and <code>..<\/code> with <code>remove_dot_segments<\/code> (section 5.2.4).<\/li>\n<li><strong>Port and empty path:<\/strong> <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9110.html\">RFC 9110<\/a> section 4.2.3 says the normal form omits the default port (80 or 443) and uses <code>\/<\/code> for an empty path.<\/li>\n<li><strong>Fragments:<\/strong> RFC 3986 section 6.1 says fragments should be excluded when comparing URIs to select a network action such as retrieval.<\/li>\n<\/ul>\n<h3>Heuristics with caveats<\/h3>\n<ul>\n<li><strong>Sorting query parameters<\/strong> suits most sites but no standard guarantees it. Scrapy's <a href=\"https:\/\/github.com\/scrapy\/w3lib\/blob\/master\/w3lib\/url.py\"><code>w3lib.url.canonicalize_url<\/code><\/a> sorts by key, then value, and drops fragments by default.<\/li>\n<li><strong>Stripping parameters:<\/strong> <code>utm_*<\/code>, <code>gclid<\/code> and <code>fbclid<\/code> are usually safe to drop. Session and view parameters are site-specific; use per-host rules.<\/li>\n<li><strong>Trailing slash, <code>www.<\/code>, <code>index.html<\/code>, http versus https<\/strong> are different URLs. Let the server's redirects tell you, and mark both source and target as seen.<\/li>\n<li><strong>Hash routes<\/strong> in single-page apps may matter if you render pages; see <a href=\"https:\/\/globalcorpora.com\/blog\/playwright-headless-browser-web-scraping-at-scale\/\">Playwright headless browser scraping at scale<\/a>.<\/li>\n<\/ul>\n<p>A stdlib-only canonicalizer with the safe rules plus sorted, de-tracked queries:<\/p>\n<pre><code>import re\nfrom urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit\n\nDEFAULT_PORTS = {\"http\": 80, \"https\": 443}\nUNRESERVED = set(\"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~\")\nTRACKING = re.compile(r\"^(utm_[a-z]+|gclid|fbclid|msclkid)$\", re.IGNORECASE)\nPCT = re.compile(r\"%([0-9A-Fa-f]{2})\")\n\n\ndef normalize_pct(s):\n    def fix(m):\n        ch = chr(int(m.group(1), 16))\n        return ch if ch in UNRESERVED else \"%\" + m.group(1).upper()\n    return PCT.sub(fix, s)\n\n\ndef remove_dot_segments(path):  # segment-stack form of RFC 3986 5.2.4\n    segments = path.split(\"\/\")\n    out = []\n    for seg in segments:\n        if seg == \".\":\n            continue\n        if seg == \"..\":\n            if len(out) &gt; 1:\n                out.pop()\n            continue\n        out.append(seg)\n    if segments[-1] in (\".\", \"..\"):\n        out.append(\"\")  # \"\/a\/b\/..\" -&gt; \"\/a\/\"\n    return \"\/\".join(out)\n\n\ndef canonicalize(url, drop=TRACKING, sort_query=True):\n    parts = urlsplit(url.strip())\n    scheme = parts.scheme.lower()\n    if scheme not in DEFAULT_PORTS:\n        raise ValueError(f\"unsupported scheme: {scheme!r}\")\n    host = parts.hostname  # lowercased, userinfo removed\n    if not host:\n        raise ValueError(f\"no host in {url!r}\")\n    if \":\" in host:\n        host = f\"[{host}]\"  # IPv6 literal\n    port = parts.port  # ValueError on a malformed port\n    netloc = host if port in (None, DEFAULT_PORTS[scheme]) else f\"{host}:{port}\"\n    path = remove_dot_segments(normalize_pct(parts.path)) or \"\/\"\n    pairs = [(k, v) for k, v in parse_qsl(parts.query, keep_blank_values=True)\n             if not drop.match(k)]\n    if sort_query:\n        pairs.sort()\n    return urlunsplit((scheme, netloc, path, urlencode(pairs), \"\"))\n\n\nprint(canonicalize(\"HTTP:\/\/www.Example.com:80\/a\/.\/b\/..\/c\/%7euser?utm_source=x&amp;b=2&amp;a=1#top\"))\n# http:\/\/www.example.com\/a\/c\/~user?a=1&amp;b=2<\/code><\/pre>\n<p>Userinfo is dropped (RFC 9110 section 4.2.4 deprecates it). Because <code>urlencode<\/code> re-encodes the query, fetch the original URL, not the key; w3lib's docstring gives the same warning.<\/p>\n<h2>Seen-URL sets: Bloom filters and exact stores<\/h2>\n<p>A Bloom filter (Bloom, <a href=\"https:\/\/doi.org\/10.1145\/362686.362692\">\"Space\/Time Trade-offs in Hash Coding with Allowable Errors\"<\/a>, CACM 1970, which the Mercator paper cites) stores set membership in <em>m<\/em> bits with <em>k<\/em> hash functions. With <em>n<\/em> items inserted:<\/p>\n<pre><code>false positive rate     p \u2248 (1 - e^(-k*n\/m))^k\noptimal hash count      k = (m\/n) * ln 2\nbits for a target p     m = -n * ln(p) \/ (ln 2)^2<\/code><\/pre>\n<table>\n<thead>\n<tr>\n<th>Target p<\/th>\n<th>Bits per URL<\/th>\n<th>Memory for n = 10^9<\/th>\n<th>k<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>1%<\/td>\n<td>4.605 \/ 0.4805 \u2248 9.59<\/td>\n<td>\u2248 1.2 GB<\/td>\n<td>7<\/td>\n<\/tr>\n<tr>\n<td>0.1%<\/td>\n<td>6.908 \/ 0.4805 \u2248 14.38<\/td>\n<td>\u2248 1.8 GB<\/td>\n<td>10<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>An exact set of 64-bit URL hashes needs 8 bytes \u00d7 10^9 = 8 GB before index overhead, and the birthday bound puts expected colliding pairs at about <em>n<\/em>\u00b2 \/ 2^65 \u2248 0.03. So the Bloom filter saves roughly 4\u20137\u00d7 memory for a known loss rate.<\/p>\n<p>Know which way the error goes: a Bloom filter sometimes says \"seen\" for a new URL, so about <em>p<\/em> of new URLs are <strong>silently skipped<\/strong>. Size it for the final <em>n<\/em>, because the rate climbs fast past it, and a plain Bloom filter cannot delete or resize.<\/p>\n<pre><code>import hashlib\nimport math\n\n\nclass BloomFilter:\n    def __init__(self, n, p):\n        self.m = math.ceil(-n * math.log(p) \/ math.log(2) ** 2)  # bits\n        self.k = max(1, round(self.m \/ n * math.log(2)))          # hash functions\n        self.bits = bytearray((self.m + 7) \/\/ 8)\n\n    def _positions(self, item):\n        d = hashlib.blake2b(item.encode(), digest_size=16).digest()\n        h1 = int.from_bytes(d[:8], \"little\")\n        h2 = int.from_bytes(d[8:], \"little\") | 1\n        return [(h1 + i * h2) % self.m for i in range(self.k)]  # double hashing\n\n    def add(self, item):\n        \"\"\"Set the item's bits; return True if it was (probably) present already.\"\"\"\n        present = True\n        for pos in self._positions(item):\n            byte, bit = divmod(pos, 8)\n            if not (self.bits[byte] &gt;&gt; bit) &amp; 1:\n                present = False\n                self.bits[byte] |= 1 &lt;&lt; bit\n        return present<\/code><\/pre>\n<p>In our check, probing 100,000 new URLs against <code>BloomFilter(100_000, 0.01)<\/code> filled with 100,000 others gave about 1.0% false positives, as predicted.<\/p>\n<h3>Exact stores<\/h3>\n<ul>\n<li><strong>RocksDB or another embedded key-value store:<\/strong> key = hash of the canonical URL, value = fetch state. It doubles as the recrawl database; a Bloom filter in RAM in front keeps most lookups off disk.<\/li>\n<li><strong>Redis:<\/strong> <a href=\"https:\/\/redis.io\/docs\/latest\/commands\/sadd\/\"><code>SADD<\/code><\/a> on a set of hashes is an atomic \"add and tell me if it was new\" across workers. Redis also has Bloom filters (<a href=\"https:\/\/redis.io\/docs\/latest\/commands\/bf.reserve\/\"><code>BF.RESERVE<\/code><\/a>, <code>BF.ADD<\/code>) and cuckoo filters (<code>CF.*<\/code>, which support deletion). It is all RAM, so the memory math still applies.<\/li>\n<li><strong>Distributed crawls:<\/strong> partition by host so each node owns its hosts' state.<\/li>\n<\/ul>\n<p>URL dedup is not content dedup: mirrors and parameter variants serve the same page under different URLs. Catch those after fetching; see <a href=\"https:\/\/globalcorpora.com\/blog\/minhash-deduplication-llm-training-data\/\">MinHash deduplication for LLM training data<\/a>.<\/p>\n<h2>Per-host queues and politeness, Mercator style<\/h2>\n<p>In the IR book's account of Mercator, a prioritizer assigns each URL to one of <em>F<\/em> front queues. Each back queue holds URLs from one host, and a heap holds the earliest time each back queue's host may be contacted again. A fetcher pops the heap, waits if needed, fetches one URL and pushes the host back with a new time. Two rules of thumb from the book: wait about ten times the last fetch duration before the next request to that host, and keep about three times as many back queues as crawler threads.<\/p>\n<p>The sketch below keeps that structure but uses a dictionary of per-host back queues instead of a fixed number of them. Put it in one file with the two blocks above.<\/p>\n<pre><code>import heapq\nimport random\nimport time\nfrom collections import deque\nfrom urllib.parse import urlsplit\n\n\nclass Frontier:\n    def __init__(self, priorities=3, min_delay=1.0, delay_factor=10.0,\n                 expected_urls=10_000_000, fp_rate=0.001):\n        self.front = [deque() for _ in range(priorities)]  # 0 = highest priority\n        self.back = {}         # host -&gt; deque of URLs\n        self.ready = []        # heap of (earliest_next_fetch, host)\n        self.not_before = {}   # host -&gt; earliest next fetch\n        self.seen = BloomFilter(expected_urls, fp_rate)\n        self.min_delay, self.delay_factor = min_delay, delay_factor\n\n    def add(self, url, priority=1):\n        if self.seen.add(canonicalize(url)):\n            return False\n        self.front[priority].append(url)\n        return True\n\n    def _refill(self, max_moves=1000):\n        weights = [2 ** (len(self.front) - i) for i in range(len(self.front))]\n        for _ in range(max_moves):\n            nonempty = [i for i, q in enumerate(self.front) if q]\n            if not nonempty:\n                return\n            i = random.choices(nonempty, [weights[j] for j in nonempty])[0]\n            url = self.front[i].popleft()\n            host = urlsplit(url).hostname\n            if host not in self.back:\n                self.back[host] = deque()\n                t = max(time.monotonic(), self.not_before.get(host, 0.0))\n                heapq.heappush(self.ready, (t, host))\n            self.back[host].append(url)\n\n    def next_url(self):\n        if not self.ready:\n            self._refill()\n        if not self.ready:\n            return None\n        t, host = heapq.heappop(self.ready)\n        return self.back[host].popleft(), host, max(0.0, t - time.monotonic())\n\n    def done(self, host, fetch_seconds):\n        t = time.monotonic() + max(self.min_delay, self.delay_factor * fetch_seconds)\n        self.not_before[host] = t\n        if self.back[host]:\n            heapq.heappush(self.ready, (t, host))\n        else:\n            del self.back[host]\n        self._refill(max_moves=100)\n\n\nf = Frontier(min_delay=0.2)\nfor u in [\"https:\/\/example.com\/\", \"https:\/\/EXAMPLE.com:443\/#top\",\n          \"https:\/\/example.org\/a?utm_source=news\", \"https:\/\/example.org\/a\",\n          \"https:\/\/example.org\/b\"]:\n    print(u, \"-&gt;\", \"queued\" if f.add(u) else \"duplicate\")\nwhile (item := f.next_url()) is not None:\n    url, host, wait = item\n    time.sleep(wait)\n    print(f\"fetch {url} (waited {wait:.2f}s)\")\n    f.done(host, fetch_seconds=0.01)<\/code><\/pre>\n<p>This queues three URLs, flags two variants as duplicates, and delays the second <code>example.org<\/code> fetch by 0.2 seconds. A host leaves the heap while its fetch is in flight, so there is one request per host at a time. Production versions keep most of each queue on disk (as the IR book notes) and add robots.txt, <code>Retry-After<\/code> and adaptive rates, covered in <a href=\"https:\/\/globalcorpora.com\/blog\/web-scraping-rate-limiting-retries-backoff\/\">web scraping rate limiting, retries and backoff<\/a>.<\/p>\n<h2>Priority signals and sitemaps<\/h2>\n<p>Useful priority signals: depth from seeds, the host's track record, URL patterns and sitemaps. The <a href=\"https:\/\/www.sitemaps.org\/protocol.html\">sitemaps.org protocol<\/a> specifies:<\/p>\n<ul>\n<li>At most <strong>50,000 URLs<\/strong> and <strong>50MB (52,428,800 bytes)<\/strong> uncompressed per sitemap; gzip is allowed. Index files may list up to 50,000 sitemaps with the same size limit, and carry a <code>lastmod<\/code> per sitemap so you can skip unchanged ones.<\/li>\n<li><code>&lt;loc&gt;<\/code> must be under 2,048 characters, and URLs must share the sitemap's host and protocol.<\/li>\n<li><code>&lt;changefreq&gt;<\/code> is \"a hint and not a command\"; <code>&lt;priority&gt;<\/code> (0.0\u20131.0, default 0.5) only ranks URLs within one site.<\/li>\n<li>Sites can advertise sitemaps with a <code>Sitemap:<\/code> line in robots.txt.<\/li>\n<\/ul>\n<p>Use sitemaps for discovery and <code>lastmod<\/code> to order recrawls within a host.<\/p>\n<h2>Recrawl scheduling with conditional requests<\/h2>\n<p>A long-running frontier also schedules revisits. Store the <code>ETag<\/code> and <code>Last-Modified<\/code> from each response and send them back as <code>If-None-Match<\/code> and <code>If-Modified-Since<\/code> (<a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9110.html\">RFC 9110<\/a> section 13). An unchanged page can then come back as <code>304 Not Modified<\/code> with no content. RFC 9110 says a recipient must ignore <code>If-Modified-Since<\/code> when <code>If-None-Match<\/code> is present, so the ETag wins. Many servers send neither validator, so also compare your own content hash.<\/p>\n<pre><code>import hashlib\n\nimport httpx\n\n\ndef revisit(client: httpx.Client, url, state):\n    \"\"\"Use httpx.Client(follow_redirects=True). state holds etag, last_modified, sha256, interval.\"\"\"\n    headers = {}\n    if state.get(\"etag\"):\n        headers[\"If-None-Match\"] = state[\"etag\"]\n    if state.get(\"last_modified\"):\n        headers[\"If-Modified-Since\"] = state[\"last_modified\"]\n    r = client.get(url, headers=headers)\n    interval = state.get(\"interval\", 86_400)\n    if r.status_code == 304:\n        changed = False\n    else:\n        r.raise_for_status()\n        digest = hashlib.sha256(r.content).hexdigest()\n        changed = digest != state.get(\"sha256\")\n        state.update(sha256=digest, etag=r.headers.get(\"ETag\"),\n                     last_modified=r.headers.get(\"Last-Modified\"))\n    # heuristic: revisit changing pages sooner, stable ones later, within 1 h to 30 days\n    state[\"interval\"] = max(3_600, min(30 * 86_400, interval \/ 2 if changed else interval * 1.5))\n    return changed<\/code><\/pre>\n<p>The halve-or-grow rule is a heuristic; the IR book warns that spam pages can change on every fetch, so weigh quality too.<\/p>\n<h2>Scrapy's built-in frontier<\/h2>\n<p>Scrapy's scheduler and dupefilter form a small URL frontier. From the <a href=\"https:\/\/docs.scrapy.org\/en\/latest\/topics\/settings.html\">settings reference<\/a> and <a href=\"https:\/\/docs.scrapy.org\/en\/latest\/topics\/jobs.html\">jobs docs<\/a> (Scrapy 2.20 at the time of writing):<\/p>\n<ul>\n<li><code>DUPEFILTER_CLASS<\/code> defaults to <code>scrapy.dupefilters.RFPDupeFilter<\/code>, which fingerprints the canonicalized URL, method and body and keeps fingerprints in memory. Scrapy 2.20 added <code>scrapy.dupefilters.DiskDupeFilter<\/code>, which keeps them in SQLite.<\/li>\n<li><code>JOBDIR<\/code> persists pending requests (<code>requests.queue\/<\/code>), fingerprints (<code>requests.seen<\/code>) and spider state, so a cleanly stopped crawl resumes. Use one directory per job and the same Scrapy version.<\/li>\n<li><code>SCHEDULER_PRIORITY_QUEUE<\/code> defaults to <code>DownloaderAwarePriorityQueue<\/code>, which the docs recommend for crawling many domains in parallel.<\/li>\n<li><code>JOBDIR_SYNC_EVERY<\/code> (2.20+) also saves state during the crawl; surviving a hard kill needs the SQLite disk queue types (2.19+).<\/li>\n<\/ul>\n<pre><code># settings.py\nJOBDIR = \"crawls\/frontier-1\"   # or: scrapy crawl myspider -s JOBDIR=crawls\/frontier-1\nDUPEFILTER_CLASS = \"scrapy.dupefilters.DiskDupeFilter\"               # 2.20+\nJOBDIR_SYNC_EVERY = 1000                                              # 2.20+, default 0\nSCHEDULER_DISK_QUEUE = \"scrapy.squeues.PickleLifoSQLiteQueue\"         # 2.19+\nSCHEDULER_START_DISK_QUEUE = \"scrapy.squeues.PickleFifoSQLiteQueue\"   # 2.19+\nDEPTH_LIMIT = 20               # default 0 = unlimited\nURLLENGTH_LIMIT = 2083         # the default<\/code><\/pre>\n<p>Requests with <code>dont_filter=True<\/code> bypass the dupefilter; unpicklable requests are lost on pause.<\/p>\n<h2>Crawler traps<\/h2>\n<p>Traps generate unique URLs without new content: endless calendars, faceted search, session IDs, relative-link bugs producing <code>\/a\/b\/a\/b\/a\/b\/<\/code>. Dedup cannot catch them. Defenses, cheapest first:<\/p>\n<ul>\n<li><strong>Hard limits:<\/strong> depth, URL length, query parameter count.<\/li>\n<li><strong>Per-host budgets:<\/strong> cap URLs queued per host, and raise the cap only for hosts that keep yielding distinct, useful documents.<\/li>\n<li><strong>Pattern counters:<\/strong> replace digits and IDs with placeholders (<code>\/calendar\/{n}\/{n}<\/code>) and cap URLs per pattern; cap distinct values per query key.<\/li>\n<li><strong>Repetition checks:<\/strong> reject paths whose segment sequence repeats.<\/li>\n<li><strong>Content feedback:<\/strong> demote patterns whose new URLs keep hashing to content you already have.<\/li>\n<\/ul>\n<p>For the full text pipeline, see <a href=\"https:\/\/globalcorpora.com\/blog\/common-crawl-llm-pretraining-data-pipeline\/\">the Common Crawl LLM pretraining data pipeline<\/a>.<\/p>\n<h2>Compliance<\/h2>\n<p>The frontier is the natural place for robots.txt checks and opt-out lists. Site terms of service often restrict automated access (YouTube's Terms of Service, for example, restrict automated access and downloading). Copyright and licensing apply to what you collect and redistribute, privacy law such as GDPR applies to personal data including faces and voices, and you should honor takedown and opt-out requests. Get legal review for your jurisdiction and use case.<\/p>\n<h2>If you would rather not run the egress yourself<\/h2>\n<p>If your frontier is in good shape and egress capacity is the bottleneck, Corpora offers <a href=\"https:\/\/globalcorpora.com\/#pricing\">dedicated 1\u201340 Gbps unmetered proxy capacity<\/a> (HTTP(S) and SOCKS5, works with custom crawlers) and a separate <a href=\"https:\/\/proxy.globalcorpora.com\/\">residential proxy<\/a> product. Keep politeness keyed on the target host either way.<\/p>\n<h2>FAQ<\/h2>\n<h3>What is a URL frontier?<\/h3>\n<p>The structure that holds URLs waiting to be fetched and decides their order, combining deduplication, priority and per-host politeness.<\/p>\n<h3>How big a Bloom filter do I need for a billion URLs?<\/h3>\n<p>About 1.2 GB at 1% false positives (7 hashes) and 1.8 GB at 0.1% (10), from <code>m = -n ln(p) \/ (ln 2)^2<\/code>.<\/p>\n<h3>Should I sort query parameters when normalizing URLs?<\/h3>\n<p>For dedup keys, usually yes; Scrapy's <code>canonicalize_url<\/code> does. No standard guarantees it, so fetch the original URL and add per-host exceptions when order matters.<\/p>\n<h3>Does Scrapy remember seen URLs between runs?<\/h3>\n<p>Only with <code>JOBDIR<\/code> set; rerunning with the same <code>JOBDIR<\/code> restores fingerprints and pending requests.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Design a URL frontier for a large crawler: RFC 3986 canonicalization, Bloom filter math, Mercator per-host queues, sitemaps, 304 recrawls and trap limits.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[114,49,118,95,117,116],"class_list":["post-42","post","type-post","status-publish","format-standard","hentry","category-guides","tag-bloom-filter","tag-deduplication","tag-politeness","tag-scrapy","tag-url-frontier","tag-web-crawling"],"_links":{"self":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/posts\/42","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/comments?post=42"}],"version-history":[{"count":0,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/posts\/42\/revisions"}],"wp:attachment":[{"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/media?parent=42"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/categories?post=42"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/globalcorpora.com\/blog\/wp-json\/wp\/v2\/tags?post=42"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}