Tag: datacenter proxies

  • Residential vs Datacenter Proxies vs ISP and Mobile for Scraping

    Residential vs datacenter proxies is the first choice most teams face when a crawler outgrows one server. ISP (static residential) and mobile proxies are the other two options. All four are billed differently, and which one fits depends on how many bytes you move, how long a session has to hold one IP, and who actually owns the address you exit from. This guide covers how each type works at the network level, how the billing models compare (with the arithmetic), and how to test a proxy against your own targets before you sign anything.

    TL;DR

    • Datacenter IPs are announced by hosting, cloud or vendor networks. They're fast, stable and cheap per IP, and the easiest type to identify as non-residential.
    • Residential IPs belong to consumer ISPs and exit through real subscriber connections. They're usually billed per GB, and throughput varies from peer to peer.
    • ISP / static residential IPs are announced by an ISP-type network but hosted on servers. They're stable and usually billed per IP.
    • Mobile IPs exit through cellular carriers, often behind carrier-grade NAT. They have the lowest throughput and are rarely suited to bulk data.
    • Cost: metered cost = GB × P, and flat cost = Gbps × F. Break-even is F / P GB per Gbps per month. At video volumes, unmetered capacity wins quickly.
    • Test before you buy: check latency, the success rate on your targets, exit-IP geolocation and ASN.

    How each type works: who announces the prefix

    Every public IP sits inside a prefix that an autonomous system (AS) announces over BGP. The origin ASN and its registered holder are the most reliable way to tell what you're really buying. RIPEstat's prefix-overview endpoint returns "whether and by whom" a prefix is announced.

    Datacenter

    The proxy runs on a server and exits from addresses belonging to the hosting company, cloud provider or vendor, and the same organisation usually announces the prefix. Throughput is limited by the port, so it's high and consistent. The ASN openly identifies a hosting network, though, and many sites apply stricter limits to hosting ranges. Shared datacenter IPs also carry whatever reputation earlier users gave them.

    Residential

    The vendor doesn't own these addresses. Traffic enters the vendor's gateway, which forwards it through a participating end-user device or router, so the exit IP belongs to a consumer ISP's AS. Exits appear and disappear, upstream bandwidth is whatever the household has, and latency includes someone's last mile. Country, region and session behaviour are usually selected with parameters in the proxy username, and the syntax differs between vendors.

    ISP (static residential)

    These are servers in a data centre that exit from address space registered to or announced by an ISP-type network. Some vendors have the ISP announce their prefix. Others lease address space and arrange for an ISP to originate it. Verify it yourself: an "ISP proxy" whose prefix is originated by a hosting ASN is a datacenter proxy. You get server-grade throughput and a stable IP, but fewer IPs per dollar, and a flagged IP stays flagged until it's replaced.

    Mobile

    Traffic exits through a modem or handset on a cellular network. Carriers commonly use carrier-grade NAT (CGN), so many subscribers share one public IPv4 address. RFC 6598, which reserved 100.64.0.0/10 for CGN, notes that "geo-location systems identify the location of the CGN server, not the end host." Rotation depends on the device getting a new address, and the radio link limits throughput. Use mobile IPs when you specifically need what carrier users see, not for bulk transfer.

    Type Who announces the prefix Common billing Throughput Sessions
    Datacenter Hosting / cloud / vendor ASN Per IP, per GB, or per Gbps (unmetered) High, consistent Static; rotating pools available
    Residential Consumer ISP ASN Per GB Variable per peer Rotating; sticky with a TTL
    ISP / static residential ISP-type ASN (verify) Per IP per month High Static
    Mobile Carrier ASN, often behind CGN Per device/port per period, or per GB Lowest Rotates on re-attach

    The billing column shows common contract structures, not a survey. This article quotes no market prices. Plug your own quotes into the formulas below.

    Rotating vs sticky sessions

    A rotating endpoint can give you a new exit IP per request or per TCP connection. A sticky session pins one exit for a time window, usually selected with a session ID in the username. Static datacenter and ISP IPs behave like a sticky session that never expires.

    • Rotation suits stateless requests, such as fetching many unrelated pages once each.
    • Stickiness is needed whenever a server ties state to your IP: cookies, signed URLs, multi-step flows and chunked downloads. The yt-dlp FAQ notes that video requests often "must come from the same IP address", which is why our yt-dlp proxy guide uses one sticky session per video.
    • Because of keep-alive, a "per-request" rotating proxy may really rotate per connection. Measure it.
    • Residential sticky sessions can end early when the peer goes offline. Make jobs resumable and requeue them on a fresh session (see rate limiting, retries and backoff).

    Billing models and the arithmetic

    • Per IP (monthly): common for static datacenter and ISP IPs. Bandwidth may be unlimited, capped or "fair use", so read the clause.
    • Per GB (metered): the norm for residential. The meter counts bytes, not outcomes, so failed requests, retries and TLS overhead usually count too.
    • Per Gbps (committed, unmetered): you reserve capacity and pay the same whatever you move through it.
    Metered:   cost_month = GB_month × P      (P = price per GB)
    Flat:      cost_month = Gbps × F          (F = price per Gbps per month)
    Break-even volume per Gbps:  GB = F / P
    
    Ceiling of 1 Gbps, 30-day month, 100% utilisation:
      1e9 bit/s × 2,592,000 s / 8 / 1e9 = 324,000 GB (324 TB)

    A worked example that needs no assumed market price. Take F = $2,000 per Gbps per month and 100 TB/month (100,000 GB):

    Flat:     1 Gbps × $2,000 = $2,000   (100 / 324 ≈ 31% utilisation)
    Metered:  100,000 GB × P
    Metered is cheaper only if 100,000 × P < 2,000  →  P < $0.02/GB

    Compare that threshold with any per-GB quote you have. At 1 PB/month you need about 3.1 Gbps sustained (provision around 5 Gbps), versus 1,000,000 × P metered. The full sizing tables are in bandwidth for large-scale video downloads. To turn media hours into GB, use the bitrate you actually measure (file size ÷ duration on a sample of real downloads):

    GB_per_hour = bitrate_Mbps × 3,600 / 8,000 = bitrate_Mbps × 0.45
    e.g. 5 Mbps average → 2.25 GB per hour

    When per-GB residential makes sense

    • Low byte volume: HTML, JSON APIs and metadata, with many requests of a few KB each.
    • Geographic breadth: many countries or cities, each with a small share of traffic.
    • Bursty or exploratory work: pilots, monthly refreshes, or a long tail of targets that fail from datacenter ranges, where reserved capacity would sit idle.

    When unmetered dedicated capacity makes sense

    • Bandwidth-heavy workloads: video, audio, large files and dataset mirrors, where the meter would dominate the bill.
    • Sustained volume: you can keep the pipe reasonably full, so you clear F / P easily.
    • Throughput-sensitive jobs: long downloads that need consistent speed and stable exits.

    A hybrid is common: residential for discovery and metadata, dedicated capacity for bulk bytes. When a server binds downloads to the requesting IP, though, an item's metadata fetch and its download must share an exit, so split traffic per item, not per request type.

    Ethical sourcing of residential IPs

    A residential exit is someone's home connection. Vendors typically recruit peers through apps or SDKs whose users agree to share bandwidth, sometimes for payment. How clear that consent is varies, and it belongs in your vendor-risk review. Ask:

    • Which apps or SDKs recruit peers, and can we see the consent screen end users see?
    • Is participation opt-in, easy to withdraw from, and compensated?
    • What limits protect peers, for example bandwidth caps or no use of metered mobile data without consent?
    • What customer KYC and abuse handling do you run?
    • Can you put this in writing for our compliance team?

    If a vendor can't answer clearly, treat that as the answer.

    HTTP(S) vs SOCKS5

    This is a protocol choice, independent of IP type.

    • HTTP proxy: for https:// targets the client sends CONNECT host:443 and the proxy opens a TCP tunnel (RFC 9110 §9.3.6). TLS stays end to end. Bad credentials return 407 Proxy Authentication Required. "HTTPS proxy" sometimes means TLS to the proxy itself, which not every client supports.
    • SOCKS5 (RFC 1928) is protocol-agnostic. It defines CONNECT, BIND and UDP ASSOCIATE, and accepts a domain name as the destination, so the proxy can resolve DNS (the socks5h:// scheme in curl and many libraries). Username/password authentication is method X'02'. Ask the vendor whether UDP is relayed.

    Choose HTTP if your toolchain includes HTTP-only downloaders such as aria2c (details in the yt-dlp proxy guide). Choose SOCKS5 for non-HTTP protocols or for DNS resolved at the exit.

    How to test a proxy

    Run tests from the region and machines you'll use in production, at a low request rate, against targets you're permitted to collect from.

    Latency breakdown with curl

    curl's --write-out variables split a request into phases. Through a proxy, time_connect is the TCP connect to the proxy, and time_appconnect adds the tunnel and the TLS handshake with the target.

    FMT='code=%{http_code} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n'
    for i in $(seq 1 20); do
      curl -s -o /dev/null --max-time 30 -x "http://USER:[email protected]:8080" -w "$FMT" "https://example.com/"
    done

    Success rate, exit IPs, ASN and country

    This script measures the success rate and latency on your own URLs, then samples exit IPs and looks each one up in RIPEstat (prefix-overview for origin ASN and holder, maxmind-geo-lite for country). For SOCKS5, install requests[socks] and use a socks5h:// URL.

    import statistics
    import time
    from collections import Counter
    
    import requests
    
    PROXY = "http://USER:[email protected]:8080"
    PROXIES = {"http": PROXY, "https": PROXY}
    TARGETS = ["https://example.com/"]  # the URLs you will actually collect
    N, PAUSE_S = 30, 1.0
    
    
    def ripestat(call, ip):
        r = requests.get(f"https://stat.ripe.net/data/{call}/data.json",
                         params={"resource": ip}, timeout=20)
        r.raise_for_status()
        return r.json()["data"]
    
    
    for url in TARGETS:
        statuses, ok_lat = Counter(), []
        for _ in range(N):
            t0 = time.perf_counter()
            try:
                status = requests.get(url, proxies=PROXIES, timeout=30).status_code
            except requests.RequestException as exc:
                status = type(exc).__name__
            statuses[status] += 1
            if status == 200:
                ok_lat.append(time.perf_counter() - t0)
            time.sleep(PAUSE_S)
        print(url, f"success={len(ok_lat)}/{N}", dict(statuses))
        if len(ok_lat) >= 2:
            p95 = statistics.quantiles(ok_lat, n=20)[18]
            print(f"  p50={statistics.median(ok_lat):.2f}s p95={p95:.2f}s")
    
    ips = set()
    for _ in range(10):
        ips.add(requests.get("https://api.ipify.org", proxies=PROXIES, timeout=30).text.strip())
        time.sleep(PAUSE_S)
    print(f"{len(ips)} distinct exit IPs in 10 requests")
    for ip in sorted(ips):
        asns = [f"AS{a['asn']} {a['holder']}" for a in ripestat("prefix-overview", ip).get("asns", [])]
        geo = ripestat("maxmind-geo-lite", ip).get("located_resources", [])
        countries = {loc["country"] for res in geo for loc in res.get("locations", [])}
        print(ip, asns or "not announced", sorted(countries))

    What to look for

    • Success rate per target. The failure type (403, 429, timeouts, resets) tells you whether the problem is reputation, rate or capacity.
    • p95 latency, not just the median. Size worker timeouts and concurrency for the tail.
    • ASN matches the label. "Residential" and "ISP" exits should originate from access-network ASNs. Read the holder names.
    • Geolocation consistency. Exits for country X should geolocate to X. Databases disagree, and CGN mobile IPs locate coarsely, so check more than one source when it matters.
    • Rotation behaviour. Count distinct IPs over a sticky session's promised lifetime, and for rotating endpoints with and without connection reuse.
    • Throughput for bulk work. Download a few real large objects and compute bytes per second per connection.

    Whatever the IP type, per-IP concurrency and request rate shape how a target treats you. For YouTube specifically, see fixing "Sign in to confirm you're not a bot".

    Compliance

    A proxy changes where requests come from, not what you're allowed to collect. Site terms still apply. YouTube's Terms of Service, for example, restrict automated access and downloading. For training data, copyright and licensing matter (filtering for Creative Commons content is one approach), as do privacy obligations for faces and voices under GDPR and similar laws, and honouring takedown and opt-out requests. Get legal review for your jurisdiction and use case.

    If you'd rather not build the proxy layer yourself

    For bandwidth-heavy collection, Corpora offers dedicated 1–40 Gbps unmetered proxy capacity (HTTP(S) and SOCKS5, compatible with yt-dlp and custom crawlers, rotating IP pools, multi-region) at $2,000 per Gbps per month, with volume discounts from 5 Gbps. See pricing. For lower-volume work that needs residential exits, there's a separate residential proxy product. Test either one the way you'd test any vendor.

    FAQ

    What is the difference between residential and datacenter proxies?

    Ownership and routing. Datacenter proxies exit from prefixes announced by hosting or cloud networks, on servers with high, stable bandwidth. Residential proxies exit through subscriber connections on consumer ISPs. They look like ordinary home users, but they're usually billed per GB and their speed varies.

    Are ISP proxies the same as residential proxies?

    No. ISP (static residential) proxies are data-centre servers using address space announced by an ISP-type network. They have datacenter speed and stability with an ISP ASN. Verify the origin ASN yourself, for example with RIPEstat.

    Is per-GB or per-Gbps pricing cheaper?

    Above F / P GB per month for each Gbps, flat pricing is cheaper. At $2,000 per Gbps, 100 TB/month on metered pricing only wins if P is below $0.02/GB.

    How do I check what kind of IP my proxy really uses?

    Fetch your exit IP through the proxy (for example from https://api.ipify.org), then look up the origin ASN and holder with RIPEstat's prefix-overview and check its location with a geolocation source. A hosting-company holder means a datacenter IP, whatever the product page says.