Learn Scrapling - Proxy Rotation & Anti-Blocking

#Introduction

In episode 11 you built your first spider: defined start_urls, wrote the parse callback, followed links between pages, configured concurrency, and used multi-session to route requests to different fetchers. All of that works smoothly — until the website starts answering with a 403 or 429 status.

This episode covers two main weapons for surviving as crawl scale grows: proxy rotation and anti-blocking. You'll learn how to choose proxy types, rotate them automatically with ProxyRotator, handle blocked requests, and use impersonate and stealthy_headers to make your requests as close as possible to a real browser.

#Why Requests Get Blocked

Before blaming proxies, understand why the server rejects your requests. Blocking is usually layered and has three main sources:

• IP reputation. Hundreds of requests per minute from one IP follow an inhuman pattern. The data centers bots use are also blacklisted by IP reputation providers.
• Rate limiting. The server signals via a 429 (Too Many Requests) or 503 status with a retry header, because you've exceeded the per-second quota.
• Fingerprinting. The server suspects requests whose TLS fingerprint, header order, or HTTP/2 settings don't resemble a real browser.

Proxies solve the IP and rate limit problems: with many IPs, the pressure is spread out. Fingerprinting is handled by impersonate, which you've known since episode 3. The two are complementary, not replacements.

#Choosing Proxy Types

Not all proxies are equal. The type determines speed, price, and how closely your IP resembles a regular user:

• Datacenter proxies. IPs from cloud providers. Fast and cheap, but their reputation is the easiest for anti-bot systems to recognize.
• Residential proxies. IPs from home-user ISPs. Highly trusted by servers, more expensive. Suitable for targets with strict protection.
• Mobile proxies. IPs from mobile networks. Best reputation, but slowest and most expensive.
• ISP proxies. Static IPs whose ownership is registered to an ISP, but hosted in a data center. A balanced middle ground.

The choice is a business strategy: for prototypes and public pages, datacenter is enough. For targets with strict Cloudflare, start with residential.

#Using a Simple Proxy with Fetcher

The fastest way to test a proxy is via the proxy parameter on Fetcher — without any spider structure at all:

simple-proxy.py

from scrapling import Fetcher
 
page = Fetcher.get(
    "https://httpbin.org/ip",
    proxy="http://user:pass@proxy.example:8080",
)
print(page.text)

The format is scheme://credentials@host:port. One proxy is used for all requests in that session. This approach is good for testing, but fragile at scale: if that IP gets banned, the whole crawl stops with it. That's where automatic rotation comes in.

#ProxyRotator: Automatic Rotation in Sessions and Spiders

ProxyRotator manages a proxy list and automatically picks the next proxy for every request. The default is cyclic rotation — taking turns in order and wrapping back to the start. It's thread-safe, so it's safe to share across parallel requests:

rotator-session.py

from scrapling import FetcherSession, ProxyRotator
 
rotator = ProxyRotator([
    "http://proxy1.example:8080",
    "http://user:pass@proxy2.example:8080",
    "http://proxy3.example:8080",
])
 
with FetcherSession(proxy_rotator=rotator, impersonate="chrome") as session:
    page1 = session.get("https://example.com/page1")
    page2 = session.get("https://example.com/page2")
    page3 = session.get("https://example.com/page3")

Note that proxy_rotator must not be used together with a static proxy in the same session — pick one. The proxy in use can be tracked via page1.meta["proxy"]. Inside a spider, the rotator is attached via configure_sessions:

rotator-spider.py

from scrapling import ProxyRotator
from scrapling.fetchers import FetcherSession
from scrapling.spiders import Spider, Response
 
class KatalogSpider(Spider):
    name = "katalog"
    start_urls = ["https://example.com/products"]
 
    def configure_sessions(self, manager):
        rotator = ProxyRotator([
            "http://proxy1.example:8080",
            "http://user:pass@proxy2.example:8080",
        ])
        manager.add("default", FetcherSession(proxy_rotator=rotator))
 
    async def parse(self, response: Response):
        for product in response.css(".product"):
            yield {
                "nama": product.css("h2::text").get(),
                "harga": product.css(".price::text").get(),
            }

With this pattern, every request automatically uses the next IP without manual code. For browser-based sessions (like StealthySession), Scrapling opens one browser context per proxy — because a browser can't switch proxies per tab.

#Custom Rotation Strategies

Cyclic rotation is easy to predict when the pool is small. ProxyRotator accepts a custom strategy function with the signature strategy(proxies, current_index) that returns a pair of the chosen proxy and the next index. Examples of random and weighted strategies:

custom-strategy.py

import random
from scrapling import ProxyRotator
 
def random_strategy(proxies, current_index):
    idx = random.randint(0, len(proxies) - 1)
    return proxies[idx], idx
 
def weighted_strategy(proxies, current_index):
    weights = [60] + [40 // (len(proxies) - 1)] * (len(proxies) - 1)
    proxy = random.choices(proxies, weights=weights, k=1)[0]
    return proxy, current_index
 
rotator = ProxyRotator(
    ["http://proxy1.example:8080", "http://proxy2.example:8080", "http://proxy3.example:8080"],
    strategy=random_strategy,
)

A random strategy spreads the load in no particular order, while a weighted strategy fits when some proxies are faster or cheaper. Also combine it with geo logic: proxies in specific locations for content that only appears in certain regions.

#Per-Request Proxy Overrides and Block Handling

Sometimes a single request needs a special proxy — for example, a page that only appears for Indonesian IPs. Overriding the rotator is as simple as the proxy argument on that request, e.g. response.follow(url, callback=self.parse, proxy="http://geo-proxy.example:8080").

To handle blocking, spiders have a built-in detection system. Statuses like 403, 429, and 503 are treated as blocks, and the request is retried with the next proxy from the rotator — up to max_blocked_retries times (default 3). The detection logic can be extended by overriding is_blocked:

block-detection.py

from scrapling.spiders import Spider, Response
 
class KatalogSpider(Spider):
    name = "katalog"
    start_urls = ["https://example.com/products"]
    max_blocked_retries = 5
 
    async def is_blocked(self, response: Response) -> bool:
        if response.status in {403, 429, 503}:
            return True
        body = response.body.decode("utf-8", errors="ignore")
        return "access denied" in body.lower()
 
    async def retry_blocked_request(self, request, response):
        request.sid = "stealth"
        return request

This combination produces a resilient cycle: a request fails because the IP hit a limit, is_blocked recognizes it, the rotator supplies a new proxy, and the retry runs with a fresh IP.

#Spoofing Headers and TLS Fingerprints

A proxy changes your IP, but the server can still guess you're a bot from your request pattern. The impersonate parameter disguises the TLS fingerprint, header order, and HTTP/2 settings to a specific browser profile. If you pass a list, Scrapling picks one at random per request — a pattern that's hard to recognize:

spoof-fingerprint.py

from scrapling import Fetcher
 
page = Fetcher.get(
    "https://example.com",
    impersonate=["chrome", "firefox", "safari"],
    stealthy_headers=True,
)

stealthy_headers=True makes request headers regenerated to be consistent with the impersonated browser version — including the header order, not just the User-Agent. In episode 13 you'll dissect how these TLS fingerprints work at the protocol level.

#Closing

You now have an arsenal to survive unfriendly territory: you understand proxy types and when to use them, rotate IPs automatically with ProxyRotator plus custom strategies, track the proxy in use via response metadata, override the proxy per request, and patch the spider's built-in block detection system. Combined with header and TLS fingerprint spoofing, your requests now look like visits from a regular user, not a bot.

The key takeaways:

• Blocks come from IP reputation, rate limits, and fingerprints — proxies only solve the first two.
• ProxyRotator provides thread-safe cyclic rotation for all session types.
• Rotation strategies are plain functions, so random, weighted, or geo-aware are all possible.
• Leverage is_blocked, max_blocked_retries, and retry_blocked_request for automatic retries with fresh proxies.
• Always combine proxies with impersonate and stealthy_headers so requests aren't recognized by header and TLS patterns.

In episode 13 we go down to the protocol level: networking and TLS impersonation. You'll dissect how JA3/JA4 fingerprints are formed, why HTTP/2 settings also leak identity, and how curl_cffi behind Scrapling makes your requests indistinguishable from a real browser.