Learn Scrapling - CAPTCHA & Anti-Bot Bypass

#Introduction

In episode 14 you paused to build an ethical compass: reading robots.txt, setting up polite rate limiting, protecting data, and securing credentials. Now we return to the technical territory — the most anticipated one: CAPTCHA and anti-bot bypass.

This episode dissects the most common anti-bot mechanisms — Cloudflare, Turnstile, and JS challenges — then shows how to deal with them using StealthyFetcher, discusses alternative captcha services, and sets up rate limit and block management as your crawl grows large. Remember, everything you learn here is only legitimate for sites you have the right to access.

#Dissecting Anti-Bot Mechanisms

Modern anti-bot systems are layered. Understand your enemy before attacking:

• Cloudflare Challenge. The page requests a cf_clearance cookie after passing the challenge. It checks JavaScript, verifies the browser, then sets a valid cookie for a few minutes.
• Turnstile. Cloudflare's newer version — a non-intrusive widget that assesses a visitor's "humanness" from behavior, without requiring typing a code. It evaluates browser fingerprint, visit history, and interaction patterns.
• JS Challenge. The site forces JavaScript to run to prove a real browser. If your request doesn't run JS (for example, purely HTTP), the server never shares its content.
• Interstitial. An intermediate page that appears before the real content, showing a challenge or waiting a few seconds.

The key point: all these mechanisms live in the browser layer. A bare HTTP request will never get through because the server needs proof that JavaScript and a real browser are actually running. That's where browser-based fetchers come in.

#When Bypassing Is Ethical

Before writing any code, answer three questions: do you have the right to access that data, is your speed human-like, and are you ready to stop when asked?

Bypassing is legitimate and ethical if: the site is public, you've read robots.txt and the ToS, you're not accessing areas locked behind paid accounts, and you respect rate limits. Bypassing is problematic if: you break through protection to scrape mass personal data, resell paid content, or keep crawling after being asked to stop.

Technical capability doesn't exempt you from ethics. solve_cloudflare in Scrapling is a legitimate tool for authorized access — not a master key for theft.

#StealthyFetcher: The Browser in Disguise

StealthyFetcher runs a modified Firefox browser — not just a regular Chromium — complete with fingerprint spoofing and various anti-detection mechanisms. The browser is launched on demand, challenges are solved automatically, and the result is returned as a regular Adaptor:

scrapling install

stealthy-fetch.py

from scrapling.fetchers import StealthyFetcher
 
page = StealthyFetcher.fetch(
    "https://nopecha.com/demo/cloudflare",
    headless=True,
    solve_cloudflare=True,
)
print(page.css("title::text").get())

The solve_cloudflare=True parameter enables automatic solving of Turnstile, interstitials, and other Cloudflare challenges. Because it solves real challenges in a real browser, your request gets through without having to manually break a CAPTCHA. Important note: this mode adds a few seconds to fetch time — enable it only when genuinely needed.

To keep a session across several pages, use the session version — the cf_clearance cookie is maintained so subsequent pages don't solve the challenge from scratch:

stealthy-session.py

from scrapling.fetchers import StealthySession
 
with StealthySession(
    headless=True,
    solve_cloudflare=True,
    network_idle=True,
) as session:
    page1 = session.fetch("https://example.com/proteksi/halaman-1")
    page2 = session.fetch("https://example.com/proteksi/halaman-2")

Note that browser fetchers use timeout in milliseconds (default 30000), unlike the HTTP Fetcher which uses seconds.

#Alternatives: Captcha Services and WebScrapingAI

When anti-bot defenses are too aggressive to solve yourself, third-party services exist. Two common categories:

• Captcha solving services. They solve CAPTCHAs manually or automatically and return valid tokens/cookies. Suitable for edge cases, but they add per-request cost and latency.
• WebScrapingAI and similar. APIs that handle the whole cycle: fetching, anti-bot bypass, even structured extraction. You send a URL, they return data. Suitable if you don't want to manage your own browser infrastructure.

Scrapling integrates with this ecosystem via the scrapling[ai] extra — which you'll dissect more deeply in episode 19. The general strategy: start with StealthyFetcher to solve it yourself; if it works, you save money. If it fails, fall back to an external service for the hard requests only.

#Rate Limit and Block Management at Scale

When the crawl grows, the challenge isn't one page but stable load distribution. Combine all the previous lessons: autothrottle for adaptive pauses, block detection, and retries. Scrapling spiders provide attributes for this:

anti-bot-spider.py

from scrapling.spiders import Spider, Response
 
class ProteksiSpider(Spider):
    name = "proteksi"
    start_urls = ["https://example.com/proteksi"]
    robots_txt_obey = True
    max_blocked_retries = 4
 
    async def is_blocked(self, response: Response) -> bool:
        if response.status in {403, 429, 503}:
            return True
        body = response.body.decode("utf-8", errors="ignore")
        return "challenge" in body.lower() and "cf-chl" in body.lower()

autothrottle automatically adjusts the pause between requests based on server responses — increasing the pause when the server starts slowing down, decreasing it when the server is calm. Combined with a smart is_blocked and the proxy rotation from episode 12, your spider becomes an adaptive system: slow when asked, fast when allowed.

#Closing

You now know the modern anti-bot landscape: Cloudflare challenges with cf_clearance, Turnstile that assesses behavior, and JS challenges that force a real browser. You know StealthyFetcher with solve_cloudflare is the main weapon — and know that technical capability doesn't replace ethical decisions. You also have an alternative roadmap: captcha services and WebScrapingAI for hard cases, plus autothrottle and block detection for large scale.

The key takeaways:

• Cloudflare, Turnstile, and JS challenges work in the browser layer — plain HTTP will never get through.
• solve_cloudflare=True on StealthyFetcher solves challenges automatically; enable it only when needed because it adds fetch time.
• Browser sessions maintain the challenge cookie, so subsequent pages don't repeat solving.
• Bypassing is only ethical for targets you have the right to access, at a human-like pace, and ready to stop.
• At scale, combine autothrottle, block detection, and proxy rotation so the load stays polite and stable.

In episode 16 we deal with content that doesn't appear without JavaScript: dynamic content and SPA handling. You'll handle lazy loading and infinite scroll, use waiting strategies so elements are truly ready before extraction, capture XHR data from SPAs, and decide when to use a fast fetcher or a browser.