Learn Scrapling - AsyncFetcher & Basic Concurrency

#Introduction

In episode 5 you went deep into advanced selection methods: find_by_text to find elements by text content, find_by_regex for patterns, find_similar to find peer elements, and attribute- and structure-based filters. All of that still uses the synchronous Fetcher, which works one request at a time. Now it's time to change that: you'll learn to fetch many pages in parallel via AsyncFetcher.

This episode is the bridge between scraping a single page and large-scale crawling. You'll get to know asynchronous programming in Scrapling, run parallel requests with asyncio.gather, limit the number of concurrent requests with a semaphore, apply rate limiting and batch requests, then close it out with a simple retry pattern. After this episode, you have the tools to speed up your scraper many times over without overwhelming the target server.

#Why Scraping Needs Asynchronous Programming

HTTP requests are operations dominated by waiting — waiting for the handshake, waiting for the server to process, waiting for the body to be sent. With the synchronous Fetcher, the program stops entirely during that waiting. Fetching 10 pages sequentially means stacking 10 wait periods, even though 99 percent of that time is idle.

Asynchronous programming solves this problem with an event loop: one thread runs many tasks at once, and every time a request is waiting on a network response, the event loop switches to another task. Because network waiting doesn't use the CPU, a single thread is enough to kick off hundreds of requests at once. This is why async scrapers can be 10-50 times faster for workloads with many small requests.

#Getting to Know AsyncFetcher

AsyncFetcher is the async twin of Fetcher. Its API is nearly identical: AsyncFetcher.get(url) and AsyncFetcher.post(url), except both are awaitable and must be called with await inside an async function. The result is still a Response object — same as Adaptor with additional attributes like status, reason, cookies, headers, and history.

basic-async-fetch.py

import asyncio
from scrapling import AsyncFetcher
 
async def main():
    page = await AsyncFetcher.get("https://quotes.toscrape.com/")
    print("status:", page.status)
    print(page.css(".quote .text::text").getall())
 
asyncio.run(main())

Notice two things: the main function starts with async def, and AsyncFetcher.get is called with await. After the await, you can select exactly as usual — page.css(...) and page.xpath(...) still work. For older Scrapling versions, the async method was named fetch_async; in modern versions that name has been replaced with an awaitable get, so make sure to match the version you have installed.

#Parallel Fetching with asyncio.gather

The core of async speed is running many requests at once. The simplest way is to build a list of coroutines and wait for them together with asyncio.gather. Compare these two approaches for fetching 10 quote pages.

The sequential version stacks 10 network wait periods. The parallel version kicks off 10 requests at once, and its total time approaches the slowest single request, not the sum of all requests. On a target with 300 ms latency per page, the sequential version takes about 3 seconds, while the parallel version only needs 300-400 ms.

#Semaphores: Limit Concurrency to Avoid Getting Blocked

Jumping from 1 to 1000 parallel requests is tempting, but it often ends in a 429 Too Many Requests status or an IP ban. Anti-bot systems and load balancers read request spikes as suspicious behavior. The solution is bounded concurrency: limiting how many requests may run at the same time.

asyncio.Semaphore is the gate. A semaphore holds a number of "tickets"; every request must grab one before running and return it when finished. If the tickets run out, the next requests queue up.

semaphore.py

import asyncio
from scrapling import AsyncFetcher
 
semaphore = asyncio.Semaphore(5)
 
async def fetch_page(url):
    async with semaphore:
        return await AsyncFetcher.get(url)
 
async def main():
    urls = [f"https://quotes.toscrape.com/page/{n}/" for n in range(1, 101)]
    pages = await asyncio.gather(*(fetch_page(url) for url in urls))
    print("total halaman:", len(pages))
 
asyncio.run(main())

With Semaphore(5){python}, at most 5 requests run at once — 100 pages still finish far faster than the synchronous version, but the target server isn't flooded with 100 requests in one second. Start from a small value like 5-10, then increase gradually while monitoring responses. If 429s start appearing, lower the concurrency before thinking about adding delay.

#Rate Limiting & Batch Requests

A semaphore controls parallelism, but it doesn't regulate how quickly requests are produced. For sensitive targets, you need rate limiting — ensuring the number of requests per unit of time doesn't exceed a reasonable limit. The simplest pattern is inserting a short pause between requests.

rate-limit.py

import asyncio
import random
from scrapling import AsyncFetcher
 
async def fetch_page(url):
    page = await AsyncFetcher.get(url)
    await asyncio.sleep(random.uniform(0.5, 1.5))
    return page
 
async def main():
    urls = [f"https://quotes.toscrape.com/page/{n}/" for n in range(1, 21)]
    pages = await asyncio.gather(*(fetch_page(url) for url in urls))
    print(len(pages))
 
asyncio.run(main())

Random pauses between 0.5 and 1.5 seconds create a more "human" rhythm than a constant pause — highly periodic request patterns are easy to detect. For very large datasets, split the URL list into batches, process one batch with a semaphore, pause briefly, then continue to the next batch. This also gives you the chance to save progress between batches.

#Error Handling & Simple Retry

Networks are never perfect: timeouts, dropped connections, or 5xx responses can appear at any time. In parallel scraping, one small error will make asyncio.gather fail entirely and throw away all the results already collected. Two fixes need to be applied. First, use return_exceptions=True so asyncio.gather returns exceptions as values instead of raising them. Second, wrap every request with a retry function that tries several times before giving up.

retry.py

import asyncio
from scrapling import AsyncFetcher
 
async def fetch_with_retry(url, retries=3):
    for attempt in range(retries):
        try:
            return await AsyncFetcher.get(url)
        except Exception as exc:
            if attempt == retries - 1:
                return exc
            await asyncio.sleep(2 ** attempt)
 
async def main():
    urls = [f"https://quotes.toscrape.com/page/{n}/" for n in range(1, 11)]
    results = await asyncio.gather(
        *(fetch_with_retry(url) for url in urls),
        return_exceptions=True,
    )
    ok = [p for p in results if isinstance(p, object) and getattr(p, "status", None)]
    print("berhasil:", len(ok))
 
asyncio.run(main())

Retrying with exponential pauses — 1 second, 2 seconds, 4 seconds — gives the server time to recover and avoids successive hits. Also note the pattern above: failed results are stored as exceptions so one failed page doesn't destroy the whole batch. In later episodes you'll encounter this pattern again in a more mature form, like AutoThrottle in the spider framework.

#Closing

Episode 6 equips you with the foundation of speed: AsyncFetcher with await AsyncFetcher.get(url), parallel fetching via asyncio.gather, concurrency control with asyncio.Semaphore, rate limiting through random pauses and batch processing, and exponential retry with return_exceptions=True. From now on your scraper can handle hundreds of pages in the time it used to take for dozens.

The key takeaways:

• AsyncFetcher provides awaitable get and post that return the same kind of Response object as Fetcher.
• asyncio.gather kicks off many requests at once, but don't forget to wrap it with a semaphore.
• Bounded concurrency is safer than unlimited concurrency — start with 5-10 concurrent requests.
• Rate limiting with random pauses makes the request rhythm look more natural.
• Use return_exceptions=True and exponential retry so one failure doesn't cancel the whole batch.

In episode 7 we enter the browser world: PlayWrightFetcher for dynamic content — rendering JavaScript and SPAs, working with headless mode, and strategies for choosing real versus headless browsers for light-to-medium protection. See you there!