Learn Scrapling - Advanced Spiders

#Introduction

The last six episodes built skills one at a time: in episode 11 you met spiders, in episode 12 proxy rotation, in episode 13 TLS impersonation, in episode 14 ethics, in episode 15 anti-bot defenses, and in episode 16 dynamic content. Now it's time to combine everything at scale with advanced spiders.

This episode covers three pillars of production crawling: pause and resume based on checkpoints so a crawl of millions of pages doesn't go to waste when the network drops, streaming with real-time stats so you can monitor and react, and scaling to thousands of pages with controlled concurrency and throttling. By the end, you'll assemble one complete spider that uses all of them at once.

#Why Long Crawls Need Checkpoints

A 100,000-page crawl takes hours. Along the way, many things can happen: the server restarts, the laptop runs out of battery, the network drops, or Ctrl+C is pressed one moment too late. Without state persistence, you'd have to start from zero — the first start_urls are already forgotten by the spider, while thousands of already-crawled pages would be reloaded.

The solution is the checkpoint: the spider periodically saves its crawl state to disk. When interrupted, run it again with the same directory, and the spider resumes exactly where it stopped — no replaying finished pages, no skipping unvisited ones.

#Pause & Resume with crawldir

Enabling it is as simple as passing crawldir to the spider constructor:

pause-resume.py

from scrapling.spiders import Spider, Response
 
class KatalogSpider(Spider):
    name = "katalog"
    start_urls = ["https://example.com/catalog"]
    concurrent_requests = 10
 
    async def parse(self, response: Response):
        for item in response.css(".product"):
            yield {
                "nama": item.css("h2::text").get(),
                "harga": item.css(".price::text").get(),
            }
        next_page = response.css(".next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page)
 
result = KatalogSpider(crawldir="crawl_data/katalog").start()

How it works behind the scenes:

• Pause: press Ctrl+C once. The spider waits for all in-flight requests to finish, saves the last checkpoint, then exits cleanly. Press Ctrl+C twice to force an immediate stop.
• Resume: run the same code with the same crawldir. The spider detects the checkpoint, restores the queue, and continues without calling start_requests again.
• Cleanup: when the crawl finishes normally, checkpoint files are cleaned up automatically.

Check the status through the result object:

check-status.py

if result.paused:
    print("Crawl dijeda. Jalankan lagi untuk melanjutkan.")
else:
    print("Crawl selesai.")
 
result.items.to_json("katalog.json", indent=True)

#State Persistence and Checkpoint Interval

What exactly gets saved? The two things that matter most for resuming a crawl precisely: the pending request queue and the set of URL fingerprints already seen. With those two pieces of data, the spider knows exactly what's unfinished and what must be skipped to avoid duplicates.

Checkpoints are saved periodically — every 5 minutes by default — and writes are atomic (write to a temp file, then rename), so a process that dies mid-write can't corrupt the file. The interval can be changed:

interval-checkpoint.py

spider = KatalogSpider(
    crawldir="crawl_data/katalog",
    interval=120.0,
)
result = spider.start()

There's also an on_start hook that receives a resuming flag — useful for notifying your team that the crawl is continuing from where it stopped:

hook-resume.py

from scrapling.spiders import Spider
 
class KatalogSpider(Spider):
    name = "katalog"
    start_urls = ["https://example.com/catalog"]
 
    async def on_start(self, resuming: bool = False):
        if resuming:
            self.logger.info("Melanjutkan dari checkpoint")
        else:
            self.logger.info("Mulai crawl baru")

This pattern lets a long crawl run over and over until it truly completes — each session stops and resumes without losing a single page.

#Streaming and Real-time Stats

A 100,000-page crawl isn't much fun if you only stare at a blank screen for hours. stream() changes that: items are passed through one by one as soon as they're scraped, and stats can be read anytime via spider.stats:

streaming.py

import anyio
 
from scrapling.spiders import Spider
 
class KatalogSpider(Spider):
    name = "katalog"
    start_urls = ["https://example.com/catalog"]
    concurrent_requests = 10
 
    async def parse(self, response):
        for item in response.css(".product"):
            yield {
                "nama": item.css("h2::text").get(),
                "harga": item.css(".price::text").get(),
            }
 
async def main():
    spider = KatalogSpider(crawldir="crawl_data/katalog")
    async for item in spider.stream():
        print(item)
        print(f"Item: {spider.stats.items_scraped} | Request: {spider.stats.requests_count}")
 
anyio.run(main)

spider.stats holds real-time metrics such as the number of items already scraped and requests made — enough to build a progress bar, a dashboard, or a notification trigger. stream() also combines with checkpoints, so the UI can be given pause/resume controls. To stop from within code, call spider.pause().

#Scaling to Thousands of Pages

Finally, speed. Three attributes control crawl aggressiveness:

scaling.py

from scrapling.spiders import Spider, Response
 
class KatalogSpider(Spider):
    name = "katalog"
    start_urls = ["https://example.com/catalog"]
    concurrent_requests = 32
    concurrent_requests_per_domain = 4
    download_delay = 0.5
    robots_txt_obey = True
 
    async def parse(self, response: Response):
        pass

• concurrent_requests — the total number of requests handled at the same time.
• concurrent_requests_per_domain — the per-domain limit, keeping a single site from being hammered too hard.
• download_delay — the pause between requests, combined with autothrottle that adapts on its own.

For very I/O-heavy crawls, an alternative event loop can boost throughput:

pip install uvloop

use-uvloop.py

result = KatalogSpider().start(use_uvloop=True)

Start with a small concurrency, watch the server's responses, then raise it gradually. 429 and 503 are signals to back off — not an excuse to blindly add more proxies.

#Closing

You now have a complete arsenal for production crawling: checkpoints with crawldir that make pause and resume seamless, safe state persistence with atomic writes, streaming via stream() with real-time statistics, and concurrency and throttle configuration for scaling to thousands of pages. Combined with proxy rotation, anti-bot defenses, and the ethics of previous episodes, your spider is ready for real-world scale.

The key takeaways:

• crawldir enables checkpoints: one Ctrl+C for a clean pause, run again to resume without repeating.
• Checkpoints save the pending queue and the set of seen URL fingerprints — with atomic writes.
• stream() yields items one at a time with real-time stats via spider.stats.
• Tune concurrent_requests, concurrent_requests_per_domain, and download_delay, then raise them gradually.
• use_uvloop=True can speed up I/O-heavy crawls if uvloop is installed.

In episode 18 we tackle performance from the numbers side: performance optimization. You'll benchmark Scrapling's parser against lxml, Parsel, Selectolax, and PyQuery, choose the right fetcher based on your workload, and cut memory and speed up execution with tree reuse and batching techniques.