Learn Scrapling - Spiders: Crawling & Architecture

#Introduction

The last ten episodes taught you to play at the request and parsing level. Now it's time to chain everything together into a complete crawling engine. In this episode 11 we enter Scrapling's spider framework — a structure similar to Scrapy but still with one common language: the Adaptor, CSS/XPath selectors, and fetchers you already know.

You'll learn why you need a spider, define your first spider with start_urls and a parse callback, follow links between pages with response.follow, configure concurrency and throttling, use multi-session for different request routes, control crawl depth, then run the spider and collect its results. After this episode, a single start_urls can explode into thousands of automatically crawled pages.

#From Fetcher to Spider

Everything you've built so far is a single-request pattern: fetch one URL, parse, done. If you want to crawl 10 thousand pages — starting from an index page, following links to detail pages, then pagination — the manual pattern becomes a mess. You'd have to manage the URL queue, deduplication, concurrency, and pauses between requests yourself.

Spider is the structured answer. It defines where to start (start_urls), what to do when a page arrives (the parse callback), and which pages to visit next (yielding new links). The framework handles the rest: the request queue, automatic URL deduplication, scheduling, and item collection. This lets one small spider give birth to a large, still-tidy crawl.

#Your First Spider

The basic spider structure is very concise. Subclass Spider, give it a name, a list of start_urls, and define parse as an async function that receives a Response. Every dictionary yielded inside parse is treated as one scraped item.

first-spider.py

from scrapling.spiders import Spider, Response
 
class QuoteSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]
 
    async def parse(self, response: Response):
        for quote in response.css(".quote"):
            yield {
                "teks": quote.css(".text::text").get(),
                "penulis": quote.css(".author::text").get(),
            }
 
QuoteSpider().start()

start_urls holds the starting pages, parse executes for every response, and every yielded dictionary is collected as an item. Note that parse is an async def — Scrapling spiders run on top of an event loop, so all the asynchronous capabilities you learned in episode 6 are available here by default.

#Link Following with response.follow

The spider's power emerges when parse doesn't just collect data, but also follows links. When you find a URL that must be visited, yield a new request. The safest way is response.follow, which automatically resolves relative URLs and inherits the referer. Here's a spider that follows pagination until the last page:

spider-pagination.py

from scrapling.spiders import Spider, Response
 
class QuoteSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]
 
    async def parse(self, response: Response):
        for quote in response.css(".quote"):
            yield {
                "teks": quote.css(".text::text").get(),
                "penulis": quote.css(".author::text").get(),
            }
 
        next_page = response.css(".next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page)
 
result = QuoteSpider().start()
print("total:", len(result.items))

This flow is nice: page one produces items and requests page two, page two produces items and requests page three, and so on until .next is no longer found. Automatic deduplication ensures an already-visited URL isn't visited again. The same pattern can be used to follow links to detail pages: yield response.follow(href, callback=self.parse_detail) with a separate callback to parse the detail page.

#Concurrency & Throttling

Spiders execute requests in parallel, and you control how aggressive they are via class attributes. concurrent_requests limits how many requests may run at once globally. concurrent_requests_per_domain limits concurrency per domain — useful when crawling many domains at once without flooding each one.

spider-throttle.py

from scrapling.spiders import Spider, Response
 
class PolitesSpider(Spider):
    name = "polites"
    start_urls = ["https://example.com/"]
    concurrent_requests = 4
    concurrent_requests_per_domain = 2
    download_delay = 1.0
 
    async def parse(self, response: Response):
        yield {"judul": response.css("title::text").get("")}

download_delay{python} adds a fixed pause before every request — simple rate limiting that keeps the spider polite. For automatic adjustment, enable AutoThrottle with autothrottle_enabled=True; the spider measures how fast the server responds and then adjusts the pause per domain — speeding up on fast servers, braking on slow ones or those starting to block.

#Multi-session and Crawl Depth

Not every page needs the same treatment. Websites often have a combination of easy public pages and anti-bot-protected pages. Spiders support multi-session: register several sessions, then route each request to the appropriate session via the sid parameter.

spider-multi-session.py

from scrapling.spiders import Spider, Request, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession
 
class HybridSpider(Spider):
    name = "hybrid"
    start_urls = ["https://example.com/indeks"]
 
    def configure_sessions(self, manager):
        manager.add("fast", FetcherSession(impersonate="chrome"))
        manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)
 
    async def parse(self, response: Response):
        for link in response.css("a::attr(href)").getall():
            if "lindungi" in link:
                yield Request(link, sid="stealth", callback=self.parse_detail)
            else:
                yield Request(link, sid="fast", callback=self.parse_detail)
 
    async def parse_detail(self, response: Response):
        yield {"judul": response.css("h1::text").get()}

configure_sessions{python} registers the fast session (cheap HTTP) and stealth (anti-detection browser, created lazily so it doesn't waste resources). Every Request is routed via sid. This is a very common production pattern: light index pages via HTTP, protected pages via a stealth browser — both in one spider.

For crawl depth, Scrapling spiders deduplicate URLs automatically, but depth is still your responsibility. The common pattern is tracking depth via request meta, then stopping link following once it passes a limit.

spider-depth.py

from scrapling.spiders import Spider, Request, Response
 
class DepthSpider(Spider):
    name = "depth"
    start_urls = ["https://example.com/"]
 
    async def parse(self, response: Response):
        kedalaman = response.meta.get("kedalaman", 0)
        yield {"url": response.url, "kedalaman": kedalaman}
 
        if kedalaman < 3:
            for link in response.css("a::attr(href)").getall():
                yield Request(
                    link,
                    callback=self.parse,
                    meta={"kedalaman": kedalaman + 1},
                )

response.meta carries data from the request that produced this response. Every time the spider follows a link, the depth increases; when it reaches the limit (here 3), the spider stops expanding — preventing an uncontrolled crawl that traverses the whole internet.

#Running and Collecting Results

Running a spider is as easy as calling .start(). The result is a result object with an item collection, statistics, and export helpers. For long crawls, Spider supports checkpoints: provide a crawldir, and the spider saves progress periodically; press Ctrl+C to stop cleanly, then run it again to resume from the last point.

run-spider.py

from scrapling.spiders import Spider, Response
 
class QuoteSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]
 
    async def parse(self, response: Response):
        for quote in response.css(".quote"):
            yield {
                "teks": quote.css(".text::text").get(),
                "penulis": quote.css(".author::text").get(),
            }
        next_page = response.css(".next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page)
 
result = QuoteSpider(crawldir="./crawl_data").start()
result.items.to_json("quotes.json")

result.items.to_json("quotes.json"){python} serializes all items into a JSON file — exactly what you learned to do manually in episode 10, but now in one line. Combine it with crawldir and your spider becomes a pipeline that can be stopped, resumed, and exported at any time. This is the foundation of the production crawl architecture you'll deepen in the next phases.

#Closing

Episode 11 closes the basic Scrapling phase by chaining everything into a spider. You've learned the spider structure with start_urls and the parse callback, following links and pagination via response.follow, controlling concurrency and throttling with concurrent_requests, download_delay, and AutoThrottle, using multi-session to route HTTP vs stealth requests, managing crawl depth with response.meta, and running the spider and exporting results with .to_json. A single request has become a full-scale crawl.

The key takeaways:

• A spider = start_urls + an async parse callback; every yielded dictionary is an item.
• response.follow follows links and handles relative URLs automatically.
• concurrent_requests, concurrent_requests_per_domain, and download_delay control politeness.
• AutoThrottle adjusts the pause per domain adaptively.
• Multi-session (sid) and response.meta for depth build a controlled hybrid crawl.

In episode 12 we cover defense in hostile environments: proxy rotation and anti-blocking — using HTTP and residential proxies, automatic rotation, and managing headers and fingerprints so requests are hard to block. See you there!