Learn Scrapling - Core Concepts & Main Architecture

#Introduction

In episode 1 you got to know who created Scrapling, its version history, and the problems it solves. Now it's time to go inside: how Scrapling is structured, and how the workflow uses each of its modules.

This episode gives you an architecture map. You won't write much code — this is conceptual understanding that forms the basis of all the technical episodes that follow. After this episode, whenever we mention Fetcher, Adaptor, or Spider, you'll already know where it fits on the big map.

#The Four Fetcher Classes

The fetcher is the first gateway of every operation: it's responsible for retrieving HTML from the target. Scrapling provides several fetcher classes, each with a different role:

• Fetcher — synchronous HTTP requests based on curl_cffi. This is the fast one: built-in TLS impersonation, no browser, ideal for static pages.
• AsyncFetcher — the async version of Fetcher using aiohttp, for large-scale parallel requests.
• StealthyFetcher — controls a real browser via the Chrome DevTools Protocol to bypass heavy anti-bot defenses like Cloudflare and Turnstile.
• PlayWrightFetcher — wraps Playwright for dynamic content and JavaScript rendering.
• DynamicFetcher — a hybrid approach that tries the fast HTTP path first, then falls back to a browser if needed.

Choosing a fetcher is a strategic decision. The rough rule: start with Fetcher, move up to AsyncFetcher for scale, and reserve browser fetchers only for sites that genuinely need JavaScript or reject plain HTTP.

#The Adaptor Parser: The Heart of Selection

The result of every fetcher is an Adaptor object — a heavily optimized lxml-based parser. If the fetcher is the hand that retrieves, Adaptor is the brain that reads. This object holds the DOM tree from the raw HTML and provides selection methods:

• .css(selector) for CSS selectors.
• .xpath(selector) for XPath.
• .get() and .getall() to retrieve one or many results.
• Advanced methods like find_by_text and find_by_regex, which we cover in episode 5.

Because it's built on lxml, all selection operations run at C speed, not pure Python interpretation. This is the reason for the claimed 400-600 times faster than BeautifulSoup, which is based on a Python parser.

#The Spider Framework: From One Page to an Entire Site

When your target is more than a few pages, you don't write manual loops. Scrapling provides Spider — a framework for defining a crawl as a series of requests, callbacks, and rules. The concept is similar to Scrapy, but it still uses all of Scrapling's advantages: self-healing selectors, stealth fetchers, and async.

With Spider you can:

• Start from one or several seed URLs.
• Extract links on a page, then follow them automatically.
• Limit crawl depth and apply throttling.
• Persist state for pause/resume if a crawl is interrupted.

Spiders are covered in depth starting from episode 11, with the advanced version in episode 17.

#CLI and Shell: Working Without Writing Files

Not all work has to be written as a Python script. Scrapling has a CLI and an interactive shell for quick exploration:

scrapling fetch https://example.com --css-selector "h1"

The command above fetches the page and immediately extracts every element matching the selector — without writing a single line in an editor. Meanwhile, the interactive shell (scrapling[shell]) provides a REPL where you can type selectors one by one and see results instantly. It's a very convenient debugging tool before committing a selector to permanent code. Episode 21 covers the CLI thoroughly.

#MCP Server: Scrapling for AI Agents

The last modern feature in this architecture is the MCP server. MCP (Model Context Protocol) allows AI agents like Claude or OpenClaw to call Scrapling as a tool. That means an agent can perform real fetching and extraction while answering questions, rather than just guessing from its internal knowledge.

This MCP server opens up new work patterns: AI agents that research prices, analyze competitors, or gather research data directly. We'll cover the practical side in episode 19 along with AI integration.

#The Main Workflow

All the modules above come together in a standard workflow. Let's look at it as one process:

1. Fetch HTML — choose the fetcher according to your needs: Fetcher.get for static pages, StealthyFetcher for anti-bot defenses, PlayWrightFetcher for JS-heavy sites.
2. Parse & select elements — the raw HTML becomes an Adaptor, then pick nodes with .css() or .xpath().
3. Extract data — take text, attributes, or structure from the selected nodes and arrange them into a dictionary or list.
4. Advanced crawling (optional) — if the page has other relevant links, continue to the next fetch, automatically via Spider.

These four steps are a cycle you'll repeat throughout the series. If you understand it now, every following episode just deepens one step.

#Trying the Workflow with a Real Example

To tie the concepts together, let's run the full cycle on one simple page. This code already uses everything we discussed — choose a fetcher, parse with Adaptor, and extract:

workflow.py

from scrapling import Fetcher
 
page = Fetcher.get("https://example.com")
 
judul = page.css("h1").text
paragraf = page.css("p").getall()
 
print("Judul:", judul)
print("Jumlah paragraf:", len(paragraf))

Notice how straight this flow is: Fetcher.get returns an Adaptor (page), then .css("h1") and .css("p") select nodes, .text retrieves the text, and .getall() retrieves all results. No intermediate libraries.

#Choosing the Right Fetcher

Because there are many fetcher options, choosing one becomes a practical question. Consider these three dimensions:

• Static or dynamic? Server-rendered pages only need Fetcher. Pages that fill content via JavaScript need PlayWrightFetcher or DynamicFetcher.
• Anti-bot present? Websites using Cloudflare or Turnstile usually reject plain HTTP requests — that's where StealthyFetcher works. Regular sites just need impersonate on Fetcher.
• Need speed? For thousands of pages, AsyncFetcher enables parallel requests without consuming as many resources as a browser.

A common misconception: browser fetchers are always "better". In reality, browsers consume far more memory and CPU. A healthy rule is to try Fetcher first for every new target, and move up to a browser only when HTTP requests fail or the page doesn't render.

#External Dependencies Under the Hood

Scrapling's architecture is also interesting because it picks the best third-party library at each layer rather than writing everything from scratch:

• curl_cffi — the foundation of Fetcher and AsyncFetcher, providing TLS impersonation and HTTP/2.
• lxml — the C parser behind Adaptor that makes selection super fast.
• playwright — the engine behind PlayWrightFetcher.
• Chrome DevTools Protocol — the communication channel between StealthyFetcher and the browser.

This pattern explains why optional extras like scrapling[fetchers] exist: you only download heavy dependencies (browsers, Playwright) when you actually need them. You feel this structure from the outside when import scrapling stays lightweight for pure HTTP usage.

#Closing

Now you have a map of Scrapling's architecture: five fetchers with their own roles, the lxml-based Adaptor parser as the heart of selection, the spider framework for scale, the CLI and shell for working speed, and the MCP server for AI integration. All of it unites in the fetch, parse, extract, and crawl cycle.

The key takeaways:

• Fetchers are the first gateway: Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher, and DynamicFetcher.
• Adaptor is lxml-based — the result of all fetchers, with selection methods .css(), .xpath(), .get(), and .getall().
• The spider framework turns one request into an automatic, full-scale crawl.
• The CLI and interactive shell speed up exploration without writing files.
• The MCP server lets AI agents use Scrapling as a tool.
• The workflow: fetch HTML, parse, extract data, then crawl further.

In episode 3 we get into the first serious hands-on practice: Fetcher and HTTP requests — sessions, TLS impersonation, and how to handle status codes and redirects. See you there!