Learn Scrapling - Performance Optimization

#Introduction

In episode 17 you learned advanced spiders: pause/resume via checkpoints, state persistence, and real-time stats for crawling thousands of pages. The larger the crawl scale, the more one thing matters: performance. One second saved per page accumulates into hours of savings when you crawl 10,000 pages. Episode 18 focuses on performance optimization.

This episode's roadmap: first we benchmark Scrapling's parser against alternatives, then discuss strategies for choosing the right fetcher for your needs, and finally wrap up with memory and speed techniques — tree reuse, limiting payloads, and batch selection optimization.

#Why Scraper Performance Is Critical

A slow scraper isn't just an inconvenience — it's a cost. The longer one page takes to process, the more resources are consumed, the higher the risk of timeouts, and the lower the data throughput you can gather in a given window of time.

There are three most common sources of waste:

• Repeated parsing — the same HTML content is parsed over and over even though it could be reused.
• Overweight fetchers — using a browser for static pages that should be handled by plain HTTP.
• Wasteful selection — repeated queries, overly generic selectors, and unnecessary iteration.

The core optimization principle of this episode is simple: don't waste time on what you don't need. Choose the fastest parser for your job, choose the lightest fetcher that still gets past the page's protections, and don't re-parse what you already hold.

#Parser Benchmark: Scrapling vs the Competition

The parser is the layer that most often becomes the bottleneck. Scrapling's Adaptor is built on lxml (a C libxml2 wrapper), so its speed is close to raw lxml — and far ahead of pure-Python parsers. A quick comparison:

To validate on your own, you can use timeit like in the following example:

Simple benchmark: parse + select

import timeit
 
from scrapling import Fetcher
from lxml import html
 
url = "https://quotes.toscrape.com/"
raw = Fetcher().get(url).body
 
 
def scrapling_path():
    page = Fetcher().get(url)
    return len(page.css(".quote .text::text").getall())
 
 
def lxml_path():
    tree = html.fromstring(raw)
    return len(tree.xpath('//div[contains(@class,"quote")]//span[contains(@class,"text")]/text()'))
 
 
scrapling_time = timeit.timeit(scrapling_path, number=20)
lxml_time = timeit.timeit(lxml_path, number=20)
print(f"Scrapling: {scrapling_time:.3f}s")
print(f"lxml: {lxml_time:.3f}s")

A point worth remembering: Scrapling almost always loses narrowly to Selectolax in pure parsing, but wins decisively in productivity because css, xpath, find_by_text, find_by_regex, and auto_match live in one consistent API. For productivity, a microsecond-per-parse difference is often not worth hours of debugging.

#Choosing the Right Fetcher

After the parser, the fetcher is the biggest performance factor. The golden rule: use the lightest fetcher that still works. Only escalate to a heavier layer when the page truly demands it.

A correct escalation example: try Fetcher first. If the result is empty or blocked, move up to PlayWrightFetcher for JS rendering. If the protection is anti-bot, only then use StealthyFetcher. This order ensures most of your requests run on the fastest path.

Efficient fetcher escalation pattern

from scrapling import Fetcher
 
page = Fetcher().get("https://shop.example.com/products")
 
if not page.css(".product"):
    from scrapling import PlayWrightFetcher
 
    page = PlayWrightFetcher().get("https://shop.example.com/products")
 
print(len(page.css(".product")))

#Tree Reuse: Parse Once, Use Many Times

The second most common mistake after choosing the wrong fetcher is re-parsing. Every Fetcher().get(url) performs a network request plus a full parse. If you need five different selections from the same page, you don't need five requests.

One fetch, many selections

from scrapling import Fetcher
 
page = Fetcher().get("https://quotes.toscrape.com/")
 
for n in range(5):
    titles = page.css(".quote .text::text").getall()
    authors = page.css(".quote .author::text").getall()
    print(len(titles), len(authors))

The code above uses the same Adaptor object five times without extra requests. The lxml tree inside is already built, so every subsequent selection is just an in-memory traversal — dozens of times faster than a new request. Keep the Adaptor around as long as it's needed, and let the garbage collector handle it afterward.

#Limit the Payload

The larger the HTML processed, the longer the parsing and the more memory consumed. Payload-limiting strategies:

• Choose the right page — if the data is available via a JSON API, fetch the API with Fetcher instead of parsing HTML.
• Discard unneeded parts — script, style, and noscript tags are usually irrelevant for text extraction.
• Use specific selectors — selectors that narrow scope early (for example, entering body then the target element) reduce the nodes traversed.

For clean text, get_all_text can be configured to discard unneeded tags in one go:

Extract clean text, drop heavy tags

text = page.get_all_text(
    separator="\n",
    ignore_tags=("script", "style", "noscript"),
)

On giant pages (such as SPARQL documents or logs), also consider fetching only part of the content via a range request from the server, or splitting the process into small batches so peak memory stays under control.

#Batch Selection Optimization

The same pattern applies to selection: grab everything first, then process. Avoid calling selection from the root repeatedly inside a loop.

Before: root query inside the loop

for i in range(50):
    titles = page.css(".product .product-title::text").getall()

After: fetch once, process per item

cards = page.css(".product")
rows = [
    {
        "title": card.css(".product-title::text").get(),
        "price": card.css(".price::text").get(),
    }
    for card in cards
]

The second version calls the root selection only once, then chains within each card's scope — far cheaper because the scope is already narrowed. Also combine similar selectors with commas (for example h1, h2, h3) and use find_all with an attribute-dictionary filter for complex conditions. Tidy batch selection isn't just fast — the result is also a ready-to-process structured dataset.

#Closing

This episode closes Phase 5 — Advanced Topics, Scaling & Optimization — on the performance side. You now know how to benchmark parsers, choose the lightest sufficient fetcher, reuse trees, limit payloads, and run selection in batch. Together, these work in production to cut costs and speed up throughput.

The key takeaways:

• Scrapling's parser approaches raw lxml speed with far higher productivity; benchmark on your own machine before deciding.
• Choose the lightest sufficient fetcher — level up only when the page demands it.
• Don't re-parse — keep and reuse the Adaptor object.
• Limit the payload up front — pick the right page and discard irrelevant tags.
• Batch selection beats query loops — grab everything first, then process per item.

In episode 19, we turn to a new direction that leverages models: AI & LLM Extraction Integration — scrapling[ai], WebScrapingAI, the MCP server for AI agents, and LLM-led parsing and validation. See you there!