Learn Scrapling - Dynamic Content & SPA Handling

#Introduction

In episode 15 you conquered anti-bot defenses with StealthyFetcher and solve_cloudflare. Now we face a subtler enemy: websites that don't show data without JavaScript. Single Page Applications (SPAs), lazy loading, and infinite scroll are the modern web standard — and the nightmare of scrapers still thinking "fetch HTML, then parse".

This episode covers how to handle dynamic content: when you truly need a browser, waiting strategies so elements are ready before extraction, techniques for infinite scroll and lazy loading, capturing XHR data that is often cleaner than the DOM, and the hybrid fetching pattern for choosing the most efficient fetcher per page.

#When a Page Needs a Browser

Not every website needs a browser. Static pages whose data already lives in the raw HTML only waste time and resources when rendered with a browser. Signs a page needs JavaScript:

• Data only appears after an additional API request (fetch or XHR).
• Elements load when scrolled into view (lazy loading) — images, product lists, comments.
• The app is an SPA — all content is drawn by JavaScript, and the raw HTML is nearly empty.
• Data lives in a JSON blob like __NEXT_DATA__ that only fills in after rendering.

The quick test: Fetcher.get and then check whether the target selector exists. If it doesn't, yet a human browser sees the data, you need a browser-based fetcher.

#Waiting Strategy: Don't Guess the Timing

The most common mistake on dynamic pages is guessing pauses: "wait 5 seconds, it'll definitely appear". Guessing is fragile — a fast page makes you wait needlessly, a slow page makes you extract empty data. DynamicFetcher gives you explicit waiting strategies:

wait-strategy.py

from scrapling.fetchers import DynamicFetcher
 
page = DynamicFetcher.fetch(
    "https://example.com/spa",
    headless=True,
    network_idle=True,
    wait_selector=".product-list",
    wait_selector_state="visible",
)
produk = page.css(".product h2::text").getall()

• network_idle=True waits for the network to go quiet — no pending requests left. Ideal for SPAs that load data via API.
• wait_selector waits for an element matching a selector to actually appear; wait_selector_state sets the desired condition — attached, visible, or hidden.
• load_dom (enabled by default) ensures all JavaScript files finish loading before wait_selector is checked.

The order: the page loads, network_idle is satisfied (if enabled), then wait_selector is satisfied (if enabled) — only then is the response returned. No magic numbers, just clear conditions.

#Infinite Scroll and Lazy Loading with page_action

page_action is a function that receives the Playwright Page object and runs right before wait_selector is checked. This is where you can scroll, click a "Load More" button, or wait for a specific element — whatever the page requires:

infinite-scroll.py

from playwright.sync_api import Page
 
from scrapling.fetchers import DynamicFetcher
 
def muat_semua(page: Page):
    for _ in range(5):
        page.mouse.wheel(0, 3000)
        page.wait_for_timeout(1500)
 
page = DynamicFetcher.fetch(
    "https://example.com/daftar-tak-habis",
    page_action=muat_semua,
)
item = page.css(".item::text").getall()

The same pattern applies to a "Load More" button: click the button, wait for the new container to appear, repeat. For the async version, write page_action as an async function and use DynamicFetcher.async_fetch.

If the page doesn't need all browser interactions, disable unimportant resources with disable_resources=True — blocking images, fonts, and media can speed up fetches by roughly 25 percent. Saves time and your proxy quota.

#Capturing XHR Data from SPAs

SPAs often load data from an internal API (XHR/fetch). Instead of waiting for the DOM to render, capture the API responses directly — usually JSON, which is far cleaner and more complete than the rendered result:

capture-xhr.py

from scrapling.fetchers import DynamicSession
 
with DynamicSession(
    headless=True,
    capture_xhr=r"https://api\.example\.com/.*",
) as session:
    page = session.fetch("https://example.com/spa")
    for req in page.captured_xhr:
        print(req.url, req.response.status)

With capture_xhr, Scrapling collects XHR requests matching the URL pattern while the page loads. Data from these APIs is often more structured than parsed DOM output — and more resilient to layout changes, because APIs change far less often than UIs.

Keep in mind: don't extract data from internal APIs of sites that forbid it, or for locked areas. Getting JSON "more easily" doesn't make it more legal — the ethics from episode 14 still apply.

#Hybrid Fetching: The Right Fetcher for Each Page

The rule of efficiency: fetch fast when you can, use the browser only when you must. The HTTP Fetcher is tens of times faster and more resource-efficient than a browser. The best strategy is to map each page type's needs, then route each request to the matching fetcher.

Inside a spider, this is as simple as defining two sessions and choosing the sid per request:

hybrid-spider.py

from scrapling.fetchers import AsyncDynamicSession, FetcherSession
from scrapling.spiders import Request, Spider, Response
 
class HybridSpider(Spider):
    name = "hybrid"
    start_urls = ["https://example.com/"]
 
    def configure_sessions(self, manager):
        manager.add("fast", FetcherSession(impersonate="chrome"))
        manager.add("browser", AsyncDynamicSession(headless=True), lazy=True)
 
    async def parse(self, response: Response):
        for link in response.css("a.statistik::attr(href)").getall():
            yield Request(link, sid="fast", callback=self.parse_item)
 
        for link in response.css("a.spa::attr(href)").getall():
            yield Request(link, sid="browser", callback=self.parse_item)

Lightweight statistic pages go through FetcherSession; heavy SPA pages go through AsyncDynamicSession, which only starts when first used (lazy=True). The server sees a regular visitor, your resources stay lean, and the crawl finishes faster.

#Closing

You can now handle the modern web without headaches: you know when a page truly needs a browser, you use network_idle and wait_selector instead of guessing timings, you scroll and click via page_action, you capture XHR data that's cleaner than the DOM, and you pick the right fetcher per page with hybrid fetching. This is the balance between data completeness and efficiency.

The key takeaways:

• Test with Fetcher first — only move to the browser if the target selector is truly empty in the raw HTML.
• Use network_idle and wait_selector with explicit states, not guessed pauses.
• page_action gives you full Playwright power for scrolling and clicking on endless pages.
• capture_xhr grabs JSON from an SPA's internal API — more stable than DOM rendering output.
• Hybrid fetching: Fetcher for what it can do, the browser for what it must, routed via sid in the spider.

In episode 17 all your skills come together at scale: advanced spiders. You'll learn checkpoints and pause/resume so long crawls don't go to waste when interrupted, streaming results with real-time stats, and strategies for scaling to thousands of pages with controlled concurrency.