Learn Scrapling - PlayWrightFetcher (Dynamic Content)

#Introduction

In episode 6 you learned to speed up fetching with AsyncFetcher and asyncio — all of that applies to static pages whose HTML is already complete from the first download. But many modern websites aren't like that: content is built by JavaScript in the browser, sometimes after waiting a few seconds, sometimes after scrolling or clicking. Plain HTTP only sees an empty skeleton. For these cases, you need a real browser, and in this episode 7 we'll use PlayWrightFetcher.

Throughout this episode you will: learn when a page must be rendered by a browser, understand PlayWrightFetcher and its newer name DynamicFetcher, use headless mode and wait for network idle, integrate Playwright selectors via wait_selector, compare real versus headless browsers for light-to-medium protection, then optimize fetching with sessions and disable_resources. Get your Chromium ready because from now on we're really running a browser.

#When a Page Must Be Rendered by a Browser

Many websites use JavaScript frameworks like React, Vue, or Svelte. The page sent by the server is only empty HTML with placeholders; the real data is loaded via API requests and then rendered in the browser. If you use Fetcher.get on such a page, all you get is the skeleton without content — selectors like .product-card will return an empty list.

Signs that a page needs rendering: content appears after the page loads or after interaction, text disappears when JavaScript is disabled, XHR requests appear in DevTools when you scroll, or the page contains words like "loading". If you've opened DevTools and seen a JavaScript app initializing, that's a signal to use a browser fetcher. For SPAs where navigation between pages happens without a full reload, a browser fetcher is almost mandatory.

#Getting to Know PlayWrightFetcher

PlayWrightFetcher is Scrapling's fetcher that controls the Chromium browser via Playwright. It opens the page, waits for JavaScript to finish, then returns the result as a Response object — the same kind of object you normally select with .css and .xpath.

basic-playwright-fetch.py

from scrapling import PlayWrightFetcher
 
page = PlayWrightFetcher.fetch(
    "https://example.com/dashboard",
    headless=True,
    network_idle=True,
)
print(page.status)
print(page.css(".widget-value::text").getall())

Before using any browser fetcher, make sure its dependencies are installed. Scrapling downloads the browser binaries via the command:

pip install "scrapling[fetchers]"
scrapling install

scrapling install{bash} downloads the required browsers complete with their dependencies. Without this step, calling PlayWrightFetcher will fail because the browser isn't found.

#Rendering JavaScript and Network Idle

JavaScript-loaded content needs time before it's ready to be retrieved. Waiting with an arbitrary time.sleep is fragile — too short gives empty results, too long wastes time. PlayWrightFetcher provides two tools for waiting intelligently.

The network_idle=True option makes the fetcher wait until network activity stops for a while — an indication that the page has finished loading resources and background requests. For content that loads slowly or with uncertain timing, use wait_selector to wait for a specific element that signals the content is present.

network-idle.py

from scrapling import PlayWrightFetcher
 
page = PlayWrightFetcher.fetch(
    "https://example.com/feed",
    headless=True,
    network_idle=True,
    wait_selector=".article-item",
    wait_selector_state="attached",
)
judul = page.css("h2.article-title::text").getall()

The wait_selector{python} argument accepts a CSS selector, and wait_selector_state{python} determines the condition to wait for — attached means the element appears in the DOM, visible means it's actually visible on screen. The combination of network_idle plus wait_selector is the most reliable pattern for pages whose content appears gradually.

#Integrating Playwright Selectors

Scrapling doesn't lock you out of the Playwright API; quite the opposite. Every browser fetcher stores a Playwright page object that you can access directly for work requiring full automation — clicking, scrolling, filling forms, or arbitrary JavaScript execution.

playwright-automation.py

from scrapling import PlayWrightFetcher
 
fetcher = PlayWrightFetcher()
page = fetcher.fetch(
    "https://example.com/katalog",
    headless=True,
    network_idle=True,
)
 
page_page = fetcher.page
page_page.mouse.wheel(delta_x=0, delta_y=3000)
page_page.click("button.load-more")
page_page.wait_for_selector(".product-card")
 
hasil = page_page.content()

After fetching finishes, fetcher.page{python} gives access to the native Playwright page. You can scroll to trigger lazy loading, click a "load more" button, then re-read the content. The .content() result can be saved and parsed with Adaptor as usual. This is how you handle infinite scroll and load-more buttons — patterns that are impossible with an HTTP fetcher.

#Headless vs Real Browser: Basic Stealth Mode

By default, PlayWrightFetcher runs in headless=True mode — a browser without a window, lightweight and fast. But headless browsers are easily recognized by anti-bot scripts: no window, a suspicious user agent, and several inconsistent JavaScript properties. For websites with light-to-medium protection — for example, pages using simple headless detection — running a real browser is often enough.

real-browser.py

from scrapling import PlayWrightFetcher
 
page = PlayWrightFetcher.fetch(
    "https://example.com/proteksi-ringan",
    headless=False,
    network_idle=True,
)
print(page.status)

headless=False{python} brings up a real browser window — just like opening Chrome manually. Protection based on headless detection will treat you as a regular user. Important note: for hard protection like Cloudflare or Turnstile, PlayWrightFetcher isn't enough — StealthyFetcher in episode 8 is the weapon. Use a real browser only for light-to-medium protection; the windowless (headless) browser remains the primary choice for speed.

#Sessions and Fetching Optimization

Opening and closing a browser for every request is expensive. If you're going to fetch many pages from the same domain, use a session so the browser stays alive and cookies and login state persist. For SPAs, a session also keeps the app from re-rendering from scratch every time.

dynamic-session.py

from scrapling.fetchers import DynamicSession
 
with DynamicSession(headless=True, disable_resources=True) as session:
    for page_url in ["https://example.com/hal/1", "https://example.com/hal/2"]:
        page = session.fetch(page_url, network_idle=True)
        print(page.css("h1::text").get())
 
page = session.fetch("https://example.com/hal/3", load_dom=False)

The disable_resources=True option blocks unnecessary images, fonts, and media — dramatically speeding up loading because the browser doesn't waste bandwidth on visual assets. The load_dom=False{python} argument tells the browser to fetch content without waiting for the DOM to fully complete, useful when you only need the initial HTML that already carries the data. The combination of sessions, disable_resources, and load_dom is the recipe for lean, fast browser fetching.

#Closing

Episode 7 puts a browser in your hands: PlayWrightFetcher for rendering JavaScript and SPAs, headless=True for speed, network_idle and wait_selector for waiting on dynamic content, direct access to the Playwright page object for clicking and scrolling, a real browser (headless=False) for light-to-medium protection, and DynamicSession with disable_resources for efficient fetching.

The key takeaways:

• SPA pages and JavaScript content must be rendered by a browser; an HTTP Fetcher only sees the HTML skeleton.
• PlayWrightFetcher (now DynamicFetcher) waits for JavaScript to finish and returns a standard Response object.
• Use network_idle and wait_selector instead of sleep to wait for dynamic content.
• Access fetcher.page for full Playwright automation: scroll, click, and JavaScript execution.
• Light-to-medium protection can be bypassed with a real browser (headless=False); hard protection needs StealthyFetcher.

In episode 8 we cover the ultimate weapon for heavily protected sites: StealthyFetcher — browser automation via the Chrome DevTools Protocol that can bypass Cloudflare and Turnstile out of the box. See you there!