Learn Scrapling - Advanced Selection Methods

#Introduction

In episode 4 you mastered basic selection: .css(), .xpath(), .get(), .getall(), and text and attribute extraction. But there are times when regular selectors aren't enough — for example, elements without a clear class, or elements that must be found by their text content. That's where advanced selection comes in.

This episode covers find_by_text, find_by_regex, searching for similar elements, and filter-based selection — four weapons that make you precise in unfriendly DOM conditions.

#Why Advanced Selection Is Needed

CSS and XPath excel when the HTML structure is tidy. But the real world is often different: classes randomly generated by frameworks, elements without clear attributes, or text being the only stable clue. A concrete example: a page has a group of buttons that all share the same class and can only be told apart by their text.

In situations like this, selecting "the button whose text is Login" makes more sense than guessing a structural selector. Scrapling provides text-based search methods designed specifically for these cases.

#Finding Elements by Text: find_by_text

find_by_text finds elements by their text content. This method is available on Adaptor and can be called directly on the tree or on a selection result:

find-by-text.py

from scrapling import Adaptor
 
html = """
<button class="btn">Login</button>
<button class="btn">Daftar</button>
"""
 
page = Adaptor(html)
 
login = page.find_by_text("Login")
print(login.text)

The result is the element containing the exact text "Login". For more flexible control, there are parameters you can combine:

find-by-text-options.py

page = Adaptor(html)
 
login = page.find_by_text("Login", case_sensitive=False)
dua = page.find_by_text("Log", first_only=False)

case_sensitive controls case matching, and first_only controls whether to return just the first element or all matches. When the text is duplicated and all elements are relevant, set first_only=False and iterate over the results.

#Finding Elements with Text Patterns: find_by_regex

When the text varies but follows a pattern — numbers, prices, dates — find_by_regex is the answer. It accepts a regex pattern and finds elements whose text matches the pattern:

find-by-regex.py

import re
from scrapling import Adaptor
 
html = """
<div>Diskon 10%</div>
<div>Diskon 25%</div>
<div>Lainnya</div>
"""
 
page = Adaptor(html)
 
diskon = page.find_by_regex(re.compile(r"Diskon \d+%"))
print([item.text for item in diskon])

The pattern Diskon \d+% matches every div containing a discount percentage. This is very useful for extracting elements whose text pattern is consistent even when their structure varies — for example, prices in various formats, or order numbers.

#Finding Similar Elements

Another classic problem: you've found one element that's exactly right, but you need other elements with a similar pattern on the same page. Scrapling provides a search for similar elements — elements whose structure resembles a reference element. The approach searches based on a structural template:

find-similar.py

from scrapling import Adaptor
 
html = """
<div class="card"><h2>Produk A</h2><span class="harga">Rp 100.000</span></div>
<div class="card"><h2>Produk B</h2><span class="harga">Rp 250.000</span></div>
<div class="banner">Promosi</div>
"""
 
page = Adaptor(html)
card_pertama = page.css("div.card").get()
mirip = page.find_similar(card_pertama)
print(len(mirip))

find_similar finds elements whose structure resembles the first card — here it will find the second card, and the banner with a different structure will be skipped. This technique is reliable when product or article lists are inconsistent in structure between items.

#Filtering by Attribute Conditions

The .filter() method filters selection results based on attribute conditions. You can filter from an existing selection:

attribute-filter.py

from scrapling import Adaptor
 
html = """
<a href="/eksternal" target="_blank">Link 1</a>
<a href="/internal/1">Link 2</a>
<a href="/internal/2" data-utama="true">Link 3</a>
"""
 
page = Adaptor(html)
link_internal = page.css("a").filter(
    href="/internal/", regex=True
)
utama = page.css("a").filter(
    {"data-utama": "true"}
)

The first example filters links whose href contains the pattern /internal/ via regex mode. The second example filters by an attribute-value pair: only elements with data-utama="true" pass. Filters can use a single string, a regex, or a multi-condition dictionary.

#Filtering by Structure

Filters aren't only about attributes — they can also be structural conditions, such as selecting elements that do or don't contain a certain descendant. This mimics the power of XPath without writing long expressions:

structure-filter.py

from scrapling import Adaptor
 
html = """
<div class="produk"><img src="a.jpg"><h3>Nama A</h3></div>
<div class="produk"><h3>Nama B</h3></div>
"""
 
page = Adaptor(html)
punya_gambar = page.css("div.produk").filter("img")
print(len(punya_gambar))

Passing the selector string "img" to filter means "only keep elements that contain an img". The result is that only the first product passes. This pattern removes incomplete items — for example, products without a photo or without a price.

#Combining Techniques in One Flow

All the methods above aren't silos; they're most powerful when combined. A real example: finding all discount prices inside similar elements:

combined.py

import re
from scrapling import Adaptor
 
page = Adaptor(html)
produk_pertama = page.css("div.card").get()
kartu_mirip = page.find_similar(produk_pertama)
 
for kartu in kartu_mirip:
    harga = kartu.find_by_regex(re.compile(r"Rp [\d.]+"))
    if harga:
        print("Harga:", harga[0].text)

The flow: find the first card, search for everything similar, then within each card search for price-formatted text. With this combination, selection keeps working even when classes change — as long as the text patterns and basic structure remain stable.

#Closing

You now have a set of advanced selection tools: find_by_text for matching text, find_by_regex for patterns, find_similar for elements resembling a reference, and .filter() for filtering by attributes or structure. All four build on the CSS/XPath foundation from episode 4.

The key takeaways:

• find_by_text finds elements by text content, with control over case and result count.
• find_by_regex matches patterned text like prices, numbers, or dates.
• find_similar finds elements whose structure resembles a reference element.
• .filter() filters results by attributes, regex, or the presence of a descendant element.
• Combine all the techniques so selection survives when the DOM structure changes.

In episode 6 we move into the speed dimension: AsyncFetcher and concurrency — parallel fetching, rate limiting, semaphores, and batch requests. See you there!