Learn Scrapling - Adaptor & Basic Parsing
Going deep into the Adaptor object: selecting elements with CSS and XPath, understanding the difference between get and getall, leveraging chaining, and extracting text and attributes. Explaining lxml-based DOM manipulation — tree traversal and element iteration — to build structured data.
#Introduction
In episode 3 you fetched web pages with Fetcher, sessions, and impersonation. Now it's time for the part you'll use most throughout your scraping career: reading HTML content with Adaptor.
This episode teaches basic selection — CSS selectors and XPath, the difference between .get() and .getall(), chaining between elements — then text and attribute extraction, all the way to lxml tree manipulation for element iteration.
#Getting to Know the Adaptor Object
Adaptor is the DOM tree representation of raw HTML. In episode 3, the page returned by Fetcher.get is an Adaptor. It can also be created directly from an HTML string — useful when the HTML is already saved or obtained from another source:
from scrapling import Adaptor
html = """
<html>
<body>
<h1 class="judul">Halo Dunia</h1>
<p class="konten">Ini paragraf pertama.</p>
<p class="konten">Ini paragraf kedua.</p>
</body>
</html>
"""
page = Adaptor(html)Now page is ready to be selected without any request at all. This object also supports operations like .get() for elements, .text, and attributes — which we'll use throughout this episode.
#Selecting with CSS: The css Method
The most common way to select elements is .css(selector) with a CSS selector. This method returns a new Adaptor object representing the selection results:
page = Adaptor(html)
judul = page.css("h1.judul")
paragraf = page.css("p.konten")The selector h1.judul selects h1 tags with the class judul, and p.konten selects all paragraphs with the konten class. The result is an Adaptor object that can be chained — for example, taking the text directly from the selection, or selecting descendants of that result.
#Selecting with XPath: The xpath Method
XPath gives finer control — pointing at elements by position, text, or relationship. The .xpath() method accepts an XPath expression:
judul = page.xpath("//h1[@class='judul']")
paragraf_kedua = page.xpath("(//p)[2]")//h1[@class='judul'] finds an h1 with the class judul anywhere in the document, while (//p)[2] grabs the second paragraph. When a CSS selector can't express a certain condition — for example, "elements whose text starts with a certain word" — XPath is usually the answer.
#get vs getall: One or All
This is the most important difference in this episode. .get() takes the first element of a selection, while .getall() returns all results as a list:
page = Adaptor(html)
pertama = page.css("p").get()
semua = page.css("p").getall()
print(pertama.text)
print(len(semua))pertama.text shows the text of the first paragraph, and len(semua) gives the total number of paragraphs. The rule of thumb: use .get() when you know you only want one result (a title, a price, a single number), and .getall() when you want an entire collection (a product list, all links).
#Chaining Selections
Selection doesn't have to be one level deep. You can select elements from a previous selection's results — that's chaining. This uses the base parameter to mark the search starting point:
page = Adaptor(html)
judul = page.css("body").css("h1", base=0)The second call searches for h1 within the body result — the result is more specific and stays concise. Chaining is very useful on complex pages with many containers: select the container first, then select elements inside it.
#Extracting Text and Attributes
Once elements are selected, extract the data. An element's text is retrieved via the .text property (which normalizes whitespace), and attributes via the attribute parameter on .get():
html = """
<a href="/artikel/1" data-category="python">Belajar Parsing</a>
"""
page = Adaptor(html)
link = page.css("a").get()
print(link.text)
print(link.attrib["href"])The element's text is automatically cleaned of extra whitespace, and attrib gives a dictionary with all attributes — href and data-category here. For specific selectors, there's also the ::text pseudo-selector that directly targets a node's text:
teks = page.css("a::text").get()
print(teks)This is a shortcut for "take the text from this element" without having to navigate the object first.
#DOM Manipulation and Element Iteration
Because Adaptor is built on lxml, its tree can be traversed programmatically. There are two patterns commonly used. First, iterating over selection results to build data:
produk_list = page.css("div.produk").getall()
data = []
for item in produk_list:
nama = item.css("h2").text
harga = item.css("span.harga").text
data.append({"nama": nama, "harga": harga})Every item from .getall() is a standalone Adaptor, so chaining inside the loop runs smoothly. This pattern becomes the foundation of structured datasets in episode 10. Second, you can navigate direct relationships between nodes:
parent = item.xpath("..").get()
siblings = item.xpath("../div")The .. expression moves one level up to the parent element, and ../div grabs all sibling divs. This ability to navigate relationships is what keeps XPath relevant alongside CSS.
Warning
Note that a selection result can be empty — a typo'd selector, or a changed DOM structure. Always check whether a .get() result is None before using its attributes, so your script doesn't crash mid-crawl.
#Building Structured Data
Now let's combine everything you've learned into a single function that produces structured data:
from scrapling import Adaptor
def parse_artikel(html):
page = Adaptor(html)
return {
"judul": page.css("h1").text,
"link": page.css("a.baca-selengkapnya").get().attrib["href"],
"paragraf": page.css("article p").getall(),
}This function takes raw HTML, selects the title with CSS, grabs the link via attributes, and collects all paragraphs. Notice the combination of .text, attrib, and .getall() in one function — that's the core of parsing you'll use over and over.
#Closing
You're now skilled at reading HTML with Adaptor: building an object from a string, selecting elements with .css() and .xpath(), understanding when to use .get() or .getall(), leveraging chaining, extracting text and attributes, and iterating over the lxml tree to build structured data.
The key takeaways:
Adaptorcan be created from an HTML string or a fetcher result; it's lxml-based, so it's very fast..css()and.xpath()return anAdaptorthat can be chained..get()takes the first result,.getall()takes all results as a list..textnormalizes text;attribprovides all element attributes.- Iterating elements with
.getall()and chaining per item is the basic pattern for building datasets.
In episode 5 we go up a level: advanced selection methods — find_by_text, find_by_regex, searching for similar elements, and filtering based on attribute conditions. See you there!