Learn Scrapling - Web Scraping Security & Ethics

#Introduction

The last two episodes felt like an arms race: in episode 12 you rotated proxies to avoid being blocked, then in episode 13 disguised your TLS fingerprint so requests look like a real browser. Those skills are important — but they must be balanced by one thing: responsibility.

This episode covers the side that can't be taught through code: legality and ethics. You'll learn to read robots.txt, understand where Terms of Service stand, set up polite rate limiting, protect the data you extract — including PII issues — and secure your scraper credentials so they don't leak. Not just to stay safe from lawsuits, but so you become a scraper people need, not one people are hostile to.

#Legal and Ethics: Two Different Things

Before writing a single line of code, distinguish these two concepts:

• Legal is what the law allows. It depends on jurisdiction. In Indonesia, collecting personal data is regulated by the Personal Data Protection Law (UU PDP), which requires a valid legal basis to process personal data. Public data isn't automatically free to use.
• Ethics is what should be done. A site may not legally forbid scraping, but it's still unethical if you hammer it with thousands of requests per second until its server collapses.

A simple rule of thumb: scraping data that is genuinely public and needed, at a human-like pace, without burdening the server, and without touching personal data — is almost always legally and ethically safe. Mass scraping of personal data, paid content, or platforms that clearly forbid it in their ToS is a red zone.

#Robots.txt: Machine-Readable Etiquette

robots.txt is a file at a site's root that tells crawlers which pages may and may not be accessed. It's not law — some sites write it loosely — but respecting it is a good habit honored by search engines and professional scrapers:

curl -s https://example.com/robots.txt

Scrapling respects this in spiders via the robots_txt_obey attribute:

polite-spider.py

from scrapling.spiders import Spider
 
class BeritaSpider(Spider):
    name = "berita"
    start_urls = ["https://example.com/berita"]
    robots_txt_obey = True

When enabled, the spider reads the Disallow rules, honors Crawl-delay, and processes Request-rate with per-domain caching. Also understand what Disallow means — it can mean "don't index" rather than "don't access". If in doubt, ask the site owner directly.

#Polite Rate Limiting

Crawl speed is the face of your ethics. Firing 100 requests per second at a small site is the fastest way to burn your own IP's reputation — and to break someone's server. Good practice: limit concurrency per domain and insert pauses between requests:

polite-rate-limit.py

from scrapling.spiders import Spider
 
class BeritaSpider(Spider):
    name = "berita"
    start_urls = ["https://example.com/berita"]
    robots_txt_obey = True
    download_delay = 2.0
    concurrent_requests_per_domain = 1
    max_blocked_retries = 3

download_delay adds a pause before each request, and concurrent_requests_per_domain limits parallel requests to one domain. Measure a reasonable limit by trial: start slow, increase gradually, and watch whether the server responds with 429. If the server signals, lower it again — that's a healthy form of negotiation.

#Security of Extracted Data

The data you collect is stored responsibility. Start with sanitization: text from web pages often carries HTML tags, double whitespace, or strange characters. Clean it before saving:

sanitize.py

import re
 
def bersihkan(raw: str) -> str:
    tanpa_tag = re.sub(r"<[^>]+>", "", raw)
    return " ".join(tanpa_tag.split())

Beyond cleanliness, there's a heavier ethical layer: PII (Personally Identifiable Information). Phone numbers, emails, addresses, or people's identities are sensitive data. Don't collect it without a clear legal basis, and never store it without need. If the target page does contain PII, apply masking or skip that field entirely:

avoid-pii.py

EMAIL_PATTERN = r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"
 
def buang_pii(teks: str) -> str:
    return re.sub(EMAIL_PATTERN, "[redacted]", teks)

The golden rule: collect as little data as possible for a clear need. Data you never stored is data that will never leak.

#Protecting Scraper Credentials

Your scraper now carries valuables: proxy credentials, API keys, and possibly session cookies. Hardcoding all of them in code is the fastest way to establish yourself as both a leak victim and a leak perpetrator. Store secrets in environment variables and read them at runtime:

read-env.py

import os
 
from scrapling import Fetcher
 
PROXY_URL = os.environ["SCRAPLING_PROXY"]
 
page = Fetcher.get("https://example.com", proxy=PROXY_URL)

Also prepare an env template in the repo without real values, so team members know which variables are needed:

SCRAPLING_PROXY=http://user:pass@proxy.example:8080
SCRAPLING_API_KEY=ganti-dengan-nilai-asli

A few other habits worth keeping: don't commit .env files, rotate credentials periodically, and never put credentials in logs. The server receiving your requests also doesn't need to know your proxy password — credentials leaked in logs are also leaked to anyone who reads those logs.

#Closing

This episode doesn't give you great new code, but it gives you a compass. You know how to read robots.txt and respect it via robots_txt_obey, set up polite rate limiting with download_delay and concurrency limits, clean and protect extracted data — including steering clear of PII — and secure credentials via environment variables. The technical skills from episodes 12 and 13 become dangerous without this ethical side.

The key takeaways:

• Legal and ethics differ: law follows jurisdiction, ethics follows the impact on servers and data owners.
• robots.txt is machine-readable etiquette; respect it via robots_txt_obey=True.
• Polite rate limiting starts with download_delay and per-domain concurrency limits, then increases gradually.
• Sanitize output and avoid storing PII without a legal basis; masking is a better last resort.
• Store credentials in environment variables, not in code, logs, or repos.

In episode 15 we enter the territory many have been waiting for: CAPTCHA and anti-bot bypass. You'll dissect Cloudflare, Turnstile, and JS challenge mechanisms, use StealthyFetcher to break through protection, and determine when — and whether — bypassing that protection is an ethical decision.