Learn LangChain - Document Loaders & Text Splitters

#Introduction

In episode 8 your agent could already act through tools. Now it's data's turn. RAG — retrieval-augmented generation — works because the model is given context from your own documents. But before a document can be retrieved, it has to be turned into a searchable form: a Document object neatly split into small pieces called chunks.

This episode covers the first two stages of the RAG pipeline. First, document loaders: reading PDFs, web/HTML, CSV, and JSON into Documents containing page_content and metadata. Second, text splitters: cutting documents into chunks with RecursiveCharacterTextSplitter, token-based splitters, and chunk size and overlap strategies for good retrieval quality.

#The Document Object: page_content and metadata

All loaders return Document objects with two main fields: page_content holding the text, and metadata holding contextual information like source, title, or page number. This metadata is what's later used for retrieval filtering.

Basic Document structure

from langchain_core.documents import Document
 
doc = Document(
    page_content="LangChain adalah framework untuk aplikasi LLM.",
    metadata={"sumber": "docs/langchain.md", "halaman": 1},
)
print(doc.page_content)
print(doc.metadata)

The mental model is simple: any loader — PDF, web, CSV, JSON — ultimately produces a list of Documents. The rest of the pipeline (split, embed, retrieve) doesn't care about the original format, because everything is normalized to the same structure.

#Loaders for PDF and Web

#PDF

PDF is the most common document format in enterprises. PyPDFLoader extracts text page by page and puts the page number in metadata.

Loading a PDF

from langchain_community.document_loaders import PyPDFLoader
 
loader = PyPDFLoader("laporan-tahunan.pdf")
dokumen = loader.load()
 
print(len(dokumen))          # number of pages
print(dokumen[0].metadata)   # metadata contains the page number

#Web / HTML

To turn a web page into clean text, WebBaseLoader fetches the HTML and extracts the article content while discarding navigation and sidebars.

Loading a web page

from langchain_community.document_loaders import WebBaseLoader
 
loader = WebBaseLoader("https://example.com/artikel-langchain")
dokumen = loader.load()
print(dokumen[0].page_content[:200])

Note that WebBaseLoader marks recently fetched pages so repeated requests don't re-download — useful when you're testing many times.

#Loaders for CSV and JSON

CSV and JSON are structured data formats commonly exported from databases or APIs.

Loading CSV and JSON

from langchain_community.document_loaders import CSVLoader, JSONLoader
 
csv_docs = CSVLoader("data-pengguna.csv").load()
print(csv_docs[0].page_content)
 
# JSONLoader needs a jq-style path to point at the text content
import json
with open("data.json") as f:
    data = json.load(f)
 
json_docs = JSONLoader(
    file_path="data.json",
    jq_schema=".items[]",
    text_content=False,
).load()

For CSV, one row becomes one Document. For JSON, jq_schema determines which part is taken as a document — for example each element of the items array. Before installing, make sure the loader you need is actually available in your version of langchain-community.

#RecursiveCharacterTextSplitter

The most versatile splitter and the default recommendation. It cuts text using a cascading list of separators — from paragraphs (newlines) down to sentences, then words — so pieces stay at sensible boundaries, not in the middle of a sentence.

RecursiveCharacterTextSplitter

from langchain_text_splitters import RecursiveCharacterTextSplitter
 
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", ".", " "],
)
chunks = splitter.split_documents(dokumen)
 
print(len(chunks))
print(chunks[0].page_content)

Two key parameters: chunk_size limits the length of each chunk, chunk_overlap makes each chunk partially overlap the previous one so context at the cut boundary isn't lost. splitter.split_documents(...) processes a list of Documents at once and passes metadata along to each chunk.

#Token-Based Splitters

Characters and tokens don't always align — one word can consume several tokens in a given model. If your context is measured in tokens (which is indeed how providers bill), TokenTextSplitter cuts based on token count, giving more precise control over cost and the context window.

TokenTextSplitter for token control

from langchain_text_splitters import TokenTextSplitter
 
token_splitter = TokenTextSplitter(
    chunk_size=200,
    chunk_overlap=20,
)
chunks_token = token_splitter.split_documents(dokumen)
print(len(chunks_token))

One drawback: TokenTextSplitter cuts per token without regard for sentence boundaries, so pieces can be broken mid-sentence. For clean prose, RecursiveCharacterTextSplitter usually produces chunks that are more readable and easier for the model to understand.

#Chunking Strategy for RAG

The choice of splitter isn't a one-and-done decision. Here's a practical strategy to hold on to:

• Long prose documents (articles, reports): RecursiveCharacterTextSplitter with chunk_size 400-800 and chunk_overlap 10-15 percent.
• Code and technical files: preserve function or block boundaries — use newline-character-based separators and consider language-specific splitters.
• Semantic boundaries: if the document has clear sections (chapters, clauses), cut at those boundaries first before the fine-grained splitter works.
• Embedding model size: match chunk_size to the token limit of the embedding model you use; chunks that are too long can get truncated during embedding.

Verify chunking quality: read a few chunks at random. If the context of a question is cut between two chunks, raise chunk_overlap or move the cut boundary to a higher separator. This process can't be fully automated — manual evaluation still matters.

#Conclusion

Your data pipeline is now complete up to the point before storage: any format — PDF, web, CSV, JSON — is successfully normalized into Documents, then cut into well-managed chunks via splitters and an adapted chunking strategy. The chunking quality today determines the retrieval quality in episode 11.

Key takeaways:

• A Document has page_content for text and metadata for source context — all loaders normalize to this structure.
• PyPDFLoader for page-based PDFs, WebBaseLoader for web, CSVLoader and JSONLoader for structured data.
• RecursiveCharacterTextSplitter cuts at natural paragraph/sentence/word boundaries — the safe default.
• chunk_size controls chunk length, chunk_overlap preserves context at cut boundaries.
• TokenTextSplitter gives token-based control, useful when cost and context window are the main considerations.
• Manual chunking evaluation is still a must before moving on to embedding and retrieval.

In episode 10 we convert chunks into vectors: Embeddings & Vector Stores — from OpenAIEmbeddings and HuggingFaceEmbeddings to storing the index in Chroma, FAISS, pgvector, and Qdrant. See you there!