Learn LangChain - Advanced RAG

#Introduction

In episode 11 you built a working basic RAG: create an index, turn it into a retriever, pull context with similarity_search, then combine it with LCEL into a retrieve-generate pipeline. That pipeline is real and runs — until it's tested with questions whose phrasing doesn't exactly match the document contents, or the corpus has tens of thousands of chunks.

Episode 12 is a step-by-step upgrade. We dissect why simple retrieval fails, then fix it one technique at a time: multi-query retrieval, query rewriting, hybrid retrieval (vector + full-text), reranking with a cross-encoder, contextual retrieval, and finally managing the context window so the token budget stays healthy.

#Why Basic RAG Isn't Enough

Pure vector retrieval works on one assumption: the query sentence and the relevant document are close in embedding space. This assumption misses for several reasons:

• Vocabulary mismatch — the document says "laptop battery lasts 8 hours", you ask "how long is the notebook's battery life". Same meaning, different vocabulary.
• Rigid top-k — a fixed k (say 4) always takes the 4 closest chunks, even when some are barely relevant.
• Complex questions — one question can touch several scattered document sections; a single embedding query will never hit them all.

That's why production uses a series of techniques that patch each other, not one magic trick.

#Multi-Query Retrieval

The idea behind multi-query retrieval is simple: ask the LLM to generate several paraphrases of the query that capture different points of view, pull documents for each query, then merge the results. The query "how long is the notebook's battery life" could be expanded to "laptop battery duration", "battery capacity hours of use", and so on — closing the vocabulary mismatch gap. In LangChain, this is already wrapped in MultiQueryRetriever:

Multi-query retrieval

from langchain_openai import ChatOpenAI
from langchain.retrievers.multi_query import MultiQueryRetriever
 
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.2)
retriever = MultiQueryRetriever.from_llm(
    retriever=vector_store.as_retriever(search_kwargs={"k": 4}),
    llm=llm,
)
results = await retriever.ainvoke("berapa lama daya tahan notebook?")

MultiQueryRetriever still accepts invoke and ainvoke, so it's a drop-in replacement for a regular retriever in your chain. k is used per query and the results of all queries are merged, so the document count can exceed k; LangChain records the generated queries as metadata on each document — useful when debugging with LangSmith.

#Rephrase Query

In chat applications, questions often depend on prior context: "so how do we evaluate it then?" can't be searched directly without knowing what "it" refers to. Query rewriting transforms the raw question into a standalone, specific search query before sending it to the retriever. Build a small rewriting chain with ChatPromptTemplate and StrOutputParser:

Query rewriting before retrieval

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
 
rewrite_prompt = ChatPromptTemplate.from_messages([
    ("system", "Tulis ulang pertanyaan menjadi kueri pencarian mandiri, "
               "tanpa kata ganti ambigu, dan pertahankan istilah teknisnya."),
    ("human", "{question}"),
])
rewriter = rewrite_prompt | llm | StrOutputParser()
query = rewriter.invoke({
    "question": "kalau begitu, gimana cara evaluasinya?"
})
docs = retriever.invoke(query)

The rewriting result is a string, not an answer sentence — that's what StrOutputParser is for. The pipeline order becomes: rewrite first, then retrieve, then generate. Check the dialog history pattern from episode 7 if you want a cleaner pipeline.

#Hybrid Retrieval: Vector + Full-Text

Embeddings are great at capturing meaning but poor at precise term matching: serial numbers, error codes, function names. Conversely, full-text search like BM25 excels at exact token matches but doesn't understand synonyms. Hybrid retrieval combines both; EnsembleRetriever merges their results via Reciprocal Rank Fusion:

Hybrid retrieval with EnsembleRetriever

from langchain_community.retrievers import BM25Retriever
from langchain.retrievers import EnsembleRetriever
 
bm25 = BM25Retriever.from_documents(docs, k=4)
vector = vector_store.as_retriever(search_kwargs={"k": 4})
hybrid = EnsembleRetriever(
    retrievers=[bm25, vector],
    weights=[0.5, 0.5],
)
docs = hybrid.invoke("kode error E401 muncul saat build")

Weights of 0.5 and 0.5 are a healthy starting point; if the corpus has more exact technical terms, shift toward BM25. BM25Retriever needs the original list of Documents as an argument, so keep it around during index building. For large corpora, replace in-memory BM25 with Elasticsearch or PostgreSQL full-text through integrations that also return a Retriever object.

#Reranking with a Cross-Encoder

Cheap retrieval (bi-encoder) often returns many documents that are "close enough" but not truly relevant. A reranker built on a cross-encoder architecture judges query-document pairs simultaneously — far more accurate, but expensive, so it's only run on the already-thinned candidates. Combine it with ContextualCompressionRetriever:

Reranking retrieval candidates

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain_community.cross_encoders import HuggingFaceCrossEncoder
 
compressor = CrossEncoderReranker(
    model=HuggingFaceCrossEncoder(model_name="BAAI/bge-reranker-v2-m3"),
    top_n=3,
)
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=hybrid,
)
final_docs = compression_retriever.invoke("apa perbedaan RAG dan fine-tuning?")

The ideal production retrieval pipeline: fetch many cheap candidates (say k of 20) from hybrid retrieval, then rerank down to a small top_n (3-5) before it enters the prompt. The context sent to the model becomes denser, more relevant, and more token-efficient. The bge-reranker-v2-m3 model runs locally without an API key — perfect for experimentation.

#Contextual Retrieval

When a document is split into small chunks, the global context gets cut too — a chunk about "that table" loses the information that it's the network configuration table. Contextual retrieval (popularized by OpenAI research) adds a context header: before storing a chunk in the index, an LLM summarizes the document context and prepends it to the chunk.

Adding a context header to each chunk

from langchain_core.documents import Document
 
context_prompt = ChatPromptTemplate.from_messages([
    ("system", "Ringkas dokumen menjadi 3-5 kalimat konteks yang menjelaskan "
               "posisi dan topik keseluruhan dokumen."),
    ("human", "{document}"),
])
context_chain = context_prompt | llm | StrOutputParser()
 
contextual_docs = []
for doc in docs:
    context = context_chain.invoke({"document": doc.page_content})
    contextual_docs.append(Document(
        page_content=f"Konteks dokumen: {context}\n\n{doc.page_content}",
        metadata=doc.metadata,
    ))

Embed and store contextual_docs into the vector store as usual. The extra cost is in the indexing process (one LLM call per chunk), so run it once in the ingestion pipeline, not per request. At retrieval time, the context header makes isolated chunks still make sense on their own.

#Optimizing the Context Window

The more techniques you stack, the higher the risk of a blown context window and ballooning cost per request. Four main controls:

1. Prompt caching — most of the cost is the same prompt prefix repeated. Many providers offer automatic or manual caching for static parts of the prompt (system prompt, instructions, document list); cached tokens are billed far cheaper.
2. Deduplication — hybrid retrieval and multi-query often return the same documents repeatedly. Drop duplicates based on content hash before the documents enter the prompt.
3. Token budget — set a maximum context budget and fill it sequentially until full; the rest is discarded.
4. Strategic truncation — if a document is too long, cut it to the most relevant section rather than discarding it entirely.

An example of budget control combined with deduplication:

Deduplication and token budget

def dedupe_documents(docs):
    seen = set()
    unique = []
    for doc in docs:
        key = hash(doc.page_content)
        if key not in seen:
            seen.add(key)
            unique.append(doc)
    return unique
 
TOKEN_BUDGET = 6000
selected = []
used = 0
for doc in dedupe_documents(final_docs):
    n = llm.get_num_tokens(doc.page_content)
    if used + n > TOKEN_BUDGET:
        continue
    selected.append(doc)
    used += n

get_num_tokens counts tokens offline using the model's tokenizer — no network call — so it's perfect for filtering before a request. Start with a budget of 40-60 percent of the context window, leaving room for the question and answer. Remember the order: deduplication first, then budget.

#Conclusion

This episode raised your RAG from "working" to "battle-hardened": multi-query retrieval closes vocabulary gaps, query rewriting makes queries standalone, hybrid retrieval combines meaning and exact matching, reranking picks the best candidates, contextual retrieval preserves document context, and prompt caching, deduplication, and token budgets keep costs down. The more layers, the better the answer quality at a still-reasonable retrieval price.

Key takeaways:

• A single embedding query isn't enough; broaden the points of view with multi-query and rewriting.
• Hybrid retrieval unites the strengths of embeddings and BM25; balance the weights to your corpus.
• Fetch many cheap candidates, then rerank with a cross-encoder down to a small top_n.
• Contextual retrieval preserves document context on each chunk, but runs at indexing time.
• Control costs with prompt caching, hash-based deduplication, and a token budget before calling the model.

Optimal RAG is just a tool for giving the model "memory". In episode 13 we give it the ability to act: agents with LangGraph — agents that decide for themselves which tool to use, with state that can be paused and resumed (human-in-the-loop). See you there!