Learn LangChain - Observability & Evaluation with LangSmith

#Introduction

In episode 18 you optimized performance and cost: streaming, batching, parallelism, async, prompt caching, model tiering, and caching layers. But optimization without measurement is dangerous — you could make your pipeline faster and simultaneously worse without realizing it. Episode 19 answers this with LangSmith: observability through tracing and evaluation through datasets, evaluators, and regression testing. As a result, every prompt or pipeline change can be justified with data, not intuition.

#Why LLM Applications Need Observability

LLM applications are non-deterministic, so their failure modes are unique too: not just visible errors, but subtly wrong answers. Imagine a RAG chain that silently answers from the model's memory instead of the retrieved context — without a complete record, you'd never know. Observability here isn't just logging; it's recording every step: the prompt that went in, the model used, the tokens consumed, the duration, and the output produced.

LangSmith provides three core capabilities we'll break down today:

• Tracing: a visual record of every run in the form of a tree.
• Dataset: a collection of input and expected-output pairs.
• Evaluator: automated scoring of output quality.

#Setup: Automatic Instrumentation

LangSmith tracing is automatic. After installing the packages and exposing the API key, almost every ecosystem component — langchain, langchain-openai, langgraph, even deepagents — is recorded immediately without extra code.

pip install langchain langchain-openai langsmith

LANGSMITH_API_KEY=lsv2_xxxx
LANGSMITH_TRACING=true
LANGSMITH_PROJECT=my-rag-app

The LANGSMITH_API_KEY variable holds your account credential, LANGSMITH_TRACING is set to the value true to turn on tracing, and LANGSMITH_PROJECT separates traces per application. After that, every invoke, ainvoke, stream, or batch automatically appears in the dashboard as a trace. Installing via pip install langsmith is enough to get started.

#Reading a Trace: Run Tree

When you open the LangSmith dashboard, each call appears as a tree of runs. The root is the chain or agent, and its children are each step: retrieval, prompt, model call, tool call, down to the parser. This tree is your debugging roadmap — you can see where latency balloons or tokens are wasted.

trace_metadata.py

from langchain_openai import ChatOpenAI
 
model = ChatOpenAI(model="gpt-5-mini")
 
model.invoke(
    "Apa itu hybrid search?",
    config={"metadata": {"fitur": "rag", "user": "tim-internal"}},
)

With a config containing metadata, you tag traces so they're easy to filter in the dashboard — for example, filtering all traces from a specific feature. Use metadata for projects, prompt versions, or experiment sessions. This habit turns the LangSmith dashboard into an analysis tool, not just a record.

#LangSmith Gateway

Managing provider API keys centrally is tedious — each developer needs their own key, and applications store many secrets. The LangSmith Gateway solves this: the application only holds one LANGSMITH_API_KEY, and the gateway forwards requests to providers while recording traces automatically.

gateway.py

from langchain_openai import ChatOpenAI
 
model = ChatOpenAI(
    base_url="https://gateway.langsmith.com/openai",
    model="gpt-5-mini",
)

The gateway serves endpoints compatible with provider SDKs, so its use is nearly invisible to your code. As a bonus, the team can share one access without sharing credentials — while making it easy to audit who uses which model.

#Datasets and Evaluators

Evaluation needs a comparison standard, and that's the dataset's role in LangSmith. A dataset contains example inputs with expected outputs or reference answers. Create it via the dashboard UI or via the SDK with Client.

dataset.py

from langsmith import Client
 
client = Client()
client.create_dataset(
    dataset_name="rag-qa-indonesia",
    description="Pertanyaan QA untuk pipeline RAG Indonesia",
)
client.create_examples(
    dataset_name="rag-qa-indonesia",
    inputs=[{"question": "Apa itu chunking?"}],
    outputs=[{"answer": "Memecah dokumen menjadi segmen kecil."}],
)

Evaluators score the produced output. There are ready-to-use built-in evaluators: exact match for literal matching, semantic similarity for meaning similarity, and criteria-based ones for assessments like correctness or helpfulness. For subtle judgments, use LLM-as-judge — an evaluator that uses another model to rate answer quality.

#Regression Testing: Evaluating Prompt and RAG

Once the dataset and evaluators are ready, run a regression test every time you change a prompt, model, or pipeline. LangSmith runs the chain against the dataset, scores each output, and presents comparable scores across experiments — exactly like unit tests, but for answer quality.

evaluate.py

from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from langsmith import evaluate
 
prompt = ChatPromptTemplate.from_template("Jawab: {question}")
chain = prompt | ChatOpenAI(model="gpt-5-mini")
 
def target(inputs: dict) -> dict:
    return {"answer": chain.invoke(inputs["question"]).content}
 
evaluate(
    target,
    data="rag-qa-indonesia",
    evaluators=["correctness"],
    experiment_prefix="rag-v1",
)

For RAG, evaluation doesn't stop at the answer. Also measure retrieval quality: faithfulness ensures the answer is supported by the retrieved context, and contextual relevance judges whether the pulled context truly answers the question. Together these capture RAG's most common failure: fluent answers not based on documents.

#Conclusion

Episode 19 equipped you with the eyes and the referee: LangSmith for structured run tracing, a gateway for centralizing API keys and traces, datasets and evaluators as the comparison standard, and regression testing that locks down prompt and RAG quality on every change. Episode 18's performance optimization now has a quality watchdog.

Key takeaways:

• LangSmith instrumentation is nearly automatic: just set LANGSMITH_API_KEY and tracing is active.
• A trace is a run tree — use metadata so it's easy to filter in the dashboard.
• The gateway centralizes API keys and traces: one key for all providers and all developers.
• Datasets plus evaluators are regression tests for answer quality.
• RAG evaluation must inspect retrieval, not just the final answer.

In episode 20 we take the application off the laptop: Deployment & Production with LangServe, the LangGraph Platform, containerization, scaling, and monitoring. See you there!