Learn ChromaDB - Scaling & Performance

#Introduction

Your application is running, data is growing, and the moment arrives for the scary question: can this serve millions of vectors? Episode 17 answers it by covering scaling and performance: how the Rust server handles large volumes, batch ingestion, index tuning, ef search optimization, metadata pre-filtering, and proper resource sizing.

Performance in ChromaDB is not a mystery — it consists of specific decisions you can control. Let us break them down one by one.

#The Rust Server and Million-Vector Scale

#Why the Rust Server Is the Foundation of Scaling

The Rust server is the foundation of modern ChromaDB scaling. Compared to the Python FastAPI server, Rust provides more deterministic performance and more efficient memory usage — the two things that matter most as data grows to millions of vectors.

The Rust server loads the entire HNSW index into memory for fast queries. This is why RAM sizing is the most important decision in this section: an index that fits in RAM is far faster than one that must be accessed from disk.

#Batch Operations for Ingestion

#Avoid One-at-a-Time

The most common performance mistake: calling add in a loop for every document. Each call carries protocol overhead and partial reindexing. The solution: batching.

Ingestion batch (benar)

batch_ids = []
batch_docs = []
batch_meta = []
 
for dok in dokumen_sumber:
    batch_ids.append(buat_id(dok))
    batch_docs.append(dok)
    batch_meta.append({"source": "ingestion"})
 
collection.upsert(
    ids=batch_ids,
    documents=batch_docs,
    metadatas=batch_meta,
)

The example above collects data, then calls collection.upsert(...) once for the entire batch. The comparison: 10,000 small calls versus 1 large call can make the latter 10-100 times faster.

#Reasonable Batch Sizes

The ideal batch size depends on embedding dimensions and RAM. A common starting point: 1,000-10,000 items per call. Do not force a giant batch into a single memory chunk — split into several 10k calls if needed:

Batch dengan chunk

for i in range(0, len(dokumen_sumber), 5000):
    batch = dokumen_sumber[i : i + 5000]
    collection.upsert(
        ids=[buat_id(d) for d in batch],
        documents=batch,
        metadatas=[{"source": "ingestion"} for _ in batch],
    )

The loop above calls collection.upsert(...) every 5,000 documents. This balances throughput against memory usage.

#Index Tuning for Performance

#Combining Parameters from Episode 9

For large datasets, start from proven settings:

Konfigurasi index untuk skala

collection = client.create_collection(
    name="skala-besar",
    metadata={
        "hnsw:space": "cosine",
        "hnsw:ef_construction": 200,
        "hnsw:M": 16,
    },
)

metadata={"hnsw:ef_construction": 200, "hnsw:M": 16} is a healthy starting point: accurate enough builds without wasting excessive memory. After the large data is in, adjust ef_search per query to balance recall and latency.

#Keeping Index Quality While Writing

Every large insert changes the index. For mass ingestion, consider: insert all data first with standard ef_construction, then — if accuracy feels insufficient — rebuild the collection with higher values. Rewriting (reindexing) is cheaper than rebuilding from scratch.

#Query Latency Optimization

#Playing with ef_search

ef_search is the knob with the most influence on query latency. Low values speed up queries but lower recall; high values do the opposite. Start with 100 and raise it if results feel insufficiently accurate:

Query dengan ef_search diatur

hasil = collection.query(
    query_texts=["topik pencarian"],
    n_results=10,
    search_params={"ef_search": 256},
)

search_params={"ef_search": 256} uses higher search quality for important queries. Measure with timing to find the optimal point for your use case.

#Metadata Pre-Filtering: Trimming the Search Space

Metadata pre-filtering — filtering before the vector search — trims the number of vectors that must be compared. For large collections, this often delivers the most dramatic latency reduction:

Pra-filter kategori

hasil = collection.query(
    query_texts=["deploy"],
    n_results=5,
    where={"category": "devops"},
    search_params={"ef_search": 200},
)

where={"category": "devops"} restricts the search to only the vectors in that category. If the collection has 5 million vectors but only 200 thousand are in the devops category, the search runs far faster without sacrificing accuracy.

#CPU and RAM Resource Sizing

#Estimating Required RAM

A rough estimate: the HNSW index needs several times the raw vector size. For 384-dimension (float32) vectors, one million vectors are roughly 1.5 GB — plus 2-4x index overhead.

1 juta vektor x 384 dimensi x 4 bytes ~= 1.5 GB
estimasi index total (overhead 2-4x) ~= 3-6 GB

Use this estimate as a starting point, then measure actual usage with the monitoring from episode 20. Enough RAM prevents the swapping that destroys latency.

#Structuring Deployment by Load

To go beyond 5 million vectors, combining episode 15 (load balancer) and episode 19 (advanced query patterns) becomes important. Do not forget: every replica needs RAM for its own index.

#Closing

Episode 17 made ChromaDB ready to grow: understanding the Rust server foundation for million-vector scale, batch ingestion 10-100 times faster, index tuning from episode 9, ef_search optimization and metadata pre-filtering for latency, and CPU and RAM sizing based on data volume.

Key takeaways:

• The Rust server is the scaling foundation; the HNSW index lives in RAM.
• Batch ingestion of 1,000-10,000 items is far faster than per-item.
• Start with ef_construction: 200 and M: 16 for scale.
• ef_search is adjusted per query; start at 100, raise if needed.
• Metadata pre-filtering dramatically trims the search space.
• RAM estimate: 3-6 GB per million 384-dimension vectors, then measure.

In the next episode, episode 18, we will discuss integration with LLM frameworks — making ChromaDB a VectorStore and retriever in LangChain, LlamaIndex and other framework adapters, and the production RAG pattern: chunking, embedding, storing, and querying in one complete pipeline. All the capabilities you have built now come together.