Learn ChromaDB - Persistence, Import & Export

#Introduction

Since episode 3 you have been using PersistentClient — but how deeply do you understand what happens behind it? Episode 11 covers persistence, import, and export: how data is actually stored on disk, how to back up and restore, and how to move collections between environments.

The data in your collections is an asset — especially after episode 10 taught you how to fill them neatly. This episode ensures that asset is safe, movable, and not dependent on a single machine.

#How Persistence Works

#Storage Directory Structure

PersistentClient stores all data in the folder you specify via path. The structure is not a single file — ChromaDB uses SQLite for metadata and segment files for the vector index:

chroma-data/
├── chroma.sqlite3
└── <collection-uuid>/
    ├── header.bin
    └── data_level0.bin

chroma.sqlite3 stores collection definitions, documents, and metadata. Each collection folder contains the HNSW index in binary format. Both must be backed up together — backing up only the SQLite will leave the index inconsistent.

#Creating a PersistentClient Correctly

PersistentClient dengan path eksplisit

client = chromadb.PersistentClient(path="/data/chroma")

chromadb.PersistentClient(path="/data/chroma") creates the folder if it does not exist. Make sure the folder is writable and has enough space — the HNSW index takes up more space than just the text documents.

#Backup and Restore

#Stop-the-World Backup

The safest and simplest method: stop the process using the collection, copy the entire folder, then start it again:

rsync -a --delete /data/chroma/ /backup/chroma-$(date +%F)/

rsync -a --delete /data/chroma/ /backup/chroma-$(date +%F)/ creates a full snapshot of the folder with a date-stamped name. Restoring is just copying it back:

rsync -a /backup/chroma-2026-08-05/ /data/chroma/

Copying while ChromaDB is writing can produce corrupt files. For automated backup schedules, pause writes or use client-server mode and back up outside peak hours.

#Exporting and Importing Data

#Exporting All Data to a Portable Format

The most portable way to move data: read the entire collection and save it as parquet or JSON. First, make sure pandas is installed:

pip install pandas pyarrow

Then export:

Export collection ke parquet

import pandas as pd
 
data = collection.get(include=["documents", "metadatas", "embeddings"])
df = pd.DataFrame({
    "id": data["ids"],
    "document": data["documents"],
    "metadata": data["metadatas"],
    "embedding": data["embeddings"],
})
df.to_parquet("export-artikel.parquet")

df.to_parquet("export-artikel.parquet") produces a single file that can be moved to another machine. Parquet is chosen because it is compressed and fast to read — an ideal format for large data.

#Importing Back into a New Collection

To import, read the parquet and upsert with the already-stored embeddings:

Import dari parquet

df = pd.read_parquet("export-artikel.parquet")
 
collection = client.get_or_create_collection("artikel-restored")
collection.upsert(
    ids=df["id"].tolist(),
    documents=df["document"].tolist(),
    metadatas=df["metadata"].tolist(),
    embeddings=[list(v) for v in df["embedding"]],
)

collection.upsert(ids=df["id"].tolist(), ..., embeddings=[...]) restores the data exactly — including the original embeddings, so no re-embedding is needed. This is the safest pattern for data migration.

#Migrating Between Environments

#Copy Folder vs Export-Import

Two migration strategies with different characteristics:

Copying the folder is fastest for dev-to-staging with the same version. Export-import is safer when versions differ or when you want to filter data — for example, only moving collections with a certain tenant.

#Ensuring Version Consistency

One common trap: ChromaDB storage files are not guaranteed compatible across major versions. Before migrating by folder copy, make sure the chromadb versions in both environments match:

pip show chromadb | grep -i version

pip show chromadb | grep -i version on both machines should show the same number. If they differ, use the parquet export-import route instead of a folder copy.

#Closing

Episode 11 made sure your data is no longer fragile: understanding the SQLite and HNSW index storage structure, backing up with rsync, restoring, exporting a collection to parquet, and importing it back — including migration strategies between environments that are safe and version-consistent.

Key takeaways:

• Persistence consists of SQLite and binary index files; backups must be complete.
• Safest backup: stop writes, rsync the entire folder, run again.
• Parquet export carries ids, documents, metadatas, and embeddings at once.
• Import uses upsert with the original embeddings to avoid re-embedding.
• Folder copy is fast; export-import is flexible for different versions.
• Make sure the chromadb versions match before migrating via copy.

In the next episode, episode 12, we will discuss client-server mode and settings — running chroma run, connecting with HttpClient, setting host and port, understanding the tenant and database model, and server configuration for local versus production deployment. Time for your data to leave a single process.