Learn Fzf - Performance & Large Datasets

#Introduction

In episode 16 you made fzf lists come alive — periodic reloads, queries forwarded to ripgrep, and even control from external processes via --listen. All of that feels magical when the list is a thousand lines. But what happens when the list is a million lines? That's where fzf's real question begins: how fast can it filter, and how do we keep it fast.

This episode covers fzf performance from two sides. The side you can control: choosing an algorithm with --algo, limiting the number of items before they enter, the --sort decision, and lightening the preview load. The side that comes from the developers: modern fzf's ability to scale linearly across CPU cores and the drastic reduction in memory cache footprint since 0.71 — plus practical strategies for handling million-item datasets.

The first principle to understand: fzf is a filter that runs in a pipeline, and a pipeline is only as fast as its slowest component. A list that's slow to build upstream won't be saved by a super-fast fzf downstream. Performance is the responsibility of the whole chain, not just the visible end.

#fzf's Performance Map

fzf's load is split into three stages that run partially in parallel:

1. Reading input — consuming lines from stdin asynchronously.
2. Matching — filtering lines against the query; the most CPU-hungry stage.
3. Displaying and previewing — rendering the interface and running the preview command.

The three aren't balanced. For large lists, matching dominates; for small lists, the preview is often the slowest part. Optimizing means knowing which stage is burning your time, not just switching on every "fast" option at once.

#Choosing an Algorithm: --algo

fzf provides two fuzzy matching algorithms. v2 is the default — an optimal scoring algorithm producing the best ranking quality. v1 is the old-style algorithm (the one fzy uses), simpler, with a coarser ranking quality trade-off.

fzf --algo v2
fzf --algo v1

#Limit Items Before They Enter

The first rule of large datasets: don't send more than you need. Every line entering fzf must be stored, indexed, and potentially scored. Lines you'll never pick are just garbage slowing everything down.

The two most effective ways: limit the source (filter upstream with fd or rg, not find), and trim the size (head) when you only care about the early part:

fd -t f . ~ | head -n 5000 | fzf

tail -f /var/log/app.log | fzf --tail 100000 --tac --no-sort --exact

The second example uses the --tail feature: from a log stream that never ends, fzf only keeps the last 100,000 lines — the list stays bounded even though the source is endless.

#The --sort Decision

Sorting is one of the most expensive operations in fzf. By default (--sort), fzf orders results by match score. When the list is huge and the order already means something — history you want to stay chronological, logs you want to stay in sequence — --no-sort removes that cost and keeps the input order as-is:

seq 1000000 | fzf --sort
seq 1000000 | fzf --no-sort

For tied results under --sort, --tiebreak defines the distinguishing criterion. On large datasets, the --no-sort --exact combination is a legitimate shortcut: items are still filtered, but the heavy scoring work is skipped.

#Minimize the Preview Load

When the list is already large, what often feels slow isn't the filter but the preview. Every time the cursor moves, the preview command reruns. Three adjustments with the most impact:

1. Use cheap commands. head and sed are far lighter than tools that parse an entire file.
2. Turn off unneeded processing. bat with --style=plain and without --color=always avoids color-parsing work.
3. Limit the scope. A small --preview-window and commands that only read a few lines limit work per move.

fzf --preview 'head -n 30 {}'
fzf --preview 'bat --style=plain {}'

#Modern Performance: Linear Across CPU Cores

Now let's get to the developer side. Since fzf 0.71, fzf's search performance has become linearly scalable across all CPU cores: the matching work is divided among all available processor cores, so a list twice as large doesn't mean twice the time if there are cores to work on it. This differs from older versions that mostly ran on a single core.

The practical impact: on modern multicore machines, fzf filters hundreds of thousands to millions of lines with latency you no longer feel — as long as the data streams through it, instead of waiting for all the input to arrive first.

#Memory Footprint: A Lighter Cache

At the same time, the 0.71 release recorded an 86x reduction in memory cache footprint per entry. That large number comes from frugal storage: the match-result cache now stores far smaller indexes than before, so lists with millions of items can fit in much more modest memory — and a smaller cache also means friendlier behavior for the CPU cache itself, speeding up repeated search iterations.

For you, this isn't just a number: it means millions of items are now reasonable to process on a laptop without making swap scream.

#Strategies for Millions of Items

Summarizing everything into a runnable recipe, here are the strategies for million-item datasets:

#Common Mistakes

#Closing

In this episode 17 you completed fzf's performance side: choosing an algorithm with --algo (v2 default, v1 for special needs), limiting items before they enter with fd, head, and --tail, deciding --sort versus --no-sort based on what the input order means, and lightening the preview load with cheap commands and plain styles. You also understood modern fzf's improvements — search that scales linearly across CPU cores and a memory cache 86x lighter since 0.71 — along with strategies for handling millions of items in a streaming fashion.

The message to take home: fzf's speed is a collaboration between a frugal upstream and an efficient downstream. A well-bounded dataset enters a well-accelerated engine — that's the recipe.

Now you have a full toolkit for using fzf correctly. But even the best setup sometimes doesn't behave as expected — a strange prompt, results that don't appear, a silent preview. In episode 18 we cover troubleshooting & debugging: turning on --debug, understanding the internal log, and diagnosing the most common problems seen in the field.