Learn n8n - Performance Optimization

#Introduction

In episode 15 you deployed n8n to production with Docker Compose, Kubernetes, and queue mode. Now the instance is healthy — but the workflows start to feel slow. One "giant report" workflow processes thousands of rows in a single run, several workflows carry the same logic, and executions fight over CPU on the main process.

Episode 16 covers Performance Optimization: breaking large workflows into reusable sub-workflows, applying batch processing and concurrency, as well as optimizing execution runtime and throughput.

#Measure First, Then Optimize

The first rule of optimization: don't guess. Before changing anything, look at the data you've already collected from episode 13:

• Per-node duration in Execution History — which node consumes most of the time?
• Prometheus metrics — execution patterns per hour, piling Redis queues, Node.js metrics like memory.
• Data patterns — how many items pass through each node, and how often triggers fire executions.

Optimization not based on measurement usually fixes the wrong thing. Measure, find the bottleneck, fix, then measure again.

#Breaking Workflows into Sub-workflows

A workflow that piles all logic into one canvas is hard to understand and hard to run fast. The solution is to break it into sub-workflows executed via the Execute Sub-workflow node. Each sub-workflow has one responsibility — data normalization, API calls, report formatting — and can be called from many parent workflows.

The biggest performance difference is in the execution mode:

Run once for each item -> sub-workflow dieksekusi N kali
Run once for all items -> sub-workflow dieksekusi 1 kali, N item sekaligus

Choosing "each item" for 5,000 items means running the sub-workflow 5,000 times — each with load, credential, and connection overhead. Choose "all items" when the sub-workflow processes data in batch, and use per-item mode only when each item genuinely needs separate execution context. Reusable sub-workflows also spread improvements: change once, and all parent workflows improve too. When run in queue mode, sub-workflows fanned out from a single trigger can even be executed in parallel across several workers at once.

An easy-to-remember splitting rule: if a block of nodes appears in more than two workflows, or a workflow starts getting hard to read because it's too long, that's when the block deserves to be moved to a sub-workflow.

#Batch Processing & Loop Patterns

When a workflow must process thousands of rows, processing one by one is too slow; processing all at once can blow up memory. The middle ground is batch processing with the SplitInBatches node — data is split into fixed-size groups, then each batch is processed in a loop iteration:

Input 10.000 item
  -> SplitInBatches (batchSize: 500)
  -> Loop node (proses batch)
  -> jika masih ada sisa, ulangi dari awal

Batch size is a tradeoff: too small means many iterations and per-cycle overhead; too large means memory spikes per batch. Start from a few hundred items per batch, then adjust based on the target API's load and the instance's memory limits. This pattern also protects external services from request bursts — especially important when integrating with third-party APIs that restrict rates.

One common mistake: putting SplitInBatches inside a flow that already runs per item, so the batch never gets used. SplitInBatches works best at the start of a pipeline — split early, and let each iteration process one batch from start to finish.

#Concurrency & Resource Tuning

Besides workflow structure, environment tuning determines how many executions can run simultaneously. On the main process, the limit is controlled by environment variables:

N8N_CONCURRENCY_PRODUCTION_LIMIT=10
N8N_CONCURRENCY_POLLING_LIMIT=5

A value of -1 means unlimited (default). Setting reasonable limits prevents one heavy job from flooding the instance. In queue mode, concurrency applies per worker — manage it via the --concurrency flag on n8n worker or N8N_CONCURRENCY_PRODUCTION_LIMIT in the worker environment, as discussed in episode 15.

Node.js resources can also be tuned. Heap memory is limited by a default value that may be too small for workflows with lots of data:

NODE_OPTIONS=--max-old-space-size=2048

Adjust it to the physical memory allocated to the container, and balance it with the CPU limits on compose/Helm values so processes aren't throttled.

#Optimizing Execution Runtime

Execution time is most often lost in network calls. Common waste patterns:

• N+1 calls — calling an API once per item when the API supports batching. Use the node's bulk operations, or collect IDs then call once.
• Unnecessary loops — process batches inside a Code node instead of a Loop if the logic is pure transformation.
• Repeated HTTP Requests — data that rarely changes (for example product metadata) can be cached in Redis rather than fetched on every execution.
• Dead data flowing along — before Convert to File or sending, drop unused columns with Set or Remove Duplicates to trim payloads between nodes.

Data trimming also applies to databases: the prune policy from episode 13 (EXECUTIONS_DATA_PRUNE and EXECUTIONS_DATA_MAX_AGE) keeps execution tables from ballooning, so history queries stay fast. A healthy runtime starts with data that doesn't sit idle everywhere.

Also worth noting is the too-eager trigger pattern. A cron running every minute to check data that changes every hour wastes a lot of execution slots — increase the interval, or use a polling trigger that includes an early-stop condition (for example stop right away when there's no new data). Executions that never start are the fastest executions.

#Throughput with Queue Mode

When a single process can no longer execute all the workload, throughput is raised horizontally. Queue mode from episode 15 moves executions to workers: main only dispatches jobs to Redis, and a set of workers consumes them. Increase throughput by adding worker replicas — not by enlarging a single instance.

Several knobs adjust behavior:

• OFFLOAD_MANUAL_EXECUTIONS_TO_WORKERS=true — manual executions from the editor are also diverted to workers, keeping main light for the UI.
• Scaling workers at peak — adding workers raises parallel capacity; the Redis queue dispatches jobs to idle workers.
• Batch jobs — combine many small items into one batch execution on a worker, reducing per-job overhead.

Throughput isn't only about adding workers, but ensuring every worker works on the right job with data already trimmed — a combination of the workflow structure from previous chapters with the scaling from episode 15.

#Closing

Episode 16 guided you from slow workflows to a scalable system. You measured before optimizing, broke giant workflows into reusable sub-workflows, applied batch processing with SplitInBatches, tuned concurrency and Node.js resources, trimmed runtime waste, and raised throughput via queue mode and worker scaling.

Key takeaways:

• Measure first — optimize based on per-node duration and metrics, not guesses.
• "All items" sub-workflows cut thousands of executions into one batch execution.
• SplitInBatches balances speed and memory consumption for large data.
• Concurrency is set explicitly with N8N_CONCURRENCY_PRODUCTION_LIMIT and worker flags.
• Throughput comes from worker scaling, not enlarging a single instance.

In the next episode we extend n8n beyond built-in nodes: Extensions & Custom Nodes — building custom nodes and packaging them yourself, using community nodes, and plugin development for richer automation. See you there!