Learning nginx - Performance Tuning & OS Kernel Optimization

#Introduction

NGINX has been fast since birth, but "fast" is relative. With the right tuning, the same server can serve tens of thousands of additional requests per second. This Episode 16 covers performance tuning at both the NGINX and Linux kernel levels.

You'll optimize the number of workers and their connections, enable sendfile and TCP options, turn on gzip and brotli compression, leverage the open file cache, and tune kernel parameters in /etc/sysctl.conf. By the end of this episode, your NGINX will run at its best on the hardware you have.

#NGINX Worker Optimization

#Number of Workers and Connections

NGINX performs best when the number of workers follows the CPU cores:

worker_processes auto;
worker_rlimit_nofile 65535;
 
events {
    worker_connections 1024;
    use epoll;
    multi_accept on;
}

• worker_processes auto; — one worker per CPU core.
• worker_rlimit_nofile — raises the file descriptor limit per worker.
• use epoll; — an efficient event loop on Linux.
• multi_accept on; — a worker accepts all new connections at once, not one by one.

Practical limit: total maximum connections is roughly worker_processes × worker_connections. Adjust worker_connections to your target load.

#Optimizing Static File Delivery

#sendfile, tcp_nopush, tcp_nodelay

These three directives determine how NGINX sends data through the kernel:

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
 
    keepalive_timeout 65;
}

• sendfile on; — files are sent directly from the kernel without passing through NGINX buffers. This is a big saving for static files.
• tcp_nopush on; — batches several packets into one before sending, works with sendfile.
• tcp_nodelay on; — disables the Nagle algorithm for small responses that must arrive immediately.

The sendfile and tcp_nopush combination is good for large files; tcp_nodelay matters for interactive applications and APIs.

#Gzip and Brotli Compression

#Enabling Gzip

Compression reduces transfer size several times over for text content:

http {
    gzip on;
    gzip_comp_level 5;
    gzip_min_length 256;
    gzip_types text/plain text/css application/json application/javascript image/svg+xml;
    gzip_vary on;
}

gzip_types must list the types to compress. Note that gzip is only for text; PNG, JPEG, and video files are usually already compressed and don't need it.

#Brotli

Brotli compresses better than gzip, but requires an additional module:

http {
    brotli on;
    brotli_comp_level 6;
    brotli_types text/plain text/css application/json application/javascript;
}

The brotli module isn't part of the default build. If your distro provides it through the nginx-module-brotli package, enable it with load_module; if not, gzip is already good enough as a standard.

#Open File Cache

#Caching File Descriptors

Every request for a static file requires opening the file, which is expensive when thousands of requests arrive. open_file_cache stores metadata and file descriptors:

http {
    open_file_cache max=10000 inactive=20s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;
}

• max=10000 — the maximum number of entries in the cache.
• inactive=20s — entries not accessed for 20 seconds are removed.
• valid=30s — the validation period for file information.
• min_uses=2 — new files are cached after being used at least 2 times.

For servers serving many static files, this cache significantly reduces disk time.

#Tuning the Linux OS Kernel

#Network Socket Buffers and Connection Queues

Part of NGINX's performance is determined by the kernel. Edit /etc/sysctl.conf:

net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.ip_local_port_range = 1024 65535

• net.core.somaxconn — the length of the connection queue waiting to be accepted by the application. Many frameworks need a value above the default of 128.
• net.ipv4.tcp_tw_reuse — allows reusing TIME_WAIT connections, important with many outgoing connections.
• ip_local_port_range — expands the ephemeral port range for many outgoing connections.

Apply the changes without rebooting:

sudo sysctl -p

sysctl -p loads the new values from /etc/sysctl.conf. Verify with sysctl net.core.somaxconn.

#Conclusion

Episode 16 squeezed maximum performance out of NGINX: you optimized workers and their connections, enabled sendfile and TCP options, turned on gzip and brotli, leveraged the open file cache, and tuned the Linux kernel.

Key takeaways:

• worker_processes auto and worker_connections determine connection capacity.
• sendfile, tcp_nopush, and tcp_nodelay optimize data delivery.
• Gzip and brotli cut the transfer size of text content.
• open_file_cache caches file descriptors for static files.
• sysctl kernel parameters raise the connection queue and TIME_WAIT reuse.
• Apply sysctl with sysctl -p without rebooting.

In the next episode we'll discuss the NGINX stream module (Layer 4 TCP/UDP load balancing) — TCP proxying for databases and services, UDP load balancing for DNS, and SSL passthrough without decrypting at NGINX.