fail2ban Deep Dive - Automated Intrusion Prevention dan Threat Response untuk Linux Servers

#Introduction

Static firewall rules melindungi terhadap ancaman yang diketahui, tetapi tidak dapat beradaptasi dengan serangan aktif. Brute force attempt di SSH, credential stuffing terhadap API Anda, atau application-level exploits memerlukan respons dinamis—mendeteksi pola berbahaya secara real-time dan memblokir sumbernya sebelum terjadi kerusakan.

fail2ban adalah framework intrusion prevention yang memonitor file log, mendeteksi perilaku berbahaya menggunakan regex patterns, dan secara otomatis memperbarui firewall rules untuk ban IP yang menyerang. Ini adalah layer dinamis di atas static iptables rules Anda, mengubah server Anda menjadi sistem pertahanan adaptif.

Ini bukan tentang menginstal fail2ban dan melupakannya. Kita akan mendalami arsitektur, sistem filter dan action, konfigurasi jail, performance tuning, dan membangun custom detection rules untuk aplikasi spesifik Anda. Di akhir, Anda akan memiliki intrusion prevention production-grade yang melindungi SSH, web servers, databases, APIs, dan custom services.

#Arsitektur fail2ban

fail2ban beroperasi pada konsep sederhana namun powerful: watch log files, match patterns, take action.

Arsitektur memiliki empat komponen inti:

Filters - Regex patterns yang match perilaku berbahaya di file log. Setiap filter mendefinisikan seperti apa "buruk" itu—failed login attempts, 404 scans, SQL injection patterns, dll.

Actions - Apa yang harus dilakukan ketika match ditemukan. Biasanya ini berarti menambahkan iptables rule untuk ban IP, tetapi actions juga dapat mengirim email, memperbarui cloud firewall rules, atau memicu custom scripts.

Jails - Kombinasi filter, action, dan konfigurasi. Jail memonitor file log tertentu menggunakan filter dan mengeksekusi actions ketika threshold terlampaui. Anda mungkin memiliki jails terpisah untuk SSH, nginx, Apache, MySQL, dan aplikasi custom.

Backends - Bagaimana fail2ban membaca file log. Opsi termasuk polling (cek file secara periodik), pyinotify (kernel-level file change notifications), gamin (file alteration monitor), dan systemd journal integration.

Alurnya: backend fail2ban memonitor file log → filter regex match pola berbahaya → jail menghitung failures per IP → threshold terlampaui → action dieksekusi (ban IP) → IP diblokir untuk waktu yang dikonfigurasi → automatic unban setelah ban time berakhir.

Bayangkan fail2ban sebagai security analyst yang tidak pernah tidur, terus membaca logs, mengidentifikasi ancaman, dan merespons secara instan.

#Instalasi dan Konfigurasi Dasar

Mari mulai dengan instalasi dan memahami struktur konfigurasi.

Ubuntu/DebianCentOS/RHEL

apt-get update
apt-get install fail2ban
systemctl enable fail2ban
systemctl start fail2ban

yum install epel-release
yum install fail2ban fail2ban-systemd
systemctl enable fail2ban
systemctl start fail2ban

File konfigurasi berada di /etc/fail2ban/:

• fail2ban.conf - Global fail2ban settings (jangan edit langsung)
• fail2ban.local - Local overrides untuk fail2ban.conf
• jail.conf - Default jail configurations (jangan edit langsung)
• jail.local - Local jail configurations (di sinilah Anda bekerja)
• filter.d/ - Filter definitions (regex patterns)
• action.d/ - Action definitions (apa yang dilakukan saat match)

File .local override file .conf. Jangan pernah edit file .conf secara langsung—mereka ditimpa saat updates. Selalu buat file .local dengan kustomisasi Anda.

#Memahami Jails

Jail adalah unit fundamental dari konfigurasi fail2ban. Mari bedah konfigurasi jail lengkap:

/etc/fail2ban/jail.local

[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
findtime = 600
bantime = 3600
action = iptables-multiport[name=SSH, port="ssh", protocol=tcp]

Memecah ini:

• enabled = true - Aktifkan jail ini
• port = ssh - Port mana yang akan di-ban (ssh resolve ke 22)
• filter = sshd - Gunakan sshd filter dari /etc/fail2ban/filter.d/sshd.conf
• logpath = /var/log/auth.log - File log mana yang dimonitor
• maxretry = 3 - Berapa banyak failures sebelum ban
• findtime = 600 - Time window dalam detik (10 menit)
• bantime = 3600 - Berapa lama ban dalam detik (1 jam)
• action = ... - Apa yang dilakukan saat ban

Logikanya: jika sebuah IP memiliki 3 failures dalam 10 menit, ban selama 1 jam.

findtime sangat kritis. Ini adalah sliding window. Jika IP gagal di menit 0, 5, dan 11, itu hanya 2 failures dalam window 10 menit ketika upaya ketiga terjadi di menit 11 (failure pertama di menit 0 berada di luar window). Memahami ini mencegah kebingungan tentang mengapa IPs tidak di-ban.

#Default Jails - Proteksi SSH

Vektor serangan paling umum adalah SSH brute force. Mari bangun proteksi SSH production-grade:

/etc/fail2ban/jail.local

[DEFAULT]
# Global settings untuk semua jails
bantime = 1h
findtime = 10m
maxretry = 3
destemail = security@example.com
sender = fail2ban@example.com
action = %(action_mwl)s
 
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 24h
findtime = 1h

Section [DEFAULT] set nilai global. Individual jails mewarisi ini tetapi dapat override.

action = %(action_mwl)s adalah predefined action yang:

• Ban IP dengan iptables
• Kirim email dengan whois information
• Sertakan relevant log lines

Predefined actions lainnya:

• %(action_)s - Hanya ban, tanpa email
• %(action_mw)s - Ban dan email dengan whois
• %(action_mwl)s - Ban, email dengan whois dan log lines

Untuk production, Anda mungkin ingin proteksi SSH yang lebih ketat:

Proteksi SSH agresif

[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 2
bantime = -1
findtime = 30m

bantime = -1 berarti permanent ban. IP tidak pernah di-unban secara otomatis. Gunakan ini dengan hati-hati—Anda mungkin mengunci legitimate users yang salah ketik password.

#Proteksi Web Server

Web servers menghadapi serangan berbeda: directory traversal, SQL injection attempts, 404 scanning, bot traffic, dan application-specific exploits.

#Proteksi nginx

nginx jails

[nginx-http-auth]
enabled = true
filter = nginx-http-auth
port = http,https
logpath = /var/log/nginx/error.log
maxretry = 3
bantime = 1h
 
[nginx-noscript]
enabled = true
port = http,https
filter = nginx-noscript
logpath = /var/log/nginx/access.log
maxretry = 6
bantime = 1h
 
[nginx-badbots]
enabled = true
port = http,https
filter = nginx-badbots
logpath = /var/log/nginx/access.log
maxretry = 2
bantime = 24h
 
[nginx-noproxy]
enabled = true
port = http,https
filter = nginx-noproxy
logpath = /var/log/nginx/access.log
maxretry = 2
bantime = 24h
 
[nginx-limit-req]
enabled = true
filter = nginx-limit-req
port = http,https
logpath = /var/log/nginx/error.log
maxretry = 10
findtime = 1m
bantime = 1h

Jails ini melindungi terhadap:

• nginx-http-auth - Failed HTTP basic auth attempts
• nginx-noscript - Upaya eksekusi scripts (.php, .asp, dll.) di static sites
• nginx-badbots - Known malicious user agents
• nginx-noproxy - Proxy abuse attempts
• nginx-limit-req - Rate limit violations (memerlukan nginx limit_req_zone)

#Proteksi Apache

Apache jails

[apache-auth]
enabled = true
port = http,https
filter = apache-auth
logpath = /var/log/apache2/error.log
maxretry = 3
bantime = 1h
 
[apache-badbots]
enabled = true
port = http,https
filter = apache-badbots
logpath = /var/log/apache2/access.log
maxretry = 2
bantime = 24h
 
[apache-noscript]
enabled = true
port = http,https
filter = apache-noscript
logpath = /var/log/apache2/error.log
maxretry = 6
bantime = 1h
 
[apache-overflows]
enabled = true
port = http,https
filter = apache-overflows
logpath = /var/log/apache2/error.log
maxretry = 2
bantime = 24h
 
[apache-modsecurity]
enabled = true
filter = apache-modsecurity
port = http,https
logpath = /var/log/apache2/modsec_audit.log
maxretry = 2
bantime = 24h

#Proteksi Database

Databases adalah high-value targets. Lindungi mereka dengan fail2ban:

#MySQL/MariaDB

Proteksi MySQL

[mysqld-auth]
enabled = true
filter = mysqld-auth
port = 3306
logpath = /var/log/mysql/error.log
maxretry = 3
bantime = 1h
findtime = 10m

Anda memerlukan custom filter untuk MySQL. Buat /etc/fail2ban/filter.d/mysqld-auth.conf:

/etc/fail2ban/filter.d/mysqld-auth.conf

[Definition]
failregex = ^%(__prefix_line)s.*Access denied for user.*\(using password: YES\).*<HOST>
            ^%(__prefix_line)s.*Access denied for user.*\(using password: NO\).*<HOST>
ignoreregex =

#PostgreSQL

Proteksi PostgreSQL

[postgresql]
enabled = true
filter = postgresql
port = 5432
logpath = /var/log/postgresql/postgresql-*-main.log
maxretry = 3
bantime = 1h

PostgreSQL filter /etc/fail2ban/filter.d/postgresql.conf:

/etc/fail2ban/filter.d/postgresql.conf

[Definition]
failregex = ^.*FATAL:  password authentication failed for user.*<HOST>
            ^.*FATAL:  no pg_hba.conf entry for host.*<HOST>
ignoreregex =

#Proteksi Aplikasi Custom

Kekuatan sebenarnya dari fail2ban adalah melindungi aplikasi custom. Mari bangun filters untuk skenario umum.

#API Rate Limiting

Lindungi API Anda dari abuse. Asumsikan aplikasi Anda log failed auth attempts:

2026-03-05 10:23:45 [ERROR] Authentication failed for IP 203.0.113.45 - Invalid API key

Buat filter /etc/fail2ban/filter.d/api-auth.conf:

/etc/fail2ban/filter.d/api-auth.conf

[Definition]
failregex = ^\S+ \S+ \[ERROR\] Authentication failed for IP <HOST> - Invalid API key
            ^\S+ \S+ \[ERROR\] Rate limit exceeded for IP <HOST>
            ^\S+ \S+ \[ERROR\] Suspicious activity detected from <HOST>
ignoreregex =

Konfigurasi jail:

API protection jail

[api-auth]
enabled = true
filter = api-auth
port = http,https
logpath = /var/log/myapp/api.log
maxretry = 5
findtime = 5m
bantime = 30m
action = iptables-multiport[name=API, port="http,https", protocol=tcp]

#Proteksi WordPress

WordPress sites menghadapi serangan konstan. Lindungi wp-login.php dan xmlrpc.php:

/etc/fail2ban/filter.d/wordpress.conf

[Definition]
failregex = ^<HOST> .* "POST /wp-login.php
            ^<HOST> .* "POST /xmlrpc.php
ignoreregex =

Konfigurasi jail:

WordPress jail

[wordpress]
enabled = true
filter = wordpress
port = http,https
logpath = /var/log/nginx/access.log
maxretry = 3
findtime = 10m
bantime = 4h

Ini ban IPs yang POST ke wp-login.php atau xmlrpc.php lebih dari 3 kali dalam 10 menit.

#Application-Specific Exploits

Deteksi SQL injection attempts di application logs:

/etc/fail2ban/filter.d/app-sqli.conf

[Definition]
failregex = ^\S+ \S+ \[SECURITY\] SQL injection attempt from <HOST>
            ^\S+ \S+ \[SECURITY\] Malicious payload detected from <HOST>
            ^\S+ \S+ \[ERROR\] Database error.*<HOST>.*UNION SELECT
            ^\S+ \S+ \[ERROR\] Database error.*<HOST>.*OR 1=1
ignoreregex =

#Teknik Filter Lanjutan

Filters adalah otak dari fail2ban. Memahami regex patterns dan filter options sangat penting untuk deteksi yang efektif.

#Anatomi Filter

Filter lengkap dengan semua opsi:

Struktur filter lanjutan

[INCLUDES]
before = common.conf
 
[Definition]
_daemon = sshd
failregex = ^%(__prefix_line)s(?:error: PAM: )?[aA]uthentication (?:failure|error|failed) for .* from <HOST>( via \S+)?\s*$
            ^%(__prefix_line)sFailed (?:password|publickey) for .* from <HOST>(?: port \d*)?(?: ssh\d*)?$
            ^%(__prefix_line)sROOT LOGIN REFUSED.* FROM <HOST>\s*$
            ^%(__prefix_line)s[iI](?:llegal|nvalid) user .* from <HOST>\s*$
ignoreregex = 
 
[Init]
journalmatch = _SYSTEMD_UNIT=sshd.service + _COMM=sshd
 
datepattern = {^LN-BEG}

Komponen kunci:

[INCLUDES] - Import common definitions. common.conf menyediakan __prefix_line yang match common log prefixes (timestamps, hostnames, dll.).

failregex - Patterns yang match failures. Multiple patterns dapat didefinisikan. <HOST> adalah special placeholder yang match IPv4/IPv6 addresses.

ignoreregex - Patterns untuk explicitly ignore. Berguna untuk mengecualikan monitoring systems atau known safe IPs dari pattern matching.

[Init] - Backend-specific initialization. journalmatch untuk systemd journal filtering.

datepattern - Cara parse timestamps. Biasanya diwarisi dari common.conf.

#Testing Filters

Sebelum deploy filter, test terhadap actual log data:

Test filter terhadap log file

fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf

Ini menunjukkan:

• Berapa banyak lines yang matched
• Regex patterns mana yang matched
• Extracted IP addresses
• Lines yang seharusnya matched tetapi tidak

#Complex Regex Patterns

Untuk application logs dengan custom formats:

Complex application filter

[Definition]
# Match JSON logs
failregex = ^\{"timestamp":"[^"]+","level":"ERROR","ip":"<HOST>","message":"Authentication failed"
            ^\{"timestamp":"[^"]+","level":"WARN","ip":"<HOST>","message":"Suspicious activity"
 
# Match structured logs dengan multiple fields
failregex = ^time=\S+ level=error ip=<HOST> msg="login failed" user=\S+ attempts=\d+
 
# Match logs dengan variable spacing
failregex = ^\s*\[\S+\]\s+ERROR\s+<HOST>\s+-\s+Failed\s+login\s+attempt

#Actions - Beyond iptables

Actions mendefinisikan apa yang terjadi ketika ban terjadi. Sementara iptables paling umum, fail2ban mendukung banyak action types.

#Anatomi Action

Actions didefinisikan di /etc/fail2ban/action.d/. Berikut simplified iptables action:

Struktur iptables action

[Definition]
actionstart = <iptables> -N f2b-<name>
              <iptables> -A f2b-<name> -j <returntype>
              <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name>
 
actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name>
             <iptables> -F f2b-<name>
             <iptables> -X f2b-<name>
 
actioncheck = <iptables> -n -L <chain> | grep -q 'f2b-<name>[ \t]'
 
actionban = <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype>
 
actionunban = <iptables> -D f2b-<name> -s <ip> -j <blocktype>
 
[Init]
name = default
protocol = tcp
chain = INPUT
blocktype = REJECT --reject-with icmp-port-unreachable
returntype = RETURN
iptables = iptables

actionstart - Dieksekusi ketika fail2ban start. Membuat iptables chain.

actionstop - Dieksekusi ketika fail2ban stop. Menghapus chain.

actioncheck - Memverifikasi action dikonfigurasi dengan benar.

actionban - Dieksekusi ketika IP di-ban. Menambahkan iptables rule.

actionunban - Dieksekusi ketika ban expires. Menghapus iptables rule.

#Custom Actions

Buat custom actions untuk infrastruktur Anda. Contoh: update AWS Security Group:

/etc/fail2ban/action.d/aws-security-group.conf

[Definition]
actionstart =
actionstop =
actioncheck =
 
actionban = aws ec2 authorize-security-group-ingress \
            --group-id <security_group_id> \
            --protocol tcp \
            --port <port> \
            --cidr <ip>/32 \
            --description "fail2ban ban"
 
actionunban = aws ec2 revoke-security-group-ingress \
              --group-id <security_group_id> \
              --protocol tcp \
              --port <port> \
              --cidr <ip>/32
 
[Init]
security_group_id = sg-0123456789abcdef
port = 22

Gunakan di jail:

Jail dengan custom action

[sshd]
enabled = true
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
action = aws-security-group[security_group_id="sg-0123456789abcdef", port="22"]

#Notification Actions

Kirim alerts ke Slack, Discord, atau PagerDuty:

/etc/fail2ban/action.d/slack-notify.conf

[Definition]
actionstart =
actionstop =
actioncheck =
 
actionban = curl -X POST <slack_webhook_url> \
            -H 'Content-Type: application/json' \
            -d '{"text":"fail2ban: Banned <ip> for <failures> failures on <name>"}'
 
actionunban = curl -X POST <slack_webhook_url> \
              -H 'Content-Type: application/json' \
              -d '{"text":"fail2ban: Unbanned <ip> on <name>"}'
 
[Init]
slack_webhook_url = https://hooks.slack.com/services/YOUR/WEBHOOK/URL

#Menggabungkan Multiple Actions

Eksekusi multiple actions saat ban:

Multiple actions

[sshd]
enabled = true
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
action = iptables-multiport[name=SSH, port="ssh", protocol=tcp]
         slack-notify[name=SSH]
         sendmail-whois[name=SSH, dest=security@example.com]

Ini ban dengan iptables, kirim Slack notification, dan email security team.

#Whitelisting dan Ignoring IPs

Jangan pernah ban IP Anda sendiri atau monitoring systems.

#Global Whitelist

Di /etc/fail2ban/jail.local:

Global whitelist

[DEFAULT]
ignoreip = 127.0.0.1/8 ::1
           10.0.0.0/8
           192.168.1.0/24
           203.0.113.50

Ini mencegah banning:

• Localhost
• Internal networks
• Specific monitoring server

#Per-Jail Whitelist

Override global whitelist untuk specific jails:

Per-jail whitelist

[sshd]
enabled = true
filter = sshd
logpath = /var/log/auth.log
ignoreip = 127.0.0.1/8 10.0.0.0/8 203.0.113.100

#Performance Tuning

fail2ban dapat berdampak pada performance di high-traffic servers. Optimalkan.

#Backend Selection

Pilih backend yang tepat untuk log monitoring:

Konfigurasi backend

[DEFAULT]
# Options: auto, pyinotify, gamin, polling, systemd
backend = systemd

systemd - Terbaik untuk systemd-based systems. Menggunakan journal langsung, tanpa file polling.

pyinotify - Menggunakan kernel inotify untuk file changes. Efisien, memerlukan python-pyinotify package.

polling - Cek files secara periodik. Paling tidak efisien, paling compatible.

auto - Secara otomatis memilih backend terbaik yang tersedia.

Untuk systemd systems, gunakan systemd backend:

systemd backend jail

[sshd]
enabled = true
filter = sshd
backend = systemd
maxretry = 3

Filter memerlukan journalmatch:

Filter dengan journalmatch

[Definition]
failregex = ^.*Failed password for .* from <HOST>
 
[Init]
journalmatch = _SYSTEMD_UNIT=sshd.service

#Database Backend

Secara default, fail2ban menggunakan pickle files untuk persistence. Untuk performance yang lebih baik, gunakan SQLite:

/etc/fail2ban/fail2ban.local

[Definition]
dbfile = /var/lib/fail2ban/fail2ban.sqlite3
dbpurgeage = 86400

dbpurgeage menghapus old ban records setelah 24 jam.

#Monitoring dan Management

Production systems memerlukan visibility ke operasi fail2ban.

#Cek Status

fail2ban status commands

# Overall status
fail2ban-client status
 
# Specific jail status
fail2ban-client status sshd
 
# Tampilkan banned IPs
fail2ban-client status sshd | grep "Banned IP"

#Manual Ban/Unban

Manual IP management

# Ban IP secara manual
fail2ban-client set sshd banip 203.0.113.45
 
# Unban IP
fail2ban-client set sshd unbanip 203.0.113.45
 
# Unban semua IPs di jail
fail2ban-client unban --all

#Reload Configuration

Setelah mengubah konfigurasi:

Reload fail2ban

# Reload semua jails
fail2ban-client reload
 
# Reload specific jail
fail2ban-client reload sshd
 
# Restart fail2ban service
systemctl restart fail2ban

#Log Analysis

Monitor fail2ban logs:

Watch fail2ban logs

# Real-time log monitoring
tail -f /var/log/fail2ban.log
 
# Tampilkan recent bans
grep "Ban" /var/log/fail2ban.log | tail -20
 
# Tampilkan recent unbans
grep "Unban" /var/log/fail2ban.log | tail -20
 
# Hitung bans per jail
grep "Ban" /var/log/fail2ban.log | awk '{print $NF}' | sort | uniq -c | sort -rn

#Integrasi dengan Cloud Firewalls

Untuk cloud infrastructure, integrasikan fail2ban dengan native firewall services.

#AWS Security Groups

/etc/fail2ban/action.d/aws-waf.conf

[Definition]
actionban = aws wafv2 update-ip-set \
            --name fail2ban-blocklist \
            --scope REGIONAL \
            --id <ip_set_id> \
            --addresses <ip>/32 \
            --region <region>
 
actionunban = aws wafv2 update-ip-set \
              --name fail2ban-blocklist \
              --scope REGIONAL \
              --id <ip_set_id> \
              --addresses "" \
              --region <region>
 
[Init]
ip_set_id = your-ip-set-id
region = us-east-1

#Cloudflare Firewall Rules

/etc/fail2ban/action.d/cloudflare.conf

[Definition]
actionban = curl -X POST "https://api.cloudflare.com/client/v4/zones/<zone_id>/firewall/access_rules/rules" \
            -H "Authorization: Bearer <api_token>" \
            -H "Content-Type: application/json" \
            --data '{"mode":"block","configuration":{"target":"ip","value":"<ip>"},"notes":"fail2ban"}'
 
actionunban = curl -X DELETE "https://api.cloudflare.com/client/v4/zones/<zone_id>/firewall/access_rules/rules/<rule_id>" \
              -H "Authorization: Bearer <api_token>"
 
[Init]
zone_id = your-zone-id
api_token = your-api-token

#Kesalahan Umum dan Pitfalls

#Kesalahan 1: Tidak Testing Filters

Deploy untested filters menyebabkan false positives atau tidak ada matches sama sekali. Selalu test:

bash

fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf

#Kesalahan 2: Banning Diri Sendiri

Lupa whitelist management IPs Anda. Selalu tambahkan IPs Anda ke ignoreip:

ini

[DEFAULT]
ignoreip = 127.0.0.1/8 YOUR.MANAGEMENT.IP.ADDRESS

#Kesalahan 3: Settings Terlalu Agresif

maxretry = 1 dengan findtime = 1m akan ban legitimate users yang salah ketik passwords. Balance security dengan usability:

Settings yang reasonable

maxretry = 3
findtime = 10m
bantime = 1h

#Kesalahan 4: Tidak Monitoring fail2ban Itself

fail2ban dapat fail secara diam-diam. Monitor statusnya:

Health check script

#!/bin/bash
if ! systemctl is-active --quiet fail2ban; then
    echo "fail2ban tidak berjalan!"
    systemctl start fail2ban
    # Kirim alert
fi

#Kesalahan 5: Mengabaikan Log Rotation

fail2ban dapat kehilangan track file log selama rotation. Pastikan proper logrotate configuration:

/etc/logrotate.d/fail2ban

/var/log/fail2ban.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0640 root adm
    postrotate
        fail2ban-client flushlogs >/dev/null 2>&1 || true
    endscript
}

#Real-World Production Configuration

Berikut adalah complete, production-ready fail2ban configuration:

/etc/fail2ban/jail.local

[DEFAULT]
# Global settings
bantime = 1h
findtime = 10m
maxretry = 3
backend = systemd
destemail = security@example.com
sender = fail2ban@example.com
action = %(action_mwl)s
 
# Whitelist
ignoreip = 127.0.0.1/8 ::1
           10.0.0.0/8
           192.168.0.0/16
           YOUR.MANAGEMENT.IP
 
# Database
dbpurgeage = 86400
 
# Proteksi SSH
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 24h
findtime = 1h
 
# Proteksi SSH DDoS
[sshd-ddos]
enabled = true
port = ssh
filter = sshd-ddos
logpath = /var/log/auth.log
maxretry = 10
findtime = 1m
bantime = 10m
 
# nginx HTTP Auth
[nginx-http-auth]
enabled = true
filter = nginx-http-auth
port = http,https
logpath = /var/log/nginx/error.log
maxretry = 3
bantime = 1h
 
# nginx Rate Limiting
[nginx-limit-req]
enabled = true
filter = nginx-limit-req
port = http,https
logpath = /var/log/nginx/error.log
maxretry = 10
findtime = 1m
bantime = 1h
 
# nginx Bad Bots
[nginx-badbots]
enabled = true
port = http,https
filter = nginx-badbots
logpath = /var/log/nginx/access.log
maxretry = 2
bantime = 24h
 
# nginx 404 Scanning
[nginx-noscript]
enabled = true
port = http,https
filter = nginx-noscript
logpath = /var/log/nginx/access.log
maxretry = 6
bantime = 6h
 
# nginx Proxy Abuse
[nginx-noproxy]
enabled = true
port = http,https
filter = nginx-noproxy
logpath = /var/log/nginx/access.log
maxretry = 2
bantime = 24h
 
# MySQL Authentication
[mysqld-auth]
enabled = true
filter = mysqld-auth
port = 3306
logpath = /var/log/mysql/error.log
maxretry = 3
bantime = 1h
 
# PostgreSQL Authentication
[postgresql]
enabled = true
filter = postgresql
port = 5432
logpath = /var/log/postgresql/postgresql-*-main.log
maxretry = 3
bantime = 1h
 
# Custom API Protection
[api-auth]
enabled = true
filter = api-auth
port = http,https
logpath = /var/log/myapp/api.log
maxretry = 5
findtime = 5m
bantime = 30m
 
# WordPress Protection
[wordpress]
enabled = true
filter = wordpress
port = http,https
logpath = /var/log/nginx/access.log
maxretry = 3
findtime = 10m
bantime = 4h

Expand code

Supporting filter untuk API protection /etc/fail2ban/filter.d/api-auth.conf:

/etc/fail2ban/filter.d/api-auth.conf

[Definition]
failregex = ^\S+ \S+ \[ERROR\] Authentication failed for IP <HOST>
            ^\S+ \S+ \[ERROR\] Rate limit exceeded for IP <HOST>
            ^\S+ \S+ \[SECURITY\] Suspicious activity from <HOST>
            ^\S+ \S+ \[ERROR\] Invalid API key from <HOST>
ignoreregex =

#Troubleshooting

#fail2ban Tidak Banning

Cek ini secara berurutan:

1. Apakah jail enabled?

bash

fail2ban-client status | grep "Jail list"

2. Apakah filter matching?

bash

fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf

3. Apakah IP whitelisted?

bash

grep ignoreip /etc/fail2ban/jail.local

4. Apakah maxretry tercapai?

bash

fail2ban-client status sshd

5. Cek fail2ban logs:

bash

tail -100 /var/log/fail2ban.log | grep ERROR

#False Positives

Legitimate users ter-ban:

1. Tingkatkan maxretry dan findtime
2. Tambahkan IPs mereka ke ignoreip
3. Review filter patterns untuk overly broad matches
4. Cek apakah monitoring systems memicu bans

#Performance Issues

fail2ban mengonsumsi terlalu banyak resources:

1. Switch ke systemd backend
2. Kurangi findtime untuk track fewer historical failures
3. Disable debug logging
4. Gunakan SQLite database backend
5. Kurangi jumlah active jails

#Skenario Lanjutan

#Distributed fail2ban

Untuk multi-server environments, sinkronisasi bans across servers:

Ban synchronization script

#!/bin/bash
# /usr/local/bin/fail2ban-sync.sh
 
SERVERS="server1.example.com server2.example.com server3.example.com"
JAIL="sshd"
 
# Dapatkan currently banned IPs
BANNED_IPS=$(fail2ban-client status $JAIL | grep "Banned IP list" | sed 's/.*://;s/\s//g')
 
# Sync ke servers lain
for SERVER in $SERVERS; do
    if [ "$SERVER" != "$(hostname -f)" ]; then
        for IP in $BANNED_IPS; do
            ssh $SERVER "fail2ban-client set $JAIL banip $IP" 2>/dev/null
        done
    fi
done

Run via cron setiap menit:

Cron job

* * * * * /usr/local/bin/fail2ban-sync.sh

#Adaptive Ban Times

Tingkatkan ban time untuk repeat offenders:

Adaptive banning

[sshd]
enabled = true
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
findtime = 10m
 
# First ban: 1 jam
bantime = 1h
 
# Tingkatkan ban time untuk repeat offenders
bantime.increment = true
bantime.factor = 2
bantime.maxtime = 30d

Ini menggandakan ban time untuk setiap subsequent ban, hingga maksimum 30 hari.

#Security Best Practices

#Defense in Depth

fail2ban adalah satu layer. Kombinasikan dengan:

1. Strong authentication (SSH keys, 2FA)
2. Network segmentation
3. Intrusion detection systems (OSSEC, Wazuh)
4. Log aggregation dan analysis (ELK, Splunk)
5. Regular security audits

#Principle of Least Privilege

Jangan expose services yang tidak perlu:

Minimalkan attack surface

# Hanya izinkan SSH dari management network
iptables -A INPUT -p tcp --dport 22 -s 10.0.0.0/8 -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j DROP
 
# Biarkan fail2ban handle sisanya

#Regular Updates

Jaga fail2ban dan filters tetap updated:

Update fail2ban

apt-get update
apt-get upgrade fail2ban

New filters ditambahkan secara regular untuk emerging threats.

#Kapan TIDAK Menggunakan fail2ban

fail2ban tidak selalu solusi yang tepat.

#Application-Level Rate Limiting

Untuk API rate limiting, gunakan application-level solutions (nginx limit_req, API gateways) alih-alih fail2ban. Mereka lebih granular dan tidak memerlukan log parsing.

#DDoS Protection

fail2ban tidak dapat handle large-scale DDoS attacks. Gunakan:

• Cloud-based DDoS protection (Cloudflare, AWS Shield)
• Hardware DDoS mitigation appliances
• Upstream ISP filtering

#Real-Time Threat Intelligence

fail2ban bereaksi terhadap serangan setelah terjadi. Untuk proactive blocking, integrasikan threat intelligence feeds langsung ke firewall Anda.

#High-Frequency Attacks

Jika Anda melihat ribuan serangan per detik, log parsing fail2ban akan terlalu lambat. Gunakan kernel-level solutions seperti eBPF-based filtering atau hardware firewalls.

#Conclusion

fail2ban mengubah static firewall rules menjadi sistem pertahanan adaptif. Dengan memonitor logs, mendeteksi patterns, dan secara otomatis merespons ancaman, ini menyediakan layer keamanan dinamis yang dibutuhkan production systems.

Prinsip kunci:

• Test filters sebelum deployment
• Whitelist IPs Anda sendiri
• Balance security dengan usability
• Monitor fail2ban itself
• Integrasikan dengan broader security stack Anda
• Jaga configurations version controlled

Mulai dengan production configuration example, kustomisasi filters untuk aplikasi Anda, dan tune thresholds berdasarkan traffic patterns Anda. Dikombinasikan dengan strong authentication, network segmentation, dan regular updates, fail2ban secara signifikan menghardening infrastruktur Anda terhadap constant stream of attacks yang dihadapi setiap internet-facing server.

Ingat: fail2ban adalah reactive, bukan proactive. Ini adalah satu layer dalam defense-in-depth. Gunakan bersama strong authentication, minimal attack surface, intrusion detection, dan regular security audits untuk membangun sistem yang truly resilient.