Learn Mailserver - Hardening & Open Relay Protection

#Introduction

With all the features installed, it's time to lock the fortress. The two threats most often hitting poorly-maintained mail servers are becoming an open relay (exploited by spammers as a sender) and brute force (repeated login attacks). This episode closes both.

We'll make sure Postfix's relay policy is correct, test it from outside, restrict the firewall to only the needed ports, install fail2ban, and enforce client identity rules. By the end of the episode, your server will pass standard open relay tests.

#What Is an Open Relay

An open relay is a mail server willing to forward email from anyone to anyone without authentication. For spammers, it's a dream: they can send in bulk with no trace. For the server's owner, it's a disaster — the IP gets blacklisted, reputation is destroyed, and legitimate email gets rejected too.

Postfix by default refuses to relay for unknown domains, but that policy can be misconfigured — for example with an overly broad mynetworks. That's why smtpd_relay_restrictions exists as an explicit defense line.

#smtpd_relay_restrictions: The Baseline

Since Postfix 2.10, relay policy is separated from recipient policy via smtpd_relay_restrictions. The standard rules:

sudo postconf -e 'smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, defer_unauth_destination'
sudo postfix reload

The meaning is simple:

• permit_mynetworks — trusted networks may relay.
• permit_sasl_authenticated — authenticated clients (via submission) may relay.
• defer_unauth_destination — everyone else is temporarily rejected (4xx), unless the destination is a domain we serve.

Check the active values:

postconf smtpd_relay_restrictions smtpd_recipient_restrictions

Both parameters must show what you expect — the difference between them is often the source of accidental open relays.

#Testing from Outside with swaks

A policy means nothing until tested from outside the server. swaks can simulate a relay attempt:

swaks --server mail.example.com --from attacker@evil.com \
  --to victim@other-domain.com

If your server is healthy, the output shows a rejection like 554 5.7.1 Relay access denied. Also test two legitimate scenarios:

• Local relay — --to admin@example.com must be accepted (our domain).
• Authenticated relay — --auth-user admin@example.com --tls must be accepted.

For mass audits, online tools like MXToolbox provide automated open relay tests. Run one after every relay policy change.

#Firewall: Only the Needed Ports

The principle: open as little as possible, close the rest. A public mail server needs four in/out ports: 25, 587, 465, 993 (plus 143/110/995 if needed, and 443 for webmail).

With ufw:

sudo ufw allow 25/tcp
sudo ufw allow 587/tcp
sudo ufw allow 465/tcp
sudo ufw allow 993/tcp
sudo ufw default deny incoming
sudo ufw enable

Important note: port 25 is only for the internet (for other MTAs), while IMAP can be restricted to internal networks if webmail is already sufficient. Verify the rules:

sudo ufw status verbose

If you use nftables or firewalld, the principle is the same — only the syntax differs. The key point: no filter ports (10024/10025), database, or SSH exposed to the public without a reason.

#fail2ban for Brute Force

SMTP and IMAP logins are brute force targets. fail2ban monitors logs and blocks IPs that fail repeatedly. Create jails for Postfix and Dovecot in /etc/fail2ban/jail.local:

[postfix-sasl]
enabled = true
port = smtp,submission,smtps
logpath = /var/log/mail.log
 
[dovecot]
enabled = true
port = imap,imaps,pop3,pop3s
logpath = /var/log/mail.log

Restart and check the status:

sudo systemctl restart fail2ban
sudo fail2ban-client status

fail2ban-client status postfix-sasl shows the number of currently banned IPs. Adjust maxretry and bantime to your policy — for example 5 attempts within 10 minutes resulting in a 1-hour ban.

#Enforcing Client Identity

Several smtpd_* parameters tighten who can talk to the server:

sudo postconf -e 'smtpd_helo_required = yes'
sudo postconf -e 'smtpd_helo_restrictions = permit_mynetworks, reject_invalid_helo_hostname, reject_non_fqdn_helo_hostname, reject_unknown_helo_hostname'
sudo postconf -e 'smtpd_sender_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unknown_sender_domain'
sudo postfix reload

• smtpd_helo_required — every client must send a helo.
• reject_invalid_helo_hostname — the helo must be a valid hostname.
• reject_unknown_sender_domain — the sender's domain must resolve.

This combination rejects many lazy spam bots that mimic the protocol poorly. Add smtpd_data_restrictions to reject lines that don't follow the specification.

#Routine Security Audit

Hardening isn't a one-time run. Build a periodic ritual:

• Test the open relay from outside every month.
• fail2ban-client status to see attack trends.
• journalctl -u postfix -u dovecot --since "24 hours ago" | grep -iE "reject|error" to spot rejection patterns.
• Update security packages: sudo apt update && sudo apt upgrade -y.

#Conclusion

Episode 16 is done. Key takeaways:

• smtpd_relay_restrictions limits relaying to mynetworks and authenticated clients only.
• Open relay tests must be run from a public IP, not localhost.
• The firewall only opens ports 25, 587, 465, 993 (and whatever's truly needed).
• fail2ban blocks brute force by monitoring Postfix and Dovecot logs.
• smtpd_helo_* and reject_unknown_sender_domain reject lazy bots.

The fortress is locked. In episode 17 we build identity: DKIM Signing (OpenDKIM), SPF Alignment & DMARC — signing every outbound email and enforcing authentication policy up to reject. See you in episode 17!