Learn Mailserver - SMTP Smuggling & Protocol Security

#Introduction

After securing identity in episode 17, now we secure the protocol. 2023 shook the email world with SMTP smuggling — an attack that exploits differences in how servers interpret the end of command lines. New variants keep being found through 2026, so this isn't a one-and-done story.

This episode dissects SMTP smuggling, installs the mitigation in Postfix, then closes with two standards that require TLS between servers: MTA-STS and DANE.

#What Is SMTP Smuggling

SMTP smuggling (CVE-2023-51764 and the 2024-2026 variants) exploits differences in how two email servers interpret end-of-line characters. The SMTP protocol defines the end of a line as CRLF, but some servers also accept a bare LF (\n without \r) as a line ending — while others don't.

Attackers exploit this gap to "smuggle" additional SMTP commands inside a single session. The effects can be fatal: bypassing authentication, SPF/DKIM, and antispam, or even turning your server into a spam tool. The most vulnerable servers are those that accept bare LF in commands.

#Why This Is Dangerous

Imagine this SMTP session seen from two different sides:

MAIL FROM:<attacker@example.com>\n\nRSET\nMAIL FROM:<victim@example.com>\n

Server A reads all lines as one session; server B sees \n\n as a separator and processes the RSET and the second MAIL FROM as if they were separate commands. As a result, email can be sent to recipients while bypassing every security policy — that's the "smuggling".

In practice, attackers can forge addresses, avoid rate limits, or flood mailboxes. Because the root cause is protocol interpretation, the main mitigation is tightening the parser on the receiving side.

#Mitigation in Postfix

Postfix 3.5.18 and later provides the smtpd_forbid_bare_newline parameter to reject bare LFs. The recommended configuration:

sudo postconf -e 'smtpd_forbid_bare_newline = yes'
sudo postconf -e 'smtpd_forbid_bare_newline_count = 5'
sudo postfix reload

What the two parameters mean:

• smtpd_forbid_bare_newline = yes — reject data containing LF without CR. The value normalize (in newer versions) normalizes instead of rejecting, suitable if legitimate clients type LF.
• smtpd_forbid_bare_newline_count — the tolerated number of bare LFs before a hard rejection.

Verify the active value and check in the logs whether rejections actually happen:

postconf smtpd_forbid_bare_newline
journalctl -u postfix --since "1 hour ago" | grep -i "bare newline"

Monitor the logs after enabling. If rejections appear from legitimate servers, raise count or switch to normalize.

#MTA-STS: Requiring TLS from the Sender's Point of View

MTA-STS (MTA Strict Transport Security, RFC 8461) tells senders: "you must use TLS when sending to this domain; never fall back to plaintext." There are two parts:

The policy record at _mta-sts.example.com:

v=STSv1; id=20260803;

The policy JSON hosted at https://mta-sts.example.com/.well-known/mta-sts.txt:

{
  "version": "STSv1",
  "mode": "enforce",
  "mx": ["mail.example.com"],
  "max_age": 86400
}

enforce mode means MTA-STS-capable senders refuse to send without TLS. Verify the record:

dig TXT _mta-sts.example.com +short
curl -s https://mta-sts.example.com/.well-known/mta-sts.txt

Note: the policy is hosted on the mta-sts subdomain, with the same certificate. Your server doesn't run MTA-STS — other senders run it on your behalf. MTA-STS protects email received by your domain.

#DANE: Requiring TLS with Keys in DNS

DANE (RFC 7672) goes further: the destination server's TLS identity is proven via a TLSA record signed by DNSSEC. Senders that validate DANE refuse connections without TLS, while also verifying that the server's certificate matches the published hash.

The TLSA record for mail.example.com port 25:

_25._tcp.mail.example.com.  IN TLSA  3 1 1  aabbcc...hash-sha256

The 3 tag indicates a hash of the full certificate (published with the hash from the full chain); 1 1 is the hash type selection. DANE only means something when the domain is DNSSEC-signed — without it, the TLSA record can be forged. Check DNSSEC support:

dig +dnssec DNSSEC example.com
dig TLSA _25._tcp.mail.example.com +short

DANE closes MTA-STS's weakness (which depends on CAs and key management) because the key is pinned directly in DNSSEC-ensured DNS.

#Mandatory TLS to the Relay

If you use a relay host for outbound email (for example a provider's SMTP relay), make sure Postfix requires TLS to it:

sudo postconf -e 'relayhost = [smtp.provider.com]:587'
sudo postconf -e 'smtp_tls_security_level = encrypt'
sudo postconf -e 'smtp_tls_mandatory_protocols = !SSLv2 !SSLv3 !TLSv1 !TLSv1.1'
sudo postfix reload

relayhost = [smtp.provider.com]:587 uses brackets so the hostname isn't resolved as an MX; smtp_tls_security_level = encrypt ensures no plaintext fallback. This combination — mandatory TLS + modern protocols — is the 2026 industry standard.

#Conclusion

Episode 18 is done. Key takeaways:

• SMTP smuggling exploits differences in LF vs CRLF interpretation between servers.
• smtpd_forbid_bare_newline rejects bare LFs; monitor the logs after enabling it.
• MTA-STS requires TLS from senders, with the policy at mta-sts.example.com.
• DANE + DNSSEC pins the server identity in the TLSA record.
• Outbound relays also require TLS via smtp_tls_security_level = encrypt.

The protocol is secured. In episode 19 we learn to listen to the server: Monitoring, Logging & Deliverability — reading mail.log, monitoring the queue, and maintaining deliverability scores. See you in episode 19!