Learn Remote Desktop - VNC Encryption (TLS/SSL & VNC Encryption)

#Introduction

Episode 12 strengthened authentication, but there's a bigger hole: VNC without encryption sends all data as plaintext. Your screen, keyboard input, and the authentication handshake travel across the network in a form readable by anyone sniffing. On public networks or open WiFi, this is like sending your screen contents to anyone.

Episode 13 closes that hole. You'll see the proof of sniffing yourself with tcpdump, enable TLS/SSL encryption in x11vnc and TigerVNC, and understand the SSH tunnel's position as the final encryption layer, which will be covered in full in episode 14.

#The VNC Plaintext Security Problem

#What Leaks Without Encryption

When a viewer connects to a VNC server without encryption, all RFB communication crosses the network raw. The leaked data includes:

• Authentication handshake: the DES challenge-response that can be captured and brute-forced offline.
• Screen framebuffer: every pixel the server sends to the viewer can be intercepted.
• Keyboard and mouse input: every keystroke and click you make is readable.

This is why raw VNC should never be exposed to the internet. Even on internal local networks, sniffing habits still matter — especially in environments with many users.

#Proving Sniffing with tcpdump

The most convincing way is to see it yourself. Run tcpdump on one machine, open a VNC session, and observe the data passing through:

sudo tcpdump -i eth0 -X -s 0 port 5900

The sudo tcpdump -i eth0 -X -s 0 port 5900 command captures packets on port 5900 and displays them in hex and ASCII. If you watch closely, the RFB protocol header is clearly visible as readable text — proof that there's no encryption at this layer.

#Encrypting x11vnc with -ssl

#Creating a Certificate and Enabling SSL

x11vnc supports built-in TLS/SSL encryption. With the -sslgen option, x11vnc automatically creates a self-signed certificate and uses it directly:

x11vnc -display :0 -ssl -sslgen -rfbauth /etc/x11vnc.pass

The x11vnc -display :0 -ssl -sslgen -rfbauth /etc/x11vnc.pass command runs the server with TLS-encrypted connections. For a certificate from an official CA — one that doesn't trigger warnings in viewers — create a certificate with openssl and point to it with -sslcert:

openssl req -x509 -newkey rsa:2048 -keyout /etc/x11vnc-key.pem -out /etc/x11vnc-cert.pem -days 365 -nodes
x11vnc -display :0 -ssl /etc/x11vnc-cert.pem -rfbauth /etc/x11vnc.pass

#Connecting from a TLS-Supporting Viewer

The TigerVNC viewer can connect to a TLS server by selecting the appropriate security type:

vncviewer -SecurityTypes TLSVnc,TLSNone 192.168.1.20:5900

#Encrypting TigerVNC with TLS

#Choosing the Security Type

TigerVNC (especially 1.16 and later) builds its TLS support on GnuTLS. The server picks the mechanism via the -SecurityTypes option, and the viewer must agree on the same mechanism:

Xvnc :1 -geometry 1920x1080 -depth 24 -SecurityTypes VncAuth,TLSVnc -passwordfile ~/.vnc/passwd

The Xvnc :1 -SecurityTypes VncAuth,TLSVnc -passwordfile ~/.vnc/passwd command creates display :1 that accepts VNC connections with password authentication encrypted by TLS. On the viewer side:

vncviewer -SecurityTypes TLSVnc localhost:5901

If the server and viewer use mismatched security types, the connection fails with a message showing the list of mechanisms each side supports.

#Checking Build Support

Not all TigerVNC builds include TLS. Verify support with this command:

vncviewer --help | grep -i -E "security|tls"

An output showing the -SecurityTypes option means your viewer build supports TLS. If it's absent, upgrade TigerVNC to 1.16 or use an SSH tunnel instead.

#The SSH Tunnel as an Encryption Layer

#Full Encryption Without Server Configuration

The most universal and reliable way is to wrap VNC in an SSH tunnel. Its advantages: no server configuration changes at all, encryption is handled by battle-tested OpenSSH, and the server can stay bound to localhost.

ssh -L 5901:localhost:5900 arman@192.168.1.20

Once the tunnel is active, the viewer on the local machine just connects to localhost:5901 and all VNC traffic passes through the encrypted SSH connection:

vncviewer localhost:5901

This is the pattern most recommended for accessing VNC from outside networks, and episode 14 will cover it thoroughly — including jump hosts and reverse tunnels.

#Conclusion

Episode 13 closed the VNC plaintext weakness: you saw the proof of sniffing with tcpdump, enabled TLS/SSL in x11vnc with -sslgen, used the TLS security type in TigerVNC, and recognized the SSH tunnel as the most universal encryption layer.

Key takeaways:

• VNC without encryption sends screen, input, and handshake as plaintext.
• tcpdump -X port 5900 proves the RFB header can be read directly.
• x11vnc uses -ssl and -sslgen for built-in TLS/SSL.
• TigerVNC uses -SecurityTypes VncAuth,TLSVnc for TLS.
• Viewer and server must agree on the same security type.
• An SSH tunnel is the most universal encryption layer without changing the server.

In the next episode, episode 14, we dissect SSH tunneling and secure remote access — local forwarding for VNC and RDP, two-way tunnels for firewall bypass, jump hosts with ssh -J, and reverse tunnels so servers behind NAT can be accessed from outside. See you there!