Learn Remote Desktop - VNC Encryption (TLS/SSL & VNC Encryption)

#Introduction

Episode 12 strengthened authentication, but there's an even bigger hole: VNC without encryption sends all data as plaintext. Your screen, keyboard input, and the authentication handshake cross the network in a form anyone sniffing can read. On public networks or open WiFi, that's the same as sending the contents of your screen to anyone.

Episode 13 closes that hole. You'll see the interception proof yourself with tcpdump, enable TLS/SSL encryption in x11vnc and TigerVNC, and understand where SSH tunneling sits as the final encryption layer, covered in full in episode 14.

#The VNC Plaintext Security Problem

#What Leaks Without Encryption

When a viewer connects to a VNC server without encryption, all RFB communication travels raw on the network. The leaked data includes:

• Authentication handshake: the DES challenge-response can be captured and brute-forced offline.
• Screen framebuffer: every pixel the server sends to the viewer can be intercepted.
• Keyboard and mouse input: every keystroke and click you make can be read.

This is why raw VNC should never be exposed to the internet. Even on internal local networks, sniffing habits still matter — especially in environments with many users.

#Proving Interception with tcpdump

The most convincing way is to see it yourself. Run tcpdump on one machine, open a VNC session, and observe the traffic passing through:

sudo tcpdump -i eth0 -X -s 0 port 5900

The sudo tcpdump -i eth0 -X -s 0 port 5900 command captures packets on port 5900 and displays them in hex and ASCII. If you pay attention, the RFB protocol header appears clearly as readable text — proof that there's no encryption at this layer.

#Encrypting x11vnc with -ssl

#Creating a Certificate and Enabling SSL

x11vnc supports built-in TLS/SSL encryption. With the -sslgen option, x11vnc generates a self-signed certificate automatically and uses it right away:

x11vnc -display :0 -ssl -sslgen -rfbauth /etc/x11vnc.pass

The x11vnc -display :0 -ssl -sslgen -rfbauth /etc/x11vnc.pass command runs the server with TLS-encrypted connections. For a certificate from an official CA — one that doesn't trigger warnings in the viewer — generate a certificate with openssl and point to it with -sslcert:

openssl req -x509 -newkey rsa:2048 -keyout /etc/x11vnc-key.pem -out /etc/x11vnc-cert.pem -days 365 -nodes
x11vnc -display :0 -ssl /etc/x11vnc-cert.pem -rfbauth /etc/x11vnc.pass

#Connecting from a TLS-Capable Viewer

The TigerVNC viewer can connect to a TLS server by choosing the appropriate security type:

vncviewer -SecurityTypes TLSVnc,TLSNone 192.168.1.20:5900

#Encrypting TigerVNC with TLS

#Choosing the Security Type

TigerVNC (1.16 and above in particular) builds TLS support on GnuTLS. The server chooses the mechanism via the -SecurityTypes option, and the viewer must agree on the same mechanism:

Xvnc :1 -geometry 1920x1080 -depth 24 -SecurityTypes VncAuth,TLSVnc -passwordfile ~/.vnc/passwd

The Xvnc :1 -SecurityTypes VncAuth,TLSVnc -passwordfile ~/.vnc/passwd command creates display :1, which accepts VNC connections with password authentication encrypted over TLS. On the viewer side:

vncviewer -SecurityTypes TLSVnc localhost:5901

If server and viewer use mismatched security types, the connection fails with a message showing the list of mechanisms each side supports.

#Checking Build Support

Not all TigerVNC builds include TLS. Verify support with the following command:

vncviewer --help | grep -i -E "security|tls"

Output that shows the -SecurityTypes option means the viewer build supports TLS. If it doesn't, upgrade TigerVNC to 1.16 or use an SSH tunnel instead.

#SSH Tunnel as an Encryption Layer

#Full Encryption Without Server Configuration

The most universal and reliable way is wrapping VNC in an SSH tunnel. Its advantages: no server configuration changes at all, encryption is handled by battle-tested OpenSSH, and the server can stay bound to localhost.

ssh -L 5901:localhost:5900 arman@192.168.1.20

Once the tunnel is active, the viewer on the local machine simply connects to localhost:5901, and all VNC traffic travels through the encrypted SSH connection:

vncviewer localhost:5901

This is the pattern most recommended for accessing VNC from outside networks, and episode 14 will cover it thoroughly — including jump hosts and reverse tunnels.

#Conclusion

Episode 13 closes the VNC plaintext weakness: you saw the interception proof with tcpdump, enabled TLS/SSL in x11vnc with -sslgen, used TLS security types in TigerVNC, and recognized SSH tunneling as the most universal encryption layer.

Key takeaways:

• VNC without encryption sends the screen, input, and handshake as plaintext.
• tcpdump -X port 5900 proves the RFB header can be read directly.
• x11vnc uses -ssl and -sslgen for built-in TLS/SSL.
• TigerVNC uses -SecurityTypes VncAuth,TLSVnc for TLS.
• Viewer and server must agree on the same security type.
• SSH tunneling is the most universal encryption layer without changing the server.

In the next episode, episode 14, we dissect SSH tunneling and secure remote access — local forwarding for VNC and RDP, two-way tunneling to bypass firewalls, jump hosts with ssh -J, and reverse tunneling so servers behind NAT can be reached from outside. See you there!