Learn Linux - Backup & Restore Strategy

#Introduction

After episode 25 where we covered Linux system security hardening — SSH key authentication, firewall, fail2ban, all the way to locking down access with strict users and sudo — your system is now much harder to breach. But there's one reality no admin, no matter how great, can avoid: no matter how well you defend, eventually you must face data loss. An SSD that dies without warning, files deleted by a wrong-target rm -rf, a database corrupted by a power outage in the middle of a transaction, or even a server held hostage by ransomware — none of that is fiction.

The question is no longer whether it will happen, but how fast you can recover. In this episode we cover the last line of defense, and the one most often neglected: backup & restore strategy. We'll break down the 3-2-1 principle, compare tools like rsync, tar, restic, and borgbackup, practice backing up databases and file servers, then close with restore procedures and drills that let you sleep well even when the server misbehaves.

#Main Discussion

#Why Backup Is Not an Option: The 3-2-1 Principle

Many beginner admins treat backup as a "when I have time" task. That thinking is dangerous. Consider your server as a cabinet full of diplomas, certificates, and important contracts. You can guard that cabinet with the strongest lock and the best alarm — but if the house burns down, everything is still lost. The sensible solution isn't just locking the cabinet, but keeping copies of important documents in another location.

In the backup world, that solution is formulated into a very famous principle: the 3-2-1 rule.

• 3 copies of data — one active production data, one primary backup, one additional backup. Three copies total of the same data.
• 2 different media — e.g. one on an internal/external hard disk and another on tape, NAS, cloud, or another server. Why different media? Because failures often come from a single point: the same hard disk can fail, and the same device can be stolen together.
• 1 copy offsite — one copy outside the primary location. If the building burns down, floods, or is hit by ransomware infecting every local machine, the offsite copy is the last savior.

+-----------------------+      +----------------------+      +------------------+
|  Production Data      |  --> |  Primary Backup      |  --> |  Offsite Backup  |
|  (main server)        |      |  (NAS/ext. disk)     |      |  (cloud/other DC)|
+-----------------------+      +----------------------+      +------------------+
  1 copy (active)          2 copies, media #1         3 copies, media #2
                                                          = 1 separate location

Now you understand why we back up. Next, how to do it — and tool choice greatly determines the quality of your backups.

#Choosing the Right Tool

Linux gives you a wide spectrum of tools, from simple to advanced. The key to choosing isn't "which tool is the coolest", but which tool fits your recovery needs. Every tool has trade-offs between simplicity, speed, storage space, and ease of restore.

#rsync: Flexible Differential Synchronization

rsync was already covered in the file sharing episode, and in the backup world it's the main workhorse. Its strength lies in incremental transfer: rsync only sends changed data, so subsequent backups are very fast and bandwidth-efficient. It also supports hardlink-based deduplication when combined with a timestamped directory structure.

rsync -avz --delete \
  /var/www/ \
  backup@nas.local:/backups/www/latest/

The pattern above is a baseline. To get daily snapshots without eating extra space, many admins use the rsync --link-dest trick: every day we back up into a dated directory, but unchanged files are just hardlinks to yesterday's backup. The result: you have 30 daily versions that effectively take only the size of one full backup.

TODAY=$(date +%F)
rsync -a --delete --link-dest=../latest \
  /var/www/ backup@nas.local:/backups/www/$TODAY/

#tar: Portable Static Archives

tar is the oldest archiving tool still alive today. Its strength: one self-contained and easily movable archive file — great for config snapshots, project archives, or server migrations. Its weakness: every full backup re-reads all data, so it's less efficient for data that changes fast in large volumes.

tar -czf /backups/etc-$(date +%F).tar.gz /etc

For rarely changing files like /etc, tar is a very reasonable choice. The combination of tar for configs and rsync for dynamic data is a healthy, common pattern in many organizations.

#restic & borgbackup: Integrated Deduplication and Encryption

For more serious scale, modern tools like restic and borgbackup offer features that used to exist only in paid enterprise solutions: data deduplication (identical data blocks are stored only once, even across different files), end-to-end encryption, and time-based snapshots.

• restic — written in Go, a single binary with no dependencies, supports local and remote repositories (SFTP, S3, Backblaze, and more). Very popular for being simple and modern.
• borgbackup — Python, with very efficient compression and deduplication, features mountable archives so you can mount a backup like a normal directory for inspection.

# 1. Initialize repository + password
restic init --repo /backups/restic-repo
 
# 2. Back up important directories
restic -r /backups/restic-repo backup /etc /home /var/www
 
# 3. List snapshots
restic -r /backups/restic-repo snapshots

Note the first line: restic immediately asks for a password to encrypt the repository. All data entering the repo is already encrypted before it leaves the server — so sending it to the cloud is not a big deal, because its contents can't be read by anyone without the password. This is why restic/borg are the top choice for offsite backups.

#Automation and Periodic Testing

A manually run backup is a backup that one day won't be run. Automation via cron or a systemd timer is a must, not a luxury. The most common pattern:

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin
 
# Backup database & file server every day at 02:30
30 2 * * * root /usr/local/sbin/backup-server.sh >> /var/log/backup.log 2>&1

One non-negotiable rule: backups must be periodically restore-tested. A backup never tested for restore is worth the same as having no backup at all — maybe even more dangerous, because it gives a false sense of security. Test restore at least once a month, and record the results.

#Backing Up Critical Databases and File Servers

Backing up files like /etc and /home is relatively simple — just copy them. Databases are a different story: copying raw data files (e.g. *.frm, *.ibd, *.pgdata) while the database is running almost certainly yields a corrupted backup. Databases write to disk in buffers and inconsistent transactions; copying in the middle of that is like photographing a book someone is copying while the pages keep turning.

The solution is using the database's built-in dump tool, which produces a consistent logical snapshot:

mysqldump --single-transaction --quick --routines \
  -u backup -p myapp_db | gzip > /backups/db/myapp-$(date +%F).sql.gz

pg_dump -U backup -Fc myapp_db > /backups/db/myapp-$(date +%F).dump

A few things to note:

• --single-transaction on mysqldump produces a consistent snapshot without locking tables for a long time.
• The -Fc (custom) format on pg_dump produces a compressed file that can be restored selectively per table.
• Databases on production servers should ideally be backed up via a replica or with an adequate buffer pool so the dump doesn't burden the primary instance.

#Restore Procedures and Integrity Verification

Restore is the side of backup that's never popular until needed — and that's when all weaknesses surface. A good procedure always runs in a clear order that anyone can practice, including someone newly joined:

1. Assess the scope of damage — which files are lost? Which database is corrupted? This determines which backup module must be restored.
2. Isolate — if the main server is damaged, don't overwrite directly; restore to a temporary environment first and verify.
3. Restore — run the restore commands for the tool being used.
4. Verify integrity — check checksums, count table rows, make sure the application can run.
5. Record and audit — document the time, tool, and results in the runbook.

For tar archives, verification can be done directly with this command:

tar -tvzf /backups/etc-2026-08-01.tar.gz | head -20

For databases, the strongest test is restoring into an empty environment then comparing row counts:

createdb myapp_restore
pg_restore -d myapp_restore /backups/db/myapp-2026-08-01.dump
psql -d myapp_restore -c "SELECT count(*) FROM orders;"

mysql -u root -p myapp_db < /backups/db/myapp-2026-08-01.sql.gz

#Practice: A restic Backup Script and Restore Drill

Time to combine everything into a single script that can be scheduled. The following script backs up critical files and the database, adds a restic snapshot, then cleans up old snapshots with a retention policy:

#!/usr/bin/env bash
set -euo pipefail
 
REPO="/backups/restic-repo"
export RESTIC_PASSWORD_FILE="/etc/restic/passphrase"
 
log() { echo "[$(date +%F_%T)] $*"; }
 
log "Backing up the database..."
mysqldump --single-transaction -u backup myapp_db \
  | gzip > /var/tmp/myapp-$(date +%F).sql.gz
 
log "Backing up the file server with restic..."
restic -r "$REPO" backup /etc /home /var/www /var/tmp/myapp-*.sql.gz
 
log "Cleaning up old snapshots (retention 7/4/6)..."
restic -r "$REPO" forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
 
log "Done."

Explanation of the important lines:

• set -euo pipefail makes the script stop and return a failure status if any single step errors — this is the basis of the alerting discussed above.
• The restic password is stored in the /etc/restic/passphrase file with 600 permissions, not inside the script. Your version of the script should pull from a secret manager, not be committed to Git.
• forget with --keep-daily/--keep-weekly/--keep-monthly is a common retention policy: keep 7 daily snapshots, 4 weekly, and 6 monthly, then delete the rest with --prune.

After the script runs, do a restore drill. A good drill simulates a real disaster: restoring a deleted /etc file, and restoring the database into an empty environment:

SNAP=$(restic -r /backups/restic-repo snapshots --latest -q)
restic -r /backups/restic-repo restore "$SNAP" \
  --target /mnt/restore --include /etc
ls -la /mnt/restore/etc/ssh/

SNAP=$(restic -r /backups/restic-repo snapshots --latest -q)
mkdir -p /mnt/restore && \
  restic -r /backups/restic-repo restore "$SNAP" --target /mnt/restore
gzip -dc /mnt/restore/var/tmp/myapp-*.sql.gz | mysql -u root myapp_db

#Common Pitfalls

After decades of backup practice, the same failure patterns keep recurring. Recognize and avoid them:

The rsync --delete case deserves special attention. If you run rsync -av --delete /source /destination and the source turns out empty or badly mounted, rsync dutifully deletes every file on the destination. This is one of the fastest ways to destroy a backup. Guard against it with --dry-run first, or use a tool with a snapshot model like restic, which never deletes old data without an explicit policy:

rsync -av --delete /var/www/ /backups/www/
rsync -av --dry-run /var/www/ /backups/www/
# Always review the dry-run output before running the real version

#Conclusion

In this episode we built a server's last line of defense: the backup & restore strategy. We understood the 3-2-1 principle as the foundation, chose tools based on restore needs — rsync for incremental synchronization, tar for static archives, and restic/borgbackup for deduplication and encryption. We also covered how to back up databases correctly via mysqldump/pg_dump, automation with cron, ordered restore procedures, and measured drills. Most importantly, we learned that a backup is not a product — it's a process that must be tested, measured, and maintained continuously.

With a solid backup, you're ready to face the worst-case scenario. But backup is only half the story; the other half is how services are run and scaled. In the next episode 27, we'll cover containerization & virtualization — Docker for application containers, LXD for system containers, and KVM for full virtual machines. You'll see how the concepts we've learned from episode 0 to 26 converge into the modern way of running services: lightweight, isolated, and easily movable. See you in episode 27!