Learn Jenkins - High Availability (HA), Backup & Disaster Recovery

#Introduction

In episode 17 we built notifications that keep builds "talking". Now we handle the most unpleasant but most critical thing: what happens if the Jenkins controller dies? Job configuration, credentials, and build history stored in JENKINS_HOME can vanish in an instant. Without a backup, all of it must be reconfigured manually — taking hours and risking the loss of critical configuration.

In this episode we cover three things:

1. Backup strategies for the JENKINS_HOME directory with ThinBackup or filesystem snapshots.
2. A from-scratch disaster recovery (DR) procedure using JCasC plus backup data.
3. Monitoring Jenkins infrastructure health with Prometheus and Grafana.

#Why JENKINS_HOME Is a Treasure

All of Jenkins's important data lives in a single JENKINS_HOME directory:

• Job and pipeline definitions.
• Global configuration (credentials, security, plugin settings).
• Build history and artifacts.
• Credentials — although stored encrypted with Jenkins's special keys.

Losing this directory is the same as losing the team's entire automation. That is why backup is not optional but mandatory.

#Backup Strategy with the ThinBackup Plugin

The ThinBackup plugin periodically backs up Jenkins configuration and jobs to a separate directory. Some of its features: scheduled automatic backups, backup rotation, and direct restore from the UI.

Configuration steps in Manage Jenkins then ThinBackup:

1. Fill in the Backup directory — preferably on a separate disk or a mount attached to external storage (e.g. NFS or a cloud volume).
2. Enable the Backup schedule using Jenkins cron, e.g. H 2 * * * for 2 AM.
3. Set the Max number of backup copies for rotation so the disk does not fill up.
4. Check the plugin list backup option so the plugin list is stored too.

For extra assurance, combine it with filesystem snapshots. In the cloud, snapshot the controller instance's volume, for example an EBS snapshot or a VM disk image. On bare-metal servers, use LVM snapshots or periodic rsync.

#Disaster Recovery Plan

A DR plan is a written procedure for recovering the controller from scratch. Since Jenkins is already declaratively defined via JCasC (episode 13), recovery becomes very fast. The sequence:

1. Prepare the infrastructure — run a fresh jenkins/jenkins:lts-jdk17 image with the same JCasC from the configuration repository.
2. Install plugins — follow the backed-up plugin list (plugins.txt file) so plugin versions match.
3. Restore data — copy the JENKINS_HOME backup contents (jobs, config, credentials) into the new controller directory.
4. Restart — the controller reads the configuration and credentials from the restored data.
5. Verify — check that jobs are available, credentials are valid, and agents are connected.

A quick command for manual restore:

sudo systemctl stop jenkins
sudo rm -rf /var/lib/jenkins
sudo tar -xzf jenkins-backup-20260803.tar.gz -C /var/lib/jenkins
sudo chown -R jenkins:jenkins /var/lib/jenkins
sudo systemctl start jenkins

#Monitoring with the Prometheus Metrics Plugin

Backups help during a disaster, but problems usually appear gradually: executors running out of slots, queues piling up, JVM memory leaks. The Prometheus Metrics plugin exposes controller metrics at the /prometheus endpoint, which is then scraped by Prometheus.

The scrape configuration in the prometheus.yml file:

scrape_configs:
  - job_name: 'jenkins-controller'
    metrics_path: '/prometheus'
    scheme: https
    static_configs:
      - targets:
          - 'jenkins.example.com'
    basic_auth:
      username: 'metrics-user'
      password: 'metrics-password'
    tls_config:
      insecure_skip_verify: true

#Key Metrics to Monitor

These metrics are visualized in Grafana using the community Jenkins dashboard. After adding the Prometheus datasource in Grafana, the dashboard displays queue size, executor usage, and JVM memory graphs all at once, complete with alert rules.

An example alert rule for a piling-up queue:

groups:
  - name: jenkins.rules
    rules:
      - alert: JenkinsQueueHigh
        expr: jenkins_queue_size_value > 20
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: Antrian build menumpuk di Jenkins

With the combination of queue size, executor usage, and JVM memory, we can decide when to add agents or raise the JVM heap — before the controller actually stalls.

#Conclusion

In this episode we covered HA, backup, and DR:

• JENKINS_HOME stores all configuration assets, jobs, and credentials, so it must be backed up.
• ThinBackup schedules automatic backups, and filesystem snapshots add a protection layer.
• From-scratch DR uses JCasC plus backup data restore, with periodic testing.
• The Prometheus Metrics plugin exposes queue size, executor usage, and JVM memory, visualized in Grafana.

A healthy controller is the foundation of a reliable pipeline. In episode 19 we go into detective mode: troubleshooting and debugging pipelines — from log analysis, the Replay feature, and the Snippet Generator, to handling OOM and disconnected agents. See you there!