Learn Linux - High Availability & Monitoring Stack

#Introduction

After episode 28 where we built centralized authentication with LDAP and SSSD — one identity directory serving dozens of servers — you now have centralized access control. But there's a question as important as "who can get in": how do we know all these servers are still alive and healthy? A server can die without sending any notification — a failed power supply, kernel panic, or a full disk — and you only notice when a user reports "the site can't be accessed".

In episode 29 we enter two operational pillars that make infrastructure truly professional: High Availability (HA) and Monitoring. We'll learn how to keep services alive even when one server dies (Keepalived + virtual IP), how to measure all servers' health centrally (Prometheus + node_exporter + Grafana), and how to receive automatic alerts before a problem becomes a disaster (Alertmanager). At the end of the episode, you'll build a real monitoring stack across two VMs — the foundation of reliable operations.

#Main Discussion

#The High Availability Concept: Redundancy and Failover

The first principle to understand: high availability doesn't mean "never down" — it means "downtime impact is minimized with automation". There's no failure-free system; what exists are systems that design for failure and provide automatic substitutes.

Think of a passenger plane. The plane isn't designed so its engines never fail — it's designed so one engine can fail without bringing the plane down. There are two engines (redundancy), and if one dies, the other keeps the plane flying (failover). The pilot doesn't panic; the system compensates automatically.

In the server world, redundancy means running two or more nodes providing the same service. Failover means the automatic mechanism moving "leadership" from the dead node to a healthy one. The most important component in failover is detection: the replacement node must know quickly that the primary node is no longer healthy.

The most popular tool for two-node active/standby scenarios on Linux is Keepalived. It uses the VRRP protocol (Virtual Router Redundancy Protocol) to create a Virtual IP (VIP) that's automatically used by the healthy node. From the outside, clients only see one IP — never knowing there are two servers behind it. When the primary node dies, the standby node takes over the VIP within seconds without human intervention.

For more complex clusters (multi-node, movable resources like services, filesystems, and IPs), there's Pacemaker + Corosync — a cluster resource manager that's far more powerful but also far more complex. Keepalived is enough for most active/standby needs; Pacemaker for enterprise-scale clusters.

#Setting Up Keepalived with a Virtual IP

We'll create two nodes — web-01 (IP 192.168.1.11) and web-02 (IP 192.168.1.12) — sharing one VIP 192.168.1.10. Both nodes run nginx. When web-01 is healthy, it holds the VIP; if it dies, web-02 takes over.

apt install -y keepalived

Config on the primary node web-01:

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 200
    advert_int 1
    virtual_ipaddress {
        192.168.1.10/24 dev eth0
    }
    track_script {
        chk_nginx
    }
}

And on the standby node web-02, almost identical except state BACKUP and priority 150:

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 150
    advert_int 1
    virtual_ipaddress {
        192.168.1.10/24 dev eth0
    }
    track_script {
        chk_nginx
    }
}

So that failover really only happens if the service (not just the OS) has a problem, we define a track_script checking nginx. This is a very important pattern: if nginx dies but the server stays alive, the VIP should still move to the other node — because users still can't access the service.

#!/usr/bin/env bash
if ! pidof nginx > /dev/null; then
    exit 1
fi
exit 0

systemctl restart keepalived
ip addr show eth0 | grep 192.168.1.10   # on web-01: the VIP shows up

#Monitoring Stack: node_exporter, Prometheus, and Grafana

HA answers "how services stay alive"; monitoring answers "how we know it's healthy". The most popular stack in the modern Linux/DevOps ecosystem is Prometheus + node_exporter + Grafana, and its architecture differs from traditional monitoring: a pull model.

• node_exporter — a lightweight agent installed on every server, exposing OS metrics (CPU, RAM, disk, network) in HTTP text format on port 9100.
• Prometheus — the metrics server that periodically pulls data from exporters, stores time-series, and evaluates alert rules.
• Grafana — the visualization platform reading data from Prometheus to build beautiful, interactive dashboards.
• Alertmanager — the Prometheus component receiving alerts and forwarding them to channels (email, Slack, Telegram, webhook).

+-----------+   scrape /metrics   +---------------+   query    +----------+
| Server 1  | <-----------------> |  Prometheus   | <--------> | Grafana  |
| node_exp  |                     |  (time-series)|            | (visual) |
+-----------+                     +-------+-------+            +----------+
+-----------+   scrape /metrics   |       |
| Server 2  | <-----------------> |  Alertmanager  | --> email/Slack
| node_exp  |                     +----------------+
+-----------+

#Step 1: Install node_exporter on all nodes

curl -LO https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar xzf node_exporter-1.8.2.linux-amd64.tar.gz
install -m 755 node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/
useradd -rs /bin/false node_exporter
install -m 644 /dev/stdin /etc/systemd/system/node_exporter.service <<'EOF'
[Unit]
Description=Node Exporter
After=network.target
 
[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter
 
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && systemctl enable --now node_exporter

Verify the metrics with curl http://localhost:9100/metrics | head — you'll see thousands of metric lines like node_cpu_seconds_total, node_memory_MemAvailable_bytes, and so on.

#Step 2: Configure Prometheus

Prometheus is configured via prometheus.yml. The most important part is scrape_configs — the list of targets whose metrics must be pulled:

scrape_configs:
  - job_name: nodes
    static_configs:
      - targets:
          - web-01:9100
          - web-02:9100
    scrape_interval: 15s

Name the job_name line according to the node's role (e.g. web, db, monitoring) so metrics are easy to group in Grafana. Down targets will show on the Prometheus Targets page with a red status.

#Step 3: Create a dashboard in Grafana

Grafana connects to Prometheus as a data source, then dashboards can be created manually or imported from the community. The most popular node-exporter dashboard is ID 1860 on grafana.com:

# Open http://<grafana-host>:3000 (default login admin/admin)
# Add data source → Prometheus → URL: http://<prometheus-host>:9090
# Dashboard → Import → ID 1860 → select the Prometheus data source

#Step 4: Alertmanager — notify before disaster

Dashboards are only useful if someone is looking at them. For healthy operations, you need automatic alerts. In Prometheus, alerts are defined with PromQL rules. Example: alert when CPU idle is nearly exhausted or the disk is full:

groups:
  - name: node-alerts
    rules:
      - alert: HighCPU
        expr: 100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) > 90
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High CPU on {{ $labels.instance }}"
      - alert: DiskSpaceLow
        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Disk / down to 10% on {{ $labels.instance }}"

Alertmanager receives alerts from Prometheus and groups them before sending to channels. The email/Slack receiver config is in alertmanager.yml:

route:
  group_by: [alertname, instance]
  receiver: ops-team
receivers:
  - name: ops-team
    email_configs:
      - to: ops@example.com
        from: alerts@example.com
        smarthost: smtp.example.com:587

#Uptime and Health Checks: systemd, monit, nagios

Beyond the Prometheus stack, there are several health check layers complementing the oversight:

• systemd — besides managing services, systemd also monitors them. Restart=on-failure makes a service restart automatically on crash, and WatchdogSec reports if a service stops responding. This is the free first line of defense.
• monit — a lightweight config-based monitoring tool for processes, filesystems, and hosts; suited for small, simple setups.
• Nagios/Icinga — traditional agent + active check monitoring still widely used by legacy organizations.

A healthy pattern combines all three: systemd for automatic recovery (restart), Prometheus for historical metrics & alerting, and external checks (like UptimeRobot or monit) to make sure services are reachable from outside — because a dead Prometheus can't report that it's dead.

[Service]
ExecStart=/usr/local/bin/myapp
Restart=on-failure
RestartSec=5
WatchdogSec=30
StartLimitIntervalSec=60
StartLimitBurst=5

#Practice: Monitoring Stack on Two VMs

Time to put it all together. Scenario: two VMs web-01 and web-02 (also the HA nodes from the previous section), plus one VM mon-01 running Prometheus, Grafana, and Alertmanager.

web-01 (nginx + node_exporter)  ----+----> mon-01 (Prometheus + Grafana + Alertmanager)
web-02 (nginx + node_exporter)  ----+---->         scrapes port 9100 every 15s
web-01 & web-02 = Keepalived VIP 192.168.1.10 (failover)

The workflow:

1. Install node_exporter on web-01 and web-02 (see Step 1 above).
2. Install Prometheus on mon-01 with scrape_configs pointing to both nodes.
3. Install Grafana on mon-01, add Prometheus as a data source, import dashboard 1860.
4. Install Alertmanager and add alert.rules.yml to Prometheus.
5. Kill one node and observe: Prometheus will mark the target down, Grafana shows empty data, and Alertmanager sends an alert.

curl http://mon-01:9090/api/v1/targets | head -20

{
  "status": "success",
  "data": {
    "activeTargets": [
      {"labels": {"instance": "web-01:9100"}, "health": "up"},
      {"labels": {"instance": "web-02:9100"}, "health": "up"}
    ]
  }
}

#Common Pitfalls

Here are the failure patterns most commonly found when building HA & monitoring:

The firewall case deserves to be an example. It's the most common cause of a down Prometheus target — not because the node is dead, but because port 9100 isn't allowed in from the Prometheus server:

# target stays down, even though curl localhost:9100 works
ufw allow from 192.168.1.20 to any port 9100 proto tcp
systemctl reload ufw

#Conclusion

In this episode we built two operational pillars that make infrastructure truly production-ready: High Availability with Keepalived and virtual IPs — services stay alive when one node dies; and the monitoring stack with node_exporter, Prometheus, Grafana, and Alertmanager — all servers measured and problems detected before they become disasters. You also understood the layered health check pattern (systemd for automatic recovery, Prometheus for observability, external checks for availability) and the common mistakes to avoid.

Almost done. You now have: hardened servers, tested backups, running containers and virtualization, centralized authentication, plus HA and monitoring. All those components are now ready to be assembled into one whole system. In episode 30 — the final episode of the Learn Linux series — we'll build a complete production-grade Linux server setup: assembling all the lessons from episode 0 to 29 into one comprehensive case study, complete with a production readiness checklist and a summary of this long journey. See you in the final episode!