Learn Vault - Vault High Availability (HA) Cluster & Raft Storage

#Introduction

After covering Vault's integration with Terraform and Ansible in episode 19 — making Vault the source of truth when infrastructure is provisioned automatically — this episode marks the start of Phase 6: HA, Hardening, Observability & Production Readiness. And we begin this final phase from the most crucial foundation: building a Vault High Availability (HA) Cluster with Raft Integrated Storage.

You might think, "why bother making a cluster? Vault already runs on my single server." True — but imagine this scenario: your apps fetch secrets from Vault on every bootstrap, dynamic database credentials are opened through Vault, and internal TLS certificates are issued by Vault. Now that Vault server dies due to disk failure or routine maintenance. What happens? Every application that needs new secrets goes down with it. A single-node Vault is a single point of failure (SPOF) — one point that can take down the entire system.

This isn't theory. In the real world, Vault is one of the components that is most "relied upon but forgotten" until it dies. Like a heart: while it beats normally you don't notice it, but the moment it stops, the whole body stops. This episode teaches you how to make Vault stay alive even when one — or even two — of its nodes die, plus understanding the protocol that guarantees data consistency behind it: Raft.

#Main Discussion

#The Single-Node Problem and Why We Need HA

Before diving into configuration, let's clarify why a single node is never acceptable in production:

1. SPOF (Single Point of Failure) — if the node dies, every secret request fails.
2. Maintenance windows — upgrading Vault or patching the OS requires downtime; in production, Vault downtime = application downtime.
3. Limited scalability — all read and write requests are loaded onto one process.
4. Compliance — many standards (PCI-DSS, SOC 2, ISO 27001) require critical systems to have redundancy and clear recovery objectives.

The HA concept in Vault is simple: it doesn't mean several nodes serve all requests in parallel. The architecture is one Active node serving read and write requests, surrounded by one or more Standby nodes that replicate all data and are ready to take over when the Active node fails. This episode focuses on how that replication works and how to manage it correctly.

#Raft Integrated Storage: Vault's Built-in Distributed Storage

Since Vault version 1.4, you no longer need to provision separate external storage to build a cluster. Vault has Raft Integrated Storage — a distributed storage backend embedded directly in the Vault binary.

The main advantages of Raft integrated storage:

• No external dependencies — no need to operate a separate Consul cluster; one system to learn operationally, not two.
• Consistency — Raft guarantees data on all nodes is always identical (strongly consistent), unlike the eventual consistency model.
• Simpler operations — provisioning, upgrades, and backups only deal with one stack.
• Snapshots for DR — built-in backup mechanism (vault operator raft snapshot save) which we'll cover in episode 22.

#Vault HA Architecture: Active, Standby, and Consensus

In one Vault cluster, there's only one Active node. All write requests (writing secrets, creating tokens, issuing certificates) and read requests are served by this node. The Standby nodes are responsible for:

• Replicating data through the Raft log — every change the Active node receives is sent to followers as a replicated log entry.
• Redirecting requests — if an app accidentally calls a Standby node, Vault responds with an HTTP 307 redirect to the Active node's address. So from the client side, there's no need to know which node is active.
• Being the replacement — when the Active node dies, Raft performs leader election and one of the Standbys is elected as the new Active within seconds (usually 5–10 seconds depending on the network and configuration).

Leader election is done with the Raft Consensus protocol. Each node has a role: leader (active) or follower (standby). When the leader stops sending heartbeats, followers start an election. A decision is only made if a majority of nodes (quorum) agree — this is where the protection against split-brain lies.

#Practice: Building a 3-Node Vault Cluster with Raft

Now it's time to practice. We'll build a 3-node cluster with the IPs:

• vault-node-1 → 10.0.0.11
• vault-node-2 → 10.0.0.12
• vault-node-3 → 10.0.0.13

The core configuration of each node is nearly identical — what differs is only node_id, api_addr, and cluster_addr. Note the storage "raft" block:

storage "raft" {
  path   = "/opt/vault/data"
  node_id = "vault-node-1"
 
  retry_join {
    leader_api_addr = "https://10.0.0.11:8200"
  }
  retry_join {
    leader_api_addr = "https://10.0.0.12:8200"
  }
  retry_join {
    leader_api_addr = "https://10.0.0.13:8200"
  }
}
 
listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_disable   = false
  tls_cert_file = "/etc/vault.d/tls/vault.crt"
  tls_key_file  = "/etc/vault.d/tls/vault.key"
}
 
api_addr     = "https://10.0.0.11:8200"
cluster_addr = "https://10.0.0.11:8201"
 
ui  = true
log_level = "Info"

storage "raft" {
  path   = "/opt/vault/data"
  node_id = "vault-node-2"
 
  retry_join {
    leader_api_addr = "https://10.0.0.11:8200"
  }
  retry_join {
    leader_api_addr = "https://10.0.0.12:8200"
  }
  retry_join {
    leader_api_addr = "https://10.0.0.13:8200"
  }
}
 
listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_disable   = false
  tls_cert_file = "/etc/vault.d/tls/vault.crt"
  tls_key_file  = "/etc/vault.d/tls/vault.key"
}
 
api_addr     = "https://10.0.0.12:8200"
cluster_addr = "https://10.0.0.12:8201"
 
ui  = true
log_level = "Info"

storage "raft" {
  path   = "/opt/vault/data"
  node_id = "vault-node-3"
 
  retry_join {
    leader_api_addr = "https://10.0.0.11:8200"
  }
  retry_join {
    leader_api_addr = "https://10.0.0.12:8200"
  }
  retry_join {
    leader_api_addr = "https://10.0.0.13:8200"
  }
}
 
listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_disable   = false
  tls_cert_file = "/etc/vault.d/tls/vault.crt"
  tls_key_file  = "/etc/vault.d/tls/vault.key"
}
 
api_addr     = "https://10.0.0.13:8200"
cluster_addr = "https://10.0.0.13:8201"
 
ui  = true
log_level = "Info"

Important things about the configuration above:

• node_id must be unique on every node — it's the node's permanent identity in the cluster. If node_id is the same, Raft will reject the node joining.
• retry_join contains the addresses of all nodes that could become the leader. When a node boots and isn't yet part of the cluster, it keeps trying to "find" the cluster through these addresses.
• api_addr is the address clients see; cluster_addr is the address for inter-node Vault communication (port 8201). Both are required so failover and redirects work correctly.

#Step 1: Initialize the First Node

Run Vault on all three nodes, then initialize only the first node:

export VAULT_ADDR=https://10.0.0.11:8200
vault operator init

The result is Unseal Keys (Shamir) and the Initial Root Token. Store them safely — we're still using manual unseal in this episode (auto-unseal will be covered in episode 21). Then unseal the first node:

vault operator unseal "Unseal Key 1"
vault operator unseal "Unseal Key 2"
vault operator unseal "Unseal Key 3"

Once the threshold is met, node-1 becomes sealed=false. This node automatically becomes the cluster leader because no other node exists yet.

#Step 2: Join the Second and Third Nodes

Now node-2 and node-3 join the cluster. The vault operator raft join command needs the leader's address (or the api_addr of one of the existing nodes):

export VAULT_ADDR=https://10.0.0.12:8200
vault operator raft join https://10.0.0.11:8200

export VAULT_ADDR=https://10.0.0.13:8200
vault operator raft join https://10.0.0.11:8200

The successful output is roughly:

Key       Value
---       -----
Joined    true

Node-2 and node-3 also need to be unsealed with the same Unseal Keys (the cluster's master key is one — shared across all nodes):

vault operator unseal "Unseal Key 1"
vault operator unseal "Unseal Key 2"
vault operator unseal "Unseal Key 3"

#Step 3: Verify Status and Peers

From any node (with a valid token), we can check the cluster's health:

export VAULT_ADDR=https://10.0.0.11:8200
vault status

Key                Value
---                -----
Sealed             false
HA Enabled         true
HA Cluster         vault-cluster-abc123
HA Mode            active
Active Since       2026-08-02T08:15:00.000Z

Note the lines HA Enabled true and HA Mode active — these are the signs that the node is serving requests. Now look at the cluster map with vault operator raft list-peers:

vault operator raft list-peers

Node          Address              State     Voter
----          -------              -----     -----
vault-node-1  10.0.0.11:8201       leader    true
vault-node-2  10.0.0.12:8201       follower  true
vault-node-3  10.0.0.13:8201       follower  true

All nodes show voter as true — meaning they participate in consensus decisions. Node-1 is the leader, the other two are followers.

#Quorum and Failure Tolerance

The key to understanding Raft is the quorum concept: the majority number of nodes that must agree before the cluster makes a decision. The formula: quorum = floor(n/2) + 1. For 3 nodes, quorum = 2. That means at least 2 nodes must be alive for the cluster to keep working.

From the table above you can see why 3 nodes is the minimum viable production configuration, and 5 nodes for highly critical environments. With 3 nodes, one node failure is safe; two node failures mean losing quorum. With 5 nodes, two node failures are still safe.

#Common Pitfalls

Real-world Vault HA operational experience is full of traps. Here are the most common:

One thing worth emphasizing: never just delete the /opt/vault/data folder on a failed node to "reset" it. If that node is still counted as a voter, the cluster still considers it part of the quorum and this can kill the cluster entirely. Use the official command to deactivate the node from the cluster's perspective before taking physical action.

#Conclusion

In this episode 20 we understood why a single-node Vault is unsuitable for production, how Raft Integrated Storage provides distributed storage without external dependencies, how the Active-Standby architecture works, and how to build a 3-node cluster step by step: the storage "raft" configuration, vault operator raft join, initialization, unseal, and verification with vault operator raft list-peers. We also learned the quorum table — 3 nodes for standard production, 5 nodes for critical environments — plus a list of common mistakes that often destroy clusters in the real world.

One thing you definitely noticed in practice: after every server restart, you must manually unseal with the Unseal Keys. In production, this is an operational nightmare — who guarantees an operator is awake at 3 AM when the maintenance window ends? The good news is there's an elegant solution: auto-unseal using Cloud KMS, which we'll cover thoroughly in episode 21. With auto-unseal, Vault unseals itself automatically at boot — without human intervention. Keep your enthusiasm up, we still have five exciting final episodes!