Learn Vault - Automated Auto-Unseal Strategy in Cloud Environments

#Introduction

After building the Vault HA Cluster with Raft Integrated Storage in episode 20 — 3 nodes that replicate data to each other and automatically elect a new leader when a node dies — this episode solves the problem we deliberately left hanging: manual unseal. You already felt it yourself in practice: every node restart, someone must be present to type the Unseal Keys one by one. In the lab, this feels like an interesting ritual. In production, it's a race against time that can take the business down.

Imagine a real scenario: at 3:00 AM, a security patch requires rebooting all servers including your three Vault nodes. After the reboot completes, the whole cluster is back to sealed. Applications trying to pull dynamic database credentials start failing, then cascade to the entire system. The only savior is an operator with the Unseal Keys — and they're asleep. This is why auto-unseal isn't just convenience, it's part of availability engineering.

This episode covers how the auto-unseal mechanism works (envelope encryption), how to configure it on the three major cloud providers (AWS KMS, GCP Cloud KMS, Azure Key Vault), how its security compares to manual unseal, and the traps you must watch out for — because a wrong KMS configuration can leave Vault locked forever.

#Main Discussion

#Recap: Why Does Vault Need Unseal?

From episode 3, we know that when Vault is initialized, the data encryption key (which protects all secrets) is split into several parts using Shamir's Secret Sharing Algorithm. This key is called the root key. Vault stores this root key in encrypted form — and to decrypt it, at least a threshold (e.g. 3 of 5) of Unseal Keys is needed. Until unsealed, Vault refuses all operations: it can't read secrets, can't create tokens, can't issue certificates. In short, sealed = the system is completely dead.

The problem: who holds the Unseal Keys? If stored by many people (good separation of duties), then gathering them is even harder in an emergency. If stored by just one person, security drops drastically. There's an eternal tension between security and availability. Auto-unseal offers an elegant way out.

#Envelope Encryption: The Concept Behind Auto-Unseal

Auto-unseal doesn't eliminate the root key concept — it moves its custodian. The concept is known as envelope encryption:

1. Vault has a root key (barrier key) that encrypts all data in storage.
2. That root key is wrapped by a KMS key in the cloud provider, producing a ciphertext stored in Vault's storage.
3. When Vault boots, it calls the Cloud KMS API to request decryption of the wrapped root key.
4. Once the root key returns in plaintext, Vault opens the barrier and becomes unsealed — automatically, without humans.

A real-world analogy: manual unseal is like a safe that can only be opened by a combination split into parts held by different people — secure, but troublesome. Auto-unseal is like a safe that opens itself after verifying its owner's fingerprint with a trusted third party (KMS). Secure, fast, and automatic.

What needs emphasis: KMS credentials and IAM roles actually become the primary target for attackers — because whoever can call KMS to decrypt the wrapped key can open Vault's seal. Therefore, auto-unseal's security depends heavily on the quality of access control to KMS.

#Configuring Auto-Unseal with AWS KMS

The first step is setting up a Customer Master Key (CMK) in AWS KMS and granting access to the Vault instance. The safest way is giving Vault access via an IAM Role attached to the EC2 instance (not a static access key in the config file).

First, create an IAM policy that only allows KMS encrypt/decrypt operations on one specific key:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt",
        "kms:Encrypt",
        "kms:GenerateDataKey",
        "kms:ReEncryptFrom",
        "kms:ReEncryptTo"
      ],
      "Resource": "arn:aws:kms:ap-southeast-1:123456789012:key/1234abcd-5678-90ef-ghij-klmnopqrstuv"
    }
  ]
}

The least privilege principle: this policy only grants access to one KMS key, not all keys in the account. Once the role is attached to the instance, the Vault configuration only needs to add the seal "awskms" block:

seal "awskms" {
  region     = "ap-southeast-1"
  kms_key_id = "1234abcd-5678-90ef-ghij-klmnopqrstuv"
  endpoint   = "https://kms.ap-southeast-1.amazonaws.com"
}
 
storage "raft" {
  path   = "/opt/vault/data"
  node_id = "vault-node-1"
 
  retry_join {
    leader_api_addr = "https://10.0.0.11:8200"
  }
}
 
listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_disable   = true
}

#Configuring Auto-Unseal with GCP Cloud KMS

In Google Cloud, the flow is similar: create a Crypto Key in Cloud KMS, grant access to the Service Account running the VM, then configure Vault using seal "gcpckms". Here are the key setup steps:

gcloud services enable cloudkms.googleapis.com
 
# Create the key ring and crypto key
gcloud kms keyrings create vault-keyring --location global
gcloud kms keys create vault-unseal \
  --location global \
  --keyring vault-keyring \
  --purpose encryption
 
# Create a service account for the Vault instance
gcloud iam service-accounts create vault-sa \
  --display-name "Vault Auto-Unseal"
 
# Grant encrypt/decrypt permission on that crypto key
gcloud kms keys add-iam-policy-binding vault-unseal \
  --location global \
  --keyring vault-keyring \
  --member "serviceAccount:vault-sa@PROJECT.iam.gserviceaccount.com" \
  --role "roles/cloudkms.cryptoKeyEncrypterDecrypter"

Note the role used: cloudkms.cryptoKeyEncrypterDecrypter — minimal, only covering encryption and decryption. Never give a broader role like cloudkms.admin to an application service account.

Then the Vault configuration:

seal "gcpckms" {
  project     = "my-gcp-project"
  region      = "global"
  key_ring    = "vault-keyring"
  crypto_key  = "vault-unseal"
}
 
storage "raft" {
  path   = "/opt/vault/data"
  node_id = "vault-node-1"
 
  retry_join {
    leader_api_addr = "https://10.0.0.11:8200"
  }
}

Same as AWS, Vault reads GCP credentials from the VM's metadata server / default application credentials — not from a key file stored on disk. Make sure the VM runs the correct service account, for example when creating the instance:

gcloud compute instances create vault-node-1 \
  --service-account vault-sa@PROJECT.iam.gserviceaccount.com \
  --scopes https://www.googleapis.com/auth/cloudkms

#Configuring Auto-Unseal with Azure Key Vault

In Azure, the flow is: create a Key Vault and a Key inside it, grant access to a managed identity or service principal, then configure Vault:

seal "azurekeyvault" {
  tenant_id      = "11111111-2222-3333-4444-555555555555"
  client_id      = "66666666-7777-8888-9999-000000000000"
  client_secret  = "REDACTED"
  vault_name     = "my-vault-kv"
  key_name       = "vault-unseal"
  environment    = "AZUREPUBLICCLOUD"
}
 
storage "raft" {
  path   = "/opt/vault/data"
  node_id = "vault-node-1"
 
  retry_join {
    leader_api_addr = "https://10.0.0.11:8200"
  }
}

This configuration uses a service principal with client_id and client_secret. For a more secure environment, Azure provides managed identity — Vault can configure authentication via managed identity by omitting client_id/client_secret and only providing tenant_id plus the client_id of the managed identity. This service principal must have Encrypt, Decrypt, Unwrap Key, and Wrap Key permissions on that key (the Key Vault Crypto Officer role, or at minimum Key Vault Crypto User for decryption).

#Init and Verification: Unseal Keys Become Recovery Keys

The most visible change when using auto-unseal is in the initialization process. Note that Vault no longer issues Unseal Keys, but Recovery Keys:

export VAULT_ADDR=https://10.0.0.11:8200
vault operator init

Unseal Keys        : (none)  ← no more!
Recovery Keys      : 5
Recovery Key 1     : abc...def
Recovery Key 2     : ...
Initial Root Token : hvs.REDACTED

Recovery Keys are not used for routine unseal — they're only used for manual recovery if access to KMS is lost, or for specific operations like vault operator generate-root. Because auto-unseal is active, Vault unseals itself without human intervention:

vault status

Key             Value
---             -----
Sealed          false
HA Enabled      true

Note the line Sealed false — even though no human typed anything. This is the magic of auto-unseal: Vault decrypts the wrapped root key via KMS at boot. To re-init across all cluster nodes, just unseal with Recovery Keys once per node (using vault operator unseal with recovery keys still works), or more practically: since the root key is the same, after one node is unsealed, other nodes can open via vault operator raft join at the setup stage.

#Auto-Unseal vs Manual Unseal Comparison

Both approaches have their place. Here's the honest comparison:

#Common Pitfalls

The auto-unseal area is one of the most classic incident sources in the Vault production world:

The last point needs emphasis: auto-unseal moves the dependency to Cloud KMS. If KMS is down when Vault needs to reboot, Vault stays sealed — same as waiting for a human, only now waiting for the cloud provider. For production, consider a transit seal or a combination (auto-unseal as primary, recovery keys as the emergency manual procedure written in the runbook and drilled periodically).

#Conclusion

In this episode 21 we covered how to eliminate manual unseal from Vault operations using auto-unseal with three cloud providers: AWS KMS, GCP Cloud KMS, and Azure Key Vault. We understood the envelope encryption mechanism that wraps the root key with a KMS key, how to prepare IAM/roles with the least privilege principle, the behavior change at init (Unseal Keys become Recovery Keys), and the honest comparison between auto-unseal and manual. Equally important, we mapped the common mistakes that can leave Vault locked forever — and how to prevent them.

Now your Vault can wake itself up after a reboot and fail over quickly. But wait — a Vault that's alive and available turns out to be not enough. How do we know who accessed what secret, when, and from where? How do we meet audit compliance demands (PCI-DSS, SOC 2) and protect Vault from its own host? That's the next big topic: Audit Logging, Security Hardening & Compliance in episode 22. Keep your enthusiasm up!