Learn Vault - Observability, Monitoring & Troubleshooting

#Introduction

After separating tenants and understanding the Open Source vs Enterprise boundaries in episode 23 — this episode enters the subject closest to the daily work of a DevOps/SRE truly operating Vault: Observability, Monitoring & Troubleshooting.

There's a fact often unrealized until an incident happens: Vault can be down without you noticing. A sealed Vault doesn't send an alarm — it just starts rejecting requests. Applications trying to log in get errors, and you only find out after users complain. By then, how many minutes has production gone without secrets? A good monitoring system answers this question early: "Vault just sealed at 02:15 — we've sounded the alarm and are handling it."

This episode equips you with three things: telemetry — how to expose Vault metrics to Prometheus and visualize them in Grafana; diagnostic commands — the operator toolbox for checking cluster health from the CLI; and troubleshooting — a practical symptom-to-cause-to-solution table for the most common problems we encounter in the field, from expired tokens to split-brain.

#Main Discussion

#Vault Telemetry: Exposing Metrics for Prometheus

Vault collects internal metrics that can be exposed via telemetry. By enabling the telemetry block in the configuration, Vault provides the /v1/sys/metrics endpoint in Prometheus format — which can be scraped directly by Prometheus without an extra exporter.

telemetry {
  prometheus_retention_time = "24h"
  disable_hostname          = true
  unauthenticated_metrics_access = false
}
 
storage "raft" {
  path   = "/opt/vault/data"
  node_id = "vault-node-1"
 
  retry_join {
    leader_api_addr = "https://10.0.0.11:8200"
  }
}
 
listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_disable   = false
  tls_cert_file = "/etc/vault.d/tls/vault.crt"
  tls_key_file  = "/etc/vault.d/tls/vault.key"
}

Several important options:

• prometheus_retention_time — how long Vault keeps metrics in memory before they can be read. A value of "24h" is sufficient for periodic scraping.
• disable_hostname = true — cleans the hostname label so metrics across nodes are easy to aggregate (a common recommendation in container/VM deployments).
• unauthenticated_metrics_access = false — forces metric requests to require a token. You can enable tokenless access only if the endpoint is truly network-isolated; in most production environments, enable authentication and provide a dedicated token for Prometheus.

After restart, test the endpoint from the same node:

curl -s -H "X-Vault-Token: $VAULT_TOKEN" \
  https://10.0.0.11:8200/v1/sys/metrics?format=prometheus | head -30

# HELP vault_core_unsealed Whether the vault is sealed.
# TYPE vault_core_unsealed gauge
vault_core_unsealed 1
 
# HELP vault_core_active_nodes Number of nodes active.
# TYPE vault_core_active_nodes gauge
vault_core_active_nodes 1
 
# HELP vault_raft_leader_last_contact Last contact with leader (ms).
# TYPE vault_raft_leader_last_contact gauge
vault_raft_leader_last_contact 0
 
# HELP vault_api_srv_http_request_count Count of API requests.
# TYPE vault_api_srv_http_request_count counter
vault_api_srv_http_request_count{code="200",method="read",operation="read",path="secret/data/app"} 42

#Key Metrics You Must Monitor

Here are the core metrics and their meanings:

#Prometheus Configuration for Scraping Vault

Prometheus needs to be pointed at all three cluster nodes (not just the leader — every node exposes its own metrics, and followers have important Raft metrics):

- job_name: 'vault'
  metrics_path: '/v1/sys/metrics'
  params:
    format: ['prometheus']
  scheme: https
  scrape_interval: 15s
  static_configs:
    - targets:
        - '10.0.0.11:8200'
        - '10.0.0.12:8200'
        - '10.0.0.13:8200'
      labels:
        namespace: 'vault-cluster'
  authorization:
    credentials_file: /etc/prometheus/vault-token

With Grafana, build a dashboard showing at least: unseal status (per node), active node, raft leader last contact, request rate, and p95 latency. HashiCorp provides reference dashboards you can import, and the community has many open source Vault dashboards — but always verify the metrics match your Vault version.

#Alert Rules You Must Have

A dashboard is only useful if there are eyes watching it. So, define alert rules for scenarios that threaten availability. Here's an example rules.yaml for Prometheus/Alertmanager following the metric table above:

groups:
  - name: vault.rules
    rules:
      - alert: VaultSealed
        expr: vault_core_unsealed == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Vault node sealed"
          description: "Vault node {{ $labels.instance }} has been sealed for over 1 minute."
 
      - alert: VaultClusterDown
        expr: vault_core_active_nodes == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "No active node"
          description: "No Vault node is active — the entire cluster is not serving requests."
 
      - alert: VaultLeaderContactLost
        expr: vault_raft_leader_last_contact > 3000
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Leader contact lost"
          description: "Node {{ $labels.instance }} hasn't received a leader heartbeat for over 3 seconds."

#Ops Diagnostic Commands (The Operator Toolbox)

Besides metrics, operators need quick commands for direct diagnosis. This is the toolkit:

# 1. Basic status + details
vault status
vault status -detailed
 
# 2. Raft cluster health
vault operator raft list-peers
vault operator raft read-config
 
# 3. Seal information (is auto-unseal active?)
vault operator seal-status

Key                      Value
---                      -----
Recovery Seal Type       awskms
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Verification Required    false
Version                  1.18.4
Storage Type             raft
HA Enabled               true
HA Mode                  standby

Note the line Recovery Seal Type awskms — this is how you confirm auto-unseal (episode 21) is active, and HA Mode standby shows this node is a follower.

vault operator raft read-config

{
  "storage": {
    "raft": {
      "node_id": "vault-node-2",
      "path": "/opt/vault/data",
      "retry_join": [
        { "leader_api_addr": "https://10.0.0.11:8200" },
        { "leader_api_addr": "https://10.0.0.12:8200" },
        { "leader_api_addr": "https://10.0.0.13:8200" }
      ],
      "max_entry_size": 1048576
    }
  }
}

#vault debug: One Command for All Logs

When an incident actually happens, you won't have time to read logs one by one. Vault provides vault debug — a command that collects logs, configuration, status, snapshots, and performance information into one zip bundle you can immediately share with the team or open for analysis:

vault debug -duration=60s -interval=10s -output=/tmp/vault-debug.zip

Collecting debug data...    https://vault.internal:8200
==> Temporary directory created for storage of collected data
    /tmp/vault-debug/
==> Bundled debug file written to
    /tmp/vault-debug.zip

This bundle contains: health status, seal-status, redacted config, Vault logs, and CPU/memory samples. vault debug is the fastest way to give a complete picture to colleagues or support — far better than copying log snippets manually.

#Troubleshooting Common Field Problems

Here's a practical symptom → likely cause → solution map, based on real Vault operations experience:

#Short Case Study: Token permission denied

One of the most common field incidents is an app suddenly getting permission denied. Let's trace it with the tools we've learned:

# 1. Is the token still alive and when does it expire?
vault token lookup
 
# 2. What capabilities does the token have on a specific path?
vault token capabilities my-token-id secret/data/app
 
# 3. If the token is valid but capability is lacking — compare with the policy
vault policy read backend-app
 
# 4. Check whether the problem is a namespace (was the token created in a different namespace?)
vault token lookup -namespace=backend my-token-id

Key            Value
---            -----
id             hvs.CAES...
policies       [default backend-app]
expire_time    2026-08-02T11:30:00Z
ttl            1h
type           service

This logic flow — check cluster health first, then check the token, policy, and namespace — saves you hours of guessing. Nine out of ten Vault permission denied cases are one of: expired token, policy lacking capability, or a token created in a different namespace than you think.

#Common Monitoring Pitfalls

#Conclusion

In this episode 24 we built comprehensive Vault observability and troubleshooting capabilities: telemetry — exposing metrics like vault_core_unsealed, vault_raft_leader_last_contact, and request latency via the telemetry block and Prometheus scraping; diagnostic toolbox — vault status, vault operator raft list-peers, vault operator raft read-config, vault debug to collect a complete bundle during an incident; and a troubleshooting map for the most common problems from sealed vaults, policy mismatches, expired tokens, quorum loss, to LDAP/OIDC failures. Don't forget: even the best monitoring is useless without proper alerts and a trained runbook.

You now have all the technical material to operate Vault in production. But you might ask: how does all of this come together as one real system? How do the HA cluster we built in episode 20, auto-unseal in 21, hardening in 22, multi-tenancy in 23, and observability in 24 work together for a real company? The answer is in our final episode: A Complete Production-Grade Vault Architecture Case Study — where we tie everything together into an end-to-end enterprise architecture. This is the summit of the entire journey!