Learn Vault - Audit Logging, Security Hardening & Compliance

#Introduction

After automating unseal with Cloud KMS in episode 21 so Vault wakes up by itself after a reboot — this episode raises Vault's security standard to the level demanded in production and compliance environments: Audit Logging, Security Hardening & Compliance.

Imagine this: one day there's a security incident. A production secret leaks and management asks, "who read this secret last? when? from which IP address? with which token?" If your Vault doesn't have audit logging enabled, your answer is: "we don't know." And in the compliance context — PCI-DSS, SOC 2, HIPAA, ISO 27001 — the words "we don't know" are a failure that can make you fail an audit, get fined, or lose certification.

On the other hand, Vault is the system that stores all of a company's secrets — it's one of the most attractive targets for attackers and also one of the most intolerant systems to configuration errors. Disabling swap without mlock, leaving API endpoints without TLS, or forgetting to revoke the Root Token are classic mistakes that often end in disaster. This episode equips you with three layers of defense: record everything (audit), harden the host & API (hardening), and be ready for the worst (backup & DR).

#Main Discussion

#Vault Audit Devices: Recording Every Request and Response

Vault has a very powerful and simple feature: audit devices. When enabled, every HTTP request entering the Vault API — along with its response — is recorded into structured logs. This includes: login operations, secret reads, token creation, certificate issuance, all of them without exception.

The three most common audit device types:

Enabling the file audit device is very easy:

vault audit enable file file_path=/var/log/vault/audit.log

Vault supports several audit devices at once (for example one local file + one syslog to ship to a SIEM). Every incoming request produces a log entry like this:

{
  "time": "2026-08-02T09:30:15.123456789Z",
  "type": "request",
  "auth": {
    "client_token": "hmac-sha256:REDACTED",
    "entity_id": "8a9f...",
    "token_type": "service",
    "policies": ["default", "app-backend"]
  },
  "request": {
    "id": "f2c1...",
    "path": "secret/data/production/database",
    "operation": "read",
    "remote_address": "10.0.4.21:51234",
    "data": {}
  },
  "response": {
    "data": {
      "data": {
        "username": "hmac-sha256:REDACTED",
        "password": "hmac-sha256:REDACTED"
      }
    }
  }
}

Note several important details from the example above:

• Tokens and sensitive data are hashed using HMAC-SHA256. Vault shows the digest, not the original value — so the audit log doesn't become a "second secret leak." The HMAC key is stored separately and can be rotated.
• However metadata like path, operation, remote_address, policies, and entity_id remains plaintext — that's what becomes the material for investigation and security analysis.
• Each line is one request and one response (there are two entries per operation: request and response).

To check the active audit devices:

vault audit list

Path     Type    Description
----     ----    -----------
file/    file    n/a

Two operational things often forgotten about audit logs:

• HMAC key and its rotation — sensitive values are hashed with an HMAC key owned by Vault. This key can be rotated (vault audit disable file/ then re-enable, or use the available options), so if the HMAC key is suspected leaked, you can change it without deleting the entire log history.
• File rotation and retention — audit logs that are never rotated will grow without bound and eventually fill the disk. Use logrotate (or a log pipeline) with a retention policy per regulation, for example 12 months as required by applicable compliance:

/etc/logrotate.d/vault

/var/log/vault/audit.log {
    daily
    rotate 365
    compress
    delaycompress
    missingok
    notifempty
    create 0640 vault vault
    sharedscripts
}

#Hardening Best Practices: Arming the Host and API

A good audit log is useless if Vault itself can be easily compromised. Here are the most important hardening measures you must apply — these are the result of field experience, not just a cosmetic checklist.

#1. Disable Swap and Enable Memory Locking (mlock)

Vault's encryption key (root key) lives in memory. If the operating system does memory swapping (moving memory pages to disk), the key could be written to disk in plaintext — and disks are easier to steal than RAM. Vault provides the mlock mechanism to lock memory pages so they can't be swapped.

disable_mlock = false

A value of false means mlock is active — Vault will call the mlock() system call to prevent itself from being swapped. However mlock() requires special privileges: Vault must run as root or have the CAP_IPC_LOCK capability. In container deployments, you need to add this capability:

services:
  vault:
    image: hashicorp/vault:1.18.0
    cap_add:
      - IPC_LOCK
    security_opt:
      - no-new-privileges:true

#2. Strict TLS Enforcement on the API Endpoint

There's no reason to let secret traffic cross the network unencrypted. Vault must be accessed over HTTPS, ideally with a certificate from an internal CA (remember PKI from episode 7!) or a public CA.

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_disable   = false
  tls_cert_file = "/etc/vault.d/tls/vault.crt"
  tls_key_file  = "/etc/vault.d/tls/vault.key"
  tls_min_version = "tls12"
}

Also enforce TLS on the client side by setting VAULT_CACERT or VAULT_CAPATH when running the CLI, so vault verifies the server certificate — not just -tls-skip-verify.

#3. Least Privilege and Policy Review

From episode 9 we already learned policies. In the production phase, this principle is strengthened into an ongoing practice:

• Default deny — all access is denied unless written in a policy. Always test with vault token capabilities.
• No root tokens — the Root Token must be revoked immediately after setup finishes (vault token revoke). For emergency operations needing root rights, use vault operator generate-root which requires Recovery Keys (episode 21).
• Audit policies periodically — a policy "forgotten" to be removed is a silent security hole. Schedule a policy review each quarter.
• Short token lifetimes — all tokens must have a reasonable TTL, not an unlimited period.

#Disaster Recovery and Raft Backup

Vault stores all data in Raft storage. What if the cluster loses data due to a disaster (data center burns down, cloud account deleted, admin misconfigures)? The answer: Raft snapshots.

Creating a manual snapshot is very easy:

vault operator raft snapshot save /backup/vault-backup.snap

To restore:

vault operator raft snapshot restore /backup/vault-backup.snap

Automation is the key. Vault provides the Vault Snapshot Agent — a separate binary running as a daemon that saves snapshots on a schedule:

storage "file" {
  path = "/backup/vault-snapshots"
}
 
snapshot_agent {
  interval = "24h"
  lock     = true
}
 
exit_after = false

The snapshot agent also supports direct delivery to cloud storage (AWS S3, GCS, Azure Blob) and does client-side encryption. For a simple approach without an extra binary, you can use a cron job wrapping the save command, then upload to a separate bucket:

# /etc/cron.d/vault-snapshot
0 2 * * *  vault /usr/local/bin/vault snapshot-save-and-upload.sh

#!/usr/bin/env bash
set -euo pipefail
 
export VAULT_ADDR=https://vault.internal:8200
export VAULT_TOKEN="$(vault login -method=approle -token-only role_id=... secret_id=...)"
 
DATE=$(date +%Y%m%d-%H%M%S)
SNAP="/backup/vault-${DATE}.snap"
 
vault operator raft snapshot save "$SNAP"
 
# A separate bucket, not the one where other apps store data
aws s3 cp "$SNAP" "s3://acme-vault-backups/${DATE}.snap" \
  --sse aws:kms
 
find /backup -name '*.snap' -mtime +14 -delete

The importance of lock: a snapshot taken during a failover can be inconsistent. The Vault snapshot agent handles this with lock; in a manual cron, use an equivalent mechanism so two snapshots never run concurrently.

#Vault Server Compliance Checklist

Here's a checklist you can use as a tool to assess Vault's compliance readiness:

#Common Pitfalls

#Conclusion

In this episode 22 we strengthened Vault from three sides: audit logging — recording every request and response with token hashes and complete metadata for investigation and compliance; hardening — mlock and disabling swap so keys can't leak to disk, mandatory TLS on the API endpoint, and least privilege with Root Token revocation; and disaster recovery — Raft snapshot automation along with a restore procedure that must be drilled. We closed with a compliance checklist you can directly use as an audit tool.

But the journey isn't over. Everything we've built so far — cluster, auto-unseal, hardening — still lives in a single namespace. What happens when Vault is shared by many teams with full isolation needs: backend team, data team, security team? And what are the feature boundaries between Vault Open Source and Vault Enterprise? That's the interesting topic of episode 23: Vault Namespaces & Multi-Tenancy (Enterprise vs Open Source). See you in the next episode!