Learn Secret Management - Audit Logging, Security Hardening and Compliance

#Introduction

In episode 16 you made sure the OpenBao server unlocks automatically after maintenance with AWS KMS, GCP Cloud KMS, and Azure Key Vault based auto-unseal. Episode 17 continues in a different direction: not how the cluster wakes up, but how the cluster talks and leaves a trail. We cover audit logging that records all activity, hardening to close physical and network gaps, and raft snapshot backup for disaster readiness — three things that are almost always mandatory requirements when an OpenBao server is audited by a compliance team or an external security assessor.

For those of you already running OpenBao in production, this episode is a moment to double-check: is every operator operation recorded? Are encryption keys safe from being swapped to disk? What is the plan if all nodes are lost? If the answers are not yet clear, the material below will answer them.

#Audit Device: A Window into Every Request and Response

OpenBao is the system that handles the most sensitive data in your organization. When something goes wrong — whether a leaked token, a mistaken policy, or suspicious access — you need accurate evidence. This is where the audit device comes in: a built-in mechanism that records every API request and response reaching the server, including who called, which token was used, which path was accessed, and what the result was.

One thing to understand from the start: an audit log is not an ordinary application log. Each entry is built from the raw request and response data, then hashed and encrypted before being written, so its contents cannot be modified without being detected. For that reason, audit log files must be treated with the same protection as the secrets themselves.

#Enabling the File Audit Device

The simplest way to start recording is to enable a file-type audit device that writes logs to a specific path:

bao audit enable file file_path=/var/log/openbao/audit.log
bao audit list

The bao audit enable file file_path=/var/log/openbao/audit.log command registers a new device. After that, every API request and response is immediately recorded to that file. Note that this command requires sudo capability on the sys/audit path, so only operators with full access can do it — and that is how it should be.

The audit log file can grow very quickly on a busy cluster. Consider log rotation with logrotate, or point the audit at devices designed for high volume such as syslog or socket forwarding to a SIEM.

#Reading and Understanding Audit Log Contents

Once the audit device is active, every operation is written as a JSON entry. If you open the audit file, a single application request reading a secret will look roughly like this:

{
  "type": "response",
  "auth": { "token_type": "service", "policies": ["web-app"] },
  "request": {
    "path": "secret/data/myapp",
    "remote_address": "10.0.4.22"
  },
  "response": { "data": { "data": { "key": "vault:v1:..." } } }
}

When reading an audit log, pay attention to three things: who the caller is via auth, what path was accessed via request, and whether the response contains data or an error. Suspicious patterns — many permission denied from a single address, or policy paths accessed outside working hours — are early signals worth chasing. Remember, sensitive data such as secret values still appear in the audit log as ciphertext, so analysis can be done without leaking the original contents.

#Hardening: Memory Locking with disable_mlock

A security concept often overlooked: secrets must stay in RAM, not be swapped to disk. Under normal operation, processes are temporarily moved to disk swap space when RAM runs low. For OpenBao, that means master key, token, and encryption key data could end up on disk — and be readable by anyone who gains access to that disk.

To prevent this, OpenBao supports memory locking via mlock on POSIX systems. The disable_mlock value in the configuration controls this behavior: true allows swapping, false forces OpenBao to lock its memory so it is never swapped.

mlock configuration in config.hcl

disable_mlock = false

With disable_mlock = false, OpenBao locks the memory pages it uses so the kernel cannot move them to swap. Note that mlock cannot be run by a process without the IPC_LOCK capability. When running OpenBao with a service manager such as systemd, make sure LimitMEMLOCK=infinity is set — and don't run OpenBao as root if it can be avoided.

#Strict TLS Enforcement on the API Endpoint

A server that accepts secrets over plain HTTP is the same as sharing all your keys over a wire that can be tapped. So the next hardening step is to force all API communication to use TLS with the listener configuration:

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_disable   = "false"
  tls_cert_file = "/etc/openbao/tls/server.crt"
  tls_key_file  = "/etc/openbao/tls/server.key"
}

Besides installing a certificate, strict TLS means rejecting all insecure variants. Summary of the hardening controls in this episode:

With this combination, all traffic to the API is forced through an encrypted channel, and clients that cannot prove their identity are rejected from the start. To lighten the workload, internal certificates can be issued automatically by the PKI secrets engine you learned about in episode 6.

#Disaster Recovery: Automating Raft Snapshot Backup

All the hardening above is useless if you lose all your nodes. Raft integrated storage keeps data distributed, but periodically taken snapshots remain the last safety net — the only way to recover data if all nodes fail at once.

bao operator raft snapshot save /backups/backup.snap

A snapshot can be taken without stopping the service. For compliance and DR purposes, schedule daily snapshots via cron or a systemd timer, send copies to a separate location (an object storage bucket, or an off-site location), then periodically test the restore in a staging environment. A snapshot whose restoration was never tested is just an illusion of protection.

#Conclusion

In this episode 17 you built the third security layer of an OpenBao cluster: an audit device that records every API request and response to an encrypted file, hardening with disable_mlock so secrets never move to swap, mandatory TLS on the API endpoint, and raft snapshot backup as the backbone of disaster recovery. All three are the standard answers when an audit team asks how you track access, protect keys, and guarantee recovery.

Key takeaways:

• The audit device records everything — hashing and encryption make the log immutable without detection.
• disable_mlock = false protects encryption keys from falling into disk swap.
• TLS is mandatory on the API endpoint, and consider mutual TLS for internal traffic.
• A snapshot without a restore test is not a backup — automation and regular testing are key.

In the next episode, episode 18, you will use all of this understanding for a rather challenging step: migrating from HashiCorp Vault to OpenBao in production — moving data and configuration without downtime, and validating compatibility along the way.