Learn Elasticsearch - Audit Logging & Compliance

#Introduction

Authentication and encryption make access secure. But security isn't complete without accountability: if an incident occurs, you must be able to answer "who accessed what, when, and with what result?". That's the role of audit logging — and that answer is also a legal requirement in many industries (GDPR, SOC 2, HIPAA).

Episode 17 covers audit logging in Elasticsearch: enabling it, event types, output formats (file and index), event filtering so it doesn't burden storage, and compliance practices — GDPR, retention policies, personal data handling, and access tracking.

#Enabling Audit Logging

Audit logging is an Elasticsearch security feature that records security- and access-related activity. Enable it in elasticsearch.yml:

xpack.security.audit.enabled: true
xpack.security.audit.logfile.events.include:
  - access_denied
  - access_granted
  - authentication_failed
  - connection_denied
  - run_as_granted
  - run_as_denied
  - anonymous_access_denied

After a restart, each selected event is recorded in the audit log file. Because this feature is sensitive to performance and storage, only enable the events you actually need.

#Audit Event Types

Audit events are grouped into several categories:

#Output: File and Index

#File Output

Audit logging writes to a file by default (logs/<cluster>-audit.log). The format is JSON per line, easy to parse:

{"type":"audit","event_type":"authentication_success","principal":"arman","request.method":"POST","request.path":"/_security/user","result.type":"success"}

Files are suitable for offline archiving and forensic analysis.

#Index Output

Audit events can also be sent to a dedicated audit index inside Elasticsearch itself — making search and dashboards easy. Additional configuration:

xpack.security.audit.index.settings.enabled: true
xpack.security.audit.index.events.include: [access_denied, access_granted, authentication_failed]

#Filtering Audit Events

Not every event deserves to be recorded — for example health check logs that appear every second. Filtering reduces noise and storage:

xpack.security.audit.logfile.events.ignore_filters:
  health_check:
    users: ["system"]
    actions: ["access_granted"]
    paths: ["/_cluster/health", "/_nodes/stats"]

With the filter above, the system user's access to health check endpoints doesn't enter the audit. Rule of thumb: filter selectively and documented — don't cancel out the benefit of auditing. Security teams usually decide what may be filtered.

#Compliance: GDPR and Access Tracking

#GDPR Principles

GDPR (and other regulations) demands three things from personal data controllers:

• Access control — only authorized parties can access personal data (episode 15: DLS/FLS, RBAC).
• Access trail — being able to show who accessed whose data, and when. Audit logging answers this requirement.
• Right to erasure — the ability to delete a person's data when requested ("right to erasure"). This means the data pipeline must be designed so personal data can be deleted/not indexed from the start.

#Data Retention Policies

Retention isn't just "keep everything forever" — storing personal data longer than needed actually violates GDPR. Apply a layered retention policy:

• Functional retention — logs needed for operations: 30–90 days.
• Compliance retention — audit logs and archives: 1–7 years per regulation.
• Anonymization — after the functional period, delete or anonymize personal fields (IP, email) instead of deleting all data.

All of this can be automated with ILM (episode 10) — a delete phase at the end of the lifecycle, plus downsampling/anonymization for data that must be retained.

#Handling Personal Data in Indexes

Before personal data enters Elasticsearch, ask: does this field really need to be indexed? Use the following pattern:

{
  "mappings": {
    "properties": {
      "email": { "type": "keyword", "index": false },
      "credit_card": { "type": "keyword", "index": false },
      "ip_address": { "type": "ip" }
    }
  }
}

A field with index: false can't be searched — disabling searchability of personal fields from the mapping onward. For data that shouldn't even appear in _source, use an ingest pipeline (episode 12) to strip it before it enters, or FLS from episode 15.

#Common Mistakes

1. Enabling all events. Storage and performance break — choose the relevant events.

2. Audit logs in the same cluster as the data. Lost during an incident — separate the cluster or archive.

3. Storing personal data indefinitely. Violates GDPR — apply retention with ILM.

4. Overly aggressive filtering. The audit trail becomes empty — document and get it approved by security.

5. Indexing searchable personal fields. Consider index: false or remove them in a pipeline.

#Conclusion

In episode 17 you mastered audit logging and compliance: enabling audit logs, event categories (authentication, access, run_as, document writes), file and index output, event filtering, and compliance practices — GDPR, layered retention policies with ILM, personal data handling with index: false and pipelines, and access tracking for DSAR.

Key takeaways:

• Audit logging answers "who accessed what, when" — the basis of accountability and compliance.
• Only enable the events you need to save storage and performance.
• Don't store audits in the same cluster if their value is critical.
• Layered retention + anonymization = GDPR compliance without throwing away all data.
• Personal fields shouldn't be searchable (index: false) from the mapping onward.

So far we've worked on a small cluster. Time to scale up. In episode 18 we'll cover scaling Elasticsearch clusters: horizontal scaling by adding nodes, shard allocation strategies, the ideal 10–50 GB shard size, hot-warm-cold implementation, when to scale up vs scale out, and SSD vs HDD storage considerations. See you there!