Learn SELinux - Audit, Monitoring & Incident Response

#Introduction

In episode 14 you locked down the system: domains confined, booleans trimmed, CIS baseline installed. Now a paradox appears that's often underestimated: the stricter the policy, the more denials it produces — and if those denials aren't monitored, you'll never know which are config bugs and which are attacks. An attacker restricted by SELinux leaves a trace; an unrecorded denial is a lost trace.

This episode turns the SELinux log from a mere file into an observation and response system. We configure auditd, read traces with ausearch and aureport, build real-time denial alerts, and then go into an incident response scenario: analyzing denial chains, forensics of changed contexts, and fast mitigation that doesn't kill production — from scoped permissive to permanent policy fixes.

#Main Discussion

#AuditD and the SELinux Denial Path

All SELinux decisions — including every denied — are forwarded to the Linux Audit subsystem, managed by the auditd daemon. An AVC denial isn't written directly by the kernel to a plain file; it's sent as an audit event, and auditd writes it to /var/log/audit/audit.log. That's why auditd is a mandatory component in a production SELinux stack.

Make sure the daemon is active:

systemctl enable --now auditd
auditctl -s

enabled 1
failure 1
pid 1827
rate_limit 0
backlog_limit 64
lost 0

Notice the failure 1 line — audit is configured to panic when it fails to write, not silently drop events. This isn't a coincidence; in incident response, losing logs is worse than downtime.

#Configuration and auditctl Rules

Two things you need to set: capacity (in /etc/audit/auditd.conf — max_log_file, num_logs, flush) and rules (what gets audited). AVC rules don't need to be written manually — the kernel always records all SELinux decisions. What you need to add are rules for sensitive non-SELinux objects, for example critical config files:

auditctl -w /etc/ssh/sshd_config -p wa -k sshd_config
auditctl -w /etc/passwd -p wa -k user_accounts
auditctl -l

This -w (watch) -p (permission) -k (key) form is the standard audit language: every write or attribute change to the watched file is recorded with the sshd_config label. For persistence across reboots, those rules are written into /etc/audit/rules.d/*.rules files — not just executed live as above.

#Finding Traces with ausearch

SELinux denials can be read directly from the audit log. The core command is ausearch — an audit event searcher with highly expressive filters:

ausearch -m avc -ts recent

time->Fri Aug  7 14:03:11 2026
type=AVC msg=audit(1754586191.123:4567): avc:  denied  { read } for
pid=5210 comm="httpd" name="index.php"
scontext=system_u:system_r:httpd_t:s0
tcontext=system_u:object_r:home_root_t:s0 tclass=file

The most-used variations in investigations:

• ausearch -m avc -ts today — all of today's denials.
• ausearch -m avc -i -ts today — -i interprets numeric values into text (UID, names).
• ausearch -m avc -ui 1001 -ts today — denials belonging to a specific user.
• ausearch -m avc -c nginx -ts today — denials belonging to a process with a specific name.

If you're not sure whether a denial is important or just noise, audit2why gives a human-readable explanation of why it was denied and the fix steps:

ausearch -m avc -ts recent | audit2why

#Summarizing with aureport

ausearch answers "what are the details", aureport answers "what's the big picture" — an aggregate summary. In an incident responder role, that's the first step for building context before diving into details:

aureport -a -ts today

Number of AVC denials: 142
Number of MAC events: 142
Number of failed syscalls: 31

A pattern of 142 denials all from one domain at the same hour usually indicates one root cause (for example a wrong label), not a distributed attack. aureport turns a pile of 142 log lines into one line of conclusion — that's its value in high-pressure moments.

#Real-time Alerts: From Journal to SIEM

A denial only noticed when you open the file isn't monitoring. Two practical alerting paths:

Path 1 — journald. Audit events are also forwarded to the systemd journal. This is the fastest way to do system-based alerting:

journalctl _TRANSPORT=audit -f | grep -i 'avc:  denied'

Path 2 — SIEM/aggregator. For production, send audit events to a centralized log platform (for example Elasticsearch, Loki, or your team's log aggregator). Filter on the receiving side: type=AVC and result=denied are the default signals worth entering the alert pipeline. One avc: denied log line is a signal; a hundred lines in one minute from one source is an indicator.

#Incident Response: Analyzing the Denial Chain

Now the scenario: your team receives an alert — dozens of avc: denied in 2 minutes from the httpd process, with an unusual user ID. Don't immediately allow. Compose questions in order:

1. Who and what — ausearch -m avc -ts today -ui 48 shows denials from UID 48 (the apache account). But something's off: that process is requesting execve from the /tmp directory.
2. When and how often — aureport -a -ts today shows the denial spike starting at 14:00, coinciding with a CPU spike.
3. What's the target context — scontext=...:httpd_t:s0, tcontext=...:home_root_t:s0. An httpd_t process trying to read files in a home directory — abnormal for a web workload.

This denial chain is a narrative: the attacker has already executed something inside the web server process (via an application vulnerability), and their attempt to touch user files was blocked by SELinux. That's not a bug — that's a defense line working. The next step is context forensics, not writing rules.

#Forensics: Changed Contexts

The most important part of an SELinux investigation is checking whether someone changed the labeling — because changed contexts are an attacker's fingerprints. Check in this order:

ls -lZ /var/www/html
semanage boolean -l -C
semanage permissive -l

semanage permissive -l
httpd_t

Notice the last finding: there's an httpd_t domain set to permissive — previously, in episode 14, there wasn't. Someone executed semanage permissive -a httpd_t on this machine, or an attacker changed it. Combined with restorecon -n -R -v /var/www showing many files outside their expected labels, the forensic picture becomes clear: labels changed, permissive activated, denials quieted — the classic pattern of an attacker trying to silence the alarm.

#Quick Mitigation: Scoped Permissive, then Policy Fix

When production is down and the team needs immediate access, the biggest temptation is setenforce 0 — disable SELinux entirely. Don't. The correct mitigation is scoped permissive: turn off enforcement only for the problematic domain, keep everything else locked:

semanage permissive -a httpd_t

The service runs immediately, but every other domain stays enforcing. Now there's time to calmly resolve the root cause:

1. Identify — ausearch -m avc -ts recent | audit2why to understand the missing rule.
2. Build the fix — ausearch -m avc -ts recent | audit2allow -M httpd_fix produces an httpd_fix.te module; review every line before using it.
3. Test — install the module with semodule -i httpd_fix.pp while the domain is still permissive; make sure the app runs and there are no new denials.
4. Lock again — semanage permissive -d httpd_t restores full enforcement.

This order ensures every step is reversible and audited. And remember the golden rule of SELinux incident response: permissive is triage, not a cure. If triage is left permanent, your last defense line dies silently — and that's exactly what an attacker looks for.

#Closing

In this episode 15, you've built the complete observation and response path: enabling and configuring auditd (auditctl -s, auditctl -w -p -k), searching denials with ausearch -m avc across various filters, summarizing the big picture with aureport -a, building real-time alerts through journald or SIEM, analyzing denial chains as an attack narrative, doing forensics on changed contexts (semanage permissive -l, semanage boolean -l -C), and running quick mitigation with scoped permissive before installing the permanent policy fix.

The essentials to take with you:

• SELinux denials are signals — build a pipeline so they're never missed.
• ausearch for details, aureport for the big picture; use both in sequence.
• A coherent denial chain is an attack narrative, not a bunch of random lines.
• Check changed contexts and lingering permissive — those are fingerprints.
• Scoped permissive is triage; policy fixes are the cure; don't reverse the order.

Observation and response are now live. But there's one layer we've been using without seeing how it works: how does SELinux actually communicate decisions between the kernel and userspace? In the next episode 16, we open the hood — selinuxfs & Runtime Inspection: dissecting /sys/fs/selinux, userspace-kernel interaction, runtime verification with matchpathcon, and live inspection with seinfo and semanage. See you in episode 16!