Learn Linux - System Logging, Rotation & Auditing

#Introduction

After episode 22 where we covered task automation with cron jobs and systemd timers — including redirecting script output to log files so every execution is recorded — in this episode we'll discuss the logs themselves: where they come from, how they're managed, and how to make them a tool that saves you when production breaks.

Imagine logs as an airplane's black box. Pilots don't write the black box for fun — it exists so that when an incident happens, experts can step back and answer the question: what really happened before, during, and after the event? In the server world, logs are the same black box. When an application errors at 2 AM, when a server is rebooted without permission, or when there's a suspicious login attempt — logs are the silent witness telling everything chronologically.

The problem is, logs don't manage themselves. Without a rotation policy, log files bloat until they consume the entire disk — and ironically, a system dying from a full disk is a very common cause of downtime. Without a retention policy, logs are useless when you need to see an event from three months ago. And without the right architecture, important logs scatter across many places, making them nearly impossible to track.

In this episode we'll dissect the Linux logging architecture: getting to know the contents of /var/log/ and its important logs, understanding how logrotate works to limit log size and age, getting to know rsyslog for centralized logging, and a brief introduction to auditd as a security trail layer. We'll close with the practice of adding a logrotate configuration for a custom application log.

#Main Discussion

#The Logging Architecture: Why Linux Records Everything

Linux adheres to the philosophy that almost all system events should be recordable. Some programs write their own logs directly to files, but the majority send messages to a central logging daemon — syslog — which then decides where those messages are stored.

On modern systems, this architecture involves two components often thought of as one:

• rsyslog — the classic logging daemon that receives messages from the kernel and applications, then writes them to text files in /var/log/ or forwards them to other servers. It's the "guard post" that reads messages, assesses their source and severity, and delivers them to the right place.
• systemd-journald — a systemd component (detailed in episode 14) that holds timestamped structured logs in binary format, accessed via journalctl. journald and rsyslog can run side by side: journald captures everything neatly, rsyslog writes it to familiar text files.

Why do you need to understand both? Because when troubleshooting, you'll meet two worlds at once: text files in /var/log/ read with grep, and the binary journal read with journalctl. An admin who only masters one often loses important information. Fortunately, rsyslog on most distros is already configured to read from the journal, so both stay in sync.

#Getting to Know the /var/log/ Directory and Important Logs

All the main logs live under /var/log/. Let's map the most important ones:

This filename difference between the Debian and RHEL families is a classic trap. An admin used to reading /var/log/auth.log on Ubuntu will be confused looking for it on a Rocky server — the answer is at /var/log/secure. Likewise syslog vs messages. When you hop between distros, get used to checking the directory contents with ls /var/log/ before assuming.

To see the latest kernel messages, dmesg is the direct window into the kernel ring buffer (covered in episode 16):

sudo dmesg -T | tail -30

And to check whether someone is trying to log in unusually, auth.log/secure is the first place you should open:

sudo grep "Failed password" /var/log/auth.log | tail -20

Aug  2 02:14:11 app-server sshd[2301]: Failed password for invalid user admin from 203.0.113.45 port 52111 ssh2
Aug  2 02:14:12 app-server sshd[2303]: Failed password for invalid user admin from 203.0.113.45 port 52113 ssh2
Aug  2 02:14:13 app-server sshd[2305]: Failed password for invalid user admin from 203.0.113.45 port 52115 ssh2
Aug  2 02:14:14 app-server sshd[2307]: Failed password for invalid user root from 203.0.113.45 port 52117 ssh2
Aug  2 02:14:15 app-server sshd[2309]: Failed password for invalid user root from 203.0.113.45 port 52119 ssh2

The pattern above — the same IP attacking continuously with changing usernames — is the telltale sign of a brute-force attack which we'll fight with fail2ban in episode 25.

#logrotate: Why Logs Must Be Rotated

This is the part most often underestimated and most often causing downtime. An unbounded log file grows forever. Verbose applications like web servers or databases can write hundreds of MB of logs per day — and when the / partition (or /var) fills up, the system becomes unstable: services fail to write, applications error, even the system can stop responding. A disk filled by logs is a confusing downtime cause because the symptoms seem unrelated to logs.

The classic solution is log rotation: periodically rotating logs — old files are moved, compressed, kept for a set period, then deleted. The tool is called logrotate. Think of it like a daily journal being archived: today's page is always ready to fill, yesterday's page moves to the archive book, and archives older than the limit are discarded.

logrotate configuration is spread across two levels. The global level is in /etc/logrotate.conf, and the per-application level is in the /etc/logrotate.d/ directory — one file per application. The global file reads all files in /etc/logrotate.d/ through the include directive. An example of a common global config:

weekly
rotate 4
create
include /etc/logrotate.d

weekly means rotation happens once a week, rotate 4 keeps 4 old files before the oldest is deleted, and include /etc/logrotate.d pulls in all application-specific configs. Each distro has slightly different defaults, but the pattern is the same.

The most-used directives at the per-application level:

#Why delaycompress and su Matter

The two directives above are often misunderstood, even though they prevent two different classes of problems.

delaycompress solves the problem of applications still holding their log file. Imagine an application writing to app.log and keeping that file open. When logrotate rotates, the old file is named app.log.1. If it's immediately compressed to app.log.1.gz, the application still writing to the old file descriptor will keep writing into an already-compressed file — producing corrupted data or weird lines. With delaycompress, compression is delayed one cycle: app.log.1 is left intact, and only older files (app.log.2 and beyond) are compressed.

su <user> <group> solves the permission problem. logrotate configurations are run by root through a cron/systemd timer, but application log files are often owned by the application user (e.g. www-data). Without the su directive, rotation can fail with the error error: state file /var/lib/logrotate/status is not owned by root or Permission denied when reading the log directory. Adding su www-data www-data makes logrotate impersonate that user for operations on that log.

#Practice: A logrotate Configuration for a Custom Application Log

Suppose our application, myapp, writes its log to /var/log/myapp/app.log as user myapp. Without any configuration, that file grows without limit. Let's add a configuration in /etc/logrotate.d/myapp:

/var/log/myapp/app.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0640 myapp myapp
    su myapp myapp
}

Line-by-line interpretation: the log rotates every day (daily), seven old files are kept (rotate 7) — meaning you have a week of history. Old files are compressed (compress), but the one-cycle delay (delaycompress) gives the application time to release the file descriptor. missingok and notifempty prevent errors when the file is empty or missing. create makes a new file with the correct owner and permissions, and su myapp myapp runs the whole process as the application user.

Now, let's see what happens when the rotation policy needs to change — for example increasing retention from 7 to 14 days. Note the before-and-after diff:

    daily
    rotate 7
    rotate 14
    compress

Before applying a production change, verify the configuration with debug and force modes:

sudo logrotate -d /etc/logrotate.d/myapp

sudo logrotate -f /etc/logrotate.d/myapp

-d (debug) shows what would be done without actually doing it — a mandatory step before changing production config. Once confident, logrotate -f forces a rotation to happen now so you can see the result immediately.

#Centralized Logging with rsyslog

The more servers you manage, the harder it becomes to check logs one by one. This is where centralized logging works: all servers send their logs to one central server, so you can search in one place. rsyslog plays two roles here — as a receiver (server) and a sender (client).

On the central server, enable reception in /etc/rsyslog.conf (uncomment the reception module, usually imtcp), then restart. On the sending server, add a forwarding file:

auth.*    @@logs.example.com:514
kern.*    @@logs.example.com:514
mail.*    @@logs.example.com:514

The @@ notation means sending via TCP (reliable, but slower); a single @ means UDP (fast, but messages can be lost). After the file is created, apply it by restarting rsyslog:

sudo systemctl restart rsyslog

With this pattern, you can build a simple SIEM (Security Information and Event Management) — collecting all auth logs from dozens of servers into one place for analysis. This is the first step toward observable security, which we'll tie to auditing in episode 25.

#A Brief Introduction to auditd: A Trail That Can't Be Denied

Normal logs record what should have happened according to the application. auditd records what actually happened at the kernel level — every system call matching an audit rule. This is what distinguishes audit logging from regular logging: auditd captures activity even from processes trying to hide from the application.

The classic use case: monitoring access to sensitive files. Want to know who reads, writes, or changes the attributes of /etc/passwd, /etc/shadow, or /etc/sudoers? auditd can answer with precision.

-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k identity

The rules above are written to /etc/audit/rules.d/audit.rules, then reloaded. The meaning: -w (watch) a specific file, -p wa monitors write and attribute change operations, and -k identity gives a label for searching. Once active, you can search all events touching those files:

sudo ausearch -k identity

For a report summary, aureport -au presents statistics:

sudo aureport -au

#Common Mistakes (Common Pitfalls)

1. logrotate not running. Most often caused by cron being disabled, the logrotate timer being masked, or the system running anacron with an incompatible configuration. Verify by running logrotate manually and checking the log file timestamps.

2. delaycompress ignored for an app still writing. Removing delaycompress when the application still holds the old file open = corrupted compressed files. Keep delaycompress for any application that can't reliably release its file descriptor.

3. Forgetting the su directive. Configs targeting logs owned by a non-root user will fail with a permission error on the state file. Add su <owner> <group> so logrotate impersonates the log's owner.

4. Mixing Debian vs RHEL log formats. auth.log vs secure, syslog vs messages. Instead of assuming, check ls /var/log/ first — or better, do centralized logging so everything lands in one place.

5. Deleting logs manually with rm. When an actively-written log file is deleted, the application still holds the file descriptor and disk space isn't freed until the app restarts — and the new file may be created with wrong permissions. Always use logrotate or truncate, not rm.

#Conclusion

In this episode 23, you've thoroughly dissected the Linux logging architecture. We got to know the two main components — rsyslog writing to text files and journald storing structured logs — then mapped /var/log/ and its important logs: syslog/messages, auth.log/secure, boot.log, and dmesg. We covered logrotate: why rotation is mandatory, how /etc/logrotate.conf and /etc/logrotate.d/ work, and directives like daily, rotate, compress, missingok, delaycompress, and su. We also met centralized logging with rsyslog and the audit layer with auditd.

Key points to take with you:

• Logs are the server's black box — but only useful if managed: rotated, kept for the right duration, and searchable.
• /var/log/auth.log (Debian) = /var/log/secure (RHEL); know the naming differences across distros.
• logrotate is run by a scheduler — always make sure the logrotate cron/timer is active.
• delaycompress protects files still held by applications; su prevents permission failures.
• Centralized logging (rsyslog forwarding) and auditd turn logs from passive archives into active investigation tools.

Now you know how to read a server's condition from logs. In the next episode 24, we'll use that ability for the most nerve-wracking thing for an admin: System Performance Tuning & Troubleshooting — identifying bottlenecks when a server is slow or down, reading load average correctly, analyzing memory and disk I/O with vmstat and iostat, understanding the OOM killer, and managing swap. See you there!