Learning AlmaLinux - Systemd, Services & Journald

#Introduction

In the previous episode, Episode 6, we set up users, permissions, and sudo — the identity of who can access the system. Now we shift to what the system runs: services and processes. Almost all services on AlmaLinux — SSH, firewall, Cockpit, databases — are managed by systemd, the init system and service manager that is the standard of the RHEL ecosystem.

This episode breaks down systemd from the unit lifecycle to boot targets, teaches you to write your own service units, manage resources with cgroups v2, and closes with journald — the logging system that unifies the trails of all services.

#Getting to Know Systemd

Systemd is PID 1 — the first process the kernel runs at boot, and the parent of all other processes. Its roles: starting, supervising, and stopping services; managing mounts, timers, and targets; and recording logs via journald.

Verify PID 1

ps -p 1 -o comm=

#The Unit Lifecycle

Every service can be in one of several states:

Command	Status / Action
`systemctl start <unit>`	Starts the unit
`systemctl stop <unit>`	Stops the unit
`systemctl restart <unit>`	Restarts (stops then starts again)
`systemctl reload <unit>`	Re-reads configuration without stopping
`systemctl enable --now <unit>`	Activates and auto-starts at boot
`systemctl status <unit>`	Status, PID, and recent logs
`systemctl is-active <unit>`	Checks if active/inactive

Manage the sshd service

sudo systemctl enable --now sshd
systemctl status sshd
systemctl is-active sshd

#Boot Targets

Systemd doesn't use the old "runlevels" — it uses targets, which are analogous to system states. A target determines the set of active units.

Target	Function
`multi-user.target`	Multi-user text mode system (default for servers)
`graphical.target`	System with a GUI
`rescue.target`	Single-user recovery mode
`emergency.target`	Minimal emergency shell

Check and change the default target

systemctl get-default
sudo systemctl set-default multi-user.target

#Writing Your Own Service Unit

When you run a custom application as a daemon, create a service unit for it. Units are stored in /etc/systemd/system/. Here's an example for a simple web application:

/etc/systemd/system/myapp.service

[Unit]
Description=My Custom Application
After=network-online.target
Wants=network-online.target
 
[Service]
User=devops
Group=devops
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp
Restart=on-failure
RestartSec=5
 
[Install]
WantedBy=multi-user.target

Load and enable the new unit

sudo systemctl daemon-reload
sudo systemctl enable --now myapp

#Overriding Without Modifying the Original File

Never edit a package's built-in unit file directly — a package update will overwrite your changes. Use an override:

Create an override for a unit

sudo systemctl edit myapp

Example override.conf

[Service]
Environment="PORT=8080"
Restart=always

#Resource Control with Cgroups v2

Systemd manages process resources through cgroups (control groups). This lets you limit a service's CPU, memory, and I/O — like a budget cap per department.

Override with resource limits

[Service]
MemoryMax=512M
CPUQuota=50%
MemoryHigh=384M

MemoryMax — the hard memory limit; processes exceeding it can be killed.
MemoryHigh — the soft limit; throttling happens before the hard limit.
CPUQuota=50% — caps usage at half of one core maximum.

Apply and check resource limits

sudo systemctl daemon-reload
sudo systemctl restart myapp
systemctl status myapp

#Journald: Centralized Logging

journald is systemd's logging daemon. It collects logs from the kernel and all services into one structured database at /var/log/journal/.

#Basic journalctl Commands

View the last boot's logs

journalctl -b

Filter logs per service

journalctl -u sshd --since "1 hour ago"

Follow logs in real time

journalctl -f -u myapp

-b limits to the current boot, -u filters per unit, --since grabs a time range, and -f follows the log like tail -f. The most common combination: journalctl -u <service> --since "30 min ago" for debugging.

#Finding Errors

Search for errors in the logs

journalctl -p err -b

Info

journald stores logs in a binary format that is rotated automatically. Older logs can be retained by setting SystemMaxUse in /etc/systemd/journald.conf. The interaction between journald, rsyslog, and logrotate will be covered in episode 12.

#Common Pitfalls

Editing a package's unit file directly. Use systemctl edit for overrides so your changes survive updates.
Forgetting daemon-reload after changing a unit file. Systemd won't read changes until reloaded.
Running services as root. Service units should run as a dedicated user with minimal privileges.
Relying on Restart=always without limits. Fast restart loops can eat resources — set RestartSec and consider on-failure.
Reading logs from old files. On modern AlmaLinux, the main logs are in journald — journalctl -u is faster than sifting through /var/log/messages.

#Conclusion

In this episode 7 you've mastered systemd and journald: the unit lifecycle with systemctl, boot targets, writing service units with overrides, resource control using cgroups v2, and effective log searching with journalctl.

Key takeaways:

Systemd is PID 1 and the manager of all services on AlmaLinux.
Master systemctl start/stop/restart/status and enable --now.
Main boot targets: multi-user.target for servers, graphical.target for desktops.
Write units in /etc/systemd/system/ and use systemctl edit for overrides.
Limit resources via MemoryMax and CPUQuota in cgroups v2.
Read logs with journalctl -u <service> --since and journalctl -p err -b.

Healthy services are the key to a stable system. In the next episode, Episode 8, we'll cover Storage, LVM & Filesystems — from lsblk, parted, mkfs, and /etc/fstab, to LVM management with lvextend and snapshots, plus Btrfs and Stratis as modern storage options. See you there!