Learn AppArmor - Scale & Multi-host Management

#Introduction

In episode 19 you understood the AppArmor 5.x transition — parser versions, cache, and policy ABI all changing between releases. Now apply that understanding at real scale: not one server, but dozens to thousands of hosts. The question is no longer is the profile correct, but do all hosts use the same profile, equivalent kernels, and a safe rollout process.

Episode 20 covers AppArmor operations at fleet scale: distributing profiles with config management, version consistency, differences in kernel behavior between hosts, gradual complain-to-enforce rollout, and regression monitoring. This is the episode that bridges "a good profile" and "a profile that survives real production".

#The Problem at Scale: Drift and Inconsistency

The biggest enemy of security policy across many hosts isn't a bad profile — it's drift: the condition where each host is slightly different. One host still uses the old profile, another hasn't been updated, a third got a new kernel. Drift happens because there's a different way to change profiles besides the official process: someone edits /etc/apparmor.d manually, or runs aa-enforce on their own.

The principle you must hold: the profile file on a host is not the source of truth — the source of truth is the repository. The host is merely the rendered result of that repository. As long as this principle is held, drift can be eliminated.

#Distributing Profiles with Config Management

The standard tool for applying the principle above is config management. Both Ansible and SaltStack can copy profiles, then reload only the changed ones.

playbook.yml — deploy AppArmor profiles

- name: Deploy AppArmor profiles
  hosts: web_servers
  become: true
  tasks:
    - name: Copy profile files
      ansible.builtin.copy:
        src: "profiles/{{ item }}"
        dest: "/etc/apparmor.d/{{ item }}"
        owner: root
        group: root
        mode: "0644"
      loop:
        - usr.sbin.nginx
        - usr.bin.myapp
      register: copied
 
    - name: Reload changed profiles
      ansible.builtin.command:
        cmd: "apparmor_parser -r /etc/apparmor.d/{{ item.item }}"
      loop: "{{ copied.results }}"
      when: item.changed

apparmor/init.sls — Salt state

/etc/apparmor.d/usr.sbin.nginx:
  file.managed:
    - source: salt://apparmor/profiles/usr.sbin.nginx
    - user: root
    - group: root
    - mode: "0644"
 
reload apparmor profiles:
  cmd.run:
    - name: apparmor_parser -r /etc/apparmor.d/usr.sbin.nginx
    - onchanges:
      - file: /etc/apparmor.d/usr.sbin.nginx

Notice the same pattern in both tools: copy the file, then reload only what changed with apparmor_parser -r (replace). This idempotency matters — running a playbook or state many times must not produce side effects. This is the same pattern as episode 8 on includes and tunables: any small change must be traceable.

Tip

Avoid restarting the apparmor service to load new profiles on a large fleet. systemctl restart apparmor brings the whole policy down and back up — a window in which hosts are unprotected. Better to reload per profile with apparmor_parser -r, which swaps policy atomically without confinement downtime.

#The Same Profile Version Across the Fleet

Config management is only half the work. The second half is versioning: all hosts must use the same profile version, not whatever happened to spread around. The practices we recommend:

Store all profiles in a single repository, complete with abstractions and tunables (the episode 8 pattern).
Use tags or versions on deployment artifacts — not "latest".
Verify the checksum or hash of profiles after deploy to be sure the file contents are actually identical.
Rollback means returning to the old version in the repository, then redeploying — not restoring files from someone's memory.

This version consistency is what makes episode 18 work: the CI matrix tests exactly the same version that will be shipped to hosts.

#Differences in Kernel Behavior Between Hosts

Even a healthy fleet almost always runs several kernel versions at once — and AppArmor treats the kernel as part of the policy. Policy is compiled against the features the target kernel supports, and the cache is invalidated when features change. The available features can be inspected directly:

Write the current kernel's feature profile

sudo aa-features-abi /etc/apparmor.d/cache/.features

List the features the kernel supports

ls /sys/kernel/security/apparmor/features

The implications are real: a profile using new features (for example from the 5.x era, episode 19) fails to load on an old kernel that doesn't support them. That's why the multi-distro test matrix isn't just a formality — apparmor_parser -Q always compiles against the features of the kernel it runs on, so validation in a CI with a different kernel can mislead you.

The control strategy: make the kernel range consistent. If the fleet must run different kernel versions, make sure each version is tested in the matrix; if possible, pin the image or a minimum kernel across the fleet so AppArmor behavior is predictable.

#Gradual Rollout: Complain to Enforce

Once profiles are distributed, the question is: how do you activate enforce without burning the whole fleet? The answer is a phased rollout in four phases:

Canary complain — one host representing each workload type is moved to complain; observe denials for a few days.
Fleet complain — all hosts in complain; denials are collected as a baseline.
Canary enforce — canary hosts are enforced; make sure no critical new denials appear.
Fleet enforce — fix all issues from phases 2 and 3, then enforce the whole fleet.

Each phase switch uses the tools from episode 18:

Phases 1 and 2: set profiles to complain

for p in /etc/apparmor.d/usr.*; do
    sudo aa-complain "$p"
done

Phases 3 and 4: set profiles to enforce

for p in /etc/apparmor.d/usr.*; do
    sudo aa-enforce "$p"
done

Warning

Never enforce the whole fleet at once. If a single legitimate denial slips through — say a cache path that only appears at peak load — the failure spreads to every host within minutes, and rollback has to touch every host too. The canary phase isn't a pleasant process to go through; it's a safeguard far cheaper than a major incident.

#Regression Monitoring

The phases above are only useful if you have a way to measure whether the rollout is going well. Regression monitoring works by comparing the count and types of denials before vs after the rollout, per host:

Count new denials since today

journalctl -k --since today | grep -c 'apparmor="DENIED"'

Summary of profile status on the host

sudo aa-status | head -20

Recommended practice: record a denial baseline per profile before the rollout, then alert if a new class of denial appears after enforce — especially on the same profile across several hosts at once (a pattern that signals a systemic problem, not a coincidence). A quick per-host summary can be obtained from aa-status. Combine this with the monitoring from episode 15: a sudden spike in denials on one host might mean the application changed behavior — or worse, someone is trying to break the policy.

#Conclusion

In episode 20 you brought AppArmor to fleet scale: the repository as source of truth rendered to each host via Ansible or SaltStack, uniform profile versioning so the CI matrix means something, kernel difference control through feature ABIs, a four-phase rollout from canary complain to fleet enforce, and regression monitoring based on denial baselines.

Key points:

A file on a host is a render of the repository, not the source of truth.
Reload per profile with apparmor_parser -r, don't restart the apparmor service.
One profile version for the whole fleet, with rollback through the repository.
The kernel determines policy features — make it consistent or test them all.
Rollout must be phased: canary complain, fleet complain, canary enforce, fleet enforce.
Monitor regressions with a per-profile denial baseline.

In episode 21 we look ahead: Modern Features & Roadmap — the 4.1.6 and 4.1.7 maintenance releases, aa-notify improvements, Python 3.14 support, the container and Kubernetes ecosystem, and LTS kernel support status. See you there!