Learn LXC - Monitoring & Logging

#Introduction

You now have many containers, and "many" means they need eyes that keep watching. In episode 20 we build observability: viewing runtime status, monitoring per-container resources, connecting to popular monitoring systems (Prometheus/Zabbix), and reading logs correctly. Without this, you'll only know a container has a problem when a user reports it.

#Monitoring Status with lxc-info

#Runtime Status

The fastest command to see a container's state:

Runtime status and details

lxc-info -n c1
lxc-ls -f

lxc-info -n c1 -s shows a concise state summary — useful in scripts:

Check state only (for scripting)

lxc-info -n c1 -s

#State Polling Pattern

For a simple monitoring script:

Check whether the container is running

if [ "$(lxc-info -n c1 -s | awk '{print $2}')" = "RUNNING" ]; then
  echo "c1 UP"
else
  echo "c1 DOWN"
fi

#Monitoring Resources via cgroup

The true resource numbers live in the cgroup (episode 7). Read them directly for accurate values:

Read resource usage from the cgroup

cat /sys/fs/cgroup/lxc/c1/memory.current
cat /sys/fs/cgroup/lxc/c1/memory.max
cat /sys/fs/cgroup/lxc/c1/cpu.max

Interactive monitoring

systemd-cgtop

systemd-cgtop shows live CPU/memory usage per cgroup — the fastest tool for finding a container that's "eating" resources.

Tip

The most accurate way to measure "a container that has exceeded its limit" is comparing memory.current vs memory.max, and checking whether memory.events records oom_kill. A rising oom_kill count is a sure sign processes inside the container are being killed for exceeding the memory limit (episode 7).

#Prometheus Integration

#node_exporter per Container Strategy

The simplest way to monitor containers with Prometheus: run node_exporter inside each container, then have Prometheus scrape each container's IP. Containers using systemd can run it as a regular service.

Run node_exporter inside the container

lxc-attach -n c1 -- apt install prometheus-node-exporter
lxc-attach -n c1 -- systemctl enable --now prometheus-node-exporter

#Scrape Configuration in Prometheus

prometheus.yml (container targets)

scrape_configs:
  - job_name: "lxc-containers"
    static_configs:
      - targets:
          - "10.0.3.10:9100"   # c1
          - "10.0.3.11:9100"   # c2
        labels:
          group: "homelab"

#Useful Metrics

node_memory_MemAvailable_bytes vs the container limit.
node_cpu_seconds_total for load patterns.
node_filesystem_avail_bytes for disk inside the container.

#Alternative: blackbox/host-side

If you don't want an agent in every container, scrape from the host via cgroup metrics (using a custom exporter or node_exporter with --collector.cgroups). Less detailed, but lighter — choose according to your needs.

#Zabbix Integration

Zabbix offers a similar agent-based approach: install the Zabbix agent inside the container, add the host in Zabbix, and monitor standard OS metrics. For LXC monitoring from the host side, Zabbix can also use custom items that run lxc-info or read cgroup files.

Custom Zabbix item for container state

lxc-info -n c1 -s

Combine with alert triggers: "container DOWN", "memory > 90% limit", or "oom_kill > 0" — that's what makes monitoring useful, not just a pretty dashboard.

#Logging

#LXC Runtime Logs

LXC writes operational logs to /var/log/lxc/:

View LXC logs

ls /var/log/lxc/
tail -50 /var/log/lxc/c1.log

These logs contain traces of lxc-start, lxc-stop operations and configuration errors — the first lines to read when a container fails to start.

#Per-Container Journal

For systemd containers, service logs inside the container are stored in their own journal. From the host, you can read them directly:

Read a service journal inside the container

lxc-attach -n c1 -- journalctl -u sshd --no-pager -n 50

All logs since the container boot

lxc-attach -n c1 -- journalctl -b

#Container Boot Logs

The container boot log (init boot log) can be viewed via the console or lxc-info:

Boot log from the console

lxc-console -n c1

Press Ctrl+A then Q to exit the console back to the host.

Warning

LXC runtime logs (/var/log/lxc/) and container journals are different things: the former is about the LXC operations themselves (start/stop/config), the latter about services inside the container. When a "container fails to start", read /var/log/lxc/c1.log first; when a "service inside the container has a problem", read the container journal.

#Building a Simple Dashboard

A reliable minimum pattern:

Alert when the container state changes to STOPPED.
Alert when memory.current passes 90% of memory.max.
Alert when oom_kill increases.
Logs sent to a centralized system (e.g. host journald, then forwarded).

Check memory and oom events

cat /sys/fs/cgroup/lxc/c1/memory.events

#Closing

Key takeaways:

lxc-info -n c1 -s and lxc-ls -f are the fastest status checks.
Accurate resources are read from the cgroup; systemd-cgtop for a live view.
Prometheus: run node_exporter per container and scrape per IP.
Zabbix: an agent inside the container or custom items based on lxc-info.
Logs: /var/log/lxc/ for LXC operations, per-container journal for services, lxc-console for boot logs.

In the next episode 21 we'll cover roadmap & community — LXC's development direction (latest kernel, cgroup2, security, LTS 7.0 coordination through 2031) and the community channels: linuxcontainers.org, GitHub lxc/lxc, the discuss.linuxcontainers.org forum, and the #lxc IRC.