Learn MetalLB - Observability & Monitoring

#Introduction

MetalLB is a system that works silently — until a Service suddenly becomes unreachable and you ask what happened. The answers to almost all those questions are already provided by MetalLB in the form of Prometheus metrics, events, and logs. Episode 11 teaches you how to use all three to monitor and diagnose the cluster.

Observability isn't a nice-to-have; it's the front line of troubleshooting. With the right metrics, you can see a pool nearing capacity before a Service fails to allocate, or a flapping BGP session before traffic breaks. This proactive monitoring is what separates professional operations from reactive operations.

#Prometheus Metrics from Speaker and Controller

#The Metrics Endpoint on Port 7472

Both the speaker and the controller expose Prometheus metrics on port 7472. The format is standard Prometheus, so any Prometheus server can scrape it directly:

kubectl get pods -n metallb-system
kubectl port-forward svc/metallb-controller 7472:7472 -n metallb-system

After kubectl port-forward svc/metallb-controller 7472:7472 is running, open http://localhost:7472/metrics to see the metric list. Note that when scraped, the port is 7472 for the controller and 7472 for the speaker as well.

#The Most Useful Metrics

A few core metrics are commonly used:

• metallb_allocator_allocated_ip_total: the number of IPs currently allocated.
• metallb_allocator_addresses_in_use_total: IPs currently in use from a pool.
• metallb_allocator_addresses_total: the total IPs available in a pool.
• metallb_bgp_session_up: BGP session status (1 for up, 0 for down).
• metallb_speaker_announced_prefixes_total: the number of prefixes currently announced.

With metallb_allocator_addresses_in_use_total and metallb_allocator_addresses_total from the same pool, you can build an alert when a pool reaches, say, 90 percent capacity.

#Setting Up a ServiceMonitor for Prometheus

#Automatic Scraping with the Prometheus Operator

If your cluster uses the Prometheus Operator, the cleanest way is to create a ServiceMonitor. The MetalLB Helm chart provides Services and labels ready to use:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: metallb
  namespace: metallb-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: metallb
  endpoints:
    - port: metrics
      interval: 30s

This ServiceMonitor with the chart's built-in label selector targets both the controller and speaker metrics endpoints at once. Once applied and Prometheus reloads, MetalLB metrics start flowing into the TSDB.

#Checking Whether the Scrape Succeeded

To confirm the targets are scraped:

kubectl get servicemonitor -n metallb-system
curl -s http://localhost:7472/metrics | head -20

curl -s http://localhost:7472/metrics shows metric lines in Prometheus text format. If they're visible, the scraping configuration is correct.

#Grafana Dashboard

#Visualizing Allocation and Peering

Although there's no dashboard officially maintained by the MetalLB team on grafana.com, many community dashboards can be imported — search with the keyword metallb. The most useful panels are usually:

• IP usage per pool as a gauge.
• BGP session status per peer.
• The number of Services using LoadBalancer.
• Announcement rates per speaker.

curl -sL https://grafana.com/api/dashboards/14035/revisions/1/download > metallb-dashboard.json

curl -sL https://grafana.com/api/dashboards/14035/revisions/1/download downloads one of the popular community dashboards. After that, import the JSON file in Grafana via the Dashboards menu.

#Events and Logging for Diagnosis

#Events as a Narrative

Events are the most readable record of what happened. For a problematic Service, the events tell you what happened in sequence:

kubectl get events --all-namespaces | grep -i metallb
kubectl describe svc nginx

kubectl get events --all-namespaces | grep -i metallb shows every event involving MetalLB components — from IP allocation to the failures that occurred.

#Speaker and Controller Logs

When events aren't enough, logs are the next source. Speaker logs show announcement and peering details, while controller logs show allocation decisions:

kubectl logs -n metallb-system -l component=speaker --tail=50
kubectl logs -n metallb-system -l component=controller --tail=50

kubectl logs -n metallb-system -l component=speaker --tail=50 shows lines like Announcing 192.168.1.200 from node worker-1 or BGP peering errors. The combination of events and logs is the main diagnostic tool we'll use again in depth in episode 19.

#Conclusion

Episode 11 completes MetalLB observability: Prometheus metrics on port 7472, a ServiceMonitor for automatic scraping, Grafana dashboards for visualization, and events and logs as daily diagnostic tools.

Key takeaways:

• The speaker and controller expose Prometheus metrics on port 7472.
• metallb_allocator_addresses_in_use_total and addresses_total monitor pool capacity.
• metallb_bgp_session_up monitors BGP peering status.
• A ServiceMonitor simplifies scraping when using the Prometheus Operator.
• Events and speaker/controller logs are the main diagnosis source.
• Community Grafana dashboards can be imported for quick visualization.

In the next episode, episode 12, we'll discuss Ingress & Nginx integration — exposing the Ingress Controller as a LoadBalancer Service, comparing LoadBalancer versus NodePort for Ingress, and building an end-to-end flow from an application to Ingress and out through MetalLB.