Learning GitOps - FluxCD - Notifications & Alerts

#Introduction

In episode 13 you learned image automation — FluxCD monitors the registry and updates Git automatically. All that automation is interesting, but automation without visibility can be dangerous. How do you know a new image has landed? How do you know a sync failed at night? The answer is in notifications.

Episode 14 covers the FluxCD event system and how to turn it into alerts that reach the team: starting from the event concept, Provider, the Alert CRD, notification examples, webhook receivers, up to custom templates.

#The Event System in FluxCD

Every FluxCD controller produces an event when something happens: sync succeeded, sync failed, artifact updated, health check failed, and so on. All events are collected by the notification-controller, the component that also handles providers and receivers.

Every event carries metadata: severity, timestamp, reason, the producing component, and the related object's information. This metadata is the raw material that the Alert filters and routes.

#Provider

A Provider is a definition of where notifications are sent. The notification-controller supports many provider types:

Provider credentials are stored in a Secret, not directly in the Provider. An example for Slack:

apiVersion: v1
kind: Secret
metadata:
  name: slack-url
  namespace: flux-system
stringData:
  address: https://hooks.slack.com/services/xxxxx

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: slack
  namespace: flux-system
spec:
  type: slack
  channel: gitops
  secretRef:
    name: slack-url

A Provider is only active in the same namespace as the Alert that references it. The same principle applies to Teams, Discord, and other providers — only the type value and the Secret contents differ.

#The Alert CRD

An Alert connects events from a specific source with a specific provider, while also filtering out unimportant events. The main components of an Alert:

• eventSources — the objects whose events are monitored, e.g. Kustomization, HelmRelease, GitRepository
• eventSeverity — the severity level that is forwarded, info or error
• providerRef — reference to the destination Provider
• inclusionList and exclusionList — filters based on the event reason
• eventMetadata — additional metadata attached to the notification

An example Alert for deployment failures:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: deploy-failure
  namespace: flux-system
spec:
  providerRef:
    name: slack
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      name: "*"
    - kind: HelmRelease
      name: "*"

The Alert above forwards all error severity events from every Kustomization and HelmRelease to the Slack channel gitops.

#Filtering with the Exclusion List

For busy channels, filter out events that are too noisy:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: discord-alerts
  namespace: flux-system
spec:
  providerRef:
    name: discord
  eventSeverity: info
  eventSources:
    - kind: Kustomization
      name: "*"
  exclusionList:
    - "reason=ReconciliationSucceeded"
    - "reason=ArtifactUpToDate"

That way the Discord channel only receives events that are truly informative.

#Notification Examples

#Deployment Success and Failure

An Alert with eventSeverity: error like the example above catches failures. For success notifications, create a separate Alert with severity info and an exclusion list for noisy reasons.

#Source Updates

Monitor GitRepository so the team knows when a new artifact is produced:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: source-updates
  namespace: flux-system
spec:
  providerRef:
    name: slack
  eventSeverity: info
  eventSources:
    - kind: GitRepository
      name: "*"
    - kind: OCIRepository
      name: "*"

#Health Status Changes

Health check failures from a Kustomization can be filtered specifically:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: health-checks
  namespace: flux-system
spec:
  providerRef:
    name: teams
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      name: "*"
  inclusionList:
    - "reason=HealthCheckFailed"

#Strategy Summary

#Webhook Receiver

Besides sending notifications out, FluxCD can also receive incoming events through a Receiver. A Receiver is useful for speeding up reconciliation: on a GitHub push, the webhook triggers Flux to sync immediately without waiting for the interval.

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
  name: github-receiver
  namespace: flux-system
spec:
  type: github
  events:
    - ping
    - push
  secretRef:
    name: receiver-token
  resources:
    - kind: GitRepository
      name: "*"

FluxCD supports many receiver types: github, gitlab, bitbucket, harbor, and generic. For the GitHub type, configure the webhook in the GitHub repository settings to point at the Receiver URL exposed by FluxCD.

#Template and Variable Substitution

The default notification message is enough for many cases, but sometimes additional context is needed. Alerts support eventMetadata which attaches extra information to every event:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: webhook-alerts
  namespace: flux-system
spec:
  providerRef:
    name: webhook
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      name: "*"
  eventMetadata:
    cluster: prod-eu
    team: platform

Providers of type webhook can receive a payload containing that metadata, so the receiving system can process it further, for example grouping alerts by cluster.

#Closing

Episode 14 explained how to make FluxCD talk to the team through targeted notifications.

The key takeaways:

• All controllers produce events and the notification-controller routes them.
• Provider defines the notification destination, and credentials are stored separately in a Secret.
• The Alert CRD connects events with a provider plus severity, reason, and source filters.
• Receiver allows Flux to be triggered from outside, for example a GitHub push, for instant sync.
• Metadata makes notifications more informative for an on-call team.

In the next episode, episode 15, we enter a new phase: Progressive Delivery with Flagger — how to deploy a new version with minimal risk using canary, A/B testing, and blue-green automatically. See you!