Learn Observability with the LGTM Stack - CI/CD & Infrastructure as Code

#Introduction

Dashboards edited by clicking through the UI are easily broken and hard to trace. In a dynamic environment, observability must be managed like code: committed to git, reviewed, tested, then deployed automatically. This is the core of GitOps for observability.

This episode covers GitOps principles, using the Grafana Terraform provider, integrating testing into CI/CD, and Jsonnet and Grafonnet for programmable dashboards.

#GitOps for Observability

#Dashboards and Alert Rules as Code

• Dashboard as code: dashboards are stored as JSON or Jsonnet in a repository.
• Alert rules as code: alert rules are written in Terraform or YAML, not clicked in the UI.
• Configuration versioning: every change is recorded and can be rolled back.
• Automated deployment: changes to the main branch are automatically applied to Grafana.

git commit -> review -> merge -> apply to Grafana

The git commit -> review -> merge -> apply pattern ensures no change slips through without a trace.

#Grafana Terraform Provider

#Managing Grafana as Code

The Grafana Terraform provider manages data sources, dashboards, and alert rules:

provider "grafana" {
  url  = "http://localhost:3000"
  auth = var.grafana_api_token
}
 
resource "grafana_data_source" "mimir" {
  type = "prometheus"
  name = "Mimir"
  url  = "http://mimir:9009/prometheus"
}

The block resource "grafana_data_source" "mimir" creates the Mimir data source declaratively — it can be recreated in any environment.

#Dashboards and Alert Rules

• Dashboard provisioning: dashboards are uploaded from JSON files to Grafana.
• Data source management: all data sources are managed as resources.
• Alert rules deployment: alert rules are managed via grafana_rule_group.

terraform init
terraform apply -auto-approve

The terraform apply -auto-approve command applies the entire observability configuration to Grafana.

#CI/CD Integration

#Automated Testing

• Automated testing of dashboards: verify dashboard JSON is valid before merging.
• Alert rule validation: make sure every alert rule contains a valid query.
• PromQL/LogQL/TraceQL linting: check query syntax automatically.
• Synthetic monitoring: automatic probes monitoring application endpoints.

steps:
  - name: Lint dashboard
    run: jq empty dashboards/*.json
  - name: Validate alert rules
    run: terraform validate

The Lint dashboard stage in the pipeline rejects pull requests with broken dashboard JSON from the start.

#Pipeline Security

Never put Grafana tokens in the repository. Use secrets from the CI environment and limit the service account scope.

#Jsonnet and Grafonnet

#Programmable Dashboards

Jsonnet is a templating language that extends JSON, and Grafonnet is a Jsonnet library for Grafana dashboards:

local grafana = import "grafonnet/grafana.libsonnet";
 
{
  grafanaDashboard('Checkout Overview')
  .addTimeSeries('Request Rate', { datasource: '$Mimir' })
}

The code grafanaDashboard('Checkout Overview') builds dashboards from reusable components.

#Grafonnet Benefits

• Reusable dashboard components: one panel block reused across many dashboards.
• Templated dashboards: variables and panels generated from data.
• Dashboard generation: dozens of dashboards generated from one template.

#Closing

In episode 30 you understood GitOps principles with dashboards and alert rules as code, using the Grafana Terraform provider for data sources and dashboards, automated testing and linting in CI/CD pipelines, and Jsonnet and Grafonnet for programmable dashboards.

The key takeaways:

• Dashboards and alert rules are managed via git, not UI clicks.
• Terraform declares data sources and alert rules.
• Pipelines test, lint, and apply configuration.
• Grafonnet enables reprogrammable dashboards.
• Never commit Grafana credentials.

In the next episode 31 we'll discuss advanced Grafana features — unified alerting, the Explore interface, transformations and overrides in dashboards, the plugin ecosystem, and enterprise features. The Grafana you've been using is far deeper than it looks.