Learn Elasticsearch - CI/CD Integration & Infrastructure as Code

#Introduction

Have you ever done repetitive "manual setup"? Elasticsearch configuration done by clicking and typing in a console is undocumented, unreviewable, and un-rollbackable. In a healthy organization, all configuration is treated like code: versioned, reviewed, tested, and deployed through a pipeline. Episode 28 covers this approach — index management as code, CI/CD for testing and deploying Elasticsearch changes, and infrastructure as code with Terraform, Ansible, and GitOps, finishing with configuration drift detection.

#Index Management as Code

Everything we discussed as "manual requests" in previous episodes can be stored as files in a repository:

• Index templates (episodes 4, 11) — mapping and settings definitions.
• ILM policies (episode 10) — index lifecycles.
• Ingest pipelines (episode 12) — data preprocessing.
• Roles and role mappings (episode 15) — security.

These files are reviewed via pull requests, tested on a staging cluster, then applied to production. An example template as a YAML file:

---
index_patterns: ["logs-*"]
data_stream: {}
template:
  settings: { number_of_shards: 2, number_of_replicas: 1, index.lifecycle.name: logs-policy }
  mappings:
    properties:
      "@timestamp": { type: date }
      level: { type: keyword }
      message: { type: text }

#CI/CD Pipeline for Elasticsearch

#Automated Query and Schema Testing

CI can run a series of tests against a temporary Elasticsearch instance (for example using a docker compose lab cluster, episode 27) — first make sure the cluster is healthy with curl -s localhost:9200/_cluster/health:

steps:
  - name: Setup cluster test
    run: docker compose up -d
  - name: Terapkan template
    run: curl -sS -X PUT localhost:9200/_index_template/logs-template --data-binary @index-template-log.yaml
  - name: Jalankan integration test
    run: python3 -m pytest tests/test_search.py

Sensible tests:

• Schema validation — make sure the deployed mapping matches expectations (fields, types, dynamic policy).
• Query tests — run the query DSL used by applications; make sure they don't error and the results make sense.
• Performance regression — compare query execution time before/after changes; detect queries that suddenly slow down.

#Performance Regression Testing

Measure a baseline with a fixed dataset, then compare: if a query that took 50 ms becomes 500 ms after a mapping change, the pipeline should fail before the change reaches production. This requires a benchmark cluster with representative data — not just "it runs".

#Deployment Strategy

Deploying Elasticsearch changes to production follows a safe order:

1. Deploy to staging, run tests and benchmarks.
2. Deploy non-breaking templates/ILM first.
3. For breaking changes (mapping): create a new index + reindex + move the alias (episode 13).
4. Monitor metrics (episode 21) before and after.
5. Always have a rollback plan — the alias can be reverted instantly.

#Infrastructure as Code

#Terraform with the Elasticsearch Provider

Terraform declares infrastructure — including Elasticsearch — as code. The community provider elastic/elasticsearch supports index, template, ILM, and user/role resources:

terraform {
  required_providers {
    elasticsearch = { source = "elastic/elasticsearch", version = "~> 8.0" }
  }
}
 
provider "elasticsearch" {
  url         = "https://node1:9200"
  username    = "terraform"
  password    = var.es_password
  insecure    = false
}
 
resource "elasticsearch_index_template" "logs" {
  name = "logs-template"
  body = jsonencode({
    index_patterns = ["logs-*"]
    template = { settings = { number_of_shards = 2, number_of_replicas = 1 } }
  })
}

With terraform plan, you see exactly what will change before terraform apply — and Terraform state detects anything differing from the declaration (drift).

#Ansible Playbooks

Ansible is great for machine configuration and bootstrap — for example setting vm.max_map_count, installing plugins, and applying elasticsearch.yml:

- name: Siapkan node Elasticsearch
  hosts: es_nodes
  become: yes
  tasks:
    - name: Set vm.max_map_count
      sysctl: { name: vm.max_map_count, value: "262144" }
    - name: Pasang repository Elastic
      yum_repository: { name: elastic, state: present }
    - name: Install elasticsearch
      dnf: { name: elasticsearch, state: present }
    - name: Terapkan elasticsearch.yml
      template: { src: elasticsearch.yml.j2, dest: /etc/elasticsearch/elasticsearch.yml }
      notify: restart elasticsearch

The elasticsearch.yml.j2 templating pattern lets one playbook serve many environments with different values.

#GitOps Workflows

GitOps treats git as the single source of truth: changes happen via pull requests, and a tool (Argo CD, Flux, or a custom pipeline) syncs the git state to the live environment. The benefits: every change is recorded in git, there's review, there's an audit trail, and drift is immediately detected because synchronization is automatic.

For Elasticsearch, GitOps means: the repository contains templates, ILM, pipelines, roles — and the pipeline applies them to the cluster when there's a commit on the main branch. Manual cluster changes that aren't in git will be reverted by the next sync.

#Configuration Drift Detection

Configuration drift happens when the cluster diverges from what's declared — for example someone changes settings via a manual console. How to detect it:

GET /_index_template/logs-template?filter_path=*.index_patterns,*.template

A common pattern: a routine pipeline fetches the cluster state (templates, ILM, roles) and compares it with the files in git — if different, the pipeline fails or restores it. Terraform (terraform plan) and GitOps sync both provide this mechanism.

#Common Mistakes

1. Configuration only in the console. Undocumented and unreviewable — treat it as code.

2. Deploying breaking changes directly. New mappings must go through reindex + alias, not a direct overwrite.

3. No performance regression tests. Slow queries reach production unnoticed.

4. No drift detection. The cluster drifts from git without anyone knowing.

5. Secrets in the repository. Passwords and API keys in git mean leakage — use secret managers and pipeline variables.

#Conclusion

In episode 28 you mastered CI/CD and infrastructure as code: index templates, ILM policies, and ingest pipelines as versioned files; automated testing (schema validation, query tests, performance regression) in the pipeline; gradual deployment strategies; and Terraform, Ansible, GitOps, and configuration drift detection.

Key takeaways:

• All configuration is code — versioned, reviewed, tested.
• Breaking schema changes need a migration plan (reindex + alias).
• CI tests queries, schema, and performance before deployment.
• Terraform/GitOps declare the desired state and detect drift.
• Never put secrets in a repository.

Documented and tested configuration makes the system more resilient. But true resilience needs one more layer. In episode 29 we'll cover high availability and disaster recovery: multi-node design with master quorum anti-split-brain, data node redundancy, zone-aware replica allocation; multi-region strategies, RPO and RTO planning, failover procedures, and periodic DR testing. See you there!