Learn Elasticsearch - Container Orchestration - Docker & Kubernetes

#Introduction

Most modern production Elasticsearch deployments run on top of containers — consistent, portable, and easy to automate. From a single docker run for experimentation to a multi-node Kubernetes cluster with an operator, this episode takes you across the entire spectrum of Elasticsearch containerization.

Episode 27 covers Docker deployment (image, environment variables, volumes), Docker Compose for multi-node clusters, then Kubernetes — especially ECK (Elastic Cloud on Kubernetes), StatefulSets, PVCs, services and ingress, ConfigMaps and Secrets — plus production considerations: resource limits, anti-affinity, init containers, and probes.

#Docker Deployment

#Official Image and Environment Variables

The official image is at docker.elastic.co/elasticsearch/elasticsearch:8.x. Configure Elasticsearch via environment variables (ES_JAVA_OPTS for heap, discovery.type for single-node mode):

docker run -d --name es-lab \
  -p 9200:9200 \
  -e "discovery.type=single-node" \
  -e "xpack.security.enabled=false" \
  -e "ES_JAVA_OPTS=-Xms2g -Xmx2g" \
  docker.elastic.co/elasticsearch/elasticsearch:8.15.3

#Multi-Node Docker Compose

For a multi-node cluster on a local machine, Compose is the most practical tool:

services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.15.3
    environment:
      - cluster.name=compose-cluster
      - node.name=es01
      - node.roles=master,data
      - discovery.seed_hosts=es02
      - cluster.initial_master_nodes=es01,es02
      - bootstrap.memory_lock=true
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
    volumes:
      - es01-data:/usr/share/elasticsearch/data
    ulimits:
      memlock: { soft: -1, hard: -1 }
    ports:
      - "9200:9200"
 
  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.15.3
    environment:
      - cluster.name=compose-cluster
      - node.name=es02
      - discovery.seed_hosts=es01
      - cluster.initial_master_nodes=es01,es02
    volumes:
      - es02-data:/usr/share/elasticsearch/data
 
volumes:
  es01-data:
  es02-data:

Run it with docker compose up -d. bootstrap.memory_lock=true and ulimits.memlock prevent JVM swapping — important for performance. In production, watch vm.max_map_count on the host (minimum 262144): sudo sysctl -w vm.max_map_count=262144.

#Volume Management

Volumes store data outside the container so data survives container restarts/recreates. Never store Elasticsearch data on the container's filesystem — containers are ephemeral; volumes (named volumes or persistent volumes in Kubernetes) are the only safe place for data.

#Kubernetes Deployment

#ECK: Elastic Cloud on Kubernetes

ECK is Elastic's official Kubernetes operator — you declare the Elasticsearch cluster as a custom resource, and the operator realizes it (deploying the StatefulSet, PVCs, service, and security setup) automatically:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: production-cluster
spec:
  version: 8.15.3
  nodeSets:
    - name: hot
      count: 3
      config:
        node.roles: [master, data_hot, ingest]
      podTemplate:
        spec:
          containers:
            - name: elasticsearch
              resources:
                requests: { memory: 8Gi, cpu: "4" }
                limits: { memory: 8Gi, cpu: "4" }
              env:
                - name: ES_JAVA_OPTS
                  value: "-Xms4g -Xmx4g"
      volumeClaimTemplates:
        - metadata: { name: elasticsearch-data }
          spec:
            accessModes: ["ReadWriteOnce"]
            resources:
              requests: { storage: 500Gi }
            storageClassName: fast-ssd

The operator handles the painful manual parts: TLS certificates, user creation, and rolling upgrades. This is the most recommended way to run Elasticsearch on Kubernetes.

#StatefulSets, PVCs, Services, and Ingress

Without an operator, Elasticsearch runs as a StatefulSet — a stateful workload that suits databases: each pod gets a stable identity (es-0, es-1) and its own PersistentVolumeClaim (PVC), so data survives pod recreates. The volumeClaimTemplates above create one PVC per pod automatically.

Kubernetes Services expose Elasticsearch inside the cluster; Ingress manages external access with TLS. ECK also creates services for Kibana and APM automatically.

#ConfigMaps and Secrets

• ConfigMaps hold non-secret configuration (for example extra elasticsearch.yml).
• Secrets hold credentials — API keys, passwords — which Kubernetes encrypts at rest.

Never put credentials in a ConfigMap; that's plaintext. Reference secrets via envFrom or volume mounts.

#Production Considerations

Example probes:

readinessProbe:
  httpGet:
    path: /_cluster/health?local=true
    port: 9200
    scheme: HTTPS
  initialDelaySeconds: 20
  periodSeconds: 10
livenessProbe:
  httpGet:
    path: /_cluster/health?local=true
    port: 9200
    scheme: HTTPS
  initialDelaySeconds: 60
  periodSeconds: 20

#Common Mistakes

1. No persistent volume. Data is lost on pod restart — PVCs/volumes are mandatory.
2. Memory lock not enabled. Swapping tanks performance — set memlock and ulimits.
3. vm.max_map_count too small. max virtual memory areas error at container startup.
4. Resource limits without requests. Oversubscription destroys nodes — make requests and limits equal.
5. No anti-affinity. All pods on one host = one host down = cluster dead.

#Conclusion

In episode 27 you mastered container orchestration: the official Docker image with environment variables, multi-node Docker Compose with volumes and memory lock, Kubernetes deployment with ECK, StatefulSets and PVCs, services and ingress, ConfigMaps and Secrets, and production considerations — resource limits, anti-affinity, init containers, and probes.

Key takeaways:

• Volumes/PVCs are mandatory — data never lives on the container filesystem.
• Compose for multi-node labs; ECK for production Kubernetes.
• Resources requests = limits; mind memlock and vm.max_map_count.
• Anti-affinity spreads pods so one host can't take down the cluster.
• Probes manage the pod lifecycle; ECK automates security and upgrades.

The infrastructure is containerized — now how do you manage it automated and documented? In episode 28 we'll cover CI/CD integration and infrastructure as code: index templates, ILM policies, and ingest pipelines as code; automated query testing, schema validation, and performance regression; and Terraform, Ansible, GitOps, and configuration drift detection. See you there!