Learn Elasticsearch - Snapshot & Restore - Backup Strategy

#Introduction

In episode 19 we nailed down performance — but performance means nothing if data can be lost. Operators lose data not because the technology is bad, but because their backups were never restore-tested. Snapshots are Elasticsearch's official backup mechanism, and this episode makes sure you can create, automate, and — most importantly — restore data with confidence. We'll cover snapshot concepts, repository types (filesystem, S3, GCS, Azure), creating repositories, snapshot lifecycle management (SLM), full and incremental snapshots, restore operations, partial and cross-cluster restore, and snapshot monitoring and security.

#Snapshot Concepts

A snapshot is a copy of an index (or the whole cluster) stored in a separate repository. Its important characteristics:

• Incremental — the second snapshot only stores what changed since the first; the first snapshot stores the full data. This is what makes repeated backups cheap.
• Independent of the cluster — snapshots live in external storage, surviving even a total cluster loss.
• Only committed data is backed up — snapshots don't capture the un-flushed translog; for real-time data, combine with replication (episode 22).

#Repository Types

The repository determines where snapshots are stored:

For production, cloud repositories (S3/GCS/Azure) are the standard — off-site, cheap, and automatically replicated. Make sure snapshots are in a different location from the cluster — a backup on the same machine isn't a backup.

#Creating a Repository

Example S3 repository (requires the repository-s3 plugin and credentials):

PUT /_snapshot/my-s3-repo

{
  "type": "s3",
  "settings": {
    "bucket": "my-es-backup-bucket",
    "region": "ap-southeast-1",
    "base_path": "elasticsearch/snapshots"
  }
}

Verify the repository is accessible:

GET /_snapshot/my-s3-repo/_verify

#Creating and Managing Snapshots

#Manual Snapshot

PUT /_snapshot/my-s3-repo/snapshot-2026.08.03?wait_for_completion=true

{
  "indices": "logs-*,produk",
  "ignore_unavailable": true,
  "include_global_state": false
}

ignore_unavailable skips indexes that don't exist; include_global_state stores cluster settings, templates, and ILM policies. ?wait_for_completion=true makes the request wait — for large snapshots, leave it false and monitor via _status.

#Snapshot Lifecycle Management (SLM)

SLM automates snapshots on a schedule — this is the heart of a hands-off backup strategy:

PUT /_slm/policy/nightly-backup

{
  "schedule": "0 30 2 * * ?",
  "name": "nightly-{now/d}",
  "repository": "my-s3-repo",
  "config": { "indices": ["logs-*", "produk", "users"] },
  "retention": { "expire_after": "30d", "max_count": 30, "min_count": 7 }
}

The schedule uses cron format (the expression above: 02:30 every day). Retention automatically deletes old snapshots by age and count. For compliance (episode 17), adjust expire_after to the organization's retention policy.

#Restore Operations

Restores don't happen automatically — a snapshot is only a backup. To restore an index from a snapshot:

POST /_snapshot/my-s3-repo/snapshot-2026.08.03/_restore

{
  "indices": "produk",
  "rename_pattern": "produk",
  "rename_replacement": "produk-restored",
  "include_global_state": false
}

Restores usually go to an index with a different name to avoid overwriting the active index — the rename_pattern/rename_replacement pattern is the most commonly used. After the data is verified, that's when the alias is moved (episode 4) to activate the restored index.

#Partial and Cross-Cluster Restore

#Partial Restore

A snapshot whose part of the data failed (for example one corrupt index) can still be restored with "partial": true — healthy indexes are recovered, damaged ones are skipped:

{
  "indices": "logs-*",
  "partial": true
}

#Cross-Cluster Restore

A snapshot doesn't belong to one cluster — another cluster can pull it. This is the basis of multi-region disaster recovery: the cluster in region A stores snapshots, the cluster in region B takes them during failover (episode 29). The way: register the same repository on the second cluster, then do a regular restore.

#Snapshot Monitoring and Security

GET /_snapshot/my-s3-repo/_status

GET /_snapshot/my-s3-repo/_all?verbose=false

The state column shows SUCCESS, FAILED, PARTIAL, or IN_PROGRESS. Set up alerts for any state other than SUCCESS (episode 21 covers alerting).

Snapshot security: cloud repositories use credentials configured in the Elasticsearch keystore (bin/elasticsearch-keystore add s3.client.default.access_key), not plaintext. Server-side encryption on the storage side is mandatory for sensitive data — and remember, snapshots contain copies of personal data, so they must be treated as strictly as the original data (episode 17).

#Common Mistakes

1. Repository in the same location as the data. A disaster destroys both — store off-site.

2. SLM without retention. Snapshots pile up without limit and burden storage costs.

3. Restoring to the same index name. Overwrites the active index — restore to a new name first, verify, then move the alias.

4. Never testing restores. A backup strategy is incomplete without periodic drills.

5. Snapshot credentials in plaintext. Use the Elasticsearch keystore.

#Conclusion

In episode 20 you mastered the backup strategy: the incremental snapshot concept, repository types (fs, S3, GCS, Azure), creating repositories, automation with SLM plus retention, restore operations with rename, partial restore, cross-cluster restore, and snapshot monitoring and security.

Key takeaways:

• Snapshots are incremental — the first is full, subsequent ones only deltas.
• Off-site repositories (S3/GCS/Azure) are the production standard.
• SLM automates scheduling and retention — monitor its failures.
• Restore to a new index name, verify, then move the alias.
• Periodic restore testing is the only proof a backup works.

Backups protect data — but how do you know the cluster is healthy before a problem occurs? In episode 21 we'll cover monitoring and observability: the cluster health API, node stats, index stats, cat APIs; JVM metrics, indexing and search rates, thread pool rejections, circuit breaker trips; and tools — Kibana Monitoring, alerting, Metricbeat, APM, and Prometheus/Grafana. See you there!