Learn Elasticsearch - Elasticsearch Clients & Application Integration

#Introduction

Up to episode 24 you interacted with Elasticsearch through curl and Kibana. But in real products, what talks to Elasticsearch is the application — backend services that send data and receive search results. The way an application connects determines stability and performance in production. Episode 25 covers the official Elasticsearch clients (Java, Python, Node.js, .NET, Go), important integration patterns (connection pooling, retry, bulk, error handling, async), and best practices — index naming conventions, schema design, mapping explosion prevention, and version compatibility.

#Official Clients

Elastic provides official clients for the major languages, with a consistent API across all of them:

All clients speak HTTP to the same REST API — meaning everything you learned in this series (indexing, query DSL, aggregations) applies directly in any language.

#Example: Python Client

Index dan search dengan Python client

from elasticsearch import Elasticsearch
 
es = Elasticsearch("https://node1:9200", api_key="dUJ4dGVzdGtleQ==")
 
es.index(
    index="produk",
    id="1",
    document={"name": "Kaos Polos Premium", "price": 99000, "category": "fashion"},
)
 
resp = es.search(index="produk", query={"match": {"name": "kaos"}}, size=5)
for hit in resp["hits"]["hits"]:
    print(hit["_source"])

Notice the api_key — per episode 15, applications use API keys, not the elastic user.

#Connection Pooling

Creating an HTTP connection per request is wasteful. Every official client has built-in connection pooling: a pool of TCP connections that get reused. Configuration to pay attention to:

Atur connection pooling dan timeout

es = Elasticsearch(
    ["https://node1:9200", "https://node2:9200"],
    max_connections=50,
    timeout=30,
    retry_on_timeout=True,
    max_retries=3,
)

The host list above gives the client multiple nodes for automatic failover. max_connections must match the application's concurrency — too small limits throughput, too large burdens the nodes.

#Retry Strategies

Networks aren't perfect: requests can fail from timeouts, dropped connections, or busy nodes. Official clients have built-in retries, but a good retry strategy is selective:

The principle: retry only transient failures (network, overload), not permanent ones (invalid requests). Blind retries only make an already-busy node worse.

#Bulk from Applications

In applications, never index documents one by one — use bulk with the client helper pattern:

Bulk indexing dari aplikasi

from elasticsearch.helpers import bulk
 
actions = [
    {"_index": "produk", "_id": str(i), "_source": {"name": f"produk {i}", "price": i * 1000}}
    for i in range(10000)
]
success, failed = bulk(es, actions, chunk_size=1000)
print(f"{success} berhasil, {failed} gagal")

The bulk helper splits actions into chunks and handles per-action retries. This is the standard pattern for applications writing mass data — combine it with the replica 0 and refresh interval settings from episode 19 when doing large loads.

#Error Handling and Async

#Correct Error Handling

Every client has its own exception classes:

Menangani error spesifik

from elasticsearch import Elasticsearch
from elasticsearch.exceptions import NotFoundError, ConnectionTimeout
 
try:
    resp = es.get(index="produk", id="999")
except NotFoundError:
    print("dokumen tidak ada")
except ConnectionTimeout:
    print("koneksi timeout, coba lagi")

Catching specific exceptions instead of a generic Exception lets the application respond correctly — for example showing a 404 for a missing document, or a retry message for a timeout.

#Async Operations

For high-load applications, the clients support async mode. In Python: AsyncElasticsearch; in Node.js: the Promise-based API is async by default:

import { Client } from "@elastic/elasticsearch";
 
const client = new Client({ node: "https://node1:9200", auth: { apiKey: "dUJ4dGVzdGtleQ==" } });
 
const results = await client.search({
  index: "produk",
  query: { match: { name: "kaos" } },
});
console.log(results.hits.hits.map(h => h._source));

Async allows many requests to run in parallel without blocking threads — important in event-driven applications.

#Integration Best Practices

#Index Naming Convention

A good index naming pattern: domain-data-year.mm.dd — for example orders-2026.08.03. Consistent names make templates, data streams, and ILM easier (episodes 10-11). Avoid special characters and leading underscores.

#Schema Design

Design the mapping (episode 5) together with the application team, not after data has arrived. Ask for each field: what's the real type, does it need to be searchable, does it need to be aggregated, and how long does this data survive. Wrong schema decisions are cheap early, expensive at reindex time.

#Preventing Mapping Explosion

In applications, never write dynamic field names from user input (for example user.<id>.score) without limits. Use dynamic: strict in production and the total_fields.limit cap (episode 5). This lesson is as important in applications as in pipelines.

#Version Compatibility

Client minor versions may differ from the server, but avoid a major version gap. Elasticsearch 8.x is compatible with 7.17+ clients (the client promises backward compatibility), but the safest approach: follow the same major version and upgrade the client before the server during a major upgrade (episode 30). Check the cluster version with curl -s localhost:9200.

#Common Mistakes

1. Indexing one-by-one in a loop. Use the bulk helper with measured chunks.

2. Retrying on 400/409 errors. Only retry transient failures.

3. No connection pooling and timeouts. Client defaults are good, but tune them to your load.

4. API keys hardcoded in source code. Store them in a secret manager and environment.

5. Unbounded dynamic fields. Mapping explosion destroys clusters — limit from design time.

#Conclusion

In episode 25 you mastered application integration: official clients for Java, Python, Node.js, .NET, and Go; connection pooling, selective retry, bulk helper, error handling with specific exceptions, and async operations; plus best practices for index naming, schema design with the team, mapping explosion prevention, and version compatibility.

Key takeaways:

• Official clients exist for all major languages with a consistent API.
• Bulk helper for mass writes; selective retry only for transient failures.
• API keys are the only application credential — store them in a secret manager.
• Design the schema from the start and limit dynamic fields.
• Follow correct client-server version compatibility.

A connected application is part of a larger system. In episode 26 we'll cover Elasticsearch in microservices architecture: centralized logging across services, structured logging and trace IDs, distributed tracing with Elastic APM and OpenTelemetry, service maps, and service discovery patterns with Elasticsearch. See you there!