Learn Elasticsearch - Elastic Stack Integration (Kibana, Logstash, Beats)

#Introduction

Elasticsearch rarely stands alone. In practice it's part of the Elastic Stack — a complete ecosystem covering Kibana (visualization and exploration), Logstash (data processing), and Beats (lightweight data collectors). This combination is what's called "ELK" and has become the industry standard for logging and observability.

Episode 24 covers the integration of the three main components: Kibana for data exploration and dashboards, Logstash with the input-filter-output pipeline concept, and Beats — Filebeat, Metricbeat, Packetbeat, and Heartbeat — as lightweight data collectors.

#Kibana: Your Window to the Data

#Setup and Configuration

Kibana needs the exact same version as Elasticsearch (episode 0) and a dedicated user to connect:

server.host: "0.0.0.0"
elasticsearch.hosts: ["https://node1:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "secret"
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/kibana.crt
server.ssl.key: /etc/kibana/kibana.key

#Discover

Discover is the exploration entry point: you search documents with the query DSL, view raw data, create filters, and explore fields. This is where you "feel" the data before building visualizations.

#Visualizations, Dashboards, and Lens

Kibana offers several ways to build visualizations:

• Lens — the modern drag-and-drop UI, fastest for creating visualizations without writing manual aggregations. Recommended as the starting point.
• Classic visualizations — bar, line, pie, data table, and so on, with full control over aggregations.
• Dashboard — arranging many visualizations into one screen, with global filters applied to all panels at once.

#Canvas

Canvas creates infographic-style presentations — Elasticsearch data combined with visual elements, text, and images on one canvas. Great for executive reports and live displays in operations rooms.

#Logstash: The Data Pipeline

#Pipeline Concepts

Logstash processes data through a pipeline with three stages: input (where data comes from), filter (transformation), and output (where data is sent). A complete pipeline example:

input {
  file {
    path => "/var/log/nginx/access.log"
    start_position => "beginning"
  }
}
 
filter {
  grok {
    match => { "message" => "%{IPORHOST:client_ip} ..." }
  }
  date {
    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
  }
  mutate {
    remove_field => ["message"]
  }
}
 
output {
  elasticsearch {
    hosts => ["https://node1:9200"]
    index => "nginx-access-%{+YYYY.MM.dd}"
  }
}

This pipeline reads nginx logs, parses them with grok (episode 12), sets the date, discards the raw field, then writes to a date-based index.

#Input and Filter Plugins

The jdbc input is very useful for copying data from PostgreSQL/MySQL into Elasticsearch for search — the dual-write pattern we discussed in episode 1.

#Logstash vs Ingest Pipeline

Logstash and the ingest pipeline (episode 12) are often compared. Logstash is more powerful (more plugins, a complete filter language, multi-output) and suits complex data from many sources; the ingest pipeline is lighter, runs inside Elasticsearch, and suffices for simple transformations. In the modern Elastic Stack, many teams replace Logstash with data streams + ingest pipeline for simplicity — unless requirements demand Logstash.

#Beats: Lightweight Data Collectors

Beats are very lightweight single-purpose agents, installed on the machines whose data needs collecting. The main categories:

#Filebeat

filebeat.inputs:
  - type: filestream
    id: nginx-logs
    paths:
      - /var/log/nginx/access.log
      - /var/log/nginx/error.log
 
output.elasticsearch:
  hosts: ["https://node1:9200"]
  username: "beats_system"
  password: "secret"

Filebeat records its read position (the registry), so when the agent restarts, no logs are missed or read twice. Output can go directly to Elasticsearch, or through Logstash if heavy filtering is needed.

#Metricbeat

Collects system metrics (CPU, memory, disk) and other service metrics — including Elasticsearch itself (episode 21). One agent collects many modules: system, nginx, docker, elasticsearch, and so on.

#Packetbeat and Heartbeat

Packetbeat captures network packets to monitor application protocols (HTTP, DNS, MySQL) — analyzing performance and errors at the network transaction level. Heartbeat sends periodic probes to endpoints to make sure services are alive and responding — the basis of uptime monitoring.

#Common Mistakes

1. Different component versions. Always match the major version across the entire Elastic Stack.

2. Kibana connecting with the elastic user in production. Use kibana_system or a service account.

3. Logstash without a date filter. Data gets timestamped when processed, not when created.

4. Filebeat re-reading old logs. Configure start_position and the registry correctly.

5. All data through Logstash without a reason. The ingest pipeline is enough for simple transformations — simplify the architecture.

#Conclusion

In episode 24 you mastered Elastic Stack integration: Kibana with Discover, visualizations, dashboards, Lens, and Canvas; Logstash with the input-filter-output pipeline concept and the file, beats, http, jdbc, grok, mutate, and date plugins; and Beats — Filebeat for logs, Metricbeat for metrics, Packetbeat for network, and Heartbeat for uptime.

Key takeaways:

• Kibana is the interaction window; start from Discover, build with Lens.
• Logstash processes complex data with input-filter-output pipelines.
• Beats are lightweight agents for collecting logs, metrics, network, uptime.
• Data streams + ingest pipelines often suffice to replace Logstash.
• The whole stack's versions must match within one major version.

The whole stack is integrated — now how do your applications talk to Elasticsearch? In episode 25 we'll cover Elasticsearch clients and application integration: the official Java, Python, Node.js, .NET, and Go clients; integration patterns for connection pooling, retry, bulk, error handling, and async; and best practices for index naming, schema design, mapping explosion prevention, and version compatibility. See you there!