Learn Elasticsearch - Machine Learning Features

#Introduction

Up to episode 22 we used Elasticsearch to search and analyze data. But Elasticsearch also has machine learning that runs inside the engine itself — without a separate ML pipeline. Episode 23 covers all three: anomaly detection, data frame analytics for outlier detection, regression, and classification, and the 8.x NLP features — ELSER, semantic search, and text embeddings integration.

#Anomaly Detection

#Concepts

Anomaly detection learns the normal patterns of time-series data, then warns when those patterns are violated. Unlike static thresholds ("CPU above 90%"), ML learns the context: a traffic spike at night might be normal for e-commerce, but an anomaly for an internal tool. The model uses an anomaly score (0–100) — the higher, the more unusual.

Practical uses: detecting error rate spikes, sudden sales drops, rising API latency, or security attacks with deviant patterns.

#Single Metric vs Multi Metric Jobs

A single metric job monitors one metric — the simplest:

{
  "job_id": "cpu-single-metric",
  "analysis_config": {
    "detectors": [
      {
        "function": "mean",
        "field_name": "cpu.usage",
        "partition_field_name": "host.name"
      }
    ]
  },
  "data_description": { "time_field": "@timestamp" }
}

A multi metric job monitors several metrics in one model, with influencers — fields that help explain why an anomaly happened:

{
  "job_id": "anomali-layanan",
  "analysis_config": {
    "detectors": [
      { "function": "count", "partition_field_name": "service.name" },
      { "function": "mean", "field_name": "latency", "partition_field_name": "service.name" },
      { "function": "sum", "field_name": "error_count", "partition_field_name": "service.name" }
    ],
    "influencers": ["service.name", "host.name", "ip"]
  }
}

Influencers answer "anomaly in whom?" — when an ML alert arrives, you immediately know which service and host is having trouble, not just "something is off".

#Population Analysis

Population analysis compares one entity against the whole population — for example "which host behaves differently from all other hosts":

{
  "job_id": "populasi-host",
  "analysis_config": {
    "detectors": [
      {
        "function": "mean",
        "field_name": "cpu.usage",
        "over_field_name": "host.name"
      }
    ]
  },
  "data_description": { "time_field": "@timestamp" }
}

over_field_name tells ML to compare each host against the population baseline. This is very powerful for fleet monitoring — finding the one server that's "odd" among hundreds of normal ones.

#Data Frame Analytics

Data frame analytics build statistical models from data to answer "what will happen" or "what's odd":

#Outlier Detection

Finds documents that deviate statistically from the majority of the data — without labels. Example: finding transactions with unusual patterns compared to other transactions:

{
  "source": { "index": "transaksi" },
  "analysis": {
    "outlier_detection": {
      "include_feature_importance": true
    }
  }
}

The result adds a ml.outlier_score field (0–1) to each document — just filter it to find suspicious transactions.

#Regression and Classification

Regression predicts continuous values (price, latency, age); classification predicts categories (fraud or not, churn or not). Both need labeled data:

{
  "source": { "index": "produk" },
  "analysis": {
    "regression": {
      "dependent_variable": "price",
      "training_percent": 75
    }
  }
}

75% of the data for training, 25% for validation. Results include accuracy evaluation and feature importance — which fields most influence predictions. Both models are used for prediction via the _ml/inference API, or exported for use in an ingest pipeline.

#NLP Features (Elasticsearch 8.x)

#ELSER and Semantic Search

Semantic search searches by meaning, not just words. The keyword "flu burung" (bird flu) can find documents about "influenza aviary" — something impossible for regular full-text. Elasticsearch 8.x does this via ELSER (Elastic Learned Sparse EncodR), an NLP model that runs directly on ML nodes:

{
  "mappings": {
    "properties": {
      "content_embedding": {
        "type": "sparse_vector",
        "inference_id": "my-elser-model"
      }
    }
  }
}

When documents are indexed, the text is turned into a sparse vector (a sparse vector that captures concepts), then semantic searches use the text_expansion query. The result: search that understands context and synonyms without a manual synonym list (episode 7).

#Text Embeddings and LLM Integration

For external embedding models, Elasticsearch 8.x has the dense_vector data type and the knn query (k-nearest neighbors):

{
  "mappings": {
    "properties": {
      "content_embedding": {
        "type": "dense_vector",
        "dims": 384,
        "index": true,
        "similarity": "cosine"
      }
    }
  }
}

The dense vector + knn query combination enables hybrid search: combining BM25 relevance and vector similarity in one result — the basic pattern of RAG (Retrieval-Augmented Generation) for chatbots that answer from company documents, opening Elasticsearch as a knowledge base for AI applications.

#Conclusion

In episode 23 you mastered the machine learning features: anomaly detection with single metric, multi metric (plus influencers), and population analysis; data frame analytics for outlier detection, regression, and classification with feature importance; and the 8.x NLP features — ELSER for semantic search, dense_vector, and hybrid search patterns for AI applications.

Key takeaways:

• Anomaly detection learns normal patterns, then flags deviations.
• Influencers explain "where" the anomaly happened.
• Population analysis compares entities against the population.
• Data frame analytics need labeled data for regression/classification.
• ELSER and dense_vector unlock semantic search and LLM integration.

ML lets data "think" — now it's time to unite the whole ecosystem. In episode 24 we'll cover Elastic Stack integration: Kibana for Discover, visualizations, dashboards, Canvas, and Lens; Logstash with input-filter-output pipelines plus the jdbc and grok plugins; and Beats — Filebeat, Metricbeat, Packetbeat, and Heartbeat. See you there!