Learn Elasticsearch - Text Analysis & Analyzers

#Introduction

In episode 6 you used the match query — but have you ever asked why match: "kaos" can find a document with "KAOS POLOS" in capital letters? The answer isn't in the query, but in the layer that runs before data is indexed: text analysis. This is what makes Elasticsearch far smarter than plain string matching. Episode 7 dissects the analysis process: character filters, tokenizers, and token filters; built-in analyzers; how to test an analyzer with the _analyze API; creating a custom analyzer; normalizers; and real-world use cases — autocomplete, case-insensitive search, stemming, and synonyms.

#The Analysis Process

When a field of type text is indexed, its text passes through three sequential stages:

1. Character filter — processes raw characters before splitting: removes HTML tags, converts symbols like & into "and".
2. Tokenizer — splits text into tokens based on rules: whitespace, words, or n-grams.
3. Token filter — modifies the resulting tokens: lowercasing, removing stop words, combining synonyms, or reducing to root words.

char filter  : KAOS POLOS Premium
tokenizer    : [KAOS] [POLOS] [Premium]
token filter : [kaos] [polos] [premium]

The order is always: char filter → tokenizer → token filter. The combination of the three is called an analyzer.

#Built-in Analyzers

standard is the default for text fields. For specific languages, there are dedicated analyzers such as english which also applies stemming — for example english turns "running" and "runs" into the root "run".

#Testing an Analyzer with the _analyze API

Before applying an analyzer to a mapping, always test it first via the POST /_analyze endpoint — it shows exactly how text is processed:

{ "analyzer": "standard", "text": "Kaos POLOS Premium!" }

{
  "tokens": [
    { "token": "kaos", "start_offset": 0, "end_offset": 4, "type": "<ALPHANUM>", "position": 0 },
    { "token": "polos", "start_offset": 5, "end_offset": 10, "type": "<ALPHANUM>", "position": 1 },
    { "token": "premium", "start_offset": 11, "end_offset": 18, "type": "<ALPHANUM>", "position": 2 }
  ]
}

Notice: the ! is dropped and all tokens are lowercased. When search results later feel off, _analyze is the first debugging tool you should reach for.

#Creating a Custom Analyzer

Built-in analyzers don't always fit. For example, for a clothing store we want to: strip dashes ("kaos-polos" becomes "kaos polos"), split on whitespace, and lowercase. The custom analyzer is defined in the index settings:

{
  "settings": {
    "analysis": {
      "char_filter": { "strip_dash": { "type": "pattern_replace", "pattern": "-", "replacement": " " } },
      "analyzer": {
        "produk_analyzer": { "type": "custom", "char_filter": ["strip_dash"], "tokenizer": "whitespace", "filter": ["lowercase"] }
      }
    }
  }
}

Then use this analyzer in the mapping:

{
  "mappings": {
    "properties": { "name": { "type": "text", "analyzer": "produk_analyzer" } }
  }
}

Important rule: the index analyzer (when data is ingested) and the search analyzer (when a query runs) don't have to be the same, but for consistent results they should be. You can set a separate search_analyzer if you really need to.

#Normalizers and N-gram

#Normalizers for Keyword Fields

keyword fields are not analyzed — but there's an exception: the normalizer. A normalizer applies only character filters and token filters (no tokenizer), so it's perfect for case-insensitive exact matching:

{
  "settings": {
    "analysis": {
      "normalizer": {
        "lowercase_normalizer": { "type": "custom", "filter": ["lowercase"] }
      }
    }
  },
  "mappings": {
    "properties": {
      "kode_produk": { "type": "keyword", "normalizer": "lowercase_normalizer" }
    }
  }
}

With this, a term query on kode_produk matches both "KAOS-001" and "kaos-001" — without sacrificing keyword performance.

#N-gram and Edge N-gram

N-gram splits text into consecutive letter chunks. edge_ngram only takes chunks from the start of a word. This is the classic technique for autocomplete: when a user types "kao", the index built with edge n-grams already has the tokens k, ka, kao, kaos — so responses come instantly.

{
  "settings": {
    "analysis": {
      "tokenizer": {
        "autocomplete_tokenizer": {
          "type": "edge_ngram", "min_gram": 2, "max_gram": 10,
          "token_chars": ["letter", "digit"]
        }
      }
    }
  }
}

Analogy: the search_analyzer here usually stays standard, while the indexing analyzer uses edge n-gram. That way, a search for "kaos" isn't limited in character count, but the index is already ready for predictions as the user types.

#Real-World Use Cases

Synonyms are configured with a synonym token filter that holds the mapping list:

{
  "filter": {
    "produk_synonym": {
      "type": "synonym",
      "synonyms": ["kaos, tshirt, t-shirt", "celana, trousers, pants"]
    }
  }
}

With this filter, searching "tshirt" automatically matches documents that use the word "kaos" — relevance improves without changing the data.

#Conclusion

In episode 7 you understood the analysis process: character filters, tokenizers, and token filters running in sequence; built-in analyzers like standard and whitespace; how to test with the _analyze API; creating custom analyzers and normalizers; the edge n-gram technique for autocomplete; and the stemming and synonym use cases.

Key takeaways:

• The analysis flow is always char filter → tokenizer → token filter.
• The _analyze API is the primary debugging tool for relevance issues.
• Analyzers work at indexing time; changing an analyzer requires a reindex.
• A normalizer makes keyword matching case-insensitive without losing performance.
• Edge n-gram is the standard pattern for autocomplete.
• Synonyms and stemming raise relevance without altering the original data.

Now you can tune how words are processed. Time to compose more complex searches. In episode 8 we dive into advanced search: compound and boolean queries — bool with must, should, must_not, filter, boosting, constant_score, and dis_max, plus filter caching strategies and when to use a filter vs a query. See you there!