· ifkarsyah

Elasticsearch Internals Series, Part 0: Overview

A roadmap through Elasticsearch 8.x internals — from inverted indexes to cluster replication. Why learning the engine makes you a better search engineer.

Search Elasticsearch
Elasticsearch Internals Overview

Elasticsearch is the backbone of billions of search requests per day. Yet most engineers who use it treat it as a black box: they send queries and hope the results are good. When performance degrades, when relevance breaks, when memory explodes — they have no mental model of what’s happening inside.

This series changes that. Over 8 parts, we’ll dissect Elasticsearch from the bottom up: how it stores and indexes text, how shards and segments work, how search queries scatter across a cluster and merge back, and how writes are made durable under the hood.

Why does this matter?

  • Debugging poor relevance requires understanding BM25 scoring, not just tweaking boost values blindly
  • Capacity planning demands knowing the difference between indexing buffers, heap, and off-heap storage
  • Query optimization becomes obvious once you understand filter caching, doc_values, and the query vs filter context
  • Production incidents — OOM errors, split-brain, hot shards — all have root causes you can reason about

This series assumes you know how to run basic Elasticsearch queries. You don’t need to know Lucene internals deeply — we’ll focus on observable behavior and the practical implications for your applications.

What We’ll Cover

Part 1: Inverted Index & Text Analysis

How Elasticsearch stores text for full-text search — analyzers, tokenizers, token filters, and the inverted index structure. You’ll use the _analyze API and _termvectors to inspect how your text is actually stored.

Key question: Why does "Quick Brown Fox" not match "quick brown fox" by default?

Part 2: Shards, Segments & Lucene

Every Elasticsearch index is split into shards. Every shard is a Lucene index. Every Lucene index is a collection of immutable segments. We’ll trace how this architecture determines write latency, search freshness, and disk I/O.

Key question: Why is there a ~1 second lag between indexing a document and being able to search it?

Part 3: Document Storage & Mappings

Field types, _source, doc_values, fielddata, and store — each field is stored in multiple ways for different purposes. We’ll cover what to enable, what to disable, and why the wrong mapping ruins performance.

Key question: Why does enabling fielddata on a text field risk OutOfMemoryError?

Part 4: Search Internals & Relevance Scoring

How a search query flows from client → coordinating node → data shards → merge. How BM25 calculates scores. How to debug relevance with the _explain API.

Key question: Why does the same query return slightly different scores when you run it against different shard counts?

Part 5: Query DSL Deep Dive

Query context vs filter context. bool query anatomy. Leaf queries (match, term, range). Pagination strategies (from/size vs search_after). We’ll build a realistic product search query from scratch.

Key question: When should you use filter instead of query, and why is it significantly faster?

Part 6: Aggregations & Analytics

How aggregations work differently from search (they use doc_values, not the inverted index). Bucket, metric, and pipeline aggs. Cardinality approximation with HyperLogLog++.

Key question: Why does a terms aggregation on a high-cardinality field have memory implications?

Part 7: Cluster Architecture & Replication

Node roles, primary vs replica shards, the write path (primary → replica), split-brain, and quorum. We’ll simulate a node failure and observe cluster recovery.

Key question: How does Elasticsearch guarantee you never lose a committed write, even when a node dies mid-write?

Part 8: Write Path & Translog

The complete lifecycle of a write: index request → indexing buffer → translog → refresh → flush → segment merge. The translog is Elasticsearch’s write-ahead log equivalent. We’ll tune durability vs throughput tradeoffs.

Key question: What exactly happens to your data between PUT /index/_doc/1 and it safely landing on disk?

How to Use This Series

Each article is standalone — you can read them in any order. But they’re designed to flow sequentially, building on concepts from earlier parts.

To follow along, you need Docker:

# docker-compose.yml — use this for the entire series
version: '3.8'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
    ports:
      - "9200:9200"
    volumes:
      - esdata:/usr/share/elasticsearch/data

  kibana:
    image: docker.elastic.co/kibana/kibana:8.12.0
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

volumes:
  esdata:
docker compose up -d

# Verify ES is up
curl -s http://localhost:9200 | jq '.version.number'
# → "8.12.0"

All code examples in this series are copy-paste ready. Run them in your own environment.

A Quick Architecture Overview

A request’s journey through Elasticsearch:

┌───────────────────────────────────────────────────────────────┐
│ Client Request  ("GET /products/_search?q=laptop")            │
└─────────────────────┬─────────────────────────────────────────┘
                      │
            ┌─────────▼──────────┐
            │ Coordinating Node  │  (any node can coordinate)
            └─────────┬──────────┘
                      │ scatter
         ┌────────────┼────────────┐
         ▼            ▼            ▼
    ┌─────────┐  ┌─────────┐  ┌─────────┐
    │ Shard 0 │  │ Shard 1 │  │ Shard 2 │  (Part 2: shards)
    │ (Lucene)│  │ (Lucene)│  │ (Lucene)│
    └────┬────┘  └────┬────┘  └────┬────┘
         │            │            │
         │            │ gather/merge
         └────────────▼────────────┘
            ┌─────────────────────┐
            │ Coordinating Node   │  (merge + rank, Part 4)
            └─────────┬───────────┘
                      │
            ┌─────────▼───────────┐
            │ Response            │
            └─────────────────────┘

Each shard is a self-contained Lucene index:

Shard (Lucene index)
├── Segment 0  (immutable, Part 2)
│   ├── Inverted index  (term → doc IDs, Part 1)
│   ├── doc_values      (doc ID → field value, Part 3)
│   └── _source store   (original JSON, Part 3)
├── Segment 1
├── Segment 2
└── In-memory buffer    (not yet flushed, Part 8)

Each layer has a story. We’ll explore them all.

Key Concepts You’ll Learn

  • Inverted index — the core data structure behind full-text search
  • Analyzers — how raw text becomes searchable tokens
  • Segments — why Elasticsearch writes are near-real-time but not instant
  • BM25 — how relevance scores are calculated
  • doc_values — how aggregations and sorting work efficiently
  • Translog — how durability is guaranteed for every write
  • Shard routing — how documents are distributed and found
  • Quorum — how the cluster survives node failures

Next Steps

In Part 1, we’ll create an index, index some documents, and use the _analyze API to see exactly how text is broken down into tokens before storage. You’ll understand why search relevance depends entirely on analysis — and how to control it.

Ready? Let’s start with the inverted index.

Part 0 complete. Next: Inverted Index & Text Analysis

← Back to Blog
Search Elasticsearch