How We Indexed 900 Million Records in Elasticsearch - In Under a Week {H1}

A pipeline that once took months, re-engineered from first principles. Here's the playbook from Redrob's data engineering trenches.

There's a moment in every data engineering project when the numbers stop being abstract and start being terrifying. For us at Redrob, that moment arrived when we stared at 900 million talent records sitting in Google Cloud Storage and realized our existing pipeline would need months to index them into Elasticsearch. Months we didn't have. So we ripped the whole thing apart and rebuilt it. This is the story of how we did it - the architecture decisions, the code, the failures, and the hard-won optimizations that compressed months into days.

The Problem: A Pipeline That Couldn't Keep Up

Redrob's core product is a talent intelligence platform. Recruiters and hiring managers search through a massive corpus of enriched professional profiles — complete with employment history, skills, company data, and more. That search experience is powered by Elasticsearch, and for it to be fast, every single record needs to be indexed, enriched, and searchable.

Our raw talent data lives in GCP Cloud Storage as compressed JSON files — hundreds of thousands of them. Each record needs to be read, merged with company enrichment data, validated, and then written to Elasticsearch. At 900 million records, even small inefficiencies cascade into catastrophic delays.

The original pipeline was straightforward: read a file, transform it, index it. Sequential. Safe. And agonizingly slow. Each document required a full network round-trip — download from cloud storage, enrich, then a single-document index call to Elasticsearch. At peak throughput, this managed roughly 100–150 documents per second. A quick napkin calculation: at 150 docs/sec, indexing 900 million records would take approximately 70 days. That's over two months of compute costs, two months of stale data, and two months of engineers babysitting a pipeline.

"We didn't need to make the pipeline faster. We needed to re-think what the pipeline was."

The Architecture: From Monolith to Stream

The first thing we did was sketch the system end-to-end on a whiteboard. Not the code — the data flow. Where do bytes live? Where do they move? What's the bottleneck at each stage? This exercise revealed that our original pipeline was I/O-bound at ingestion, CPU-bound at enrichment, and write-bound at indexing — three entirely different problems wearing the same trenchcoat.

We decomposed the pipeline into four discrete stages, each independently tunable: Ingest, Enrich, Buffer, and Index.

Streaming From the Cloud - Not Downloading

Our first instinct was to download all files from GCP to local disk, then process them. This is a trap. With hundreds of thousands of compressed JSON files totalling terabytes, local disk becomes a bottleneck before you've even started transforming data.

The fix was Python's smart_open library, which provides a file-like interface for streaming directly from cloud storage. Instead of downloading a 500MB gzip file, waiting for it to decompress, and then parsing it — we streamed it. Bytes flow directly from GCP into our Python process, decompressed on the fly, parsed line-by-line. Memory footprint: negligible. Startup latency: near-zero.

💡Key Insight

smart_open supports gs://, s3://, and azure:// URIs natively. It handles gzip decompression transparently. One line of code eliminated our entire file-download step, cutting ingestion latency by over 70%.

Multi-threading: Saturating Every Core

Streaming solved the I/O problem, but we still had idle CPU cycles. Our enrichment step — merging talent records with company data, normalizing fields, computing derived attributes — is CPU-intensive work. Running it sequentially on a single thread was like driving a Ferrari in first gear.

We used Python's ThreadPoolExecutor for I/O-heavy tasks (reading from GCS, writing to Redis, making Elasticsearch bulk calls) and ProcessPoolExecutor for CPU-heavy enrichment. The key was finding the right balance: too many threads and you thrash the GIL; too few and you waste resources. We settled on 8 I/O threads and a process pool sized to the machine's core count minus two (leaving headroom for the OS and monitoring)

The orchestration pattern here is deliberate: we pipeline the stages so that while one batch is being enriched, the next is being read, and the previous is being indexed. No stage waits for another to finish completely. This overlap was responsible for roughly a 3× throughput improvement on its own.

Redis as the Resilience Layer

At 900 million records, failures aren't a possibility — they're a certainty. Networks blip, Elasticsearch nodes restart, malformed records cause serialization errors. The question isn't if something will fail, but when — and whether you can recover without reprocessing everything from scratch.

We inserted Redis as an intermediate buffer between enrichment and indexing. Every enriched batch gets written to Redis with a deterministic key derived from its source file and offset. This gave us three critical capabilities: resumability (restart from where you left off), decoupling (enrichment and indexing can run at different speeds), and idempotency (replaying a batch produces the same result).

⚠️ Lesson Learned

Early on, we tried using Redis Lists (LPUSH / RPOP) for a queue pattern. This caused memory issues at scale because Redis keeps lists entirely in memory. Switching to keyed batches with TTLs kept memory predictable and gave us free garbage collection.

Elasticsearch: The Art of the Index

This is where most teams get it wrong, and where we got the biggest gains. Elasticsearch is phenomenal at search, but it was never designed for one-shot ingestion of 900 million documents. Its defaults are tuned for steady-state mixed read/write workloads, not bulk loading. We had to fight every default.

Small Indexes + Aliases = Exponential Gains

Instead of creating a single monolithic index with 900M documents, we partitioned the data across multiple smaller indexes — each containing roughly 20 million records. An Elasticsearch alias (my_search_alias) points to all of them simultaneously. To the querying application, it looks like one big index. Under the hood, each sub-index is independently manageable.

Why does this matter? Because Elasticsearch's merge operations, segment management, and recovery behaviors all scale super-linearly with index size. A single index with 900M documents spends enormous amounts of time in segment merges. Twelve indexes with 75M documents each can be indexed in parallel, merged independently, and if one fails, you only re-index that slice — not the whole corpus.

Zero Replicas + Async Translog + Write-Optimized Routing

This cluster of optimizations — applied together — cut our indexing time nearly in half. Here's the reasoning behind each setting:

Setting number_of_replicas: 0 eliminates the overhead of writing every document twice. By default, Elasticsearch writes to both the primary shard and all replica shards synchronously. During bulk ingestion, replicas provide no value — they just double your write load. We restore them to 1 after indexing for fault tolerance.

Disabling refresh (refresh_interval: -1) prevents Elasticsearch from creating new Lucene segments every second. During ingestion, nobody is searching the in-progress index, so this overhead is pure waste. On read-only indexes, refresh becomes irrelevant entirely — which is why even post-ingestion, the effective cost is zero until new data arrives.

Using translog.durability: async allows Elasticsearch to batch disk writes instead of calling fsync on every single operation. This dramatically reduces I/O pressure during bulk loading. Post-ingestion, we switch back to request durability to guarantee per-write durability for live user queries.

Finally, routing shard allocation to write-optimized nodes (data_hot preference) during ingestion ensures writes land on nodes with fast NVMe SSDs and high IOPS. After ingestion, we move data to balanced read/search nodes (data_content preference) optimized for serving user queries.

Right-Sizing Shards: 40GB Target, 3 Million Records Each

Elasticsearch's documentation recommends shard sizes between 10–50GB. Through benchmarking, we converged on a 40GB target per primary shard as our sweet spot. The math was straightforward: with roughly ~20TB of total indexed data, that gives us approximately 250 primary shards. Adding one replica doubles it to 500 total shards — a number well within Elasticsearch's comfort zone for cluster management.

The key insight was indexing data in chunks of approximately 3 million records per shard. This ensured balanced shard sizing — no single shard ended up bloated with 80GB while others sat at 10GB. Uneven shards cause hot spots where one node does all the work during search while others idle. We pre-computed record counts per partition to ensure even distribution before indexing even began.

An important pre-flight check we added: disk space validation before every bulk operation. Bulk updates create new document versions internally, which temporarily increases disk usage as old segments coexist with new ones. We built an automated check that queries the cluster's disk usage via the _cat/allocation API and aborts the batch if free space drops below 20%. This saved us from at least two potential shard failure incidents during early pipeline runs.

Batch Indexing with the Bulk API

Single-document indexing is a non-starter at this scale. Elasticsearch's _bulk API lets you send thousands of documents in a single HTTP request, amortizing network overhead and allowing ES to optimize its internal segment writes. We experimented with batch sizes from 1,000 to 20,000 and settled on 5,000 documents per bulk request — a balance between memory usage and throughput.

A critical lesson from our internal runbook: bulk update is always safer than delete-and-reinsert. When we needed to update records with enriched company data (matched via source_reference_id → enrichment_key), we used the Bulk API's update action targeting documents by their _id. This approach produces fewer I/O operations, lower shard pressure, reduced risk of data inconsistency, and easier rollback compared to delete-reinsert workflows.

One more caveat that cost us a full day of debugging: avoid search_as_you_type fields on ingestion-heavy indices. These fields have an extremely high indexing cost because they generate multiple sub-fields with edge n-gram analyzers. We stripped them during bulk loading and added them post-ingestion via reindexing into a dedicated autocomplete index.

Blue-Green Deployments for Zero Downtime

Here's the thing about re-indexing 900 million records: users don't stop searching while you're doing it. Our production cluster needed to serve live queries at sub-second latency while we were hammering it with bulk writes. The answer was a blue-green deployment pattern adapted for Elasticsearch.

We maintained two sets of indexes: the "green" set (currently live, pointed to by the my_search_alias alias) and the "blue" set (being built). The blue indexes lived on the same cluster but had zero replicas and disabled refresh intervals, so they consumed minimal resources. Once indexing completed on blue, we ran validation checks, enabled replicas, forced a refresh, and then performed an atomic alias swap. Total downtime: zero. The entire switchover took under a second.

✅ Production Win

Blue-green deployments also gave us a free rollback mechanism. If the new index had data quality issues, we could swap the alias back to the old indexes instantly — no re-indexing required.

Error Logging: The Pipeline's Immune System

When you're processing 900 million records, a 0.01% failure rate still means 90,000 broken documents. Without granular observability, those failures are invisible — they accumulate silently until someone notices search results are incomplete weeks later. We built error tracking into every stage of the pipeline, not as an afterthought, but as a first-class subsystem.

Every failed record gets captured with full context: the source file, the line offset, the enrichment stage where it failed, the exception type, and a truncated payload for debugging. These failures are written to a structured dead-letter queue (a dedicated Redis stream) and simultaneously aggregated into a pipeline health report that runs at the end of each batch cycle.

The report generation step was critical. After each major batch completed, the pipeline automatically generated a JSON sanity report with success rates, error breakdowns by type, and a list of sample failures. If the success rate dropped below 99.5%, the pipeline flagged the batch for manual investigation rather than proceeding blindly. This caught several subtle issues early — a malformed date format in a subset of records from a specific data provider, a rare Unicode encoding edge case in company names, and an intermittent timeout from a third-party enrichment API.

🔴 The Error That Almost Got Away

During our second full pipeline run, we noticed the health report showing a 0.3% failure rate in enrichment — all KeyError exceptions. Investigation revealed that roughly 2.7 million records were missing a company_id field entirely. Without structured error logging, these records would have been silently dropped. Instead, we patched the enrichment logic to handle missing company IDs gracefully, re-queued the failed records from the dead-letter stream, and recovered all 2.7 million records in under an hour.

Tuning the Elasticsearch Coordinator Node

Here's something that doesn't show up in most Elasticsearch tutorials but matters enormously at scale: the coordinator node. In Elastic Cloud, the coordinator node is the entry point for all client requests — it parses queries, fans them out to data nodes, aggregates results, and returns them to the client. During heavy bulk indexing, the coordinator becomes a bottleneck if under-provisioned.

We learned this the hard way. Midway through our first large-scale indexing attempt, the coordinator started rejecting requests with 429 Too Many Requests errors. The bulk thread pool was saturated, the coordinating queue was full, and the cluster entered a cascading rejection spiral where retries from our client only made things worse.

The fix was a combination of infrastructure and configuration changes on the Elastic Cloud deployment:

Beyond configuration, we also right-sized the coordinator instance itself. We scaled it from a 4GB RAM / 2 vCPU instance to 8GB RAM / 4 vCPU during the indexing window, then scaled back down afterward. This temporary vertical scaling — combined with the queue size and circuit breaker tuning — eliminated all 429 errors and kept the cluster stable even as we pushed 40,000+ documents per second during peak bursts.

💡 Pro Tip

On Elastic Cloud, you can adjust node sizes independently. During indexing, over-provision the coordinator and data nodes. During steady-state search, you can scale back down. We saved ~40% on cluster costs by right-sizing after indexing completed.

Read Optimization: Making 900M Records Searchable at Speed

Indexing is only half the story. The entire point of this exercise was to make 900 million records searchable in sub-second response times. Once the data was indexed, we turned our attention to query performance — and discovered that a naively indexed 900M-record corpus delivers disappointingly slow search results without careful optimization. Here are the nine specific techniques from our internal performance playbook that brought latency down to production SLA.

1. Enable Request Cache for Dashboards

Our recruiter-facing dashboards fire the same aggregation queries repeatedly — "top skills in London," "candidate count by experience band," and so on. Enabling the request cache at the index level meant that repeated identical queries return instantly from cache instead of re-executing across all shards. The impact on dashboard load times was dramatic.

2. Country-Level Indexes for Talent Search

This was one of our highest-impact read optimizations. Instead of querying a single global index for every search, we maintain separate indexes per country. When a recruiter selects "India" in the location filter — which accounts for a huge proportion of our searches — the query hits only the India index, not all 900M records spread across 250+ shards. This reduced search scope by 80–90% for geo-filtered queries and cut average latency by roughly 35%.

3. Dedicated Autocomplete Index

For talent and company data, we identified the most frequently searched fields — job titles, company names, skills — and stored their unique, normalized values in a dedicated lightweight autocomplete index. This index uses search_as_you_type fields (which we deliberately excluded from the main ingestion indexes due to their high write cost). The result: instant typeahead suggestions without touching the 900M-record main index.

4. Fetch Only Required Fields

Our initial queries retrieved the full _source document — which for enriched talent profiles averaged 8–12KB each. For a search returning 25 results, that's 200–300KB of payload, most of it unused. We switched to _source filtering and fields parameter to fetch only the 6–8 fields actually displayed in search results. This reduced payload size by 70% and cut network transfer time significantly.

5. Force Merge to a Single Segment

After indexing completes and the index becomes read-only, we run a force merge to consolidate all Lucene segments into a single optimized segment per shard. The internal mechanics: Elasticsearch creates a new larger segment, copies all live documents into it (permanently excluding soft-deleted documents), then deletes the old segments. The benefits compound across multiple dimensions:

⚠️ Force Merge Caveats

Force merge rewrites the entire index — it's CPU and I/O intensive and can take hours on large indexes. Run it during off-peak hours, never while the index is serving live traffic. It also temporarily increases disk usage by 10–50% as old and new segments coexist. We scheduled force merges during our blue-green "dark" window when the index wasn't serving live queries.

6. Block Writes for Maintenance Windows

During snapshot operations, reindexing, or any maintenance activity, we lock the index to prevent accidental writes. This stabilizes the cluster by stopping segment refresh/merge work, reducing disk I/O, lowering CPU usage, and minimizing GC pauses.

7. Optimized Mapping Design

We were aggressive about choosing the right field types. Fields used only for filtering (like country_code or experience_years) were mapped as keyword or short — never text. Fields needing full-text search (like job_title or skills) used text with carefully chosen analyzers. And fields never queried but only returned in results were set to enabled: false to skip indexing entirely, saving disk and CPU.

8. Filter Context for Non-Scoring Queries

We used filter context (inside bool.filter) for all non-scoring clauses, which allows Elasticsearch to cache results and skip the expensive TF-IDF scoring step. For paginated results, we replaced deep from/size pagination with search_after — because deep pagination forces every shard to return from + size documents to the coordinator for re-sorting. At page 1000 with 25 results, that's 25,000 docs per shard across 250+ shards.

9. Cluster Architecture: The Final Setup

After all optimizations, here's the architecture we converged on for our ~20TB indexed dataset:

✅ Read Performance Results {H4}

After all optimizations — request cache, country-level routing, force merge, field filtering, and mapping tuning — our p95 search latency across 900M records settled at 80–150ms for geo-filtered queries and under 300ms for complex aggregations. Well within our 500ms SLA.

11

The Money Question: Cost Optimization at Scale {H2}

Running an Elasticsearch cluster capable of indexing and serving 900 million records is not cheap. At one point during our early experiments, our projected annual Elastic Cloud bill was spiraling. Through a series of deliberate infrastructure and operational decisions informed by extensive benchmarking, we achieved a 40–50% overall cost reduction on our Elasticsearch infrastructure.

The Storage Optimised Dense Breakthrough {H3}

This was our single biggest cost lever. Elastic Cloud offers two node profiles: Storage Optimised and Storage Optimised Dense. The dense nodes pack significantly more disk per unit of RAM:

The conventional wisdom is that dense nodes sacrifice performance for capacity. But our benchmarking revealed a critical nuance: when you keep disk usage below 60%, dense performance is nearly identical to standard storage optimised. Here's why — Elasticsearch performance depends heavily on OS filesystem cache, JVM heap efficiency, and disk read pressure. When disk usage is low, more data stays in filesystem cache, fewer disk seeks are needed, and GC pressure drops. At 50% usage on a 6TB dense node, you're effectively getting 3TB of fast, cache-friendly storage.

For our workload — read-heavy, large dataset, low continuous ingest, all data searchable — dense nodes with disciplined disk management gave us the same effective performance at 40–50% lower cost. The key constraint: never exceed 65% disk usage. We built monitoring alerts to enforce this.

CPU is identical between both profiles — only disk-per-RAM differs. Since RAM is the primary performance driver for search workloads, and we had sufficient RAM, the dense profile was the clear winner.

Force Merge: The Hidden Cost Optimizer {H3}

Force merging to a single segment after indexing isn't just a performance optimization — it's a cost optimization. By consolidating segments and permanently purging soft-deleted documents, force merge reduced our disk usage by 10–30% across the cluster. On a multi-TB deployment, that translates directly to lower storage costs and delayed need for capacity expansion.

Additional Cost Levers {H3}

Beyond the major wins, we pulled several smaller levers that compounded into significant savings: reducing availability zones from 3 to 2 for our staging environment (50–70% savings on non-production clusters), enabling autoscaling to scale down during low traffic, and reducing replica shards to 0 on non-critical backup indexes (33–50% storage savings).

✅ Bottom Line

Combined infrastructure cost reduction: 40–50% on Elasticsearch. The Storage Optimised Dense switch alone was worth the investigation — same performance, dramatically lower bill.

12

The Results: From Months to Days {H2}

Here's the before-and-after picture. These aren't theoretical benchmarks — this is what we measured on our production pipeline processing the full 900M+ record corpus.


The Journey, Visualized {H3}

13

What We'd Tell Our Past Selves {H2}

If we could go back to day one and give ourselves a cheat sheet, here's what it would say:

Partition your indexes. A single monolithic index at this scale is a trap. Small indexes behind an alias give you parallelism, fault isolation, and independent lifecycle management. The performance gains are not linear — they're exponential.

Stream, don't download. If your data lives in the cloud, it should stay in the cloud until the last possible moment. smart_open turned a multi-hour download step into a zero-latency streaming pipeline.

Redis is your safety net. At billion-record scale, failures are a mathematical certainty. An intermediate buffer that enables resumability and decoupling is not a nice-to-have — it's infrastructure.

Fight every Elasticsearch default. Replicas, refresh intervals, translog durability, thread pool sizes — the defaults are tuned for a very different workload than bulk ingestion. Know what each setting does and override it deliberately.

Cost is a feature. The switch to Storage Optimised Dense nodes — at under 60% disk utilization — delivered nearly identical query latency (90–170ms vs 80–150ms) at 40–50% lower cost. Force merging reclaimed 10–30% of disk space. Infrastructure spending that isn't regularly audited is infrastructure spending that's being wasted.

Blue-green isn't just for apps. The same deployment pattern that gives you zero-downtime application releases gives you zero-downtime index rebuilds. The alias swap is the most underrated feature in Elasticsearch.

Design for search from day one. Country-level indexes, dedicated autocomplete indexes, request caching, field filtering, and force-merging to single segments — these aren't afterthoughts. They're the difference between 800ms search latency and 80ms.

"The pipeline that once felt like an unscalable mountain became a well-oiled system — one that not only handled 900 million records but was ready for the next billion."

14

What's Coming Next {H2}

The pipeline described in this article got us from zero to 900 million records in under a week. But we're not done. The next iteration is already in progress — and it changes how we think about both the enrichment layer and the Elasticsearch infrastructure underneath it.

PySpark for Country-Wise Enrichment at Scale {H3}

Our current Python-based enrichment pipeline works, but for the next full re-index cycle we're moving the heavy lifting to PySpark. The strategy: process member data country-by-country using distributed Spark joins against company data, then write the enriched output as JSON files in chunks of 5,000 records each. These small, independently readable files then get streamed directly into Elasticsearch.

Why JSON chunking instead of large Parquet files? If we wrote large Parquet files and then reloaded the entire 4TB enriched dataset back into Spark for ES indexing, it would require massive cluster memory, increase compute cost, and keep the full dataset in memory during writes. Instead, 5,000-record JSON files give us lower memory usage, controlled ES write throughput, easier retry (re-run a single failed file), and dramatically reduced cluster cost during the indexing phase.

Target: 20,000 Docs/Sec Sustained Throughput {H3}

With JSON chunking, each 5,000-record file can be processed in 0.25 seconds at peak throughput. Running 8–12 parallel bulk workers, we're targeting a sustained 20,000 documents per second — which means the full 900M record corpus could be indexed in approximately 23 hours. 

The math…

20,000 × 3,600 = 72M documents per hour, and 

900M ÷ 72M ≈ 12.5 hours for indexing. 

Additionally, Spark enrichment is expected to take approximately 10.5 hours. A massive improvement over even our current sub-7-day pipeline.

The enrichment phase will run on an 8–10 node Spark cluster (16 vCPU / 64GB RAM each), estimated at $120–150 per full run — or 60–70% cheaper on spot instances. The indexing phase needs only 4–6 smaller compute nodes focused on bulk ingestion, estimated at $40–80 per run.

Elasticsearch Cluster Architecture — Scaled for 25TB {H3}

For the next iteration, we're scaling the Elasticsearch cluster to handle the growing dataset with a production-grade architecture: 10 data nodes (64GB RAM, 32GB heap, 16–32 vCPU, NVMe/SSD-backed) plus 3 dedicated master nodes (8GB RAM each). With a 40GB target shard size and 15–25TB of indexed data, that works out to 400–450 primary shards with one replica post-indexing. Total cluster heap: 320GB, with the remaining RAM dedicated to OS page cache — critical for indexing performance.

Further Cost Optimization: Squeezing Another 30–40% {H3}

Beyond infrastructure right-sizing, we've identified several data-level optimizations that should reduce our index footprint by another 30–40%:

Field pruning: A significant portion of our current index size comes from raw or debug fields, large arrays, and unused metadata. Stripping these before indexing could yield a 10–30% data size reduction.

Source compression: Enabling index.codec: best_compression on our indexes should deliver another 20–30% storage reduction. Since _source must remain enabled for updates and reindexing, compression is the next best lever.

Shard consolidation: Right-sizing shards to 40–50GB each and reducing shard count lowers heap memory usage, speeds up cluster management, and reduces operational overhead.

External raw data storage: Maintaining the original dataset in Parquet files and JSON archives in object storage, rather than relying on Elasticsearch as the source of truth, means we can rebuild indexes from scratch without data loss concerns.

✅ The Trajectory {H4}

From a 70-day pipeline to under 7 days, and soon to under 13 hours. From $16,000/mo to $9,280/mo, and targeting $5,500–$6,500/mo. Each iteration compounds — and the foundation we built in this article makes every future optimization possible.

How This Powers Redrob’s Talent Search Experience {H2}

At Redrob, infrastructure decisions like these directly shape the product experience. Faster indexing, resilient enrichment pipelines, and search-optimized Elasticsearch architecture help us make massive volumes of talent data searchable, structured, and usable for recruiters and hiring teams. As our data systems scale, so does our ability to deliver a more reliable talent intelligence platform built for speed, relevance, and operational efficiency.

Want to see how Redrob handles talent data at scale?
[Book a Demo]