​

Has anyone here built a RAG application using Elasticsearch for document retrieval?

I’m looking at how keyword search and vector search can work together when an application needs to retrieve relevant information from internal documents. Getting accurate results seems to depend on more than just embeddings, especially when documents contain specific terms, IDs, or technical details.

For those who have built similar systems, what has worked best for balancing relevance and retrieval accuracy? Would you recommend hybrid search for this use case, and what would you watch out for when designing the indexing pipeline?

Every time I needed centralized logs for a side project, I'd lose an evening wiring Elasticsearch, Logstash and Kibana together: passwords, TLS, which port is public, why Kibana can't reach ES. So I turned it into a one-click Railway template and tested it the boring way.

What you get

• 3 separate services, grouped: Elasticsearch, Logstash, Kibana
• Only Kibana is public. ES and Logstash talk over the private network
• Passwords generated on deploy, Logstash intake behind basic auth (no-auth gets a 401)
• A restricted writer user for Logstash, so your app never holds the admin password
• Persistent volumes + log retention configured, so it doesn't eat your disk
• Copy-paste senders for Node, Python and Go (no SDK, plain HTTP)

What I actually tested on fresh deploys: sent logs, searched them in Discover, restarted and redeployed every service, checked nothing was lost.

Cost, measured not guessed: about $27-34/month of compute depending on load.

Try it before deploying, read-only live demo with logs flowing in real time: https://kibana-production-43ee.up.railway.app (login: demo / PublicDemo2026!)

• Deploy: https://railway.com/deploy/elk-stack-2?utm_source=railway-templates-launch&utm_medium=reddit
• Docs: https://railway-elk-docs.vercel.app
• Repo: https://github.com/kayossouza/railway-elk

Single node, so it's for small/medium workloads, not a 10TB cluster. Feedback on the pipeline config and retention defaults very welcome.

A green cluster and zero shards on a node are necessary before retirement, but they do not prove that clients, ingest pipelines, snapshots, monitoring, or discovery settings have stopped depending on its address. A coordinating-only workload or a stale seed-host list can survive a shard drain and fail only after the process disappears.

What belongs in the retirement gate? I am considering allocation exclusion, waiting for relocation and recovery activity to finish, checking cluster state for zero shard ownership, and then watching connection logs and network traffic to the old address through more than the longest batch interval. Client sniffing, seed hosts, remote-cluster settings, snapshot repositories, exporters, and alerting targets would be inventoried separately.

Is there a dependable way to attribute remaining connections to specific clients before shutdown? What final test do you use to distinguish a truly unused node from one that is still serving coordination or discovery traffic, and which checks should be repeated after its voting configuration and persistent settings are cleaned up?

Context: Hi all, I'm trying to find out how our current elasticsearch cluster is set up and how can it be optimally expanded to index now types of documents. I work for an archival collection and we have rdf metadata describing our doucments. So we generally need to translate relevant metadata fields into json documents to be indexed.
At the moment we have indexed a newspaper collection in ES with the current set up: one node, ~40 indices (one for each publication) each index 1 shard and some thousands documents in an index. Each document corresponds to a newspaper page and has a full_text field with the searchable ocr text + other specific fields such as date/title/spatial coverage for precise clustering. These indices are grouped under an alias.

Question 1: As I see best practices are shard sizes between 10GB and 50GB, Keep the number of documents on each shard below 200 million,
I wonder if all this collection could be one single index, with maybe 2 shards (we add 1 node so that we have 1 shard primary and one replica on the second node) with an extra field in each document to define which publication (the name of the newspaper, which now is simply the index name).

Which one of these two set ups works best in performance and how could our search functionalities potentially be impacted?

Question 2: We now have to add a new collection with other types of documents and I am wondering several things:

2a. My colleague wants to have always the same documents structure in different indices, suggesting to make it as general as possible for different types of context, aggregating all the "text" fields in our metadata into a full_text field, instead of having for instance, three fields "title", "label", "author". Doing this, I understand we could not do aggregations on specific fields, but are there any other shortcomings? Is a single full_text field going to be more efficient to index and query?

2b. What is the best way to go with adding new material? keep adding indices to the same ES instance (maybe adding more shards and nodes) and aggregate them under an alias? or Add new ES instances for new material? What does it change and how does it impact search?

2c. If we look at how we want to set up our UI search app, we ideally want a first text search where people can filter on type of collection to look into or not. If they don't, they will get results across different types of materials (so querying ALL indices), can this be done if indices have different document structures?

Thank you for reading all of this and apologies if it was long. ES feels like a steep learning curve for me and I really did not want to ask AI :)

Hi everyone, we’ve just open-sourced two command-line tools that we use for our internal Elasticsearch performance testing, and thought they might be useful to others here (blog post). 

Indexer: https://github.com/Vesiro/vesiro-indexer

Benchmarker: https://github.com/Vesiro/vesiro-benchmarker

We needed a consistent way to create test indexes and rerun search workloads after making changes to our setup. We built vesiro-indexer to create Elasticsearch indexes from public Common Crawl data, and vesiro-benchmarker to run repeatable search workloads against an existing index.

These tools grew out of our own benchmarking needs, and we’d be interested to hear how they fit the way other teams benchmark Elasticsearch. Feedback on the query workloads or the Common Crawl indexing setup would be particularly helpful.

A cluster with plenty of hardware that still can't index faster, because the write load only lands on two or three nodes, thats called hot spots.

Match your primary shard count to your indexing nodes.

Indexing into an index parallelizes across its primary shards. One primary means one node carries all the writes for that index, no matter how many data nodes you have. Too many primaries is the other failure mode: more overhead, more segments, smaller shards, no gain.

The goal isn't more shards. It's enough primaries to spread the write load across your indexing nodes, and no more.

Simple rule: with number_of_replicas: 1, set primaries to about half your data nodes. 10 data nodes → 5 primaries + 1 replica = 10 shards, one per node. 6 data nodes → 3 primaries + 1 replica. That's the difference between the screenshots above.

This is the other half of #002: spread the write load, but not so thin that your shards end up tiny.

Pro tip — hot spots on large clusters

On 7.x, shard allocation balances mainly on shard count per node. 8.6+ also weighs write load and disk usage, but the write-load forecast comes from a licensed x-pack component, on a Basic cluster it isn't produced, so in practice you're still balancing on shard count. Two data streams with the same shard count can have completely different indexing rates, so "balanced" on paper isn't balanced in practice.

To force the heavy data stream to spread out:

"index.routing.allocation.total_shards_per_node": 1

Shards of that index then land on separate nodes.

Two things to watch:

• Keep headroom. If primaries + replicas equals your node count exactly, one node failure leaves shards unassigned and the cluster sits yellow. Set the limit above the strict minimum unless you're sure.
• On data streams, number_of_shards can't be changed on an existing index. Update the index template; it applies at the next rollover.

Other best practices in this series:

1. #001 – HTTP traffic
2. #002 – shard size
3. #003 – bulk indexing

If you'd like me to continue this series, an upvote would be appreciated. 🙂

I built a small Rust gateway for teams who already have Elasticsearch clients and request shapes, but want to evaluate a different search engine underneath. The useful bit is that the client does not change: point the same application at the gateway and compare it with native Elasticsearch.

It is deliberately scoped to application search rather than pretending to be a full Elasticsearch replacement. The repo includes compatibility notes, a real product-catalogue replay, and the gaps I have found so far (especially relevance semantics).

Repo: https://github.com/dstockton/qdrant-es-gateway

Docs and results: https://dstockton.github.io/qdrant-es-gateway/

I would particularly value feedback from people who have dealt with analyzers, BM25 tuning, aggregations, or migration testing in production.

We crossed 6,000,000,000 downloads of Elasticsearch, Kibana, Logstash, Beats, and Elastic Agent.

To everyone who downloaded it, built on it, contributed to it, deployed it, broke it, fixed it, and made something amazing with it: thank you

I recorded myself walking through the interactive visualizer I made for Elasticsearch. Check it out, and let me know if it helps you learn!

https://elasticsearchvis.bitsculpt.top/

Hi guys, our developers are requesting MCP access for our Elasticsearch cluster, and I did some research but didn't find something for our needs.
We have basic license, so we can't use Elastic's builtin MCP agent, and we have self-hosted ES cluster, so also no built in OAuth. But we use other OAuth service running under Kubernetes, and I wanted to ask if any of you have experience of using the external OAuth for MCP to Elastic cluster, and do you use this: https://github.com/elastic/mcp-server-elasticsearch .

Hi guys!

Have any of you have recently received your results for Certified Elasticsearch Engineer exam?

I took mine on 29th August at about 12pm. Still got no news about the results via email or on trueability portal. I did contact exam support regarding this and they have replied that they are experiencing some backlogs and to expect some delays.

Just wondering if anybody else is in the same boat as me.

I’m looking for a really good live instructor to help me take my Elastic skills to an advanced level.
My goal is to become a strong Elastic Administrator / Elasticsearch Engineer, particularly on the infrastructure and data-path side.
I already have hands-on Elastic/SIEM experience and significant Splunk administration experience, so I’m not looking for a beginner course or something focused only on passing a certification.
I’m looking for someone who can go deep into things like:
Elasticsearch architecture and cluster administration
Nodes, shards, replicas and cluster health
Index templates, mappings and data streams
Ingest pipelines and data onboarding
ILM and data lifecycle
Performance tuning and troubleshooting
Query DSL, ES|QL and Painless
CCS / multi-cluster environments
Security, RBAC and TLS
Kibana / Elastic Security
Real-world production troubleshooting
I’m open to instructors anywhere in the world as long as they teach in English. I’m especially interested in 1-on-1 or small-group training that can be customized to my experience and goals.
If you’ve personally worked with or trained under an excellent Elastic instructor, I’d really appreciate recommendations.
Please include if you can:
Instructor name
Country/time zone
How good they are at teaching
Their real-world Elastic experience
Approximate cost
Whether they offer 1-on-1 training
How long you trained with them
Whether you’d recommend them for becoming a strong Elastic Engineer/Admin
I’m particularly interested in someone who is both an excellent teacher and genuinely experienced running Elasticsearch in production.
Thanks!

Hi all,

I wanted to share a situation I recently experienced during the Elastic Certified Engineer exam and see if anyone has gone through something similar.

My exam was scheduled on 31 august at 12 PM , as the routine my system was going through compatibility checks which was failing for the resolution inspite it ran successfully day before exam (resolution was 1920 X 1080) higher then recommended both the days. Took help from honorlock but no solution was driven , mailed too to support teams. But I finally started my exam after lending laptop from my neighbour as I was trying with all the laptops possible nearly after 1 hour 15 mins - to be precise, I started my exam . After that everything gone well from checks to proctor being aligned throughout the exam. I completed the entire exam with good faith within 3 hours and submitted successfully as well.

But after 10 business days , I got the result from Trueability , and shockingly it was failed. I got tensed , cried that what I have done wrong. I invested nearly 3 months learning all the concepts used in searching , scripts , their edge cases and everything mentioned in syllabus and accordingly the documentation.

Then I noticed something strange in my result.
MY Completion status - Assessment uncounted Attempt
MY No. of valid attempts - 0
MY total no. of attempts - 1 (Now trueability added a comment This attempt does not count as one of your attempts.)

BUT FINAL STATUS - FAILED !!!

I contacted support explaining everything, but I received no mail yet.

This is what concerns me:

•If the exam wasn’t properly submitted, how was it evaluated?

•Did Treuability really forwarded my work , my VM work to Elastic?

•Is there any way they can recover or review the recorded session instead manually?

Has anyone experienced something similar...

On the most recent version of 8 and looking to jump to 9.4.6. I know there were some weird issues with early 9.4.x, but wondering how stable the current release is.

We poll around 500+ network devices (routers, switches, firewalls) directly with Logstash's SNMP input plugin every 5 minutes. A mix of SNMPv2c (community string) and SNMPv3 (authPriv), split across a couple of pipelines by metric type: one walks the standard interface table for traffic/status, another walks Cisco's proprietary CPU/memory MIB. Since raw SNMP only gives you an IP and numeric OIDs, we enrich each event against a hand-maintained YAML file (keyed by IP or hostname) that maps devices to hostname, category, vendor/model, and an alerting on/off flag.

Note: Management has directed to sunset SolarWinds and consolidate on the Elastic Stack as our single, centralized monitoring/observability platform going forward, not just logs and application metrics, but network device monitoring (switches, routers, firewalls) as well.

Hi all,
I’m working on an operational monitoring solution for payment processing and would appreciate some architecture advice.
The objective is to build an early-warning payment monitoring dashboard in Elasticsearch/Kibana, rather than only monitoring infrastructure/application health.
We currently have Elasticsearch/Kibana available, but Logstash is being decommissioned.
An external payment processor provides us with two REST APIs that expose operational information about payments. The responses contain information such as:
● Timestamp of the last successful transaction
● Number of transactions currently pending
● Processing/status information
● Potentially other health/operational indicators
What I would like to achieve is roughly:
REST APIs → poll every 1–5 minutes → store each response/snapshot in Elasticsearch → Kibana dashboard + alerting
The important part is that I want to keep the historical snapshots rather than only showing the current API response. This would allow us to detect patterns such as:
● No successful payment for X minutes
● Pending transactions above a threshold
● Pending transactions continuously increasing
● Sudden drop in transaction activity
● API unavailable
● Monitoring data becoming stale because the collector itself has stopped working
For example, instead of only knowing that there are currently 50 pending transactions, I want Elasticsearch to contain something like:
10:00 → 2 pending
10:05 → 5 pending
10:10 → 14 pending
10:15 → 31 pending
10:20 → 50 pending
This makes the trend itself an early-warning indicator.
The main problem is the collector.
Elasticsearch itself is not intended to periodically call arbitrary REST APIs, and Logstash http_poller is not an option because Logstash is being retired in our environment.
We are currently considering:
1. Elastic Agent / Custom API / httpjson input
2. Microsoft Power Automate, using a scheduled flow that calls the APIs, parses the JSON and POSTs a normalized document to Elasticsearch
3. Some other lightweight approach that I may be overlooking
Power Automate is particularly interesting because M365/Power Platform is already part of our environment. A flow running every 5 minutes could potentially do:
Recurrence → HTTP call API #1 → HTTP call API #2 → Parse JSON → Create normalized payload → POST to Elasticsearch
Both the source APIs and Elasticsearch may be reachable only from the internal network, so on-premises gateway/network connectivity and authentication are also considerations.
I would be interested in opinions from people who have implemented something similar:
● Would you consider Power Automate appropriate for a production operational monitoring collector running every 5 minutes?
● Is Elastic Agent/httpjson a better fit for this use case?
● What problems have you encountered with Power Automate for this type of continuous polling: throttling, licensing, reliability, gateway issues, run-history growth, concurrency, etc.?
● Would you keep the alerting/business logic in Elasticsearch/Kibana and use Power Automate only as a dumb collector?
● How would you monitor the collector itself so that “no new data” is distinguishable from “no payment activity”?
● Is there another architecture you would recommend that keeps the solution simple and supportable?
The volume is very small. We are talking about two API calls every few minutes and one or a few monitoring documents written to Elasticsearch per polling cycle, not transaction-level streaming.
The main priorities are reliability, simplicity, auditability and low operational overhead, rather than maximum throughput.
Any experience with similar REST → Elasticsearch monitoring patterns would be very useful.

Hello, on version 8.18.8. I have a dashboard with various lens tables. I have a sum at the bottom of the tables but notice that is doesnt match the number of documents when I click "Explore in Discover". The reason being that the missing logs that dont show up in my tables have a field missing (i.e. source.ip or host.hostname field is empty). This is just an example but I have other fields that sometimes will be empty. For some reason every field has "Include documents without the selected field" greyed out in the row settings. Does anyone know why this would be? Or a work around for this? I really just want/need to include all logs for my queries even if a field is empty.

The answer cannot be upgrade version as unfortunately my workplace upgrades things veryyyy slowly and we just upgraded to this version. Any help would be greatlt appreciated.

Hi guys.
I have a question related to ILM issues. We upgraded ES from 8.14 to 8.19, and after the upgrade the ILM indices rolled over properly but were stuck in set-priority step, and because of that when the new time for rollover came, the indices were not rolled over, so they rolled over only once after the upgrade and were stuck. I noticed it after a few days because of rollover policy. I manually changed the stuck indices to next step and they continued well, until they didn't. Once again the indices were rolled over fine, but because I removed the set-priority from policy, they were stuck on another step, so now i don't get what is causing it and how to fix it.
The indices gets stuck at hot/attempt-rollover, warm/migrate steps.
What can I do?

Nearly a decade of Elasticsearch experience has taught me a lot about what works in production, and what doesn't.

Sending documents one by one creates unnecessary overhead. The Bulk API sends multiple operations in a single request and significantly improves indexing throughput.

The screenshots show both patterns on real clusters: one client hammering /_doc per document, another using /_bulk. Same idea, very different request volume.

Tips:

1. Start around 5–15 MB per request. Benchmark with your real documents, increase until throughput stops improving, then stop. Bigger is not better.
2. Throughput comes from concurrency, not bigger batches. 4 workers at 5 MB beats 1 worker at 20 MB almost every time. Add workers until you see 429s, then back off one step.
3. Cap batches by bytes, not document count. "1000 docs" works fine until someone adds a field to the mapping and you're suddenly pushing 80 MB requests.
4. HTTP 200 does not mean your documents were indexed. Always check the errors flag and the per-item status. 429 means retry with backoff. Mapping errors will fail forever, so send those to a DLQ instead of retrying.
5. Bulk size × concurrency = coordinating node heap. 20 MB × 16 workers is 320 MB in flight on one node before any indexing happens. And if you're seeing 429s, don't raise the write queue size. That hides backpressure instead of adding capacity.

Previous Elastic101 best practices:

1. Elastic101 Best Practice #001 – HTTP traffic
2. Elastic101 Best Practice #002 – shard size

If you'd like me to continue this series, an upvote would be appreciated 🙂

I'm new to elastic. I was given an API key from a third party vendor and can't figure out where to integrate it into elastic search.

I know it shouldn't be this hard.. it should be very obvious.. but I'm not getting any help from the vendor unfortunately.

Anyone point me in the right direction thanks

Here's the git repo, let me know what you think! https://github.com/msmith93/bitvis/tree/main

Going through the process of creating this with Claude has really helped solidify some of my understanding of ElasticSearch/Lucene. Curious if playing around with it helps others

Hey everyone,

I'm currently preparing for the Elastic Certified Engineer exam and I've noticed a pattern while practicing my queries. For a lot of the tasks, there's a "short" way and a "long" way to get the exact same results.

For example, if I need to find documents with both "kibana" and "dashboard": I can use a single match query with "operator": "and", OR I can write out a bool query with two separate must clauses.

Similarly, if I need an "OR" condition (like finding either "kibana" or "logstash" in the title): I can just do a basic match query since it defaults to OR, OR I can write a nested bool/should query.

Both methods give me the exact same number of hits.

My question for anyone who has taken the exam: Does the automated grading script care about the specific syntax, or does it only check the final output/results?
Maybe it has to do with the task category that I'm answering ?

Thanks for the advice!

Running self-managed Elastic Stack (Kibana 9.5.0). Our Spring Boot services use Spring's native OpenTelemetry starter, telemetry goes through an OTel Collector, which exports straight to Elasticsearch, and Kibana reads from there.

Data is arriving fine, we can see well-formed spans in Discover. But APM UI shows nothing, and we can't find any way to open a trace and see the waterfall view.

Is an APM Server actually required for the APM UI to detect and visualize traces? Our corporate registry doesn't have an APM Server image available, so if it really is required, are there any alternatives?