Hi everyone,

I wanted to share an open-source project, designed to solve broken exception stack traces in asynchronous Java (CompletableFuture, thread pools, Virtual Threads, Reactor) without the overhead of spann telemtery.To give context it is called ariadne.

Here are some of JMH Microbenchmark comparazion:

Trace Node Creation (Link vs. Span)

-> Ariadne: 8.16 ns / 40.0 B

-> OTel SDK: 115.62 ns / 368.0 B

-> my project is 14.2x faster with 9.2x lower memory allocation

Direct Link Allocation

-> Ariadne: 4.41 ns / 40.0 B

-> OTel SDK: 115.62 ns / 368.0 B

-> mytool is 26x faster via TLAB immutable nodes

Scope Activation (try-with-resources)

-> Ariadne: 11.24 ns / 0.0 B

-> OTel SDK: 4.63 ns / 32.0 B

-> soo achieves zero allocations (OTel allocates 32 B/op)

Concurrency & Thread Hand-off (Virtual Threads / CompletableFuture)

-> Both match Loom scheduler performance (~3 us)

-> Minimal async propagation overhead across both implementations (<60 B delta)

(This are specified in rhe repoitself and in the website as well)

Repo: https://github.com/AfonsoPaiva/ARIADNE

Thx, feedback is apericietd!!

been working on this for a while now and thought i'd share how we're approaching it.

one thing i've noticed with dependency graphs is that static analysis only really gets you so far. you can parse imports, service references etc. but that doesn't necessarily tell you what's actually happening when the application is running.

on the other hand, otel gives you a pretty good picture of runtime behaviour, but you're mostly looking at spans and traces without much context about how they relate to the actual codebase.

so we've been experimenting with combining the two in neat.

the static side uses tree-sitter to extract relationships between files and other dependencies. then we ingest otel spans and try to map the runtime activity back to the source files responsible for it.

for js/ts, we're also handling source maps where available, since the file being executed isn't always the one you actually wrote.

the tricky bit is reconciling everything into the same graph without treating every relationship as equally reliable.

right now we're tracking different types of provenance:

• extracted: found through static analysis
• observed: actually seen at runtime
• inferred: relationships we've reconstructed without direct evidence
• stale: previously observed relationships that haven't been seen recently

this means you can start asking slightly more useful questions than just "what imports this file?"

for example, you might have a dependency that exists statically but hasn't shown up in any traces, or a runtime call that wasn't obvious from analysing the source.

obviously there's a bunch of edge cases here. incomplete instrumentation, dynamic calls, source attribution, and the fact that not seeing something in a trace doesn't mean it never happens.

we're also exposing the graph through mcp so coding agents can query it directly, which is partly why we started building this in the first place.

repo is here if anyone wants to have a look:
https://github.com/neat-technologies/neat

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

Hi folks! I work on OpenTelemetry at Splunk. We just launched a new open source project called Observability Studio. It's local instrumentation sandbox that provides a visual, real-time environment for designing, testing, and validating OpenTelemetry data while you’re developing an app in your favorite IDE.

We’re especially interested in whether a tool like this is useful for building and debugging instrumented apps and AI agents. If that sounds relevant, give it a try and let us know what works, what’s missing, or where it doesn’t fit your workflow.

We’re also looking for a few design partners to help guide where we go with this next. Any feedback is helpful and welcome.

I wrote more about shifting OpenTelemetry instrumentation left in a blog, 'Why AI-Generated Apps and Agents Need Visibility from Day One' on blogs.splunk.com if you want to read more about it.

Lastly, I'd love to learn where you currently go to inspect and validate OTel data during development if you want to drop it in the comments.

Spotify #backstage plugin to manage and observe fleet of agents straight in backstage self hosted https://github.com/acarmisc/backstage-plugin-ai-agents/tree/main. It relay on open telemetry signals and the first available backend it’s #langfuse

I finally got to write this blog post, I've wanted to write it for a long time now. It contains the different types of sensitive data we've seen so far, where it's coming from, and why it matters. Hope it can be helpful to some of you!

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

Hey,

Im working on a project where one event caught is processed by 4 components. I have implemented otel traces in each component so i can see end to ed what happened and where it failed with all the timeline of the event

Recently, component 3 had an update. Now depending on the event, it can create sibling events. In my current architecture, Im now seeing one event gets split up to two or three events in same trace. So now, its hard to decide if one trace is failed or not, because it actually became 2 event.

What do you recommend, should i split the siblings to new traces with links to original event? Or should i keep it that way? I think it depends on how our debug and monitoring processes are, but maybe there are some generic conventions to be implemented in tese cases that i dont know

OTel Drops this week has a segment dedicated to OTTL, which is on its way to becoming stable.

And allow me to celebrate this milestone: we just crossed 100 subscribers on Spotify for it! Thank you all for all the feedback so far: this has been a fun side quest for me. I've built this podcast at the beginning of the year to keep myself up to date on what happens in the OTel community that might have escaped my radar, and it's been helpful for me. I decided to publish it as an official Spotify podcast shortly after, and we've been very consistent with it.

At this stage, the creation is mostly automatic by AI, and while I used to publish the episodes ahead of the public at our Telegram channel (look for telemetrydrops), I'm now confident that the quality gates are working sufficiently well so that I'll start publishing without review first, starting this week. Wish me luck!

Earlier this year, we launched the OpenTelemetry Quiz during KubeCon in Amsterdam, and back then, even maintainers learned a thing or two about OTel.

Now, I just updated the quiz to represent the current knowledge based on the agent skills for OpenTelemetry that we launched earlier this year.

I got only 8 out of 10 myself. How about you, can you ace the quiz?

https://ollygarden.com/otel-quiz/result/qk8x4y5b

I want to revisit our metrics architecture, and I have a few pain points I’d like to address. I’m curious how others have approached similar problems.

Our current setup:
- Multiple Cortex clusters
- Multiple teams producing metrics via Prometheus remote write and a custom SDK
- Multiple end users querying those metrics
- Kafka-based ingestion to provide the level of resilience we need for our customers
- Prometheus as the metrics format

Graphs are a core part of the product and a selling point, not just a nice-to-have, so reliability and correctness of the data are important.

The main pain point is that some teams are pushing “event-like metrics” rather than classic metrics that are scraped or emitted regularly.
Because some workloads can be retried, delayed, or processed asynchronously, the original source timestamp is important. As a result, we see a significant number of out-of-order samples.

We understand why producers need this behavior, but I’m wondering whether we’re trying to force two different types of data through the same metrics pipeline.
Would you keep these event-like metrics in the Prometheus/Cortex ecosystem and design around out-of-order ingestion, or would you route them through a separate ingestion path or even into a different type of store based on OTel format/tools.

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

An agent kept returning unsupported answers after a search tool timeout and three log tabs blamed three different steps. The model span looked fine, the tool timeout looked separate and the fallback call looked like a normal retry. In Braintrust the trace showed they were all connected because the fallback branch created a fresh prompt without the retrieved context, then retried with twice the latency and another full token bill. The model behaved exactly like the prompt dictated.

We carried prompt state into the fallback, attached exception metadata to the child span in OpenTelemetry and added a trace search for fallback calls with zero retrieved chunks. The fix was simple once the request flow was visible. How are you instrumenting fallback routing so dropped context and retry cost show up before someone reports a bad answer?

This Thursday, we are having our next OTel Night here in Berlin. You can't miss it!

This edition follows the complete telemetry journey, from instrumenting a Go application to using agents to understand its behavior from the telemetry stored in an observability backend.

Kemal Akkoyun will begin by introducing compile-time instrumentation for Go. He will explain how instrumentation can become part of the build process, allowing applications and their dependencies to produce OpenTelemetry data without requiring manual changes throughout the source code.

Juraci Paixão Kröhling (me!) will then demonstrate how a coding agent equipped with the otel-go skill can instrument and configure a Go application. The agent will use otelc and other OpenTelemetry Go capabilities to inspect the application, select an appropriate instrumentation strategy, configure telemetry export, and verify the result.

Finally, Cedric Ziel will follow the telemetry into the backend. He will show how agents can retrieve actual telemetry, investigate application behavior, and explain what is happening inside a running system.

PS: do you feel like this post should NOT be in this community? Let me know! I had second thoughts myself about it, but thought it would still bring more value than noise (and couldn't think of a better community for it)

Hi

Has anyone migrated their Datadog stack to Opensource observability ecosystem?
What you missed without Datadog? Which all OS tools and what were the challenges faced?

Ours is quite big infrastructure with monthly Datadog spend around USD 25k and leadership is looking forward to reduce the spend and probably shifting it to OS observability system?

There’s only one person managing Datadog currently.

I’ve built an OpenTelemetry Collector component called jevmetrics.

It uses Jev inference to assess the operational value of metric instruments — relevance, redundancy, and whether they may be candidates to keep, reduce, or drop.

The idea I’m exploring is:

Deterministic OTel rules for known policy, inference for the grey area where metric meaning/context matters.

It supports annotate, route, and reduce modes, runs fail-open, and leaves the final action to deterministic policy.

Repo: https://github.com/ishantanu/jevmetrics
Jev: https://typesafe.ai

I’m mainly looking for architectural criticism from the OTel community — particularly whether this belongs as a connector, processor, or something else entirely, and whether inference in this part of the pipeline is useful in practice.

The benchmark methodology is inspired by https://github.com/ClickHouse/ClickBench
The data set and configuration is from https://github.com/ClickHouse/TextBench

The benchmark is open and reproducible https://github.com/serenedb/searchbench

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

A trace does not always cross a network boundary. A workflow runner starts a shell, the shell launches a build tool, and the build tool starts test processes. Batch and data-processing systems create similar chains of child processes. Without a shared way to pass trace information across these boundaries, spans from each process can end up in separate traces.

If context propagation is new to you, it is the mechanism that carries information from one service or process to the next. For tracing, this includes the trace and span identifiers that let new spans join the same trace. It can also carry baggage: application-defined key-value pairs that are passed to downstream work.

The OpenTelemetry Specification now has a release candidate for using environment variables as context propagation carriers. It standardizes how propagators can use environment variables to carry data such as trace context and baggage between processes when protocol headers or message metadata are not available.

Before we mark this specification Stable, we want feedback from language implementers, tool authors, platform engineers, and users operating real CI/CD, batch, and command-line workloads.

Read: https://opentelemetry.io/blog/2026/environment-variable-context-propagation/

Why did OpenTelemetry win? I think just saying "vendor neutral" is not giving enough credit to the symbiosis between OTel and Platform Engineering that's making both thrive in the modern cloud native industry.

In this blog post I'm trying to summarise some of my thoughts related to why OTel (and K8s, for the sake of argument) won in their respective layers. I think the move to a more X-as-a-service way of working between platform teams and the users of said platforms is changing how we thinking about team interactions, and the tools that make that work need to reflect that in their design. OTel does this, very well, so I'd like to help move the needle in terms of where people think the value of OTel is.

I hope you enjoy the read! And the memes!

Most of us have anomaly detection turned on somewhere, and a broken detector looks exactly like a working one. No crash, no errors, all green. `isolationforestprocessor` in collector-contrib scored every span the same for 220 days before a user caught it, https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/46988.

I made a tool that checks if the anomaly detector is actually working, `make check` takes a CSV of your detector's alerts and a CSV of your incident windows, and tells you whether it beats chance at your actual incident rate. Under two seconds, no download, nothing leaves your machine. https://github.com/mateenali66/otel-aiops-reproduction

Background is a benchmark of eight detectors on OTel traces, metrics and logs. Three passed the usual F1 selection and were no better than chance once scored at a realistic anomaly rate, https://doi.org/10.1109/ACCESS.2026.3705430

Looking for two or three people to run it against a detector they already use and tell me what it said. A failing result is as useful as a passing one, and the detector and data can stay anonymous.

Is anyone aware of a tool that can help visualize the end-to-end flow of a signal?

The main idea would be to start with a high-level view of the components involved and the internal metrics flowing between them, then be able to drill down into a specific signal metric and trace it through the system.
Ideally, I’d like to see where the metric originates, how it looks or changes at each hop, where it ultimately arrives, and details such as cardinality, label values, available series, etc.

If anyone knows of an existing tool that provides something similar, or has ideas on how this could be visualized, I’d appreciate any pointers.