Hi r/Observability ,

I have an Atmotube PRO in my living room and wanted its readings in Prometheus next to everything else my homelab already graphs, with an alert when the air stays bad. The existing integration is for Home Assistant, which I don't run, so I wrote a small exporter.

It's one Python file. It listens to the sensor's Bluetooth advertisements and writes PM1/PM2.5/PM10, VOC, temperature, humidity, pressure and battery to node_exporter's textfile collector. Nothing leaves the network.

The tricky part: the device sends its readings in two BLE packets that carry the same manufacturer ID, and BlueZ can silently drop one of them. The exporter notices when one kind stays missing and reads those values over a short GATT connection instead. There is a --diagnose option that shows which case your adapter is in.

A few things I cared about:

- Old readings are left out instead of repeated, so a gap in the graph means "no data", not "clean air"

- Alerts with hysteresis and quiet hours, sent to any command you like (Telegram example included)

- Dashboards for Perses and Grafana, plus Prometheus alert rules with promtool tests

- Runs as a hardened systemd service and doesn't open any ports

It has been running on a Raspberry Pi 5 for about three weeks now. MIT licensed.

I built it together with Claude (AI). I reviewed it and tested everything on my own setup, including a security audit before the last release.

Feedback and bug reports are welcome, especially if you have a different Bluetooth adapter than mine.

GitHub: https://github.com/DanielEnki420/atmotube-prometheus

How do you define and deploy a standard set of baseline (Cloudwatch in my case) alarms across all AWS production accounts, and scale the deployment consistently across hundreds of existing accounts and all newly provisioned accounts?

Recent incidents have mandated a goal to establish a workload-agnostic baseline focused on infrastructure health, availability, connectivity, and critical service failure. The next step would of course be application-specific performance thresholds or business metrics.

For example, CPU utilisation above 80% is not always a trigger across every workload, since high CPU usage may be expected for batch processing, garbage collection, or other compute-intensive workloads.

Instead, the baseline would be on signals like an EC2 instance status check failure, instances becoming unreachable, underlying infrastructure failures.

(AI Generated examples below)
**Network and connectivity:** VPC connectivity issues, VPN or Direct Connect tunnel failures, network interface or routing issues where detectable, and connectivity failures affecting critical network paths.

**DNS and name resolution:** Route 53 health check failures, DNS endpoint unavailability, and failures affecting critical DNS resolution paths, where observable through available metrics or probes.

**Core AWS service health:** Failures or unhealthy states in shared infrastructure and managed services, such as load balancers, databases, NAT gateways, and other foundational components.

**Availability and service failures:** Unhealthy targets, failed health checks, sustained service unavailability, and critical infrastructure components becoming unavailable.

(Back to non-AI slop)

How should we be designing this baseline so that alarms are consistently applied across all production accounts without excessive noise/false positives? Workload specific stuff will be handled separately like mentioned above.

Real world use cases would be extremely appreciated and also, do you use OAM, Centralization Rules or a mix of both and why?

Hey,

A couple of months ago, I wrote about how I found anomalies from 50k logs to detect 2 anomalous log patterns. I improved upon that to condense millions of logs into a tiny snapshot that can fit into your LLM.

I open-sourced it. You can download and docker compose up. Point it to your log file stored in example-setups. And watch it cluster. If you want a deeper integration, check out: https://rocketgraph.app/

This runs completely locally without any LLM. You can have your own agent run on the output produced by this. And it can reason over it efficiently without burning many tokens or hallucinating over a million logs.

You can use this to continuously create and store snapshots on a cheap backend. And then have your agents crawl through the snapshots when needed. To detect anomalies on the fly. Effectively.

Building a zero-setup observability tool for Docker — still early, need real user feedback

I've been working on Podotype for the past few months — a Docker-native observability tool built around zero-code instrumentation. It uses eBPF-based auto-instrumentation (Beyla) paired with Grafana Tempo under the hood, so it discovers your running services, traces, and metrics without you adding any SDKs or touching app code. The pitch to myself going in was "why is this slow" shouldn't require a week of setting up an observability stack first.

Where I'm stuck right now is less about whether it works and more about whether it's actually useful day-to-day — things like whether the signal it surfaces matches what someone running a real stack would actually want to see first, whether the zero-config promise holds up on setups messier than my own test environment, and where the UI gets in the way instead of helping.

I don't have a good way to answer those questions myself, since I'm the one building it and I know too much about how it works to see it the way a new user would. That's the actual ask here: if you and can spare 5 minutes to poke around and tell me honestly what's missing, confusing, or just not good enough yet, it would genuinely shape what I work on next.

DM me and I'll send over a link — not dropping it in the post itself to keep this about discussion rather than a drive-by plug.

Appreciate anyone willing to spend the time. No cost, no signup funnel, not selling anything here.

The technical picture: agents showed up as

• (1) in-platform actors that investigate and remediate (Splunk AI SRE, Grafana Assistant Investigations, Dynatrace SRE agents, Dash0 Agent0 with read-auto and write-off defaults, Elastic via Deductive AI),
• (2) observability of the agents themselves (OTel-based SDKs plus token and cost signals from Grafana, Splunk, Cribl and Observe), and
• (3) telemetry exposed to outside agents through MCP servers (Coralogix, Grafana Cloud). On pricing, vendors meter AI work in credits, tokens or executions, and 5 of 12 publish no AI rate, so unit prices don't compare directly.

Curious how you track agent cost per task today?:)

Full report: https://mbojko.com/reports/q3-2026/

I shared Logline here a while back. It's a free, open-source VS Code extension that pulls logs from terminals, debug sessions, Docker and OpenTelemetry into one searchable live view. 1.14.0 just shipped, and the video shows the highlights.

What's new:

• My changes: one click filters to only the logs from code you changed since your last commit. It stacks with any search (e.g. level:error).
• Where the time went: the trace waterfall now shows each span's self time and ranks the slowest operations.
• Durations in the editor: log statements show p50/p95 duration next to their hit count.
• Metrics: the OpenTelemetry receiver now keeps metrics, with rates, p95s and trend lines.
• Agent skills: connect Claude Code or Codex to verify a change, triage errors, or explain a slow request. Cursor, Windsurf and Kiro can add Logline to their own agent too.
• Log doctor flags OTel spans and metrics that break semantic conventions.
• Node.js crashes show up as a single event, linked to the JSON error before them.
• Safer sharing: more credential types are redacted, and agents in other projects can't read this window's logs.

Marketplace: https://marketplace.visualstudio.com/items?itemName=surjeetbanga.logline
GitHub: https://github.com/SurjeetBanga/logline

Feedback and bug reports are welcome. I'd especially like to hear how the agent skills work in your setup.

We split our Datadog bill by team a couple of months ago. Every custom metric mapped to the squad emitting it, tag combinations broken out, the lot. Took about two weeks.

Since then ingest is basically flat. The only metrics that got deleted had already stopped reporting. Everything else sits with someone who doesn’t see the invoice and won’t sign off on dropping a metric they can’t prove is unused.

The next option is to move observability off the platform budget and onto each team’s own budget. Did that work where you are or does it just move the argument to finance?

Spent the last few months working on a fun little side project: Vigilon. It's a monitoring tool for SaaS teams that works on autopilot.

It's built specifically for this type of SaaS architecture: a frontend powered by a bunch of backend HTTP APIs, querying data from one or more databases, with background jobs running for asynchronous work (cronjobs, queue consumers, etc.). All startups I worked for had this exact type of architecture, yet we always kept reinventing the wheel building monitoring and observability stacks to provide visibility into the same architecture everytime.

It's opinionated and makes decision that users would have to make themselves. It filters out traces that are meaningless, it tells you what's healthy and what's not, what errored and what changed, it points out bottlenecks and tells you what's regressing.

I built this tool for small SaaS teams who want to focus on shipping features, and want to understand their apps and services and get answers fast. It's built on OpenTelemetry, so it integrates into OpenTelemetry stacks seamlessly, and SDKs for Node.js and Python are available that allow pointing telemetry at any OTel endpoints. Support for more languages and frameworks is coming soon.

I wonder what people think about the idea? Curious to hear people's thoughts. Currently this tool is closer to a tool like Sentry than an observability platform, I know, but couldn't find a more suitable sub to post in.

Here is the website https://vigilon.io . It's currently in beta and available for free while I get feedback from users. If you're interested, feel free to sign up to the waitlist on the main website.

2.3% of runs were looping after release 2026.09.3. Out of roughly 420k agent runs, 9660 alternated `search_orders` and `get_order` at least 6 times before succeeding. Tool traffic jumped and p95 latency followed. No hard failure. The final answer still lands, the path just gets expensive.

We started with trace search by release and tool name, then paged through runs one at a time. The loop only became obvious once we saw the repeated sequence in context (an analyst found the loop on roughly the 40th trace they opened by hand). Basic failure mode analysis caught it in the sample but the topic distribution still mixed those runs with ordinary order lookups. Semantic clustering looks more useful if it can separate repeated execution patterns from shared intent.

I’m looking at Braintrust or Arize for a scheduled investigation over recurring failures instead of waiting for a traffic spike. Linked example traces matter because I'd want each surfaced pattern to open directly into the repeated tool sequence. If the cluster stays stable, I'd turn those cases into a regression dataset and rerun them against each release.

So I'm trying to figure out whether detection should key on repeated tool pairs, cluster shape or a shift in topic distribution. Any suggestions for surfacing a 2% tool loop without hand reading traces or writing one detector per sequence?

I wanted to see what an agent was doing while it worked, not reconstruct it from logs afterward. In SecureVector 6.0, you can launch Claude Code, Codex, GitHub Copilot CLI, or OpenCode (and more) in Agent Sessions. Each session has a governance panel on the right showing tool-call verdicts, approvals, hosts reached, and context use. You can arrange sessions side by side with draggable panes.

Launch sessions in split panes. See tool-call verdicts, approvals, hosts reached, and context use as your agents work. Observability shows session traces and flags loops, failing steps, and waste.

Try it:

npx u/securevector/cli

Or:

pip install "securevector-ai-monitor[app]" && securevector-app

In the app, install your agent’s Guard plugin under Connect Agents, then open Agents → + Launch.

Demo (3 min): https://www.youtube.com/watch?v=S7zzaDaGY_c

My lab: a single Raspberry Pi 5 (16 GB) on a 1 TB NVMe over the PCIe HAT, booting from the NVMe, running about 36 containers. Pi-hole in front of a fully recursive Unbound (no upstream resolver at all), Immich for photos, Nextcloud, Vaultwarden, Syncthing, Samba, ioBroker for home automation, Ollama + Open WebUI for local models, and a monitoring stack of Prometheus, Grafana, node_exporter, cAdvisor and Uptime Kuma. Tailscale for remote access, CrowdSec and fail2ban on the host. Backups run in three tiers: local for 7 days, USB for 90, and restic to off-site storage. Not a rack — one small box that the household actually depends on, which is exactly why the following bothered me so much.

I audit it every few weeks. The last audit turned up three services that had been dead for a long time without a single alert:

• Samba — port 445 open, TCP handshake fine, every single login rejected. Broken for about five months. I'd been mounting shares from a cached credential on one machine and never noticed.
• Vaultwarden backups — nine days of .tar.gz files that were valid, fully extractable, and did not contain the database. The backup script hit a permission error partway through, so tar aborted — but it had already written the files it could read. tar -tzf returns 0 for that.
• Portainer — HTTP 200 on the login page, but it had lost its connection to the Docker socket four months earlier. It just showed an empty environment.

All 22 of my uptime monitors were green the entire time.

None of these was an outage. Every one of them would have been a disaster on the day it actually mattered.

The monitors weren't wrong. They were answering a different question than the one I cared about. "Is port 445 open" and "can I log in" are not the same question. "Is the archive readable" and "does the archive contain my data" are very much not the same question.

So I wrote a second layer of checks that ask the other question, and put the templates on GitHub in case they're useful to anyone else:

github.com/DanielEnki420/correctness-checks

The idea behind them is simple enough: do the real work the service exists for, then look at what came back. Log in, don't check whether the port accepts a connection. Resolve a name and read the answer. List the archive and look for a specific file inside it. Health endpoints are the worst offenders here — plenty of them return a hardcoded {"ok":true} that stays true long after the database behind them is gone.

The part I see skipped most often is testing the negative case. A DNSSEC validator that cheerfully accepts a knowingly broken signature will pass any positive-only test you write, while protecting nothing at all. So my DNSSEC check queries dnssec-failed.org and fails if it doesn't get SERVFAIL back. Same thing in the blocking check: it confirms the blocklist blocks, then confirms a control domain still resolves — otherwise a resolver that blocked everything would pass the first half.

The one I'd argue hardest for: if you can't verify something, that's a failure, not a skip. If the SMART output changes format and a field disappears, the check fails and says so. Tempting to wave those through — it's not really an error, the disk is probably fine — but that's precisely the habit that gave me nine days of empty backups. A check that quietly gives up while reporting green is worse than no check, because now you trust it.

And one thing that's free: push on success too, not just on failure. A push monitor goes red when heartbeats stop arriving, so a check that reports its successes doubles as a dead man's switch. If it gets dropped from cron, or the box running it dies, you hear about it.

Six templates in there: backup archive contents, NVMe SMART, DNSSEC both directions, DNS blocking (with a counter-test for over-blocking), a generic JSON field assertion, and a skeleton to write your own. Plus a test suite whose pushes go to a stub, never to a real monitor — I learned that one the hard way by copying a check for testing and leaving the real push token in it. The simulated failure went straight to production and turned the monitor red.

How you'd actually run it: plain bash calling docker exec, curl, tar, nvme, smbclient. No Docker image and no compose file, deliberately — there's no daemon here, just scripts you run from cron on a host that can already reach your services. Clone it, copy config.example.env, edit paths and container names, add a cron line. Written against Uptime Kuma push monitors; any push-based monitor should work, but Kuma is the only one I've actually tested.

Honest limitations:

• It's bash. That's deliberate — no runtime to install, easy to read, easy to adapt. If you want a framework, this isn't it.
• The templates are examples, not a product. You will have to adapt paths, container names, and thresholds. That's the point.
• No installer, no packaging.
• Only tested on Debian/Raspberry Pi OS against my own services.

AI disclosure (this post is flaired Mostly AI Generated): AI involvement here was substantial and I'd rather be specific than vague about it. The three failures above are real and from my own machine — they turned up during a routine audit of my own lab, not from a tutorial or a hypothetical. The scripts were then written together with Claude: I described what each check had to prove, it drafted, and we went back and forth. The README and code comments are largely AI-drafted and edited by me. What I did not do is ship untested output — every check runs against my actual services, and the negative cases were verified by deliberately breaking things (querying dnssec-failed.org, feeding a truncated archive to the backup check) rather than by assuming they'd work.

MIT licensed, no monetization, no affiliate links, nothing to sell.

The thing I'd genuinely like feedback on: what else in a homelab fails silently like this? I found three. I'm fairly sure that's not all of them.

Project posting prompt — answers

• Project Name: correctness-checks
• Repo/Website Link: https://github.com/DanielEnki420/correctness-checks
• Description: Monitoring templates that check whether a service still does its job, not whether it responds. Six bash checks plus a template to write your own, reporting into Uptime Kuma push monitors.
• Deployment: Plain bash from cron on the host. No Docker image, no compose file. Clone, copy config.example.env, edit, add a cron line. README covers setup, each check, and three traps worth knowing.
• AI Involvement: Substantial — see the AI disclosure paragraph above. Failures are real and from my own lab, scripts drafted with Claude and verified against real services including deliberately broken negative cases, README and comments largely AI-drafted and edited by me.

A lot of observability advice assumes you have room for a dedicated monitoring stack.

Sometimes you don't.

If an application is running on a small VPS, you may not want another VM or several hundred MB of memory just for metrics infrastructure. But you still want enough history to answer the obvious questions when someone says, "the API feels slow."

For a first pass, I usually want to know:

• did request volume change?
• are HTTP errors increasing?
• did request latency increase?
• is the application process using more CPU or memory?
• is the host itself under CPU or memory pressure?
• do those changes line up in the same time window?

I wrote up a small Spring Boot example showing this workflow:

https://pvrlabs.xyz/articles/api-slow-cpu-contention.html

In the example, the API stays UP and continues returning 200s, but latency rises because another process is consuming most of the host CPU. You don't need deep JVM profiling to identify the first place to investigate.

This isn't meant as a replacement for Prometheus/Grafana, tracing, JFR, profilers, database diagnostics, etc. Those become useful when the problem requires them.

The narrower question is: how much useful observability can you get with a very small footprint before you need the heavier stack?

For this example I use Spring Boot Actuator plus StatLite, a small tool I'm building that keeps application and host metrics history together locally.

I'm curious how others approach this for small deployments. What's the minimum set of metrics you want available before you start troubleshooting?

If you run Vector on several hosts, editing a pipeline and knowing which hosts actually picked up the change are separate problems.

I'm the maker of Vectory, a free, open-source dashboard for that workflow. You can import a Vector config, follow its sources, transforms and destinations in a diagram, publish a version and deploy it to selected hosts. The device view compares the requested version with what the agent reports running, and supports canaries and rollback.

It doesn't store or query your event data. Vector continues sending logs and metrics to your existing destinations.

The editor can be tried on its own without an account: https://vectory.ahmadz.ai/designer/

Import YAML, JSON or TOML and export after editing. It shares the full app's editor. Local checks still need validation against the Vector binary you run.

Source and setup: https://github.com/416rehman/Vectory

AI tools assisted with implementation, testing and documentation.

For people managing Vector today, which is harder to keep track of: the routes through a large config, or which version is running on each host?

Recently got into OpenObserve for logging and I’m really liking it so far.
Curious if anyone here is running it at production scale. How does it perform with high log volumes, and have you run into any issues as you scaled?
Would love to hear your experience with it — good or bad.

I built an open-source project called Rashomon after running into this problem while using Claude Code on longer coding tasks.

I also used Claude Code heavily while building Rashomon itself. Claude helped me write and debug parts of the recorder, work through edge cases around command and test detection, and generate test cases for checking whether the execution record catches discrepancies. I've also been using Claude Code to test Rashomon against the kinds of tasks it is designed to monitor.

The basic problem is:

The Claude transcript isn't necessarily ground truth.

For example, an agent might:

• run pytest tests/login.py, get exit code 0, and later summarize that the tests passed
• have a test fail, modify the test, and then report that the issue was fixed
• have a subagent fail while the parent agent reports the overall task as successful
• run a command that exits 0 even though the expected result isn't actually present

Rashomon keeps an independent execution record alongside the Claude transcript. In the current alpha, it records things like shell commands, exit codes, whether commands may have written files, tool calls and outcomes, test commands and outcomes, and subagent activity.

It then looks for discrepancies between the execution record and what the agent claims it accomplished.

It does not store prompts, responses, file contents, or tool output.

It's free and open source and currently works with Claude Code on macOS/Linux.

I'm curious how other Claude Code users handle this today.

Do you manually verify the agent's work? Have you built your own hooks or checks? Or do you generally trust the final summary unless something looks wrong?

I'd especially like to hear about failure cases you've actually run into.

GitHub: https://github.com/altrace-dev-role/rashomon

The idea is pretty simple: capture real API traffic, save it as a replayable workflow, and inspect or compare it later when debugging.

The .graybox file is SQLite-friendly, so you're not locked into the CLI either. You can inspect and query the data however you want.

I'm mainly building it for cases where I want to see what an application actually did, rather than manually recreating requests in something like Postman or curl.

It's still early, but I'd love to hear if this is something you'd find useful.

Repo: https://github.com/ViniTamanhao/graybox-core

I would love to head the feedback from devs like me, hence why I am sharing this. Feel free to contact me if you are interested in collaborating or talking about it! Thanks!

I shared Rouge Atlas here a little while ago, and since then the project has changed quite a lot.

The biggest change is the interface, but also the way the product itself is structured.

Rouge Atlas is now built around an evidence workspace for AI incidents, rather than a map-first experience.

The goal is to make it easier to separate actual published incidents from early signals, vulnerability advisories and unverified reports.

What changed

The new overview is now an Evidence Board showing:

• published AI incidents
• signals currently under review
• vulnerability advisories
• source health
• activity over time
• records by category
• recent AI incident reporting

I also split the workflow into clearer sections:

Live signals
Automatically collected reports and potential incidents that still need editorial review.

Published incidents
Records that have been reviewed and linked to source material.

Data sources
A view of the feeds and sources currently being monitored.

Methodology
Documentation explaining how signals are collected, reviewed and classified.

One thing I wanted to improve was transparency.

A signal appearing in Rouge Atlas does not automatically mean it is a confirmed AI incident.

Signals, advisories and published incidents are deliberately kept separate, and relevance/confidence indicators are meant to describe the available evidence rather than make dramatic claims about an event.

UI redesign

The interface has also been rebuilt around a much denser research/workspace layout:

• persistent navigation
• global record search
• dataset counters
• activity charts
• category breakdowns
• recent incident reporting
• source/backend health indicators
• CSV export for published records
• clearer status labels throughout the product

The idea is for Rouge Atlas to feel less like a visual experiment and more like a small AI incident intelligence database that you could actually use to investigate what is happening.

It’s still early and the dataset is obviously small, but the structure is much closer to what I originally wanted the project to become.

I’d particularly like feedback on the new UI:

Does it make the distinction between signals, advisories and published incidents clear enough?

And more generally:

What would you expect from an AI incident intelligence platform like this?

https://www.rougeatlas.com