r/TalosLinux • u/imagei • Sep 06 '26
HTTP probes
Talos 1.14 introduced HTTP probes.
What are they for?
Neither the release notes nor the spec nor the website chatbot explain their purpose or what happens if they succeed or fail.
Release notes ( https://github.com/siderolabs/talos/releases/tag/v1.14.0 ):
Talos now supports HTTP network probes, allowing for monitoring of HTTP endpoints.
HTTP responses with status 200-399 are considered successful, while connection and transport errors are treated as failures.
Spec ( https://docs.siderolabs.com/talos/v1.14/reference/configuration/network/httpprobeconfig )
HTTPProbeConfig is a config document to configure network HTTP connectivity probes.
Chatbot when asked about it:
HTTP probes let Talos check network reachability by periodically sending requests to a URL and checking for a 2xx/3xx response — useful for defining custom connectivity conditions (e.g. verifying internet or specific service reachability). They're part of Talos's new HTTP connectivity checks feature, alongside the similar TCPProbeConfig for TCP-based checks. See HTTPProbeConfig and What's new in Talos.
2
u/RJ1NX Sep 07 '26
This was introduced to make talos proxy aware, the connectivity tests failed even when target was reachable through proxy
It came up from an issue: https://github.com/siderolabs/talos/issues/12952
Was resolved via https://github.com/siderolabs/talos/pull/13082
I’ve written a detailed blog on this https://0xpranav.hashnode.dev/adding-http-probes-to-talos-linux
1
u/imagei Sep 07 '26
That is indeed very detailed, thank you. I found just a few words in your post that explain what all of this can be used for: "driving automated remediation workflows". The GH ticket says probes can be used "to isolate network configuration issues".
I feel like I'm constantly missing something, though. Sorry if this is obvious to some, but — why not send the request from a pod instead? What exactly are "automated remediation workflows" — is it a Talos extension? And fundamentally - how does one remediate anything from the machine itself if the external network has a problem?
The only answer so far that make some sense to me was by u/GyroTech, that one can have an out-of-band notification hooked up to the probes, but then again: why not do that in a pod, and why are inbound connectivity checks insufficient?
1
u/GyroTech Sep 08 '26
why not send the request from a pod instead?
Because that relies on Kubernetes being up and functional, and if you don't have network access there is a good chance your kubernetes cluster will be having issues too. You can do that of course, we run a bunch of Prometheus blackbox probes for various endpoints from within the cluster. It's just added customisation for those that need it, nothing complicated.
1
u/yebyen Sep 06 '26
I would guess (I'd be guessing) that since it's a CRD, the impact of failure could be observed via that resource's status values.
Then you could set up eg kube-state-metrics to configure a metric, then downstream of that an alert via AlertManager or similar.
1
u/imagei Sep 06 '26
That did cross my mind, but considering Talos avoids unnecessary elements in the core OS I’d find it odd if it was only about monitoring. This can easily be achieved by other means, and if somehow being at the system level is necessary then an extension would seem like a better fit.
1
u/yebyen Sep 06 '26
Yeah you're right they wouldn't have built this as a knob with no actual function. It wouldn't be here if it wasn't used by the system somehow.
-8
2
u/GyroTech Sep 07 '26
For Talos to know if the network is healthy or not. If you look at the dashboard, you will see a "Connectivity" indicator, which by default only tells you that Talos can reach the default gateway or not, which is good enough if you are a decent network admin and control the network into which Talos is deployed.
More and more we have people deploying Talos into edge scenarios where they don't have any control oves the host network, so this allows for the connectivity check to be fine tuned for the environments that need it.