[Question]: How to handle Watchdog, which are always critical

[wattebausch]

Ask a question

Hello everyone,

We use prometheus for our kubernetes cluster. As written in your article (https://blog.netways.de/blog/2023/07/25/check_prometheus-ist-jetzt-oeffentlich-verfuegbar/), we define all rules there. And we use the default Rules like general.rules

We would like to use your plugin, but we have a question about how you handle the watchdog. This is set to active by default.
https://runbooks.prometheus-operator.dev/runbooks/general/watchdog
(If not firing then it should alert external systems that this alerting system is no longer working.)

If we now query ‘alerts’ in this way, we always have a critical state.

thanks for your plugin, looks great.

[martialblog]

Hi, thanks for the feedback.

I think you raise an interesting point regarding "dead man's switch" alerts, which is a common use case.
Not sure yet how the plugin should handle this, but I'll give it some thought.

Any ideas are welcome.

Cheers,
Markus