I have add one alert rule about one timeseries .. it will notify something when match , like
- alert: Y
expr: prometheus_notifications_sent_total < 40
for: 10s
labels:
A: A
annotations:
summary: "{{$value}}"
but when I want know more detail about it , I add another alert rule , like:
- alert: X
expr: prometheus_notifications_sent_total{environment="dev"} < 20
for: 10s
labels:
A: A
annotations:
summary: "{{$value}}"
so that , when prometheus_notifications_sent_total(environment="dev") == 10 matched, I will received alert Y and alert X ... but alert Y is no need and meanless .. now we only want alert X.
the best is to fix alert Y expr not include alert X expr like:
- alert: Y
expr: prometheus_notifications_sent_total unless prometheus_notifications_sent_total{environment="dev"}< 40
for: 10s
labels:
A: A
annotations:
summary: "{{$value}}"
it maybe work , but question is:
when I add alert X, I must to fix every alert relate with alert X about diff scope! it is difficult to find it.. alert X and alert Y will add by different user, every know less about scope .... anyone has idea to manager the alert rule ??