prometheus alert rule with different scope

2019-08-20 11:02发布

问题:

I have add one alert rule about one timeseries .. it will notify something when match , like

- alert: Y
  expr: prometheus_notifications_sent_total < 40
  for: 10s
  labels:
    A: A
  annotations:
    summary: "{{$value}}"

but when I want know more detail about it , I add another alert rule , like:

- alert: X
  expr: prometheus_notifications_sent_total{environment="dev"} < 20
  for: 10s
  labels:
    A: A
  annotations:
    summary: "{{$value}}"

so that , when prometheus_notifications_sent_total(environment="dev") == 10 matched, I will received alert Y and alert X ... but alert Y is no need and meanless .. now we only want alert X.

the best is to fix alert Y expr not include alert X expr like:

- alert: Y
  expr: prometheus_notifications_sent_total unless  prometheus_notifications_sent_total{environment="dev"}< 40
  for: 10s
  labels:
    A: A
  annotations:
    summary: "{{$value}}"

it maybe work , but question is:

when I add alert X, I must to fix every alert relate with alert X about diff scope! it is difficult to find it.. alert X and alert Y will add by different user, every know less about scope .... anyone has idea to manager the alert rule ??