Dynamic label values in Promethues alerting rules

问题:

I am a bit of a novice with the monitoring world. Here is my question.

I want to fire an alert only for a set of assets based on asset-id.

My metrics looks like the below.

test_value{asset_id="123"} 0.215

My alert manager rules looks like the below.

name: iot_rules
  rules:
  - alert: threshhold_alert
    expr: test_value >= 4
    #for: 1m
    labels:
      severity: critical      
      probableCause: Communication failure
    annotations:      
      summary: 'Error detected on {{$labels.assset_id}}'

I get the templating feature on the annotation. However, my promQL expression does not allow me to template. Basically, i am looking for writing a expression as below.

expr: test_value{asset_id=$1} >= 4.

The value for $1 will come from elsewhere (list of assets).

Is this a possibility? I don't want to hard code the asset-id in the expression and there by creating same rule for each asset. Basically the assets-id are unknown at the development time and I don't want my customer to create the rules.

回答1:

PromQL itself does not support templating. You do have a few choices of doing this, though:

• Have whatever deployment tool you're using (Ansible, Chef, Puppet) populate that $1 with a regular expression that lists all the assets you're interested in (and use the =~ matcher instead of = in your PromQL expression).

• Create another metric (either by pushing it to a Pushgateway or by defining it in a separate rule file) with an asset_id label populated with all the asset IDs you're interested in, e.g.:

  should_alert{asset_id="123"} 1
  should_alert{asset_id="124"} 1
  should_alert{asset_id="125"} 1
  

  and then define your alert expression as:

  expr: test_value >= 4 and on (asset_id) should_alert