Prometheus: how to drop a target based on Consul t

Question

My Prometheus server gets its list of targets (or "services", in Consul's lingo) from Consul. I only want to monitor a subset of these targets. This should be possible via Prometheus's regex mechanism, but the correct configuration eludes me. How is this done?

Answer 1

I've used a very similar solution to the problem using the following config. It allows to scrape only the services with a specific tag, rather than excluding services with a given tag.

Here's the scrape_configs section of my config:

scrape_configs:
  - job_name: 'consul_registered_services'
    scrape_interval: 5s
    metrics_path: '/prometheus'
    consul_sd_configs:
    - server: 'my-consul-server:8500'
      token: 'xyz'
    relabel_configs:
    - source_labels: ['__meta_consul_tags']
      regex: '^.*,metrics_method=prometheus-servlet,.*$'
      action: keep
    - source_labels: ['__meta_consul_node']
      target_label: instance
    - source_labels: ['__meta_consul_service']
      target_label: service
    - source_labels: ['__meta_consul_tags']
      target_label: tags

I then make sure to register all relevant services with the metrics_method=prometheus-servlet tag, and the rest will be ignored.

The documentation for the relabeling configuration is available here: https://prometheus.io/docs/operating/configuration/#relabel_config.

The documentation for the Consul service discovery configuration is available here: https://prometheus.io/docs/operating/configuration/#consul_sd_config.

Answer 2

I've scoured the web and there is not a single example showing how its done, so for posterity - the following configuration will drop all consul services marked with the 'ignore-at-prometheus' tag

# ignore consul services with 'ignore_at_prometheus' tag
# https://www.robustperception.io/little-things-matter/
relabel_configs
- source_labels: ['__meta_consul_tags']
  regex: '(.*),ignore-at-prometheus,(.*)'
  action: drop