I'm attracted to prometheus by the histogram (and summaries) time-series, but I've been unsuccessful to display a histogram in either promdash or grafana. What I expect is to be able to show:
- a histogram at a point in time, e.g. the buckets on the X axis and the count for the bucket on the Y axis and a column for each bucket
- a stacked graph of the buckets such that each bucket is shaded and the total of the stack equals the inf bucket
A sample metric would be the response time of an HTTP server.
Grafana v5+ provides direct support for representing Prometheus histograms as heatmap.
http://docs.grafana.org/features/panels/heatmap/#histograms-and-buckets
Heatmaps are preferred over histogram because a histogram does not show you how the trend changes over time. So if you have a time-series histogram, then use the heatmap panel to picture it.
To get you started, here is an example (for Prometheus data):
Suppose you've a histogram as follows,
http_request_duration_seconds_bucket(le=0.2) 1,
http_request_duration_seconds_bucket(le=0.5) 2,
http_request_duration_seconds_bucket(le=1.0) 2,
http_request_duration_seconds_bucket(le=+inf) 5
http_request_duration_seconds_count 5
http_request_duration_seconds_sum 3.07
You can picture this histogram data as a heatmap by using the query:
The Answer from @brian-brazil above works almost, with some additional, not mentioned things, to be done.
You can do a standard non-stacked graph of the rate a histogram, and as Prometheus histograms are cumulative you'll get the result you're looking for.
- The x-axis in Grafana needs to be in mode series
- You need to aggregate the results by the le label, if you use additional distinct labels for the series
After that you'll get a beautiful histogram. The only thing that grinds my gears is that the x-axis sort order of grafana is natural string sorting. So the x axis starts with +Inf, then 0.1,0.2, .. 1,1.5,10,2,...
PS: In grafana 5.1 there will be working heatmap with prometheus datasource out-of-box. There was an issue for native support for heatmap visualization, which is also appropriate (if you want to see tendency/history) for visualizing histograms over time.
I don't believe that Grafana supports a barchart for a histogram.
You can do a standard non-stacked graph of the rate a histogram, and as Prometheus histograms are cumulative you'll get the result you're looking for.