Skip to content

Prometheus recording rules

Monitoring · Prometheus
Studied

Problem

Grafana dashboards take 8–15 seconds to open. During an incident the team waits for graphs to load instead of reading them. The cause: heavy histogram_quantile + rate() + sum by expressions are recomputed on every refresh, for every user.

Solution

From the Prometheus docs: "Recording rules allow you to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series. Querying the precomputed result will then often be much faster than executing the original expression every time it is needed. This is especially useful for dashboards." Move heavy PromQL into recording rules — the dashboard reads a ready series: milliseconds instead of seconds.

Example

prometheus/rules/http.yaml:

groups:
  - name: http-golden-signals
    interval: 30s
    rules:
      - record: job:http_request_duration_seconds:p95_5m
        expr: |
          histogram_quantile(0.95,
            sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))

      - record: job:http_requests:rate_5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_errors:ratio_5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
            /
          sum by (job) (rate(http_requests_total[5m]))

Wire it up in prometheus.yml:

rule_files:
  - /etc/prometheus/rules/*.yaml

In Grafana, a single line now replaces the big expression:

job:http_request_duration_seconds:p95_5m{job="api"}

Naming follows the official convention: level:metric:operations. It's obvious this is a recording rule, not a "raw" metric.

When to use

  • Dashboards opened by more than one person at a time.
  • Alerts on complex PromQL — record it and alert on the ready series instead of recomputing it in the rule.
  • Formulas that appear in 2+ dashboards — define once as a recording rule, reuse everywhere.

Source

How to apply it yourself

  1. Find the 3 slowest dashboards in Grafana and look at their PromQL queries.
  2. Move shared expressions into rules/*.yaml.
  3. Rewrite the dashboard panels to use the ready series with the job:... prefix.
  4. Measure load time before/after — it should be several times faster. Once the difference is felt during an incident — status Studied.