HolmesGPT: AI analysis of Prometheus alerts

AI tools · Operations

Not studied

Added 2026-10-29.

Problem

An alert fires at night. The on-call engineer spends the first 20 minutes on the same queries every time: metrics, logs, recent changes.

Solution

HolmesGPT is an "open-source AI agent for investigating production incidents and finding root causes. Works with any stack — Kubernetes, VMs, cloud providers, databases, and SaaS platforms", and a CNCF sandbox project. It gathers data from Prometheus, Grafana, Loki, Kubernetes and other sources on its own and proposes a hypothesis for the cause. By design it is read-only and respects RBAC. Route the model through LiteLLM so metrics and logs don't leave for an external cloud.

Example

~/.holmes/config.yaml — the metrics source:

toolsets:
  prometheus/metrics:
    enabled: true
    config:
      prometheus_url: http://127.0.0.1:9090

The model through an OpenAI-compatible gateway:

export OPENAI_API_BASE=http://127.0.0.1:4000/v1
export OPENAI_API_KEY=<LiteLLM virtual key>

holmes ask "why is latency growing for devops-kb?" --model="openai/local-llm"
holmes investigate alertmanager --alertmanager-url http://127.0.0.1:9093 \
  --alertmanager-label "severity=critical"

Limitations

Quality depends on the model and the available data. The output is a hypothesis for the on-call engineer, not a diagnosis. If you only need Kubernetes, there's a simpler alternative — k8sgpt.

Source

• robusta-dev/holmesgpt — Apache-2.0
• The Prometheus toolset
• OpenAI-compatible models
• Investigating Alertmanager alerts

How to apply it yourself

1. On a test environment, connect Prometheus and a model via LiteLLM.
2. Run three historical alerts through it.
3. Compare the conclusions with the real causes from the postmortems.
4. Mark it "Studied" if the hypothesis matched in at least two cases out of three.