Skip to content

LiteLLM: a single LLM gateway

MLOps / LLMOps · Gateway
Not studied

Added 2026-10-29.

Problem

Cloud model keys are scattered across services, spending isn't visible, and switching providers means code changes in every application.

Solution

LiteLLM AI Gateway is an "open source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format." Applications talk to one address, providers and local models are configured in one config, and virtual keys get budgets and limits.

Example

config.yaml:

model_list:
  - model_name: local-llm
    litellm_params:
      model: ollama/llama3
      api_base: http://127.0.0.1:11434

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

litellm_settings:
  callbacks: ["prometheus"]

Before running, verify the image signature — the project publishes one for every release:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/e82a3c646908ba0a6ee0beddc4310d7a6cac4065/cosign.pub \
  ghcr.io/berriai/litellm:<release-tag>

Virtual keys with a budget (max_budget) are issued via /key/generate and require a database (DATABASE_URL).

Versions 1.82.7 and 1.82.8

On March 24, 2026 malicious litellm 1.82.7 and 1.82.8 were published to PyPI: the publishing token was stolen through an unpinned Trivy in the project's CI. Pin the version and the image digest, never use latest. If these versions were installed, treat the machine as compromised and rotate the secrets reachable from it.

When to use

When two or more applications use LLMs, or spending needs to be tracked per team.

Source

How to apply it yourself

  1. Run the proxy in Docker with a pinned digest and a verified signature.
  2. Connect one cloud model and one local model.
  3. Issue a virtual key with a budget and check that the limit triggers.
  4. Mark it "Studied" once metrics show up in Prometheus and applications go only through the gateway.