vLLM: serving LLMs on your own hardware

MLOps / LLMOps · Inference

Not studied

Added 2026-10-29.

Problem

Data can't be sent to an external cloud — an air-gapped network, security requirements. You need your own model with decent throughput and a familiar API.

Solution

vLLM — "easy, fast, and cheap LLM serving for everyone" — "is a fast and easy-to-use library for LLM inference and serving". It gets its speed from PagedAttention (efficient management of attention key and value memory) and continuous batching of incoming requests, and exposes OpenAI- and Anthropic-compatible APIs. It supports GPUs from different vendors as well as CPUs. It plugs into LiteLLM as a regular provider.

Example

docker run --rm --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v vllm-cache:/root/.cache/vllm \
  -p 127.0.0.1:8000:8000 \
  vllm/vllm-openai:v0.32.0 \
  meta-llama/Llama-3.1-8B-Instruct

The port is published on 127.0.0.1 only; external access goes through a gateway with authentication. The vllm-cache volume keeps compiled artifacts between restarts and speeds up startup.

When to use

When you have a GPU and a requirement to keep data in-house. CPU mode is fine for learning, not for load.

Limitations

A fast release cycle — pin the image version. Check model weight licenses separately.

Source

• vllm-project/vllm
• Deploying with Docker

How to apply it yourself

1. Run a small model on a rented GPU.
2. Collect time to first token and throughput in Grafana.
3. Connect it to LiteLLM as a provider.
4. Mark it "Studied" once an application works through vLLM instead of a cloud model.