// Error · Run models on your own machine
vLLM fails at start-up with CUDA out of memory
At start-up vLLM pre-allocates a share of GPU memory and reserves KV cache for the maximum context length. If the model plus context does not fit, it fails immediately.
verified 2026-10-101 official sources
Scenario: Run models on your own machine →
🖥️Self-hosted computeSelf-hosted compute: decide → size → build → run, with calculators→Symptoms
- The log shows
CUDA out of memoryor says there is not enough KV cache space
Common causes
- Max context length defaults to the model’s maximum, so the KV cache requirement is too large
- Other processes are using GPU memory
- The model is unquantised and the weights alone exceed one GPU
How to fix
- Lower the maximum context with
--max-model-lento what you actually need - Adjust pre-allocation with
--gpu-memory-utilization(lower it if the GPU is shared) - With several GPUs, split the model with
--tensor-parallel-size - Use a quantised model (AWQ, GPTQ, FP8, depending on hardware support)
- Check with
nvidia-smithat no other process holds GPU memory
Official references
Compiled from official documentation and public discussions on 2026-10-10; not yet reproduced in our own environment. Follow the official docs if they differ.