Step 2 · Tier & memory
Pick a tier, then size the memory
Two questions decide the tier: what you do (inference or training) and for how many people. Memory then decides which machine in that tier – and memory bandwidth decides how fast it generates.
By use and scale
| Use | Workload | Tier | What matters |
|---|---|---|---|
| Chat / coding assistant for yourself | Inference, 1 user | Personal | Fit weights + context; bandwidth decides speed |
| Internal tool for a small team | Inference, 2–20 concurrent | StudioTeam | KV cache × concurrency becomes the main cost |
| Production API / customer-facing | Inference, 100+ concurrent | TeamEnterprise | Throughput per $ and per watt; redundancy (≥2 nodes) |
| LoRA / QLoRA fine-tuning | Training (adapters) | PersonalStudio | QLoRA fine-tuned a 65B model on one 48GB GPUarxiv.org |
| Full fine-tuning | Training (all weights) | TeamEnterprise | Mixed-precision Adam ≈ 16 bytes per parameter before activations (ZeRO paper) → 70B ≈ 1.1TBarxiv.org |
| Pre-training / large RL | Training, multi-node | Cluster | Fabric (800G per GPU), storage and reliability dominate |
Inference memory
Weights = parameters × bits per weight ÷ 8. BF16 = 2 bytes, FP8 = 1 byte, 4-bit ≈ 0.5–0.6 bytes (Q4_K_M ≈ 4.85 bits).github.com
KV cache = 2 × layers × KV heads × head dim × context × concurrent requests × bytes. It grows linearly with context and users – on a busy server it can exceed the weights.
MoE: all experts must be in memory, so size by total parameters; only speed scales with active parameters.
Speed: single-user generation is roughly memory bandwidth ÷ bytes read per token, which is why a 273 GB/s DGX Spark is slower than a 1.8 TB/s RTX 5090 on models that fit both.nvidia.comnvidia.com
Training memory
Full fine-tuning with mixed-precision Adam needs ≈ 16 bytes per parameter for weights, gradients and optimizer states, before activations (ZeRO paper). Sharding (ZeRO / FSDP) spreads that across GPUs; activations add more and depend on batch and sequence length.arxiv.org
LoRA trains small adapters on a frozen model; QLoRA keeps the frozen model in 4-bit – the paper fine-tuned a 65B model on a single 48GB GPU.arxiv.org
Rule: inference needs ≈ weights + KV; full training needs ≈ 8× the BF16 weight size plus activations.
Quick reference: weights + 10% overhead
| Parameters | BF16 | FP8 | 4-bit (Q4_K_M) |
|---|---|---|---|
| 8B | 18 GB | 10 GB | 6 GB |
| 32B | 71 GB | 38 GB | 22 GB |
| 70B | 155 GB | 82 GB | 47 GB |
| 235B | 518 GB | 275 GB | 157 GB |
| 405B | 892 GB | 474 GB | 271 GB |
| 671B | 1.48 TB | 785 GB | 448 GB |
| 1T | 2.20 TB | 1.17 TB | 667 GB |
Formula estimate, no KV cache – add it with the calculator below.
VRAM calculator
Architecture details (for KV cache)
KV cache = 2 × layers × KV heads × head dim × context × concurrency × bytes. MLA models (DeepSeek-V3, Kimi K2) cache a 576-dim latent per layer; we enter it as 1 head × 288.
GPUs and servers (dedicated memory)
- 8GB GPU (e.g. RTX 4060)
- 12GB GPU (e.g. RTX 3060 12G)
- 16GB GPU (e.g. RTX 5060 Ti 16G)
- GeForce RTX 4090 · 24GB
- GeForce RTX 5090 · 32GB
- 2 × RTX 5090 · 64GB
- H100 SXM · 80GB
- RTX PRO 6000 Blackwell · 96GB
- DGX Spark · 128GB
- H200 SXM · 141GB
- B200 SXM · 180GB
- Instinct MI300X · 192GB
- B300 SXM · 288GB
- Instinct MI355X · 288GB
- 4 × RTX PRO 6000 · 384GB
- 8 × H100 · 640GB
- 8 × H200 · 1,128GB
- 8 × B200 · 1,440GB
- 8 × Instinct MI300X · 1,536GB
- 8 × B300 · 2,304GB
- 8 × Instinct MI355X · 2,304GB
Mac / DGX Spark (unified memory)
macOS lets the GPU use roughly two-thirds of unified memory on 36GB-and-smaller machines and about three-quarters on larger ones; we use those ratios. For DGX Spark we assume ~90% of 128GB is usable (to be verified on your OS image).
Assumptions: overhead = 10% of weights + 0.5GB per process. Multi-GPU rows assume tensor/pipeline parallelism splits weights and KV evenly and ignore per-GPU duplicated buffers. Green = fits, amber = ≤ 10% to spare, grey = does not fit. Real usage varies by engine (vLLM, SGLang, llama.cpp, Ollama, MLX) – vLLM, for example, pre-allocates most free memory for KV cache.
VRAM table for every open-weight model → · Open-weight models and their sizes → · Local inference tools →