// Models · deploy & compute

Self-hosted AI compute

Decide whether to build at all, then size, buy, install and run it — from one RTX 5090 or Mac Studio to HGX B300 clusters. Every spec links to its source.

How much VRAM do I need? ↓ HGX B300 build plan →

6 steps in decision orderdata checked 2026-10-21unconfirmed specs marked “to be verified”

01Should you self-host? Three questions

Is your monthly volume large and steady?

Yes · keep going; run the break-even calculator.

No · use an API (pay per token).

Must data stay inside your own network?

Yes · self-host (or a dedicated private cloud).

No · API or cloud GPUs are fine.

Do you train or fine-tune?

Yes · Short bursts → rent; continuous → buy.

No · Inference only: size for memory and concurrency.

Build vs API vs cloud GPUs: break-even calculator →

02VRAM calculator

Sizing guide →

Pick a model size, precision and context length to see how much memory you need and which hardware fits.

Model (503 open models + custom) Parameters (billions, total for MoE) Weight precision / quantisation

Context (tokens) Concurrent requests

KV cache precision

Architecture details (for KV cache)

Layers KV heads Head dim

KV cache = 2 × layers × KV heads × head dim × context × concurrency × bytes. MLA models (DeepSeek-V3, Kimi K2) cache a 576-dim latent per layer; we enter it as 1 head × 288.

Full table: what fits on my GPU →

—Total estimate (GB)

—Weights

—KV cache

—Overhead

GPUs and servers (dedicated memory)

8GB GPU (e.g. RTX 4060)
12GB GPU (e.g. RTX 3060 12G)
16GB GPU (e.g. RTX 5060 Ti 16G)
GeForce RTX 4090 · 24GB
GeForce RTX 5090 · 32GB
2 × RTX 5090 · 64GB
H100 SXM · 80GB
RTX PRO 6000 Blackwell · 96GB
DGX Spark · 128GB
H200 SXM · 141GB
B200 SXM · 180GB
Instinct MI300X · 192GB
B300 SXM · 288GB
Instinct MI355X · 288GB
4 × RTX PRO 6000 · 384GB
8 × H100 · 640GB
8 × H200 · 1,128GB
8 × B200 · 1,440GB
8 × Instinct MI300X · 1,536GB
8 × B300 · 2,304GB
8 × Instinct MI355X · 2,304GB

Mac / DGX Spark (unified memory)

Mac 16GBMac 24GBMac 32GBMac 36GBMac 48GBMac 64GBMac 96GBMac 128GBMac 256GBMac 512GBDGX Spark 128GB

macOS lets the GPU use roughly two-thirds of unified memory on 36GB-and-smaller machines and about three-quarters on larger ones; we use those ratios. For DGX Spark we assume ~90% of 128GB is usable (to be verified on your OS image).

Assumptions: overhead = 10% of weights + 0.5GB per process. Multi-GPU rows assume tensor/pipeline parallelism splits weights and KV evenly and ignore per-GPU duplicated buffers. Green = fits, amber = ≤ 10% to spare, grey = does not fit. Real usage varies by engine (vLLM, SGLang, llama.cpp, Ollama, MLX) – vLLM, for example, pre-allocates most free memory for KV cache.

Open-model VRAM table: 799 models, 41 familiesQwen, DeepSeek, Llama, Gemma, GLM, Kimi, MiniMax, gpt-oss, Mistral… BF16 / FP8 / 4-bit memory, licence and which GPU or Mac fits.Open →

03Featured: HGX B300

The complete HGX B300 build plan8-GPU server config, OEM models (Supermicro, Dell, HPE, Lenovo), single node to SuperPOD networking, model capacity, pricing and export compliance – all sourced.Open →

04The six steps

STEP 1Build or notCost and break-even vs API and cloud GPUsOpen →STEP 2Tier & memoryPick a tier by use and scale; how to size memoryOpen →STEP 3Build plansConcrete configs from consumer GPUs to data centresOpen →STEP 4Software stackDrivers, CUDA / ROCm, engines, orchestrationOpen →STEP 5Power, cooling, networkFacility: power, cooling, fabric, floor loadingOpen →STEP 6Operations & pitfallsMonitoring, failures, upgrades and common trapsOpen →

05Tiers at a glance

Tier	Who	Typical hardware	Model size
Personal	One developer: learning, local private assistant	One box (a consumer GPU / a Mac / a DGX Spark)	7–32B quantised; big-memory Mac / Spark reach 70–200B quantised (slower)
Studio	2–10 people: internal tools, fine-tuning experiments	Workstation (2–4 × 96GB pro cards) or a 512GB Mac	70B FP8, 100B+ quantised; QLoRA on 70B
Team	Tens to hundreds of concurrent users, production serving	One 8-GPU server (H100 / H200 / MI300X or RTX PRO Server)	70B BF16 at high concurrency; 8 × H200 run a 600B-class MoE in FP8
Enterprise	Company-wide or customer-facing; frontier open models; continuous fine-tuning	1–16 HGX B200 / B300 (or MI355X) nodes + fast fabric	1T-parameter MoE (FP8) fits one node; long-context reasoning
Cluster	Pre-training, large-scale RL, compute operators	Tens to thousands of nodes, or GB200 / GB300 NVL72 racks	Train and serve any size; bottlenecks are power, fabric and operations

06Hardware specs (all sourced)

Hardware	Tier	Memory	Bandwidth	Power	Price	Good for	Sources
GeForce RTX 4090NVIDIA	Personal	24GBGDDR6X	To be verified	450 W	Street price varies; to be verifiedquote / TBV	7–32B quantised models on one card; LoRA on 7–8B	nvidia.com
GeForce RTX 5090NVIDIA	Personal	32GBGDDR7	1.79 TB/s	575 W	Launch MSRP $1,999 (Jan 2025); street prices are often higher	32B-class 4-bit with longer context; a common single-GPU dev box	nvidia.com nvidianews.nvidia.com
Mac Studio (M5 Max)Apple	Personal	128GBunified	614 GB/s	480 W (max continuous, whole system)	US Apple Store standard configs from $2,499 (read 2026-10-21)	Quiet and low-power; single-user 70B-class 4-bit inference; no CUDA training	apple.com apple.com
DGX Spark (GB10)NVIDIAvendor claim	Personal	128GBLPDDR5x	273 GB/s	240 W PSU (GB10 TDP 140 W)	Founders Edition MSRP raised to $4,699 in Feb 2026; later higher listings reported – to be verifiedquote / TBV	Desktop CUDA dev box: vendor says inference up to 200B, fine-tuning up to 70B; 273 GB/s bandwidth means slower generation than a discrete GPU	nvidia.com forums.developer.nvidia.com
Mac Studio (M5 Ultra)Apple	Studio	512GBunified	1.2 TB/s	480 W (max continuous, whole system)	Standard configs up to $6,799; 256/512GB are build-to-order – see Apple	256/512GB unified memory holds several-hundred-B quantised MoE models; limited concurrent throughput	apple.com apple.com
RTX PRO 6000 Blackwell (Workstation)NVIDIA	Studio	96GBGDDR7 ECC	1.79 TB/s	600 W	Quote required (no public price on NVIDIA page)quote / TBV	96GB per card: 70B FP8 or 100B+ 4-bit; 2–4-card workstations for QLoRA / small full fine-tunes	nvidia.com
H100 SXM (HGX H100 ×8)NVIDIA	Team	80GB × 8HBM3	3.35 TB/s	up to 700 W / GPU	Quote required; cloud rental prices on Servicesquote / TBV	8×80GB = 640GB: 70B BF16 serving, 30–70B full fine-tunes; largest installed base	nvidia.com docs.nvidia.com
H200 SXM (HGX H200 ×8)NVIDIA	Team	141GB × 8HBM3e	4.8 TB/s	up to 700 W / GPU	Quote requiredquote / TBV	8 GPUs ≈ 1.1TB: a 600B+ MoE (DeepSeek-V3 class) in FP8 on one node	nvidia.com docs.nvidia.com
Instinct MI300X (×8 OAM)AMD	Team	192GB × 8HBM3	5.3 TB/s	750 W / GPU	Quote requiredquote / TBV	8 GPUs = 1.5TB; ROCm stack; vLLM / SGLang support ROCm – benchmark your own models first	amd.com rocm.docs.amd.com
B200 SXM (HGX B200 ×8)NVIDIA	Enterprise	180GB × 8HBM3e	8 TB/s	To be verified	Quote requiredquote / TBV	8 GPUs = 1.44TB; FP4/FP8 inference and mid-to-large training	nvidia.com docs.nvidia.com
B300 SXM (HGX B300 ×8)NVIDIA	Enterprise	288GB × 8HBM3e	8 TB/s	1,100 W / GPU (Dell XE9780 config)	Quote required (no official public price found)quote / TBV	8 GPUs ≈ 2.3TB: a 1T-parameter MoE in FP8 on one node, long-context reasoning; see the B300 build plan	nvidia.com docs.nvidia.com dell.com
Instinct MI355X (×8 OAM)AMD	Enterprise	288GB × 8HBM3E	8 TB/s	1,400 W TBP / GPU	Quote requiredquote / TBV	288GB per GPU like B300; e.g. HPE XD685 offers both B300 and MI355X	amd.com buy.hpe.com
GB200 NVL72 (rack)NVIDIA	Cluster	13.4TBHBM3E	576 TB/s	To be verified	Quote requiredquote / TBV	72 GPUs + 36 Grace CPUs in one NVLink domain (130 TB/s), liquid-cooled rack	nvidia.com
GB300 NVL72 (rack)NVIDIA	Cluster	20TBHBM3e	576 TB/s	To be verified	Quote requiredquote / TBV	72 Blackwell Ultra GPUs + 36 Grace; 20TB GPU memory, 37TB fast memory; fully liquid-cooled	nvidia.com

Data checked 2026-10-21. Prices change quickly; data-centre systems are sold by quote only.

Where the old pages went

Self-hosting content used to be spread over eight places. The calculator moved here; the other pages stay and link back to this section.