Skip to content
English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

Step 4 · Software stack

Software stack, bottom-up

Install in this order. Most “it doesn’t work” problems are a mismatch between two adjacent layers.

1. Operating system

LinuxUbuntu LTS for servers; DGX systems ship DGX OS 7 (Ubuntu 24.04-based). DGX Spark runs DGX OS.docs.nvidia.comnvidia.com
Windows / macOSPersonal tier only: Windows + WSL2 for CUDA tools, macOS for MLX / llama.cpp.github.comgithub.com

2. Driver & fabric

NVIDIA driverUse the data-centre driver packages for servers. Blackwell needs recent driver and CUDA releases – check the CUDA release notes for the minimum (version to be verified for your OS).docs.nvidia.comdocs.nvidia.com
Fabric ManagerRequired on NVSwitch systems (HGX / DGX 8-GPU); without it NVLink between GPUs does not come up.docs.nvidia.com
AMD ROCmDriver + runtime stack for Instinct MI300X / MI355X.rocm.docs.amd.com

3. CUDA & containers

CUDA ToolkitInstall per the Linux guide; pin one version per cluster.docs.nvidia.com
Container ToolkitLets Docker / containerd see GPUs – the easiest way to run vLLM / SGLang images.docs.nvidia.com
KubernetesGPU Operator installs driver, toolkit, device plugin and DCGM exporter on every node.docs.nvidia.comkubernetes.io

4. Inference engines

llama.cppGGUF quantisation, CPU + GPU offload, Apple Silicon; the base of many desktop apps.github.com
OllamaEasiest local runner; binds 127.0.0.1:11434 by default.docs.ollama.comdocs.ollama.com
MLX / mlx-lmApple’s framework for Apple Silicon.github.com
vLLMProduction serving with an OpenAI-compatible API; pre-allocates GPU memory (gpu-memory-utilization).docs.vllm.aidocs.vllm.ai
SGLangHigh-throughput serving runtime with an OpenAI-compatible API.docs.sglang.ai
TensorRT-LLMNVIDIA’s own optimised inference library.github.com
DynamoDistributed serving across nodes (disaggregated prefill / decode); used in NVIDIA’s GB300 benchmarks.github.comnvidia.com

5. Training & fine-tuning

PyTorch FSDP / DeepSpeedShard weights, gradients and optimizer states across GPUs (ZeRO).docs.pytorch.orgarxiv.org
Fine-tuning stacksUnsloth, LLaMA-Factory, TRL – see our stack page.

6. Scheduling & management

SlurmBatch scheduler used in SuperPOD-style training clusters.slurm.schedmd.comdocs.nvidia.com
Mission Control / Base CommandNVIDIA’s management software for DGX / NVL72 clusters.nvidia.comnvidia.com

Local inference tools in our catalogue → · Stacks: local private LLM, fine-tuning → · Learn: serving engines → · Learn: model gateways →

Data updated: Compute hardware