// I want to solve · Individuals / Developers / Researchers
🖥️ Run models on your own machine
Estimate what your VRAM or RAM can run, pick an inference runtime, and configure context length and chat templates.
5 steps4 alternatives5 stacks · 5 problemschecked 2026-10-22
How to do it
- 01Use the VRAM calculator to see model size and quantisation you can run
- 02Beginners install LM Studio or Ollama and start with 4–8B models
- 03Add Open WebUI for a web interface
- 04Raise the context-length setting if conversations get truncated
- 05Move to vLLM / SGLang for multi-user or production
What it costs
Free start
All software is free
Hardware
GPU VRAM or Apple Silicon memory decides model size; see self-hosted compute
Team / enterprise
Multi-GPU servers: see compute build guides
Prices change; the vendor’s official page wins. Subscriptions compared · API prices
Alternatives: which to pick
✓ Good fit
- Privacy, offline, unlimited free use
✗ Not a good fit
- You need frontier capability
Ready-made stacks
All stacks Run LLMs on your own computerBeginner From one-click installs to command-line tuning: download open-weight models, run them locally and expose an OpenAI-compatible API to other tools. 5 tools Local private LLM Keep data on-device while running chat, APIs and simple agents. 4 tools Local private knowledge baseBeginner Documents never leave your machine: a local model answers and a local vector store retrieves. Good for individuals and small teams turning PDFs and notes into a Q&A knowledge base. 4 tools Local voice assistantIntermediate Speech recognition, a local LLM and speech synthesis all running on your machine, without cloud APIs. 4 tools Fine-tune open-weight modelsAdvanced Prepare data, fine-tune on modest GPUs, track experiments, then serve the result as an API. 6 tools
Software that helps
All software Ollama Local inference & servingv0.40.8 ★ 184k Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models. Open source CLIAPI llama.cpp Local inference & servingb11804 ★ 132k LLM inference in C/C++ Open source vllm Local inference & servingv0.31.0 ★ 95k A high-throughput and memory-efficient inference and serving engine for LLMs Open source APICLI cherry-studio Local inference & servingv2.1.4 ★ 53k AI productivity studio with smart chat, autonomous agents, and 300+ assistants. Unified access to frontier LLMs Open source App LocalAI Local inference & servingv4.11.0 ★ 50k LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required. Open source API exo Local inference & servingv1.0.71 ★ 48k Run frontier AI locally. Open source CLIAPI sglang Local inference & servingv0.5.21 ★ 38k SGLang is a high-performance serving framework for large language models and multimodal models. Open source API mlx Local inference & servingv0.32.3 ★ 29k MLX: An array framework for Apple silicon Open source SDK llamafile Local inference & serving0.10.6 ★ 26k Distribute and run LLMs with a single file. Open source CLIAPI ktransformers Local inference & serving ★ 20k A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations Open source TensorRT-LLM Local inference & servingv1.4.0rc1 ★ 15k TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way. Open source SDKAPI ai-dynamo/dynamo Local inference & servingv1.5.0-kimi-k3-post.1 ★ 8.4k A Datacenter Scale Distributed Inference Serving Framework Open source API lmdeploy Local inference & servingv0.18.0 ★ 8.1k LMDeploy is a toolkit for compressing, deploying, and serving LLMs. Open source APICLI mlx-lm Local inference & serving ★ 7.4k Run LLMs with MLX Open source llama-swap Local inference & servingv262 ★ 5.9k Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc Open source API gpustack Local inference & servingv2.2.3 ★ 5.9k A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances. Open source API lms Local inference & serving ★ 5.4k LM Studio CLI Open source CLI llm-d Local inference & servingv0.10 ★ 4.9k Achieve state of the art inference performance with modern accelerators on Kubernetes Open source API ramalama Local inference & servingv0.25.0 ★ 3.1k RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers. Open source CLI LM Studio Local inference & serving0.4.26（Homebrew 当前） Desktop app to download and run local LLMs with an OpenAI-compatible server. Closed / commercial Free macOSWindowsLinux NVIDIA driver Local inference & serving615.78.08 Open source CUDA Toolkit / cuDNN Local inference & servingCUDA 13.4.2 Closed / commercial stable-diffusion-webui Local inference & servingv1.10.1 ★ 165k Stable Diffusion web UI Open source API textgen Local inference & servingv4.9 ★ 48k Open-source desktop app for local LLMs. Text, vision, tool-calling, OpenAI/Anthropic-compatible API. 100% private. Open source API
Problems & solutions
All problems- ChoiceWhat model size can my VRAM / RAM run?Rule of thumb: weight memory ≈ parameters × bits per parameter ÷ 8, plus context (KV cache) and runtime overhead. At 4-bit quantisation that is roughly 0.5–0.7 GB per billion parameters.
- ErrorOllama truncates long documents and the model “forgets” earlier textOllama’s default context length is much smaller than many models support (the default changes between versions). When input exceeds it, earlier content is dropped.
- Errorllama.cpp prints garbage, mixes up roles or never calls toolsThe chat template turns messages into the format the model was trained on. With the wrong template the model emits odd tokens, talks to itself or fails to call tools properly.
- ErrorvLLM fails at start-up with CUDA out of memoryAt start-up vLLM pre-allocates a share of GPU memory and reserves KV cache for the maximum context length. If the model plus context does not fit, it fails immediately.
- How-toBuild a private ChatGPT on your own computer (Ollama + Open WebUI)Run open-weight models locally with Ollama and get a ChatGPT-like web interface with Open WebUI; chats and files never leave your machine.
Needs in this scenario
-
DEV-08Running locally: matching VRAM/hardware and configuring parameters
Developers · People search for: what llm can i run on 24gb vram; ollama best settings; llama.cpp chat template
-
IND-03What can my hardware run?
Individual developers & researchers · People search for: what model can i run on 8gb vram; local llm hardware calculator
-
IND-05Comparing low-cost GPU rental
Individual developers & researchers · People search for: cheapest gpu cloud; runpod vs vast
Learn
All tutorials- CMU 10-414/714 Deep Learning Systems
CMU · Compute & systems · English
- Hugging Face TRL 文档
Hugging Face · Fine-tuning & training · English
- Hugging Face PEFT 文档
Hugging Face · Fine-tuning & training · English
- Unsloth 文档
Unsloth · Fine-tuning & training · English
- Finetuning Large Language Models
DeepLearning.AI · Fine-tuning & training · English
- Hugging Face smol-course
Hugging Face · Fine-tuning & training · English
- OpenAI Spinning Up in Deep RL
OpenAI · Fine-tuning & training · English
- Illustrating RLHF
Hugging Face · Fine-tuning & training · English
- RLHF Book
Nathan Lambert · Fine-tuning & training · English
- OpenAI Fine-tuning 指南
OpenAI · Fine-tuning & training · English
Official references
Compiled from official documentation on 2026-10-22; not reproduced in our own environment. Follow the official docs if they differ.