// Stage 6 · Inference, Deployment & Efficiency
Running LLMs Locally
llama.cpp, Ollama, MLX, memory estimates and choosing a model.
about 4 hours3 algorithms0 paperslesson text in progress
01Goals
- Pick a model and quantisation level for your memory
- Run models with Ollama/llama.cpp and plug them into apps
- Know MLX on Apple Silicon
Do these first: Quantization, Pruning & Distillation
02Algorithms to master
Each card has the original paper, prerequisites and resources.
03Courses & videos
- Ollama ↗ToolsEN
- ggml-org/llama.cpp ↗Reference codeEN
- ml-explore.github.io ↗Official docsEN
04Hands-on: project, code & exercises
Free, official or author-published pages. Most run in the browser or on a free Colab GPU.
PROJECT
Run a quantised 7B–14B model on your own computer and connect it to a local document Q&A script.
- 1
- 2
- 3
05Self-test
Answer in your head, then open to check.
Roughly how much memory does a 4-bit 7B model need (excluding KV)?
About 4–5 GB.
What is GGUF?
The model file format used by llama.cpp.
The main advantages of local models?
Privacy, offline use and no per-token fees.
On these routes: Application Developers