// Local inference & serving
Run models locally
Run models on your machine or servers: Ollama, llama.cpp, vLLM and more.
83 tools81 open sourcechecked 2026-10-22Compare side by side →
How to choose · start with these three
- · Getting started: Ollama, LM Studio; high-throughput serving: vLLM, SGLang
- · Check the VRAM calculator first
83 / 83 items
Ollama v0.40.8 ★ 184k Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models. Open source CLIAPI llama.cpp b11804 ★ 132k LLM inference in C/C++ Open source vllm v0.31.0 ★ 95k A high-throughput and memory-efficient inference and serving engine for LLMs Open source APICLI airllm ★ 40k Run 70B inference on a single 4GB GPU. Open source Python colibri ★ 43k Pure-C runtime to run frontier MoE models on hardware you already own. Open source local LM Studio 0.4.26（Homebrew 当前） Desktop app to download and run local LLMs with an OpenAI-compatible server. Closed / commercial Free macOSWindowsLinux ODS ★ 7.6k Turn your PC into an AI server: LLM inference, chat UI, voice and agents. Open source local llmfit ★ 38k One command to find which models and providers fit your hardware. Open source CLI tilelang ★ 8.7k DSL to streamline high-performance GPU/CPU kernel development. Open source DSL ds4 ★ 24k Local inference engine for DeepSeek 4 Flash and PRO. Open source local omlx ★ 23k LLM inference server with continuous batching for Apple Silicon. Open source macOS needle ★ 14k Automation foundation model for tiny devices. Open source embedded OpenLLM ★ 13k Run open-source LLMs as OpenAI-compatible API endpoints. Open source server osaurus ★ 8.1k Native macOS harness for AI agents — any model, memory, autonomy. Open source macOS vllm-omni v0.30.0 ★ 7.3k Framework for efficient inference with omni-modality models. Open source server magnitude ★ 7.2k Open inference server that runs the best local models for your hardware. Open source server whichllm ★ 6.8k Find the local LLM that actually runs and performs on your hardware. Open source CLI cactus ★ 6.1k Quantization, kernels and inference for mobiles and wearables. Open source mobile LightLLM ★ 4.3k Python-based LLM inference and serving framework. Open source server sie ★ 3.4k Open-source inference server for all the models your agent needs. Open source server stable-diffusion-webui v1.10.1 ★ 165k Stable Diffusion web UI Open source API cherry-studio v2.1.4 ★ 53k AI productivity studio with smart chat, autonomous agents, and 300+ assistants. Unified access to frontier LLMs Open source App LocalAI v4.11.0 ★ 50k LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required. Open source API exo v1.0.71 ★ 48k Run frontier AI locally. Open source CLIAPI textgen v4.9 ★ 48k Open-source desktop app for local LLMs. Text, vision, tool-calling, OpenAI/Anthropic-compatible API. 100% private. Open source API BitNet ★ 41k Official inference framework for 1-bit LLMs Open source CLI sglang v0.5.21 ★ 38k SGLang is a high-performance serving framework for large language models and multimodal models. Open source API mediapipe v1.1.0 ★ 37k Cross-platform, customizable ML solutions for live and streaming media. Open source SDK diffusers v0.41.0 ★ 35k 🤗 Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch. Open source SDK tinygrad v0.14.0 ★ 34k You like pytorch? You like micrograd? You love tinygrad! ❤️ Open source SDK mlx v0.32.3 ★ 29k MLX: An array framework for Apple silicon Open source SDK llamafile 0.10.6 ★ 26k Distribute and run LLMs with a single file. Open source CLIAPI flash-attention v2.8.3.post2 ★ 25k Fast and memory-efficient exact attention Open source SDK ncnn 20260526 ★ 24k ncnn is a high-performance neural network inference framework optimized for the mobile platform Open source SDK mlc-llm v0.20.0 ★ 23k Universal LLM Deployment Engine with ML Compilation Open source APISDK onnxruntime v1.31.0 ★ 22k ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator Open source SDK candle 0.11.0 ★ 21k Minimalist ML framework for Rust Open source SDK ktransformers ★ 20k A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations Open source web-llm v0.2.86 ★ 19k High-performance In-browser LLM Inference Engine Open source SDK DeepInfer ★ 18k The open-source release of the inference framework behind DeepSeek’s production services (repo created 2026-10-16, ~18k stars in a week): currently DeepSeek-V4.1-Flash only, offering offline inference with the core features for serving at scale — FlashMLA / DeepGEMM / DeepEP kernels, attention DP + MoE EP/EPLB, draft-token decoding (DSpark), async scheduling, batch-invariant and deterministic sampling, structured outputs. Runs on NVIDIA SM100 (Blackwell) and Huawei Ascend 950 only; the KV-cache system, prefill/decode disaggregation and load balancing needed for a full online service are not included Open source transformers.js 4.3.1 ★ 16k State-of-the-art Machine Learning for the web. Run 🤗 Transformers directly in your browser, with no need for a server! Open source SDK MNN 3.6.1 ★ 16k MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI. Open source SDK TensorRT-LLM v1.4.0rc1 ★ 15k TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way. Open source SDKAPI tvm v0.27.0.post1 ★ 14k Open Machine Learning Compiler Framework Open source SDK TensorRT v11.4 ★ 13k NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT. Open source SDK FlashMLA ★ 13k FlashMLA: Efficient Multi-head Latent Attention Kernels Open source SDK LMCache v0.5.6 ★ 12k LMCache: Supercharge Your LLM with the Fastest KV Cache Layer Open source SDK koboldcpp v1.122.1 ★ 12k KoboldCpp - Run GGUF AI models locally with a KoboldAI UI. One File. Zero Install. Open source API server v2.73.0 ★ 11k The Triton Inference Server provides an optimized cloud and edge inferencing solution. Open source API openvino 2026.4.1 ★ 11k OpenVINO™ is an open source toolkit for optimizing and deploying AI inference Open source SDK llama-cpp-python v0.3.36-hip-radeon ★ 11k Python bindings for llama.cpp Open source SDKAPI 3FS ★ 10k A high-performance distributed file system designed to address the challenges of AI training and inference workloads. Open source API DeepEP v1.2.1 ★ 10k DeepEP: an efficient expert-parallel communication library Open source SDK inference v3.5.0 ★ 9.6k Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API. Open source API mlx-examples ★ 9.0k Examples in the MLX framework Open source CLI BentoML v1.4.39 ★ 8.9k The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more! Open source SDKAPI ai-dynamo/dynamo v1.5.0-kimi-k3-post.1 ★ 8.4k A Datacenter Scale Distributed Inference Serving Framework Open source API lmdeploy v0.18.0 ★ 8.1k LMDeploy is a toolkit for compressing, deploying, and serving LLMs. Open source APICLI stable-diffusion.cpp master-987-a77d362 ★ 7.7k Diffusion model(SD,Flux,Wan,Qwen Image,Z-Image,...) inference in pure C/C++ Open source CLI mlx-lm ★ 7.4k Run LLMs with MLX Open source flashinfer v0.7.2 ★ 6.7k FlashInfer: Kernel Library for LLM Serving Open source SDK kserve v0.21.0 ★ 6.1k Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes Open source API llama-swap v262 ★ 5.9k Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc Open source API gpustack v2.2.3 ★ 5.9k A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances. Open source API mlx-vlm v0.7.10 ★ 5.6k MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX. Open source CLISDK lms ★ 5.4k LM Studio CLI Open source CLI aibrix v0.7.0 ★ 5.2k Cost-efficient and pluggable Infrastructure components for GenAI inference Open source API executorch 23323 ★ 5.1k On-device AI across mobile, embedded and edge for PyTorch Open source SDK text-embeddings-inference v1.9.4 ★ 5.1k A blazing fast inference solution for text embeddings models Open source API llm-d v0.10 ★ 4.9k Achieve state of the art inference performance with modern accelerators on Kubernetes Open source API exllamav2 v0.3.2 ★ 4.6k A fast inference library for running LLMs locally on modern consumer-class GPUs Open source SDK llm-compressor 0.10.0.4 ★ 3.9k State-of-the-art LLM compression, built for production inference with vLLM Open source SDK LiteRT v2.3.0 ★ 3.5k LiteRT, successor to TensorFlow Lite. is Google's On-device framework for high-performance ML & GenAI deployment on edge platforms, via efficient conversion, runtime, and optimization Open source SDK optimum v2.3.0 ★ 3.5k 🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools Open source SDK ik_llama.cpp t0002 ★ 3.3k llama.cpp fork with additional SOTA quants and improved performance Open source CLI distributed-llama v0.16.5 ★ 3.1k Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference. Open source CLI ramalama v0.25.0 ★ 3.1k RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers. Open source CLI production-stack vllm-stack-0.1.13 ★ 2.7k vLLM’s reference system for K8S-native cluster-wide deployment with community-driven performance optimization Open source Helm exllamav3 v1.6.0 ★ 1.7k An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs Open source SDK mlx-engine ★ 1.2k LM Studio Apple MLX engine Open source SDK CUDA Toolkit / cuDNN CUDA 13.4.2 Closed / commercial NVIDIA driver 615.78.08 Open source text-generation-inference v3.3.7 ★ 11k Large Language Model Text Generation Inference Open source API Archived
About this data
- Default order is “recommended”: editorial picks first, then by popularity; general-purpose developer software and archived projects are listed last.
- Data checked 2026-10-22. Official links only. JSON API