Skip to content
English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

// Run models locally

TensorRT-LLMv1.4.0rc1

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

Open source Maintained For developers

Official site ↗ GitHub ↗

IN ONE PARAGRAPH

TensorRT-LLM is tensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

Good for: Developers and teams who want to run models locally.

01How to start

  1. 1Read the official quick-start (link above).
  2. 2Install from the official package registry: pip install tensorrt-llm
  3. 3Try it on a small example before wiring it into a project.

PyPI: tensorrt-llm

02Alternatives & comparison

Compare side by side →
vllm A high-throughput and memory-efficient inference and serving engine for LLMs ★ 95k flashinfer FlashInfer: Kernel Library for LLM Serving ★ 6.7k sglang SGLang is a high-performance serving framework for large language models and multimodal models. ★ 38k gpustack A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances. ★ 5.9k LMCache LMCache: Supercharge Your LLM with the Fastest KV Cache Layer ★ 12k

03Related

Versions, why it is listed, for AI to read

Recent releases

For AI agents

Data updated: AI software & hot ranks · Software profiles & needs(UTC+8)