Skip to content
English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

// I want to solve · Developers / Enterprise IT

📈 Evaluate and monitor AI quality

Build regression tests for prompts and agents, record traces, catch silent model degradation, and tell vendor-reported scores from independent ones.

4 steps3 alternatives2 stacks · 2 problemschecked 2026-10-22

How to do it

  1. 01Build a gold set of 30–100 real questions and expected answers
  2. 02Run regressions before prompt or model changes (promptfoo / DeepEval)
  3. 03Record traces, cost and latency in production (Langfuse / Phoenix)
  4. 04Tell vendor-reported scores from independent ones on public leaderboards

What it costs

Free start

Main tools are open source

Team / enterprise

Hosted versions bill by trace volume; see their sites

Pay as you go

LLM-as-judge adds model call costs

Prices change; the vendor’s official page wins. Subscriptions compared · API prices

Alternatives: which to pick

OptionChoose it when
promptfooRegression tests and red-teaming
LangfuseTraces and scores
DeepEvalUnit-test style

✓ Good fit

  • LLM features in or near production

✗ Not a good fit

  • Prototyping

Ready-made stacks

All stacks
LLM app monitoring & evaluationIntermediate Gate quality with regression tests before release, then record traces, cost and latency in production to catch silent degradation. 5 tools Observability & evals Trace agents, cost and failures; run regression evals. 3 tools

Software that helps

All software

🧪 Evaluation, red-teaming & guardrails📈 Observability & tracing

promptfoo Evaluation, red-teaming & guardrails0.124.1 ★ 26k Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic. Open source CLI openai/evals Evaluation, red-teaming & guardrails3.0.1 ★ 20k Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks. Open source CLI deepeval Evaluation, red-teaming & guardrailsv4.1.5 ★ 19k The LLM Evaluation Framework Open source SDK ragas Evaluation, red-teaming & guardrailsv0.4.3 ★ 16k Supercharge Your LLM Application Evaluations 🚀 Open source SDK lm-evaluation-harness Evaluation, red-teaming & guardrailsv0.4.13 ★ 14k A framework for few-shot evaluation of language models. Open source CLI opencompass Evaluation, red-teaming & guardrails0.5.4 ★ 7.5k OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety. Open source CLI SWE-bench Evaluation, red-teaming & guardrailsv5.0.1 ★ 6.1k SWE-bench: Can Language Models Resolve Real-world Github Issues? Open source CLI giskard-oss Evaluation, red-teaming & guardrailsv1.0.6 ★ 5.9k 🐢 Open-Source Evaluation & Testing library for LLM Agents Open source SDK agenta Evaluation, red-teaming & guardrailsv0.122.10 ★ 4.8k Agenta is a workspace where you and your team build agents and automations. Open source API trulens Evaluation, red-teaming & guardrailstrulens-2.15.0 ★ 3.6k Evaluation and Tracking for LLM Experiments and AI Agents Open source SDK OSWorld Evaluation, red-teaming & guardrailsv0.1.16 ★ 3.2k [NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments Open source CLI inspect_ai Evaluation, red-teaming & guardrails0.3.283 ★ 3.1k Inspect: A framework for large language model evaluations Open source CLISDK terminal-bench-1 Evaluation, red-teaming & guardrails ★ 2.6k A benchmark for LLMs on complicated tasks in the terminal Open source CLI tau2-bench Evaluation, red-teaming & guardrailsv1.0.1 ★ 2.2k τ-Bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains Open source CLI BrowserGym Evaluation, red-teaming & guardrailsv0.14.3 ★ 1.4k 🌎💪 BrowserGym, a Gym environment for web task automation Open source SDK autoevals Evaluation, red-teaming & guardrailspy-0.4.0 ★ 1.0k AutoEvals is a tool for quickly and easily evaluating AI model outputs using best practices. Open source SDK LangSmith Evaluation, red-teaming & guardrails Closed / commercial Braintrust Evaluation, red-teaming & guardrails Closed / commercial FastChat Evaluation, red-teaming & guardrailsv0.2.36 ★ 40k An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena. Open source CLIAPI iFixAi Evaluation, red-teaming & guardrailsv4.2.1 ★ 32k Independent auditing of AI agents: what they did and whether it was safe. Open source CLI gorilla Evaluation, red-teaming & guardrailsv1.3 ★ 13k Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls) Open source CLI e2e Evaluation, red-teaming & guardrails ★ 11k Next-generation e2e testing framework for web and mobile. Open source SDK simple-evals Evaluation, red-teaming & guardrails ★ 4.7k Open source CLI lmms-eval Evaluation, red-teaming & guardrailsv0.7.3 ★ 4.5k One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks Open source CLI

49 more by category →

Problems & solutions

All problems

Needs in this scenario

  • DEV-04Evaluation, regression and silent model degradation

    Developers · People search for: llm eval framework; model quality degradation; promptfoo alternative

  • DEV-12Observability: multi-agent runs and MCP servers without logs

    Developers · People search for: llm observability open source; otel genai; mcp server logs

  • DEV-02AI-generated code is hard to verify and review

    Developers · People search for: ai code review tool; verify ai generated code

  • ENT-01Choosing the control plane: gateways, observability, security, identity

    Enterprise IT · People search for: enterprise llm gateway comparison; ai observability platform

Learn

All tutorials

Data updated: Solutions & stack guides(UTC+8)