// Evaluate & guard
openai/evals3.0.1
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Open source No commits in 6+ months For developers
IN ONE PARAGRAPH
openai/evals is evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Good for: Developers and teams who want to evaluate & guard.
Not for: teams that need active fixes (no commits in the last 6 months).
01How to start
- 1Read the official quick-start (link above).
- 2Install from the official package registry:
pip install evals - 3Try it on a small example before wiring it into a project.
02Alternatives & comparison
Compare side by side → promptfoo Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic. ★ 26k deepeval The LLM Evaluation Framework ★ 19k ragas Supercharge Your LLM Application Evaluations 🚀 ★ 16k lm-evaluation-harness A framework for few-shot evaluation of language models. ★ 14k opencompass OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety. ★ 7.5k
03Related
Versions, why it is listed, for AI to read
Recent releases
- 3.0.1 2024-05-01
- 3.0.0 2024-04-17
- 2.0.0 2024-01-12
- 1.0.3.post1 2023-04-17
- 1.0.3 2023-04-17
For AI agents
- Interfaces：CLI
- Machine-readable record：/api/v1/software/openai-evals.json · llms.txt
- Added 2026-10-10 · updated 2026-10-22