// Evaluate & guard
lmms-evalv0.7.3
One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
Open source Maintained For developers
IN ONE PARAGRAPH
lmms-eval is one-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks.
Good for: Developers and teams who want to evaluate & guard.
01How to start
- 1Read the official quick-start (link above).
- 2Install from the official package registry:
pip install lmms_eval - 3Try it on a small example before wiring it into a project.
02Alternatives & comparison
Compare side by side → OSWorld [NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments ★ 3.2k promptfoo Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic. ★ 26k opencompass OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety. ★ 7.5k VLMEvalKit Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks ★ 4.5k deepeval The LLM Evaluation Framework ★ 19k