// How-to · Evaluate and monitor AI quality
Catch silent quality drops after model or prompt changes
Build a fixed regression suite for prompts and agents and run it before and after every change or model switch, like unit tests.
verified 2026-10-101 official sources
Scenario: Evaluate and monitor AI quality →
Symptoms
- After a model or prompt change users say it got worse but nobody can say how
- You suspect the provider’s model got “dumber” but have no evidence
Common causes
- There is no fixed test set; judgement is based on feel
How to fix
- Collect 50–200 representative real inputs and write expected outputs or grading criteria
- Define tests with promptfoo, DeepEval or Inspect: exact match, required keywords, JSON schema checks, LLM-as-judge and so on
- Run evals in CI and compare pass rates whenever prompts or models change
- Run the same suite against production models regularly (e.g. weekly), keep history and use the data to judge degradation
- Public benchmarks are a reference only; results on your own tasks decide
Official references
Compiled from official documentation and public discussions on 2026-10-10; not yet reproduced in our own environment. Follow the official docs if they differ.