Evals (évaluation d'un système LLM)
The test set that measures whether your model-based system does what it should, on cases you picked in advance. Without evals, you tweak a prompt, look at three answers, decide it feels better, and have no idea what you just broke elsewhere. With evals, a change becomes a number going up or down, and a regression shows up before it ships.
Strengths
- Turns a prompt change into something measurable instead of a feeling
- Catches silent regressions, the ones that break nothing but degrade the answer
Limitations
- Model-judged metrics burn API calls: a large suite costs on every run
- Easy to measure what is easy to measure rather than what matters to the user
Best for
- Proving with numbers that a prompt change actually improved the system