Evals (évaluation d'un système LLM)
The test set that measures whether your model-based system does what it should, on cases you picked in advance. Without evals, you tweak a prompt, look at three answers, decide it feels better, and have no idea what you just broke elsewhere. With evals, a change becomes a number going up or down, and a regression shows up before it ships.
Strengths
- Two pairs to tell apart. Offline evals: prepared questions or tasks, often with the expected answer, run before deployment. Online evals: real interactions in production, which catch what your test set missed. On feedback, explicit is what the user tells you (thumbs up or down), implicit is what you infer from their actions (they click a link in the answer) ([Instagram reel DezVgr8gjd1](https://www.instagram.com/reel/DezVgr8gjd1/))
- Turns a prompt change into something measurable instead of a feeling
- Catches silent regressions, the ones that break nothing but degrade the answer
Limitations
- Model-judged metrics burn API calls: a large suite costs on every run
- Easy to measure what is easy to measure rather than what matters to the user
Best for
- Proving with numbers that a prompt change actually improved the system