DeepEval
Open source LLM evaluation framework (Apache 2.0) that works like Pytest but for language-model systems. You write test files, run `deepeval test run`, and get metrics: G-Eval (customizable LLM-as-a-judge), hallucination, answer relevancy, RAG metrics (faithfulness, contextual recall and precision) and agentic metrics (task completion, tool correctness). Most run locally on your machine. The framework is free; Confident AI is the optional paid cloud platform on top (datasets, tracing, prod monitoring).
Strengths
- Pytest logic: any dev already knows how to write and run the tests, zero new concept
- Rich metrics (G-Eval, hallucination, RAG, agentic) running locally, no paid platform needed
- Plugs into OpenAI, LangChain, CrewAI and most LLM frameworks
Limitations
- LLM-as-a-judge metrics burn API calls: a large suite costs tokens on every run
- Easy to over-measure: stacking decorative metrics instead of defending the real claim
Best for
- Proving with metrics that an LLM replication holds the paper's claim
- Adding eval tests to a RAG, an agent or a chatbot before showing it
- Catching regressions in an LLM system the way you catch a broken test