promptfoo
A command-line tool, MIT licensed, for testing prompts, agents and RAG systems. Everything is declared in a `promptfooconfig.yaml` file: the prompts, the model providers to compare (OpenAI, Anthropic, Bedrock, Ollama and others), the variables of each case and the expected assertions, deterministic (`contains`, `is-json`, JavaScript code) or scored by a judge model (`llm-rubric`). `promptfoo view` opens a report where every failed output can be read. It also runs attack tests against your application (prompt injection, data leaks). OpenAI announced its acquisition on March 9, 2026; the project stays open source under MIT and keeps supporting other providers, according to the announcement and the README.
Best for
- Failing a PR when a prompt change lowers the score on your reference cases
- Comparing two models or two prompt versions on exactly the same cases
- Scoring subjective outputs (tone, polite refusal, a summary's faithfulness) with a judge model declared in YAML