Promptfoo

Active
GitHub TypeScript MIT

Description

Promptfoo is an evaluation and regression testing tool for LLM apps and agents, useful for comparing prompts, tool-call results, and model outputs over time.

Key Features

  • Automated LLM evaluations — Batch test prompts, models, and RAG pipeline output quality
  • Red team security testing — Built-in vulnerability scanning and adversarial testing for LLM security
  • Multi-model comparison — Side-by-side comparison of OpenAI, Anthropic, Azure, Bedrock, Ollama models
  • CI/CD integration — Automate evaluation checks in continuous integration pipelines
  • Code scanning — Review LLM-related security and compliance issues in pull requests
  • Result sharing — Share evaluation results with team members for collaborative analysis

Use Cases

💡 Evaluate LLM prompt quality before production deployment
💡 Run red team security testing on AI applications to find vulnerabilities
💡 Choose the best model among multiple LLM providers
💡 Establish continuous regression testing baselines for LLM applications

Tags

evaluation testing prompts typescript

Categories

📊 Observability

Quick Start

npm install -g promptfoo
export OPENAI_API_KEY=sk-xxx
promptfoo init --example getting-started
cd getting-started
promptfoo eval
promptfoo view
Visit GitHub Visit Website View Docs

Related Projects