Inspect AI
ActiveDescription
A framework for large language model evaluations developed by the UK AI Safety Institute (AISI), providing comprehensive model capability assessment tools with support for safety and alignment testing.
Key Features
- Comprehensive LLM evaluation framework created by the UK AI Safety Institute (AISI)
- 200+ pre-built evaluations ready to run on any model for capability assessment
- Built-in prompt engineering, tool usage, multi-turn dialog, and model-graded evaluations
- Extensible architecture with Python package-based plugins for new elicitation and scoring techniques
- Web UI frontend with TypeScript/React for visualizing evaluation results
- Full CLI and Python API for programmatic evaluation orchestration
Use Cases
💡 Benchmarking and comparing LLM capabilities across safety, alignment, and task performance dimensions
💡 Government and enterprise AI safety testing before production deployment
💡 Running standardized evaluations to detect model regressions and capability drift
💡 Research institutions conducting reproducible AI capability assessments
💡 Building custom evaluation suites with the extensible plugin system
Categories
Quick Start
1. Clone repo: git clone https://github.com/UKGovernmentBEIS/inspect_ai.git && cd inspect_ai
2. Install: pip install -e ".[dev]" or uv sync --extra dev
3. Run evaluation: inspect eval <eval-path> --model <model>
4. Browse 200+ pre-built evals at https://inspect.aisi.org.uk/evals/
5. Docs: https://inspect.aisi.org.uk/