HELM
ActiveDescription
HELM (Holistic Evaluation of Language Models) is Stanford CRFM's open-source framework for holistic, reproducible, and transparent evaluation of foundation models including LLMs and multimodal models.
Key Features
- Standardized benchmarks including MMLU-Pro, GPQA, IFEval, WildBench
- Unified interface to evaluate models from OpenAI, Anthropic, Google and more
- Metrics beyond accuracy — efficiency, bias, toxicity, fairness
- Web UI for inspecting individual prompts and model responses
- Official leaderboards for capabilities, safety, and vision-language models
Use Cases
💡 Benchmarking new LLM releases against established baselines
💡 Comparing model performance across safety and bias dimensions
💡 Reproducing published evaluation results for research papers
💡 Building internal leaderboards for model selection in enterprises
💡 Evaluating multimodal models on vision-language tasks
Categories
Quick Start
Install: `pip install crfm-helm`. Run a benchmark: `helm-run --run-entries mmlu:subject=philosophy,model=openai/gpt2 --suite my-suite --max-eval-instances 10`. Summarize: `helm-summarize --suite my-suite`. View results: `helm-server --suite my-suite` then open http://localhost:8000/.