LM Evaluation Harness
ActiveDescription
A framework for few-shot evaluation of language models by EleutherAI, providing standardized evaluation pipelines supporting hundreds of benchmark tasks and widely adopted as a core LLM evaluation tool in the community.
Key Features
- Over 60 standard academic benchmarks with hundreds of subtasks
- Support for transformers, vLLM, GPT-NeoX, and multiple inference backends
- Commercial API model evaluation for OpenAI, Anthropic, and more
- Publicly available prompts ensure reproducibility and cross-paper comparability
- Adapter evaluation support including LoRA via PEFT library
- Lightweight base install with optional model backends installed separately
Use Cases
💡 LLM performance benchmark comparison
💡 Pre-release model quality assessment
💡 Model evaluation for academic papers
💡 Enterprise model selection and comparison
💡 Open LLM Leaderboard evaluation tasks
Categories
Quick Start
Clone the repo and run pip install -e . for the base framework. Install the HuggingFace backend with pip install "lm_eval[hf]", then run lm_eval --model hf --model_args pretrained=EleutherAI/gpt-j-6B --tasks hellaswag --device cuda:0 to evaluate.