Lighteval

Active
GitHub Python MIT

Description

HuggingFace's all-in-one toolkit for evaluating LLMs across multiple backends, deeply integrated with the HuggingFace ecosystem and providing flexible evaluation metrics and benchmark configuration.

Key Features

  • 1000+ evaluation tasks — covers knowledge, math, code, chat, multilingual, and NLU benchmarks
  • Multi-backend support — evaluate via Accelerate, VLLM, SGLang, Nanotron, or remote inference endpoints
  • Custom tasks and metrics — create tailored evaluation pipelines with custom task and metric definitions
  • Sample-by-sample results — save detailed per-sample outputs for debugging and model comparison
  • HuggingFace ecosystem integration — push results to Hub and use the Open Benchmark Index for discovery
  • Python API and CLI — use either the command-line interface or programmatic Python API for evaluation

Use Cases

💡 Benchmarking LLMs against standard evaluations like MMLU, GSM8K, and HumanEval
💡 Comparing model performance across multiple backends and hardware configurations
💡 Custom evaluation pipelines for domain-specific or multilingual model assessment
💡 CI/CD integration for automated model quality gates before deployment
💡 Research reproducibility with standardized evaluation reporting and tracking

Tags

llm-evaluation evaluation-framework huggingface benchmark python

Categories

📊 Observability

Quick Start

# Install
pip install lighteval

# Evaluate via CLI
lighteval eval "hf-inference-providers/openai/gpt-oss-20b" gpqa:diamond

# Or use Python API
from lighteval.pipeline import Pipeline, PipelineParameters
pipeline = Pipeline(model=model, pipeline_parameters=params, tasks='gsm8k')
results = pipeline.evaluate()
Visit GitHub Visit Website

Related Projects