Hugging Face Evaluate

Active
GitHub Python Apache-2.0

Description

A library by Hugging Face for easily evaluating machine learning models and datasets, providing a wide range of metrics and evaluation methods.

Key Features

  • Standardized metrics library — load dozens of popular metrics with `load('accuracy')`, covering NLP, CV, and dataset-specific tasks
  • Framework-agnostic evaluation — works across Numpy, Pandas, PyTorch, TensorFlow, and JAX without code changes
  • Type-safe inputs — automatic type checking ensures correct input formats for each metric
  • Metric cards — each metric includes documentation of values, limitations, ranges, and usage examples
  • Community Hub integration — publish custom evaluation modules to Hugging Face Hub and share with others
  • CLI for metric creation — scaffold new evaluation modules with `evaluate-cli create` and push to Hub Spaces

Use Cases

💡 Comparing multiple ML models on standard benchmarks like accuracy, F1, and BLEU
💡 Evaluating dataset quality and bias using measurement modules
💡 Building custom evaluation pipelines for domain-specific NLP or CV tasks
💡 CI/CD model regression testing with automated metric reporting
💡 Collaborative evaluation projects where teams share metrics via the Hub

Tags

evaluation llm python huggingface framework

Categories

📊 Observability

Quick Start

# Install
pip install evaluate

# Usage
import evaluate
accuracy = evaluate.load('accuracy')
results = accuracy.compute(predictions=[1, 2, 3], references=[1, 2, 4])
print(results)
Visit GitHub Visit Website

Related Projects