Harbor
ActiveDescription
Framework for running agent evaluations and creating RL environments to measure and improve agent performance
Key Features
- Evaluate arbitrary agents like Claude Code, OpenHands, and Codex CLI with standardized benchmarks
- Build and share custom benchmarks and evaluation environments
- Run experiments across thousands of environments in parallel via Daytona and Modal providers
- Generate rollouts for reinforcement learning optimization of agent models
- Official harness for Terminal-Bench-2.0 agent evaluation
- CLI-driven workflow for easy integration into CI/CD pipelines
Use Cases
💡 Benchmark and compare different coding agents on standardized tasks
💡 Generate training data for reinforcement learning from agent rollouts
💡 Run large-scale agent evaluations in cloud environments
💡 Integrate agent performance testing into continuous integration workflows
Categories
Quick Start
Install with `uv tool install harbor` or `pip install harbor`. Set your API key (e.g., `ANTHROPIC_API_KEY`), then run `harbor run --dataset terminal-bench@2.0 --agent claude-code --model anthropic/claude-opus-4-1` to execute a benchmark locally via Docker.