AgentBench
StaleDescription
A comprehensive benchmark to evaluate LLMs as agents (ICLR 2024), covering operating systems, databases, knowledge graphs, digital card games and more.
Key Features
- First comprehensive benchmark evaluating LLMs as autonomous agents across 8 diverse environments
- Covers OS interaction, database operations, knowledge graphs, digital card games, and lateral thinking puzzles
- AgentBench FC integrates function-calling style prompts with AgentRL reinforcement learning framework
- Fully containerized deployment via Docker Compose for all 5 core tasks (ALFWorld, DB, KG, OS, WebShop)
- Published leaderboard tracking model performance with continuous community contributions and updates
- Supports both Dev and Test dataset splits with multi-turn interaction requiring thousands of LLM generations
Use Cases
💡 Benchmark and compare different LLM models on real-world agent capabilities
💡 Evaluate function-calling performance of models in structured tool-use scenarios
💡 Train and validate agent behaviors using the integrated AgentRL reinforcement learning framework
💡 Research agent failure modes across diverse task environments including OS, databases, and web
Tags
Categories
Quick Start
Clone the repo and set up with `conda create -n agent-bench python=3.9 && conda activate agent-bench && pip install -r requirements.txt`. Build required Docker images for the tasks you want to test. Configure your OpenAI API key in `configs/agents/openai-chat.yaml`. Start task workers and run benchmarks.