SWE-bench
ActiveDescription
SWE-bench is a benchmark for evaluating language models on real-world GitHub issue resolution, featuring genuine problems from popular Python repositories, now a core standard for measuring AI coding agent capabilities.
Key Features
- LLM coding capability benchmark based on real GitHub issues from popular Python repositories
- Multiple subsets: SWE-bench Lite, Verified (human-validated), and Multimodal editions
- Fully containerized Docker evaluation framework ensuring reproducible results
- Supports local and cloud-based evaluation (Modal, AWS) with flexible environment choices
- Companion SWE-Llama models and BM25/Oracle retrieval-augmented datasets
- ICLR 2024 Oral paper, now the core standard for measuring AI coding agent capabilities
Use Cases
💡 Evaluate and compare different LLM/Agent capabilities in solving real software engineering problems
💡 Train and fine-tune models specialized for code generation (with SWE-smith training data toolkit)
💡 Research AI reasoning and code modification abilities in software development scenarios
💡 Provide standardized evaluation and ranking benchmarks for AI coding agents
Strengths & Limitations
✅ Strengths
- • Actively maintained, recent updates
- • High community interest (5.6k stars)
- • Permissive open-source license (MIT)
- • Established track record (2 years in production)
Tags
Categories
Quick Start
Install Docker, clone the repo and run pip install -e . Use python -m swebench.harness.run_evaluation with dataset and predictions path to run evaluation. Recommended on x86_64 with 120GB+ storage and 16GB+ RAM.