RouteLLM
StaleDescription
RouteLLM is a framework for serving and evaluating LLM routers, enabling cost reduction without compromising quality through intelligent request routing across multiple model tiers.
Key Features
- Drop-in replacement for OpenAI client to automatically route queries between strong and weak models
- Pre-trained routers reducing costs by up to 85% while maintaining 95% GPT-4 quality
- Multiple routing strategies (Matrix Factorization, Swapping Rollouts, Sentence BERT) with benchmark comparisons
- OpenAI-compatible server mode for integration with any existing LLM client
- Configurable cost threshold for fine-grained cost vs. quality tradeoff control
- Broad model support via LiteLLM — Anthropic, Gemini, Bedrock, Together, Ollama, and more
Use Cases
💡 Reducing LLM inference costs in production by routing simple queries to cheaper models
💡 A/B testing and comparing different LLM routing strategies on real workloads
💡 Building cost-aware AI applications that dynamically select model tiers based on query complexity
💡 Deploying an OpenAI-compatible proxy that intelligently balances response quality and spend
💡 Research and evaluation of LLM routing algorithms across standardized benchmarks
Categories
Quick Start
pip install 'routellm[serve,eval]' → Set OPENAI_API_KEY and provider keys → Initialize Controller with routers=['mf'] → Set strong_model (e.g., gpt-4) and weak_model (e.g., Mixtral) → Calibrate threshold for target cost ratio → Use model='router-mf-{threshold}' in completions → Optional: launch OpenAI-compatible server for any client