OpenLLM
ActiveDescription
Run any open-source LLMs such as DeepSeek and Llama as OpenAI-compatible API endpoints in the cloud. Supports fine-tuning, quantization, and distributed inference for production-grade LLM deployment.
Key Features
- Any Open-Source LLM Support: Run DeepSeek, Llama, Qwen and any open-source models
- OpenAI-Compatible APIs: Provides endpoints compatible with OpenAI API for seamless integration
- Built-in Chat UI: Ready-to-use chat interface
- State-of-the-Art Inference Backends: Supports latest inference optimization techniques
- Model Repository: Manage model versions and dependencies, support custom model repositories
- Production-Grade Deployment: Supports Docker, Kubernetes, and BentoCloud cloud deployment
Use Cases
π‘ Self-Hosting LLMs: Deploy open-source large language models locally or in the cloud
π‘ API Endpoint Services: Provide OpenAI-compatible API services for applications
π‘ Model Fine-Tuning: Fine-tune open-source models for specific tasks
π‘ Production Deployment: Deploy large models to production with high concurrency support
π‘ Development Testing: Quickly start models for development and testing
Categories
Quick Start
1. Install OpenLLM: pip install openllm
2. Run interactive experience: openllm hello
3. Start LLM server: openllm serve llama3.2:1b
4. Visit http://localhost:3000 to use chat UI
5. Or use API: from openai import OpenAI; client = OpenAI(base_url='http://localhost:3000/v1', api_key='na')