OpenRLHF
ActiveDescription
OpenRLHF is a high-performance agentic RL framework based on Ray and vLLM, offering PPO, DAPO, and REINFORCE++ algorithms for large-scale training of agents and vision-language models.
Key Features
- First production-ready open-source RLHF framework built on Ray + vLLM distributed architecture
- Supports PPO, DAPO, REINFORCE++, GRPO, and RLOO reinforcement learning algorithms
- Hybrid engine scheduling where all models share GPUs to minimize idle time
- VLM (Vision-Language Model) RLHF support for training models with image inputs
- Multi-turn agent RL with complex environment interactions and custom reward functions
- DeepSpeed ZeRO-3 integration enabling training of 70B+ parameter models
Use Cases
💡 Training large language models with RLHF for alignment and safety
💡 Fine-tuning reasoning models using REINFORCE++ and GRPO algorithms
💡 Training vision-language models with multi-turn image interactions
💡 Building custom agent reward functions for domain-specific RL training
Strengths & Limitations
✅ Strengths
- • Actively maintained, recent updates
- • High community interest (9.9k stars)
- • Permissive open-source license (Apache-2.0)
- • Established track record (3 years in production)
Categories
Quick Start
Install via pip: `pip install openrlhf`. Requires Ray and vLLM. Run SFT training with `ray_start.sh` and example scripts. For agent RL: define a custom reward function and use `--train.agent_func_path`. Check examples/scripts/ for ready-to-use training configurations.