AppAgent
StaleDescription
AppAgent is an LLM-based multimodal agent framework designed to operate smartphone apps like a human, supporting touch interaction and autonomous exploration.
Key Features
- LLM-based multimodal agent that operates smartphone apps via tapping and swiping
- Supports both autonomous exploration and human demonstration learning modes
- Simplified action space design operates across apps without system backend access
- Integrates GPT-4V and Qwen-VL multimodal models for decision making
- Grid overlay feature precisely locates unlabeled UI elements on screen
- Works with Android Studio emulator — no physical device required
Use Cases
💡 Automate social media operations like following users and posting content
💡 Assist visually impaired users in navigating smartphone applications
💡 Batch automate repetitive mobile tasks at scale
💡 Research human-computer interaction and GUI automation
Categories
Quick Start
1. Install Android Debug Bridge (adb) and connect your Android device via USB
2. Clone the repo and install dependencies: `pip install -r requirements.txt`
3. Configure your OpenAI API key and request interval in `config.yaml`
4. Start the exploration phase — choose autonomous exploration or human demonstration
5. Run the deployment phase to execute your target task