HomeProjectsArticlesPK BoardAbout
Explore Projects

UFO

Active
GitHub Python MIT

Description

UFO is a Windows GUI automation agent by Microsoft that understands screen interfaces and executes complex OS tasks through natural language commands.

Key Features

  • UFO³ multi-device Agent Galaxy with cross-device DAG workflow orchestration and parallel execution
  • Deep Windows OS integration with hybrid GUI and API action capabilities
  • Declarative task decomposition into dynamic DAGs with runtime adaptive adjustments and continuous graph evolution
  • Unified Agent Interaction Protocol (AIP) with WebSocket-based secure coordination and automatic fault recovery
  • Dual modes: UFO² desktop AgentOS and UFO³ Galaxy covering simple to complex automation scenarios
  • Supports heterogeneous platforms including Windows, Linux, Android with MCP integration for tool augmentation

Use Cases

💡 Cross-device office automation: operate multiple computers simultaneously via natural language instructions
💡 AI-driven automation testing and UI regression testing for Windows desktop applications
💡 Execute batch system configuration and fault troubleshooting tasks via natural language in IT operations
💡 Build enterprise-grade intelligent assistants with multi-device capabilities for cross-platform data processing
💡 Educational and demo scenarios showcasing how AI agents understand screen interfaces and execute operations

Strengths & Limitations

✅ Strengths

  • • Actively maintained, recent updates
  • • High community interest (9.5k stars)
  • • Permissive open-source license (MIT)
  • • Established track record (2 years in production)

Tags

gui-agent windows automation python

Categories

🌐 Browser Agent

Quick Start

1. Install Python 3.10+ and dependencies
2. Clone the repo and run pip install -e .
3. Configure OpenAI API Key
4. Run UFO² (simple scenarios): python -m ufo
5. Run UFO³ Galaxy (multi-device): refer to galaxy/README.md
6. Start automation by describing tasks in natural language
Visit GitHub Visit Website View Docs

Related Projects

Agent S

12.2k · Python
Active A+

Open-source agentic framework that uses computers like a human, capable of completing complex GUI tasks with autonomous learning and experience accumulation.

computer-usegui-agentautomation +1
  • · First framework to surpass human-level performance on OSWorld (72.60%) benchmark
  • · Cross-platform support for Linux, macOS, and Windows with single-monitor GUI interaction
  • · Modular grounding with UI-TARS-1.5-7B and support for multiple LLM backends (OpenAI, Anthropic, Gemini)
View full details → View on GitHub

ScaleCUA

1.1k · Python
Stale C

Open-sourced computer use agents that can operate on cross-platform environments including Windows, macOS, Ubuntu, and Android. ICLR 2026 Oral paper project.

browserpythonagent +1
  • · Cross-platform GUI operation: supports Windows, macOS, Ubuntu, Android, and Web environments
  • · Large-scale training data: operation trajectories across 6 operating systems and 3 task domains
  • · State-of-the-art performance: MMBench-GUI L1-Hard 94.4%, OSWorld-G 60.6%, WebArena-Lite-v2 47.4%
View on GitHub

PyWinAssistant

1.3k · Python
Stale C

The first open-source Artificial Narrow Intelligence generalist agent that fully operates GUIs using only natural language. Uses Visualization-of-Thought and Chain-of-Thought reasoning for spatial perception and HID simulation.

browseragentpython +2
  • · First open-source Computer-Using-Agent that operates GUIs through native Windows Accessibility APIs without OCR or pixel-level imaging
  • · Symbolic spatial mapping with hierarchical element tracking via OS-native parent/child relationships and coordinate systems
  • · Natural language to GUI automation: parse intent, map to UI elements, and generate adaptive workflows
View on GitHub