PyWinAssistant

Stale
GitHub Python MIT

Description

The first open-source Artificial Narrow Intelligence generalist agent that fully operates GUIs using only natural language. Uses Visualization-of-Thought and Chain-of-Thought reasoning for spatial perception and HID simulation.

Key Features

  • First open-source Computer-Using-Agent that operates GUIs through native Windows Accessibility APIs without OCR or pixel-level imaging
  • Symbolic spatial mapping with hierarchical element tracking via OS-native parent/child relationships and coordinate systems
  • Natural language to GUI automation: parse intent, map to UI elements, and generate adaptive workflows
  • Cross-application state awareness treating all apps as nodes in a single UIA-accessible graph
  • Self-healing workflows that auto-adjust to UI changes reducing maintenance overhead
  • AI-powered script generation that translates natural language into executable code in any IDE

Use Cases

💡 Automating complex Windows desktop workflows using only natural language commands
💡 Cross-application data transfer and orchestration between GUI apps without API integration
💡 Building accessible computer control systems for users with disabilities
💡 Automating repetitive Windows tasks like form filling, data entry, and report generation
💡 Creating end-to-end test automation for Windows desktop applications

Tags

browser agent python automation desktop

Categories

🌐 Browser Agent

Quick Start

1. Install dependencies with `pip install pywinassistant`. 2. Set your LLM API key. 3. Write a natural language command like `assistant('Play Daft Punk on Spotify')`. 4. PyWinAssistant uses Windows UIA APIs to detect UI elements and execute the workflow without OCR or screenshots.
Visit GitHub

Related Projects