HomeProjectsArticlesPK BoardAbout
Explore Projects

OmniParser

Active
GitHub Jupyter Notebook CC-BY-4.0

Description

OmniParser is Microsoft's open-source screen parsing tool that turns GUI screenshots into structured elements, enabling reliable vision-only agent actions.

Key Features

  • Vision-only — works from screenshots, no DOM or UI tree required
  • Structured output — parses into recognizable icon and text regions
  • GPT-4V friendly — gives the vision model precise coordinates
  • Boosts GUI agents — enables reliable clicking and typing
  • Open-source — model and code released under open licenses
  • Multi-platform — Web, desktop and mobile apps

Use Cases

💡 Powering vision-only GUI agents with screen parsing
💡 Accessibility and automation over legacy desktop apps
💡 Bringing unparseable legacy apps into agent workflows
💡 Generating labeled screenshot data for training

Strengths & Limitations

✅ Strengths

  • • Actively maintained, recent updates
  • • High community interest (25.3k stars)
  • • Responsive to issues, low backlog

⚠️ Limitations

  • • No clear open-source license

Tags

vision gui-agent screen-parser microsoft ui

Categories

🌐 Browser Agent ⚡ Agent Tools

Quick Start

git clone https://github.com/microsoft/OmniParser.git
cd OmniParser
pip install -r requirements.txt
python gradio_demo.py
Visit GitHub

Related Projects

Agent S

12.2k · Python
Active A+

Open-source agentic framework that uses computers like a human, capable of completing complex GUI tasks with autonomous learning and experience accumulation.

computer-usegui-agentautomation +1
  • · First framework to surpass human-level performance on OSWorld (72.60%) benchmark
  • · Cross-platform support for Linux, macOS, and Windows with single-monitor GUI interaction
  • · Modular grounding with UI-TARS-1.5-7B and support for multiple LLM backends (OpenAI, Anthropic, Gemini)
View full details → View on GitHub

Chrome DevTools MCP

49.3k · TypeScript
Active A+

MCP server providing Chrome DevTools capabilities to coding agents, enabling web debugging, performance analysis, and DOM manipulation automation.

mcpchrome-devtoolsbrowser-debug +2
  • · Performance Insights — Record Chrome performance traces and extract actionable optimization recommendations
  • · Advanced Browser Debugging — Analyze network requests, take screenshots, check console with source-mapped stack traces
  • · Reliable Automation — Puppeteer-based browser automation with automatic wait for action results
View full details → View on GitHub

Vision Agents

8.1k · Python
Active A+

Open Vision Agents by Stream. Build voice and vision agents quickly with any model or video provider, using Stream's edge network for ultra-low latency realtime interactions.

voiceagentpython +3
  • · Real-time video AI with YOLO, Roboflow integration combined with Gemini/OpenAI for live visual understanding
  • · Ultra-low latency audio/video under 30ms via Stream's edge network with 500ms join time
  • · Native SDK methods from OpenAI, Gemini, and Claude for always-latest LLM capabilities
View full details → View on GitHub