Midscene.js

Active

GitHub TypeScript MIT

Description

AI-powered vision-driven UI automation that lets you describe actions in natural language instead of writing selectors, supporting browser and mobile platforms

Key Features

AI-powered vision-driven UI automation using natural language descriptions instead of selectors
Cross-platform support: Web (Puppeteer/Playwright), Android (adb), and iOS (WebDriverAgent)
Pure vision localization — no DOM dependency for element interaction, reducing token cost
Supports multiple vision-language models: Qwen3.x, Doubao-Seed-2.0, GLM-4.6V, UI-TARS
Three API layers: Interaction API, Data Extraction API, and Utility API (aiAssert, aiLocate, aiWaitFor)
MCP integration exposing Midscene Agent actions as tools for upper-layer agent orchestration

Use Cases

💡 Automating web UI testing with natural language test scripts across browsers

💡 Building cross-platform mobile automation for Android and iOS app testing

💡 Creating AI agent workflows that interact with any user interface visually

💡 Prototyping UI automation without writing fragile CSS/XPath selectors

Quick Start

npm install @midscene/web && npx @midscene/web playground

Visit GitHub Visit Website

Related Projects

MCP Playwright

5.6k · TypeScript

Stale

Playwright Model Context Protocol server for automating browsers and APIs in Claude Desktop, Cline, Cursor IDE and other AI coding tools

mcpplaywrightbrowser-automation +2

HyperAgent

1.5k · TypeScript

Normal

HyperAgent is a Playwright-based AI browser automation framework offering high-level APIs like page.ai(), page.perform(), and page.extract(). It features built-in MCP client support and action caching, enabling AI agents to browse, interact, and extract data using natural language.

browser-automationplaywrightai-agent +3

AIPex

1.2k · TypeScript

Active

AI browser automation assistant as a Chrome extension, privacy-first with MCP support, alternative to Claude Chrome and Manus Browser Operator

browser-automationchrome-extensionmcp +2

Magnitude

4.1k · TypeScript

Stale

An open-source, vision-first browser agent that drives web automation through visual understanding, supporting complex web interaction tasks for QA testing and workflow automation.

vision-firstbrowser-automationweb-agent +2

Browser AgentWeb 自动化Playwright

Browser Agents in Practice: Architecture and Pitfalls of AI-Controlled Browsers

Breaking down three abstraction layers for browser automation—from raw Playwright to structured extraction—with production patterns, runnable code, and common pitfalls.