Browser Agent Data Extraction and Form Filling: From LLM Vision to DOM Selectors

Browser Agents are among the most densely implemented agent types in enterprises during 2025. Two high-frequency scenarios — extracting structured data from web pages in batch, and auto-filling forms for submission — account for 80% of real demand.

1. Essential Differences Between Two Tasks

Dimension	Data Extraction	Form Filling
Operation direction	Read (read-only)	Write
Error cost	Low (failure does not submit)	High (wrong submission)
Speed requirement	Batched (multiple pages per second)	Serial (must wait for previous step)
Fault tolerance need	Medium (retryable)	High (must verify)
Success rate	95%+ acceptable	Must be 99%+
Anti-bot resistance	Medium	High

Key difference: data extraction is read-only, batchable; form filling is write, has side effects. Form filling errors include duplicate charges, wrong orders, garbage data, compliance risks.

2. Three Action Localization Strategies

Strategy 1: Vision Only LLM

Send the entire page screenshot to GPT-4o / Claude / Gemini, model directly outputs coordinates. Advantages: no need to parse DOM, no requirement on frontend stack. Disadvantages: poor coordinate precision (±10-20 pixels), high latency (1-3 seconds), high cost (1000+ tokens per screenshot).

Strategy 2: DOM-Based Selectors

Extract page's accessibility tree / DOM tree, serialize to text, let LLM output CSS/XPath selectors. Advantages: high precision, fast speed (<500ms), low cost (500-2000 tokens), explainable. Disadvantages: DOM noisy, Shadow DOM hard, iframe nesting, dynamic class.

Key optimization: use accessibility tree instead of raw DOM — browser natively extracted ARIA tree only contains semantic elements, node count usually 5%-10% of raw DOM. This is core approach of Browser-Use, Stagehand, MCP Playwright.

Strategy 3: Hybrid Coordinates

Combination: first use DOM to extract candidate element list, then use vision LLM on reduced screenshot for final confirmation. Advantages: high precision, fast speed (small image inference), occlusion-resistant. Disadvantages: complex implementation.

Three Strategies Comparison

Dimension	Vision LLM	DOM Selector	Hybrid Coordinate
Precision	Medium	High	High
Speed	Medium	High	Medium
Cost	Medium	High	Medium
Implementation difficulty	Simple	Medium	Complex
Style change resistance	High	Medium	Medium
Anti-bot resistance	Medium	High	Medium
Applicable pages	Minimal/legacy	Modern SPA	Complex hybrid

Rules of thumb: 90% of modern web applications → DOM selector; anti-bot strict pages → Vision LLM; high precision requirement tasks → Hybrid coordinate.

3. Data Extraction Practice

Case 1 (batch e-commerce): key optimization points include scroll loading trigger (IntersectionObserver, agent must simulate scroll to bottom), pagination vs infinite scroll (Taobao/JD/Xiaohongshu all use infinite scroll), data validation (price sometimes displays as "from ¥99.00", must parse to number), deduplication.

Case 2 (news article extraction): use Stagehand + DOM extraction, call page.extract with instruction and schema to extract title, author, publish time, body.

4. Form Filling Practice

Case 3 (auto registration): agent task description includes entering email, username, password, clicking terms checkbox, clicking submit, waiting for page jump, screenshot confirmation. Key: form operations must have second confirmation, wait for server response before next step.

Case 4 (office reporting): direct Playwright operation skipping LLM — LLM form filling failure rate is high (5-15%), pure Playwright is 0%. Each submit must wait_for_selector — form submission involves async network.

Key experience: use pure Playwright when possible, not LLM; every submit must wait_for_selector; failure retry + skip strategy; sensitive operations must have human approval.

5. Four Frameworks Comparison

Browser-Use

Most popular Python framework in 2024 (30k+ stars). Advantages: pure DOM route, fastest, cheapest. Disadvantages: weak iframe nesting support, no vision fallback, complex pages need refined prompts.

Stagehand (by Browserbase)

TypeScript framework emphasizing code + AI hybrid. Advantages: code/AI hybrid switching, TypeScript type-friendly, cloud Browserbase hosting. Disadvantages: cloud version paid.

Playwright MCP (by Microsoft)

Microsoft official MCP Server, exposes Playwright as MCP tools. Advantages: deep integration with Claude/Cursor, native multi-tab support, Microsoft official maintenance. Disadvantages: limited to LLM clients, cannot directly use for production backend.

Skyvern

Vision LLM + computer vision for browser automation. Advantages: pure vision route, effective for legacy/inaccessible DOM systems. Disadvantages: high cost, slow speed.

Framework Comparison

Dimension	Browser-Use	Stagehand	Playwright MCP	Skyvern
Route	DOM	DOM + Code	DOM	Vision
Language	Python	TypeScript	MCP (any)	Python/Cloud
Speed	High	High	Medium	Low
Cost	High	Medium	Medium	Low
Ease of use	High	High	High	Medium
Production-grade	Medium	High	Medium	High
Anti-bot resistance	Medium	Medium	Medium	High

6. Anti-Bot and Production Deployment

Anti-bot: headless=False has higher pass rate; configure user_agent, locale, timezone, proxy; disable-blink-features=AutomationControlled removes webdriver flag.

Production deployment architecture: API layer receives tasks and allocates Browser instances; scheduling layer uses Redis queue + priority + retry on failure; execution layer is Browser-Use worker pool + browser isolation; browser pool is local Playwright or Browserbase cloud.

Key architecture principles: browser isolation (each task independent BrowserContext), resource recycling (tasks end must close page), rate limiting (same domain QPS ≤ 2), failure retry (exponential backoff, max 3 times), full-chain logging (each action screenshot + DOM snapshot).

7. Common Traps

1. All buttons found by LLM: login, submit, confirm — these buttons have fixed selectors, always use CSS selectors.
2. Submit without verification: form filling must wait for server response, screenshot confirm.
3. headless = default: anti-bot strict sites must use headless=False + anti-detection parameters.
4. iframe is black box: cross-iframe operations must first switch_to_frame.
5. CAPTCHA recognition bypass: CAPTCHA is anti-automation by design, bypassing has legal risks.

8. Selection Recommendations

• Enterprise internal tools / R&D self-use: Browser-Use
• SaaS product integration: Stagehand
• AI clients (Claude/Cursor): Playwright MCP
• Government/financial legacy systems: Skyvern
• Batch data collection: self-developed Browser-Use + proxy pool + rate limiting

Final word: Browser Agent success rate is 70% from prompt engineering, 30% from framework choice.

9. Cost Optimization Strategies

Browser Agent costs come from three places:

1. LLM inference cost: GPT-4o is $5/1M input tokens. Each DOM extraction averages 2K tokens. 100K pages = $1K just for LLM.
2. Browser hosting cost: Browserbase charges $0.10/hour per browser. 24/7 operation = $72/browser/month.
3. Proxy cost: Residential proxies $1-3/GB. Anti-bot scraping may need 10GB+/day.

Optimization paths:

• Cache repeated DOM extractions (same URL re-scraped returns same content)
• Use gpt-4o-mini for simple extraction, gpt-4o for complex judgment
• Batch operations: extract 100 pages in parallel, not sequentially
• Compress accessibility tree: only send semantic elements, skip styling nodes

10. Compliance and Legal Considerations

Browser Agent usage must consider:

1. Terms of Service: Many websites explicitly prohibit automated scraping (LinkedIn, Facebook). Violating ToS can lead to legal action.
2. robots.txt: Respect robots.txt. Even if you can bypass, legally should not.
3. Copyright: Scraped content may be copyrighted. Storage and redistribution require legal review.
4. Personal data: GDPR, CCPA. If scraped data contains personal information, need lawful basis for processing.
5. Rate limiting: Even without technical limits, ethical scraping limits request rate.

For enterprise use, recommend legal team review before deployment.

11. Monitoring and Observability

Production Browser Agent requires monitoring:

1. Success rate per site: Track per-domain success rate. Sites below 90% need prompt tuning or fallback strategy.
2. Average time per page: P50, P95, P99. Slow sites may need different strategy.
3. Token usage per extraction: High token usage indicates noisy DOM, needs optimization.
4. Browser pool utilization: Idle vs busy ratio. Underutilization = over-provisioning.
5. Error type distribution: Network errors, parsing errors, anti-bot blocks. Each requires different response.

Tools: Langfuse (LLM tracing), Browserbase built-in monitoring, custom Prometheus metrics.