Scrapling
ActiveDescription
An adaptive web scraping framework that intelligently handles anti-bot measures, from single requests to full-scale crawls, designed for AI agent data collection.
Key Features
- Adaptive selectors — Parser learns from website structure changes and automatically relocates elements when pages update
- Anti-bot bypass — Built-in StealthyFetcher and DynamicFetcher bypass Cloudflare Turnstile and other anti-bot systems
- Multi-mode fetchers — Fetcher, AsyncFetcher, StealthyFetcher, and DynamicFetcher for different scraping scenarios
- Large-scale spider framework — Supports concurrent multi-session crawls with pause/resume and automatic proxy rotation
- Real-time stats & streaming — Real-time crawl statistics and streaming output for monitoring large-scale operations
- AI Agent integration — Agent Skill and MCP server for direct invocation by AI agents for data collection
Use Cases
💡 Scrape e-commerce product prices and inventory with adaptive selectors that survive page redesigns
💡 Provide AI agents with real-time web data collection capabilities via direct MCP server invocation
💡 Large-scale crawling with proxy rotation and concurrency control to avoid IP bans and rate limiting
💡 Monitor competitor website changes with adaptive detection of page structure updates and automatic rule adjustment
💡 Build data pipelines that continuously collect data from multiple sources and stream to downstream systems
Categories
Quick Start
pip install scrapling
from scrapling.fetchers import Fetcher, StealthyFetcher
# Basic fetch
page = Fetcher.get('https://example.com')
products = page.css('.product')
# Stealth mode to bypass anti-bot
StealthyFetcher.adaptive = True
page = StealthyFetcher.fetch(
'https://example.com',
headless=True,
network_idle=True
)
products = page.css('.product', auto_save=True)