Crawlee

Active
GitHub TypeScript Apache-2.0

Description

A web scraping and browser automation library for Node.js to build reliable crawlers, supporting Puppeteer, Playwright, Cheerio, and raw HTTP. Extract data for AI, LLMs, RAG, or GPTs with proxy rotation and both headful and headless modes.

Key Features

  • Multi-engine unified interface — same API supports Puppeteer, Playwright, Cheerio and raw HTTP
  • Anti-detection stealth — default config simulates human behavior to bypass major bot protections
  • Proxy rotation and session management — built-in proxy rotation, fingerprint generation and session persistence
  • Persistent queue — supports breadth-first and depth-first URL crawling queues
  • Automatic scaling — adjusts crawling concurrency based on available system resources
  • Structured data storage — built-in Dataset and KeyValueStore for tabular and file storage

Use Cases

💡 Batch scrape web content for AI/RAG applications to build knowledge base data sources
💡 Crawl e-commerce product info for price monitoring and competitive analysis
💡 Automate news and social media content collection for sentiment analysis
💡 Extract structured data from JavaScript-rendered SPA websites
💡 Build scheduled crawling pipelines to monitor data changes on target websites

Tags

typescript javascript data-processing tools rag automation

Categories

📚 RAG Tools ⚡ Agent Tools

Quick Start

npx crawlee create my-crawler
cd my-crawler
npm start
Visit GitHub

Related Projects