DocETL
ActiveDescription
An agentic LLM-powered data processing and ETL system. Enables complex data transformations using natural language-defined pipelines, turning unstructured data into structured, analyzable outputs with LLM intelligence.
Key Features
- LLM-powered data processing pipeline engine with operators like map, reduce, resolve, gleaning, and filter for complex document transformations
- DocWrangler interactive UI playground for iterative prompt engineering — experiment with prompts and see results in real-time
- Python package for production pipelines runnable from command line or Python code with full programmatic control
- Natural language pipeline definition — describe data processing tasks in plain English and let the LLM generate pipeline configurations
- AWS Bedrock integration with configurable profiles and regions for enterprise-grade LLM backend support
- Claude Code integration for pipeline authoring assistance — use Claude to help write and debug pipeline configurations
Use Cases
💡 Extracting structured information from unstructured documents like research papers, legal contracts, or reports
💡 Building ETL pipelines that use LLM intelligence to handle edge cases traditional rule-based systems miss
💡 Iteratively developing and testing data processing prompts using the visual DocWrangler playground
💡 Processing large document corpora with AI-powered deduplication, merging, and quality evaluation
💡 Creating RAG-ready datasets from raw documents through intelligent chunking and metadata extraction
Categories
Quick Start
Install with `pip install docetl`. Set `OPENAI_API_KEY` in `.env`. For the UI playground, run `make docker` (requires Docker) and visit http://localhost:3000/playground. For production pipelines, define operations in a YAML config and run with `docetl run pipeline.yaml`.