Dagster
ActiveDescription
An orchestration platform for developing, producing, and observing data assets and AI workflows, with built-in asset definitions, scheduling, and monitoring.
Key Features
- Declarative asset-based programming model for defining data pipelines as Python functions
- Integrated lineage graph with built-in data quality checks and dependency tracking
- Web UI for real-time pipeline monitoring, scheduling, and asset health dashboards
- Extensive integration library covering cloud warehouses, orchestration tools, and ML frameworks
- CI/CD-friendly with local development, unit testing, staging, and production parity
- Multi-tenant orchestration engine that scales organizationally with built-in observability
Use Cases
💡 Building and maintaining ETL/ELT data pipelines across cloud data warehouses
💡 Orchestrating ML model training, evaluation, and deployment workflows
💡 Defining and tracking data assets with lineage for regulatory compliance
💡 Coordinating multi-step data processing jobs with scheduling and retries
💡 Managing complex data platform infrastructure with centralized monitoring
Categories
Quick Start
Install via `uv add dagster dagster-webserver dagster-dg-cli`. Define assets as decorated Python functions, load them into the Dagster UI, and use `dagster dev` to launch a local development server with the web interface.