Prefect

Active
GitHub Python Apache-2.0

Description

A workflow orchestration framework for building resilient data pipelines and AI workflows in Python, with task scheduling, state management, and failure recovery from local to distributed deployments.

Key Features

  • Flow and task decorators — declare Python functions as orchestrable workflows with @flow and @task
  • Auto-retry and error handling — built-in task-level retries, caching, parametric execution
  • Scheduled deployments — cron expression and event-triggered automated scheduling
  • Visual monitoring dashboard — Prefect Server UI shows real-time workflow status, logs, dependencies
  • Distributed execution — elastic deployment from local to Kubernetes/Docker, parallel task support
  • Event-driven automation — trigger downstream actions based on workflow events for reactive pipelines

Use Cases

💡 Building resilient ETL data pipelines (extract, transform, load)
💡 Orchestrating AI/ML training and inference workflows
💡 Automating scheduled data reports and monitoring tasks
💡 Managing multi-step data science experiment pipelines
💡 Building scalable data workflows from local dev to production

Tags

workflow automation python observability framework

Categories

🔄 Workflow

Quick Start

pip install -U prefect

from prefect import flow, task
import httpx

@task(log_prints=True)
def get_stars(repo: str):
    url = f"https://api.github.com/repos/{repo}"
    count = httpx.get(url).json()["stargazers_count"]
    print(f"{repo} has {count} stars!")

@flow(name="GitHub Stars")
def github_stars(repos: list[str]):
    for repo in repos:
        get_stars(repo)

if __name__ == "__main__":
    github_stars(["PrefectHQ/prefect"])
Visit GitHub Visit Website

Related Projects