MLflow

Active
GitHub Python Apache-2.0

Description

MLflow is the open-source AI engineering platform for debugging, evaluating, monitoring, and optimizing AI agents and LLM applications, with model and data access management.

Key Features

  • LLM Observability — Capture complete traces of Agent and LLM applications via OpenTelemetry
  • Systematic Evaluation — 50+ built-in metrics and LLM Judges for automatic output quality assessment
  • Prompt Registry & Optimization — Version-controlled prompts with auto-optimization algorithms
  • AI Gateway — Unified API gateway managing multiple LLM providers with rate limiting, fallbacks, and A/B testing
  • Experiment Tracking — Record model parameters, metrics, and evaluation results across experiments
  • Model Registry & Deployment — Collaborative ML model lifecycle management with Docker/K8s/AWS deployment

Use Cases

💡 Monitoring latency, cost, and quality metrics of LLM applications in production
💡 Systematically evaluating Prompt version effectiveness and detecting regressions before deployment
💡 Managing access to multiple LLM providers (OpenAI, Anthropic, etc.) through a unified AI gateway
💡 Tracking ML experiment iterations and comparing effects of different model parameters and training data
💡 Deploying trained models to cloud or Kubernetes clusters for inference serving

Tags

mlflow llmops evaluation observability python

Categories

📊 Observability

Quick Start

pip install mlflow
uvx mlflow server

# In Python code:
import mlflow
mlflow.set_tracking_uri("http://localhost:5000")
mlflow.openai.autolog()

from openai import OpenAI
client = OpenAI()
response = client.responses.create(
    model="gpt-4o-mini",
    input="Hello!",
)
print(response)
Visit GitHub Visit Website

Related Projects