AgentOps

Normal

Description

AgentOps is an observability platform for AI agents, providing monitoring, debugging, and evaluation to help developers optimize agent performance.

Key Features

AI Agent observability platform with session replay and step-by-step execution graphs
LLM cost management: track spending with foundation model providers
Native integrations with CrewAI, AG2, LangGraph and other major agent frameworks
Enable agent monitoring with just two lines of code
Self-hostable deployment, MIT open-source license

Use Cases

💡 Agent development debugging: visualize execution paths to quickly identify issues

💡 Production monitoring: real-time tracking of agent performance and LLM calls

💡 Cost optimization: analyze LLM usage patterns to reduce API spending

💡 Agent evaluation: analyze agent behavior quality through session replays

Strengths & Limitations

✅ Strengths

• High community interest (5.8k stars)
• Permissive open-source license (MIT)
• Established track record (3 years in production)

Quick Start

pip install agentops && call agentops.init('<API_KEY>') in code and agentops.end_session('Success') at end

Visit GitHub Visit Website

Related Projects

Claude Code Hooks Multi-Agent Observability

1.5k · Python

Stale C

A real-time observability toolkit for Claude Code agents that tracks hook events to monitor multi-agent coding workflows.

observabilityclaude-codehooks +2

· Real-time event streaming via WebSocket from Claude Code hook events to a Vue dashboard
· Supports all 12 Claude Code hook event types (PreToolUse, PostToolUse, SessionStart, etc.)
· Multi-agent session tracking with colored agent identification and event filtering

View on GitHub

Evidently

7.8k · Jupyter Notebook

Active A

Evidently is an open-source ML and LLM observability framework with 100+ metrics for evaluating, testing, and monitoring any AI-powered system or data pipeline.

observabilityevaluationmonitoring +2

· 100+ built-in evaluation metrics for data drift, ML models, and LLM outputs
· LLM-as-a-judge evaluators for generative AI quality assessment
· Reports and Test Suites with pass/fail conditions for CI/CD integration

View full details → View on GitHub

OpenLLMetry

7.4k · Python

Active A

OpenLLMetry is an open-source observability tool for LLM applications based on OpenTelemetry, providing tracing, metrics, and monitoring capabilities.

observabilityopentelemetryllm +2

· OpenTelemetry-Based Observability — built on OpenTelemetry extensions for complete LLM application tracing and monitoring
· 20+ LLM Provider Instrumentation — supports OpenAI, Anthropic, Cohere, Google Gemini, Mistral, Ollama, HuggingFace, and more
· Vector DB Tracing — instrument Chroma, Pinecone, Qdrant, Weaviate, Milvus, LanceDB, and Marqo

View full details → View on GitHub

LLMTracer

2.6k · Python

Stale B

LLMTracer is a tracing tool for agent and LLM applications, helping developers capture call paths, tool execution, and state transitions for debugging and incident analysis.

observabilitytracingdebugging +1

· Tracing tool for Agent and LLM applications
· Records complete call paths and tool execution
· Real-time monitoring and recording of state transitions

View on GitHub

AI Agent可观测性链路追踪

Building Agent Observability: From Distributed Tracing to Automated Evaluation

A systematic guide to the three pillars of agent observability — distributed tracing, metrics monitoring, and automated evaluation — for building production-grade agent monitoring.