Observability

Netdata is an open-source real-time observability platform with second-resolution metrics, AI health checks and zero-config deployment, widely used for monitoring AI agent services.

The cloud-native API and AI Gateway providing LLM request routing, rate limiting, load balancing and observability for AI agent applications.

PostHog is a full-stack observability platform for AI agents and self-driving products, combining analytics, session replay, feature flags and error tracking.

Open-source LLM engineering platform providing tracing, evaluations, prompt management, and dataset management with integrations for LangChain, OpenAI, Anthropic, and more.

An AI prompt optimizer that helps users write better prompts and achieve improved AI results.

SigNoz is an open-source OpenTelemetry-native observability platform combining APM, logs, metrics and alerts.

MLflow LLM / GenAI extensions. One-stop LLMOps: prompt versioning, evaluation, model registry, tracing.

MLflow is the open-source AI engineering platform for debugging, evaluating, monitoring, and optimizing AI agents and LLM applications, with model and data access management.

What are the principles we can use to build LLM-powered software that is actually good enough to put in the hands of production customers?

Test and evaluate LLM prompts, agents, and RAG pipelines. Built-in red teaming and security evaluation for reliable AI applications.

Opik is an open-source LLM observability platform providing agent tracing, evaluation testing, and prompt experiment management to help developers monitor and optimize AI agent systems.

End-to-end, code-first tutorials for building production-grade GenAI agents. From prototype to enterprise deployment.

OpenObserve is a high-performance observability platform for logs, metrics, and traces, well suited for monitoring AI agent runtimes and tool calls.

OpenAI's framework for evaluating LLMs and LLM systems, providing an open-source registry of benchmarks and tools for systematic model assessment.

Analyze coding (agent) CLI token usage and costs from local data.

DeepEval is an open-source evaluation framework for LLM applications. It provides rich evaluation metrics and tools, supporting unit testing and integration testing to help developers build reliable LLM applications.

RagaAI Catalyst is an observability, monitoring, and evaluation framework for Agent AI, supporting agent/LLM/tool tracing, multi-agent debugging, and self-hosted dashboard analytics.

Ragas is a framework for evaluating RAG (Retrieval Augmented Generation) systems. It provides various evaluation metrics including faithfulness, answer relevance, context precision, helping developers optimize RAG application performance.

OpenMetadata is a unified metadata platform for data and AI, providing data asset discovery, lineage, governance, and agent context retrieval capabilities.

A framework for few-shot evaluation of language models by EleutherAI, providing standardized evaluation pipelines supporting hundreds of benchmark tasks and widely adopted as a core LLM evaluation tool in the community.

eBPF-powered network observability for Kubernetes. Indexes L4/L7 traffic with full K8s context, queryable by AI agents via MCP and humans via dashboard.

TensorZero is an open-source LLMOps platform that unifies an LLM gateway, observability, evaluation, optimization, and A/B testing, designed for production agents.

Crucix is a personal intelligence agent that watches the world from multiple data sources and pings you when something changes, helping you stay on top of information in real time.

Weights & Biases is an experiment tracking, visualization, and collaboration platform for ML and LLM applications, covering agent training evaluation, hyperparameter management, and model registry workflows.