TruLens

Active

Description

TruLens is an open-source tool for evaluating and tracking LLM apps. It provides specialized evaluation for RAG applications including context relevance, groundedness, and answer relevance.

Key Features

OpenTelemetry-based tracing with structured OTEL spans
7 agentic evaluators: consistency, efficiency, plan adherence, quality, tool selection, tool calling, tool quality
Batch and inline evaluation with configurable workers
MCP tool call instrumentation for latency and output tracking
RAG Triad evaluation: context relevance, groundedness, answer relevance
Multi-provider support: OpenAI, Anthropic, Google, Bedrock, Snowflake, HuggingFace

Use Cases

💡 Systematically evaluating LLM application quality during development

💡 Monitoring RAG pipeline performance with the RAG Triad metrics

💡 Instrumenting agentic workflows for failure mode detection

💡 Running batch evaluations on datasets to compare model versions

💡 Integrating observability into existing OpenTelemetry infrastructure

Strengths & Limitations

✅ Strengths

• Actively maintained, recent updates
• High community interest (3.5k stars)
• Permissive open-source license (MIT)
• Established track record (5 years in production)

Quick Start

pip install trulens-core, then pip install trulens-providers-openai (or your provider). Import instrument decorator, wrap your RAG functions with @instrument, define feedback functions, and run evaluations via the dashboard or Python API.

Visit GitHub Visit Website View Docs

Related Projects

Ragas

15.3k · Python

Stale B

Ragas is a framework for evaluating RAG (Retrieval Augmented Generation) systems. It provides various evaluation metrics including faithfulness, answer relevance, context precision, helping developers optimize RAG application performance.

ragevaluationllm +1

· Objective evaluation metrics using both LLM-based and traditional approaches for precise LLM app assessment
· Automatic test data generation covering a wide range of scenarios for RAG systems
· Seamless integrations with LangChain, major observability tools, and popular LLM frameworks

View full details → View on GitHub

SwanLab

4.2k · Python

Active A+

An open-source, modern-design AI training tracking and visualization tool. Supports PyTorch, Transformers and more. Monitor and evaluate AI agent training processes.

pythonobservabilityevaluation +2

· Seamless integration with 50+ mainstream frameworks: native support for PyTorch, Transformers, HuggingFace Accelerate, PaddleNLP, NVIDIA NeMo RL and more, with two lines of code to connect training pipelines
· Rich visualization system: supports line charts, scalar plots, PR curves, ROC curves, confusion matrices, 3D point clouds, molecular structures, ECharts custom charts and 20+ chart types
· Multi-dimensional hardware monitoring: real-time monitoring of GPU (NVIDIA/AMD ROCm/Hygon DCU/Cambricon MLU/Moore Threads/Muxi/Iluvatar/Kunlun), disk utilization, network traffic and other hardware metrics

View on GitHub

OpenInference

1.1k · Python

Active A

OpenTelemetry instrumentation for AI observability, providing standardized tracing, metrics collection, and span definitions for LLM inference processes to help developers monitor and debug AI agent systems.

observabilitypythonllm +2

· OpenTelemetry-based instrumentation spec for tracing LLM inference and application context
· Instrumentation libraries for 15+ ML frameworks: OpenAI, LlamaIndex, LangChain, DSPy, CrewAI, Agno, and more
· Semantic conventions for standardized span attributes across LLM applications

View on GitHub

DeepEval

17.6k · Python

Active A

DeepEval is an open-source evaluation framework for LLM applications. It provides rich evaluation metrics and tools, supporting unit testing and integration testing to help developers build reliable LLM applications.

llmevaluationtesting +1

· Pytest-compatible LLM evaluation framework with ready-to-use metrics for agents, RAG, and chatbots
· Agentic metrics including Task Completion, Tool Correctness, Step Efficiency, and Plan Adherence
· RAG metrics covering Answer Relevancy, Faithfulness, Contextual Recall/Precision/Relevancy, and RAGAS

View full details → View on GitHub

RAG评估Ragas

RAG System Evaluation in Practice: Building High-Quality RAG Apps with Ragas and DeepEval

Learn how to evaluate RAG systems using Ragas and DeepEval, including measuring key metrics like faithfulness, answer relevance, and context precision.