LangWatch

Active
GitHub TypeScript Apache-2.0

Description

Platform for LLM evaluations and AI agent testing, providing comprehensive tracing, evaluation, and quality monitoring to help teams build reliable AI applications.

Key Features

  • End-to-end agent simulations — run realistic scenarios against your full stack to pinpoint where agents break
  • Unified eval + observability + prompt management in one loop — trace, dataset, evaluate, optimize, re-test
  • OpenTelemetry/OTLP-native — framework-agnostic and LLM-provider-agnostic, no vendor lock-in
  • AI Gateway with virtual keys, hierarchical budgets, inline guardrails, and automatic provider fallback
  • Collaboration tools with annotations, queues, and GitHub-integrated prompt versioning linked to traces
  • Multi-framework integration supporting LangChain, LangGraph, Vercel AI SDK, CrewAI, and more

Use Cases

💡 Pre-release regression testing for LLM-powered agents in production-like scenarios
💡 Real-time production observability and monitoring of LLM calls, costs, and quality
💡 Prompt engineering optimization with version control and A/B testing
💡 Multi-provider LLM cost governance and automatic failover
💡 Team collaboration on AI quality with annotation workflows and shared evaluation datasets

Tags

observability evaluation llm-testing tracing typescript

Categories

📊 Observability

Quick Start

1. Create a free account at app.langwatch.ai or run `npx @langwatch/server` locally
2. Create a project and get your API key
3. Install the SDK: `pip install langwatch` (Python) or `npm install langwatch` (TypeScript)
4. Instrument your LLM calls with `@langwatch.trace()` decorator
5. Run your first agent simulation via the Scenario feature in the dashboard
Visit GitHub Visit Website

Related Projects