Giskard

Active
GitHub Python Apache-2.0

Description

An open-source evaluation and testing library for LLM agents providing automated model scanning, bias detection, performance benchmarking, and compliance checks.

Key Features

  • Scenario API for creating evaluations that test non-deterministic LLM outputs
  • Built-in checks including Groundedness, Conformity, and LLM-as-judge assessments
  • Red-teaming vulnerability scanner generating adversarial test suites across OWASP LLM Top-10 categories
  • Multi-turn conversation testing for full agent dialog evaluation
  • Modular v3 architecture with focused packages: giskard-checks, giskard-scan, and planned giskard-rag
  • Async-first design with lightweight dependencies for efficient CI/CD integration

Use Cases

💡 Catch regressions after model or prompt changes with automated eval suites
💡 Validate RAG quality by checking if answers are grounded in retrieved context
💡 Red-team AI agents to identify prompt injection, data leakage, and bias vulnerabilities
💡 Generate evaluation datasets for RAG applications from knowledge bases
💡 Enforce content safety policies with conformity checks on agent outputs

Tags

evaluation testing llm-safety bias-detection compliance python

Categories

📊 Observability 🛡️ Security & Guardrails

Quick Start

pip install giskard-checks && pip install giskard-scan
Visit GitHub Visit Website View Docs

Related Projects

Related Articles

Agent 评估LLM 评测自动化测试

Agent Evaluation and Testing: From Vibe Checks to End-to-End Pipelines

Most teams evaluate agents by checking a few examples. Real evaluation needs layered metrics, non-rotting datasets, and judges that push back. This article provides runnable code patterns and a practical decision framework.

Read more →