UQLM
ActiveDescription
CVS Health's open-source uncertainty quantification library for language models, providing UQ-based hallucination detection with confidence scoring and mitigation tools to identify and reduce unreliable LLM outputs.
Key Features
- Five categories of uncertainty quantification scorers: Black-Box (consistency), White-Box (token probability), LLM-as-a-Judge, Ensemble, and Long-Text
- Black-Box scorers measure response consistency through multiple generations and comparisons, compatible with any LLM, off-the-shelf
- White-Box scorers leverage token probabilities for uncertainty estimation, single-generation scoring with minimal latency and cost
- Ensemble scorers flexibly combine multiple methods, beginner-friendly with tuning options for advanced users
- Supports hallucination mitigation — use_best=True automatically selects the response with lowest uncertainty
- Published in JMLR and TMLR with comprehensive documentation and example notebooks
Use Cases
💡 Detect hallucinations in medical QA systems to ensure information provided to doctors is reliable
💡 Add confidence scores to customer service bot responses, auto-transfer low-confidence answers to humans
💡 Identify high-uncertainty content in financial report generation, flagging sections needing manual review
💡 Evaluate reliability of different LLMs on specific tasks to assist model selection decisions
💡 Build safety guardrails that automatically filter low-confidence outputs in agent workflows
Categories
Quick Start
Install: pip install uqlm. Black-Box example: from uqlm import BlackBoxUQ, create ChatOpenAI LLM instance, initialize BlackBoxUQ(llm=llm, scorers=["semantic_negentropy"], use_best=True), call await bbuq.generate_and_score(prompts=prompts, num_responses=5) for results with confidence scores.