Download
UQLM is a Python library developed to detect hallucinations and quantify uncertainty in the outputs of large language models. The system implements a variety of uncertainty quantification techniques that assign confidence scores to model responses. These scores help developers determine how likely a generated answer is to contain errors or fabricated information. The library includes both black-box and white-box approaches to uncertainty estimation. Black-box methods evaluate model outputs through multiple generations or comparative analysis, while white-box methods rely on token probabilities produced during inference. UQLM also supports ensemble strategies and model-as-judge approaches for evaluating responses. By combining multiple uncertainty metrics, the system provides more reliable indicators of when language model outputs may be unreliable.
Features
- Python library for hallucination detection in language models
- Confidence scoring system for evaluating LLM outputs
- Support for black-box and white-box uncertainty quantification methods
- Techniques including semantic entropy and semantic density metrics
- Ensemble evaluation strategies using multiple model generations
- Tools for calibrating uncertainty scores across different models
Project Samples
