Semantic Volume: Quantifying and Detecting both External and Internal Uncertainty in LLMs

The introduction of Semantic Volume marks a significant advancement in understanding and managing uncertainty in large language models (LLMs). Traditional methods primarily focused on internal uncertainty, which arises from the model's limitations, while Semantic Volume also accounts for external uncertainty stemming from ambiguous user queries. By perturbing queries and responses and analyzing their semantic embeddings, this novel approach provides a robust, unsupervised method for detecting uncertainty. Extensive experiments have demonstrated that Semantic Volume consistently outperforms existing baselines in both internal and external uncertainty detection tasks. This not only enhances the reliability of LLMs but also links the measure to differential entropy, offering a unified framework that extends previous uncertainty measures. As LLMs continue to be integrated into various applications, the ability to quantify and manage uncertainty is crucial for ensuring accurate and trustwor…