Selective Risk Certification for LLM Outputs via Information-Lift Statistics: PAC-Bayes, Robustness, and Skeleton Design

Large language models (LLMs) have demonstrated remarkable abilities in natural language processing, image recognition, and multimodal tasks, positioning them as pivotal in the technological landscape. This article reviews the foundational technologies behind LLMs, such as prompt engineering and fine-tuning, while also exploring the historical evolution of the Science of Science (SciSci). It anticipates future applications of LLMs in scientometrics and discusses the potential of AI-driven models for scientific evaluation.

The COMPASS (Context-Modulated PID Attention Steering System) is introduced as a framework designed to mitigate hallucinations in large language models (LLMs). It incorporates a feedback loop within the decoding process, utilizing the Context Reliance Score (CRS) to assess how attention heads utilize contextual evidence. This system aims to ensure factual consistency in generated outputs without the need for retraining or multiple decoding passes.

Large language models (LLMs) exhibit label length bias, where labels of varying lengths are treated inconsistently despite normalization efforts. This paper introduces normalized contextual calibration (NCC), a method that normalizes predictions at the full-label level, effectively addressing this bias. NCC demonstrates statistically significant improvements across multiple datasets and models, achieving up to 10% gains in F1 scores. Additionally, it extends bias mitigation to tasks like multiple-choice question answering, showing reduced sensitivity to few-shot example selection.