Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs

The study on context faithfulness in large language models (LLMs) addresses a critical issue: these models often fail to ground their responses in the provided context, leading to irrelevant outputs. To tackle this, the researchers introduce Router Lens, a method designed to identify experts within the model that specialize in context utilization. This identification is crucial as it allows for targeted optimization. Building on this, they propose Context-faithful Expert Fine-Tuning (CEFT), a lightweight approach that selectively fine-tunes these context-faithful experts. The experiments conducted demonstrate that CEFT not only matches but can also surpass the performance of full fine-tuning methods while being significantly more efficient. This advancement is vital as it enhances the reliability of LLMs in context-dependent scenarios, paving the way for more effective applications in various fields.