On the Entropy Calibration of Language Models

This study addresses the k-means clustering problem for a set of segments in Euclidean space, focusing on finding k centers that minimize the total distance from each point along a segment to a center. The research introduces the first coreset construction that effectively handles arbitrary input segments, allowing for efficient computation in various contexts. The findings have implications for applications such as real-time video tracking and clustering in high-dimensional spaces.

The paper titled 'FreDN: Spectral Disentanglement for Time Series Forecasting via Learnable Frequency Decomposition' presents a novel approach to time series forecasting. It addresses the challenges posed by spectral entanglement in non-stationary time series, which complicates the analysis of trends and periodicities. The proposed Frequency Decomposition Network (FreDN) includes a learnable Frequency Disentangler module that effectively separates these components in the frequency domain. Additionally, a ReIm Block is introduced to simplify complex-valued learning, enhancing computational effi…

Deep neural networks have shown remarkable capabilities in image classification but are susceptible to adversarial examples. Traditional adversarial training improves robustness but often overlooks inter-class feature overlap, which contributes to vulnerability. This study introduces a new adversarial training framework that reduces inter-class proximity by projecting out dependencies from both adversarial and clean samples in the feature space. The method enhances feature separability and theoretically lowers the Lipschitz constant of neural networks, improving generalization.

The article presents a novel approach to deep hashing called Center-Reassigned Hashing (CRH), which enhances traditional methods by dynamically reassigning hash centers from a preset codebook. This end-to-end framework optimizes the hash function while avoiding the inefficiencies of local similarity optimization and the complexities of two-stage methods. By adapting hash centers to data distribution without explicit optimization phases, CRH aims to improve performance and streamline the learning process in semantic hashing.

Recent advancements in large language model (LLM) pruning have demonstrated state-of-the-art compression results without the need for post-training or retraining, while still maintaining high predictive performance. However, prior research predominantly focused on English text for calibration, overlooking the multilingual capabilities of modern LLMs. This paper presents a comprehensive empirical study analyzing the effects of different calibration languages on pruning multilingual models, revealing significant insights into performance and internal representation changes.

This paper presents a theoretical analysis of data scaling laws in linear regression, particularly focusing on the effects of training on limited datasets over multiple epochs. It investigates how much larger a dataset must be to achieve the same performance as training on a smaller dataset for multiple epochs. The study introduces the concept of the effective reuse rate, which quantifies the necessary dataset growth for one-pass training to match the test loss of multi-epoch training.

The paper titled 'QA-Noun: Representing Nominal Semantics via Natural Language Question-Answer Pairs' introduces a new framework called QA-Noun, aimed at capturing noun-centered semantic relations. This framework utilizes nine question templates to address both explicit and implicit roles of nouns, producing interpretable question-answer pairs that enhance existing verbal QA-SRL methods. The authors provide a dataset of over 2,000 annotated noun mentions and a trained model that integrates with QA-SRL, achieving significant coverage of noun arguments and revealing additional contextual relatio…

VIR-Bench is a new benchmark designed to evaluate the geospatial and temporal understanding of multimodal large language models (MLLMs) through the reconstruction of travel video itineraries. It consists of 200 travel videos, addressing a gap in current benchmarks that primarily focus on indoor or short-range outdoor activities. The study highlights the challenges faced by state-of-the-art MLLMs in handling extended geospatial-temporal trajectories, which are crucial for real-world applications like AI planning and navigation.