Know Your Limits: Entropy Estimation Modeling for Compression and Generalization

The article discusses the constraints of language prediction imposed by informational entropy, highlighting limits on the accuracy of language models and a lower bound on language compression. Current efficient language compression algorithms are causal large language models, but estimating language entropy accurately with these models is computationally infeasible. The authors introduce encoder-augmented causal decoder model architectures that demonstrate superior training efficiency and achieve higher compression than causal transformers, even on modest hardware. They show that entropy estimates can be obtained on a per-token basis and that models trained to approach the entropy of their training data generalize better than those trained solely to minimize loss beyond this point.