Normalization in Attention Dynamics

The study titled 'Normalization in Attention Dynamics' investigates the impact of various normalization schemes on token representations within deep transformers, modeling their behavior as interacting particles on a sphere. It reveals that normalization functions as a form of speed regulation, influencing clustering dynamics and representation collapse. By analyzing schemes such as Post-LN, Pre-LN, Mix-LN, Peri-LN, and nGPT, the research offers a unified framework for understanding their effects across layers. Notably, Peri-LN emerges as a particularly effective choice, enhancing the clarity of token representations. This research is significant as it provides a principled basis for comparing normalization methods, which is essential for optimizing AI models that utilize transformer architectures.