Relational Rank Geometry in Transformers: Detecting and Steering Hidden-State Relation Frames

A new study published on arXiv explores the rank-indexed geometry of relations among token tuples in Transformer hidden states, focusing on the Llama-family models. The research utilizes Plucker sign entropy to analyze the orientation signatures of true relation tuples compared to scrambled ones, revealing consistent orientation-sign patterns across various model checkpoints.

This development enhances the understanding of how Transformers process relational information, potentially leading to improved model designs and applications in artificial intelligence, particularly in tasks requiring complex relational reasoning.