arXiv:2311.02960v5 Announce Type: replace 
Abstract: Over the past decade, deep learning has proven to be a highly effective tool for learning meaningful features from raw data. However, it remains an open question how deep networks perform hierarchical feature learning across layers. In this work, we attempt to unveil this mystery by investigating the structures of intermediate features. Motivated by our empirical findings that linear layers mimic the roles of deep layers in nonlinear networks for feature learning, we explore how deep linear networks transform input data into output by investigating the output (i.e., features) of each layer after training in the context of multi-class classification problems. Toward this goal, we first define metrics to measure within-class compression and between-class discrimination of intermediate features, respectively. Through theoretical analysis of these two metrics, we show that the evolution of features follows a simple and quantitative pattern from shallow to deep layers when the input data is nearly orthogonal and the network weights are minimum-norm, balanced, and approximate low-rank: Each layer of the linear network progressively compresses within-class features at a geometric rate and discriminates between-class features at a linear rate with respect to the number of layers that data have passed through. To the best of our knowledge, this is the first quantitative characterization of feature evolution in hierarchical representations of deep linear networks. Empirically, our extensive experiments not only validate our theoretical results numerically but also reveal a similar pattern in deep nonlinear networks which aligns well with recent empirical studies. Moreover, we demonstrate the practical implications of our results in transfer learning. Our code is available at https://github.com/Heimine/PNC_DLN.

دراسة حديثة نُشرت على arXiv تستكشف التعلم العميق للتمثيلات من خلال ضغط الميزات والتفريق بين الطبقات، كاشفةً كيف يمكن للشبكات الخطية العميقة أن تتعلم بفعالية الميزات الهرمية من البيانات الخام. تركز الدراسة على تحويل البيانات المدخلة إلى ميزات مخرجة عبر الطبقات في مهام التصنيف متعددة الفئات، مع وضع مقاييس لقياس ضغط الميزات داخل الفئة والتفريق بين الفئات.

Un estudio reciente publicado en arXiv investiga el aprendizaje profundo de representaciones a través de la compresión y discriminación de características por capas, revelando cómo las redes lineales profundas pueden aprender de manera efectiva características jerárquicas a partir de datos en bruto. La investigación se centra en la transformación de datos de entrada en características de salida a través de las capas en tareas de clasificación multiclase, estableciendo métricas para medir la compresión dentro de la clase y la discriminación entre clases.

Une étude récente publiée sur arXiv examine l'apprentissage de représentations profondes à travers la compression et la discrimination des caractéristiques par couche, révélant comment les réseaux linéaires profonds peuvent efficacement apprendre des caractéristiques hiérarchiques à partir de données brutes. La recherche se concentre sur la transformation des données d'entrée en caractéristiques de sortie à travers les couches dans des tâches de classification multi-classes, établissant des métriques pour mesurer la compression intra-classe et la discrimination inter-classe.

A recent study published on arXiv investigates deep representation learning through layerwise feature compression and discrimination, revealing how deep linear networks can effectively learn hierarchical features from raw data. The research focuses on the transformation of input data into output features across layers in multi-class classification tasks, establishing metrics for measuring within-class compression and between-class discrimination.