arXiv:2605.27734v1 Announce Type: new 
Abstract: Generative models, from diffusion models to large language models, achieve remarkable performance but at a cost in training data orders of magnitude larger than what biological learners require. An alternative paradigm has emerged in which networks are trained to predict their \emph{own} latent representations of related views or masked regions, as in data2vec and JEPA -- an idea related to predictive-coding accounts of the cortex. Despite strong empirical results, the theoretical understanding of these methods remains limited. Central questions include: by how much does latent prediction actually improve data efficiency? Is there a benefit to stacking such methods into multi-scale hierarchies? We answer both using as data a tractable probabilistic context-free grammar that captures the compositional structure of natural language and images. Such a grammar generates strings of visible tokens by recursively applying production rules along a tree of hidden symbols of depth $L$. For such data, supervised or token-level SSL require a number of samples \emph{exponential} in $L$ to recover the latent tree; we prove that latent prediction achieves this with a number of samples \emph{constant} in $L$, up to logarithmic factors. We confirm this bound with (i) a hierarchical clustering algorithm, (ii) an end-to-end neural network whose predictor-clusterer modules predict their own latents at each level via gradient descent, and (iii) the first sample-complexity analysis of data2vec, which we show implicitly performs hierarchical latent prediction. This suggests that explicit stacking such as H-JEPA is largely redundant.

دراسة حديثة نُشرت على arXiv تناقش نموذجًا جديدًا في النماذج التوليدية، حيث يتم التركيز على تدريب الشبكات لتوقع تمثيلاتها الكامنة الخاصة بدلاً من الاعتماد على بيانات تدريب ضخمة. يرتبط هذا النهج بنظريات الترميز التنبؤي، ويهدف إلى تحسين كفاءة البيانات في تطبيقات التعلم الآلي.

Un estudio reciente publicado en arXiv discute un nuevo paradigma en los modelos generativos, enfatizando el entrenamiento de redes para predecir sus propias representaciones latentes en lugar de depender de datos de entrenamiento extensos. Este enfoque, vinculado a teorías de codificación predictiva, busca mejorar la eficiencia de los datos en aplicaciones de aprendizaje automático.

Une étude récente publiée sur arXiv aborde un nouveau paradigme dans les modèles génératifs, mettant l'accent sur l'entraînement de réseaux pour prédire leurs propres représentations latentes plutôt que de s'appuyer sur des données d'entraînement massives. Cette approche, liée aux théories du codage prédictif, vise à améliorer l'efficacité des données dans les applications d'apprentissage automatique.

A recent study published on arXiv discusses a new paradigm in generative models, emphasizing the training of networks to predict their own latent representations rather than relying on extensive training data. This approach, linked to predictive coding theories, aims to enhance data efficiency in machine learning applications.