arXiv:2405.04370v5 Announce Type: replace 
Abstract: Understanding how humans would behave during hand-object interaction is vital for applications in service robot manipulation and extended reality. To achieve this, some recent works have been proposed to simultaneously forecast hand trajectories and object affordances on human egocentric videos. The joint prediction serves as a comprehensive representation of future hand-object interactions in 2D space, indicating potential human motion and motivation. However, the existing approaches mostly adopt the autoregressive paradigm for unidirectional prediction, which lacks mutual constraints within the holistic future sequence, and accumulates errors along the time axis. Meanwhile, these works basically overlook the effect of camera egomotion on first-person view predictions. To address these limitations, we propose a novel diffusion-based interaction prediction method, namely Diff-IP2D, to forecast future hand trajectories and object affordances concurrently in an iterative non-autoregressive manner. We transform the sequential 2D images into latent feature space and design a denoising diffusion model to predict future latent interaction features conditioned on past ones. Motion features are further integrated into the conditional denoising process to enable Diff-IP2D aware of the camera wearer's dynamics for more accurate interaction prediction. Extensive experiments demonstrate that our method significantly outperforms the state-of-the-art baselines on both the off-the-shelf metrics and our newly proposed evaluation protocol. This highlights the efficacy of leveraging a generative paradigm for 2D hand-object interaction prediction. The code of Diff-IP2D is released as open source at https://github.com/IRMVLab/Diff-IP2D.

تناقش الورقة المعنونة 'Diff-IP2D: توقع تفاعل اليد-الشيء القائم على الانتشار في مقاطع الفيديو الذاتية' طريقة جديدة لتوقع تفاعلات اليد-الشيء في مقاطع الفيديو الذاتية. تهدف هذه الطريقة، المسماة Diff-IP2D، إلى تحسين دقة التنبؤ بمسارات اليد وخصائص الأشياء، مع معالجة القيود التي تواجه النماذج التلقائية الحالية. تبرز الأبحاث أهمية فهم سلوك البشر في تفاعلات اليد-الشيء لتطبيقات في تشغيل الروبوتات الخدمية والواقع الممتد.

El artículo titulado 'Diff-IP2D: Predicción de Interacción Mano-Objeto Basada en Difusión en Videos Egocéntricos' presenta un nuevo método para predecir interacciones mano-objeto en videos egocéntricos. Este método, denominado Diff-IP2D, busca mejorar la precisión en la previsión de trayectorias de mano y affordances de objetos, abordando las limitaciones de los modelos autorregresivos existentes. La investigación destaca la importancia de entender el comportamiento humano en las interacciones mano-objeto para aplicaciones en la manipulación de robots de servicio y realidad extendida.

L'article intitulé 'Diff-IP2D: Prédiction d'interaction main-objet basée sur la diffusion dans des vidéos égocentriques' présente une nouvelle méthode pour prédire les interactions main-objet dans des vidéos égocentriques. Cette méthode, nommée Diff-IP2D, vise à améliorer la précision de la prévision des trajectoires de main et des affordances d'objet, en abordant les limitations des modèles autoregressifs existants. La recherche souligne l'importance de comprendre le comportement humain dans les interactions main-objet pour des applications dans la manipulation de robots de service et la réal…

The paper titled 'Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videos' discusses a new method for predicting hand-object interactions in egocentric videos. This method, named Diff-IP2D, aims to improve the accuracy of forecasting hand trajectories and object affordances, addressing limitations of existing autoregressive models. The research highlights the importance of understanding human behavior in hand-object interactions for applications in service robots and extended reality.

Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videos

arXiv:1906.07172v5 Announce Type: replace-cross 
Abstract: Equivariant neural networks are a class of neural networks designed to preserve symmetries inherent in the data. In this paper, we introduce a general method for modifying a neural network to enforce equivariance, a process we refer to as equivarification. We further show that group convolutional neural networks (G-CNNs) arise as a special case of our framework.

Equivariant neural networks and equivarification

arXiv:2311.13745v4 Announce Type: replace-cross 
Abstract: Diffusion models have become the most popular approach to deep generative modeling of images, largely due to their empirical performance and reliability. From a theoretical standpoint, a number of recent works have studied the iteration complexity of sampling, assuming access to an accurate diffusion model. In this work, we focus on understanding the sample complexity of training such a model; how many samples are needed to learn an accurate diffusion model using a sufficiently expressive neural network? Prior work showed bounds polynomial in the dimension, desired Total Variation error, and Wasserstein error. We show an exponential improvement in the dependence on Wasserstein error and depth, along with improved dependencies on other relevant parameters.

Improved Sample Complexity Bounds for Diffusion Model Training

arXiv:2410.22995v2 Announce Type: replace-cross 
Abstract: A hallmark of advanced artificial intelligence is the capacity to progress from passive visual perception to the strategic modification of visual information to facilitate complex reasoning. This advanced capability, however, remains critically underdeveloped in current Large Multi-modal Models (LMMs). The deficiency is often masked by evaluation metrics that prioritize final-answer accuracy, creating an illusion of competence where genuine reasoning is absent. Using the domain of geometric problem-solving as a precise instrument, we probe this issue through tasks that require constructing visual aids. To this end, we introduce \textbf{VisAidMath}, a challenging benchmark, and our novel Three-Layered Funnel Evaluation Framework. This framework moves beyond simple accuracy (ACCU) to scrutinize the generation of valid visual aids (PVA) and the soundness of subsequent reasoning steps (SPRS). Our extensive experiments on state-of-the-art models, including Doubao-Seed-1.6 and o4, reveal a profound ``Reasoning Illusion''. We observe that high surface-level accuracy conceals a catastrophic failure in the models' ability to produce valid visual aids or to reason from them. Our findings expose a fundamental schism between visual perception and logical deduction in modern LMMs. We host an evaluation platform at CodaBench for testing publicly. Homepage: https://nlp2ct.github.io/VisAidMathHomepage/ Evaluation: https://www.codabench.org/competitions/7634/

VisAidMath: Benchmarking Visual-Aided Mathematical Reasoning

arXiv:2508.01249v3 Announce Type: replace-cross 
Abstract: Large Language Model (LLM) agents offer a powerful new paradigm for solving various problems by combining natural language reasoning with the execution of external tools. However, their dynamic and non-transparent behavior introduces critical security risks, particularly in the presence of prompt injection attacks. In this work, we propose a novel insight that treats the agent runtime traces as structured programs with analyzable semantics. Thus, we present AgentArmor, a program analysis framework that converts agent traces into graph intermediate representation-based structured program dependency representations (e.g., CFG, DFG, and PDG) and enforces security policies via a type system. AgentArmor consists of three key components: (1) a graph constructor that reconstructs the agent's runtime traces as graph-based intermediate representations with control and data flow described within; (2) a property registry that attaches security-relevant metadata of interacted tools \& data, and (3) a type system that performs static inference and checking over the intermediate representation. By representing agent behavior as structured programs, AgentArmor enables program analysis for sensitive data flow, trust boundaries, and policy violations. We evaluate AgentArmor on the AgentDojo benchmark, the results show that AgentArmor can reduce the ASR to 3\%, with the utility drop only 1\%.

AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injection

arXiv:2511.14691v1 Announce Type: cross 
Abstract: Attention is the brain's ability to selectively focus on a few specific aspects while ignoring irrelevant ones. This biological principle inspired the attention mechanism in modern Transformers. Transformers now underpin large language models (LLMs) such as GPT, but at the cost of massive training and inference energy, leading to a large carbon footprint. While brain attention emerges from neural circuits, Transformer attention relies on dot-product similarity to weight elements in the input sequence. Neuromorphic computing, especially spiking neural networks (SNNs), offers a brain-inspired path to energy-efficient intelligence. Despite recent work on attention-based spiking Transformers, the core attention layer remains non-neuromorphic. Current spiking attention (i) relies on dot-product or element-wise similarity suited to floating-point operations, not event-driven spikes; (ii) keeps attention matrices that suffer from the von Neumann bottleneck, limiting in-memory computing; and (iii) still diverges from brain-like computation. To address these issues, we propose the Spiking STDP Transformer (S$^{2}$TDPT), a neuromorphic Transformer that implements self-attention through spike-timing-dependent plasticity (STDP), embedding query--key correlations in synaptic weights. STDP, a core mechanism of memory and learning in the brain and widely studied in neuromorphic devices, naturally enables in-memory computing and supports non-von Neumann hardware. On CIFAR-10 and CIFAR-100, our model achieves 94.35\% and 78.08\% accuracy with only four timesteps and 0.49 mJ on CIFAR-100, an 88.47\% energy reduction compared to a standard ANN Transformer. Grad-CAM shows that the model attends to semantically relevant regions, enhancing interpretability. Overall, S$^{2}$TDPT illustrates how biologically inspired attention can yield energy-efficient, hardware-friendly, and explainable neuromorphic models.

Attention via Synaptic Plasticity is All You Need: A Biologically Inspired Spiking Neuromorphic Transformer

arXiv:2511.08955v2 Announce Type: replace-cross 
Abstract: Simulating microstructure evolution (MicroEvo) is vital for materials design but demands high numerical accuracy, efficiency, and physical fidelity. Although recent studies on deep learning (DL) offer a promising alternative to traditional solvers, the field lacks standardized benchmarks. Existing studies are flawed due to a lack of comparing specialized MicroEvo DL models with state-of-the-art spatio-temporal architectures, an overemphasis on numerical accuracy over physical fidelity, and a failure to analyze error propagation over time. To address these gaps, we introduce MicroEvoEval, the first comprehensive benchmark for image-based microstructure evolution prediction. We evaluate 14 models, encompassing both domain-specific and general-purpose architectures, across four representative MicroEvo tasks with datasets specifically structured for both short- and long-term assessment. Our multi-faceted evaluation framework goes beyond numerical accuracy and computational cost, incorporating a curated set of structure-preserving metrics to assess physical fidelity. Our extensive evaluations yield several key insights. Notably, we find that modern architectures (e.g., VMamba), not only achieve superior long-term stability and physical fidelity but also operate with an order-of-magnitude greater computational efficiency. The results highlight the necessity of holistic evaluation and identify these modern architectures as a highly promising direction for developing efficient and reliable surrogate models in data-driven materials science.

MicroEvoEval: A Systematic Evaluation Framework for Image-Based Microstructure Evolution Prediction

arXiv:2409.02638v2 Announce Type: replace 
Abstract: Understanding human intentions and actions through egocentric videos is important on the path to embodied artificial intelligence. As a branch of egocentric vision techniques, hand trajectory prediction plays a vital role in comprehending human motion patterns, benefiting downstream tasks in extended reality and robot manipulation. However, capturing high-level human intentions consistent with reasonable temporal causality is challenging when only egocentric videos are available. This difficulty is exacerbated under camera egomotion interference and the absence of affordance labels to explicitly guide the optimization of hand waypoint distribution. In this work, we propose a novel hand trajectory prediction method dubbed MADiff, which forecasts future hand waypoints with diffusion models. The devised denoising operation in the latent space is achieved by our proposed motion-aware Mamba, where the camera wearer's egomotion is integrated to achieve motion-driven selective scan (MDSS). To discern the relationship between hands and scenarios without explicit affordance supervision, we leverage a foundation model that fuses visual and language features to capture high-level semantics from video clips. Comprehensive experiments conducted on five public datasets with the existing and our proposed new evaluation metrics demonstrate that MADiff predicts comparably reasonable hand trajectories compared to the state-of-the-art baselines, and achieves real-time performance. We will release our code and pretrained models of MADiff at the project page: https://irmvlab.github.io/madiff.github.io.

يقدم المقال MADiff، وهي طريقة جديدة للتنبؤ بمسارات اليدين في مقاطع الفيديو الذاتية باستخدام نماذج الانتشار. تهدف هذه الطريقة إلى تحسين فهم النوايا والأفعال البشرية، وهو أمر حاسم للتقدم في الذكاء الاصطناعي المجسد. يتم تناول التحديات المتعلقة بالتقاط النوايا البشرية عالية المستوى وتأثيرات حركة الكاميرا الذاتية، مما يجعل هذه الطريقة مهمة للتطبيقات في الواقع الممتد والتلاعب بالروبوتات.

El artículo presenta MADiff, un método novedoso para predecir trayectorias de manos en videos egocéntricos utilizando modelos de difusión. Este enfoque busca mejorar la comprensión de las intenciones y acciones humanas, lo cual es crucial para los avances en inteligencia artificial encarnada. Se abordan los desafíos de capturar intenciones humanas de alto nivel y los efectos de la egomoción de la cámara, lo que hace que este método sea significativo para aplicaciones en realidad extendida y manipulación robótica.

L'article présente MADiff, une méthode novatrice pour prédire les trajectoires des mains dans des vidéos égocentriques en utilisant des modèles de diffusion. Cette approche vise à améliorer la compréhension des intentions et des actions humaines, ce qui est crucial pour les avancées en intelligence artificielle incarnée. Les défis de la capture des intentions humaines de haut niveau et des effets de l'égomotion de la caméra sont abordés, rendant cette méthode significative pour les applications en réalité étendue et en manipulation robotique.

The article presents MADiff, a novel method for predicting hand trajectories in egocentric videos using diffusion models. This approach aims to enhance the understanding of human intentions and actions, which is crucial for advancements in embodied artificial intelligence. The challenges of capturing high-level human intentions and the effects of camera egomotion interference are addressed, making this method significant for applications in extended reality and robot manipulation.

Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videos

Was this article worth reading? Share it