arXiv:2510.20877v1 Announce Type: new 
Abstract: Multimodal learning systems often encounter challenges related to modality imbalance, where a dominant modality may overshadow others, thereby hindering the learning of weak modalities. Conventional approaches often force weak modalities to align with dominant ones in "Learning to be (the same)" (Positive Learning), which risks suppressing the unique information inherent in the weak modalities. To address this challenge, we offer a new learning paradigm: "Learning Not to be" (Negative Learning). Instead of enhancing weak modalities' target-class predictions, the dominant modalities dynamically guide the weak modality to suppress non-target classes. This stabilizes the decision space and preserves modality-specific information, allowing weak modalities to preserve unique information without being over-aligned. We proceed to reveal multimodal learning from a robustness perspective and theoretically derive the Multimodal Negative Learning (MNL) framework, which introduces a dynamic guidance mechanism tailored for negative learning. Our method provably tightens the robustness lower bound of multimodal learning by increasing the Unimodal Confidence Margin (UCoM) and reduces the empirical error of weak modalities, particularly under noisy and imbalanced scenarios. Extensive experiments across multiple benchmarks demonstrate the effectiveness and generalizability of our approach against competing methods. The code will be available at https://github.com/BaoquanGong/Multimodal-Negative-Learning.git.

تسلط دراسة جديدة حول أنظمة التعلم متعددة الوسائط الضوء على التحديات المتعلقة بعدم توازن الوسائط، حيث يمكن أن تطغى الوسائط الأقوى على الأضعف. وغالبًا ما يؤدي ذلك إلى قمع المعلومات الفريدة من الوسائط الأضعف. تقترح الدراسة نهجًا جديدًا للتعلم يهدف إلى الحفاظ على تميز جميع الوسائط، بدلاً من إجبارها على الت conform إلى الوسائط السائدة. هذا مهم لأنه يمكن أن يعزز فعالية أنظمة الوسائط المتعددة في تطبيقات مختلفة، مما يضمن الاستفادة من الرؤى القيمة من جميع الوسائط.

Un nuevo estudio sobre sistemas de aprendizaje multimodal destaca los desafíos del desequilibrio de modalidades, donde las modalidades más fuertes pueden eclipsar a las más débiles. Esto a menudo conduce a suprimir la información única de las modalidades más débiles. La investigación propone un nuevo enfoque de aprendizaje que busca preservar la singularidad de todas las modalidades, en lugar de forzarlas a conformarse a las dominantes. Esto es significativo, ya que podría mejorar la efectividad de los sistemas multimodales en diversas aplicaciones, asegurando que se utilicen valiosos conocimientos de todas las modalidades.

Une nouvelle étude sur les systèmes d'apprentissage multimodal met en lumière les défis de l'imbalance des modalités, où les modalités plus fortes peuvent éclipser les plus faibles. Cela conduit souvent à une suppression des informations uniques provenant des modalités plus faibles. La recherche propose une nouvelle approche d'apprentissage qui vise à préserver la singularité de toutes les modalités, plutôt que de les forcer à se conformer à celles qui dominent. Cela est significatif car cela pourrait améliorer l'efficacité des systèmes multimodaux dans diverses applications, garantissant que des informations précieuses de toutes les modalités soient utilisées.

A new study on multimodal learning systems highlights the challenges of modality imbalance, where stronger modalities can overshadow weaker ones. This often leads to a suppression of unique information from the weaker modalities. The research proposes a novel approach to learning that aims to preserve the distinctiveness of all modalities, rather than forcing them to conform to the dominant ones. This is significant as it could enhance the effectiveness of multimodal systems in various applications, ensuring that valuable insights from all modalities are utilized.

Multimodal Negative Learning

arXiv:2512.07747v1 Announce Type: new 
Abstract: Unified understanding and generation is a highly appealing research direction in multimodal learning. There exist two approaches: one trains a transformer via an auto-regressive paradigm, and the other adopts a two-stage scheme connecting pre-trained understanding and generative models for alignment fine-tuning. The former demands massive data and computing resources unaffordable for ordinary researchers. Though the latter requires a lower training cost, existing works often suffer from limited task coverage or poor generation quality. Both approaches lack the ability to parse input meta-information (such as task type, image resolution, video duration, etc.) and require manual parameter configuration that is tedious and non-intelligent. In this paper, we propose Unison which adopts the two-stage scheme while preserving the capabilities of the pre-trained models well. With an extremely low training cost, we cover a variety of multimodal understanding tasks, including text, image, and video understanding, as well as diverse generation tasks, such as text-to-visual content generation, editing, controllable generation, and IP-based reference generation. We also equip our model with the ability to automatically parse user intentions, determine the target task type, and accurately extract the meta-information required for the corresponding task. This enables full automation of various multimodal tasks without human intervention. Experiments demonstrate that, under a low-cost setting of only 500k training samples and 50 GPU hours, our model can accurately and automatically identify tasks and extract relevant parameters, and achieve superior performance across a variety of understanding and generation tasks.

تم تقديم إطار عمل جديد يسمى Unison، مصمم لفهم وتوليد موحد في التعلم متعدد الوسائط. يعتمد هذا الإطار على مخطط من مرحلتين يستخدم بفعالية نماذج مدربة مسبقًا مع تقليل تكاليف التدريب بشكل كبير، مما يعالج قيود الأساليب الحالية التي تتطلب إما بيانات ضخمة أو تعاني من ضعف جودة التوليد.

Se ha presentado un nuevo marco llamado Unison, diseñado para la comprensión y generación unificadas en el aprendizaje multimodal. Este marco adopta un esquema de dos etapas que utiliza eficazmente modelos preentrenados mientras reduce significativamente los costos de entrenamiento, abordando las limitaciones de los enfoques existentes que requieren grandes cantidades de datos o sufren de mala calidad en la generación.

Un nouveau cadre nommé Unison a été introduit, conçu pour la compréhension et la génération unifiées dans l'apprentissage multimodal. Ce cadre adopte un schéma en deux étapes qui utilise efficacement des modèles pré-entraînés tout en réduisant considérablement les coûts d'entraînement, répondant ainsi aux limitations des approches existantes qui nécessitent soit des données extensives, soit souffrent d'une mauvaise qualité de génération.

A new framework named Unison has been introduced, designed for unified understanding and generation in multimodal learning. This framework adopts a two-stage scheme that effectively utilizes pre-trained models while significantly reducing training costs, addressing the limitations of existing approaches that either require extensive data or suffer from poor generation quality.

Multimodal Negative Learning

Was this article worth reading? Share it

LucidQuery AI

Magicley AI

Augmeta