MM-CoT:A Benchmark for Probing Visual Chain-of-Thought Reasoning in Multimodal Models

A new framework has been proposed to predict GPU memory usage during the training of multimodal models, addressing the common issue of out-of-memory (OoM) errors that disrupt training processes. This framework analyzes model architecture and training behavior, decomposing models into layers to estimate memory usage accurately.

A new approach to multimodal language models (MLLMs) has been introduced, focusing on a progressive reparameterization strategy that replaces dense feed-forward network blocks with Parameterized Hypercomplex Multiplication (PHM) layers. This method aims to compress models while maintaining performance, facilitating faster inference without compromising output quality.

ReCAD has been introduced as a reinforcement learning framework that utilizes pretrained large models to generate precise parametric CAD models from multimodal inputs, enhancing the capabilities of vision-language models in computer-aided design. This approach allows for complex CAD operations with minimal functional input, contrasting with traditional methods that rely heavily on supervised fine-tuning.

A new technique called Dropout Prompt Learning has been proposed to enhance the robustness of vision-language models by applying dropout to both textual and visual tokens, allowing for flexible dropout probabilities based on token significance. This method aims to improve generalization in challenging scenarios such as low-shot learning and out-of-distribution generalization.

The introduction of TV2TV marks a significant advancement in video generation models, addressing challenges related to complex outputs that require semantic branching and high-level reasoning. This unified framework integrates language modeling and video flow matching through a Mixture-of-Transformers architecture, allowing for an interleaved generation process of text and video frames.