Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance

This paper presents a tractable algorithm for estimating an unknown Lipschitz function from noisy observations, establishing an upper bound on its convergence rate. The approach extends max-affine methods from convex shape-restricted regression to a broader Lipschitz setting. A key component is a nonlinear feature expansion that maps max-affine functions into delta-convex functions, achieving the minimax convergence rate under squared loss and subgaussian distributions.

This paper addresses the challenges of learning representations for recipes and food images in cross-modal retrieval. It highlights that treating a recipe solely as a text source can create bias in image-and-recipe similarity judgments. The authors propose a causal theory model to mitigate this bias, emphasizing that factors like cooking processes and image conditions affect the representation learning process.

The paper introduces a novel task called code-to-style image generation, which aims to create images with unique and consistent visual styles based solely on numerical style codes. This approach addresses challenges faced by existing generative methods that rely on extensive textual prompts or reference images. The authors present CoTyle, the first open-source method for this task, filling a gap in academic research on visual stylization, which has been largely dominated by industry players like Midjourney.

The paper presents StreamingTalker, an autoregressive diffusion model designed for speech-driven 3D facial animation. This model addresses the limitations of previous methods that process audio sequences in a single pass, which can lead to poor performance with longer inputs and increased latency. By processing audio in a streaming manner, StreamingTalker offers flexibility with varying audio lengths and reduces latency, enhancing the realism and synchronization of facial animations.

Large Language Models (LLMs) are advanced linguistic tools that can produce outputs that may sound plausible but are often factually incorrect, a phenomenon known as hallucination. This study introduces a mathematical framework to analyze, quantify, and mitigate these hallucinations. It employs probabilistic modeling and Bayesian uncertainty estimation to develop refined metrics and strategies, including contrastive decoding and retrieval-augmented grounding, aimed at enhancing the reliability of LLMs.

This research explores retrieval augmented generation as a method for automatic context discovery in context-aware Automatic Speech Recognition (ASR) systems, aiming to enhance transcription accuracy, especially with rare or out-of-vocabulary terms. The study introduces an embedding-based retrieval approach and evaluates its effectiveness against large language model alternatives. Experiments show a reduction in word error rate (WER) by up to 17% compared to no-context, with oracle context achieving a 24.1% reduction.

The paper presents a novel physical adversarial attack targeting stereo matching models used in autonomous driving. Unlike traditional attacks that utilize 2D patches, this approach employs a 3D physical adversarial example (PAE) with global camouflage texture, enhancing visual consistency across various viewpoints. Additionally, a new 3D stereo matching rendering module is introduced to align the PAE with real-world positions in binocular vision, addressing the disparity effects of stereo cameras.

The paper introduces 'Insert In Style', a zero-shot generative framework designed to improve the insertion of real-world objects into stylized domains. This framework addresses the limitations of existing methods that either lack fidelity or require impractical fine-tuning. Key innovations include a multi-stage training protocol that separates identity, style, and composition, and a masked-attention architecture that enforces this separation during generation, trained on a new dataset of 100,000 samples.