arXiv:2510.12089v2 Announce Type: replace 
Abstract: Recent advances in diffusion models have significantly improved audio-driven human video generation, surpassing traditional methods in both quality and controllability. However, existing approaches still face challenges in lip-sync accuracy, temporal coherence for long video generation, and multi-character animation. In this work, we propose a diffusion transformer (DiT)-based framework for generating lifelike talking videos of arbitrary length, and introduce a training-free method for multi-character audio-driven animation. First, we employ a LoRA-based training strategy combined with a position shift inference approach, which enables efficient long video generation while preserving the capabilities of the foundation model. Moreover, we combine partial parameter updates with reward feedback to enhance both lip synchronization and natural body motion. Finally, we propose a training-free approach, Mask Classifier-Free Guidance (Mask-CFG), for multi-character animation, which requires no specialized datasets or model modifications and supports audio-driven animation for three or more characters. Experimental results demonstrate that our method outperforms existing state-of-the-art approaches, achieving high-quality, temporally coherent, and multi-character audio-driven video generation in a simple, efficient, and cost-effective manner.

أدت التطورات الأخيرة في نماذج الانتشار إلى تحسينات كبيرة في توليد مقاطع الفيديو البشرية المدفوعة بالصوت، متفوقة على الأساليب التقليدية من حيث الجودة والتحكم. ومع ذلك، لا تزال هناك تحديات في دقة مزامنة الشفاه، والتماسك الزمني لتوليد مقاطع الفيديو الطويلة، والرسوم المتحركة متعددة الشخصيات. يقترح الإطار المعتمد على محول الانتشار (DiT) توليد مقاطع فيديو واقعية لأشخاص يتحدثون دون الحاجة إلى تدريب. يتم استخدام استراتيجية تدريب قائمة على LoRA مع نهج استدلال لتحويل الموضع، مما يعزز كل من مزامنة الشفاه والحركة الطبيعية للجسم من خلال التغذية الراجعة للمكافآت.

Los avances recientes en modelos de difusión han mejorado significativamente la generación de videos humanos impulsados por audio, superando a los métodos tradicionales en calidad y control. Sin embargo, aún existen desafíos en la precisión de la sincronización labial, la coherencia temporal para la generación de videos largos y la animación de múltiples personajes. El marco propuesto utiliza un transformador de difusión (DiT) para generar videos realistas de personas hablando sin necesidad de entrenamiento. Se emplea una estrategia de entrenamiento basada en LoRA y un enfoque de inferencia de…

Les avancées récentes dans les modèles de diffusion ont considérablement amélioré la génération de vidéos humaines audio-dirigées, surpassant les techniques traditionnelles en termes de qualité et de contrôle. Cependant, des défis subsistent en matière de précision de synchronisation labiale, de cohérence temporelle pour les vidéos longues et d'animation multi-personnages. Le cadre proposé utilise un transformateur de diffusion (DiT) pour générer des vidéos réalistes de personnes parlant, sans nécessiter d'entraînement. Il intègre une stratégie basée sur LoRA et une méthode d'inférence par déc…

Recent advancements in diffusion models have led to significant improvements in audio-driven human video generation, outperforming traditional techniques in quality and controllability. However, challenges remain in achieving lip-sync accuracy, maintaining temporal coherence in long videos, and creating multi-character animations. The proposed framework utilizes a diffusion transformer (DiT) to generate realistic talking videos of any length without the need for training. It incorporates a LoRA-based strategy and a position shift inference method, enhancing lip synchronization and natural body…

Playmate2: Training-Free Multi-Character Audio-Driven Animation via Diffusion Transformer with Reward Feedback

arXiv:2601.07833v2 Announce Type: replace 
Abstract: We present RefVFX, a new framework that transfers complex temporal effects from a reference video onto a target video or image in a feed-forward manner. While existing methods excel at prompt-based or keyframe-conditioned editing, they struggle with dynamic temporal effects such as dynamic lighting changes or character transformations, which are difficult to describe via text or static conditions. Transferring a video effect is challenging, as the model must integrate the new temporal dynamics with the input video's existing motion and appearance. % To address this, we introduce a large-scale dataset of triplets, where each triplet consists of a reference effect video, an input image or video, and a corresponding output video depicting the transferred effect. Creating this data is non-trivial, especially the video-to-video effect triplets, which do not exist naturally. To generate these, we propose a scalable automated pipeline that creates high-quality paired videos designed to preserve the input's motion and structure while transforming it based on some fixed, repeatable effect. We then augment this data with image-to-video effects derived from LoRA adapters and code-based temporal effects generated through programmatic composition. Building on our new dataset, we train our reference-conditioned model using recent text-to-video backbones. Experimental results demonstrate that RefVFX produces visually consistent and temporally coherent edits, generalizes across unseen effect categories, and outperforms prompt-only baselines in both quantitative metrics and human preference. See our website https://tuningfreevisualeffects-maker.github.io/Tuning-free-Visual-Effect-Transfer-across-Videos-Project-Page/

تم تقديم إطار جديد يسمى RefVFX، مما يتيح نقل التأثيرات الزمنية المعقدة من فيديو مرجعي إلى فيديو أو صورة مستهدفة بطريقة مباشرة. تعالج هذه الابتكار التحديات المتعلقة بالتأثيرات الزمنية الديناميكية، مثل تغييرات الإضاءة وتحولات الشخصيات، والتي يصعب التعبير عنها من خلال النص أو الشروط الثابتة.

Se ha presentado un nuevo marco llamado RefVFX que permite la transferencia de efectos temporales complejos de un video de referencia a un video o imagen objetivo de manera directa. Esta innovación aborda los desafíos de los efectos temporales dinámicos, como los cambios de iluminación y las transformaciones de personajes, que son difíciles de describir a través de texto o condiciones estáticas.

Un nouveau cadre nommé RefVFX a été introduit, permettant le transfert d'effets temporels complexes d'une vidéo de référence vers une vidéo ou une image cible de manière directe. Cette innovation répond aux défis des effets temporels dynamiques, tels que les changements d'éclairage et les transformations de personnages, qui sont difficiles à exprimer par le texte ou des conditions statiques.

A new framework named RefVFX has been introduced, enabling the transfer of complex temporal effects from a reference video to a target video or image in a feed-forward manner. This innovation addresses challenges in dynamic temporal effects, such as lighting changes and character transformations, which are difficult to articulate through text or static conditions.

Tuning-free Visual Effect Transfer across Videos

One More Thing in AI – Your Shortcut to AI Mastery

Playmate2: Training-Free Multi-Character Audio-Driven Animation via Diffusion Transformer with Reward Feedback

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

Synthesia

Interactive Avatar

Rendora AI

AiReelGenerator.com

Ready to build your own newsroom?