arXiv:2511.15552v2 Announce Type: replace-cross 
Abstract: Multimodal large language models (MLLMs) are currently at the center of research attention, showing rapid progress in scale and capabilities, yet their intelligence, limitations, and risks remain insufficiently understood. To address these issues, particularly in the context of the Russian language, where no multimodal benchmarks currently exist, we introduce Mera Multi, an open multimodal evaluation framework for Russian-spoken architectures. The benchmark is instruction-based and encompasses default text, image, audio, and video modalities, comprising 18 newly constructed evaluation tasks for both general-purpose models and modality-specific architectures (image-to-text, video-to-text, and audio-to-text). Our contributions include: (i) a universal taxonomy of multimodal abilities; (ii) 18 datasets created entirely from scratch with attention to Russian cultural and linguistic specificity, unified prompts, and metrics; (iii) baseline results for both closed-source and open-source models; (iv) a methodology for preventing benchmark leakage, including watermarking and licenses for private sets. While our current focus is on Russian, the proposed benchmark provides a replicable methodology for constructing multimodal benchmarks in typologically diverse languages, particularly within the Slavic language family.

تقدم الورقة Mera Multi، وهو إطار تقييم متعدد الوسائط مفتوح مصمم للهياكل اللغوية الروسية. يعالج نقص المعايير متعددة الوسائط في اللغة الروسية من خلال تقديم 18 مهمة تقييم عبر النص والصورة والصوت والفيديو. يهدف هذا الإطار إلى تعزيز فهم نماذج اللغة الكبيرة متعددة الوسائط (MLLMs) وقدراتها وقيودها ومخاطرها، خاصة في سياق الخصوصية الثقافية واللغوية الروسية.

El artículo presenta Mera Multi, un marco de evaluación multimodal abierto diseñado para arquitecturas en lengua rusa. Aborda la falta de referencias multimodales en el idioma ruso al introducir 18 tareas de evaluación en modalidades de texto, imagen, audio y video. Este marco tiene como objetivo mejorar la comprensión de los modelos de lenguaje multimodal (MLLM) y sus capacidades, limitaciones y riesgos, especialmente en el contexto de la especificidad cultural y lingüística rusa.

L'article présente Mera Multi, un cadre d'évaluation multimodal ouvert conçu pour les architectures en langue russe. Il répond au manque de références multimodales dans la langue russe en introduisant 18 tâches d'évaluation à travers les modalités de texte, d'image, d'audio et de vidéo. Ce cadre vise à améliorer la compréhension des modèles de langage multimodaux (MLLM) et de leurs capacités, limitations et risques, en particulier dans le contexte de la spécificité culturelle et linguistique russe.

The paper presents Mera Multi, an open multimodal evaluation framework designed for Russian-language architectures. It addresses the lack of multimodal benchmarks in the Russian language by introducing 18 evaluation tasks across text, image, audio, and video modalities. The framework aims to enhance understanding of multimodal large language models (MLLMs) and their capabilities, limitations, and risks, particularly in the context of Russian cultural and linguistic specificity.

Multimodal Evaluation of Russian-language Architectures

Was this article worth reading? Share it

LucidQuery AI

Magicley AI

VideoDubber Video Translator

Dubsmart LLC

Https

OpenL Translator

Ready to build your own newsroom?