TOPIC · MULTIMODAL · 52 PAPERS

Multimodal

LLMs Reasoning Alignment Agents Multimodal Visual generation Vision & 3D Robotics & embodied AI Efficient training & inference Safety & interpretability AI for Science Code & SWE Benchmarks & data Reinforcement learning

Last 30 days 51

#03 ▲223 ★12k ❝1 2026-09-27

YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality

YuE2 plans a readable score first and then renders full songs, unifying symbolic and audio music generation.

Multimodal Art Projection · Ruibin Yuan, Jiahao Pan, Junyan Jiang +32

Multimodal Visual generation arXiv Code HF

#08 ▲375 ★210 ❝1 2026-10-13

VideoJev: System One Model for Video Understanding and Interaction

VideoJev scores time spans and answer options directly, unifying temporal grounding and video QA for fast decisions.

Multimedia Computing Group-Nanjing University · Yuhan Zhu, Xiangyu Zeng, Qingyi Si +15

Multimodal Efficient training & inference arXiv Code HF

#09 ▲485 ★200 ❝0 2026-09-25

Omni-IO Skills: Harnessing Your Agent Omni-Native

Omni-IO Skills makes existing agents omni-modal through hierarchical skills, a common execution interface and an asset registry.

National University of Singapore · Yanlin Li, Mingyang Hao, Shengqiong Wu +3

Agents Multimodal arXiv Code HF

#13 ▲417 ★261 ❝0 2026-09-29

LoopVL: Recurrent Visual Intelligence

LoopVL brings looped (recurrent) Transformers to vision-language models, iteratively refining a shared vision-language state.

Renmin University of China · Zhe Qian, Ziyang Gong, Zhongxing Xu +13

Multimodal Reasoning arXiv Code HF

#15 ▲184 ★200 ❝6 2026-09-12

Realtime-Venus: A full-duplex interaction system with asynchronous delegation

Realtime-Venus is a full-duplex audio-visual interaction system that listens, watches and talks while delegating tasks asynchronously.

inclusionAI · Ant Group

Multimodal Agents arXiv Code HF

#17 ▲238 ★228 ❝3 2026-10-01

OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction

OneStreamer lets streaming video LLMs keep hierarchical caption memories and respond proactively once evidence is sufficient.

Nanjing University · Xiangyu Zeng, Yuandong Yang, Zhiqiu Zhang +21

Multimodal arXiv Code HF

#27 ▲560 ★17 ❝0 2026-10-15

MJev-Spatial: System One Spatial Intelligence via Native Attention Matching

MJev-Spatial casts spatial questions as classification over candidates, answering relation, distance and point queries faster than text generation.

Mind Lab · Luozheng Qin, Yuhan Zhu, Changhai Zhou +9

Multimodal Efficient training & inference arXiv Code HF

#33 ▲270 ★175 ❝0 2026-10-09

S-Space: Exploring Spatial Workspace in Multimodal Language Models

Finds S-Space, a linear subspace inside MLLMs where object tokens encode continuous coordinates, acting as an internal spatial workspace.

MirroS · Zhiyuan Shi, Xiang Li, Xingzhuo Guo +8

Multimodal Safety & interpretability arXiv Code HF

#36 ▲224 ★47 ❝0 2026-09-28

Rethinking Latent Visual Reasoning: Grounding Latent Reasoning in Visual Evidence

Shows latent visual reasoning tokens barely respond to image evidence and proposes grounding them in visual evidence during training.

Amazon · Xi Xiao, Tianchen Zhao, Youngeun Kim +10

Reasoning Multimodal arXiv Code HF

#39 ▲174 ★364 ❝0 2026-10-04

Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation

Kandinsky 6.0 Video generates 5-second clips with synchronised 44 kHz audio and lip-sync, in 3B and 29B variants.

Kandinsky Lab · Team Kandinsky, Julia Agafonova, Bulat Akhmatov +85

Visual generation Multimodal arXiv Code HF

#59 ▲83 ★131 ❝3 2026-10-01

UniWAM: Unified World-Action Model

UniWAM unifies a physical reasoner, a world generator and an action predictor in one model for robot control.

HKUSTGZ · Wenxuan Song, Jiayi Chen, Jingbo Wang +13

Robotics & embodied AI Multimodal arXiv Code HF

#61 ▲102 ★152 2026-10-19

OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

OpenVLHarness gives MLLMs a tool capability layer and persistent multimodal memory for visual tasks needing external evidence.

Apple · Yu Zhou, Cheng-Fu Yang, Rui Sun +8

Multimodal Agents arXiv Code HF

#62 ▲143 ★30 ❝0 2026-10-10

Learning Where to Split: Dynamic Band Splitting for Audio Source Separation

Dynamic Band Splitting adapts frequency band boundaries per mixture to improve audio source separation within a fixed token budget.

Tsinghua Machine Learning Group · Bingzhe Zhang, Kai Li, Xiaolin Hu

Multimodal arXiv Code HF

#67 ▲116 ★59 ❝0 2026-09-29

Beyond the Timeline: Augmenting Long-Video Memory with Grounded Entity Biographies

GEB groups observations of the same physical object into entity biographies to answer questions over long videos.

Amazon Science · Hui Ren, Lei Fan, Henry Pao +5

Multimodal arXiv Code HF

#71 ▲225 ★12 ❝0 2026-10-19

Focused Omni-Modal GUI Agents

FocusGUI is a lightweight adapter that picks the step-relevant omni-modal evidence for a frozen GUI agent, curbing compounding distraction.

Nanyang Technological University · Fuyao Zhang, Wenwen He, Jiaming Zhang +6

Agents Multimodal arXiv Code HF

#78 ▲266 ❝0 2026-09-30

UniEvo-VL: An On-policy Self-Distillation Training Recipe for Multimodal Model Self-improvement

UniEvo-VL has one multimodal model act as both teacher and student, using its own critiques for on-policy self-distillation.

Stanford NLP · Fang Wu, Da Xing, Yanjie Huang +16

Multimodal arXiv HF

#79 ▲151 ★70 ❝0 2026-09-28

PanoVLN: Towards Effective Panoramic Vision-and-Language Navigation

PanoVLN studies how to make vision-and-language navigation actually benefit from panoramic observations.

Zhejiang University · Zhen Wang, Changpeng Wang, Zhe Liu +5

Robotics & embodied AI Multimodal arXiv Code HF

#99 ▲206 ★6 ❝0 2026-10-13

RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignment

RoTA-Bench evaluates VLMs' 3D spatial reasoning by having them rotate objects to match target silhouettes interactively.

MAC-AutoML · Qingchuan Ma, Yuexiao Ma, Xiang Meng +7

Multimodal Benchmarks & data Reasoning arXiv Code HF

#102 ▲70 ★91 ❝0 2026-10-06

VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMs

VisionWeave lets MLLMs learn end-to-end where and how finely to spend visual tokens, adapting to image content.

Qwen · Yuan Feng, Qize Yang, Ruizhe Chen +9

Multimodal Efficient training & inference arXiv Code HF

#110 ▲78 ★38 ❝2 2026-09-29

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

Imagine3D-LLM teaches MLLMs to imagine a 3D scene before answering multi-view spatial questions.

KAIST AI · Jaewoo Jung, Hyeonseo Yu, Honggyu An +10

Multimodal Reasoning arXiv Code HF

#111 ▲92 ❝6 2026-10-08

MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement

Technical report of Xiaomi's omni-modal MiMo-V2.6, focused on scaling RL compute along several dimensions toward self-improvement.

Xiaomi MiMo · Xiaomi LLM-Core Team, :, Zongming Qiao +147

LLMs Reinforcement learning Multimodal arXiv HF

#121 ▲89 ❝11 2026-09-19

Transferring the Intelligence of VLMs to Robotic Control

RoboDawn exposes robot control as simple discrete commands to an agentic VLM, testing whether digital intelligence transfers to the physical world.

Affiliation unverified · Meng-Hao Guo, Zhe-Han Mo, Jia-Jun Wang +6

Robotics & embodied AI Multimodal arXiv HF

#127 ▲34 ★227 ❝1 2026-09-29

PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation

PixelUMM unifies image and video understanding and generation directly in pixel space without a visual encoder.

NVIDIA · Cong Wei, Xuanchi Ren, Bryan Chu +8

Multimodal Visual generation arXiv Code HF

#143 ▲51 ★81 ❝0 2026-10-14

Intern Lumina U2: A Multi-Codebook Diffusion Large Language Model for Omni-Visual Understanding and Image Generation

Intern Lumina U2 is a multi-codebook diffusion LLM unifying visual understanding and image generation in one space.

Shanghai AI Laboratory · Qian Yu, Zongyang Qiu, Juncheng Yan +40

Multimodal Visual generation arXiv Code HF

#146 ▲134 ★7 ❝0 2026-09-26

VoxMem: Benchmarking Multimodal Memory in Large Audio Language Models

VoxMem benchmarks whether audio LLMs remember who said what, how, and what was audible—information beyond the transcript.

The University of Melbourne · Yang Xiao, Vidhyasaharan Sethu, Eun-Jung Holden +1

Multimodal Benchmarks & data arXiv Code HF

#148 ▲60 ★135 ❝0 2026-10-13

OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

OmniCompass evaluates video grounding and understanding on separate vision-language and audio-visual tracks.

VCLab · Xuyang Liu, Shihao Wang, Xinyu Wei +5

Multimodal Benchmarks & data arXiv Code HF

#152 ▲64 ★90 ❝0 2026-09-19

Spatial-Interactor: Learning Spatial Reasoning through Interaction with the Observable Physical World

Spatial-Interactor trains VLM spatial reasoning on observation-action-observation interaction trajectories.

ZJU-OmniAI · Kaixiang Yao, Xu Wang, Miao Pan +10

Multimodal Reasoning arXiv Code HF

#155 ▲77 ★58 ❝0 2026-09-21

Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings

Ovis-Embedding encodes text, images, video and audio into one shared embedding space using an omni-modal backbone.

ATH-MaaS · Ovis-Embedding Team

Multimodal arXiv Code HF

#168 ▲38 ★625 ❝0 2026-09-29

HelixWorld: A Real-time Interactive Audio-Visual World Model

HelixWorld is a real-time interactive world model where visuals and spatial stereo sound co-evolve with user input.

NoizAI · Lei Ke, Jiahao Pan, Zeyue Tian +13

Visual generation Multimodal arXiv Code HF

#178 ▲37 ★86 ❝0 2026-10-14

DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding

DeltaStreamer mimics human observation with change-aware memory and query-driven replay for streaming video understanding.

ByteDance · Boyuan Sun, Deshui Miao, Wenzhao Gao +8

Multimodal Agents arXiv Code HF

#187 ▲90 ❝1 2026-09-25

Duplex-MPE: Benchmarking Multi-Party Interaction in Full-Duplex Dialogue

Duplex-MPE uses 2,000 multi-party scenarios to test when a full-duplex voice assistant should answer, stay silent or stop.

Peking University · Chengqian Ma, Wenhao Feng, Weixuan Jin +7

Multimodal Benchmarks & data arXiv HF

#190 ▲175 ❝0 2026-10-15

Learning to Think and Explore in 3D World

Trains VLMs with RL to reason and actively move their viewpoint in 3D, seeing one image at a time like a human.

University of California at San Diego · Yijiang Li, Bingyang Wang, Nuno Vasconcelos

Multimodal Reinforcement learning Reasoning arXiv HF

#195 ▲41 ★233 ❝0 2026-10-06

WorldSonus: Bringing Sound to Worlds

WorldSonus adds real-time, spatially aligned stereo sound to world models with mid-stream sound control.

NoizAI · Pengjun Fang, Jingyi Fa, Kam Man Wu +9

Visual generation Multimodal arXiv Code HF

#198 ▲57 ★17 ❝0 2026-09-29

OmniTaskonomy: When Does Visual Generation Improve Visual Understanding?

Studies when visual generation training improves understanding, finding image-to-image training helps under the right recipe.

UC Berkeley · Jiaxin Ge, Yiming Qin, Ji Xie +13

Multimodal Vision & 3D arXiv Code HF

#200 ▲106 ★4 ❝0 2026-09-30

WorldAuditBench: Interactive 3D World Auditing with Multimodal Agents

WorldAuditBench has multimodal agents navigate 3D worlds to find anomalies like floating objects or walk-through walls.

University of California, Santa Barbara · Ziyan Jiang, Jingbo Yang, Jiabao Ji +5

Multimodal Agents Benchmarks & data arXiv Code HF

#209 ▲145 ❝0 2026-10-05

VepAgent: Bridging Causal-Transition via Tool-Augmented Reinforcement Learning for Video Event Prediction

VepAgent combines causal-transition reasoning with tool-augmented RL for video event prediction.

UCSC-VLAA · Qiutong Chen, Yuchan Guo, Zhenlong Yuan +10

Multimodal Reinforcement learning Reasoning arXiv HF

#218 ▲55 ★37 ❝0 2026-10-14

Seeing, Maintaining, and Learning: The Evolution of Multimodal Agent Memory

A survey of multimodal agent memory, from perceiving and maintaining memories to learned memory-management policies.

Fudan University · Minghao Guo, Qingyue Jiao, Xiujin Liu +77

Multimodal Agents arXiv Code HF

#224 ▲25 ★185 ❝0 2026-10-14

BlenderWorld: Benchmarking Visual Agents for 3D Creation from Real-World References

Qwen's BlenderWorld benchmarks visual agents creating and editing 3D content in Blender from real-world references.

Qwen · Xinyu Fang, Yuchi Wang, Weihao Yu +12

Benchmarks & data Agents Multimodal arXiv Code HF

#227 ▲68 ★14 ❝0 2026-09-30

Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generation

StereoBind ties stereo sound position to on-screen source motion for immersive joint video-audio generation.

Affiliation unverified · Hanmo Chen, Chengcheng Liu, Tianxiao Chen +7

Multimodal Visual generation arXiv Code HF

#228 ▲75 ★11 ❝0 2026-09-30

EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos

EgoTools targets tool-centric reasoning in real egocentric videos, covering hand-tool-object relations and procedures.

Ropedia · Shulin Tian, Junsu Kim, Shuai Liu +17

Multimodal Reasoning Benchmarks & data arXiv Code HF

#232 ▲25 ★826 ❝0 2026-10-15

Intern-Decision: Efficient Multimodal Decision Making with Autoregressive Masked Language Modeling

Intern-Decision (0.8B–4B) returns structured choices with probabilities instead of free text, making agent decisions faster.

Intern Large Models · Tong Zhu, Zhe Xu, Zefeng He +7

Multimodal Agents Efficient training & inference arXiv Code HF

#248 ▲45 ★45 ❝0 2026-10-16

Xiaomi-Lens-0 Technical Report

Xiaomi-Lens-0 is a 4B MLLM on Qwen3.5 that predicts segmentation masks from its own visual representations.

SeerRay-Lab · Pei Fu, Yike Liu, Yue Sun +13

Vision & 3D Multimodal arXiv Code HF

#250 ▲52 ★34 ❝0 2026-10-09

Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction

Work2Skill captures human GUI demonstrations and induces reusable skills for computer-use agents.

Nex AGI · Yuqi Huang, Hui Li, Yifei Cao +9

Agents Multimodal arXiv Code HF

#251 ▲60 ★24 ❝0 2026-10-01

Spatial Memory Intelligence: Endowing World Models with Understanding-Driven Long-Term Memory

Uses MLLM spatial understanding to manage long-term memory so world models stay spatially consistent in long video generation.

Affiliation unverified · Ying Yang, Guiyu Zhang, Lianghua Huang +5

Visual generation Multimodal arXiv Code HF

#253 ▲50 ★31 ❝0 2026-09-28

Learning Native Reflection in Unified Models with Interleaved Reinforcement Learning

Interleaved RL jointly trains reflection text and image generation so unified models can diagnose and repair their own images.

MMLab@NTU · Yijia Fan, Ziqi Huang, Zhongang Cai +5

Multimodal Reinforcement learning Visual generation arXiv Code HF

#267 ▲115 ❝0 2026-09-29

MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation

MotorMind scaffolds general vision-language models for zero-shot robot manipulation without specialised VLA training.

University of Illinois at Urbana-Champaign · Bingxuan Li, Siqi Song, Yizhuo Wu +3

Robotics & embodied AI Multimodal arXiv HF

#274 ▲46 ★22 ❝0 2026-10-10

BridgeWAM: Bridging Video and Action Experts via Latent Bridge Queries

BridgeWAM links video and action experts through 32 latent bridge queries instead of dense layer-wise coupling, enabling a two-layer action head.

Fudan University · Jiaxin Fang, Ruitong Liu, Zeyu Lai +6

Robotics & embodied AI Multimodal arXiv Code HF

#275 ▲34 ★14 ❝0 2026-10-15

Can Visual Generation Expand What Unified Multimodal Models Know?

A controlled probe shows unified multimodal models can transfer knowledge learned through image generation into text answers, though unevenly.

Z.ai · Wenkai Li, Wenyi Hong, Zhuoyi Yang +1

Multimodal arXiv Code HF

#276 ▲69 ❝1 2026-09-30

PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop

PhysVista evaluates VLMs' physical intelligence through a perception-reasoning-assessment loop.

university of science and technology of china · Xinge Peng, Yiting Lu, Tianwu Zhi +4

Multimodal Benchmarks & data Reasoning arXiv HF

#277 ▲29 ★19 ❝0 2026-10-13

Open-o3-Video-v2: Seek, Ground, and Reason with Verifiable Spatio-Temporal Evidence

Open-o3-Video-v2 actively gathers more frames and writes timestamps and boxes into its reasoning as verifiable evidence.

Peking University · Jiahao Meng, Zhongyuan Bao, Yue Tan +7

Multimodal Reasoning Agents arXiv Code HF

#293 ▲28 ★6 ❝0 2026-10-05

MemPilot: Orchestrating On-Demand Multimodal Memory Curation for LLM Agents

MemPilot trains an RL policy to curate multimodal agent memory on demand under performance–cost–latency preferences.

Nanyang Technological University · Haozhen Zhang, Haodong Yue, Quanyu Long +6

Agents Multimodal arXiv Code HF

Earlier (archive) 1

▲82 ★216 ❝0 2026-09-18

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

Defines native audio-visual dialogue for omni models and provides synthetic data, a benchmark and training recipes.

Qwen · Haolin He, Yunfei Chu, Qi Chen +15

Multimodal Benchmarks & data arXiv Code HF

Search this topic in the archive →