Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation

一句话要点

腾讯提出 FAPL（因素感知预测学习），让视频 DiT 真正用上多模态大模型的潜表示：先冻结 MLLM 训练 DiT 依赖其潜变量，再用成对视频训练只改目标因素（角色、场景、动作、镜头）而保持其余不变，最后冻结 DiT 反过来优化 MLLM 和连接器，使其能从稀疏输入预测时序。在稀疏到视频、因素编辑、多镜头生成三项任务上优于基线。

Tencent's FAPL makes video DiTs actually use MLLM latents for cinematic generation: ground the DiT on frozen-MLLM latents, train factor-aware edits (character, scene, action, camera) with paired videos, then freeze the DiT and optimize the MLLM and connector for temporal inference from sparse inputs; it beats baselines on sparse-to-video, factor editing and multi-shot generation.