Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation

One-line takeaway

Tencent's FAPL makes video DiTs actually use MLLM latents for cinematic generation: ground the DiT on frozen-MLLM latents, train factor-aware edits (character, scene, action, camera) with paired videos, then freeze the DiT and optimize the MLLM and connector for temporal inference from sparse inputs; it beats baselines on sparse-to-video, factor editing and multi-shot generation.

腾讯提出 FAPL（因素感知预测学习），让视频 DiT 真正用上多模态大模型的潜表示：先冻结 MLLM 训练 DiT 依赖其潜变量，再用成对视频训练只改目标因素（角色、场景、动作、镜头）而保持其余不变，最后冻结 DiT 反过来优化 MLLM 和连接器，使其能从稀疏输入预测时序。在稀疏到视频、因素编辑、多镜头生成三项任务上优于基线。