Foundation4D: Towards a unified perception and reconstruction model for autonomous vehicles

One-line takeaway

Foundation4D unifies perception and reconstruction, achieving state-of-the-art results in depth, detection, and occupancy with co-training benefits.

Foundation4D提出一个统一模型，融合多摄像头视频实现几何估计、3D检测和语义占位等任务，在多个基准上取得最优性能，并通过联合训练提升检测与几何精度。

Key points

Foundation4D presents a unified feed-forward model for autonomous driving perception and reconstruction, leveraging factorized spatiotemporal attention to jointly estimate geometry, 3D detection, semantic occupancy, and Gaussian surfaces from multi-camera videos. It incorporates pose graph optimization for trajectory refinement and object tracking. The model achieves state-of-the-art performance on depth, detection, and occupancy benchmarks through task co-training, demonstrating mutual benefits between perception tasks.

• Foundation4D unifies geometry estimation, 3D detection, and semantic occupancy via a shared backbone with factorized spatiotemporal attention.
• Pose graph optimization refines camera trajectories and enables cross-window object tracking in video sequences.
• Outperforms existing methods on depth AbsRel, detection mAP, and occupancy mIoU across Waymo, nuScenes, and OpenOccupancy.
• Co-training detection with geometry improves detection mAP by 1.2 points and reduces camera-to-rig translation error by 13.4%.

Written only from the official arXiv abstract by an open-source model, then checked automatically (every number and name must appear in the abstract). The paper is authoritative.