DOSSIER

AUTONEWS

U4D: Uncertainty-aware 4D world modeling from LiDAR sequences

Modeling dynamic 3D environments from LiDAR sequences is central to building reliable 4D worlds for autonomous driving and embodied AI. Existing generative frameworks, however, often treat all spatial regions uniformly, overlooking the varying uncertainty across real-world scenes. This uniform generation leads to artifacts in complex or ambiguous regions, limiting realism and temporal stability. In this work, we present U4D, an uncertainty-aware framework for 4D LiDAR world modeling. Our approach first estimates spatial uncertainty maps from a pretrained segmentation model to localize semantically challenging regions.

Modeling dynamic 3D environments from LiDAR sequences is fundamental to constructing reliable 4D world models that enable autonomous systems to perceive, simulate, and interact with the physical world over time. LiDAR provides precise geometric and depth information, forming the basis for perception, mapping, and planning in autonomous driving, robotics, and 3D scene reconstruction. However, collecting large-scale, diverse, and annotated LiDAR data remains costly and labor-intensive, motivating increasing interest in generative LiDAR modeling for scalable simulation, data augmentation, and pretraining.

Recent advances have explored LiDAR scene generation via adversarial, variational, and diffusion-based generative frameworks . Early efforts focus on object-level point clouds, whereas recent methods such as LiDARGen, LiDM, and LiDARCrafter synthesize large-scale and even dynamic LiDAR scenes. Despite these advances, existing methods treat spatial regions equally during generation, ignoring the varying semantic difficulty of real-world data. This uniform assumption often causes unreliable reconstruction in geometrically or semantically complex regions, such as thin poles, moving objects, and distant surfaces, where predictive confidence is low.

We observe that reliable LiDAR world modeling requires understanding the underlying uncertainty itself. Real LiDAR observations exhibit inherently non-uniform difficulty: while some areas are well-defined, others – such as occluded areas, small-scale structures, or semantically ambiguous regions – remain uncertain. Ignoring this asymmetry leads to geometric artifacts and temporal instability. Inspired by how humans resolve ambiguous regions before perceiving global context, we propose to model uncertainty explicitly, generating difficult regions first as structural anchors for the rest of the scene.

To this end, we propose U4D, an uncertainty-aware framework for 4D LiDAR world modeling. U4D leverages spatial uncertainty as a structural prior to guide scene generation. Our framework first estimates an uncertainty map from a pretrained LiDAR segmentation network to localize semantically ambiguous or geometrically unreliable regions. It then performs two sequential stages of generation: An uncertainty-region diffusion stage, which focuses on reconstructing high-entropy regions with fine geometric fidelity, and an uncertainty-conditioned completion stage, which synthesizes the remaining areas conditioned on these reconstructed structures. The two stages share a unified latent scene representation, enabling global contextual cues to refine local uncertainty and ensuring geometric and temporal consistency across the generated 4D scenes.

To further ensure stable temporal evolution, U4D integrates a Mixture of Spatio-Temporal (MoST) block, which explicitly decomposes and adaptively fuses spatial and temporal representations within the diffusion process. This design enables the generation of LiDAR sequences that are both geometrically precise and temporally coherent. Extensive experiments on the nuScenes and SemanticKITTI datasets demonstrate that U4D consistently outperforms existing LiDAR generation frameworks in terms of geometric fidelity, temporal stability, and downstream generalization

We propose U4D, the first uncertainty-aware generative framework for LiDAR scene synthesis. U4D generates LiDAR scenes in a “hard-to-easy” manner through two sequential stages. It estimates an uncertainty map from a real scan using a pretrained segmentation model and employs an unconditional diffusion process to reconstruct high-fidelity uncertain regions. Conditioned on these reconstructed areas, U4D completes the remaining scene to ensure structural integrity and global coherence. Both stages share latent representations, enabling global context to refine local uncertainty. To maintain temporal stability, U4D integrates a Mixture of Spatio-Temporal (MoST) block within the diffusion backbone. The MoST block decouples and adaptively fuses spatial and temporal features, enabling the generation of LiDAR sequences that are both geometrically precise and temporally coherent.

Uncertainty measurement in 3D...Real-world LiDAR scenes exhibit non-uniform difficulty across spatial regions. Some structures, such as ground or buildings, are geometrically stable and semantically consistent, while others are inherently uncertain due to factors such as distance-induced sparsity, occlusion, small-scale objects, or semantic ambiguity between visually similar categories. These uncertainty-prone regions frequently appear at long ranges, around object boundaries, or in areas of low point density, leading to inconsistent predictions in safety-critical perception tasks. Explicitly identifying and modeling these regions allows generative models to focus first on structurally unstable and perceptually ambiguous areas before extending to the entire scene, thereby producing more realistic priors and improving downstream robustness.

Uncertainty measurement in 3D...Real-world LiDAR scenes exhibit non-uniform difficulty across spatial regions. Some structures, such as ground or buildings, are geometrically stable and semantically consistent, while others are inherently uncertain due to factors such as distance-induced sparsity, occlusion, small-scale objects, or semantic ambiguity between visually similar categories. These uncertainty-prone regions frequently appear at long ranges, around object boundaries, or in areas of low point density, leading to inconsistent predictions in safety-critical perception tasks. Explicitly identifying and modeling these regions allows generative models to focus first on structurally unstable and perceptually ambiguous areas before extending to the entire scene, thereby producing more realistic priors and improving downstream robustness.

Uncertainty-conditioned scene completion...While sparse uncertainty scenes emphasize semantically ambiguous regions or structurally unstable regions, they only capture partial geometric information. To generate complete and coherent LiDAR frames, we design an uncertainty-conditioned diffusion model that synthesizes full scenes under the guidance of these uncertainty priors.

Mixture of spatio-temporal...Temporal coherence is essential for modeling dynamic real-world environments. While prior methods  focus on spatial reconstruction, maintaining consistent temporal evolution across frames remains challenging in dynamic modeling. To jointly ensure spatial fidelity and temporal consistency, we propose the Mixture of Spatio-Temporal (MoST) block. MoST is a core diffusion component that simultaneously captures fine-grained spatial geometry within each frame and smooth temporal transitions between frames, producing LiDAR sequences that are both geometrically accurate and temporally stable.

Comparative study...Scene-Level Fidelity. We first benchmark U4D against state-of-the-art LiDAR scene generation methods from a scene-level spatial fidelity perspective. Specifically, we sample 

 sequences and evaluate the first frame of each sequence to ensure a fair comparison with single-frame generation baselines producing the same number of scenes. As shown in Tab. 1 and Tab. 2, U4D consistently outperforms existing methods, achieving an FRD of 

For BEV-based metrics, including JSD and MMD, U4D also achieves competitive or superior performance, demonstrating robust spatial consistency across viewpoints. These results highlight U4D’s strong ability to generate geometrically accurate and perceptually consistent LiDAR scenes.

Table below:Comparison of state-of-the-art LiDAR scene generation methods on the SemanticKITTI [5] dataset. Metrics marked with  ↓ indicate that lower values are better. The MMD scores are reported in units of 10−4 The best and second-best scores are highlighted in bold and underline, respectively.

Method	Venue	FRD $\downarrow$	FPD $\downarrow$	JSD $\downarrow$	MMD $\downarrow$
LiDARGen [124]	ECCV’22	$735.49$	$119.69$	$0.13$	$21.90$
LiDM [83]	CVPR’24	-	$496.78$	$0.08$	$9.20$
R2DM [73]	ICRA’24	$262.85$	$12.06$	$\mathbf{0.03}$	$0.89$
Text2LiDAR [99]	ECCV’24	$567.47$	$16.78$	$0.08$	$4.24$
U4D	Ours	$\mathbf{245.73}$	$\mathbf{10.92}$	$0.04$	$\mathbf{0.85}$

Maintaining temporal coherence is crucial for sequential LiDAR generation, as inconsistent frame-to-frame predictions can lead to unrealistic scene dynamics. We evaluate U4D against recent methods, including UniScene, OpenDWM, and LiDARCrafter, on sequences sampled at 2Hz. TTCE measures deviations between predicted and ground-truth transformations via point cloud registration, while CTC computes Chamfer distances between consecutive frames. U4D consistently achieves the lowest TTCE scores across all frame intervals and maintains competitive CTC scores, reflecting its ability to generate temporally stable sequences with smooth and realistic motion patterns. These results highlight the effectiveness of the MoST block in capturing both spatial and temporal dependencies within sequences.

Design of MoST Block...The Mixture of Spatio-Temporal (MoST) block serves as a key component of our diffusion backbone, designed to adaptively fuse spatial and temporal cues for coherent LiDAR scene generation. In this ablation, we investigate how different fusion strategies affect generation quality. We first follow prior video generation approaches  and apply spatial and temporal operations in a cascaded manner. This configuration yields suboptimal results, likely because the deeper cascaded structure hampers convergence and optimization. We then decompose features into spatial and temporal branches and fuse them in parallel using either element-wise addition or concatenation. Both strategies significantly improve generation quality, as they expand network width rather than depth, facilitating more stable optimization. Finally, we introduce an adaptive fusion mechanism inspired by the mixture-of-experts paradigm, where the model learns to dynamically balance spatial and temporal information. This design achieves the best generation quality, demonstrating the effectiveness of adaptive spatio-temporal fusion for coherent 4D LiDAR generation.

To further examine how the MoST block fuses spatial and temporal information across different network stages, we analyze the relative activation weights of its two branches. As shown in the top-right of we visualize the averaged weighting distribution of spatial and temporal branches throughout the diffusion network. We observe that near the input and output layers, the spatial branch contributes more prominently, as these stages mainly focus on reconstructing local geometric details and structural integrity of LiDAR frames. In contrast, the temporal branch exhibits stronger activations in intermediate layers, where the model captures motion dynamics and ensures temporal consistency across frames. This observation validates our design intuition that spatial cues dominate at the boundaries for geometric fidelity, while temporal cues become essential in the latent space to model scene evolution and motion continuity. The adaptive allocation of activations allows MoST to balance geometric reconstruction and motion modeling, enabling the diffusion network to generate LiDAR sequences that are both spatially accurate and temporally coherent.

 

by Autonews