CA-World: Multi-Object Counterfactual Alignment for Efficient Interactive-Ready Reconstruction

Xin Dong^1,2, Weijian Deng¹, Lihan Zhang¹, Tianru Dai¹, Wenfeng Deng², Yansong Tang¹

¹Shenzhen International Graduate School, Tsinghua University, ²Pengcheng Laboratory

Abstract

Reconstructing interaction-ready 3D worlds is essential for physical simulation, virtual reality, robotics, and autonomous driving. However, existing methods mainly optimize static and holistic visual fidelity, with limited support for multi-object interaction. We argue that an interaction-ready reconstruction should anticipate potential scene changes and preserve geometric completeness, visual quality, multi-object spatial relationship, and physical plausibility under potential interactions. To this end, motivated by the causal intervention, we propose CA-World, an efficient framework that integrates counterfactual alignment learning into a decoupling–reintegration reconstruction pipeline. Specifically, we formulate foreground–background decoupling as a visual intervention, separate object generation and background inpainting as counterfactual generation, and scene reintegration as an inverse intervention. According to counterfactual consistency, reversing the intervention should recover the factual world, motivating three alignment objectives between the reintegrated and original scenes: appearance, spatial, and physical consistency. This formulates interaction-ready reconstruction as counterfactual alignment learning with direct supervision. Moreover, leveraging the locality of object-level interventions, CA-World constrains counterfactual states using the observed scene, enabling efficient and coherent reintegration without jointly optimizing all object states, thereby reducing computational cost and error accumulation. Experiments on object completeness, spatial accuracy, outdoor background completion, rendering quality, simulated dynamics, and downstream applications demonstrate the effectiveness of CA-World.

Video comparisons

Ours

Feature Splatting

Ours

Feature Splatting

Ours

Feature Splatting

Ours

Feature Splatting

Comparison with baselines

To validate the counterfactual consistency for interactive simulation, we show comparisons with Feature Splatting in multi-object condition. As shown in following figure, our method delivers more accurate object decoupling and restoration, with good simulated dynamics. In contrast, Feature Splatting is dependent on natural language for localization and segmentation, failing in multi-object scenarios to separate objects from the background, leading to fragmented objects, incomplete scene, and severe tearing effects.

User study and LMM-as-judge evaluation

We invited 12 participants and showed each of them the rendered videos generated by different methods across all scenes. As shown in the table, our method significantly outperforms the baselines in both motion realism and visual quality. Additionally, Large Multimodal Model (LMM) have recently emerged as a powerful tool for assessing the quality of rendered videos. The results output from the LMM align closely with human judgments, validating the effectiveness of our approach.

Ablation studies

To validate the contribution of each consistency objective, we report ablation results in following table, employing edge error, PSNR, and SSIM to quantify the ablated visual degradation. The results indicate that appearance consistency is critical for enhancing visual metrics, whereas spatial and physical consistency are vital for preserving geometric fidelity.

Additionally, sub-figure(a) of following figure illustrates the object alignment in the bear scene and a ball in the outdoor scene. As optimization progresses, the object masks exhibit increasingly precise alignment with the target regions, and the object positions converge to their correct locations. In sub-figure(b), we ablate spatial consistency, exploiting metric cues from the 3D reconstruction to compute pointmaps, helping 3D generative models with more accurate object initialization in 3D space.

BibTeX

@article{dong2026sam3d, title={CA-World: Multi-Object Counterfactual Alignment for Efficient Interactive-Ready Reconstruction}, author={Dong, Xin and Deng, Weijian and Zhang, Lihan and Dai, Tianru and Deng, Wenfeng and Tang, Yansong}, journal={arXiv preprint arXiv:2605.30239}, year={2026} }