[CVPR 2026] Physical Simulator In-the-Loop Video Generation

Lin Geng Foo^1,5, Mark He Huang^2,3, Alexandros Lattas⁴, Stylianos Moschoglou⁴, Thabo Beeler⁴, Christian Theobalt^{1,5 1}Max Planck Institute for Informatics, Saarland Informatics Campus, ²SUTD, ³A*STAR, ⁴Google, ⁵Saarbrücken Research Center for Visual Computing, Interaction and Artificial Intelligence

[Project Page] | [Paper] | [Video]

Usage

Please refer to the instructions.md for environment setup and pipeline execution.

Disclaimer: This repository offers a reference implementation of the pipeline described in the paper. The code is intended for research purposes only, is not optimized for robustness or production environments, and may not be actively maintained.

License

This project is released under the Apache 2.0 License.

BibTeX

@InProceedings{Foo_2026_CVPR,
    author    = {Foo, Lin Geng and Huang, Mark He and Lattas, Alexandros and Moschoglou, Stylianos and Beeler, Thabo and Theobalt, Christian},
    title     = {Physical Simulator In-the-Loop Video Generation},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2026},
    pages     = {4301-4311}
}

Acknowledgments

This project is built upon invaluable open-source research projects: ViPE, PhysGen3D, Taichi, Taichi Elements, Mitsuba, GroundingDINO, SAM, LaMa, InstantMesh, lang-segment-anything, Go-with-the-Flow, cogvideox-factory, CogVideoX, HunyuanVideo.

We also thank these research projects for sharing their code: pisa-experiments, MotionClone, SG-I2V, ImageConductor, DragAnything, VBench.