4D Gaussian Splatting for Real-Time Dynamic Scene Rendering

Arxiv Preprint

Project Page| Arxiv Paper

Guanjun Wu^1*, Taoran Yi^2*, Jiemin Fang³, Lingxi Xie³,
Xiaopeng Zhang³, Wei Wei¹,Wenyu Liu², Qi Tian³ , Xinggang Wang^2✉

¹School of CS, HUST ²School of EIC, HUST ³Huawei Inc.

Our method converges very quickly. And achieves real-time rendering speed.

Environmental Setups

Please follow the 3D-GS to install the relative packages.

git clone https://github.com/hustvl/4DGaussians --recursive
cd 4DGaussians
conda create -n Gaussians4D python=3.7 
pip install -r requirements.txt

In our environment, we use pytorch=1.13.1+cu116

Data Preparation

For synthetic scenes:
The dataset provided in D-NeRF is used. You can download the dataset from dropbox.

For real dynamic scenes:
The dataset provided in HyperNeRF is used. You can download scenes from Hypernerf Dataset and organize them as Nerfies. Meanwhile, Plenoptic Dataset could be downloaded from their offical websites, to save the memory, you should extract the frames of each video, twhen organize your dataset as follows.

├── data
│   | dnerf 
│     ├── mutant
│     ├── standup 
│     ├── ...
│   | hypernerf
│     ├── interp
│     ├── misc
│     ├── virg
│   | dynerf
│     ├── cook_spinach
│       ├── cam00
│           ├── images
│               ├── 0000.png
│               ├── 0001.png
│               ├── 0002.png
│               ├── ...
│       ├── cam01
│           ├── images
│               ├── 0000.png
│               ├── 0001.png
│               ├── ...
│     ├── cut_roasted_beef
|     ├── ...

Training

For training synthetic scenes such as lego, run

python train.py -s data/dnerf/bouncingballs --port 6017 --expname "dnerf/bouncingballs" --configs arguments/dnerf/bouncingballs.py

You can custom your training config through the config files.

Rendering

Run the following script to render the images.

python render.py --model_path "output/dnerf/bouncingballs/"  --skip_train --configs arguments/dnerf/bouncingballs.py  &

Evaluation

Run the following script to evaluate the model.

python metrics.py --model_path "output/dnerf/bouncingballs/"

Scripts

There are some helpful scripts in scripts/, please feel free to use them.

Some source code of ours is borrowed from 3DGS, k-planes,HexPlane, TiNeuVox. We sincerely appreciate the excellent works of these authors.

Citation

If you find this repository/work helpful in your research, welcome to cite the paper and give a ⭐.

@article{wu20234dgaussians,
  title={4D Gaussian Splatting for Real-Time Dynamic Scene Rendering},
  author={Wu, Guanjun and Yi, Taoran and Fang, Jiemin and Xie, Lingxi and Zhang, Xiaopeng and Wei Wei and Liu, Wenyu and Tian, Qi and Wang Xinggang},
  journal={arXiv preprint arXiv:2310.08528},
  year={2023}
}

4.5 KiB Raw Blame History Unescape Escape