2025-07-02 16:07:39 +08:00

5.0 KiB

Raw Blame History

SpatialTrackerV2: 3D Point Tracking Made Easy

CAD&CG, Zhejiang University; University of Oxford; Ant Research; Pixelwise AI; NUS

Yuxi Xiao, Jianyuan Wang, Nan Xue, Nikita Karaev, Iurii Makarov, Bingyi Kang, Xin Zhu, Hujun Bao, Yujun Shen, Xiaowei Zhou

Project Page | BibTeX | Goolge Drive

🚀 Latest Updates & News

🎉 What's New & Exciting! 🎉

🔥 Stay tuned for the most exciting developments! We're constantly pushing the boundaries of 3D tracking technology! 🔥

📅 Recent Highlights

🏆 25 June 2025: 🥇 SpatialTrackerV2 has been ACCEPTED by ICCV 2025! 🥇
📄 See you in Hawaii!

🎯 23 June 2025: 🤖 Try our amazing Huggingface Space Demo: https://huggingface.co/spaces/Yuxihenry/SpatialTrackerV2
✨ Experience the magic of 3D point tracking in your browser!

💡 Coming Soon: More incredible features on the way! Keep watching this space! 👀

TODO List

Release Offline Version
- SpaTrack2 + Moge ➔ supports unposed RGB as input.
- SpaTrack2+ MegaSAM ➔ supports Posed RGBD as input.
- SpaTrack2 + VGGT ➔ make VGGT works in Dynamic Scenes.
Release Online Version
- Sliding windows version.
More Releases
- Some Ceres Python Bindings designed for SpatialTracker and Dynamic Reconstruction.
- More supports for other Depth Model, i.e., DepthAnything, StereoFoundation, UniDepth, Metric3D.

Set up the environment

To set up the environment for running the SpaTrack model, follow these steps:

Clone the Repository:

git clone git@github.com:henry123-boy/SpaTrackerV2.git
cd SpaTrackerV2

Create a Virtual Environment: It's recommended to use a virtual environment to manage dependencies.
```
conda create -n SpaTrack2 python=3.11
conda activate SpaTrack2
```

Install Dependencies:

Install the torch dependencies pip (tested with torch2.4).

python -m pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu124

Install the required Python packages using pip.

python -m pip install -r requirements.txt

Install SpaTrack2 Visualizer:
```
cd viser
python -m pip install -e .
```

By following these steps, you should have a working environment ready to run the SpaTrack model.

Download the Checkpoints

mkdir -p checkpoints

Step1: Download the checkpoint of Moge from here, and put the model.pt into ./checkpoints/

Step2: Download the checkpoint of SpaTrack2 from GoolgeDrive, and place it into ./checkpoints/

Quick Start

We gave two examples to illustrate the usage of SpaTrack2. Firstly, please download ckpts and examples via:

sh scripts/download.sh

Type1: Posed RGBD video (Example0)

We provide an example who has Posed RGBD input with MegaSAM.

python inference.py --data_type="RGBD" --data_dir="assets/example0" --video_name="snowboard" --fps=1

Type2: unposed RGB video (Example1)

python inference.py --data_type="RGB" --data_dir="assets/example1" --video_name="xhs" --fps=6

Visualize your results

We provide two types of visualization, i.e. viser and TAPIP3D. The guidance will be displayed in the terminal after running inference.py.

🌟 Recommended: Gradio Demo with SAM 🌟

Please follow the instructions in the app_3rd README to configure the dependencies. Then,

python -m pip install gradio==5.31.0 pako

Our gradio demo enable the user to track the points on the target object easily, just try:

python app.py

Here is an example with our GUI:

5.0 KiB Raw Blame History