WavePlanes: Compact Feature Planes for Dynamic Novel View Synthesis

Adrian Azzarelli, Nantheera Anantrasirichai, David R Bull
Visual Information Laboratory, Univesity of Bristol
Paper Code

Dynamic Novel View Synthesis (Dynamic NVS) enhances NVS technologies to model moving 3-D scenes. However, current methods are resource intensive and challenging to compress. To address this, we present WavePlanes, a fast and more compact hex plane representation, applicable to both dynamic Neural Radiance Fields and Gaussian Splatting methods. Rather than modeling many feature scales separately (as done previously), we use the inverse discrete wavelet transform to reconstruct features at varying scales. This leads to a more compact representation and allows us to explore wavelet-based compression schemes for further gains. The proposed compression scheme exploits the sparsity of wavelet coefficients, by applying hard thresholding to the wavelet planes and storing nonzero coefficients and their locations on each plane in a Hash Map. Compared to the state-of-the-art (SotA), WavePlanes is significantly smaller, less resource demanding and competitive in reconstruction quality. Compared to small SotA models, WavePlanes outperforms methods in both model size and quality of novel views.

MY ALT TEXT

Model Overview: (a-f) illustrates the main pipeline, (1) outlines the compression pipeline (accomplished after training)

D-NeRF Synthetic Scenes
Dy-NeRF Real Scenes
Visual Comparisons
Additional Ablations

The ablation results, below, were carried out on "K-Planes-W" (our WavePlanes applied to K-Plane pipeline) on the D-NeRF dataset. The configurations were carried over to 4D-GS-W (WavePlanes applied to 4D-GS pipeline) without significant modification, aside from that we only model a single feature plane for 4D-GS-W while K-Planes-W still requires modelling two sets of feature planes.

MY ALT TEXT

For the D-NeRF scenes we found that lower resolutions are better for the more primitive scenes. This ablation shows that lowering the scale of the feature planes can be massively beneficial for both time and quality, as with the Bouncing Balls

MY ALT TEXT

We test the various wavelet families, though it should be noted that the number of parameters will change depending on the how the wavelet is processed. Some wavelet families use more verbose functions extending training time and computation. Indepth results from testing varying wave-levels is provided below

MY ALT TEXT

The ablation on wavelet level reveals significant quality benefit for methods using a shallow wavelevel. Unlike MWT (the most relevant static wavelet-tri-plane method) which uses N=4, our method is capable of producing good results with just N=2. This comes with the benefit of being faster to train.

MY ALT TEXT

This final ablation tests the effectiveness of selecting a singular wavelet direction to regularize. When wavelet coeffs are reconstructed into signals they have directionality (horizontal, diagonal and vertical), so we test the proposed regularizing method on each individual direction (e.g.) only regularizing coefficients along the temporal axis (horizontal axis). The results indicate that the proposed regularizer is much stronger when all directions are regularized.

BibTeX

@article{azzarelli2023waveplanes,
        title={WavePlanes: Compact Feature Planes for Dynamic Novel View Synthesis},
        author={Azzarelli, Adrian and Anantrasirichai, Nantheera and Bull, David R},
        journal={arXiv preprint arXiv:2312.02218},
        year={2023}
      }