Learn how to Prune Pixels for Multi-view Neural Image-based Synthesis
This addresses bandwidth limitations for real-time or constrained applications in neural rendering, representing an incremental improvement over existing pruning methods.
The paper tackles the problem of high data volume in image-based rendering by introducing LeHoPP, a method that prunes irrelevant input pixels to reduce bandwidth without retraining, achieving an average gain of 0.9 to 3.6 dB in synthesis quality compared to baselines.
Image-based rendering techniques stand at the core of an immersive experience for the user, as they generate novel views given a set of multiple input images. Since they have shown good performance in terms of objective and subjective quality, the research community devotes great effort to their improvement. However, the large volume of data necessary to render at the receiver's side hinders applications in limited bandwidth environments or prevents their employment in real-time applications. We present LeHoPP, a method for input pixel pruning, where we examine the importance of each input pixel concerning the rendered view, and we avoid the use of irrelevant pixels. Even without retraining the image-based rendering network, our approach shows a good trade-off between synthesis quality and pixel rate. When tested in the general neural rendering framework, compared to other pruning baselines, LeHoPP gains between $0.9$ dB and $3.6$ dB on average.