ResVR: Joint Rescaling and Viewport Rendering of Omnidirectional Images
This work addresses the issue of inferior visual experiences for VR users by improving viewport rendering from compressed omnidirectional images, representing an incremental advance in domain-specific VR technology.
The paper tackles the problem that current omnidirectional image rescaling methods focus on equirectangular projection quality, leading to poor viewport experiences in VR headsets, and proposes ResVR, a joint rescaling and viewport rendering framework that outperforms existing methods in visual quality across various conditions while maintaining low transmission overhead.
With the advent of virtual reality technology, omnidirectional image (ODI) rescaling techniques are increasingly embraced for reducing transmitted and stored file sizes while preserving high image quality. Despite this progress, current ODI rescaling methods predominantly focus on enhancing the quality of images in equirectangular projection (ERP) format, which overlooks the fact that the content viewed on head mounted displays (HMDs) is actually a rendered viewport instead of an ERP image. In this work, we emphasize that focusing solely on ERP quality results in inferior viewport visual experiences for users. Thus, we propose ResVR, which is the first comprehensive framework for the joint Rescaling and Viewport Rendering of ODIs. ResVR allows obtaining LR ERP images for transmission while rendering high-quality viewports for users to watch on HMDs. In our ResVR, a novel discrete pixel sampling strategy is developed to tackle the complex mapping between the viewport and ERP, enabling end-to-end training of ResVR pipeline. Furthermore, a spherical pixel shape representation technique is innovatively derived from spherical differentiation to significantly improve the visual quality of rendered viewports. Extensive experiments demonstrate that our ResVR outperforms existing methods in viewport rendering tasks across different fields of view, resolutions, and view directions while keeping a low transmission overhead.