Super-Resolution Appearance Transfer for 4D Human Performances
This addresses the issue of poor rendering quality in 4D human performance capture for applications like virtual reality or film, but it is incremental as it builds on existing capture and super-resolution techniques.
The paper tackles the problem of low-quality dynamic textures in 4D human performance reconstruction from multi-view video, which arises due to small person occupancy in large capture volumes, by proposing a super-resolution appearance transfer method from high-resolution static captures, resulting in significant qualitative and quantitative improvements in rendering.
A common problem in the 4D reconstruction of people from multi-view video is the quality of the captured dynamic texture appearance which depends on both the camera resolution and capture volume. Typically the requirement to frame cameras to capture the volume of a dynamic performance ($>50m^3$) results in the person occupying only a small proportion $<$ 10% of the field of view. Even with ultra high-definition 4k video acquisition this results in sampling the person at less-than standard definition 0.5k video resolution resulting in low-quality rendering. In this paper we propose a solution to this problem through super-resolution appearance transfer from a static high-resolution appearance capture rig using digital stills cameras ($> 8k$) to capture the person in a small volume ($<8m^3$). A pipeline is proposed for super-resolution appearance transfer from high-resolution static capture to dynamic video performance capture to produce super-resolution dynamic textures. This addresses two key problems: colour mapping between different camera systems; and dynamic texture map super-resolution using a learnt model. Comparative evaluation demonstrates a significant qualitative and quantitative improvement in rendering the 4D performance capture with super-resolution dynamic texture appearance. The proposed approach reproduces the high-resolution detail of the static capture whilst maintaining the appearance dynamics of the captured video.