Learned Single-Pass Multitasking Perceptual Graphics for Immersive Displays
This addresses the problem of computational efficiency for immersive display applications, offering a flexible solution for dynamic creative processes, though it is incremental in combining existing perceptual graphics methods with text guidance.
The paper tackles the challenge of running multiple perceptual graphics methods on resource-constrained devices by proposing a learned single-pass multitasking model that uses text prompts to perform tasks like foveated rendering and denoising, achieving consistent high quality with reasonable compute as validated by user studies.
Emerging immersive display technologies efficiently utilize resources with perceptual graphics methods such as foveated rendering and denoising. Running multiple perceptual graphics methods challenges devices with limited power and computational resources. We propose a computationally-lightweight learned multitasking perceptual graphics model. Given RGB images and text-prompts, our model performs text-described perceptual tasks in a single inference step. Simply daisy-chaining multiple models or training dedicated models can lead to model management issues and exhaust computational resources. In contrast, our flexible method unlocks consistent high quality perceptual effects with reasonable compute, supporting various permutations at varied intensities using adjectives in text prompts (e.g. mildly, lightly). Text-guidance provides ease of use for dynamic requirements such as creative processes. To train our model, we propose a dataset containing source and perceptually enhanced images with corresponding text prompts. We evaluate our model on desktop and embedded platforms and validate perceptual quality through a user study.