Multiple output samples per input in a single-output Gaussian process
This addresses the problem of incorporating output uncertainty from multiple annotations in subjective tasks for researchers and practitioners using Gaussian Processes, though it is incremental as it builds on standard GP methods.
The paper tackles the problem of Gaussian Processes (GP) being limited to single output samples per input by proposing a generalization that accommodates multiple output samples, such as from multiple human raters in subjective tasks like spoken language assessment. The result shows that this approach allows the GP to compute test set output distributions more similar to reference outputs from multiple raters, as evaluated on the speechocean762 dataset.
The standard Gaussian Process (GP) only considers a single output sample per input in the training set. Datasets for subjective tasks, such as spoken language assessment, may be annotated with output labels from multiple human raters per input. This paper proposes to generalise the GP to allow for these multiple output samples in the training set, and thus make use of available output uncertainty information. This differs from a multi-output GP, as all output samples are from the same task here. The output density function is formulated to be the joint likelihood of observing all output samples, and latent variables are not repeated to reduce computation cost. The test set predictions are inferred similarly to a standard GP, with a difference being in the optimised hyper-parameters. This is evaluated on speechocean762, showing that it allows the GP to compute a test set output distribution that is more similar to the collection of reference outputs from the multiple human raters.