Adversarially Contrastive Estimation of Conditional Neural Processes
This work addresses a bottleneck in CNPs for researchers in machine learning, offering an incremental improvement by enhancing expressivity while preserving foundational properties.
The paper tackles the limited expressivity of Conditional Neural Processes (CNPs) for high-dimensional observations by proposing an adversarial training scheme that calibrates CNPs with an energy-based model using noise contrastive estimation, resulting in improved predictions without significant computational overhead.
Conditional Neural Processes~(CNPs) formulate distributions over functions and generate function observations with exact conditional likelihoods. CNPs, however, have limited expressivity for high-dimensional observations, since their predictive distribution is factorized into a product of unconstrained (typically) Gaussian outputs. Previously, this could be handled using latent variables or autoregressive likelihood, but at the expense of intractable training and quadratically increased complexity. Instead, we propose calibrating CNPs with an adversarial training scheme besides regular maximum likelihood estimates. Specifically, we train an energy-based model (EBM) with noise contrastive estimation, which enforces EBM to identify true observations from the generations of CNP. In this way, CNP must generate predictions closer to the ground-truth to fool EBM, instead of merely optimizing with respect to the fixed-form likelihood. From generative function reconstruction to downstream regression and classification tasks, we demonstrate that our method fits mainstream CNP members, showing effectiveness when unconstrained Gaussian likelihood is defined, requiring minimal computation overhead while preserving foundation properties of CNPs.