Stable Distribution Alignment Using the Dual of the Adversarial Distance
This work addresses optimization challenges in adversarial methods for distribution alignment, offering a more stable alternative for tasks like domain adaptation, though it is incremental as it builds on existing adversarial frameworks.
The paper tackles the instability and convergence issues in adversarial distribution alignment methods by proposing a dual formulation that replaces the maximization part with its dual, leading to more stable and monotonic improvement in aligning synthetic point clouds and real-image domain adaptation tasks.
Methods that align distributions by minimizing an adversarial distance between them have recently achieved impressive results. However, these approaches are difficult to optimize with gradient descent and they often do not converge well without careful hyperparameter tuning and proper initialization. We investigate whether turning the adversarial min-max problem into an optimization problem by replacing the maximization part with its dual improves the quality of the resulting alignment and explore its connections to Maximum Mean Discrepancy. Our empirical results suggest that using the dual formulation for the restricted family of linear discriminators results in a more stable convergence to a desirable solution when compared with the performance of a primal min-max GAN-like objective and an MMD objective under the same restrictions. We test our hypothesis on the problem of aligning two synthetic point clouds on a plane and on a real-image domain adaptation problem on digits. In both cases, the dual formulation yields an iterative procedure that gives more stable and monotonic improvement over time.