Adversarial Representation Learning With Closed-Form Solvers
This work addresses a practical optimization problem in adversarial representation learning for applications like fair classification and privacy protection, though it is incremental as it builds on existing methods with improved stability and efficiency.
The paper tackles the instability and unreliability of adversarial representation learning by proposing OptNet-ARL, which uses closed-form solvers to learn representations that remove sensitive information while maintaining utility for target tasks. Results show it is stable, converges 3-5 times faster, and offers better trade-offs in utility-bias and utility-privacy scenarios compared to existing methods.
Adversarial representation learning aims to learn data representations for a target task while removing unwanted sensitive information at the same time. Existing methods learn model parameters iteratively through stochastic gradient descent-ascent, which is often unstable and unreliable in practice. To overcome this challenge, we adopt closed-form solvers for the adversary and target task. We model them as kernel ridge regressors and analytically determine an upper-bound on the optimal dimensionality of representation. Our solution, dubbed OptNet-ARL, reduces to a stable one one-shot optimization problem that can be solved reliably and efficiently. OptNet-ARL can be easily generalized to the case of multiple target tasks and sensitive attributes. Numerical experiments, on both small and large scale datasets, show that, from an optimization perspective, OptNet-ARL is stable and exhibits three to five times faster convergence. Performance wise, when the target and sensitive attributes are dependent, OptNet-ARL learns representations that offer a better trade-off front between (a) utility and bias for fair classification and (b) utility and privacy by mitigating leakage of private information than existing solutions.