Private Online Prediction from Experts: Separations and Faster Rates
This work addresses privacy-preserving online learning for experts, showing fundamental separations in regret under different adversary models and privacy types, which is incremental but provides key theoretical insights for the field.
The paper tackles the problem of online prediction from experts under privacy constraints, achieving improved regret bounds for non-adaptive adversaries with algorithms that provide $ ilde{O}(\sqrt{T \log d} + \log d/\varepsilon)$ for stochastic settings and $ ilde{O}(\sqrt{T \log d} + T^{1/3} \log d/\varepsilon)$ for oblivious adversaries, and proving new lower bounds that reveal separations between adaptive and non-adaptive adversaries and between pure and approximate differential privacy.
Online prediction from experts is a fundamental problem in machine learning and several works have studied this problem under privacy constraints. We propose and analyze new algorithms for this problem that improve over the regret bounds of the best existing algorithms for non-adaptive adversaries. For approximate differential privacy, our algorithms achieve regret bounds of $\tilde{O}(\sqrt{T \log d} + \log d/\varepsilon)$ for the stochastic setting and $\tilde{O}(\sqrt{T \log d} + T^{1/3} \log d/\varepsilon)$ for oblivious adversaries (where $d$ is the number of experts). For pure DP, our algorithms are the first to obtain sub-linear regret for oblivious adversaries in the high-dimensional regime $d \ge T$. Moreover, we prove new lower bounds for adaptive adversaries. Our results imply that unlike the non-private setting, there is a strong separation between the optimal regret for adaptive and non-adaptive adversaries for this problem. Our lower bounds also show a separation between pure and approximate differential privacy for adaptive adversaries where the latter is necessary to achieve the non-private $O(\sqrt{T})$ regret.