Hebbian learning inspired estimation of the linear regression parameters from queries
This work addresses a theoretical bottleneck in zeroth-order optimization for machine learning, with incremental improvements in query efficiency.
The paper tackles the problem of estimating linear regression parameters using only loss function queries, showing that a Hebbian learning rule achieves near-optimal rates and outperforms non-adaptive methods.
Local learning rules in biological neural networks (BNNs) are commonly referred to as Hebbian learning. [26] links a biologically motivated Hebbian learning rule to a specific zeroth-order optimization method. In this work, we study a variation of this Hebbian learning rule to recover the regression vector in the linear regression model. Zeroth-order optimization methods are known to converge with suboptimal rate for large parameter dimension compared to first-order methods like gradient descent, and are therefore thought to be in general inferior. By establishing upper and lower bounds, we show, however, that such methods achieve near-optimal rates if only queries of the linear regression loss are available. Moreover, we prove that this Hebbian learning rule can achieve considerably faster rates than any non-adaptive method that selects the queries independently of the data.