Forward-Backward Greedy Algorithms for General Convex Smooth Functions over A Cardinality Constraint
This work addresses scalability issues in sparse feature selection algorithms for researchers and practitioners in machine learning, though it is incremental as it builds on existing greedy methods with improved theoretical bounds.
The paper tackles the inefficiency of forward-backward greedy algorithms for sparse feature selection by analyzing FoBa-gdt, which uses gradient information to improve scalability, and shows it achieves the same theoretical performance as FoBa-obj under restricted strong convexity conditions, with applications in sensor selection for activity recognition where it outperforms other methods.
We consider forward-backward greedy algorithms for solving sparse feature selection problems with general convex smooth functions. A state-of-the-art greedy method, the Forward-Backward greedy algorithm (FoBa-obj) requires to solve a large number of optimization problems, thus it is not scalable for large-size problems. The FoBa-gdt algorithm, which uses the gradient information for feature selection at each forward iteration, significantly improves the efficiency of FoBa-obj. In this paper, we systematically analyze the theoretical properties of both forward-backward greedy algorithms. Our main contributions are: 1) We derive better theoretical bounds than existing analyses regarding FoBa-obj for general smooth convex functions; 2) We show that FoBa-gdt achieves the same theoretical performance as FoBa-obj under the same condition: restricted strong convexity condition. Our new bounds are consistent with the bounds of a special case (least squares) and fills a previously existing theoretical gap for general convex smooth functions; 3) We show that the restricted strong convexity condition is satisfied if the number of independent samples is more than $\bar{k}\log d$ where $\bar{k}$ is the sparsity number and $d$ is the dimension of the variable; 4) We apply FoBa-gdt (with the conditional random field objective) to the sensor selection problem for human indoor activity recognition and our results show that FoBa-gdt outperforms other methods (including the ones based on forward greedy selection and L1-regularization).