Focus-and-Expand: Training Guidance Through Gradual Manipulation of Input Features
This addresses the need for more robust and accurate neural network solutions in domains like bias removal and image classification, though it appears incremental as it builds on Parameter Continuation methods.
The paper tackles the problem of guiding neural network training toward solutions that emphasize specific input features by proposing Focus-and-Expand (\fax), a method that gradually shifts the input domain from a subset of features to all features. The result includes state-of-the-art bias removal and improvements in computer vision tasks, though specific numerical gains are not detailed in the abstract.
We present a simple and intuitive Focus-and-eXpand (\fax) method to guide the training process of a neural network towards a specific solution. Optimizing a neural network is a highly non-convex problem. Typically, the space of solutions is large, with numerous possible local minima, where reaching a specific minimum depends on many factors. In many cases, however, a solution which considers specific aspects, or features, of the input is desired. For example, in the presence of bias, a solution that disregards the biased feature is a more robust and accurate one. Drawing inspiration from Parameter Continuation methods, we propose steering the training process to consider specific features in the input more than others, through gradual shifts in the input domain. \fax extracts a subset of features from each input data-point, and exposes the learner to these features first, Focusing the solution on them. Then, by using a blending/mixing parameter $α$ it gradually eXpands the learning process to include all features of the input. This process encourages the consideration of the desired features more than others. Though not restricted to this field, we quantitatively evaluate the effectiveness of our approach on various Computer Vision tasks, and achieve state-of-the-art bias removal, improvements to an established augmentation method, and two examples of improvements to image classification tasks. Through these few examples we demonstrate the impact this approach potentially carries for a wide variety of problems, which stand to gain from understanding the solution landscape.