Sparsely Grouped Input Variables for Neural Networks
This work addresses the need for efficient and robust neural network training in domains like genomics and image recognition by enabling group-level variable elimination, though it is incremental as it builds on existing group sparsity methods.
The paper tackled the problem of selecting sparse groups of input variables in neural networks to reduce data acquisition costs and prevent overfitting, achieving group sparsity by excluding 89.9% of stimuli in an eye-tracking experiment and 60% of image rows in MNIST while maintaining satisfactory results.
In genomic analysis, biomarker discovery, image recognition, and other systems involving machine learning, input variables can often be organized into different groups by their source or semantic category. Eliminating some groups of variables can expedite the process of data acquisition and avoid over-fitting. Researchers have used the group lasso to ensure group sparsity in linear models and have extended it to create compact neural networks in meta-learning. Different from previous studies, we use multi-layer non-linear neural networks to find sparse groups for input variables. We propose a new loss function to regularize parameters for grouped input variables, design a new optimization algorithm for this loss function, and test these methods in three real-world settings. We achieve group sparsity for three datasets, maintaining satisfying results while excluding one nucleotide position from an RNA splicing experiment, excluding 89.9% of stimuli from an eye-tracking experiment, and excluding 60% of image rows from an experiment on the MNIST dataset.