Correlation-based construction of neighborhood and edge features
This work addresses feature engineering for classification tasks, offering an incremental improvement over existing methods like boosting on raw pixels or Haar filters.
The paper tackles unsupervised feature construction by grouping correlated features into neighborhoods and edges, and validates it on multi-class classification problems, achieving a test error of 0.94% on MNIST and outperforming boosting on raw pixels and Haar filters on CIFAR-10.
Motivated by an abstract notion of low-level edge detector filters, we propose a simple method of unsupervised feature construction based on pairwise statistics of features. In the first step, we construct neighborhoods of features by regrouping features that correlate. Then we use these subsets as filters to produce new neighborhood features. Next, we connect neighborhood features that correlate, and construct edge features by subtracting the correlated neighborhood features of each other. To validate the usefulness of the constructed features, we ran AdaBoost.MH on four multi-class classification problems. Our most significant result is a test error of 0.94% on MNIST with an algorithm which is essentially free of any image-specific priors. On CIFAR-10 our method is suboptimal compared to today's best deep learning techniques, nevertheless, we show that the proposed method outperforms not only boosting on the raw pixels, but also boosting on Haar filters.