On the Bias Against Inductive Biases
This work addresses the challenge for typical AI researchers who lack resources to train large transformer models, offering insights into more accessible network designs.
The paper tackles the problem of understanding the impact of inductive biases in small to moderately-sized isotropic networks for unsupervised visual feature learning, showing that removing these biases is not always ideal, with results indicating performance improvements in specific tasks.
Borrowing from the transformer models that revolutionized the field of natural language processing, self-supervised feature learning for visual tasks has also seen state-of-the-art success using these extremely deep, isotropic networks. However, the typical AI researcher does not have the resources to evaluate, let alone train, a model with several billion parameters and quadratic self-attention activations. To facilitate further research, it is necessary to understand the features of these huge transformer models that can be adequately studied by the typical researcher. One interesting characteristic of these transformer models is that they remove most of the inductive biases present in classical convolutional networks. In this work, we analyze the effect of these and more inductive biases on small to moderately-sized isotropic networks used for unsupervised visual feature learning and show that their removal is not always ideal.