On Random Weights for Texture Generation in One Layer Neural Networks
This work addresses a theoretical gap in understanding texture generation mechanisms for researchers in computer vision and neural networks, but it is incremental as it builds on prior experimental findings.
The paper investigates why one-layer convolutional neural networks with random filters can generate textures, showing theoretically that such architectures without non-linearity preserve and modulate frequency coefficients similarly to pretrained weights, and that adding a ReLU non-linearity can reduce variability by limiting solutions.
Recent work in the literature has shown experimentally that one can use the lower layers of a trained convolutional neural network (CNN) to model natural textures. More interestingly, it has also been experimentally shown that only one layer with random filters can also model textures although with less variability. In this paper we ask the question as to why one layer CNNs with random filters are so effective in generating textures? We theoretically show that one layer convolutional architectures (without a non-linearity) paired with the an energy function used in previous literature, can in fact preserve and modulate frequency coefficients in a manner so that random weights and pretrained weights will generate the same type of images. Based on the results of this analysis we question whether similar properties hold in the case where one uses one convolution layer with a non-linearity. We show that in the case of ReLu non-linearity there are situations where only one input will give the minimum possible energy whereas in the case of no nonlinearity, there are always infinite solutions that will give the minimum possible energy. Thus we can show that in certain situations adding a ReLu non-linearity generates less variable images.