Abstract Learning via Demodulation in a Deep Neural Network
This provides foundational insights into abstract learning in DNNs, potentially explaining similar processes in the human brain.
The paper tackles the problem of understanding how deep neural networks learn abstract representations, demonstrating that they do so through a process of demodulation, with results showing improved performance when optimized for this process.
Inspired by the brain, deep neural networks (DNN) are thought to learn abstract representations through their hierarchical architecture. However, at present, how this happens is not well understood. Here, we demonstrate that DNN learn abstract representations by a process of demodulation. We introduce a biased sigmoid activation function and use it to show that DNN learn and perform better when optimized for demodulation. Our findings constitute the first unambiguous evidence that DNN perform abstract learning in practical use. Our findings may also explain abstract learning in the human brain.