CODA: Constructivism Learning for Instance-Dependent Dropout Architecture Construction
This work addresses an incremental improvement in dropout techniques for deep learning applications, potentially benefiting domains where instance-specific structural information is critical.
The paper tackles the problem of existing dropout methods failing to differentiate among instances when constructing dropout architectures, which is a deficiency for many applications. The proposed CODA method, inspired by constructivism learning and using Uniform Process Mixture Models, was evaluated on 5 real-world datasets and demonstrated effectiveness compared to state-of-the-art dropout techniques.
Dropout is attracting intensive research interest in deep learning as an efficient approach to prevent overfitting. Recently incorporating structural information when deciding which units to drop out produced promising results comparing to methods that ignore the structural information. However, a major issue of the existing work is that it failed to differentiate among instances when constructing the dropout architecture. This can be a significant deficiency for many applications. To solve this issue, we propose Constructivism learning for instance-dependent Dropout Architecture (CODA), which is inspired from a philosophical theory, constructivism learning. Specially, based on the theory we have designed a better drop out technique, Uniform Process Mixture Models, using a Bayesian nonparametric method Uniform process. We have evaluated our proposed method on 5 real-world datasets and compared the performance with other state-of-the-art dropout techniques. The experimental results demonstrated the effectiveness of CODA.