Do autoencoders need a bottleneck for anomaly detection?
This work addresses a fundamental design assumption in autoencoders for anomaly detection, potentially improving performance in applications like security or quality control.
The paper challenges the belief that autoencoders need a bottleneck for anomaly detection by showing that non-bottlenecked architectures, such as overparameterized or skip-connected ones, can outperform bottlenecked versions, achieving an AUROC of 0.857 vs. 0.696 on CIFAR vs. SVHN tasks.
A common belief in designing deep autoencoders (AEs), a type of unsupervised neural network, is that a bottleneck is required to prevent learning the identity function. Learning the identity function renders the AEs useless for anomaly detection. In this work, we challenge this limiting belief and investigate the value of non-bottlenecked AEs. The bottleneck can be removed in two ways: (1) overparameterising the latent layer, and (2) introducing skip connections. However, limited works have reported on the use of one of the ways. For the first time, we carry out extensive experiments covering various combinations of bottleneck removal schemes, types of AEs and datasets. In addition, we propose the infinitely-wide AEs as an extreme example of non-bottlenecked AEs. Their improvement over the baseline implies learning the identity function is not trivial as previously assumed. Moreover, we find that non-bottlenecked architectures (highest AUROC=0.857) can outperform their bottlenecked counterparts (highest AUROC=0.696) on the popular task of CIFAR (inliers) vs SVHN (anomalies), among other tasks, shedding light on the potential of developing non-bottlenecked AEs for improving anomaly detection.