WeNet: Weighted Networks for Recurrent Network Architecture Search
This addresses the need for efficient architecture search in deep learning, particularly for recurrent networks, though it appears incremental as it builds on existing concepts like mixture of experts.
The paper tackles the problem of automatic architecture search in deep learning by proposing WeNet, a method using weighted networks to find recurrent architectures, achieving state-of-the-art performance on Penn Treebank and WikiText-2 datasets.
In recent years, there has been increasing demand for automatic architecture search in deep learning. Numerous approaches have been proposed and led to state-of-the-art results in various applications, including image classification and language modeling. In this paper, we propose a novel way of architecture search by means of weighted networks (WeNet), which consist of a number of networks, with each assigned a weight. These weights are updated with back-propagation to reflect the importance of different networks. Such weighted networks bear similarity to mixture of experts. We conduct experiments on Penn Treebank and WikiText-2. We show that the proposed WeNet can find recurrent architectures which result in state-of-the-art performance.