Balancing Sparse RNNs with Hyperparameterization Benefiting Meta-Learning
This work addresses model optimization for sparse RNNs in meta-learning, though it appears incremental as it builds on existing sparse RNN methods with new hyperparameters and metrics.
The paper tackles the problem of optimizing sparse Recurrent Neural Networks (RNNs) by introducing alternative hyperparameters that vary sparsity in weight matrices, combined with a novel hidden proportion metric to balance unknowns. This approach yields significant performance gains and improves a priori performance expectations for meta-learning applications.
This paper develops alternative hyperparameters for specifying sparse Recurrent Neural Networks (RNNs). These hyperparameters allow for varying sparsity within the trainable weight matrices of the model while improving overall performance. This architecture enables the definition of a novel metric, hidden proportion, which seeks to balance the distribution of unknowns within the model and provides significant explanatory power of model performance. Together, the use of the varied sparsity RNN architecture combined with the hidden proportion metric generates significant performance gains while improving performance expectations on an a priori basis. This combined approach provides a path forward towards generalized meta-learning applications and model optimization based on intrinsic characteristics of the data set, including input and output dimensions.