Growing Neural Network with Shared Parameter
This addresses parameter inefficiency in neural networks for AI practitioners, though it appears incremental as it builds on existing network growth and transfer learning concepts.
The paper tackles the problem of inefficient parameter usage in neural networks by proposing a method to grow networks with shared parameters, matching trained subnetworks to new inputs. The method improved performance with higher parameter efficiency and enabled transfer learning across tasks without retraining.
We propose a general method for growing neural network with shared parameter by matching trained network to new input. By leveraging Hoeffding's inequality, we provide a theoretical base for improving performance by adding subnetwork to existing network. With the theoretical base of adding new subnetwork, we implement a matching method to apply trained subnetwork of existing network to new input. Our method has shown the ability to improve performance with higher parameter efficiency. It can also be applied to trans-task case and realize transfer learning by changing the combination of subnetworks without training on new task.