Guided Transfer Learning
This addresses resource constraints in ML for practitioners, but it appears incremental as it builds on existing transfer learning methods.
The paper tackles the problem of high data and computational demands in machine learning by proposing guided transfer learning, which introduces guiding parameters for each network weight and bias to control changes during new task learning, resulting in reduced resource needs and enabling learning from small data or with smaller networks.
Machine learning requires exuberant amounts of data and computation. Also, models require equally excessive growth in the number of parameters. It is, therefore, sensible to look for technologies that reduce these demands on resources. Here, we propose an approach called guided transfer learning. Each weight and bias in the network has its own guiding parameter that indicates how much this parameter is allowed to change while learning a new task. Guiding parameters are learned during an initial scouting process. Guided transfer learning can result in a reduction in resources needed to train a network. In some applications, guided transfer learning enables the network to learn from a small amount of data. In other cases, a network with a smaller number of parameters can learn a task which otherwise only a larger network could learn. Guided transfer learning potentially has many applications when the amount of data, model size, or the availability of computational resources reach their limits.