Prototypical Contrastive Transfer Learning for Multimodal Language Understanding
This addresses the challenge of smooth human-robot interaction in domestic settings, though it appears incremental as it builds on existing transfer learning frameworks.
The paper tackled the problem of domestic service robots struggling to identify target objects from natural language instructions by proposing Prototypical Contrastive Transfer Learning (PCTL), which achieved 78.1% accuracy compared to 73.4% for simple fine-tuning.
Although domestic service robots are expected to assist individuals who require support, they cannot currently interact smoothly with people through natural language. For example, given the instruction "Bring me a bottle from the kitchen," it is difficult for such robots to specify the bottle in an indoor environment. Most conventional models have been trained on real-world datasets that are labor-intensive to collect, and they have not fully leveraged simulation data through a transfer learning framework. In this study, we propose a novel transfer learning approach for multimodal language understanding called Prototypical Contrastive Transfer Learning (PCTL), which uses a new contrastive loss called Dual ProtoNCE. We introduce PCTL to the task of identifying target objects in domestic environments according to free-form natural language instructions. To validate PCTL, we built new real-world and simulation datasets. Our experiment demonstrated that PCTL outperformed existing methods. Specifically, PCTL achieved an accuracy of 78.1%, whereas simple fine-tuning achieved an accuracy of 73.4%.