Natural Language Robot Programming: NLP integrated with autonomous robotic grasping
This addresses the problem of intuitive robot programming for users in robotics, but it is incremental as it builds on existing natural language and grasping methods.
The paper tackles robot programming for pick-and-place tasks by developing a grammar-based natural language framework that uses a custom dictionary for action words, validated through simulation and real-world experiments with a robotic arm, achieving a high system usability score.
In this paper, we present a grammar-based natural language framework for robot programming, specifically for pick-and-place tasks. Our approach uses a custom dictionary of action words, designed to store together words that share meaning, allowing for easy expansion of the vocabulary by adding more action words from a lexical database. We validate our Natural Language Robot Programming (NLRP) framework through simulation and real-world experimentation, using a Franka Panda robotic arm equipped with a calibrated camera-in-hand and a microphone. Participants were asked to complete a pick-and-place task using verbal commands, which were converted into text using Google's Speech-to-Text API and processed through the NLRP framework to obtain joint space trajectories for the robot. Our results indicate that our approach has a high system usability score. The framework's dictionary can be easily extended without relying on transfer learning or large data sets. In the future, we plan to compare the presented framework with different approaches of human-assisted pick-and-place tasks via a comprehensive user study.