Deep Learning-Driven Multimodal Detection and Movement Analysis of Objects in Culinary
This work addresses the need for automated cooking assistance in daily life, though it is incremental as it fine-tunes existing models rather than introducing fundamentally new approaches.
The researchers tackled the problem of automatically generating cooking instructions by developing a multimodal system that combines YOLOv8 segmentation, LSTM motion analysis, and ASR with TinyLLaMa to predict recipes and create step-by-step guides from video data, achieving a robust task-specific system for complex kitchen environments.
This is a research exploring existing models and fine tuning them to combine a YOLOv8 segmentation model, a LSTM model trained on hand point motion sequence and a ASR (whisper-base) to extract enough data for a LLM (TinyLLaMa) to predict the recipe and generate text creating a step by step guide for the cooking procedure. All the data were gathered by the author for a robust task specific system to perform best in complex and challenging environments proving the extension and endless application of computer vision in daily activities such as kitchen work. This work extends the field for many more crucial task of our day to day life.