BenchSeg: A Large-Scale Dataset and Benchmark for Multi-View Food Video Segmentation
This work addresses the need for better dietary analysis tools by providing a dataset and benchmark for multi-view food segmentation, though it is incremental as it builds on existing datasets and methods.
The authors tackled the problem of limited multi-view data and poor generalization in food image segmentation by introducing BenchSeg, a large-scale multi-view food video dataset and benchmark, and demonstrated that memory-augmented methods improve performance, with their best model outperforming prior work by ~2.63% mAP.
Food image segmentation is a critical task for dietary analysis, enabling accurate estimation of food volume and nutrients. However, current methods suffer from limited multi-view data and poor generalization to new viewpoints. We introduce BenchSeg, a novel multi-view food video segmentation dataset and benchmark. BenchSeg aggregates 55 dish scenes (from Nutrition5k, Vegetables & Fruits, MetaFood3D, and FoodKit) with 25,284 meticulously annotated frames, capturing each dish under free 360° camera motion. We evaluate a diverse set of 20 state-of-the-art segmentation models (e.g., SAM-based, transformer, CNN, and large multimodal) on the existing FoodSeg103 dataset and evaluate them (alone and combined with video-memory modules) on BenchSeg. Quantitative and qualitative results demonstrate that while standard image segmenters degrade sharply under novel viewpoints, memory-augmented methods maintain temporal consistency across frames. Our best model based on a combination of SeTR-MLA+XMem2 outperforms prior work (e.g., improving over FoodMem by ~2.63% mAP), offering new insights into food segmentation and tracking for dietary analysis. We release BenchSeg to foster future research. The project page including the dataset annotations and the food segmentation models can be found at https://amughrabi.github.io/benchseg.