No Free Lunch in Annotation either: An objective evaluation of foundation models for streamlining annotation in animal tracking
This addresses the annotation bottleneck for researchers in animal behavior tracking, showing an incremental but practical improvement over blind automation.
The paper tackles the problem of using foundation models to automate annotation for animal tracking, finding that automated annotations alone yield poor results (IDF1 score 65.6), but a thoughtful combination with manual annotations significantly improves performance (IDF1 score 80.8).
We analyze the capabilities of foundation models addressing the tedious task of generating annotations for animal tracking. Annotating a large amount of data is vital and can be a make-or-break factor for the robustness of a tracking model. Robustness is particularly crucial in animal tracking, as accurate tracking over long time horizons is essential for capturing the behavior of animals. However, generating additional annotations using foundation models can be counterproductive, as the quality of the annotations is just as important. Poorly annotated data can introduce noise and inaccuracies, ultimately compromising the performance and accuracy of the trained model. Over-reliance on automated annotations without ensuring precision can lead to diminished results, making careful oversight and quality control essential in the annotation process. Ultimately, we demonstrate that a thoughtful combination of automated annotations and manually annotated data is a valuable strategy, yielding an IDF1 score of 80.8 against blind usage of SAM2 video with an IDF1 score of 65.6.