Coarse or Fine? Recognising Action End States without Labels
This addresses a specific computer vision problem for action understanding, but it is incremental as it builds on existing datasets and methods.
The paper tackles the problem of recognizing the end state of a cutting action in images, specifically predicting whether an object was cut coarsely or finely, by proposing an augmentation method to synthesize training data from less than a hundred images, achieving successful recognition and generalization to unseen objects despite a domain gap.
We focus on the problem of recognising the end state of an action in an image, which is critical for understanding what action is performed and in which manner. We study this focusing on the task of predicting the coarseness of a cut, i.e., deciding whether an object was cut "coarsely" or "finely". No dataset with these annotated end states is available, so we propose an augmentation method to synthesise training data. We apply this method to cutting actions extracted from an existing action recognition dataset. Our method is object agnostic, i.e., it presupposes the location of the object but not its identity. Starting from less than a hundred images of a whole object, we can generate several thousands images simulating visually diverse cuts of different coarseness. We use our synthetic data to train a model based on UNet and test it on real images showing coarsely/finely cut objects. Results demonstrate that the model successfully recognises the end state of the cutting action despite the domain gap between training and testing, and that the model generalises well to unseen objects.