Task Programming: Learning Data Efficient Behavior Representations
This work addresses the problem of reducing the burdensome and time-consuming annotation effort for domain experts in automated behavior analysis, particularly in fields like behavioral neuroscience.
This paper introduces TREBA, a multi-task self-supervised learning method that uses "task programming" to encode domain expert knowledge into tasks, rather than relying on extensive data annotation. Applied to behavioral neuroscience data (mice and fruit flies), TREBA reduced annotation burden by up to a factor of 10 while maintaining accuracy compared to state-of-the-art features.
Specialized domain knowledge is often necessary to accurately annotate training sets for in-depth analysis, but can be burdensome and time-consuming to acquire from domain experts. This issue arises prominently in automated behavior analysis, in which agent movements or actions of interest are detected from video tracking data. To reduce annotation effort, we present TREBA: a method to learn annotation-sample efficient trajectory embedding for behavior analysis, based on multi-task self-supervised learning. The tasks in our method can be efficiently engineered by domain experts through a process we call "task programming", which uses programs to explicitly encode structured knowledge from domain experts. Total domain expert effort can be reduced by exchanging data annotation time for the construction of a small number of programmed tasks. We evaluate this trade-off using data from behavioral neuroscience, in which specialized domain knowledge is used to identify behaviors. We present experimental results in three datasets across two domains: mice and fruit flies. Using embeddings from TREBA, we reduce annotation burden by up to a factor of 10 without compromising accuracy compared to state-of-the-art features. Our results thus suggest that task programming and self-supervision can be an effective way to reduce annotation effort for domain experts.