TRESTLE: Toolkit for Reproducible Execution of Speech, Text and Language Experiments
This addresses reproducibility issues for researchers in computational linguistics and healthcare AI, though it is incremental as it builds on existing datasets and methods.
The paper tackles the problem of inconsistent and non-comparable results in computational studies of cognitive impairment detection from language data by introducing TRESTLE, an open-source toolkit that standardizes data pre-processing and selection for reproducible experiments, successfully deployed in a hackathon at AAAI 2022.
The evidence is growing that machine and deep learning methods can learn the subtle differences between the language produced by people with various forms of cognitive impairment such as dementia and cognitively healthy individuals. Valuable public data repositories such as TalkBank have made it possible for researchers in the computational community to join forces and learn from each other to make significant advances in this area. However, due to variability in approaches and data selection strategies used by various researchers, results obtained by different groups have been difficult to compare directly. In this paper, we present TRESTLE (\textbf{T}oolkit for \textbf{R}eproducible \textbf{E}xecution of \textbf{S}peech \textbf{T}ext and \textbf{L}anguage \textbf{E}xperiments), an open source platform that focuses on two datasets from the TalkBank repository with dementia detection as an illustrative domain. Successfully deployed in the hackallenge (Hackathon/Challenge) of the International Workshop on Health Intelligence at AAAI 2022, TRESTLE provides a precise digital blueprint of the data pre-processing and selection strategies that can be reused via TRESTLE by other researchers seeking comparable results with their peers and current state-of-the-art (SOTA) approaches.