HealthSLM-Bench: Benchmarking Small Language Models for Mobile and Wearable Healthcare Monitoring
This work addresses privacy and efficiency issues in healthcare monitoring for mobile and wearable device users, but it is incremental as it benchmarks existing SLMs rather than introducing new methods.
The authors tackled the problem of evaluating small language models (SLMs) for mobile and wearable healthcare monitoring, finding that SLMs achieve performance comparable to large language models with substantial gains in efficiency and privacy, though challenges like class imbalance persist.
Mobile and wearable healthcare monitoring play a vital role in facilitating timely interventions, managing chronic health conditions, and ultimately improving individuals' quality of life. Previous studies on large language models (LLMs) have highlighted their impressive generalization abilities and effectiveness in healthcare prediction tasks. However, most LLM-based healthcare solutions are cloud-based, which raises significant privacy concerns and results in increased memory usage and latency. To address these challenges, there is growing interest in compact models, Small Language Models (SLMs), which are lightweight and designed to run locally and efficiently on mobile and wearable devices. Nevertheless, how well these models perform in healthcare prediction remains largely unexplored. We systematically evaluated SLMs on health prediction tasks using zero-shot, few-shot, and instruction fine-tuning approaches, and deployed the best performing fine-tuned SLMs on mobile devices to evaluate their real-world efficiency and predictive performance in practical healthcare scenarios. Our results show that SLMs can achieve performance comparable to LLMs while offering substantial gains in efficiency and privacy. However, challenges remain, particularly in handling class imbalance and few-shot scenarios. These findings highlight SLMs, though imperfect in their current form, as a promising solution for next-generation, privacy-preserving healthcare monitoring.