PulmoFusion: Advancing Pulmonary Health with Efficient Multi-Modal Fusion
This work addresses pulmonary health monitoring for patients, offering an incremental improvement through efficient multimodal fusion.
The paper tackles the problem of imprecise remote spirometry by proposing a non-invasive multimodal approach using RGB or thermal video with patient metadata, achieving 92% accuracy on breathing cycles and 99.5% patient-wise, with state-of-the-art performance in PEF regression and FEV1/FVC predictions.
Traditional remote spirometry lacks the precision required for effective pulmonary monitoring. We present a novel, non-invasive approach using multimodal predictive models that integrate RGB or thermal video data with patient metadata. Our method leverages energy-efficient Spiking Neural Networks (SNNs) for the regression of Peak Expiratory Flow (PEF) and classification of Forced Expiratory Volume (FEV1) and Forced Vital Capacity (FVC), using lightweight CNNs to overcome SNN limitations in regression tasks. Multimodal data integration is improved with a Multi-Head Attention Layer, and we employ K-Fold validation and ensemble learning to boost robustness. Using thermal data, our SNN models achieve 92% accuracy on a breathing-cycle basis and 99.5% patient-wise. PEF regression models attain Relative RMSEs of 0.11 (thermal) and 0.26 (RGB), with an MAE of 4.52% for FEV1/FVC predictions, establishing state-of-the-art performance. Code and dataset can be found on https://github.com/ahmed-sharshar/RespiroDynamics.git