Learning from Higher-Layer Feature Visualizations
This work addresses the need for efficient sleep apnea detection on small devices, but it is incremental as it builds on existing knowledge transfer and visualization methods.
The paper tackled the problem of enabling sleep apnea monitoring on mobile devices by using interpretation-based indirect knowledge transfer, where a student classifier learns from synthetic data generated from a teacher network's feature visualizations, achieving 97.8% accuracy on MNIST and up to 89.5% on an Apnea-ECG dataset.
Driven by the goal to enable sleep apnea monitoring and machine learning-based detection at home with small mobile devices, we investigate whether interpretation-based indirect knowledge transfer can be used to create classifiers with acceptable performance. Interpretation-based indirect knowledge transfer means that a classifier (student) learns from a synthetic dataset based on the knowledge representation from an already trained Deep Network (teacher). We use activation maximization to generate visualizations and create a synthetic dataset to train the student classifier. This approach has the advantage that student classifiers can be trained without access to the original training data. With experiments we investigate the feasibility of interpretation-based indirect knowledge transfer and its limitations. The student achieves an accuracy of 97.8% on MNIST (teacher accuracy: 99.3%) with a similar smaller architecture to that of the teacher. The student classifier achieves an accuracy of 86.1% and 89.5% for a subset of the Apnea-ECG dataset (teacher: 89.5% and 91.1%, respectively).