MetaCLBench: Meta Continual Learning Benchmark on Resource-Constrained Edge Devices
This provides a benchmark for researchers and practitioners implementing Meta-CL on resource-constrained edge devices, though it is incremental as it extends existing methods to new data types.
The paper tackled the lack of evaluation of Meta-Continual Learning methods for sequential time-series data like audio on edge devices, finding that while these methods can learn new classes for both image and audio modalities, they impose significant computational and memory costs, with pre-training and meta-training improving performance.
Meta-Continual Learning (Meta-CL) has emerged as a promising approach to minimize manual labeling efforts and system resource requirements by enabling Continual Learning (CL) with limited labeled samples. However, while existing methods have shown success in image-based tasks, their effectiveness remains unexplored for sequential time-series data from sensor systems, particularly audio inputs. To address this gap, we conduct a comprehensive benchmark study evaluating six representative Meta-CL approaches using three network architectures on five datasets from both image and audio modalities. We develop MetaCLBench, an end-to-end Meta-CL benchmark framework for edge devices to evaluate system overheads and investigate trade-offs among performance, computational costs, and memory requirements across various Meta-CL methods. Our results reveal that while many Meta-CL methods enable to learn new classes for both image and audio modalities, they impose significant computational and memory costs on edge devices. Also, we find that pre-training and meta-training procedures based on source data before deployment improve Meta-CL performance. Finally, to facilitate further research, we provide practical guidelines for researchers and machine learning practitioners implementing Meta-CL on resource-constrained environments and make our benchmark framework and tools publicly available, enabling fair evaluation across both accuracy and system-level metrics.