Representation learning with function call graph transformations for malware open set recognition
This addresses the challenge of classifying known malware families while detecting unknown ones in security applications, representing an incremental advancement.
The paper tackles the open set recognition problem in malware classification by introducing a self-supervised pre-training approach with function call graph transformations and a statistical thresholding method, resulting in improved performance across different downstream loss functions.
Open set recognition (OSR) problem has been a challenge in many machine learning (ML) applications, such as security. As new/unknown malware families occur regularly, it is difficult to exhaust samples that cover all the classes for the training process in ML systems. An advanced malware classification system should classify the known classes correctly while sensitive to the unknown class. In this paper, we introduce a self-supervised pre-training approach for the OSR problem in malware classification. We propose two transformations for the function call graph (FCG) based malware representations to facilitate the pretext task. Also, we present a statistical thresholding approach to find the optimal threshold for the unknown class. Moreover, the experiment results indicate that our proposed pre-training process can improve different performances of different downstream loss functions for the OSR problem.