Small Effect Sizes in Malware Detection? Make Harder Train/Test Splits!
This addresses the challenge for industry practitioners who need to detect small effect sizes in malware detection but are constrained by small academic datasets, though it is incremental as it focuses on benchmark design rather than new detection methods.
The paper tackles the problem of evaluating small accuracy improvements in malware detection by creating configurable difficulty benchmarks from limited public datasets, showing that using a less accurate secondary model with different features effectively produces benchmarks for evaluating more sophisticated target models.
Industry practitioners care about small improvements in malware detection accuracy because their models are deployed to hundreds of millions of machines, meaning a 0.1\% change can cause an overwhelming number of false positives. However, academic research is often restrained to public datasets on the order of ten thousand samples and is too small to detect improvements that may be relevant to industry. Working within these constraints, we devise an approach to generate a benchmark of configurable difficulty from a pool of available samples. This is done by leveraging malware family information from tools like AVClass to construct training/test splits that have different generalization rates, as measured by a secondary model. Our experiments will demonstrate that using a less accurate secondary model with disparate features is effective at producing benchmarks for a more sophisticated target model that is under evaluation. We also ablate against alternative designs to show the need for our approach.