Towards Generic Deobfuscation of Windows API Calls
This addresses the challenge for malware analysts and anti-malware scanners in detecting obfuscated malicious behavior, though it is an incremental improvement over existing methods.
The paper tackles the problem of malware analysis by introducing a static analysis technique for deobfuscating Windows API calls, achieving 87.60% accuracy in identifying API names.
A common way to get insight into a malicious program's functionality is to look at which API functions it calls. To complicate the reverse engineering of their programs, malware authors deploy API obfuscation techniques, hiding them from analysts' eyes and anti-malware scanners. This problem can be partially addressed by using dynamic analysis; that is, by executing a malware sample in a controlled environment and logging the API calls. However, malware that is aware of virtual machines and sandboxes might terminate without showing any signs of malicious behavior. In this paper, we introduce a static analysis technique allowing generic deobfuscation of Windows API calls. The technique utilizes symbolic execution and hidden Markov models to predict API names from the arguments passed to the API functions. Our best prediction model can correctly identify API names with 87.60% accuracy.