Malware uses API functions to execute malicious code, manipulate processes and threads, hide its presence, or access sensitive information. Thus, malware detection based on API instruction sequence is important for the security of Windows systems. Traditionally, static and dynamic detection techniques are two primary categories for malware detection. Static detection models identify known malware without executing it, while dynamic detection models can detect unknown malware more efficiently by executing it in controlled environments.

Compared with static methods, dynamic detection models typically exhibit superior performance in terms of detection accuracy and false positive rate. However, the complexity of the feature selection and extraction process and the reliance on expert knowledge can cause difficulties in malware classification. To overcome these problems, recent research has focused on using recurrent neural networks to automate the feature extraction and classification process.

Malware Detection API Overview: How to Identify & Block Malware

In particular, Malware detection API overviews are effective in reducing the number of false positives and improving classification accuracy by training a model to recognise features with multiple dimensions. This study uses a recurrent GAN with a boosted gradient descent (BGGD) algorithm to classify API instruction sequences of executable software samples after sandbox program simulation. The model was tested on Cuckoo sandbox data, which consists of 42,797 malicious and 10,792 benign API call sequences. The class imbalance of the dataset can adversely affect model performance, so a synthetic minority over-sampling technique is applied to balance the classes.

The API instruction sequences of the software samples are reorganized by type and order to form an API block. The API codes, API devotion and the API sequential index are extracted from each API block to generate a feature image for classification. The reorganised feature images are fed into the recurrent GAN for further analysis, and the results show that this method improves detection accuracy and false positive rate.