Abstract
Malware have become the scourge of the century, as they are continuously evolving and becoming more complex with increasing damages. Therefore, an adequate protection against such threats is vital. Behavior-based malware detection techniques have shown to be effective at overcoming the weaknesses of the signature-based ones. However, they are known for their high false alarms, which is still a very challenging problem. In this article, we address this shortcoming by proposing a rule-based behavioral malware detection system, which inherits the advantages of both signature and behavior-based approaches. We apply the proposed detection system on a combined set of three types of dynamic features, namely, (1) list of application programming interface calls; (2) application programming interface sequences; and (3) network traffic, which represents the IP addresses and domain names used by malware to connect to remote command-and-control servers. Feature selection and construction techniques, that is, term frequency–inverse document frequency and longest common subsequence, are performed on the three extracted features to generate new set of features, which are used to build behavioral Yet Another Recursive Acronym rules. The proposed malware detection approach is able to achieve an accuracy of 97.22% and a false positive rate of 4.69%.