Abstract
For the enormous growth and the hysterical impact of undocumented malicious software, otherwise known as Zero-Day malware, specialized practices were joined to implement systems capable of detecting these kinds of software to avert possible disastrous consequences. Owing to the nature of developed Zero-Day malware, distinct evasion tactics are used to remain stealth. Hence, there is a need for advance investigations of the methods that can identify such kind of malware. Machine learning (ML) is among the promising techniques for such type of predictions, while the sandbox provides a safe environment for such experiments. After thorough literature review, carefully chosen ML techniques are proposed for the malware detection, under Cuckoo sandboxing (CS) environment. The proposed system is coined as Zero-Day Vigilante (ZeVigilante) to detect the malware considering both static and dynamic analyses. We used adequate datasets for both analyses incorporating sufficient samples in contrast to other studies. Consequently, the processed datasets are used to train and test several ML classifiers including Random Forest (RF), Neural Networks (NN), Decision Tree (DT), k-Nearest Neighbor (kNN), Naïve Bayes (NB), and Support Vector Machine (SVM). It is observed that RF achieved the best accuracy for both static and dynamic analyses, 98.21% and 98.92%, respectively.