Abstract
The explosive growth of Internet and the recent increasing trends in automation using intelligent applications have provided a veritable playground for malicious software (malware) attackers. With a variety of devices connected seamlessly via the Internet and large amounts of data collected, the escalating malware attacks and security risks are a big concern. While a number of malware detection methods are available, new methods are required to match with the scale and complexity of such a data-intensive environment. We propose a novel and unified hybrid deep learning and visualization approach for an effective detection of malware. The aim of the paper is two-fold: 1. to present the use of image-based techniques for detecting suspicious behavior of systems, and 2. to propose and investigate the application of hybrid image-based approaches with deep learning architectures for an effective malware classification. The performance is measured by employing various similarity measures of malware behavior patterns as well as cost-sensitive deep learning architectures. The scalability is benchmarked by testing our proposed hybrid approach with both public and privately collected large malware datasets that show high accuracy of our malware classifiers. (C) 2019 Published by Elsevier Ltd.