Abstract
With the continuous advancements of technology, cyber-criminals accordingly develop sophisticated tactics to exploit vulnerabilities in individual systems, organization networks, and nation-states. Enterprises routinely collect huge amount of security-relevant data such as log events of people, networks, and software applications for future forensic analysis. Existing traditional security analysis tools fail to work well with large scales of data and usually produces high false alarms especially when the enterprises moves to cloud architecture and collect more data. Moreover, the detection of recent and more sophisticated attacks, like advanced persistent threats (APTs), requires continuous monitoring and analysis of huge security related data, accurately and rapidly. Big Data analytics has been in active use in several fields such as financial transactions, healthcare and industrial applications among others. Recently, it has attracted the attention of information security audience due to its promised ability in correlating security related data and draw insights efficiently at unprecedented scale. In this paper, we analyze the traditional technology/systems and Security Information and Event Management (SIEM) tools and show their shortcomings in dealing with huge data scales and advanced sophisticated threats. We then explore the requirements for Big Data analytics to be successfully adopted in cyber threat intelligence and cyber-security landscape to deal with high data scales and sophisticated threats. Finally, we highlight the challenges resulted from such adoption, and suggest some recommendations to overcome adoption challenges in future research.