Abstract
The widespread adoption of web vulnerability scanners and the differences in the functionality provided by these tool-based vulnerability detection approaches increase the demand for testing their detection effectiveness. Despite the advantages of dynamic testing approaches, the literature lacks studies that systematically evaluate the performance of open source web vulnerability scanners. The main objectives of this study are to assess the performance of open source scanners from multiple perspectives and to examine their detection capability. This paper presents the results of a comparative evaluation of the security features as well as the performance of four web vulnerability detection tools. We followed this comparative assessment with a case study in which we evaluate the level of agreement between the results reported by two open source web vulnerability scanners. Given that the results of our comparative evaluation did not show significant performance differences among the scanners while the results of the conducted case study revealed high level of disagreement between the reports generated by different scanners, we conclude that the inconsistencies between the reports generated by different scanners might not necessarily correlate with their performance properties. We also present some recommendations for helping developers of web vulnerabilities scanners to improve their tools’ capabilities.