Abstract
•Reviewed 77 papers on cloud service evaluation method-based MCDM.•Presented a systematic literature review based on evaluation theory components.•Sixteen research deficiencies were identified.
A substantial effort has been made to solve the cloud-service evaluation problem. Different Cloud Service Evaluation Methods (CSEMs) have been developed to address the problem. Cloud services are evaluated against multiple criteria, which leads to a Multi-Criteria Decision-Making (MCDM) problem. Yet, studies that assess, analyse, and summarize the unresolved problems and shortcomings of current CSEM-based MCDM are limited. In the existing review studies, only individual parts of CSEMs, rarely the full solution, are reviewed and examined. To investigate CSEMs comprehensively, we present a systematic literature review based on Evaluation Theory, a theory that generalizes six evaluation components, target, criteria, yardstick, data gathering techniques, synthesis techniques, and evaluation process. These six evaluation components and the CSEMs validation approach are the seven dimensions used to assess and analyse 77 papers published from 2006 to 2016. Sixteen research deficiencies were identified. The results confirm that the majority of the studies of the proposed CSEMs were either incomplete or lacked sufficient evidence. This research not only provides the relative strengths and weaknesses of the different CSEMs but also offers a basis for researchers and decision makers to develop improved CSEMs.