Abstract
This article addresses the need of applying testing activities with the purpose of checking the consistency and the correctness of distributed and dynamically adaptable systems whose structure and behavior evolve dynamically at runtime. We identified in the literature a number of studies that used some traditional testing techniques (e.g., regression testing) applied at development time and runtime verification techniques (e.g., runtime monitoring) to ensure the trustworthiness of software systems. Since traditional testing techniques are no longer adequate for dynamically adaptable systems because some behavioral/structural evolutions are unknown at development time, the runtime testing was introduced (Brenner et al., 2007). It is defined as any testing method that has to be carried out on the final execution environment by performing tests during deployment or in-service time. However, there is a lack of recent comprehensive survey on this emerging verification an validation technique. To fill this gap, we conduct a survey to investigate research done in this area. A list of 43 research studies are deeply discussed to address answers of eight research questions. In particular, this work gives the runtime testing taxonomy, challenges and introduces several approaches dealing with such technique and discusses challenges as well as recent advances in this area. Moreover, it also points out some future directions.