Understanding test evolution: from highly‑configurable systems to software ecosystems

Abstract

Software evolution is inevitable if the systems are planned to survive in the long-term. Equally, well-understood is the necessity of having a good test suite available to ensure the quality of the current state of the software system and to ease future changes. This is especially true in the context of reusable systems since they are planned to attend for a long time a specific market niche and need to support a large number of configuration options. However, developing and maintaining a test suite is time-intensive and costly. This situation is challenging for the projects: on the one hand, tests are essential for the success of software; on the other hand, tests become a severe burden during maintenance.

Even though a substantial body of literature has studied testing in reusable environments, test evolution analysis has not been addressed. In general, researchers have looked into analyzing test strategies, dynamic test selection techniques, and co-evolution of tests along with other systems artifacts. This thesis intends to improve the test evolution body of knowledge in reusable systems, investigating characteristics that indicate the effort to develop and maintain the test suite and unveiling how the reusable aspects affect the tests. The set of evidence can help researchers and practitioners to better planning the test development and evolution.

This way, we employed a multi-method approach to develop the understanding of test evolution in configurable systems and unveil evidence on the topic from various sources. In the first phase of the research program, we provided an overview of the existing research related to this thesis’s subjects and presented related work to our investigation. The second phase was composed of four empirical studies. First, we performed a case study to analyze the test evolution of a large configurable system. Next, we performed a comparative study to evaluate the test evolution in 18 open-source projects from various sizes and domains in configurable systems and their similarities and differences to 18 Single Systems projects. Third, we conducted an extended study to analyze the test suite evolution in another category of reusable systems to verify whether some observations are recurring and gather new data that support the findings. Finally, we surveyed test contributors to investigate the test evolution from the development point of view and improve the findings in previous stages.

This work collected a set of findings of test evolution, and these findings were strengthened by using different research methods. Our work provided a better understanding of test evolution in configurable systems by documenting evidence observed in open-source projects and test contributors. Moreover, in this Thesis, we synthesized the gathered evidence and identified open issues in this research topic. These findings are an important step to establish guidelines for addressing test evolution in configurable systems.