Comparing the effort and effectiveness of automated and manual tests. An industrial case study

Abstract

This paper presents three case studies that compare the effort and effectiveness of automated versus manual testing, in the context of a multinational services organization. Effort is measured in terms of the total test time, which includes script creation and test execution in the case of automated testing, and comprises test execution and reporting in the case of manual testing. Effectiveness is measured in terms of the number and severity of defects found. The software under test is a set of Java web applications. The testing process was carried out by two testers within the organization. Our results show that automated testing needs a higher initial effort, mainly caused by the creation of the scripts, but this cost can be amortized in time as automated tests are executed multiple times for regression testing. Results also show that automated testing is more effective than manual testing at finding defects.