Test Automation Process Improvement in a DevOpsTeam: Experience Report
This provides insights for software organizations aiming to improve test automation in continuous development, though it is incremental as it builds on prior literature with a specific case study.
The paper presents an experience report on test automation process improvement (TAPI) in a DevOps team at F-Secure, finding that the team defines maturity through indicators like increased release speed and productivity, and identifies critical success factors such as incremental approaches and whole-team effort.
How to successfully conduct test automation process improvement (TAPI) for continuous development, consisting of iterative software development, continuous testing, and delivery, is the challenge faced by many software organizations. In this paper, we present an experience report on TAPI in one DevOps team in F-Secure (a Finnish software company). The team builds Windows application software and exists in F-Secure's TAPI culture. The team self-reports high satisfaction and maturity in test automation for continuous development. To study their TAPI, we reviewed a collection of experience notes, team reflection reports and telemetry result reports. Then several meetings were held to discuss the details. We found that based on the understanding of the team, test automation maturity for continuous development is defined as a set of indicators, e.g., the increasing speed to release, improving the productivity of the team, high test efficiency. Second, the team indicated that a set of critical success factors have a major impact on successfully carrying out its TAPI, e.g., incremental approach, the whole team effort, test tool choice and architecture, telemetry. Third, we compare the TAPI practices in the observed team with the practices described in prior literature. The team believes that the existing test automation maturity approaches should include the identified practices like the whole team effort to build a more comprehensive test automation improvement model for the software industry.