Flakiness goes live: Insights from an In Vivo testing simulation study

Test flakiness is a topmost concern in software test automation. While conducting pre-deployment testing, those tests that are flagged as flaky are put aside for being either repaired or discarded. We hypothesise that some flaky tests could provide useful insights if run in the field, and could help identify hard-to-detect failures that escape testing and present themselves in operation. Our study investigates the behaviour of flaky tests when moved from the development environment to a sufficiently realistic simulation of operations in the field. The experimentation, against known flaky tests from an open-source library, provides a first confirmation that moving from the laboratory to the field the behaviour of tests changes. In particular, the results of our study suggest that the identification of an intermittently failing behaviour could be a valuable hint for a test engineer, and hence flaky tests should not be dismissed right away