Identifying and mitigating flaky tests in JavaScript : a thesis presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Computer Science, Massey University, Palmerston North, New Zealand

Abstract

Flaky tests, tests that may pass in one execution and fail in another without changes to the test code or the code under test, pose a persistent challenge to software development by reducing confidence in test results and increasing maintenance costs. Although flaky tests have been studied in several programming ecosystems, their causes, manifestations, and mitigation strategies in JavaScript remain less well understood, despite the language’s widespread use and highly heterogeneous execution environments.

This thesis investigates flaky tests in JavaScript through an empirical paradigm, focusing on three complementary objectives: understanding the main causes of flakiness in practice, detecting order-dependent flaky tests, and controlling flaky tests caused by environmental dependency. To address these objectives, the thesis combines large-scale mining of open-source repositories, systematic experimentation, and tool development.

First, the thesis presents an empirical study of flaky-test-related commits collected from popular JavaScript projects hosted on GitHub. By manually analysing commit messages, code changes, and associated issue discussions, the study identifies the dominant causes of flaky tests and examines how developers respond to them. The results show that flakiness in JavaScript is primarily driven by concurrency, asynchronous waiting, operating system dependencies, and network-related issues. Together, these causes account for more than 70% of the observed flaky-test commits. The study further shows that developers usually address flaky tests through direct fixes, while skipping, quarantining, or removing tests are used as pragmatic alternatives when immediate fixes are difficult.

Second, the thesis investigates order-dependent flakiness, where test outcomes depend on the execution order of tests. Building on the empirical findings, it introduces a systematic test reordering approach and implements it in a tool called JS-TOD for Jest-based projects. JS-TOD reorders tests at multiple structural levels and reruns them to expose order-dependent behaviour. An evaluation on real-world projects shows that order-dependent flakiness does occur in JavaScript, although it is less prevalent than reported in other ecosystems. The detected cases are primarily caused by shared state between tests, including shared files and shared mocking state, with the latter identified as an important and previously underreported source of order dependency in JavaScript.

Finally, the thesis examines environmental flakiness, which arises when tests fail under specific execution environments such as particular operating systems, Node.js versions, or browsers. Through systematic reruns across multiple environments, the study demonstrates that environmental flakiness is common and diverse in JavaScript projects. To mitigate its impact, the thesis proposes a lightweight annotation-based approach, implemented as a tool js-env-sanitizer, that sanitizes environment-dependent tests by conditionally skipping and reporting them under known problematic configurations. An empirical evaluation shows that js-env-sanitizer can effectively control environmental flaky tests without introducing incorrect annotations.

Overall, this thesis provides an evidence-based understanding of flaky tests in JavaScript and demonstrates practical techniques for detecting and controlling two important classes of flakiness. The findings highlight the need for ecosystem-specific analysis and offer actionable guidance for improving the reliability of automated testing in JavaScript projects.