Abstract
Modern software quality assurance (QA) relies heavily on Continuous Integration (CI), which automatically builds and tests each code change to provide rapid feedback on its quality. However, automation produces QA signals rather than QA decisions. Developers still have to make three decisions themselves. Before a build runs, they need to know whether it will take a long time, which is common in machine learning (ML) projects. When a build fails, they need to judge whether their own change caused the failure. After a discovered defect is fixed, they need to find the earlier change that introduced it. When these decisions are slow or wrong, they weaken the rapid feedback that automated QA is meant to provide. Despite their practical importance, there are still limited empirical studies of how ML can support such QA decisions in a consistent and effective way.
In this thesis, we perform three empirical studies on ML-based decision support for QA. First, we study long CI builds in ML projects using 287,541 GitHub Actions builds from 107 open-source ML projects. We find that the top 10% longest builds consume about 80% of the total build time and are associated with CI workflows containing more jobs and more complex job dependencies. Based on these results, we build prediction models to identify the top 10% longest builds before execution. The best model, XGBoost trained on our full set of 47 features, achieves an F1 of 0.623, compared with 0.413 for XGBoost trained on baseline features derived from prior studies. Second, we study unrelated build failures using 77,354 failed CI builds from seven open-source Apache projects. We find that unrelated failures are common, and that 20% of the failures deemed unrelated to the triggering code changes are caused by unrelated test failures. Our semi-supervised learning models to identify unrelated build failures achieve AUC scores ranging from 0.63 to 0.97 across projects. Third, we explore the identification of bug-inducing commits after bugs have been fixed. We propose ML-SZZ, an ML-based approach that combines multiple features to identify bug-inducing commits. On four benchmark datasets, ML-SZZ improves F1 by 6.8% to 31.1% over the best existing SZZ variant.
Taken together, the three studies show that important QA decisions can be supported by ML using historical CI data, but that effective support is not provided by a single uniform solution. Instead, each decision problem requires task-specific features, modeling choices, and workflow pipelines. Overall, this thesis shows both the promise and the current limits of ML for QA decision support.