API evolution and compatibility: A data corpus and tool evaluation

Abstract

The development of software components with independent release cycles is nowadays widely supported by multiple languages and frameworks. A critical feature of any such platform is to safeguard composition by ensuring backward compatibility of substituted components. In recent years, some tooling has been developed to help developers and DevOps engineers to establish whether components are backward compatible by means of static analysis. We investigate the state of the art in this space by benchmarking such tools for Java. For this purpose, we have developed a compact benchmark data set of less than 200KB. Using this dataset, we study possible API changes of Java libraries, and whether the tools investigated can detect them. We find that only a small number of tools suitable to analyse API evolution exist. Those tools are only infrequently maintained by small communities. All tools investigated have some shortcomings in that they fail to detect certain API incompatibilities.