Citadel: Context Similarity Based Deep Learning Framework Bug Finding

With the application of Deep Learning (DL) technology, tools of DL framework testing are in high demand.Existing DL framework testing tools have limited coverage of bug types. For example, they lack the capabilityof effectively finding performance bugs, which are critical for DL models regarding performance, economics,and the environment. Moreover, existing tools are inefficient, generating hundreds of test cases with fewtrigger bugs. In this article, we propose Citadel, a method that accelerates bug finding in terms of efficiencyand effectiveness. We observe that many DL framework bugs are similar due to the similarity of operatorsand algorithms belonging to the same family. Orthogonal to existing bug-finding tools, Citadel aims to findnew bugs that are similar to reported ones that have known test oracles. Citadel defines context similarity tomeasure the similarity of DL framework API pairs and automatically generates test cases with oracles forAPIs that are similar to the problematic APIs in existing bug reports. Citadel effectively detects 58 and 66API bugs on PyTorch and TensorFlow (excluding those rejected by developers or duplicates of prior reports),many of which, e.g., 13 performance bugs, cannot be detected by existing tools. Moreover, 35.40% of test casesgenerated by Citadel can trigger bugs significantly transcending the state-of-the-art method (3.90%).