ABCA: Android Black-box Coverage Analyzer of Mobile App without Source Code

 Coverage criteria [5, 4] are indispensable in evaluating the adequacy of testing effort and progress of real-world software projects. On one hand, there are many traditional coverage criteria, including line coverage, branch coverage, data-flow coverage, method coverage, for white-box testing when the source code are available. On the other hand, there are also coverage criteria, defined on the system or module interfaces, for black-box testing. An empirical study [3] observed that coverage-based techniques are also effective in detecting software faults. Recently, the booming of Android app market in the cloud has prompted many small teams and individuals to adventure in this market. We present a coverage analysis tool, called Android Black-box Coverage Analyzer (ABCA), that does not need the source code of the apps under test (AUT) but can still provide source code level coverage data. ABCA can thus be beneficial in the following two aspects. • Clients need protect their intellectual properties (IP) and would be reluctant to release the source code of their applications to the test service providers. In this aspect, ABCA could be a useful bridge between the clients and the test service providers and could eventually promote the test service industry to the mobile application developers. • ABCA makes it possible for the test service providers to gain business without leaving concerns of IP infringement to their potential clients. As a research project, ABCA also shows the solution techniques for collecting source code coverage data without access to the program source code. With the detailed coverage data, ABCA can also help diagnosing why some parts of the code cannot be reached or identifying those code segments related to certain bug traces. Academia can also use ABCA in researching coverage-based techniques for Android apps. To the best of our knowledge, the only coverage tool similar to ABCA and available to the public for Android apps is by Horváthy, Bognáry, Gergely, et al [1]. However their tool can only report coverage on the methods. In contrast, ABCA can report detailed coverage reports on lines, methods, and classes of the source code of the execution of Android APKs. ABCA runs on Microsoft Windows on PC and connects to Android devices via a USB line. It consists of two parts. • ABCA-instrumentor (ABCAI), that instruments the byte-code of an Android Application Package (APK) so that coverage data can be collected out of the execution of the APK. • ABCA-reporter (ABCAR), that monitors the execution of the instrumented APK and collects the coverage data from the APK to compile the final coverage report. In Figure 1, we show a typical sequence diagram of the activities in using ABCA. The sequence diagram shows the following steps in sequence. 1) A test engineer first gets the APK under test (AUT) in bytecode from her client. 2) Then the test engineer can use ABCAI to instrument the APK to include bytecodes for providing coverage data. 3) The test engineer connects the Android device to the PC and with the help of ABCAR, install the instrumented APK to the Android device via the USB line. At this step, the test engineer can also use option of ABCAR to import a coverage report from a previous test session. With this option, the test engineer can continue test sessions and accumulate test coverage across test sessions in a testing project. 4) Then the test engineer starts the instrumented APK with ABCAR. 5) The test engineer then feeds test input to the AUT on the device and the coverage data is collected by ABCAR via the USB line. This step can be repeated several times until the user decides to stop the test session. Various test case generation modules and tools can also be employed for high coverage and bug exposure. 6) After the test session is done, ABCAR generates a coverage report file. At the moment, the coverage report includes line coverage, method coverage, and class coverage. In this manuscript, we shall present the algorithms and techniques used in ABCA in section 2. We also explain some technical issues in our implementation in section 3. Then we report experiment of ABCA with several Android apps in the Google Play in section 4. Via the experiment, we can see that ABCA is effective in generating white-box coverage data without source code of the apps of Android.