Fundamentals of Testing for applications

Fundamentals of Testing

Users interact with your app on a variety of levels, from pressing a button to downloading information onto their device. Accordingly, you should test a variety of use cases and interactions as you iteratively develop your app.

Organize your code for testing

As your app expands, you might find it necessary to fetch data from a server, interact with the device's sensors, access local storage, or render complex user interfaces. The versatility of your app demands a comprehensive testing strategy.

Create and test code iteratively

When developing a feature iteratively, you start by either writing a new test or by adding cases and assertions to an existing unit test. The test fails at first because the feature isn't implemented yet.

It's important to consider the units of responsibility that emerge as you design the new feature. For each unit, you write a corresponding unit test. Your unit tests should nearly exhaust all possible interactions with the unit, including standard interactions, invalid inputs, and cases where resources aren't available. Take advantage of Jetpack libraries whenever possible; when you use these well-tested libraries, you can focus on validating behavior that's specific to your app.

Figure 1. The two cycles associated with iterative, test-driven development

The full workflow, as shown in Figure 1, contains a series of nested, iterative cycles where a long, slow, UI-driven cycle tests the integration of code units. You test the units themselves using shorter, faster development cycles. This set of cycles continues until your app satisfies every use case.

View your app as a series of modules

To make your code easier to test, develop your code in terms of modules, where each module represents a specific task that users complete within your app. This perspective contrasts the stack-based view of an app that typically contains layers representing the UI, business logic, and data.

For example, a "task list" app might have modules for creating tasks, viewing statistics about completed tasks, and taking photographs to associate with a particular task. Such a modular architecture also helps you keep unrelated classes decoupled and provides a natural structure for assigning ownership within your development team.

It's important to set well-defined boundaries around each module, and to create new modules as your app grows in scale and complexity. Each module should have only one area of focus, and the APIs that allow for inter-module communication should be consistent. To make it easier and quicker to test these inter-module interactions, consider creating fake implementations of your modules. In your tests, the real implementation of one module can call the fake implementation of the other module.

As you create a new module, however, don't be too dogmatic about making it full-featured right away. It's OK for a particular module to not have one or more layers of the app's stack.

To learn more about how to define modules in your app, as well as platform support for creating and publishing modules, see Android App Bundles.

Configure your test environment

When setting up your environment and dependencies for creating tests in your app, follow the best practices described in this section.

Organize test directories based on execution environment

A typical project in Android Studio contains two directories in which you place tests. Organize your tests as follows:

• The androidTest directory should contain the tests that run on real or virtual devices. Such tests include integration tests, end-to-end tests, and other tests where the JVM alone cannot validate your app's functionality.
• The test directory should contain the tests that run on your local machine, such as unit tests.

Consider tradeoffs of running tests on different types of devices

When running your tests on a device, you can choose among the following types:

• Real device
• Virtual device (such as the emulator in Android Studio)
• Simulated device (such as Robolectric)

Real devices offer the highest fidelity but also take the most time to run your tests. Simulated devices, on the other hand, provide improved test speed at the cost of lower fidelity. The platform's improvements in binary resources and realistic loopers, however, allow simulated devices to produce more realistic results.

Virtual devices offer a balance of fidelity and speed. When you use virtual devices to test, use snapshots to minimize setup time in between tests.

Consider whether to use test doubles

When creating tests, you have the option of creating real objects or test doubles, such as fake objects or mock objects. Generally, using real objects in your tests is better than using test doubles, especially when the object under test satisfies one of the following conditions:

• The object is a data object.
• The object cannot function unless it communicates with the real object version of a dependency. A good example is an event callback handler.
• It's hard to replicate the object's communication with a dependency. A good example is a SQL database handler, where an in-memory database provides more robust tests than fakes of database results.

In particular, mocking instances of types that you don't own usually leads to brittle tests that work only when you've understood the complexities of someone else's implementation of that type. Use such mocks only as a last resort. It's OK to mock your own objects, but keep in mind that mocks annotated using @Spy provide more fidelity than mocks that stub out all functionality within a class.

However, it's better to create fake or even mock objects if your tests try to perform the following types of operations on a real object:

• Long operations, such as processing a large file.
• Non-hermetic actions, such as connecting to an arbitrary open port.
• Hard-to-create configurations.

Tip: Check with the library authors to see if they provide any officially-supported testing infrastructures, such as fakes, that you can reliably depend on.

Write your tests

After you've configured your testing environment, it's time to write tests that evaluate your app's functionality. This section describes how to write small, medium, and large tests.

Levels of the Testing Pyramid

Figure 2. The Testing Pyramid, showing the three categories of tests that you should include in your app's test suite

The Testing Pyramid, shown in Figure 2, illustrates how your app should include the three categories of tests: small, medium, and large:

• Small tests are unit tests that validate your app's behavior one class at a time.
• Medium tests are integration tests that validate either interactions between levels of the stack within a module, or interactions between related modules.
• Large tests are end-to-end tests that validate user journeys spanning multiple modules of your app.

As you work up the pyramid, from small tests to large tests, each test increases in fidelity but also increases in execution time and effort to maintain and debug. Therefore, you should write more unit tests than integration tests, and more integration tests than end-to-end tests. Although the proportion of tests for each category can vary based on your app's use cases, we generally recommend the following split among the categories: 70 percent small, 20 percent medium, and 10 percent large.

To learn more about the Android Testing Pyramid, see the Test-Driven Development on Android session video from Google I/O 2017, starting at 1:51.

Write small tests

The small tests that you write should be highly-focused unit tests that exhaustively validate the functionality and contracts of each class within your app.

As you add and change methods within a particular class, create and run unit tests against them. If these tests depend on the Android framework, use a unified, device-agnostic API, such as the androidx.test APIs. This consistency allows you to run your test locally without a physical device or emulator.

If your tests rely on resources, enable the includeAndroidResources option in your app's build.gradle file. Your unit tests can then access compiled versions of your resources, allowing the tests to run more quickly and accurately.

app/build.gradle

android {
    // ...

    testOptions {
        unitTests {
            includeAndroidResources = true
        }
    }
}

Note: Android Studio 3.4 and higher provide compiled versions of your resources by default.

Local unit tests

Use the AndroidX Test APIs whenever possible so that your unit tests can run on a device or emulator. For tests that always run on a JVM-powered development machine, you can use Robolectric.

Robolectric simulates the runtime for Android 4.1 (API level 16) or higher and provides community-maintained fakes called shadows. This functionality allows you to test code that depends on the framework without needing to use an emulator or mock objects. Robolectric supports the following aspects of the Android platform:

• Component lifecycles
• Event loops
• All resources

Instrumented unit tests

You can run instrumented unit tests on a physical device or emulator. This form of testing involves significantly slower execution times than local unit tests, however, so it's best to rely on this method only when it's essential to evaluate your app's behavior against actual device hardware.

When running instrumented tests, AndroidX Test makes use of the following threads:

• The main thread, also known as the "UI thread" or the "activity thread", where UI interactions and activity lifecycle events occur.
• The instrumentation thread, where most of your tests run. When your test suite begins, the AndroidJUnitTest class starts this thread.

If you need a test to execute on the main thread, annotate it using @UiThreadTest.

Write medium tests

In addition to testing each unit of your app by running small tests, you should validate your app's behavior from the module level. To do so, write medium tests, which are integration tests that validate the collaboration and interaction of a group of units.

Use your app's structure and the following examples of medium tests (in order of increasing scope) to define the best way to represent groups of units in your app:

1. Interactions between a view and view model, such as testing a Fragment object, validating layout XML, or evaluating the data-binding logic of a ViewModel object.
2. Tests in your app's repository layer, which verify that your different data sources and data access objects (DAOs) interact as expected.
3. Vertical slices of your app, testing interactions on a particular screen. Such a test verifies the interactions throughout the layers of your app's stack.
4. Multi-fragment tests that evaluate a specific area of your app. Unlike the other types of medium tests mentioned in this list, this type of test usually requires a real device because the interaction under test involves multiple UI elements.

To carry out these tests, do the following:

1. Use methods from the Espresso Intents library. To simplify the information that you're passing into these tests, use fakes and stubbing.
2. Combine your use of IntentSubject and Truth-based assertions to verify the captured intents.

Use Espresso when running instrumented medium tests

Espresso helps keep tasks synchronized as you perform UI interactions similar to the following on a device or on Robolectric:

• Performing actions on View objects.
• Assessing how users with accessibility needs can use your app.
• Locating and activating items within RecyclerView and AdapterView objects.
• Validating the state of outgoing intents.
• Verifying the structure of a DOM within WebView objects.

To learn more about these interactions and how to use them in your app's tests, see the Espresso guide.

Write large tests

Although it's important to test each class and module within your app in isolation, it's just as important to validate end-to-end workflows that guide users through multiple modules and features. These types of tests form unavoidable bottlenecks in your code, but you can minimize this effect by validating an app that's as close to the actual, finished product as possible.

Note: For each large, workflow-based test that you write, you should also write medium tests that check the functionality of each module included in the workflow. This testing structure makes it easier to determine which step of a critical user journey exhibits unexpected behavior.

If your app is small enough, you might need only one suite of large tests to evaluate your app's functionality as a whole. Otherwise, you should divide your large test suites by team ownership, functional verticals, or user goals.

Typically, it's better to test your app on an emulated device or a cloud-based service like Firebase Test Lab, rather than on a physical device, as you can test multiple combinations of screen sizes and hardware configurations more easily and quickly.

Synchronization support in Espresso

In addition to supporting medium-sized instrumentation tests, Espresso provides support for synchronization when completing the following tasks in large tests:

• Completing workflows that cross your app's process boundaries. Available only on Android 8.0 (API level 26) and higher.
• Tracking long-running background operations within your app.
• Performing off-device tests.

To learn more about these interactions and how to use them in your app's tests, see the Espresso guide.

Complete other testing tasks using AndroidX Test

This section describes how to use elements of AndroidX Test to further refine your app's tests.

Create more readable assertions using Truth

The Guava team provides a fluent assertions library called Truth. You can use this library as an alternative to JUnit- or Hamcrest-based assertions when constructing the validation step—or then step—of your tests.

Usually, you use Truth to express that a particular object has a specific property using phrases that contain the conditions you're testing, such as the following:

• assertThat(object).hasFlags(FLAGS)
• assertThat(object).doesNotHaveFlags(FLAGS)
• assertThat(intent).hasData(URI)
• assertThat(extras).string(string_key).equals(EXPECTED)

AndroidX Test supports several additional subjects for Android to make Truth-based assertions even easier to construct:

• BundleSubject
• IntentSubject
• MotionEventSubject
• NotificationActionSubject
• NotificationSubject
• PendingIntentSubject
• PointerCoordsSubject
• PointerPropertiesSubject

The AndroidX Test API helps you carry out common tasks related to mobile app testing, which the following sections discuss.

Write UI tests

Espresso allows you to programmatically locate and interact with UI elements in your app in a thread-safe way. To learn more, see the Espresso guide.

Run UI tests

The AndroidJUnitRunner class defines an instrumentation-based JUnit test runner that lets you run JUnit 3- or JUnit 4-style test classes on Android devices. The test runner facilitates loading your test package and the app under test onto a device or emulator, running your tests, and reporting the results.

To further increase these tests' reliability, use Android Test Orchestrator, which runs each UI test in its own Instrumentation sandbox. This architecture reduces shared state between tests and isolates app crashes on a per-test basis. For more information about the benefits that Android Test Orchestrator provides as you test your app, see the Android Test Orchestrator guide.

Interact with visible elements

The UI Automator API lets you interact with visible elements on a device, regardless of the activity or fragment that has focus.

Caution: We recommend testing your app using UI Automator only when your app must interact with the system UI or another app to fulfill a critical use case. Because UI Automator interacts with a particular system UI, you must re-run and fix your UI Automator tests after each platform version upgrade and after each new release of Google Play services.

As an alternative to using UI Automator, we recommend adding hermetic tests or separating your large test into a suite of small and medium tests. In particular, focus on testing one piece of inter-app communication at a time, such as sending information to other apps and responding to intent results. The Espresso-Intents tool can help you write these smaller tests.

Add accessibility checks to validate general usability

Your app's interface should allow all users, including those with accessibility needs, to interact with the device and complete tasks more easily in your app.

To help validate your app's accessibility, Android's testing library provides several pieces of built-in functionality, which is discussed in the following sections. To learn more about how to validate your app's usability for different types of users, see the guide on testing your app's accessibility.

Robolectric

Enable accessibility checks by including the @AccessibilityChecks annotation at the beginning of your test suite, as shown in the following code snippet:

KOTLINJAVA

import org.robolectric.annotation.AccessibilityChecks

@AccessibilityChecks
class MyTestSuite {
    // Your tests here.
}

Espresso

Enable accessibility checks by calling AccessibilityChecks.enable() in your test suite's setUp() method, as shown in the following code snippet.

For more information on how to interpret the results of these accessibility checks, see the Espresso accessibility checking guide.

KOTLINJAVA

import androidx.test.espresso.accessibility.AccessibilityChecks

@Before
fun setUp() {
    AccessibilityChecks.enable()
}

Drive activity and fragment lifecycles

Use the ActivityScenario and FragmentScenario classes to test how your app's activities and fragments respond to system-level interruptions and configuration changes. To learn more, see the guides on how to test activities and test fragments.

Note: Given the device-specific conditions under which fragments behave, it's best to work with FragmentScenario as part of your instrumented tests.

Manage service lifecycles

AndroidX Test includes code for managing the lifecycles of key services. To learn how to define these rules, see the JUnit4 Rules guide.

Evaluate all variants of behavior that differ by SDK version

If your app's behavior depends on the device's SDK version, use the @SdkSuppress annotation, passing in values for minSdkVersion or maxSdkVersion depending on how you've branched your app's logic:

KOTLINJAVA

@Test
@SdkSuppress(maxSdkVersion = 27)
fun testButtonClickOnOreoAndLower() {
    // ...
}

@Test
@SdkSuppress(minSdkVersion = 28)
fun testButtonClickOnPieAndHigher() {
    // ...
}

Additional resources

For more information about testing on Android, consult the following resources.

Samples

• Android testing samples

Codelabs

• Android Testing Codelab

enroll corses