What is Android Test Orchestrator?

问题:

Google released Android Testing Support Library 1.0 recently. After reading the overview, I'm a little confused with Android Test Orchestrator.

It said

Typically, AndroidJUnitRunner runs all tests in the same instrumentation process, which can cause a number of problems.

Can you explain which kinds of problems will be caused by using the same instrumentation process?

if one test crashes, it prevents the remainder of the test suite from running

Through my experience, one test crash won't prevent from other test cases from running. Please point out what I misunderstood here?

And from Android Testing Orchestrator developer guide,

For completeness, Android Test Orchestrator runs pm clear after each test.

So Android Test Orchestrator will run pm clear [test_package_name] after each test, right?

Through my test, pm clear [app_package_name] won't be executed after each test. That means the data of application under test will not be cleared. So test cases might still have dependencies on each other. For example:

• Test case A stores a SharedPreference key-value
• Test case B which runs after test case A can read out the value stored by test case A

Overall, after some trial, I did not find any advantage of Android Test Orchestrator. Can somebody help to address my confusion? Thanks.

回答1:

After researching the issue a bit I can provide the following answers:

Typically, AndroidJUnitRunner runs all tests in the same instrumentation process, which can cause a number of problems.

As mentioned, AndroidJUnitRunner runs on the same instrumentation process so basically your tests are running state-full, this might wreak havoc if your tests have some sort of dependency on the process state. In Android test orchestrator each test is run in its own process so dependencies aren't an issue.

if one test crashes, it prevents the remainder of the test suite from running

The crash in question here is a crash of the process, not the activity/application. You can test this by inserting in one your tests System.exit(0); Typically, this will stop the whole test run while in Android test orchestrator the tests continue as expected.

For completeness, Android Test Orchestrator runs pm clear after each test.

This is an oversight of google and has been retracted from the official documentation as can be observed here.

Basically, The advantages of using Android test orchestrator is all about the separate process for each test which improves stability and ensures full execution of tests.

回答2:

Android Test Orchestrator is a tool which allows you to run each of your app’s tests within its own invocation of Instrumentation. This means that each test (method annotated with @Test ) will be run on a separate instance of AndroidJUnitRunner.

What issues does it solve? When dealing with UI tests we identified 2 major problems which occur from time to time when run on CI or locally:

1. Occasional crashes which stop the entire test suite.
2. Test overlap.