Flaky Android Espresso Test - Snackbar

1) All devices/emulators being tested have animations disabled.

2) I have an @BeforeClass which builds my Credentials object.

3) I have an IntenServiceIdlingResource and an EventBusIdlingResource, registered in @Before.

4) When the sign in button is clicked, the IntentService fires off. In this case, the server (a mocked server) is returning a 500 error. That information is posted back to the UI from the IntentService, via greenrobot's EventBus, and a Snackbar is shown with the error message.

Here's the code for the test:

@Test
public void a_userNamePasswordTest() throws Exception {
    // email input
    ViewInteraction userNameView = onView(withId(R.id.email));

    // verify it's on screen and enabled
    userNameView.check(matches(isDisplayed())).check(matches(isEnabled()));

    // set the username
    userNameView.perform(scrollTo(), replaceText(credentials.username), closeSoftKeyboard());

    // password input
    ViewInteraction passwordView = onView(withId(R.id.password));

    // verify it's on screen and enabled
    passwordView.check(matches(isDisplayed())).check(matches(isEnabled()));

    // set the password.
    passwordView.perform(scrollTo(), replaceText(credentials.password), closeSoftKeyboard());

    // sign in button
    ViewInteraction signInButton = onView(withId(R.id.email_sign_in_button));

    // verify the button
    signInButton.check(matches(allOf(
            isDisplayed(), isEnabled(), withText("Sign In"), withContentDescription("Sign In")
    )));

    // clickity click the button
    signInButton.perform(scrollTo(), click());

    // verify the snackbar text
    onView(withText(startsWith("Server Error: 500"))).check(matches(isDisplayed()));

}

This is the exception I usually get:

SignInExceptionTest > a_userNamePasswordTest[Nexus_6P_API_23(AVD) - 6.0] FAILED android.support.test.espresso.NoMatchingViewException: No views in hierarchy found matching: with text: a string starting with "Server Error: 500"

According to my logging, my idling resources are working. But looking at the timestamps for the logs, the exception is occurring about 5 seconds after the idling resources have gone idle.

It seems like there's a delay between when the resources go idle, and when it attempts to look for the view.

Other possibly pertinent details:

minSdk: 20
compile & targetSdk: 25
buildTools: 25.0.2
support-library: 25.1.1
espresso-core: 2.2.2
gradle plugin 2.3.0-beta3

How can I fix this test so it's not flaky, besides inflating how long my snackbars are displayed?

Espresso enforces a 5 second wait after an IdlingResource becomes active before it is checked again for being idle.

This has 2 negative impacts in my case.

First, it increases how long it takes the test to run. An extra 5 seconds (or multiple of 5) on every test can really add up.

Second, since the snackbar is displayed immediately, and only shows for about 3.5 seconds, pretty much any time you're waiting for an IdlingResource, the snackbar will be gone before Espresso realizes that the resource is idle, making it difficult to test.

ConditionWatcher, described here: https://medium.com/azimolabs/wait-for-it-idlingresource-and-conditionwatcher-602055f32356#.9rms52osh

And code here: https://github.com/AzimoLabs/ConditionWatcher

Provides a solution to both these issues. Rather than 5 seconds, it defaults to 250ms, and this value can be adjusted (setWatchInterval).