Xcode 5 says tests failed but shows green checkmar

Question

I have written a few unit tests in the new XCTest framework of Xcode 5.

The tests seem to pass successfully most of the time, but I've come across a strange behaviour where if I run the tests enough times consecutively, eventually I'll get a 'test failed' message and the console will show Program ended with exit code 0, but there are green checkmarks beside all the tests.

What's going wrong here?

Answer 1

This is definitely a bug. Submit a report via https://bugreport.apple.com/ feel free to reference my report (15251050).

Answer 2

Until the bug is fixed, close the project and re-open it.

Answer 3

This bug has been corrected in Xcode 5.1 :

Xcode 5.1 release notes

Notes specifically mention :

Test target [test name] encountered an error (Test process exited with code -1)

Attempt recovery by quitting and restarting the simulator. (15929053)

Answer 4

Having just run into this in my own updated XCode 5 thought I'd share my similar resolutions.

I notice if I hit any breakpoint when running the test and simply continue it reports as passing, very odd indeed and unacceptable for running your whole suite...

I also noticed adding a usleep as someone mentioned, as well as doing a clean build prior to running the tests seems to resolve the issue. However creating a new test often re-introduces the issue at least for that new test. Hope Apple fixes this soon!

Answer 5

For me, I think this problem was linked to incorrect OCMock configuration. Removing OCMock (completely) and re-adding it to the project exactly as recommended by OCMock seems to have fixed it.

A couple specific things I did wrong:

1. I was using #import "OCMock.h instead of #import "OCMock/OCMock.h. Xcode autocompleted the former.
2. I had a bad link leftover in library search paths from previously trying to install OCMock with a different directory structure.
3. This is dumb, but in trying to debug this issue, I commented out the instantiation of my OCMockObject in my setup method.. and didn't remember to uncomment it. The nil OCMockObject caused similar behavior as described in the other answers here. This is odd.. In my current project I have two tests that use the OCMockObject that is instantiated in my setup. Both should fail. However, one of them passes, and the only difference that I can see between the two is that the one that is passing previously passes legitimately...

Anyways, with OCMock setup correctly my tests are working as they should

Answer 6

This appears to have been fixed in XCode 5.1.1