How to unit test asynchronous APIs?

Question

I have installed Google Toolbox for Mac into Xcode and followed the instructions to set up unit testing found here.

It all works great, and I can test my synchronous methods on all my objects absolutely fine. However, most of the complex APIs I actually want to test return results asynchronously via calling a method on a delegate - for example a call to a file download and update system will return immediately and then run a -fileDownloadDidComplete: method when the file finishes downloading.

How would I test this as a unit test?

It seems like I'd want to the testDownload function, or at least the test framework to 'wait' for fileDownloadDidComplete: method to run.

EDIT: I've now switched to using the XCode built-in XCTest system and have found that TVRSMonitor on Github provides a dead easy way to use semaphores to wait for async operations to complete.

For example:

- (void)testLogin {
  TRVSMonitor *monitor = [TRVSMonitor monitor];
  __block NSString *theToken;

  [[Server instance] loginWithUsername:@"foo" password:@"bar"
                               success:^(NSString *token) {
                                   theToken = token;
                                   [monitor signal];
                               }

                               failure:^(NSError *error) {
                                   [monitor signal];
                               }];

  [monitor wait];

  XCTAssert(theToken, @"Getting token");
}

Answer 1

I find it very convenient to use https://github.com/premosystems/XCAsyncTestCase

It adds three very handy methods to XCTestCase

@interface XCTestCase (AsyncTesting)

- (void)waitForStatus:(XCTAsyncTestCaseStatus)status timeout:(NSTimeInterval)timeout;
- (void)waitForTimeout:(NSTimeInterval)timeout;
- (void)notify:(XCTAsyncTestCaseStatus)status;

@end

that allow very clean tests. An example from the project itself:

- (void)testAsyncWithDelegate
{
    NSURLRequest *request = [NSURLRequest requestWithURL:[NSURL URLWithString:@"http://www.google.com"]];
    [NSURLConnection connectionWithRequest:request delegate:self];
    [self waitForStatus:XCTAsyncTestCaseStatusSucceeded timeout:10.0];
}

- (void)connectionDidFinishLoading:(NSURLConnection *)connection
{
    NSLog(@"Request Finished!");
    [self notify:XCTAsyncTestCaseStatusSucceeded];
}

- (void)connection:(NSURLConnection *)connection didFailWithError:(NSError *)error
{
    NSLog(@"Request failed with error: %@", error);
    [self notify:XCTAsyncTestCaseStatusFailed];
}

Answer 2

I implemented the solution proposed by Thomas Tempelmann and overall it works fine for me.

However, there is a gotcha. Suppose the unit to be tested contains the following code:

dispatch_async(dispatch_get_main_queue(), ^{
    [self performSelector:selector withObject:nil afterDelay:1.0];
});

The selector may never be called as we told the main thread to lock until the test completes:

[testBase.lock lockWhenCondition:1];

Overall, we could get rid of the NSConditionLock altogether and simply use the GHAsyncTestCase class instead.

This is how I use it in my code:

@interface NumericTestTests : GHAsyncTestCase { }

@end

@implementation NumericTestTests {
    BOOL passed;
}

- (void)setUp
{
    passed = NO;
}

- (void)testMe {

    [self prepare];

    MyTest *test = [MyTest new];
    [test run: ^(NSError *error, double value) {
        passed = YES;
        [self notify:kGHUnitWaitStatusSuccess];
    }];
    [test runTest:fakeTest];

    [self waitForStatus:kGHUnitWaitStatusSuccess timeout:5.0];

    GHAssertTrue(passed, @"Completion handler not called");
}

Much cleaner and doesn't block the main thread.

Answer 3

Looks like Xcode 6 will solve the issue. https://developer.apple.com/library/prerelease/ios/documentation/DeveloperTools/Conceptual/testing_with_xcode/testing_3_writing_test_classes/testing_3_writing_test_classes.html

Answer 4

I appreciate that this question was asked and answered almost a year ago, but I can't help but disagree with the given answers. Testing asynchronous operations, particularly network operations, is a very common requirement, and is important to get right. In the given example, if you depend on actual network responses you lose some of the important value of your tests. Specifically, your tests become dependent on the availability and functional correctness of the server you're communicating with; this dependency makes your tests

• more fragile (what happens if the server goes down?)
• less comprehensive (how do you consistently test a failure response, or network error?)
• significantly slower imagine testing this:

Unit tests should run in fractions of a second. If you have to wait for a multi-second network response each time you run your tests then you're less likely to run them frequently.

Unit testing is largely about encapsulating dependencies; from the point of view of your code under test, two things happen:

1. Your method initiates a network request, probably by instantiating an NSURLConnection.
2. The delegate you specified receives a response via certain method calls.

Your delegate doesn't, or shouldn't, care where the response came from, whether from an actual response from a remote server or from your test code. You can take advantage of this to test asynchronous operations by simply generating the responses yourself. Your tests will run much faster, and you can reliably test success or failure responses.

This isn't to say you shouldn't run tests against the real web service you're working with, but those are integration tests and belong in their own test suite. Failures in that suite may mean the web service has changes, or is simply down. Since they're more fragile, automating them tends to have less value than automating your unit tests.

Regarding how exactly to go about testing asynchronous responses to a network request, you have a couple options. You could simply test the delegate in isolation by calling the methods directly (e.g. [someDelegate connection:connection didReceiveResponse:someResponse]). This will work somewhat, but is slightly wrong. The delegate your object provides may be just one of multiple objects in the delegate chain for a specific NSURLConnection object; if you call your delegate's methods directly you may be missing some key piece of functionality provided by another delegate further up the chain. As a better alternative, you can stub the NSURLConnection object you create and have it send the response messages to its entire delegate chain. There are libraries that will reopen NSURLConnection (amongst other classes) and do this for you. For example: https://github.com/pivotal/PivotalCoreKit/blob/master/SpecHelperLib/Extensions/NSURLConnection%2BSpec.m

Answer 5

I ran into the same question and found a different solution that works for me.

I use the "old school" approach for turning async operations into a sync flow by using a semaphore as follows:

// create the object that will perform an async operation
MyConnection *conn = [MyConnection new];
STAssertNotNil (conn, @"MyConnection init failed");

// create the semaphore and lock it once before we start
// the async operation
NSConditionLock *tl = [NSConditionLock new];
self.theLock = tl;
[tl release];    

// start the async operation
self.testState = 0;
[conn doItAsyncWithDelegate:self];

// now lock the semaphore - which will block this thread until
// [self.theLock unlockWithCondition:1] gets invoked
[self.theLock lockWhenCondition:1];

// make sure the async callback did in fact happen by
// checking whether it modified a variable
STAssertTrue (self.testState != 0, @"delegate did not get called");

// we're done
[self.theLock release]; self.theLock = nil;
[conn release];

Make sure to invoke

[self.theLock unlockWithCondition:1];

In the delegate(s) then.

Answer 6

If you're using a library such as AFNetworking or ASIHTTPRequest and have your requests managed via a NSOperation (or subclass with those libraries) then it's easy to test them against a test/dev server with an NSOperationQueue:

In test:

// create request operation

NSOperationQueue* queue = [[NSOperationQueue alloc] init];
[queue addOperation:request];
[queue waitUntilAllOperationsAreFinished];

// verify response

This essentially runs a runloop until the operation has completed, allowing all callbacks to occur on background threads as they normally would.