Sending SPDY requests results in “The request time

Question

My iOS app loads images from an nginx HTTP server. After I send 400+ such requests the networking 'gets stuck' and all subsequent HTTP requests result in "The request timed out" error. I can make the images load again only when I restart the app.

Details:

1. I am using NSURLSession.sharedSession().dataTaskWithURL to send four hundred HTTP GET requests to jpeg files.
2. Requests are sent sequentially, one after another. The interval between requests is 10 ms.
3. Each previous unfinished request is cancelled with cancel() method of NSURLSessionDataTask object.

Interestingly:

1. I can only have this issue with HTTPS requests and when SPDY is enabled on the server.
2. Non-secure HTTP requests work fine.
3. Non-SPDY HTTPS requests work fine. I tested it by turning SPDY off on the server side, in the nginx config.
4. Problem appears both on iOS 8 and 9, on physical device and in the simulator. Both on Wi-Fi and LTE.
5. When I look at nginx access logs, I can still see the 'stuck' requests coming in. Important nuance: the request log record appears at the exact moment when the iOS app is giving up on it after the time out period ends.
6. I was hoping to analyze HTTP requests with Charles Proxy but the problem cures itself when requests go through Charles. That is - everything works with Charles, much like effect in quantum mechanics when the fact of looking influences the outcome.
7. I was able to reproduce the issue when the iOS app connected to two different servers with vastly different nginx configurations. This probably means that the issue is not related to a particular nginx setup.
8. I analyzed the app using "Activity Monitor" instrument. The number of threads it is using during the bulk HTTP requests jumps from 5 to 10. In comparison, when I send just a single HTTP requests the number of threads jumps to 8. CPU load rarely goes above 30%.

What can be the cause of the issue? Can anyone recommend other ways or tools for analysing and debugging it?

Analysing with scheduling instrument

Demo app

This demo app reproduces the issue 100% of the time for me.

https://github.com/exchangegroup/ImageLoadDemo

Versions and settings

My nginx config: http://pastebin.com/pYYjdxfP

OS X: 10.10.4 (14E46), iOS: 8 and 9, Xcode: 7.0 (7A218), nginx: 1.9.4

Not ideal workaround

I managed to keep requests working only if I create a new NSURLSession for each individual request and clear the previous session with finishTasksAndInvalidate or invalidateAndCancel.

// Request 1

let configuration = NSURLSessionConfiguration.defaultSessionConfiguration()     
let session = NSURLSession(configuration: configuration)
session.dataTaskWithURL ...

// Request 2

// clear the previous request
session.finishTasksAndInvalidate()
let session2 = NSURLSession(configuration: configuration)
session2.dataTaskWithURL ...

Answer 1

One possibility is that iOS started sending the request, and then packet loss prevented the headers and request body from being fully delivered.

Another possibility that comes to mind is that your server may not be logging the request until it actually finishes trying to deliver it, which would make the time stamps in the server logs line up with when the connection was closed, rather than when it was opened. (IIRC, that's what Apache does; I haven't worked with nginx, so I can't speak for its behavior.) If that's the case, then this is just a simple connection stall. As for why it is stalling, I couldn't guess.

Does the problem occur exclusively for HTTPS traffic? If you can reproduce it with HTTP, you don't need Charles Proxy; just use OS X's "Internet Sharing" feature, and capture the packets with tcpdump or wireshark, listening on the bridge interface. If you can't reproduce it with HTTP, my money would be on a problem with fetching the CRLs or performing the OCSP check while validating the server's certificate.

Is your app ending up with a huge number of threads as a result of excessive async dispatching to new queues, by any chance? Because that could easily cause all sorts of odd misbehavior.

How long is the timeout? If it is too short, your app might simply be running up against performance limitations of the hardware while processing the results of 400 requests delivered in only four seconds.

Also, are you trying to schedule these requests simultaneously? Because I seem to recall reading about a bug that causes NSURLSession to hit a brick wall if you start too many tasks in a single session at the same time. You might try adding tasks only after the number of tasks in a session drops below some threshold and see if that fixes the problem.