AFNetworking + NsOperationQueue - Download thousan

I am working on a task (iOS5 + only) that involves downloading thousands of images from the server. The images belong to certain categories and each category can have hundreds of images. What I need to do is this :-

1) Make sure the app downloads any missing images in background if the app is active (even when the user is browsing some other areas of the app that are not related to photos).

2) When the user clicks on a Photo Category, the images in that Category must be downloaded as high priority because those are the ones that need to be visible immediately.

All of the above happens only if the image is not already available offline. Once it's downloaded, the image will be used from local storage.

To solve this, the logic I am using is this :-

1) In AppDelegate.m, in applicationDidBecomeActive, I start downlading any missing images. To do this, I make a Core Data query, find out which images are missing, and start downloading them in a thread with BACKGROUND priority. Something like this :-

 dispatch_queue_t imageDownloadQueue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND, 0);
dispatch_async(imageDownloadQueue, ^{
    [DataDownloader downloadMissingImages];
});
dispatch_release(imageDownloadQueue);

The downloadMissingImages code looks like this :-

NSOperationQueue *downloadQueue = [[NSOperationQueue alloc] init];
        downloadQueue.maxConcurrentOperationCount = 20;

        for(MyImage *img in matches)
        {
            NSURLRequest *request = [NSURLRequest requestWithURL:img.photoUrl];
            AFImageRequestOperation *operation = [AFImageRequestOperation imageRequestOperationWithRequest:request success:^(UIImage *image) {

                [MyImage imageFromAPI:image inManagedObjectContext:document.managedObjectContext];

                NSLog(@"Successfully downloaded image for %@", img.title);      
            }];

            [downloadQueue addOperation:operation];
        }

This works, but it blocks the main UI and the app crashes after a while. This is when I try to download about 700 images. With more images, it would certainly crash.

2) When a user clicks on a category, I need to download those images first as they must be shown to the user immediately. I am not sure how I can interrupt the missingImages call and tell it to start downloading certain images before others.

So, basically, I need to download all the missing images in the background but if the user is browsing photo category, those images must take high priority in the download queue.

I am at a loss how to get this working efficiently. Any thoughts?

The crash logs look like this

PAPP(36373,0xb065f000) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
PAPP(36373,0xb065f000) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Jun 24 11:39:45 MacBook-Pro.local PAPP[36373] <Error>: ImageIO: JPEG    Insufficient memory (case 4)

Thanks in advance.

About the crash, I guess that your app is killed due to either of two options:

the app becoming unresponsive (and thus not responding to the iOS sentinel process);
too much memory used in the loop creating over 700 request operations.

To clarify what is really happening, you should provide more info about the crash (the console log). In any case, the fix would be loading the images in chunks of maybe 10 or 20 each (you could go even 1 by 1, if you like, I don't see much problem with that).

About the second point, what about this:

download a higher priority image in the main thread (via an async download, of course, to avoid blocking the UI);
before starting downloading an "offline" image, you check whether the image has been already downloaded in the meanwhile through a "higher-priority" download.

To handle point 2 well, you would probably need to queue your custom operation instead of an AFImageRequestOperation in order to do the check before the actual download.

EDIT:

About downloading images in chunk, what you could do is using dispatch groups to group your networking operations:

dispatch_group_t group = dispatch_group_create();

<your_core_data_query>

for (...) {
    dispatch_group_enter(group);

        NSURLRequest *request = [NSURLRequest requestWithURL:img.photoUrl];
        AFImageRequestOperation *operation = [AFImageRequestOperation imageRequestOperationWithRequest:request success:^(UIImage *image) {
            [MyImage imageFromAPI:image inManagedObjectContext:document.managedObjectContext];
            NSLog(@"Successfully downloaded image for %@", img.title);

            dispatch_group_leave(group);     //<== NOTICE THIS
        }];

}

dispatch_group_wait(group, DISPATCH_TIME_FOREVER);
dispatch_release(group);

In this sample, I am using a dispatch group to group a number of async operations and wait for them being executed all; when dispatch_group_wait returns, you can execute another round of that (querying core data then dispatching ops).

About your other question (how do I check whether a higher priority queue has already downloaded a certain image), you should do a core data query before executing each AFImageRequestOperation; one possibility is deriving your own class of it and overriding the start method to do the check.

On both accounts, you could simplify much the logics of all this by downloading the images one at the time (i.e., the for (...) loop would not be there; you simply query the next image to download and download it; before downloading you check it is not already there.

I would suggest going this easier path.

Hope it helps.