Best way to access all movie frames in iOS

问题:

Im trying to edit existing movie with added effects on top thus I need ability to scan all movie frames, get them as UIImage, apply effect and then either update that frame or write it into new movie. What I found is people suggesting to use AVAssetImageGenerator. Below is my final edited sample of how Im doing it:

-(void)processMovie:(NSString*)moviePath {
    NSURL* url = [NSURL fileURLWithPath:moviePath];
    AVURLAsset *asset=[[AVURLAsset alloc] initWithURL:url options:nil];
    float movieTimeInSeconds = CMTimeGetSeconds([movie duration]);
    AVAssetImageGenerator *generator = [[AVAssetImageGenerator alloc] initWithAsset:asset];
    generator.requestedTimeToleranceBefore = generator.requestedTimeToleranceAfter = kCMTimeZero;
    generator.appliesPreferredTrackTransform=TRUE;
    [asset release];

    // building array of time with steps as 1/10th of second
    NSMutableArray* arr = [[[NSMutableArray alloc] init] retain];
    for(int i=0; i<movieTimeInSeconds*10; i++) {
        [arr addObject:[NSValue valueWithCMTime:CMTimeMake(i,10)]];
    }

    AVAssetImageGeneratorCompletionHandler handler = ^(CMTime requestedTime, CGImageRef im, CMTime actualTime, AVAssetImageGeneratorResult result, NSError *error){
        if (result == AVAssetImageGeneratorSucceeded) {
            UIImage* img = [UIImage imageWithCGImage:im];
            // use img to apply effect and write it in new movie
            // after last frame do [generator release];
        }
    };

    [generator generateCGImagesAsynchronouslyForTimes:arr completionHandler:handler];
}

2 problems with this approach:

1. I need to guess what timesteps movie has or as in my example assume that its 10FPS for movie. Actual timesteps are not evenly spaced plus sometime we have skipped frames.
2. It is slow to scan through the frames. If movie is recorded in high resolution I get around 0.5 seconds to retrieve UIImage for each frame.

It seems unnatural. Question: is there better way to scan all original frames of the movie?

回答1:

Finally found what I was looking for. Below is my code that scans all samples in the movie. BTW using AVAssetImageGenerator was working but was very slow. This approach is fast.

inputUrl = [NSURL fileURLWithPath:filePath];
AVURLAsset* movie = [AVURLAsset URLAssetWithURL:inputUrl options:nil];

NSArray* tracks = [movie tracksWithMediaType:AVMediaTypeVideo];
AVAssetTrack* track = [tracks firstObject];

NSError* error = nil;
AVAssetReader* areader = [[AVAssetReader alloc] initWithAsset:movie error:&error];
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
                         [NSNumber numberWithInt:kCVPixelFormatType_32ARGB], kCVPixelBufferPixelFormatTypeKey,
                         nil];

AVAssetReaderTrackOutput* rout = [[AVAssetReaderTrackOutput alloc] initWithTrack:track outputSettings:options];
[areader addOutput:rout];

[areader startReading];
while ([areader status] == AVAssetReaderStatusReading) {
    CMSampleBufferRef sbuff = [rout copyNextSampleBuffer];
    if (sbuff) {
        dispatch_sync(dispatch_get_main_queue(), ^{
            [self writeFrame:sbuff];
        });
    }
}

回答2:

You could get the frame rate of the video. Have a look at the code here: Given a URL to a movie, how can retrieve it's info?

You only really need to get the AVAssetTrack and its nominalFrameRate.

回答3:

Although this may not help you with the skipped frame issue you could do a cal on the AVAssetTrack along with the nominalFrameRate to achieve the desired results.