Take ownership of memory from CVImageBufferRef

Question

I am making a simple pipeline that gets images from AVCaptureSession, processes them in OpenCV, and then renders them in OpenGL. It is based on RosyWriter but without the audio and recording capabilities. The OpenCV processing looks like

- (void)processPixelBuffer: (CVImageBufferRef)pixelBuffer 
{
CVPixelBufferLockBaseAddress( pixelBuffer, 0 );
int bufferWidth = CVPixelBufferGetWidth(pixelBuffer);
int bufferHeight = CVPixelBufferGetHeight(pixelBuffer);
unsigned char *pixel = (unsigned char *)CVPixelBufferGetBaseAddress(pixelBuffer);

cv::Mat image = cv::Mat(bufferWidth,bufferHeight,CV_8UC4,pixel);
//do any processing
[self setDisplay_matrix:image];
CVPixelBufferUnlockBaseAddress( pixelBuffer, 0 );
}

In this function thus far I have not copied any memory and I would like to keep it that way. The issue is that pixelBuffer may still own the memory contained in display_image. The processing code may or may not allocate new memory and store it in image. If the processing did not allocate new memory I have to pass pixelBuffer around with display_matrix to keep the data from being erased. Is there a way for me to take ownership of the memory? I want to destroy pixelBuffer without destroying the memory it points too.

On a related note, what exactly does LockBaseAddress do? If I were passing around a cv::Mat, CVImageBufferRef pair would I have to lock the base address every time i want to modify/use the data with cv::Mat?

Answer 1

You can create data provider from base address data without copying, and then create UIImage from this data provider. To avoid buffer reusing while you referring this image you need to retain sample buffer and lock base address. They should be unlocked and released automatically when you will forget this image object:

- (void)captureOutput:(AVCaptureOutput *)captureOutput 
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
       fromConnection:(AVCaptureConnection *)connection
{
    // Retain sample buffer and lock base address
    CFRetain(sampleBuffer);
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    CVPixelBufferLockBaseAddress(imageBuffer, 0);

    size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
    size_t height = CVPixelBufferGetHeight(imageBuffer);
    size_t width = CVPixelBufferGetWidth(imageBuffer);
    void *baseAddress = (void *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);

    UIImage *image = imageFromData(baseAddress, width, height, bytesPerRow, sampleBuffer);

    // Now you can store this UIImage as long as you want
}

I've got imageFromData from this project https://github.com/k06a/UIImage-DecompressAndMap/blob/master/UIImage%2BDecompressAndMap.m and adopted it a little bit:

UIImage *imageFromData(void *data, size_t width, size_t height, size_t bytesPerRow, CMSampleBufferRef sampleBuffer)
{
    CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
    CGDataProviderRef provider = CGDataProviderCreateWithData((void *)sampleBuffer, data, bytesPerRow * height, munmap_wrapper);
    CGImageRef inflatedImage = CGImageCreate(width, height, 8, 4*8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst, provider, NULL, NO, kCGRenderingIntentDefault);

    CGColorSpaceRelease(colorSpace);
    CGDataProviderRelease(provider);

    UIImage *img = [UIImage imageWithCGImage:inflatedImage scale:scale orientation:UIImageOrientationUp];
    CGImageRelease(inflatedImage);
    return img;
}

You also need to provide unlock_function:

void unlock_function(void *info, const void *data, size_t size)
{
    // Unlock base address release sample buffer
    CMSampleBufferRef sampleBuffer = (CMSampleBufferRef)info;
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
    CFRelease(sampleBuffer);
}