How do I draw onto a CVPixelBufferRef that is plan

问题:

I have received a CMSampleBufferRef from a system API that contains CVPixelBufferRefs that are not RGBA (linear pixels). The buffer contains planar pixels (such as 420f aka kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange aka yCbCr aka YUV).

I would like to modify do some manipulation of this video data before sending it off to VideoToolkit to be encoded to h264 (drawing some text, overlaying a logo, rotating the image, etc), but I'd like for it to be efficient and real-time. Buuuut planar image data looks suuuper messy to work with -- there's the chroma plane and the luma plane and they're different sizes and... Working with this on a byte level seems like a lot of work.

I could probably use a CGContextRef and just paint right on top of the pixels, but from what I can gather it only supports RGBA pixels. Any advice on how I can do this with as little data copying as possible, yet as few lines of code as possible?

回答1:

CGBitmapContextRef can only paint into something like 32ARGB, correct. This means that you will want to create ARGB (or RGBA) buffers, and then find a way to very quickly transfer YUV pixels onto this ARGB surface. This recipe includes using CoreImage, a home-made CVPixelBufferRef through a pool, a CGBitmapContextRef referencing your home made pixel buffer, and then recreating a CMSampleBufferRef resembling your input buffer, but referencing your output pixels. In other words,

1. Fetch the incoming pixels into a CIImage.
2. Create a CVPixelBufferPool with the pixel format and output dimensions you are creating. You don't want to create CVPixelBuffers without a pool in real time: you will run out of memory if your producer is too fast; you'll fragment your RAM as you won't be reusing buffers; and it's a waste of cycles.
3. Create a CIContext with the default constructor that you'll share between buffers. It contains no external state, but documentation says that recreating it on every frame is very expensive.
4. On incoming frame, create a new pixel buffer. Make sure to use an allocation threshold so you don't get runaway RAM usage.
5. Lock the pixel buffer
6. Create a bitmap context referencing the bytes in the pixel buffer
7. Use CIContext to render the planar image data into the linear buffer
8. Perform your app-specific drawing in the CGContext!
9. Unlock the pixel buffer
10. Fetch the timing info of the original sample buffer
11. Create a CMVideoFormatDescriptionRef by asking the pixel buffer for its exact format
12. Create a sample buffer for the pixel buffer. Done!

Here's a sample implementation, where I have chosen 32ARGB as the image format to work with, as that's something that both CGBitmapContext and CoreVideo enjoys working with on iOS:

{
    CGPixelBufferPoolRef *_pool;
    CGSize _poolBufferDimensions;
}
- (void)_processSampleBuffer:(CMSampleBufferRef)inputBuffer
{
    // 1. Input data
    CVPixelBufferRef inputPixels = CMSampleBufferGetImageBuffer(inputBuffer);
    CIImage *inputImage = [CIImage imageWithCVPixelBuffer:inputPixels];

    // 2. Create a new pool if the old pool doesn't have the right format.
    CGSize bufferDimensions = {CVPixelBufferGetWidth(inputPixels), CVPixelBufferGetHeight(inputPixels)};
    if(!_pool || !CGSizeEqualToSize(bufferDimensions, _poolBufferDimensions)) {
        if(_pool) {
            CFRelease(_pool);
        }
        OSStatus ok0 = CVPixelBufferPoolCreate(NULL,
            NULL, // pool attrs
            (__bridge CFDictionaryRef)(@{
                (id)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_32ARGB),
                (id)kCVPixelBufferWidthKey: @(bufferDimensions.width),
                (id)kCVPixelBufferHeightKey: @(bufferDimensions.height),
            }), // buffer attrs
            &_pool
        );
        _poolBufferDimensions = bufferDimensions;
        assert(ok0 == noErr);
    }

    // 4. Create pixel buffer
    CVPixelBufferRef outputPixels;
    OSStatus ok1 = CVPixelBufferPoolCreatePixelBufferWithAuxAttributes(NULL,
        _pool,
        (__bridge CFDictionaryRef)@{
            // Opt to fail buffer creation in case of slow buffer consumption
            // rather than to exhaust all memory.
            (__bridge id)kCVPixelBufferPoolAllocationThresholdKey: @20
        }, // aux attributes
        &outputPixels
    );
    if(ok1 == kCVReturnWouldExceedAllocationThreshold) {
        // Dropping frame because consumer is too slow
        return;
    }
    assert(ok1 == noErr);

    // 5, 6. Graphics context to draw in
    CGColorSpaceRef deviceColors = CGColorSpaceCreateDeviceRGB();
    OSStatus ok2 = CVPixelBufferLockBaseAddress(outputPixels, 0);
    assert(ok2 == noErr);
    CGContextRef cg = CGBitmapContextCreate(
        CVPixelBufferGetBaseAddress(outputPixels), // bytes
        CVPixelBufferGetWidth(inputPixels), CVPixelBufferGetHeight(inputPixels), // dimensions
        8, // bits per component
        CVPixelBufferGetBytesPerRow(outputPixels), // bytes per row
        deviceColors, // color space
        kCGImageAlphaPremultipliedFirst // bitmap info
    );
    CFRelease(deviceColors);
    assert(cg != NULL);

    // 7
    [_imageContext render:inputImage toCVPixelBuffer:outputPixels];

    // 8. DRAW
    CGContextSetRGBFillColor(cg, 0.5, 0, 0, 1);
    CGContextSetTextDrawingMode(cg, kCGTextFill);
    NSAttributedString *text = [[NSAttributedString alloc] initWithString:@"Hello world" attributes:NULL];
    CTLineRef line = CTLineCreateWithAttributedString((__bridge CFAttributedStringRef)text);
    CTLineDraw(line, cg);
    CFRelease(line);

    // 9. Unlock and stop drawing
    CFRelease(cg);
    CVPixelBufferUnlockBaseAddress(outputPixels, 0);

    // 10. Timings
    CMSampleTimingInfo timingInfo;
    OSStatus ok4 = CMSampleBufferGetSampleTimingInfo(inputBuffer, 0, &timingInfo);
    assert(ok4 == noErr);

    // 11. VIdeo format
    CMVideoFormatDescriptionRef videoFormat;
    OSStatus ok5 = CMVideoFormatDescriptionCreateForImageBuffer(NULL, outputPixels, &videoFormat);
    assert(ok5 == noErr);

    // 12. Output sample buffer
    CMSampleBufferRef outputBuffer;
    OSStatus ok3 = CMSampleBufferCreateForImageBuffer(NULL, // allocator
        outputPixels, // image buffer 
        YES, // data ready
        NULL, // make ready callback
        NULL, // make ready refcon
        videoFormat,
        &timingInfo, // timing info
        &outputBuffer // out
    );
    assert(ok3 == noErr);

    [_consumer consumeSampleBuffer:outputBuffer];
    CFRelease(outputPixels);
    CFRelease(videoFormat);
    CFRelease(outputBuffer);
}