How to convert a kCVPixelFormatType_420YpCbCr8BiPl

Question

i have captured video using AVFoundation .i have set (video setting )and get in outputsamplebuffer kCVPixelFormatType_420YpCbCr8BiPlanarFullRange format. But i need YUV420 format for further processing.

For that i use libyuv framework.

LIBYUV_API
int NV12ToI420(const uint8* src_y, int src_stride_y,
           const uint8* src_uv, int src_stride_uv,
           uint8* dst_y, int dst_stride_y,
           uint8* dst_u, int dst_stride_u,
           uint8* dst_v, int dst_stride_v,
           int width, int height);

 libyuv::NV12ToI420(src_yplane, inWidth ,
                   src_uvplane, inWidth,
                   dst_yplane, inWidth,
                   dst_vplane, inWidth / 2,
                   dst_uplane, inWidth / 2,
                   inWidth,  inHeight);

But i am getting output buffer is full green color? i done any mistake for that process pls help me?

Answer 1

You need convert your data to I420, I am processing camera too, but on Android. I think it should be similar on iOS. Android raw camera is NV21 or NV16 format, I convert from NV21 or NV16 to YV12, I420 is almost the same as YV12:

BYTE m_y[BIG_VIDEO_CX * BIG_VIDEO_CY], 
     m_u[(BIG_VIDEO_CX/2) * (BIG_VIDEO_CY/2)],
     m_v[(BIG_VIDEO_CX/2) * (BIG_VIDEO_CY/2)];

    void NV21_TO_YV12(BYTE *data)
{
    int width = BIG_VIDEO_CX;
    int height = BIG_VIDEO_CY;
        m_y2=data;
    data=&data[width*height];
    for (uint32_t i=0; i<(width/2)*(height/2); ++i)
    {
        m_v[i]=*data;
        m_u[i]=*(data+1);
        data+=2;
    }
}
void NV16_TO_YV12(BYTE *data)
{
    int width = BIG_VIDEO_CX;
    int height = BIG_VIDEO_CY;
    m_y2=data;
    const BYTE* src_uv = (const BYTE*)&data[width*height];
    BYTE* dst_u = m_u;
    BYTE* dst_v = m_v;
    for (uint32_t y=0; y<height/2; ++y)
    {
        const BYTE* src_uv2 = src_uv + width;
        for (uint32_t x=0; x<width/2; ++x)
        {
            dst_u[x]=(src_uv[0]+src_uv2[0]+1)>>1;
            dst_v[x]=(src_uv[1]+src_uv2[1]+1)>>1;
            src_uv+=2;
            src_uv2+=2;
        }
        src_uv=src_uv2;
        dst_u+=width/2;
        dst_v+=width/2;
    }
}

Answer 2

Here is how I do it on iOS in my captureOutput after I get a raw video frame from AVCaptureSession(kCVPixelFormatType_420YpCbCr8BiPlanarFullRange):

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection 
{

    CVImageBufferRef videoFrame = CMSampleBufferGetImageBuffer(sampleBuffer);

    CFRetain(sampleBuffer);

    CVPixelBufferLockBaseAddress(videoFrame, 0);
    size_t _width = CVPixelBufferGetWidth(videoFrame);
    size_t _height = CVPixelBufferGetHeight(videoFrame);

    const uint8* plane1 = (uint8*)CVPixelBufferGetBaseAddressOfPlane(videoFrame,0);
    const uint8* plane2 = (uint8*)CVPixelBufferGetBaseAddressOfPlane(videoFrame,1);
    size_t plane1_stride = CVPixelBufferGetBytesPerRowOfPlane (videoFrame, 0);
    size_t plane2_stride = CVPixelBufferGetBytesPerRowOfPlane (videoFrame, 1);

    size_t plane1_size = plane1_stride * CVPixelBufferGetHeightOfPlane(videoFrame, 0);
    size_t plane2_size = CVPixelBufferGetBytesPerRowOfPlane (videoFrame, 1) * CVPixelBufferGetHeightOfPlane(videoFrame, 1);

    size_t frame_size = plane1_size + plane2_size;

    uint8* buffer = new uint8[ frame_size ];
    uint8* dst_u = buffer + plane1_size;
    uint8* dst_v = dst_u + plane1_size/4;

    // Let libyuv convert
    libyuv::NV12ToI420(/*const uint8* src_y=*/plane1, /*int src_stride_y=*/plane1_stride,
                /*const uint8* src_uv=*/plane2, /*int src_stride_uv=*/plane2_stride,
                   /*uint8* dst_y=*/buffer, /*int dst_stride_y=*/plane1_stride,
                   /*uint8* dst_u=*/dst_u, /*int dst_stride_u=*/plane2_stride/2,
                   /*uint8* dst_v=*/dst_v, /*int dst_stride_v=*/plane2_stride/2,
                   _width, _height);

    CVPixelBufferUnlockBaseAddress(videoFrame, 0);
    CFRelease( sampleBuffer)

    // TODO: call your method here with 'buffer' variable. note that you need to deallocated the buffer after using it
  }

I made the code a bit more descriptive for clarity.

Answer 3

Android is NV21, which libyuv supports with Arm as well as Intel. It can also rotate by 90, 180 or 270 as part of the conversion if necessary for orientation. The Arm optimized version is about 2x faster than C

C
NV12ToI420_Opt (782 ms)
NV21ToI420_Opt (764 ms)

Arm (Neon optimized)
NV12ToI420_Opt (398 ms)
NV21ToI420_Opt (381 ms)

Curious you use NV16 on Android. I'd expect NV61 for consistency with NV21. Your code looks correct, but would nicely optimize into Neon using vrhadd.u8. File a libyuv issue if you'd like to see that. https://code.google.com/p/libyuv/issues/list

Answer 4

Looks right. Make sure your src_uvplane points to src_yplane + inWidth * inHeight