How to change orientation of camera preview callba

This is a variation on a question often asked hereabouts but I don't see this exact situation, so I'll throw it out there.

I have an onPreviewFrame callback set up. This gets a byte[] with NV21 data in it. We h.264 encode it and send it out as a video stream. On the other side, we see the video skewed, either 90 or 270 degrees, depending on the phone.

So the question is, how to rotate the data, not just the preview image? Camera.Parameters.setRotation only affects taking the picture, not video. Camera.setDisplayOrientation specifically says it only affects the displaying preview, not the frame bytes:

This does not affect the order of byte array passed in onPreviewFrame(byte[], Camera), JPEG pictures, or recorded videos.

So is there a way, at any API level, to change the orientation of the byte array? Failing that, can you even rotate the NV21 (YVU) format that this come in, or do I need to RGB it first?

回答1:

Turns out you do need to rotate each frame yourself before sending it off. We ended up using libyuv, which has a very convenient function that both rotates and converts it - libyuv::ConvertToI420

https://code.google.com/p/libyuv/

回答2:

I think that you would need to rotate the picture yourself. I did it once using the NDK and the leptonica library. A look at my code should get you started. Performance was okayish on a Samsung Galaxy S2 (i think i got around 15 frames or so). Since i was pushing the result into an openGL texture i had to also swizzle the color bytes around.. You could speed it up by rotating the image directly in the loop which decodes the yuv data..

mPix32 and mPix8 were previously allocated to hold the converted data.You would need to replace with your own image data structure of course..

jint Java_de_renard_ImageFilter_nativeProcessImage(JNIEnv *env, jobject javathis, jbyteArray frame) {
    ....
    jbyte *data_buffer = env->GetByteArrayElements(frame, NULL);
    l_uint8 *byte_buffer = (l_uint8 *) data_buffer;
    yuvToPixFast(byte_buffer, mPix32, mPix8);
    env->ReleaseByteArrayElements(frame, data_buffer, JNI_ABORT);
    ....
}

static inline void yuvToPixFast(unsigned char* pY, Pix* pix32, Pix* pix8) {
    int i, j;
    int nR, nG, nB;
    int nY, nU, nV;
    l_uint32* data = pixGetData(pix32);
    l_uint32* data8 = pixGetData(pix8);
    l_int32 height = pixGetHeight(pix32);
    l_int32 width = pixGetWidth(pix32);
    l_int32 wpl = pixGetWpl(pix32);
    l_int32 wpl8 = pixGetWpl(pix8);
    l_uint8 **lineptrs = pixSetupByteProcessing(pix8, NULL, NULL);
    l_uint8* line8;

    //memcpy(data8,pY,height*width);

    unsigned char* pUV = pY + width * height;

    for (i = 0; i < height; i++) {
        nU = 0;
        nV = 0;
        unsigned char* uvp = pUV + (i >> 1) * width;
        line8 = lineptrs[i];
        memcpy(line8, pY, wpl8 * 4);

        for (j = 0; j < width; j++) {

            if ((j & 1) == 0) {
                nV = (0xff & *uvp++) - 128;
                nU = (0xff & *uvp++) - 128;
            }
            // Yuv Convert
            nY = *(pY++);
            //*line8++ = (l_uint8) nY;
            nY -= -16;

            if (nY < 0) {
                nY = 0;
            }
            int y1192 = nY * 1192;

            /*double saturation to increase cartoon effect*/
            //nU<<=1;
            //nV<<=1;

            nB = y1192 + 2066 * nU;
            nG = y1192 - 833 * nV - 400 * nU;
            nR = y1192 + 1634 * nV;

            if (nR < 0) {
                nR = 0;
            } else if (nR > 262143) {
                nR = 262143;
            }
            if (nG < 0) {
                nG = 0;
            } else if (nG > 262143) {
                nG = 262143;
            }
            if (nB < 0) {
                nB = 0;
            } else if (nB > 262143) {
                nB = 262143;
            }
            //RGBA
            //ABGR
            *data++ = ((nR << 14) & 0xff000000) | ((nG << 6) & 0xff0000) | ((nB >> 2) & 0xff00) | (0xff);
            //*data++ = (0x00 << 24) | (0xff<<16) | (0x00<<8) | ( 0xff) ;
            //*data++ = (0xff << 24) | ((nB << 6) & 0xff0000) | ((nG >> 2) & 0xff00) | ((nR >> 10) & 0xff);
        }
    }
    pixCleanupByteProcessing(pix8, lineptrs);

}