Android: how to display camera preview with callba

Question

What I need to do is quite simple, I want to manually display preview from camera using camera callback and I want to get at least 15fps on a real device. I don't even need the colors, I just need to preview grayscale image. Images from camera are in YUV format and you have to process it somehow, which is the main performance problem. I'm using API 8.

In all cases I'm using camera.setPreviewCallbackWithBuffer(), that is faster than camera.setPreviewCallback(). It seems that I cant get about 24 fps here, if I'm not displaying the preview. So there is not the problem.

I have tried these solutions:

1. Display camera preview on a SurfaceView as a Bitmap. It works, but the performance is about 6fps.

baos = new ByteOutputStream();
yuvimage=new YuvImage(cameraFrame, ImageFormat.NV21, prevX, prevY, null);

yuvimage.compressToJpeg(new Rect(0, 0, prevX, prevY), 80, baos);
jdata = baos.toByteArray();

bmp = BitmapFactory.decodeByteArray(jdata, 0, jdata.length); // Convert to Bitmap, this is the main issue, it takes a lot of time

canvas.drawBitmap(bmp , 0, 0, paint);

---

2. Display camera preview on a GLSurfaceView as a texture. Here I was displaying only luminance data (greyscale image), which is quite easy, it requires only one arraycopy() on each frame. I can get about 12fps, but I need to apply some filters to the preview and it seems, that it can't be done fast in OpenGL ES 1. So I can't use this solution. Some details of this in another question.

---

3. Display camera preview on a (GL)SurfaceView using NDK to process the YUV data. I find a solution here that uses some C function and NDK. But I didn't manage to use it, here some more details. But anyway, this solution is done to return ByteBuffer to display it as a texture in OpenGL and it won't be faster than the previous attempt. So I would have to modify it to return int[] array, that can be drawn with canvas.drawBitmap(), but I don't understand C enough to do this.

---

So, is there any other way that I'm missing or some improvement to the attempts I tried?

Answer 1

I think Michael's on the right track. First you can try this method to convert from RGB to Grayscale. Clearly it's doing almost the same thing as his,but a little more succinctly for what you want.

//YUV Space to Greyscale
static public void YUVtoGrayScale(int[] rgb, byte[] yuv420sp, int width, int height){
    final int frameSize = width * height;
    for (int pix = 0; pix < frameSize; pix++){
        int pixVal = (0xff & ((int) yuv420sp[pix])) - 16;
        if (pixVal < 0) pixVal = 0;
        if (pixVal > 255) pixVal = 255;
        rgb[pix] = 0xff000000 | (pixVal << 16) | (pixVal << 8) | pixVal;
    }
}

}

Second, don't create a ton of work for the garbage collector. Your bitmaps and arrays are going to be a fixed size. Create them once, not in onFramePreview.

Doing that you'll end up with something that looks like this:

    public PreviewCallback callback = new PreviewCallback() {
    @Override
    public void onPreviewFrame(byte[] data, Camera camera) {
        if ( (mSelectView == null) || !inPreview )
            return;
        if (mSelectView.mBitmap == null)
        {
            //initialize SelectView bitmaps, arrays, etc
            //mSelectView.mBitmap = Bitmap.createBitmap(mSelectView.mImageWidth, mSelectView.mImageHeight, Bitmap.Config.RGB_565);
           //etc

        }
        //Pass Image Data to SelectView
        System.arraycopy(data, 0, mSelectView.mYUVData, 0, data.length);
        mSelectView.invalidate();
    }
};

And then the canvas where you want to put it looks like this:

class SelectView extends View {
Bitmap mBitmap;
Bitmap croppedView;
byte[] mYUVData;
int[] mRGBData;
int mImageHeight;
int mImageWidth;

public SelectView(Context context){
    super(context);
    mBitmap = null;
    croppedView = null;
}

@Override
protected void onDraw(Canvas canvas){
    if (mBitmap != null)
    {
        int canvasWidth = canvas.getWidth();
        int canvasHeight = canvas.getHeight();
        // Convert from YUV to Greyscale
        YUVtoGrayScale(mRGBData, mYUVData, mImageWidth, mImageHeight);
            mBitmap.setPixels(mRGBData, 0, mImageWidth, 0, 0, mImageWidth, mImageHeight);
            Rect crop = new Rect(180, 220, 290, 400);
        Rect dst = new Rect(0, 0, canvasWidth, (int)(canvasHeight/2));
        canvas.drawBitmap(mBitmap, crop, dst, null);
    }
    super.onDraw(canvas);
}

This example shows a cropped and distorted selection of the camera preview in real time, but you get the idea. It runs at high FPS on a Nexus S in greyscale and should work for your needs as well.

Answer 2

For very basic and simple effects, there is

Camera.Parameters parameters = mCamera.getParameters();
parameters.setColorEffect(Parameters.EFFECT_AQUA);

I figured out that this effects do DIFFERENTLY depending on the device. For instance, on my phone (galaxy s II) it looks kinda like a comic effect as in contrast to the galaxy s 1 it is 'just' a blue shade.

It's pro: It's working as live-preview.

I looked around some other camera apps and they obviously also faced this problem. So what did they do? They are capturing the default camera image, applying a filter to the bitmap data, and show this image in a simple ImageView. It's for sure not that cool as in live preview, but you won't ever face performance problems.

Answer 3

Is this not what you want? Just use a SurfaceView in your layout, then somewhere in your init like onResume():

SurfaceView surfaceView = ...
SurfaceHolder holder = surfaceView.getHolder();
...
Camera camera = ...;
camera.setPreviewDisplay(holder);

It just sends the frames straight to the view as fast as they arrive.

If you want grayscale, modify the camera parameters with setColorEffect("mono").

Answer 4

I'm working on exactly the same issue, but haven't got quite as far as you have.

Have you considered drawing the pixels directly to the canvas without encoding them to JPEG first? Inside the OpenCV kit http://sourceforge.net/projects/opencvlibrary/files/opencv-android/2.3.1/OpenCV-2.3.1-android-bin.tar.bz2/download (which doesn't actually use opencv; don't worry), there's a project called tutorial-0-androidcamera that demonstrates converting the YUV pixels to RGB and then writing them directly to a bitmap.

The relevant code is essentially:

public void onPreviewFrame(byte[] data, Camera camera, int width, int height) {
    int frameSize = width*height;
    int[] rgba = new int[frameSize+1];

    // Convert YUV to RGB
    for (int i = 0; i < height; i++)
        for (int j = 0; j < width; j++) {
            int y = (0xff & ((int) data[i * width + j]));
            int u = (0xff & ((int) data[frameSize + (i >> 1) * width + (j & ~1) + 0]));
            int v = (0xff & ((int) data[frameSize + (i >> 1) * width + (j & ~1) + 1]));
            y = y < 16 ? 16 : y;

            int r = Math.round(1.164f * (y - 16) + 1.596f * (v - 128));
            int g = Math.round(1.164f * (y - 16) - 0.813f * (v - 128) - 0.391f * (u - 128));
            int b = Math.round(1.164f * (y - 16) + 2.018f * (u - 128));

            r = r < 0 ? 0 : (r > 255 ? 255 : r);
            g = g < 0 ? 0 : (g > 255 ? 255 : g);
            b = b < 0 ? 0 : (b > 255 ? 255 : b);

            rgba[i * width + j] = 0xff000000 + (b << 16) + (g << 8) + r;
        }

    Bitmap bmp = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
    bmp.setPixels(rgba, 0/* offset */, width /* stride */, 0, 0, width, height);
    Canvas canvas = mHolder.lockCanvas();
    if (canvas != null) {
        canvas.drawBitmap(bmp, (canvas.getWidth() - width) / 2, (canvas.getHeight() - height) / 2, null);
        mHolder.unlockCanvasAndPost(canvas);
    } else {
        Log.w(TAG, "Canvas is null!");
    }
    bmp.recycle();
}

Of course you'd have to adapt it to meet your needs (ex. not allocating rgba each frame), but it might be a start. I'd love to see if it works for you or not -- i'm still fighting problems orthogonal to yours at the moment.

Answer 5

I believe I read in a blog that the grayscale data is in the first x*y bytes. Yuv should represent luminance, so the data is there, although it isn't a perfect grayscale. Its great for relative brightness, but not grayscale, as each color isn't as bright as each other in rgb. Green is usually given a stronger weight in luminosity conversions. Hope this helps!

Answer 6

Is there any special reason that you are forced to use GLES 1.0 ?

Because if not, see the accepted answer here: Android SDK: Get raw preview camera image without displaying it

Generally it mentions using Camera.setPreviewTexture() in combination with GLES 2.0. In GLES 2.0 you can render a full-screen-quad all over the screen, and create whatever effect you want.

It's most likely the fastest way possible.