Capturing H264 stream with OpenCV

2020-07-22 10:17发布

问题:

So. I have been trying to get my Raspberry Pi 2 to capture H264 stream with OpenCV from my Logitech C920 for quite some time now. I have been scavenging the internet for info, but with no luck.

A short system description:

  • Raspberry Pi 2, running Raspbian, Kernel 3.18
  • Logitech HD Pro Webcam c920
  • OpenCV 2.4.11
  • boneCV - Credits to Derek Molloy (https://github.com/derekmolloy/boneCV)
  • libx264 and FFMPEG (built with x264 support)
  • libv4l-dev, v4l-utils, qv4l2, v4l2ucp

I know OpenCV forces format to BGR24 (MJPG). This is specified in cap_libv4l.cpp. It looks like this(line 692->):

  /* libv4l will convert from any format to V4L2_PIX_FMT_BGR24 */
  CLEAR (capture->form);
  capture->form.type                = V4L2_BUF_TYPE_VIDEO_CAPTURE;
  capture->form.fmt.pix.pixelformat = V4L2_PIX_FMT_BGR24;
  capture->form.fmt.pix.field       = V4L2_FIELD_ANY;
  capture->form.fmt.pix.width = capture->width;
  capture->form.fmt.pix.height = capture->height;

I can set the pixelformat manualy with v4l2-ctl --set-fmt-video

pi@raspberrypi ~/boneCV$ v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=H264
pi@raspberrypi ~/boneCV$ v4l2-ctl --get-fmt-video
Format Video Capture:
        Width/Height  : 1920/1080
        Pixel Format  : 'H264'
        Field         : None
        Bytes per Line: 3840
        Size Image    : 4147200
        Colotspace    : SRGB

And if I now run "./boneCV" - A very simple capture program that captures a picture and does a canny edge detection. (I'll add the code in the end). I get this:

pi@raspberrypi ~/boneCV$ ./boneCV
pi@raspberrypi ~/boneCV$ v4l2-ctl --get-fmt-video
Format Video Capture:
        Width/Height  : 1920/1080
        Pixel Format  : 'MJPG'
        Field         : None
        Bytes per Line: 0
        Size Image    : 4147200
        Colorspace    : SRGB

As you can se the "Pixelformat" and the "Bytes per Line" changes. The "Field" stays at None and the "Colourspace" stays at SRGB. Then I tried to replace every "V4L2_PIX_FMT_BGR24" with "V4L2_PIX_FMT_H264" in cap_lib4vl.cpp and rebuilded OpenCV. When I then ran the "./boneCV" my two .png images are only black with one or two stripes of white color.

To find out if it is libv4l or OpenCV I ran "./capture" script that follow Derek Molloys boneCV. It uses libv4l directly and captures an H264 video stream with no problems at all. I then have to use "./raw2mpg4" to be able to watch it. The .mp4 file is 1920x1080 at 30 fps with no glitches. And after this I checked "v4l2-ctl --get-fmt-video" again and got this:

pi@raspberrypi ~/boneCV$ v4l2-ctl --get-fmt-video
Format Video Capture:
        Width/Height  : 1920/1080
        Pixel Format  : 'H264'
        Field         : None
        Bytes per Line: 3840
        Size Image    : 4147200
        Colotspace    : SRGB

Exactly the same as when I did set everything manualy.

I have come to the conclusion that if I want OpenCV to be able to capture raw H264 streams I'll have to change the cap_libv4l.cpp, but I have no idea how. I think it may be because the difference in bits per frame and/or colorspace. Do anybody know how to do this or how to make an workaround so that I stil can use OpenCVs "VideoCapture" function?

I know alot of Raspberry Pi and BeagleboneBlack users would be ever so gratefull if there was any solution to this problem. I have tried to cover everything that I think is relevant, if there is anything more I could provide to paint the picture better, please say so.

Her some links to the mentioned scripts and programs: (edit. I tried to post the links to each of the programs, but I didn't have enough reputation. Go to Derek Molloys github page and you'll find boneCV there.)

And no I can not use the "CV_FOURCC('H','2','6','4');" because this function is not implemented for linux yet.