Convert an h264 byte string to OpenCV images

Question

In Python, how do I convert an h264 byte string to images OpenCV can read, only keeping the latest image?

Long version:

Hi everyone.

Working in Python, I'm trying to get the output from adb screenrecord piped in a way that allows me to capture a frame whenever I need it and use it with OpenCV. As I understand, I need to constantly read the stream because it's h264.

I've tried multiple things to get it working and concluded that I needed to ask for specific help.

The following gets me the stream I need and works very well when I print stream.stdout.read(n).

import subprocess as sp

adbCmd = ['adb', 'exec-out', 'screenrecord', '--output-format=h264', '-']
stream = sp.Popen(adbCmd, stdout = sp.PIPE, universal_newlines = True)

Universal newlines was necessary to get it to work on Windows.

Doing:

sp.call(['ffplay', '-'], stdin = stream.stdout, universal_newlines = True)

Works.

The problem is I am now trying to use ffmpeg to take the input h264 stream and output as many frames as possible, overwriting the last frame if needed.

ffmpegCmd = ['ffmpeg', '-f', 'image2pipe', '-pix_fmt', 'bgr24', '-vcodec', 'h264', 'fps=30', '-']
ffmpeg = sp.Popen(ffmpegCmd, stdin = stream.stdout, stdout = sp.PIPE, universal_newlines = True)

This is what I think should be used, but I always get the error "Output file #0 does not contain any stream".

Edit:

Final Answer

Turns out the universal_newlines option was ruining the line endings and gradually corrupting the output. Also, the ffmpeg command was wrong, see LordNeckbeard's answer.

Here's the correct ffmpeg command to achieve what was used:

ffmpegCmd = ['ffmpeg', '-i', '-', '-f', 'rawvideo', '-vcodec', 'bmp', '-vf', 'fps=5', '-']
ffmpeg = sp.Popen(ffmpegCmd, stdin = stream.stdout, stdout = sp.PIPE)

And then to convert the result into an OpenCV image, you do the following:

fileSizeBytes = ffmpeg.stdout.read(6)
fileSize = 0
for i in xrange(4):
    fileSize += fileSizeBytes[i + 2] * 256 ** i
bmpData = fileSizeBytes + ffmpeg.stdout.read(fileSize - 6)
image = cv2.imdecode(np.fromstring(bmpData, dtype = np.uint8), 1)

This will get every single frame of a stream as an OpenCV image.

Answer 1

Use any of these:

ffmpeg -i - -pix_fmt bgr24 -f rawvideo -
ffmpeg -i pipe: -pix_fmt bgr24 -f rawvideo pipe:
ffmpeg -i pipe:0 -pix_fmt bgr24 -f rawvideo pipe:1

• You didn't provide much info about your input so you may need to add additional input options.

• You didn't specify your desired output format so I just chose rawvideo. You can see a list of supported output formats (muxers) with ffmpeg -muxers (or ffmpeg -formats if your ffmpeg is outdated). Not all are suitable for piping, such as MP4.

• See FFmpeg Protocols: pipe.

Answer 2

It works well just a minor changes: This will read the stream in a loop and show each time the last image

adbCmd = ['adb', 'exec-out', 'screenrecord', '--output-format=h264', '-']
stream = sp.Popen(adbCmd, stdout = sp.PIPE)

ffmpegCmd =['ffmpeg', '-i', '-', '-f', 'rawvideo', '-vf', 'scale=324:576', 
'-vcodec', 'bmp',  '-']
ffmpeg = sp.Popen(ffmpegCmd, stdin = stream.stdout, stdout = sp.PIPE)

while True:
    fileSizeBytes = ffmpeg.stdout.read(6)
    fileSize = 0
    for i in xrange(4):
        fileSize += array.array('B',fileSizeBytes[i + 2])[0] * 256 ** i
    bmpData = fileSizeBytes + ffmpeg.stdout.read(fileSize - 6)
    image = cv2.imdecode(np.fromstring(bmpData, dtype = np.uint8), 1)
    cv2.imshow("im",image) 
    cv2.waitKey(25)