Raspberry Pi 3 Python and OpenCV Face Recognition

2020-03-30 09:15发布

I'm attempting to stream video over a WiFi network using a raspberry pi model 3 and the camera module but have come across some inconsistencies from other examples I have found online. I'm attempting to achieve the results of the 3rd test performed in this video: https://www.youtube.com/watch?v=sYGdge3T30o

I can see the data being streamed over the network and into the fifo264 file that is created, I also have a functioning python script that operates for a few minutes before the video feed can no longer be processed.

The video that is displayed on the OS X machine eventually crashes. I'm not sure if my problem exists within how I'm generating and sending the video (From the Raspberry Pi side), or with how I'm consuming the video with python and processing it.

I'm attempting to determine platform differences that can be contributing to my problems, this question focuses mkfifo and netcat permissions and possible differences between how these programs behave across platforms.

This bash script is being used on OS X 10.12.6. I'm attempting to listen on port 777 for the feed that is being created and sent to this machine.

#!/bin/bash

if [ -p fifo264 ]
then
    rm fifo264
fi
mkfifo fifo264 
nc -l -v 777 > fifo264

When the above script is executed ./makeFifo.sh the pipe fifo264 is created in the working directory and it inherits the user permissions of the user.

sudo ./makeFifo.sh results in ls -l as

prw-r--r-- 1 root staff 0 Feb 4 12:12 fifo264

AND

./makeFifo.sh results in ls -l as

prw-r--r-- 1 user staff 0 Feb 4 12:12 fifo264

I've also altered the bash script to supply a -m 0777 with mkfifo result mkfifo -m 0777 fifo264. The pipe is created with the appropriate permissions, but netcat fails to establish an open 777/tcp open multiling-http. I'm running nmap -sS -O localhost to test. Results are the same if I use the machines ip address.

If I do not use the bash script to create a pipe and execute netcat to listen on port 777 from the command line, I can get netcat to create an open 777 port on OS X 10.12.6 by entering sudo nc -l -v 777 > fifo264.

This results in creating an open 777/tcp open multiling-http but does not create a pipe. ls -l outputs:

-rw-r--r-- 1 user staff 0 Feb 4 12:26 fifo264 note the missing p from the permission string.

If I create the pipe with mkfifo fifo264 from the command line, then proceed to execute sudo nc -l -v 777 > fifo264, the pipe is created but netcat will not listen on port 777.

Would sending data to a non pipe file create any latency issues? Would this have an impact on the following python script that I'm using to display the feed? I'm using openCV 3.4, with Python 3.6.4 on OS X (the machine I'm trying to process the video feed on). videoFeed.py is below:

import cv2
import subprocess as sp
import numpy

FFMPEG_BIN = "ffmpeg"
command = [ FFMPEG_BIN,
        '-i', 'fifo264',             # fifo is the named pipe
        '-pix_fmt', 'bgr24',      # opencv requires bgr24 pixel format.
        '-vcodec', 'rawvideo',
        '-an','-sn',              # we want to disable audio processing (there is no audio)
        '-f', 'image2pipe', '-']    
pipe = sp.Popen(command, stdout = sp.PIPE, bufsize=10**8)

while True:
    # Capture frame-by-frame
    raw_image = pipe.stdout.read(640*480*3)
    # transform the byte read into a numpy array
    image =  numpy.fromstring(raw_image, dtype='uint8')
    image = image.reshape((480,640,3))          # Notice how height is specified first and then width
    if image is not None:
        cv2.imshow('Video', image)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break
    pipe.stdout.flush()

cv2.destroyAllWindows()

From the RaspberryPi Model 3 B running RASPBIAN STRETCH WITH DESKTOP (Nov. 2017) I'm using the following bash script to transmit the video streamVideo.sh.

#!/bin/bash

raspivid -vf -n -w 640 -h 480 -o - -t 0 -b 1200000 | nc -v 10.0.0.3 777

To put this altogether for what I have been able to get to work thus far is as follows:

On the OS X machine, I issue sudo nc -l -v 777 > fifo264 from the terminal which results in the following nmap scan:

Host is up (0.00015s latency).
Not shown: 993 closed ports
PORT     STATE SERVICE
80/tcp   open  http
445/tcp  open  microsoft-ds
548/tcp  open  afp
777/tcp  open  multiling-http

Then from the RaspberryPi I start streamVideo.sh: Connection to 10.0.0.3 777 port [tcp/moira-update] succeeded!

On the OS X machine I start my python script to access the fifo264 feed.

python3 videoFeed.py

If the camera is focused on a still object without any movement in the frame, it will stream to the fifo264 feed. If I start videoFeed.py after a few seconds have elapsed, the python script will display the video feed from the beginning with a significant delay, although the quality is ok.

If I start videoFeed.py immediately after executing the streamVideo.sh script from the RaspberryPI, the latency is non-existent, but the picture quality is horrible. This screenshot was taken of me moving my hand in front of the camera.

enter image description here

Below you will find the entire output of the videoFeed.py python script that is attempting to consume the video feed.

ffmpeg version N-78264-g9079e99-tessus Copyright (c) 2000-2016 the FFmpeg developers
  built with Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
  configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --as=yasm --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libass --enable-libbluray --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzmq --enable-version3 --disable-ffplay --disable-indev=qtkit --disable-indev=x11grab_xcb
  libavutil      55. 15.100 / 55. 15.100
  libavcodec     57. 22.102 / 57. 22.102
  libavformat    57. 23.100 / 57. 23.100
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6. 27.100 /  6. 27.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
Input #0, h264, from 'fifo264':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuv420p, 640x480, 25 fps, 25 tbr, 1200k tbn, 50 tbc
Output #0, image2pipe, to 'pipe:':
  Metadata:
    encoder         : Lavf57.23.100
    Stream #0:0: Video: rawvideo (BGR[24] / 0x18524742), bgr24, 640x480, q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc57.22.102 rawvideo
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> rawvideo (native))
Press [q] to stop, [?] for help
[h264 @ 0x7f813c831800] error while decoding MB 6 6, bytestream -5
[h264 @ 0x7f813c831800] concealing 1003 DC, 1003 AC, 1003 MV errors in P frame
[h264 @ 0x7f813c873200] error while decoding MB 30 20, bytestream -2884320.0kbits/s speed=1.43x    
[h264 @ 0x7f813c873200] concealing 419 DC, 419 AC, 419 MV errors in P frame
[NULL @ 0x7f813c808000] missing picture in access unit with size 919
[h264 @ 0x7f813c866000] no frame!
[h264 @ 0x7f813c824600] error while decoding MB 7 15, bytestream -7
[h264 @ 0x7f813c824600] concealing 642 DC, 642 AC, 642 MV errors in P frame
Error while decoding stream #0:0: Invalid data found when processing input.0kbits/s dup=2 drop=0 speed=1.42x    
[h264 @ 0x7f813c858e00] error while decoding MB 39 17, bytestream -6
[h264 @ 0x7f813c858e00] concealing 530 DC, 530 AC, 530 MV errors in P frame
[h264 @ 0x7f813c821600] error while decoding MB 17 28, bytestream -6
[h264 @ 0x7f813c821600] concealing 112 DC, 112 AC, 112 MV errors in P frame
[h264 @ 0x7f813c880400] error while decoding MB 31 21, bytestream -5
[h264 @ 0x7f813c880400] concealing 378 DC, 378 AC, 378 MV errors in P frame
[h264 @ 0x7f813c83ea00] error while decoding MB 38 18, bytestream -5
[h264 @ 0x7f813c83ea00] concealing 491 DC, 491 AC, 491 MV errors in P frame
[h264 @ 0x7f813c880400] error while decoding MB 20 24, bytestream -5184320.0kbits/s dup=7 drop=0 speed=1.43x    
[h264 @ 0x7f813c880400] concealing 269 DC, 269 AC, 269 MV errors in P frame
[h264 @ 0x7f813c84bc00] error while decoding MB 29 27, bytestream -6
[h264 @ 0x7f813c84bc00] concealing 140 DC, 140 AC, 140 MV errors in P frame
[h264 @ 0x7f813c83ea00] error while decoding MB 36 0, bytestream -10
[h264 @ 0x7f813c83ea00] concealing 1200 DC, 1200 AC, 1200 MV errors in P frame
[h264 @ 0x7f813c873200] error while decoding MB 15 7, bytestream -9=184320.0kbits/s dup=9 drop=0 speed=1.42x    
[h264 @ 0x7f813c873200] concealing 954 DC, 954 AC, 954 MV errors in P frame
[h264 @ 0x7f813c824600] error while decoding MB 39 0, bytestream -25
[h264 @ 0x7f813c824600] concealing 1200 DC, 1200 AC, 1200 MV errors in P frame
[h264 @ 0x7f813c84bc00] error while decoding MB 34 24, bytestream -6
[h264 @ 0x7f813c84bc00] concealing 255 DC, 255 AC, 255 MV errors in P frame
[h264 @ 0x7f813c821600] error while decoding MB 32 21, bytestream -6184320.0kbits/s dup=15 drop=0 speed=1.44x    
[h264 @ 0x7f813c821600] concealing 377 DC, 377 AC, 377 MV errors in P frame
[h264 @ 0x7f813c858e00] error while decoding MB 28 22, bytestream -6
[h264 @ 0x7f813c858e00] concealing 341 DC, 341 AC, 341 MV errors in P frame
[h264 @ 0x7f813c880400] error while decoding MB 4 29, bytestream -8
[h264 @ 0x7f813c880400] concealing 85 DC, 85 AC, 85 MV errors in P frame
[h264 @ 0x7f813c873200] error while decoding MB 7 18, bytestream -6=184320.0kbits/s dup=17 drop=0 speed=1.44x    
[h264 @ 0x7f813c873200] concealing 522 DC, 522 AC, 522 MV errors in P frame
[h264 @ 0x7f813c866000] error while decoding MB 31 24, bytestream -10
[h264 @ 0x7f813c866000] concealing 258 DC, 258 AC, 258 MV errors in P frame
[h264 @ 0x7f813c84bc00] error while decoding MB 23 28, bytestream -30
[h264 @ 0x7f813c84bc00] concealing 106 DC, 106 AC, 106 MV errors in P frame
[h264 @ 0x7f813c880400] error while decoding MB 15 29, bytestream -9184320.0kbits/s dup=21 drop=0 speed=1.44x    
[h264 @ 0x7f813c880400] concealing 74 DC, 74 AC, 74 MV errors in P frame
[h264 @ 0x7f813c84bc00] error while decoding MB 26 26, bytestream -5
[h264 @ 0x7f813c84bc00] concealing 183 DC, 183 AC, 183 MV errors in P frame
[h264 @ 0x7f813c880400] error while decoding MB 37 14, bytestream -5
[h264 @ 0x7f813c880400] concealing 652 DC, 652 AC, 652 MV errors in P frame
[h264 @ 0x7f813c831800] error while decoding MB 36 24, bytestream -6
[h264 @ 0x7f813c831800] concealing 253 DC, 253 AC, 253 MV errors in P frame
[h264 @ 0x7f813c880400] error while decoding MB 25 26, bytestream -5184320.0kbits/s dup=23 drop=0 speed=1.44x    
[h264 @ 0x7f813c880400] concealing 184 DC, 184 AC, 184 MV errors in P frame
[h264 @ 0x7f813c824600] error while decoding MB 37 2, bytestream -6
[h264 @ 0x7f813c824600] concealing 1132 DC, 1132 AC, 1132 MV errors in P frame
[h264 @ 0x7f813c84bc00] error while decoding MB 33 23, bytestream -7
[h264 @ 0x7f813c84bc00] concealing 296 DC, 296 AC, 296 MV errors in P frame
[h264 @ 0x7f813c821600] error while decoding MB 32 16, bytestream -5184320.0kbits/s dup=26 drop=0 speed=1.44x    
[h264 @ 0x7f813c821600] concealing 577 DC, 577 AC, 577 MV errors in P frame
[h264 @ 0x7f813c880400] error while decoding MB 1 26, bytestream -6
[h264 @ 0x7f813c880400] concealing 208 DC, 208 AC, 208 MV errors in P frame
[h264 @ 0x7f813c84bc00] error while decoding MB 32 12, bytestream -6184320.0kbits/s dup=28 drop=0 speed=1.43x    
[h264 @ 0x7f813c84bc00] concealing 737 DC, 737 AC, 737 MV errors in P frame
[h264 @ 0x7f813c880400] error while decoding MB 27 26, bytestream -8
[h264 @ 0x7f813c880400] concealing 182 DC, 182 AC, 182 MV errors in P frame
[h264 @ 0x7f813c84bc00] error while decoding MB 27 27, bytestream -5
[h264 @ 0x7f813c84bc00] concealing 142 DC, 142 AC, 142 MV errors in P frame
[h264 @ 0x7f813c831800] error while decoding MB 23 20, bytestream -7
[h264 @ 0x7f813c831800] concealing 426 DC, 426 AC, 426 MV errors in P frame
[h264 @ 0x7f813c858e00] error while decoding MB 29 26, bytestream -5184320.0kbits/s dup=31 drop=0 speed=1.43x    
[h264 @ 0x7f813c858e00] concealing 180 DC, 180 AC, 180 MV errors in P frame
[h264 @ 0x7f813c821600] error while decoding MB 26 26, bytestream -5
[h264 @ 0x7f813c821600] concealing 183 DC, 183 AC, 183 MV errors in P frame
[h264 @ 0x7f813c858e00] error while decoding MB 27 27, bytestream -9
[h264 @ 0x7f813c858e00] concealing 142 DC, 142 AC, 142 MV errors in P frame
[h264 @ 0x7f813c824600] error while decoding MB 30 4, bytestream -8
[h264 @ 0x7f813c824600] concealing 1059 DC, 1059 AC, 1059 MV errors in P frame
[h264 @ 0x7f813c858e00] error while decoding MB 9 22, bytestream -5=184320.0kbits/s dup=33 drop=0 speed=1.42x    
[h264 @ 0x7f813c858e00] concealing 360 DC, 360 AC, 360 MV errors in P frame
[h264 @ 0x7f813c873200] error while decoding MB 25 13, bytestream -12
[h264 @ 0x7f813c873200] concealing 704 DC, 704 AC, 704 MV errors in P frame
[h264 @ 0x7f813c880400] error while decoding MB 14 1, bytestream -9
[h264 @ 0x7f813c880400] concealing 1195 DC, 1195 AC, 1195 MV errors in P frame
[h264 @ 0x7f813c821600] error while decoding MB 29 23, bytestream -9
[h264 @ 0x7f813c821600] concealing 300 DC, 300 AC, 300 MV errors in P frame
[h264 @ 0x7f813c824600] error while decoding MB 30 25, bytestream -10
[h264 @ 0x7f813c824600] concealing 219 DC, 219 AC, 219 MV errors in P frame
[h264 @ 0x7f813c831800] error while decoding MB 27 27, bytestream -8
[h264 @ 0x7f813c831800] concealing 142 DC, 142 AC, 142 MV errors in P frame
[h264 @ 0x7f813c83ea00] error while decoding MB 13 23, bytestream -6184320.0kbits/s dup=34 drop=0 speed=1.42x    
[h264 @ 0x7f813c83ea00] concealing 316 DC, 316 AC, 316 MV errors in P frame
[h264 @ 0x7f813c84bc00] error while decoding MB 37 12, bytestream -5
[h264 @ 0x7f813c84bc00] concealing 732 DC, 732 AC, 732 MV errors in P frame
[h264 @ 0x7f813c866000] error while decoding MB 27 2, bytestream -6
[h264 @ 0x7f813c866000] concealing 1142 DC, 1142 AC, 1142 MV errors in P frame
[h264 @ 0x7f813c873200] error while decoding MB 22 25, bytestream -10
[h264 @ 0x7f813c873200] concealing 227 DC, 227 AC, 227 MV errors in P frame
[h264 @ 0x7f813c880400] error while decoding MB 5 15, bytestream -6
[h264 @ 0x7f813c880400] concealing 644 DC, 644 AC, 644 MV errors in P frame
[h264 @ 0x7f813c84bc00] error while decoding MB 27 13, bytestream -8
[h264 @ 0x7f813c84bc00] concealing 702 DC, 702 AC, 702 MV errors in P frame
[h264 @ 0x7f813c858e00] error while decoding MB 33 15, bytestream -12
[h264 @ 0x7f813c858e00] concealing 616 DC, 616 AC, 616 MV errors in P frame
[h264 @ 0x7f813c866000] error while decoding MB 39 16, bytestream -5
[h264 @ 0x7f813c866000] concealing 570 DC, 570 AC, 570 MV errors in P frame
[h264 @ 0x7f813c831800] error while decoding MB 27 29, bytestream -8
[h264 @ 0x7f813c831800] concealing 62 DC, 62 AC, 62 MV errors in P frame
[h264 @ 0x7f813c873200] error while decoding MB 35 29, bytestream -5184320.0kbits/s dup=38 drop=0 speed=1.42x    
[h264 @ 0x7f813c873200] concealing 54 DC, 54 AC, 54 MV errors in P frame
[h264 @ 0x7f813c821600] error while decoding MB 32 14, bytestream -14
[h264 @ 0x7f813c821600] concealing 657 DC, 657 AC, 657 MV errors in P frame
[h264 @ 0x7f813c824600] error while decoding MB 29 22, bytestream -7
[h264 @ 0x7f813c824600] concealing 340 DC, 340 AC, 340 MV errors in P frame
[h264 @ 0x7f813c831800] error while decoding MB 8 13, bytestream -7
[h264 @ 0x7f813c831800] concealing 721 DC, 721 AC, 721 MV errors in P frame
[NULL @ 0x7f813c808000] missing picture in access unit with size 1448
[h264 @ 0x7f813c83ea00] no frame!
[h264 @ 0x7f813c84bc00] error while decoding MB 9 16, bytestream -19
[h264 @ 0x7f813c84bc00] concealing 600 DC, 600 AC, 600 MV errors in P frame
[h264 @ 0x7f813c858e00] error while decoding MB 24 18, bytestream -16
[h264 @ 0x7f813c858e00] concealing 505 DC, 505 AC, 505 MV errors in P frame
Error while decoding stream #0:0: Invalid data found when processing input
[h264 @ 0x7f813c866000] error while decoding MB 22 13, bytestream -5
[h264 @ 0x7f813c866000] concealing 707 DC, 707 AC, 707 MV errors in P frame
[h264 @ 0x7f813c880400] error while decoding MB 11 29, bytestream -6
[h264 @ 0x7f813c880400] concealing 78 DC, 78 AC, 78 MV errors in P frame
[h264 @ 0x7f813c821600] error while decoding MB 38 14, bytestream -6
[h264 @ 0x7f813c821600] concealing 651 DC, 651 AC, 651 MV errors in P frame
[h264 @ 0x7f813c824600] error while decoding MB 6 17, bytestream -6
[h264 @ 0x7f813c824600] concealing 563 DC, 563 AC, 563 MV errors in P frame
[h264 @ 0x7f813c83ea00] error while decoding MB 19 27, bytestream -5
[h264 @ 0x7f813c83ea00] concealing 150 DC, 150 AC, 150 MV errors in P frame
[h264 @ 0x7f813c84bc00] error while decoding MB 18 17, bytestream -9
[h264 @ 0x7f813c84bc00] concealing 551 DC, 551 AC, 551 MV errors in P frame
[h264 @ 0x7f813c858e00] error while decoding MB 38 9, bytestream -16
[h264 @ 0x7f813c858e00] concealing 851 DC, 851 AC, 851 MV errors in P frame
[h264 @ 0x7f813c880400] error while decoding MB 8 19, bytestream -7=184320.0kbits/s dup=41 drop=0 speed=1.42x    
[h264 @ 0x7f813c880400] concealing 481 DC, 481 AC, 481 MV errors in P frame
[h264 @ 0x7f813c821600] error while decoding MB 24 21, bytestream -14
[h264 @ 0x7f813c821600] concealing 385 DC, 385 AC, 385 MV errors in P frame
[h264 @ 0x7f813c873200] error while decoding MB 26 23, bytestream -5
[h264 @ 0x7f813c873200] concealing 303 DC, 303 AC, 303 MV errors in P frame
[h264 @ 0x7f813c83ea00] error while decoding MB 31 26, bytestream -25
[h264 @ 0x7f813c83ea00] concealing 178 DC, 178 AC, 178 MV errors in P frame
[h264 @ 0x7f813c866000] error while decoding MB 25 28, bytestream -6
[h264 @ 0x7f813c866000] concealing 104 DC, 104 AC, 104 MV errors in P frame
[h264 @ 0x7f813c821600] error while decoding MB 23 29, bytestream -1284320.0kbits/s dup=44 drop=0 speed=1.42x    
[h264 @ 0x7f813c821600] concealing 66 DC, 66 AC, 66 MV errors in P frame
[h264 @ 0x7f813c831800] error while decoding MB 23 20, bytestream -19
[h264 @ 0x7f813c831800] concealing 426 DC, 426 AC, 426 MV errors in P frame
[h264 @ 0x7f813c84bc00] error while decoding MB 27 26, bytestream -10
[h264 @ 0x7f813c84bc00] concealing 182 DC, 182 AC, 182 MV errors in P frame
[h264 @ 0x7f813c858e00] error while decoding MB 31 26, bytestream -10
[h264 @ 0x7f813c858e00] concealing 178 DC, 178 AC, 178 MV errors in P frame
[h264 @ 0x7f813c866000] error while decoding MB 8 25, bytestream -8
[h264 @ 0x7f813c866000] concealing 241 DC, 241 AC, 241 MV errors in P frame
[h264 @ 0x7f813c873200] error while decoding MB 39 27, bytestream -8
[h264 @ 0x7f813c873200] concealing 130 DC, 130 AC, 130 MV errors in P frame
[h264 @ 0x7f813c83ea00] error while decoding MB 26 13, bytestream -10
[h264 @ 0x7f813c83ea00] concealing 703 DC, 703 AC, 703 MV errors in P frame
[h264 @ 0x7f813c866000] error while decoding MB 5 23, bytestream -6
[h264 @ 0x7f813c866000] concealing 324 DC, 324 AC, 324 MV errors in P frame
[h264 @ 0x7f813c880400] error while decoding MB 39 4, bytestream -9=184320.0kbits/s dup=47 drop=0 speed=1.41x    
[h264 @ 0x7f813c880400] concealing 1050 DC, 1050 AC, 1050 MV errors in P frame
[h264 @ 0x7f813c821600] error while decoding MB 27 27, bytestream -5
[h264 @ 0x7f813c821600] concealing 142 DC, 142 AC, 142 MV errors in P frame
[h264 @ 0x7f813c84bc00] error while decoding MB 18 3, bytestream -7
[h264 @ 0x7f813c84bc00] concealing 1111 DC, 1111 AC, 1111 MV errors in P frame
[h264 @ 0x7f813c873200] error while decoding MB 39 19, bytestream -8
[h264 @ 0x7f813c873200] concealing 450 DC, 450 AC, 450 MV errors in P frame
[h264 @ 0x7f813c866000] error while decoding MB 28 23, bytestream -6
[h264 @ 0x7f813c866000] concealing 301 DC, 301 AC, 301 MV errors in P frame
[h264 @ 0x7f813c824600] error while decoding MB 35 7, bytestream -5=184320.0kbits/s dup=51 drop=0 speed=1.41x    
[h264 @ 0x7f813c824600] concealing 934 DC, 934 AC, 934 MV errors in P frame
[h264 @ 0x7f813c83ea00] error while decoding MB 19 29, bytestream -7
[h264 @ 0x7f813c83ea00] concealing 70 DC, 70 AC, 70 MV errors in P frame
[h264 @ 0x7f813c873200] error while decoding MB 37 16, bytestream -6
[h264 @ 0x7f813c873200] concealing 572 DC, 572 AC, 572 MV errors in P frame
[h264 @ 0x7f813c831800] error while decoding MB 16 12, bytestream -5
[h264 @ 0x7f813c831800] concealing 753 DC, 753 AC, 753 MV errors in P frame
[h264 @ 0x7f813c866000] error while decoding MB 29 13, bytestream -6
[h264 @ 0x7f813c866000] concealing 700 DC, 700 AC, 700 MV errors in P frame
[h264 @ 0x7f813c821600] error while decoding MB 31 8, bytestream -6=184320.0kbits/s dup=55 drop=0 speed=1.42x    
[h264 @ 0x7f813c821600] concealing 898 DC, 898 AC, 898 MV errors in P frame
[h264 @ 0x7f813c84bc00] error while decoding MB 11 22, bytestream -6
[h264 @ 0x7f813c84bc00] concealing 358 DC, 358 AC, 358 MV errors in P frame
frame=  400 fps= 35 q=-0.0 Lsize=  360000kB time=00:00:16.00 bitrate=184320.0kbits/s dup=56 drop=0 speed=1.41x    
video:360000kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
Traceback (most recent call last):
  File "videoFeed.py", line 19, in <module>
    image = image.reshape((480,640,3))          # Notice how height is specified first and then width
ValueError: cannot reshape array of size 0 into shape (480,640,3)

I've read somewhere that problems can arise if you do not start reading from a pipe as soon as it starts receiving data, but if anyone can point out potential issues that would be helpful.

Update

I can obtain an quality camera feed using netcat while piping the output to mplayer:

sudo nc -l 777 | mplayer -fps 200 -demuxer h264es -

So I can only assume this has more to do with how I'm processing the feed than how the pipe is being constructed.

More Tests

Using the following script I can read video from the fifo264 pipe, but it is also terminating.

import cv2
import sys

video_capture = cv2.VideoCapture(r'fifo264')
video_capture.set(cv2.CAP_PROP_FRAME_WIDTH, 640);
video_capture.set(cv2.CAP_PROP_FRAME_HEIGHT, 480);

while True:
    # Capture frame-by-frame
    ret, frame = video_capture.read()
    if ret == False:
        pass

    cv2.imshow('Video', frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# When everything is done, release the capture
video_capture.release()
cv2.destroyAllWindows()

Streaming from the Raspberry PI with:

#!/bin/bash

raspivid -ih -vs -vf -n -w 640 -h 480 -o - -t 0 -b 2000000 | nc -v 10.0.0.3 777

Error and Output Produced:

[h264 @ 0x7fb25005d400] error while decoding MB 22 28, bytestream -8
[h264 @ 0x7fb250068c00] error while decoding MB 8 24, bytestream -9
[h264 @ 0x7fb25005d400] error while decoding MB 2 18, bytestream -5
[h264 @ 0x7fb250060400] error while decoding MB 23 11, bytestream -7
[h264 @ 0x7fb250073000] error while decoding MB 22 18, bytestream -11
[h264 @ 0x7fb250073600] error while decoding MB 36 9, bytestream -5
[h264 @ 0x7fb250067a00] error while decoding MB 29 23, bytestream -8
[NULL @ 0x7fb251155a00] missing picture in access unit with size 1448
[h264 @ 0x7fb251155400] Got unexpected packet size after a partial decode
[h264 @ 0x7fb251155400] Got unexpected packet size after a partial decode
[h264 @ 0x7fb251155400] Got unexpected packet size after a partial decode
[NULL @ 0x7fb251155a00] missing picture in access unit with size 1448
[h264 @ 0x7fb251155400] Got unexpected packet size after a partial decode
[h264 @ 0x7fb25005d400] error while decoding MB 35 2, bytestream -6
[NULL @ 0x7fb251155a00] missing picture in access unit with size 1200
[h264 @ 0x7fb25005d400] No start code is found.
[h264 @ 0x7fb25005d400] Error splitting the input into NAL units.
OpenCV(3.4.0-dev) Error: Assertion failed (size.width>0 && size.height>0) in imshow, file /Users/jamie/opencv/modules/highgui/src/window.cpp, line 356
Traceback (most recent call last):
  File "video3.py", line 14, in <module>
    cv2.imshow('Video', frame)
cv2.error: OpenCV(3.4.0-dev) /Users/jamie/opencv/modules/highgui/src/window.cpp:356: error: (-215) size.width>0 && size.height>0 in function imshow

1条回答
我只想做你的唯一
2楼-- · 2020-03-30 09:52

Ultimately this was not a result of the pipe, the platform or permissions. The video that was being generated on the Raspberry Pi and piped to the python script was not being handled properly.

I ended up adapting this picamera python recipe

On the Raspberry Pi: (createStream.py)

import io
import socket
import struct
import time
import picamera

# Connect a client socket to my_server:8000 (change my_server to the
# hostname of your server)
client_socket = socket.socket()
client_socket.connect(('10.0.0.3', 777))

# Make a file-like object out of the connection
connection = client_socket.makefile('wb')
try:
    with picamera.PiCamera() as camera:
        camera.resolution = (1024, 768)
        # Start a preview and let the camera warm up for 2 seconds
        camera.start_preview()
        time.sleep(2)

        # Note the start time and construct a stream to hold image data
        # temporarily (we could write it directly to connection but in this
        # case we want to find out the size of each capture first to keep
        # our protocol simple)
        start = time.time()
        stream = io.BytesIO()
        for foo in camera.capture_continuous(stream, 'jpeg', use_video_port=True):
            # Write the length of the capture to the stream and flush to
            # ensure it actually gets sent
            connection.write(struct.pack('<L', stream.tell()))
            connection.flush()

            # Rewind the stream and send the image data over the wire
            stream.seek(0)
            connection.write(stream.read())

            # Reset the stream for the next capture
            stream.seek(0)
            stream.truncate()
    # Write a length of zero to the stream to signal we're done
    connection.write(struct.pack('<L', 0))
finally:
    connection.close()
    client_socket.close()

On the machine that is processing the stream: (processStream.py)

import io
import socket
import struct
import cv2
import numpy as np

# Start a socket listening for connections on 0.0.0.0:8000 (0.0.0.0 means
# all interfaces)
server_socket = socket.socket()
server_socket.bind(('0.0.0.0', 777))
server_socket.listen(0)

# Accept a single connection and make a file-like object out of it
connection = server_socket.accept()[0].makefile('rb')
try:
    while True:
        # Read the length of the image as a 32-bit unsigned int. If the
        # length is zero, quit the loop
        image_len = struct.unpack('<L', connection.read(struct.calcsize('<L')))[0]
        if not image_len:
            break
        # Construct a stream to hold the image data and read the image
        # data from the connection
        image_stream = io.BytesIO()
        image_stream.write(connection.read(image_len))
        # Rewind the stream, open it as an image with opencv and do some
        # processing on it
        image_stream.seek(0)
        image = Image.open(image_stream)

        data = np.fromstring(image_stream.getvalue(), dtype=np.uint8)
        imagedisp = cv2.imdecode(data, 1)

        cv2.imshow("Frame",imagedisp)
        cv2.waitKey(1)  #imshow will not output an image if you do not use waitKey
        cv2.destroyAllWindows() #cleanup windows 
finally:
    connection.close()
    server_socket.close()

This solution has similar results to the video I referenced in my original question. Larger resolution frames increase latency of the feed, but this is tolerable for the purposes of my application.

First you need to run processStream.py, and then execute createStream.py on the Raspberry Pi

查看更多
登录 后发表回答