Replacing detected object in a frame with an image

Question

Overview:

I am working on a video creation project. The technology I am using are: imageMagick, php, ffmpeg.

Current Status:

Currently the project is able to create videos using images and texts and few basic transitions. The way I am doing it is using imagemagick to create gif using input images(with transition effects in them) and then converting all gifs to videos and atlast concatenating the video together.

Next Move (My question):

I am now set to take it to the next level. So, what I am having is a video(1920x1080) with some white frames(1280x720) that keeps shifting in each frame. I want to replace those white frames appearing in some frames of the video with some images(1280x720) that I wish to use. Please see the image here and you will get an idea: These are just two frames from my video. If you can see carefully the images are shifting(white space is not constant).

Expectation:

So, I want to fill those white space with one of my own image. If the case would have been for only one frame I could have used ffmpeg to overlay image on the exact width and height. But here the white space is not fixed and keeps shifting in all the frames and there are a lot of frames. So, I am looking for something like opencv or some other technology that can be used for object detection in a video or in a set of frames and replace the detected area with some other image.

I just need a kick. So, if anyone has already worked on something like this just suggest me what technology can I use. Thanks in advance.

Answer 1

It all depends on exactly what you can assume :

If you can safely assume that your rectangle's boundary is never occluded (hidden) somehow, you can try finding the edges in your image (like OpenCV's Canny edge) and then look for rectangular shape (corners forming a warped rectangle, or the very popular Hough Lines).

If the rectangle you're looking for is always white, you can threshold the image in a colorspace like HSV to look for maximum value (the V in HSV ~ brightness) then rectangular shape search in a binary image.

If your corners are occluded sometimes you'll have to do some tweaking with your image, like morphological operations ("grow and contract" binary thresholded image), then Hough Lines could do the trick.

Note that this answer assumes that once you know where the rectangle is, "you're done", and you just have to overwrite the rectangle with custom content. I also do not check for any time-continuity : you video frame might jump around based on the frame-by-frame appearance of rectangle. You'd have to include some knowledge about previous positions.