Optical flow based segmentation

Question

Hi I would like to segment objects in a video sequence based on optical flow (as suggested in this answer), I am using opencv and have been able to generate a flow field for two successive frames using the calcOpticalFlowFarneback function. Could someone explain how I would go about segmenting an image based on this flow field?

Answer 1

One very straightforward way to go about it will be to provide the algorithm with a key frame which has the ground truths. What it means is that, segment the first frame manually (you could use a grabcut here) and then propagate the segmentation to subsequent frames using optical flow.

The caveat is that optical flow is notoriously famous for being error prone. Hence instead of using the propagated segmentation directly, you can erode (using some structuring element) your propagated segmentation and use the eroded image as a foreground seed for grabcut. Similarly, dilate the propagated segementation and use a band around the dilated boundary as a background seed.

In short: For the first frame give the ground truth, use that to automatically perform grabcut on subsequent frames.

Does this help?

Answer 2

Since you want to use Optical Flow to segment object so your decision making will be based on the type of motion your objects under go...You can have a look at this video on youtube. You can measure the direction and the magnitude of the displacement each pixel of interest undergoes in two successive frames to get an idea of their movement pattern. In the link to the video the car is segmented from the background on the basis of Optical Flow...