How to align face images c++ opencv

Question

I am developing a C++ application for face authentication. First, I have to detect the face and pre-process the image.

1. For face detection I have used the HaarCascadeClassifier. The problem is that the this tool or this algorithm gives me a facial region detected by a little bit large rectangle that englobes hair and some of the background. Is there a solution to change the dimension of this rectangle? I used "frontalfacecascaadclassifier.xml".
2. For face pre-processing i want to do face alignment exactly like this technique. How would I go about accomplishing this?

Answer 1

Finding the accurate position of the eyes in a given image is far from trivial. The Haar-cascades for finding the eyes in OpenCV produce too many false positive to be useful, moreover this approach won't be robust to image rotation (it may compensate slight rotation though, I don't know the training images). If I were you I'd start a breadth-first search on http://scholar.google.com for relevant papers of this research area.

You'll need a robust head pose estimation for aligning face images. I did some research myself and I think sharing algorithms and code is useful here. The most interesting approaches I have seen are:

• Gary B. Huang, Vidit Jain, and Erik Learned-Miller. Unsupervised joint alignment of complex images. International Conference on Computer Vision (ICCV), 2007. (Project page), (PDF Online available), (Source code)

• X. Zhu, D. Ramanan. Face Detection, Pose Estimation and Landmark Localization in the Wild Computer Vision and Pattern Recognition (CVPR) Providence, Rhode Island, June 2012. (Project page), (PDF Online available), (Source code)

Answer 2

Can't you then use another Haar classifier to find each eye (eyes are very easy to find) then assuming the person has two eyes and we define a 'level' face to mean the eyes are horizontal.

Simply measure the anlge between the two eyes and rotate the image by that angle.

angle = atan ( eye1.Y - eye2.Y ) / (eye1.X - eye2.X )

Answer 3

Have a look at CSIRO face analysis SDK (website and demos, source code) software, it does face alignment, tracking with 66 fiducial points. It is fast and very accurate.

Answer 4

I tried the following face alignment code from the Labelled Faces in the Wild project page. It works really well and does not require detecting facial feature points. The C++ code can be downloaded from: http://vis-www.cs.umass.edu/faceAlignment/

If you still wish to find face key points, I find that the Viola-Jones detector is not very robust and accurate. I personally recommend using the Flandmark face keypoint detector: http://cmp.felk.cvut.cz/~uricamic/flandmark/ which is much much more robust and accurate. C code can be downloaded from the above site.

Answer 5

After searching all day for an algorithm to accomplish this with, I found "Face Detection By Finding The Facial Features And The Angle of Inclination of Tilted Face " by Hemlata et al. after switching from Google to DuckDuckGo. It supports faces that are tilted at an angle larger than 45 degrees.

As for how to implement in code, that's another problem that I'm currently working on, but at least this is a starting point.

Answer 6

Detecting misaligned faces make face recognition difficult. Sometimes you want to fix the alignment, sometimes it is sufficient to exclude the ones that aren't aligned correctly (for instance if you're detecting faces in a video stream). I took the latter approach and trained a special Haar Cascade to only detect correctly aligned, well-lit faces. Details here: http://rwoodley.org/?p=417.

If you use my cascade let me know how it works for you. I'm curious what results others would obtain. It met my needs.