Copyright is owned by the Author of the thesis. Permission is given for a copy to be downloaded by an individual for the purpose of research and private study only. The thesis may not be reproduced elsewhere without the permission of the Author. DDETECTING LIVE PERSON FFOR THE FACE RECOGNITION PPROBLEM By Alrashed, H. H. Sam@iSam.co.nz 07227310 Submitted in partial fulfilment of the requirements for the degree of Master of Information Sciences Massey University 2016 Detecting Live Person For The Face Recognition Problem ii Table of Contents 1. Introduction ............................................................. 4 1.1. Motivation and Objective ................................................... 6 .................................................................................................. 7 2. Literature review ..................................................... 8 2.1. Object Detection ............................................................... 9 2.1.1. Viola and Jones method ................................................. 9 2.1.2. Face Detection ............................................................. 12 2.1.3. Eyes Detection ............................................................. 12 2.1.3.1. Chrominance-based method ........................................ 13 2.1.3.2. Skin Detection-Based Method ..................................... 14 2.1.4. Mouth Detection ........................................................... 15 2.2. Face Pre-processing ......................................................... 15 2.3. Learning a Collection of Faces and Training the system ...... 21 2.3.1. Fisherfaces (also referred to as Linear Discriminant Analysis) 21 2.3.2. Hidden Markov Models .................................................. 22 2.3.3. Eigenfaces .................................................................... 23 2.4. Face Recognition ............................................................. 28 2.5. Eye Blinking Detection ..................................................... 29 2.5.1. Optical and normal flow ................................................ 29 2.5.2. Neural network ............................................................. 31 Detecting Live Person For The Face Recognition Problem iii 2.5.2.1. The back-propagation Training .................................. 35 2.6. Smile Detection ............................................................... 37 2.7. Test Data ........................................................................ 38 2.8. OpenCV Library .............................................................. 39 2.9. QT Creator ..................................................................... 40 2.10. Summary of the Literature review .................................. 40 3. Methodology .......................................................... 42 3.1. Face detection ................................................................. 42 3.2. Face processing ............................................................... 44 3.3. Face Images Acquisition ................................................... 45 3.4. Learning faces ................................................................. 47 3.5. Recognizing Face ............................................................. 48 3.6. Eye Detection .................................................................. 50 3.7. Eye Blinking Method ........................................................ 51 3.8. Smile Detection ............................................................... 54 3.9. Random Instructions ........................................................ 54 3.10. System Flowchart ......................................................... 56 4. Work done and Outcome ....................................... 60 4.1. Training ......................................................................... 60 4.2. Recognise Module ............................................................ 63 4.3. Issues during the Development .......................................... 66 5. Experimental Results ............................................ 67 5.1.1. Test face detection ........................................................ 68 Detecting Live Person For The Face Recognition Problem iv 5.1.2. Test the eye detection ................................................... 71 5.1.3. Test the mouth detection ............................................... 75 5.2. Test the face recognition on static image ........................... 81 5.3. Test the eye blinking and smile detection from recorded video 83 5.4. Integrated System Test ..................................................... 84 6. Conclusion and Future Work ................................ 86 7. Bibliography .......................................................... 88 8. Appendix ............................................................... 95 8.1. Source Code .................................................................... 95 Detecting Live Person For The Face Recognition Problem v Detecting Live Person For The Face Recognition Problem vi List of Tables Table 1 face two pixels P = pixel ............................................................. 24 Table 2 accuracy of the 3 different approaches for the smile detection [25] .......................................................................................................... 38 Table 3 Comparing Haar-cascade vs. LBP with face in the image ........... 42 Table 4 Comparing Haar-cascade vs. LBP with No face in the image ..... 43 Table 5 results on testing the closed eye detection on the left eye ........... 53 Table 6 results on testing the closed eye detection on the right eye ........ 53 Table 7 face detection Experimental Results ........................................... 68 Table 8 FERET image test results table for the face detection ............... 68 Table 9 CMU_MIT_images image test results table for the face detection .......................................................................................................... 69 Table 10 eye detection test results on FERER [27] dataset ..................... 71 Table 11 results of the image eye detection without mask. ...................... 72 Table 12 results from the eye detection with mask covering the lower part of the face. ......................................................................................... 74 Detecting Live Person For The Face Recognition Problem vii Table 13 results from the eye detection with mask covering the lower and top left part of the face. .................................................................... 75 Table 14 mouth detection test results on FERET images ........................ 76 Table 15 mouth detection applied on the whole face without mask ......... 76 Table 16 mouth detection applied on the masked face ............................. 78 Table 17 face detection experiment results ............................................... 78 Table 18 Eye detection experiment results ............................................... 79 Table 19 Mouth detection experiment results .......................................... 80 Table 20 Face recognition on static image results ................................... 82 Table 21 Test the eye blinking and smile detection from video natural video speed ........................................................................................ 83 Table 22 eye blinking and smile detection test from slow motion video .. 84 Table 23 Experimental test results on the system. 1 = true, 0 = false E: examinee ....................................................................................... 84 Detecting Live Person For The Face Recognition Problem viii Table of Figures Figure 1 system Flow-chart ....................................................................... 7 Figure 2 Type of features for Haar-like [11] ............................................. 11 Figure 3 select a small region of the face to calculate its features [11] ..... 11 Figure 4 Chrominance based method flow [13] ......................................... 14 Figure 5 Skin Detection-Based Method flow [13] ..................................... 15 Figure 6 Face in the image was detected and surrounded by a green rectangle ............................................................................................ 16 Figure 7 face image converted to gray scale and cropped ........................ 16 Figure 8 the Histogram of the image equalized ........................................ 17 Figure 9 Image histogram [11] .................................................................. 17 Figure 10 ideal Equalized histogram of the image of the above Figure 9 [11] ..................................................................................................... 18 Figure 11 histogram was not equalized [11] .............................................. 18 Figure 12 image after applying the histogram equalization [11] ............... 19 Detecting Live Person For The Face Recognition Problem ix Figure 13 Filter applied on the face ......................................................... 20 Figure 14 the right image is a result of applying the bilateral filter. Source [11] ......................................................................................... 20 Figure 15 Elliptical mask applied on the image ....................................... 20 Figure 16 convert an image from 2D to 1D .............................................. 24 Figure 17 the resulting vector from subtracting the average from every image [8] ............................................................................................ 26 Figure 18 example of flow fields showing eye open [19]. ........................... 30 Figure 19 dominating field motion is downward i.e. the eye is blinked [19]. .......................................................................................................... 30 Figure 20 eye features extracted from pre-defined sub-regions of the eye 34 Figure 21 A Multilayer Feed-forward Network ........................................ 35 Figure 22 The ORL face database [27] ..................................................... 39 Figure 23 Comparing frame rate per second Haar-cascade vs. LBP with and without face in the image ........................................................... 43 Figure 24 A face detected on the frame ................................................... 44 Figure 25 the processing steps from the detected to final image .............. 45 Figure 26 the Label of the face is added in order to train the classifier ... 46 Detecting Live Person For The Face Recognition Problem x Figure 27 accuracy of recognising between Eigenfaces and Fisherfaces with 9 images [11] .............................................................................. 47 Figure 28 face image passed to the recognition function .......................... 49 Figure 29 mask apply to the face image to extract the eye region. .......... 50 Figure 30 eye region after applying the mask as well as the elliptic mask on the face ......................................................................................... 51 Figure 31 extracting and processing one eye only .................................... 52 Figure 32 the full system flowchart .......................................................... 57 Figure 33 flow chart of the face recognition module ................................ 59 Figure 34 The application main modules ................................................. 60 Figure 35 detecting faces without training in the training window .......... 61 Figure 36 training will start after adding a name .................................... 61 Figure 37 multiple snapshots will be capture per training ....................... 62 Figure 38 showing the progress bar and the actual image to be trained. . 62 Figure 39 person has been recognised and the liveness detection is activated. ........................................................................................... 63 Figure 40 the set of instructions are Blink, Blink Smile and detected the first blink ........................................................................................... 64 Detecting Live Person For The Face Recognition Problem xi Figure 41 new set of instructions created after a failed attempt to follow the instructions ................................................................................. 65 Figure 42 adding the action log and new mouth window ......................... 66 Figure 43 CMU_MIT_images sample image ........................................... 70 Figure 44 CMU_MIT_images sample image ........................................... 71 Figure 45 masks cover the lower part of the face ..................................... 73 Figure 46 masks cover the lower part of the face ..................................... 73 Figure 47 mask covering all face except the eye region ............................ 74 Figure 48 left image is the original face and right one is the image with mask covering all face except the eye region ..................................... 75 Figure 49 mask applied to the face 70% the upper part covered and 30% the lower part .................................................................................... 77 Figure 50 Face recognition on static image results on column chart ........ 82 H ALRASHED Detecting Live Person For The Face Recognition Problem 1 H ALRASHED Detecting Live Person For The Face Recognition Problem 2 Abstract Face recognition has been a challenging problem for computer vision scientists for the last few decades. Hence it was the center of attention for computer vision researchers. The purpose of this research is to improve the security of the face recognition system by identifying the liveness of a person in front of a camera to be recognised. The objective was to detect if the images used to be recognised reflect a real person’s face, i.e., a live person’s face instead of just a static image of the face. This can be achieved by randomly asking the person to carry out certain tasks. Simple tasks such as blinking an eye or smiling can then be repeated randomly according to the instructions given by the new system, so even a video of the target face made previously would not be able to perform the authentication easily. Each component of the system were tested separately. The accuracy of the face detection component was impressively at 98.93%. The eye blinking detection uses a new proposed method with a high accuracy of 91%. Face recognition component was also tested and had a high recognition rate of 96%. Keywords: Face Recognition, Face Detection, Eigenfaces, OpenCV, Face Anti-Spoofing, Eye Detection, Smile Detection, Eye Blinking Detection H ALRASHED Detecting Live Person For The Face Recognition Problem 3 AAcknowledgements I would like to express my sincere gratitude and regards to my supervisors Doctor Andre Barczak and Doctor Napoleon Reyes for their guidance and support during the course of my thesis. I must express my very profound gratitude to my beloved wife for providing me with unfailing support and continuous encouragement throughout my years of study and through the process of researching and writing this thesis. This accomplishment would not have been possible without her. Thank you. Finally, I would also like to acknowledge Intel for providing an open source library that is heavily used in this project that is OpenCV library.