Lightweight and Effective Facial Landmark Detection using Adversarial Learning with Face Geometric Map Generative Network

This paper was published jointly by KAIST IVY Lab. and Genesis Lab as part of the IITP (Institute for Information & Communications Technology Planning & Evaluation) 2018 ICT R&D Voucher Project.

Original Paper

Lightweight and Effective Facial Landmark Detection using Adversarial Learning with Face Geometric Map Generative Network

Facial Landmarks are feature points that represent key components of the face (eyes, eyebrows, nose, mouth, and jawline). These features are generally represented by 68 points on the face and are used to locate faces within an image. Therefore, Facial Landmark Detection is very important for precise face detection. Facial landmarks detected through algorithms can be used not only for face detection but also in various fields of computer vision, such as head pose estimation and emotion recognition.

Facial Landmark

Facial landmark detection is the task of finding the locations of important facial components, providing essential information for computer vision tasks. Representative methodologies include optimization-based methods, which predict the direction of movement to align initially detected facial elements with actual facial locations, and regression-based methods, which directly predict facial element locations in an image using learned parameters. Of course, recently, deep learning-based methods have shown good performance, similar to other tasks. As deep learning-based research has progressed actively, various methodologies have been studied, such as multi-task learning methods that aim to solve tasks like face detection and head pose estimation at once. Since these research projects based on facial landmark detection need to be applied to applications such as mobile or web, models must be simple yet capable of accurate performance. However, when using a simple CNN structure, performance is very poor for images with misaligned facial contours. To address this, research using two sub-networks to predict internal facial elements and contours has alleviated this issue, but it is difficult to say it has been fully resolved.

Facial landmark detection results on the 300W dataset with 68 facial landmark points. These examples include several challenging conditions such as various head poses, expressions, and occlusions.

Introduction

Facial landmark detection is a task to locate key components of the face, providing essential information for computer vision tasks. Representative methodologies include optimization-based methods, which predict the direction of movement to align initially detected facial elements with actual facial locations, and regression-based methods, which directly predict facial element locations in an image using learned parameters. Of course, recently, deep learning-based methods have shown good performance, similar to other tasks. As deep learning-based research has progressed actively, various methodologies have been studied, such as multi-task learning methods that aim to solve tasks like face detection and head pose estimation at once. Since these research projects based on facial landmark detection need to be applied to applications such as mobile or web, models must be simple yet capable of accurate performance. However, when using a simple CNN structure, performance is very poor for images with misaligned facial contours. To address this, research using two sub-networks to predict internal facial elements and contours has alleviated this issue, but it is difficult to say it has been fully resolved.

In this paper, we propose a Geometric Prior-Generative Adversarial Network based on GAN, which adversarially trains a generative model and a discriminative model. Unlike existing methods that learn the discrepancy between real Facial Landmarks and predicted Facial Landmarks using L1 or L2 loss, the proposed model utilizes an adversarial loss that considers geometric characteristics of the face. In this paper, a Generator is trained to receive the output of an Encoder trained to predict facial landmark coordinates from face images and predict a geometric map of the facial contours and a geometric map of the facial interior. In addition, the Discriminator was designed to learn to distinguish between real facial landmarks and the facial landmarks created by the generative model.

Facial Geometric Generative Adversarial Network

Model Overview

Figure 2. Overall architecture of the proposed facial geometric GAN. It shows (a) facial landmark estimator, (b) facial inner geometric map generator, (c) facial contour geometric map generator, (d) facial inner geometric map discriminator, and (e) facial contour geometric map discriminator. In the generator, two types of adversarial geometric maps (inner/contour) are generated. The generator and discriminator are trained in an adversarial minimax game. The estimator predicts the inner/contour facial landmarks, while the discriminator determines whether the geometric map is real/fake and predicts each landmark. As shown in the figure, the facial geometric map consists of a binary map.

Training Face Geometric Map Generator

The generator consists of one encoder that predicts facial inner and contour landmarks from an input image, and two decoders that generate geometric maps from the encoder's output. Geometric facial features are important for accurately predicting landmarks in images containing various noises, such as cropped or angled faces.

Conventional L1/L2 methods do not sufficiently capture these features because they only consider the difference between ground truth and predicted values. On the other hand, in this paper, the generator's encoder predicts the inner/contour landmarks respectively, and the generator's decoders utilize them to generate geometric maps.

When training the generator, not only the adversarial loss but also the prediction loss of the discriminator is considered, while the parameters of the discriminator are frozen.

Figure 3. (a) Dice coefficient used for facial geometric evaluation. (b) Matching and Dice coefficient evaluation between ground truth facial geometric map and generated geometric map.

Training the Discriminator

Each discriminative model learns to determine whether the input geometric map is real or generated, and to predict the coordinates of the landmarks. Among the loss functions of the discriminant function, there are also loss functions computed during the training of the generative model. Since the parameters of the discriminant function are not updated when training the generative model, the generative model learns to generate geometric maps that look more realistic to minimize the loss. Similarly, when training the discriminative model, the parameters of the generative model are not updated.

Experiment Result

The datasets used in the experiments are HELEN and 300-W. HELEN comes with two annotation information types: one with 194 landmarks and the other with 68 landmarks. 300-W consists of four subsets (AFW, LFPW, HELEN, IBUG) with 3,148 training data (AFW: 377 + HELEN: 2,000 + LFPW: 811) and 689 testing data (LFPW: 224 + HELEN: 300 + IBUG: 135). In addition, translation, rotation, and magnification were used for data augmentation.

Experiments for Performance Comparison

As shown in the table above, the performance is better than existing studies, and we can confirm that it performs better than TCDCN pre-trained with MAFL Database and RCFA which uses RNN for face alignment.

The table above shows that the model proposed in this paper achieves the best performance even on 300-W, which includes images where landmarks are difficult to predict accurately, indicating that it is more robust than other models.

Experiment Results for Usefulness of Contour Map

In this paper, we measured the effectiveness of utilizing the geometric map of facial contours through experiments.

The graph above shows experiments using three models; each model has the same structure but a different training method. CNN8 is a model optimized using L1, and ContourGeometric8 is a model trained excluding the Facial Inner map. Looking at the experimental results above, the method using facial contours performs better than the method using general L1 loss, and we can see that performance increases even further when utilizing internal facial information as well.

Conclusion

Through this study, we confirmed that utilizing facial geometric information for adversarial training in FLD tasks shows superior performance compared to previous research. In addition, we found that accurate facial contours help locate inner facial landmarks better. Since only the encoder is required to extract landmarks during the test phase, it can be concluded that the goal of creating a simple and effective FLD network that can be easily applied to various applications has been achieved.

References

[2] Asthana et al., “Robust Discriminative Response Map Fitting with Constrained Local Models,” CVPR 2013.

[5] Zhang et al., “Learning Deep Representation for Face Alignment with Auxiliary Attributes,” IEEE TPAMI 2016.

[6] Lv et al., “A Deep Regression Architecture with Two-Stage Re-initialization for High Performance Facial Landmark Detection,” CVPR 2017.

[7] Cao et al., “Face Alignment by Explicit Shape Regression,” IJCV 2014.

[13] Zhang et al., “Coarse-to-Fine Auto-Encoder Networks (CFAN) for Real-Time Face Alignment,” ECCV 2014.

[14] Wang et al., “Multiscale Recurrent Regression Networks for Face Alignment,” Applied Informatics 2017.

[24] Zhu et al., “Face Alignment by Coarse-to-Fine Shape Searching,” CVPR 2015.

[25] Xiong et al., “Supervised Descent Method and Its Applications to Face Alignment,” CVPR 2013.

[26] Burgos-Artizzu et al., “Robust Face Landmark Estimation under Occlusion,” ICCV 2013.

[27] Tzimiropoulos et al., “Gauss-Newton Deformable Part Models for Face Alignment In-the-Wild,” CVPR 2014.

[28] Wang et al., “Recurrent Convolutional Face Alignment,” ACCV 2016.

[29] Ren et al., “Face Alignment at 3000 FPS via Regressing Local Binary Features,” CVPR 2014.

[30] Xu et al., “Joint Head Pose Estimation and Face Alignment Framework Using Global and Local CNN Features,” IEEE FG 2017.

[31] Trigeorgis et al., “Mnemonic Descent Method: A Recurrent Process Applied for End-to-End Face Alignment,” CVPR 2016.

[32] Hou et al., “Face Alignment Recurrent Network,” Pattern Recognition 2018.

[33] Zhang et al., “Combining Data-Driven and Model-Driven Methods for Robust Facial Landmark Detection,” IEEE TIFS 2018.

[34] Lai et al., “Deep Recurrent Regression for Facial Landmark Detection,” IEEE TCSVT 2016.

[35] Kowalski et al., “Deep Alignment Network: A Convolutional Neural Network for Robust Face Alignment,” CVPRW 2017.

[36] Liu et al., “Learning Deep Sharable and Structural Detectors for Face Alignment,” IEEE TIP 2017.