Encoding features robust to unseen modes of variation with attentive long short-term memory

This paper is a result of a joint academic-industrial project between KAIST IVY Lab and Genesis Lab.

Original Paper

Encoding features robust to unseen modes of variation with attentive long short-term memory

In performing facial expression recognition tasks using video, the mode of variation, such as different appearances, poses, and light diversity, reduces the classification capability of the extracted features. This paper presents a method to improve classification performance by minimizing the effects of the mode of variation.

Introduction

When performing human facial expression recognition tasks using deep learning, various types of information that can interfere with this task coexist in the data. For example, a person's appearance, body movements, facial angles, and even the direction of light, act as factors that hinder accurate expression recognition, despite being information unrelated to recognizing expressions [2,3]. These cognitive distractors are called the mode of variation [1]. To create a model that is unaffected by distractors, a method of training the model with as many diverse distractors as possible is sometimes used, allowing the model to recognize the distractors on its own. However, if one wishes to perform a desired task using continuous data where the length of the input information is not fixed, such as video, an almost infinite amount of training data is required. Ultimately, we must derive a good recognition rate by making the most of a model trained with limited data, but there will inevitably be unseen modes of variation, which are distracting information not entered during training. This paper proposes the following method to minimize the impact of this unseen mode of variation.

Proposed Method

Figure 1. Overview of the proposed method for encoding features robust to unseen modes of variation

This paper separates continuous input data, such as video, into two types: task-relevant dynamic sequence features and task-irrelevant static sequence features. Task-relevant dynamic sequence features are the parts where there are changes between the previous frame and the current frame when the video is divided into frames. These change elements are used to extract dynamic features. Task-irrelevant static sequence features are the parts of the video that do not change. Task-irrelevant static sequence features are used to extract the mode of variation, which is an element that interferes with the feature to be recognized. Through the feature information extracted separately in this way, we can obtain spat-temporal features robust to unseen mode variations. Using this feature information, we can construct a robust model that is not affected by distracting information that could not be used during training. The details are as follows.

Figure 2. Proposed attentive mode variational LSTM

Input Signal Separator

The Input Signal Separator in Fig 2 was designed by taking inspiration from the concept of the element-wise attention gate proposed in [4]. The input signal separator serves to separate task-relevant dynamic sequence features, $x_{t}^{att}$, and task-irrelevant static sequence features, $\hat x_{t}^{att}$. To obtain $x_{t}^{att}$ and $\hat x_{t}^{att}$, $a_{t}$ (the element-wise dynamics attention) is required, which can be obtained through the element-wise dynamics attention gate. The method of obtaining $a_{t}$ using the gate is as follows.

$a_{t} = \alpha(W_{xa}x_{t}+W_{ha}H_{t-1}+b_{a})$

$x_{t}^{att}$ can be obtained by multiplying $a_{t}$ by the input data according to time t. The $x_{t}^{att}$ obtained in this way is used to emphasize the characteristics of active regions (dynamic features) of the input data [4].

$x_{t}^{att}=a_{t} \odot x_{t}$

However, while paper [4] completely ignores static regions, this paper uses these static regions to obtain the features of the mode of variation that interferes with cognitive identification. For task-irrelevant static sequence features, the remaining regions of the entire input data, excluding $a_{t}$, are used.

$\hat x_{t}^{att}=(1-a_{t}) \odot x_{t}$

Encoding dynamic features and mode of variation

The dynamic features extracted by passing $x_{t}^{att}$ to the input gate and the forget gate respectively are stored in the memory cell $c_{t}$. $\hat x_{t}^{att}$ is also passed to the input gate and forget gate in the same way to obtain the mode of variation, and that information is stored in another memory cell $\hat c_{t}$. The feature information stored in the two memory cells passes through a single shared output gate, and by repeating this several times, spatio-temporal feature information that is unaffected by unseen mode variation can be obtained. There are two reasons for sharing a single output gate to obtain this feature information: one is to synchronize the dynamics feature and the mode of variation, and the other is to minimize the impact on the extracted dynamics features by carrying along the characteristic values of the current and previous modes of variation.

Experiments

To verify the effectiveness of the proposed model, this paper performed two tasks: facial expression recognition and human action recognition. This experiment uses the Oulu-CASIA dataset, which features 6 basic emotions (anger, disgust, fear, happiness, sadness, surprise) filmed under 3 different lighting environments targeting 80 subjects; the AFEW dataset, which collected data from real-world video clips; and the KAIST Face Multi-pose Multi-illumination data, which simultaneously filmed 104 subjects showing 7 emotions (the 6 basic emotions plus a neutral emotion) using 13 webcams.

Experiment 1 was conducted to compare performance with previously proposed models. Tables 1, 2, and 3 show the results for each dataset, respectively. It can be confirmed that the currently proposed model exceeds the state-of-the-art (highest recognition rate).

Experiment 2 tests robustness against unseen modes of variation. This experiment divided the Oulu-CASIA dataset into Asian and Caucasian groups to check the degradation of the model's recognition rate based on appearance or looks. Table 4 shows the results when a model trained on Asians recognized the expressions of Caucasians, and when a model trained on Caucasians recognized the expressions of Asians. This indicates that the model proposed in this paper extracts features that are more robust to changes in the subject's appearance.

To confirm the recognition rate under changes in lighting, the next experiment was also conducted using the KAIST Face MPMI dataset. Using training data with lighting under room conditions, the model was made to recognize human expressions when the room was bright, when light shone from the left, and when light shone from the right, respectively. The proposed model shows an excellent recognition rate regardless of the direction and intensity of the light.

Conclusion

In this paper, we propose a method to design a model to extract features that minimize the impact on the unseen mode of variation. The core of this proposal, the attentive mode variational LSTM, uses the concept of attention to separate input information into task-relevant dynamic sequence features and task-irrelevant static sequence features. Each of the above features is used to extract dynamics feature information and the mode of variation, and the extracted information is converted into spatio-temporal features robust to unseen mode variations through a single shared output gate. Experiments performed with two tasks, facial expression recognition and human action recognition, prove the effectiveness of this feature and have updated the state of the art (SOTA) in each domain.

References

[1] Wang et al., “Disentangling the Modes of Variation in Unlabelled Data,” IEEE TPAMI 2017.

[2] Shahroudy et al., “NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis,” CVPR 2016.

[3] Zhang et al., “View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data,” ICCV 2017.

[4] Zhang et al., “Adding Attentiveness to the Neurons in Recurrent Neural Networks,” ECCV 2018.

[16] Baddar et al., “Mode Variational LSTM Robust to Unseen Modes of Variation: Application to Facial Expression Recognition,” AAAI 2019.