In previous literature, there are two types of data identified; static facial image and moving face. Initially, studies begin with static facial image detection and recognize a few regions of interest on the facial. In the study conducted by [45], participants have to restrict their head movement. In the real world, it is an impractical approach to instruct the subject to be static. This limitation emphasizes the importance to track the facial in motion. Several studies focused on automating facial detection in thermal imaging where the participant can move freely. The majority of studies conducted self-experiment to collect their dataset. A laboratory experiment was conducted in [62,68,69,74] to collect thermal images and other physiological signals.

Face detection is the first step in stress detection based on the facial thermal image. Thermal images are commonly used in many circumstances where ordinary perception is limited, hindered, or inadequate. For example, during night surveillance and fugitive searches. The facial detection algorithm in the thermal image was inefficient in the beginning and was also not sophisticated for visual RGB images. Many studies reported this as a limitation that affects the experiment methodology and findings. Several stress detection methods prohibit head movement during the data collection phase. This is because thermal imaging has limitations for many head movements. As in [62], the authors willingly crop regions of interest (ROIs) manually due to this limitation. The common practice in face detection in the scope of stress detection based on the thermal image is knowledge-based techniques feature invariant facial approaches, template matching method, appearance-based method, colour information, and fusion with visible spectrum imaging. In previous literature, the methodology involved in facial detection in thermal imaging is classified as appearance-based [75,76], feature extraction [77–79], fusion with RGB image [80–85] and multimodal analysis.

Zheng [86] proposed the Projection Project Analysis (PPA) algorithm for face detection algorithm. Studies related to facial thermal images adapted this algorithm to detect face regions. Zheng [75] continued the experiment that adapted PPA to detect faces that have eyeglasses. One of the main limitations of thermal imaging is facial occlusion, which may occur from glass opacity in individuals who are eye-glass wearers. Therefore, occlusion will prevent thermal sensors from reading the heat pattern produced in that region. Several models have been introduced to handle such limitations; different facial ROI is being explored to tackle this challenge. Basu et al. [87] proposed a thermal-based occluded images model by applied Kotani Thermal Facial Expression Database (KTFE) as input images and the Viola-Jones algorithm for facial detection. The study applied a median filter to remove noise and Contrast Limited Adaptive Histogram Equalization (CLAHE) for image enhancement. The study selected six facial patches: forehead, eyes, right and left cheeks, nose, and mouth. Hu's [88] seven-moment invariant methods were chosen for feature extraction and the classification process. The study selected multi-scale SVM to classify four basic emotions and the average accuracy was 87.5%.

A number of studies [45,89,90] applied a state-of-the-art face detection algorithm, the Haar-based Viola-Jones face detection algorithm [91]. Reese et al. [90] conclusively proven that learning-based methods; Viola-Jones and Gabor have a high tendency to detect a face, and the Viola-Jones algorithm can be used as mainstream for face detection in thermal images. Basbrain et al. [92] suggested strategies for improving the Viola-Jones algorithm's face detection performance in thermal images. The findings suggest that the Viola-Jones method with LBP performs significantly better in thermal images vs. Haar-like features. The study also found that using the Otsu technique in the pre-processing stage improves the detection rate. In a recent study, Tran et al. [93] proposed to integrate cross-examples into a proposed scheme, which effectively improved the face detection accuracy. Kowalski et al. [94] evaluated three common face detection algorithms, the Viola-Jones, YOLO, and CNN. The deep learning network outperforms the Viola-Jones algorithm. Faster R-CNN has better performance with a near-perfect detection rate and a low false detection rate. Researchers presented the bioheat model, a special identification method, in [95], by creating a thinned vascular network, similar to work done in [96]. Cho et al. [97] integrated Modified Hausdorff Distance features to improve the precision performance implemented by [95].

Kopazka and his colleagues [98] have proven that a state-of-the-art machine learning algorithm designated for visual images performs better than dedicated algorithms in detecting facial in thermal imaging in terms of accuracy and false-positive rate. Five algorithms were selected; Haar Cascade Viola-Jones, Haar Cascade classifier with local binary patterns (LBPs), Histograms of Oriental Gradients (HOG), the Deformable Parts Model (DPM), and Pixel Intensity Comparisons Organized in Decision Trees (PICO) to compare with two algorithms; Eye Corner Detection (ED) [99], Projection Profile Analysis (PPA) [100] dedicated for facial detection in thermal imaging. These comparative studies exposed the algorithms have a sensitivity towards a change in pose and facial expression as it impacts the accuracy rate. VJ-LBP and HOG produce good similar detection rates. DPM performs best in the detection and false-positive rates at the cost of having the longest computation time. The authors recommend the PICO method performs fast and produce better results. Kopaczka et al. [100,101] proposed one of the earliest works on thermal facial landmark detection based on active appearance models [102]. The authors applied PCA to landmark data and trained the AAM model based on dense HOG and SIFT features. Chu et al. [103] tested a theory by applying an image transfer model by transferring a thermal image to visible and used one of the landmark detectors designated for visual images. The result indicated that the dedicated facial landmark detector need for thermal imaging. This leads the authors to propose a thermal facial landmark detection based on deep multi-task learning. Kopaczka et al. [101] suggested the face tracker method by using thermal videos and images based on AAM. The study focused on strong landmarks to detect and track ROI within head pose and rotation. Furthermore, the study proposed several enhancement algorithms such as sharp masking, USM with bilateral filtering and USM with a Gaussian kernel. The study used several descriptors with AAM like scale-invariant feature transform (SIFT) and Histogram Oriented Gradient (HOG) for fitting the algorithm. The study also used Project-out inverse compositional, alternating inverse compositional (IC) and Simultaneous Inverse compositional.

Sonkusare et al. [104] proposed a novel deep-learning assisted facial landmark to detect method for the thermal image. This is to extract thermal signals from the facial regions. The authors applied the sudden auditory stimulus of a loud stimulus to invoke the physiological responses. The authors aimed to characterize the spatial changes in temperatures of different facial regions i.e., nose-tip, right and left cheeks and forehead). The GSR and HR is selected as bench marker. In this work, the authors compared two methods; (1) a task-constrained deep convolutional network (TCDCN) and (2) an OpenPose detector. For method (1), the authors implemented TCDCN which trained on RGB images and then fine-tune by further learning on thermal images. Method (2) OpenPose detector is presented by [105] that employs a robust multi-view bootstrapping architecture. Then this two methods are combined to improve the landmark localization accuracy.

Several studies [90] demonstrated that face detection in the thermal image is possible without being aided by a visual image. Researchers explored the artificial intelligence method. Mohd et al. [59] suggested a BoCNN architecture framework to overcome one’ limitation, detect occluded facial in a thermal image. A variety of CNN models proven to perform well in thermal imaging [106–110]. Hong [45] proposed a multi-subject correlation [100] method to detect ROIs; forehead, nose, and mouth in a thermal image.

There are many stresses stimulus widely used in literature; Stroop test, Trier Social Stress Test (TSST) [55,62,74], arithmetical questionnaire [61,68,69,71], mock crime scenario setup [70] and physical activities [55]. In [45], the participants were required to run on a treadmill to induce physical stress as this study aimed to differentiate physical stress and baseline status. This study achieves an accuracy rate of 90%.

Thermal images, which have a distinct texture, appearance, and form than RGB images, play a crucial role to identify human stress. Various forms of thermal descriptors have been used in previous research. The effectiveness of facial features from the entire face or facial parts (ROI) is required in stress state identification. Numerous types of feature extraction are shown in Tab. 3 of this survey. Based on Tab. 3, the majority of studies have used statistical features. For example, He et al. [69] used maximum temperature, and mean. Derakhshan et al. [70] extracted the six temporal features including mean, minimum, maximum, standard deviation, means of the absolute values of the first and seconds’ derivatives of the pre-processed signals. Vasavi et al. [65] calculated heart rate based on the mean value of the frame over time. The authors [64] extracted the mean of the top 10% thermally hot pixels, minimum, maximum, and standard deviation. The authors [71] calculated the mean value of emissivity and performed normalization by using min-max normalization. In [61] the authors used the mean of the top 10% of the pixels. Cross et al. [55] extracted average pixel value, maximum pixel value and the mean of the top 10% of the pixels. Berlovskaya et al. [111] also extract the average value of the pixel and standard deviations.

The researchers conducted several studies to establish the correlation between thermal features with stress state of a human [74]. suggested that the individual stress state recognized by the facial temperature is measurable with thermal imaging. Kan Hong and his colleagues [62] found a Pearson correlation value less than 1 between facial thermal signatures and known stress indicator such as heart rate (HR) and cortisol level. In this study, the authors proposed new physiological signal extraction; the Eulerian magnification algorithm, to amplify the physiological signals. This study also shows the significant correlation between perinasal ROI and ground truth. A 96% accuracy is achieved by the proposed algorithm compared to ground truth features.

The authors [64] applied the thermal imaging to classify the challenged participants among the threatened participants. The findings show that features extracted from forehead and nose regions yield better accuracy of stress detection. The stress classification based on individual features such as forehead and the nose yields 80% accuracy. When the data from the forehead and nose are combined, the accuracy improves even further. This research reveals insight temperature variations in the facial region that can be used to identify different human emotional states.

The authors [71] studied the impact of psychophysical stimuli on facial thermal emissions. In this study, the authors attempt to distinguish the facial pattern produced by physical activity and mental stress. The findings show that thermal variation caused by psychological stimuli has more changes in pixel intensity compared to the caused by physical stimuli. Reference [68] investigated the impact of student cognitive load on the facial thermographic during knowledge assessment test. The output of the study provides strong evidence to support the correlations between student cognitive load and thermal changes. The authors agreed that thermal signature changes on facial as criteria for stress detection.

Vasavi et al. [66] attempted to identify the most engaging feature for better stress detection by applying regressing modelling. The findings support that the periorbital region is the most engaging thermal feature for stress detection. Authors [112] investigated the correlation between stress and topography of facial temperature changes over time. With Bonferroni-corrected pairwise comparisons, the correlation between induced stress state and the temperature changes in forehead, cheeks and perioral is acknowledged. He et al. [69] justify the usefulness of using facial temperature to evaluate mental stress. The authors also attempted to evaluate the effectiveness of employing face temperature as a mental stress biomarker in comparison to other established biomarkers; HRV, TLI, and PDM. The results established that the face temperature can provide an accountable indicator for human stress recognition in a non-contact approach. Derakhshan et al. [70] confirmed that perinasal and chin areas mostly correlated to stress state in studies. Hong et al. [54] found that number of hot pixels increases in the periorbital region when participants have induced physical stress. When participant experiences emotional stress, prefrontal region has its pixel increases temperatures. This exhibits characteristics of the thermal distribution.

Researchers investigated the stress classification based on the extracted thermal signature of the face. Table 2 provides the summary.

In [60], the authors experimented to detect human acute stress by integrating physiological features and thermal features. The authors evaluate the performance of the chosen feature individually and in combination. The performance of thermal features as individual features achieves the highest accuracy of 73% as similar to the fusion of four physiological features, state-of-the-art biomarkers of stress detection. These findings provide evidence to support that the thermal features hold high accountability to measure stress remotely. The authors have suggested a combination of the thermal and respiration rate features can improve the accuracy of stress detection. This combination contributes 26% to improve accuracy. The decision tree classifier is employed for stress classification and validated by the leave-one-subject-out-validation.

Cross et al. [55] demonstrated that the system yields the highest accuracy in classifying mental stress vs. physical stress. The features used are frequency analysis of the respiratory and cardiovascular pulse. The authors extracted statistical descriptors, such as pixel value, maximum value, and the mean of the 10% hottest pixel. The authors also compared four classifiers; AAN, Naiye Bayes classifier, linear discriminant analysis, and SVM. The accuracy rate of classifications is 96.4% (AAN), 100% (LDA), 92.9% (SVM), and 82.1% (NB). Each classifier is validated with four-fold cross-validation. Among the four classifiers, LDA was found to perform better that provides the high accuracy classification in a short time of computation. The authors claimed that isolated face regions have the potential to improve classification accuracy.

Baltaci et al. [61] proposed a method to separate the stress state of a computer user. The authors investigated the classification accuracy for each feature and fusion. The results show that individually thermal features perform better with an accuracy of 76%, while individual pupil features achieved 73% accuracy. The combined features of thermal and pupil achieved 83% accuracy. Two classifiers were compared which are Adaboost with Random Forest outperform Decision trees. 10-fold cross-validation is applied. The stress simulation used in this study revealed limitation emotional that influences the result. The mean of the hottest pixel of the periorbital region is used as descriptors in this study.

Derakhshan et al. [70] conducted an experiment to discriminate deception and truth by comparing four machine learning techniques which SVM, KNN, LDA, and decision tree (DT). This experiment aimed to improve the accuracy of thermal imaging and also to identify the ROIs that can show significant results. From the perspective of physiology, deceptive anxiety leads to spontaneous physiological signs including perspiration, increased heart rate, blood flow changes and so on [70]. According to Cannon, this physiological reaction to acute stress is called the “fight or flight” response [113]. The raw measurement obtained from thermal data is maximum and minimum values. The six temporal features are extracted from these raw measurements: mean, minimum, maximum, standard deviation, and means of absolute values of the first and second derivatives of the signals. In this study, the authors carried two activities to trigger deceptive, mock crime, and best friend scenarios. For mock crime scenarios, the classification accuracies of thermal data, GSR, and PPG are 83.8%, 67.7% and 64.5%, respectively. Thermal data performs better. While in best friend scenarios, accuracies for thermal data, GSR, and PPG are 62.9%, 66.6%, and 79.6%. Physiological signals show its discrimination property is stronger than thermal data. After the feature reduction technique is applied, the accuracy of the thermal data jumps from 41% to 90%. The DT performs better than other models. LDA classifier achieves 905 accuracies after feature reduction. The authors also compared the four thermal reduction methods: t-test, relative entropy, ROC, and MWW. The result shows that ROC and MWW produce high accuracies and the t-test shows improvement in other classifiers. The authors used the leave one out validation method to get classifier accuracy. This study established that thermal features outperform gold standard measurement and the accuracy can be improved with the feature reduction method. The authors also found that perinasal and chin contributed to high accuracy classification with help of the feature reduction method.

The authors [65] presented a framework to measure thermal signatures to detect cardiovascular and stress. In this study, the authors extracted thermal signatures such as card pulse, stress responses (heart rate and heart rate variability), breath rate, and sudomotor responses. They categorized the stress state based on rules. For stress responses, the carried two methods, the first method uses FFT and the second method applied wavelet and FFT to calculate heart rate. The first method achieves 91% accuracy while the second method achieves 90.3%. The proposed method performs better than the similar method proposed in [114].

Hong [45] proposed a contact-free model to detect the physical stress of the human body by maximizing thermal signature in the facial region. After extract the ROIs by using the multi-object correlation method, the stress signal was extracted, converted into an independent component by the blind source separation (ICA) method and then amplified by Euclidean Magnification (EM) algorithm. They applied the deep learning algorithm model to classify the baseline and physical stress and achieve 90% accuracy. Before [45], the authors also conducted studies to detect stress by analyzing facial temperature. They achieve 96% accuracy between the proposed EM algorithm and ground truth that consists of the established stress markers. The authors magnified the stress signal after preprocessed with the FFT algorithm.

In the recent years, very few studies explored the deep learning technique to produce better high accuracy stress classification. Reshma [72] presented a hybrid deep learning network for stress detection in the thermal image. Z-normalization based on the mean and standard deviation is applied for better training. In this work, the authors combine two deep learning neural networks. Raw image provided as input to first network DNN-1, and generate frequency features based on wavelet transform technique. This frequency feature is given as input to the second network, DNN-2. This hybrid system performance is compared with the machine learning technique. The comparison shows the proposed system produces high accuracies, 96.2%. This system is also compared with two transfer learning networks. The hybrid system outperforms these transfer learning techniques, Alexnet and Vgg-16. Alexnet accuracy is 92%, Vgg-166 accuracy is 94.5 and proposed system accuracy is 96.2%.

Kumar et al. [73] presented a novel deep learning-based methodology that explored the new feature ISTI to detect stress in the thermal videos. This study also proposed emission representation modules that can be used to model variations in emitted radiation due to the motion of blood and head movements. The authors explored the neural network based on facial skin temperature and established evidence to introduce a new feature that has similar performance to the state-of-the-art features. Biomarkers of Stress State (BOSS) and Cold Pressor test used to invoke stress. Mean Squared Error (MSE) and Pearson's correlation coefficient (R) were used to evaluate ISTI prediction. The findings show that ISTI extracted by the proposed model have a high prediction rate. Average precision (AP) as a validation metric is applied for stress detection. Predicted ISTI signal is better in detecting stress state than HR (12% higher AP) and HRV (4% higher AP). Also, higher AP with the ground truth ISTI signal confirms that ISTI is the most performing index of stress state in the experiment. Panasiuk et al. [71] compared the numeral analysis and deep learning method for stress state detection. Both techniques depend on heat emissions as a feature. The proposed deep learning methods achieved a high classification accuracy of 88.21%. The numerical analysis produces an accuracy of 76.40% and 78.10% for psychological and physical tests.

Bara et al. [115] presented a preliminary approached based on deep learning towards multi modal stress detection. The authors evaluated a different set of deep learning method. The multi-modal used in this works are thermal video; RGB Closeup Video; RGB Wideangle Video; Audio; the QA and monologues; Physiological signals: (1) heart rate, (2) body temperature, (3) skin conductance, (4) breathing rate; and text: transcripts extracted from the QA. In this work, the proposed architecture is based on Convolutional-Autoencoders and Recurrent Neural Networks. The Gated Recurrent Unit (GRU) is used for implementation. The subject-based leave-one-out cross-validation is used for validation. Results demonstrated that the deep-learning methods can generate rich state representations related to stress, regardless of relatively limited amount of data.

Gupta [116] proposed a stress detection method based on a deep learning technique. The authors employed the deep learning model that consists of LSTM layer and a fully connected layer. The output from fully connected layer is channeled into a softmax function for stress prediction of a person. The authors used the 5-fold cross-validation to train the model. The results show the average accuracy of classification by this model is 87%. Proposed model performs better than similar accuracy produced in [117].