Journal of Clinical & Experimental Dermatology Research
Open Access

The Accuracy of Facial Landmark Models in Different Facial Expressions

Author info »

Introduction

In clinical practice, the accurate documentation and assessment of facial lesions present significant challenges due to the variability introduced by different photographic angles and facial expressions. These variations complicate the quantification of facial skin lesions, making it difficult to track skin disease progression and calculate severity levels using conventional software tools. This limitation is prevalent among many facial recognition systems and has spurred our efforts to develop advanced methods for facial landmark analysis. The process of quantifying facial lesions involves several critical steps, beginning with facial landmark image calibration. This initial phase ensures that facial images are accurately standardized, addressing potential distortions such as variations in facial pose, lighting conditions, and geometric discrepancies. Proper calibration is essential for the accurate representation of facial features in subsequent analyses. Following calibration, the next step employs deep learning-based automated lesion detection [1,2]. In this phase, advanced deep learning algorithms are utilized to train models capable of autonomously identifying and segmenting lesions within facial images. The model undergoes rigorous training on annotated datasets, enabling it to learn and recognize the distinct characteristics of lesions and thereby automate the detection process. The third step involves image binarization and edge detection optimization. The facial images are converted into a binary format, which simplifies the analysis and processing. Subsequently, edge detection techniques are applied to enhance the clarity of lesion boundaries [3]. This optimization is important for accurately delineating the shape and borders of lesions, ensuring precise measurement and evaluation. Finally, proportion and severity calculation is performed. This step involves a detailed analysis of the lesions, including their size, quantity, and distribution across the facial area. By quantifying these aspects, the severity of the condition can be assessed, and the effectiveness of any treatments can be evaluated based on the calculated proportions and severity metrics. Our investigation into existing literature on facial landmarks recognition systems revealed several models with varying levels of utility.

Materials and Methods

We conducted an experiment comparing the 68-point [4], 83- point [5], and 106-point [6] facial landmark models. In the experiment, as shown in Figure 1, a person was photographed with both a neutral and a smiling expression. We annotated these photos using the three models, with green dots indicating landmarks for the neutral expression and red dots for the smiling expression (Figure 1). We further assessed the models by calculating the Standard Deviation (SD) of the coordinates to measure alignment accuracy in Figure 2. Standard deviation is a statistical measure that indicates the extent of variation or dispersion of a set of values [7]. A smaller standard deviation signifies that the data points are closer to the mean, reflecting more precise alignment.

Figure 1: The neutral expressions and smiling expression in 68-point, 83-point and 106-point facial landmark models.

Figure 2: The standard deviation of facial landmark markings for smiling and neutral expressions of the same individual is analyzed across different facial landmark models.

In the analysis of the aligned and overlaid photos, where green dots marks represent neutral expressions and red dots marks indicate smiling expressions, it is evident that the 83-point facial landmark model demonstrates superior accuracy in key facial features such as the eyes, nose, mouth, and nasolabial folds compared to the other systems.

In this coordinate graph, it is evident that the 83-point facial landmark model exhibits the smallest standard deviation in unrefined states, indicating it is the most optimized and precise model overall. Here’s how standard deviation is calculated:

• Calculate the mean: Compute the average of all data points in the set.

• Compute the deviations: Subtract the mean from each data point to find the deviation of each point.

• Square the deviations: Square each deviation to ensure all values are positive.

• Compute the mean of squared deviations: Sum all the squared deviations and divide by the number of data points to get the variance.

• Calculate the standard deviation: Take the square root of the variance to get the standard deviation. The mathematical formula for standard deviation is:

Where,

• N is the number of data points

• x_i is the -th data point

• SD is the Standard Deviation

In the context of image alignment, the standard deviation calculation involves the following steps:

• Detect landmarks in both images: Detect facial landmarks or key points in both images.

• Align landmarks: Apply an affine transformation to align the landmarks of the second image to the landmarks of the first image.

• Compute deviation of aligned landmarks: Calculate the deviation between the aligned landmarks and the target landmarks.

• Calculate the standard deviation: Compute the standard deviation of these deviations to quantify the accuracy of the alignment.

Results and Discussion

Our results showed that the 83-point model had a smaller standard deviation, underscoring its superior alignment accuracy. The 83-point facial landmark model demonstrates superior accuracy over the 68-point model in both the long (Yaxis) and short (X-axis) axis. Although the 106-point model shows better performance in the short axis compared to the 83- point model, it significantly lags behind in the long axis. A close examination of the photos with different expressions reveals that the 83-point model still provides the best alignment and consistency across facial features in Figure 2.

Figure 2, illustrates that the 83-point facial landmark model outperforms the 68-point and 106-spoint models in matching expressions along both the long axis (Y-axis) and short axis (Xaxis) of the photos. Table 1, presents the precise standard deviation values for the accuracy of the three facial landmark models. This superiority highlights the model’s higher consistency and precision in landmark coordinates across different facial expressions. The 83-point model consistently demonstrated the highest accuracy and consistency between the two expressions.

Table 1: The precise standard deviation values for the accuracy of the three facial landmark models.

The 21-point facial landmarks system, designed primarily for biometric recognition, focuses on key facial features and the chin contour [8]. Despite its practical use, this system’s limited granularity constrains its effectiveness in more nuanced facial analyses. To address this, the 68-point facial landmarks system was developed to enhance biometric accuracy [4]. Although it offers improved granularity compared to the 21-point facial landmarks system, many of its additional points do not contribute significantly to overall accuracy due to their lack of meaningful definition [8]. An 81-point facial landmarks system, although has not published yet, includes the hairline in its contour analysis. This addition, however, is less effective due to the variability in hairline shapes among different individuals, which can undermine its practical application. The 106-point facial landmarks system increases the density of contour points but lacks specific definitions for each additional point, limiting its practical utility [6].

During our research, we tested these systems and identified that variations in photographic angles and facial expressions led to substantial calibration inaccuracies. This discovery led us to explore facial landmarks or features that remain relatively stable despite such variations, which we refer to as invariants. We identified facial retaining ligaments as critical components in this analysis. These fibrous structures anchor the skin and subcutaneous tissues to the underlying facial skeleton, thereby providing stability and maintaining the spatial relationships of facial features. Because retaining ligaments are less affected by facial movements and expressions, they serve as ideal reference points for accurate facial feature analysis [9]. Unlike connective tissues in other body parts, these ligaments are important for accommodating dynamic facial expressions and supporting facial fat compartments, significantly influencing the aging process and aesthetic outcomes. In response to the limitations of existing models, we developed a facial landmark model incorporating 83 points which inspired by facial retaining ligaments. This model provides a more precise framework for facial landmarks analysis by identifying stable and reliable reference points that remain consistent across various facial expressions and angles. By focusing on essential anatomical features, our 83-point system enhances accuracy compared to other models [4,6,8]. While incorporating the 106-point facial contour model and physiological invariants proposed for 2D photos could potentially enhance accuracy, our validation revealed that overfitting might introduce distortion into the photographs, which could adversely affect the model's overall precision. Therefore, while further integration of the 106-point model and 15 invariants could offer additional benefits, careful consideration must be given to the risks associated with overfitting condition.

Looking ahead, we are exploring the concept of selfquantification of the human body. Statistical data from various populations indicate that the iris, with a major diameter ranging from approximately 10.2 mm to 13.0 mm, is a stable dimension with minimal variation across ethnic groups [10]. This dimension provides a reliable self-scaling reference for accurately calculating the size and proportion of lesions in individuals. Our whole concept is applicable to both 2D and 3D models. However, our ongoing research is focused on determining whether similar self-scaling and invariant dimensions can be identified in other parts of the human body.

The integration of our facial feature models with advanced Artificial Intelligence (AI) and deep learning techniques holds significant potential for enhancing various applications. These models facilitate automatic detection and calibration of landmarks, which can be transformative across several domains. In the field of disease management, automated systems can provide more accurate and consistent evaluations of disease severity, improving clinical decision-making and treatment planning. Additionally, precise measurements and tracking capabilities enable effective monitoring of disease progression over time.

AI-driven analysis can also reveal correlations between disease severity and treatment outcomes, guiding personalized treatment approaches. In the cosmetic industry, advanced facial analysis can lead to customized beauty and cosmetic recommendations and even product development based on individual facial features and preferences. Moreover, accurate facial analysis can enhance product recommendation systems for anti-aging products by providing insights into user preferences and needs. In manufacturing, facial recognition technology can improve personnel management and security.

Remote healthcare is another potential application of our facial landmark models. Telemedicine enables patients to receive medical care and consultations without physically visiting a healthcare facility. By integrating precise facial landmark models into remote healthcare systems, which we called telemedicine [11], we can enhance remote monitoring. For example, changes in facial skin lesions can be tracked over time, allowing healthcare providers to assess skin disease progression and adjust treatment plans as needed. Accurate facial landmark models can also improve diagnostic accuracy during teleconsultations, providing doctors with detailed insights into patients' conditions and facilitating more effective remote evaluations. Additionally, remote healthcare services powered by advanced facial analysis models can improve access to care for patients in remote or underserved areas, reducing the need for travel and associated costs

Conclusion

In conclusion, our research underscores the critical need for precise and consistent facial landmark models for accurate disease assessment and analysis. The development and validation of the 83-point facial landmark model demonstrate its superiority in maintaining accuracy across different facial expressions. While there is potential for further enhancement through integration with advanced AI and self-scaling dimensions, challenges such as excessive manual calibration and distortion must be addressed. The integration of these models into remote healthcare platforms holds potential for improving patient monitoring, teleconsultations, and personalized treatment, ultimately enhancing the quality and accessibility of medical care. Continued research and refinement of facial feature analysis models are essential for advancing applications across various domains and improving clinical and practical outcomes.

References

1. Shih C, Lin CH, Weng YC, Jian JY, Lin YC. Implementation of weakly supervised vitiligo treatment evaluation system. In 2022 IEEE 4th Eurasia conference on biomedical engineering, healthcare and sustainability (ECBIOS) .2022:44-47

   [Google Scholar]

2. Bian Z, Xia S, Xia C, Shao M. Weakly supervised vitiligo segmentation in skin image through saliency propagation. Comput Med Imaging Graph. 2020;85:101779.

   [Crossref] [Google Scholar] [PubMed]

3. Shih CH, Weng YC, Lin CH, Jian CY. Edge detection methods suitable for vitiligo treatment evaluation. National Safety Council. 2021.
4. Amato G, Falchi F, Gennaro C, Vairo C. A comparison of face verification with facial landmarks and deep features. In10th International Conference on Advances in Multimedia (MMEDIA). 2018:1-6.

   [Google Scholar]

5. Jhen‐Yu J, Yeh AT, Yu‐Ching W. Eighty‐three facial landmarks for facial image alignment. J Cosmet Dermatol. 2024.

   [Crossref] [Google Scholar] [PubMed]

6. Liu Y, Shen H, Si Y, Wang X, Zhu X, Shi H, et al. Grand challenge of 106-point facial landmark localization. In 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). 2019: 613-616.

   [Google Scholar]

7. Lovric M. International encyclopedia of statistical science. 2011.

   [Google Scholar]

8. Koestinger M, Wohlhart P, Roth PM, Bischof H. Annotated facial landmarks in the wild: A large-scale, real-world database for facial landmark localization. In 2011 IEEE International Conference on Computer Vision workshops (ICCV workshops). 2011: 2144-2151.

   [Google Scholar]

9. Alghoul M, Codner MA. Retaining ligaments of the face: Review of anatomy and clinical applications. Aesthet Surg J. 2013;33(6):769-782.

   [Crossref] [Google Scholar] [PubMed]

10. Li Y, Huang D. Pupil size and iris thickness difference between asians and caucasians measured by optical coherence tomography. Invest Ophthalmol Vis Sci. 2009;50(13):5785.

    [Google Scholar]

11. Field MJ, editor. Telemedicine: A guide to assessing telecommunications for health care. 1996.

    [Google Scholar]