Sage Journals: Discover world-class research

Abstract

The integration of artificial intelligence (AI) in healthcare, particularly in the domain of facial processing tasks, has witnessed substantial growth in the 21st century. However, this requires sufficient appraisal for clinicians and researchers to adequately understand nomenclature and key concepts commonly used in this field. This article aims to elucidate the diverse applications of facial processing tasks, such as facial landmark extraction, face detection, face tracking, facial expression recognition and action unit detection, and their relevance to ophthalmology and other medical specialties. The keywords ‘ophthalmology’, ‘facial artificial intelligence’, ‘facial recognition’ and ‘periorbital measurements’ were used on PubMed and Ovid, between September 2012 and September 2022, to identify and screen for eligible articles. Studies reporting on human patients in ophthalmology, plastic, maxillofacial and cosmetic surgery with ocular lesions whose facial biometrics were processed by AI and written in the English language were included. A total of 291 and 513 articles were identified on PubMed and Ovid respectively. Twenty articles were included for analysis in this literature review after duplicates, inaccessible articles and articles without full manuscripts were excluded. Although fully automated algorithms can share the workload in healthcare systems and relieve strains on manpower, rigorous testing is crucial, followed by the challenges of convincing management bodies that it would work in reality, coupled with the costs of implementing specialised functional hardware and software. While patients have a valid concern that it would reduce physical contact with clinicians, it is important for clinicians not to replace clinical decision-making with AI alone.

Plain language summary

Study on reviewing basic and complex artificial intelligence tasks that involve the face in eye surgery and medicine

Aims and purpose of the research: The use of artificial intelligence (AI), especially those which tasks that process faces, is getting more and more common in medicine. This development has been huge within this century. In order to use these tools properly, doctors and scientists ought to understand basic concepts and terms. This study aims to achieve that and discuss usages of the diverse array of facial AI tools. Methods and research design: The researchers looked up key terms related to the subject in matter to identify other studies that looked into facial AI tasks on databases. The researchers were particularly interested in identifying English language studies that looked at human patients under eye surgeons, plastic surgeons, and face surgeons whose faces were used by facial AI. Results and importance: The researchers identified 20 studies suitable to be looked at in this study and discussed the usage of facial AI tasks mentioned by these studies. It is important as it can potentially make healthcare systems more efficient, however there will be obstacles to be addressed before these tools can be fully implemented in daily usage in medicine. It is also important to note that a human doctor’s decision making in medicine cannot be replaced by these tools as well.

Keywords

facial artificial intelligence facial processing tasks ophthalmology review

Introduction

Medical advancement over the 21st century has been heavily intertwined with technological innovation and computer dependency. The last decade alone has seen artificial intelligence (AI) increasingly applied across different arms of healthcare and medical specialties for a plethora of specific purposes.^1,2 However, ubiquitous AI application in the medical context, as with all good practice, requires sufficient appraisal and with this, clinicians and researchers must adequately understand nomenclature and the concepts defining these terms.

The concept of AI was first coined by McCarthy et al. in 1956, who proceeded to define AI as intelligent machines capable of the following aspects to mimic problem-solving competency of the human mind: using language, forming abstractions and concepts, solving problems and self-improvement.³ Facial AI is an overarching category of AI that encompasses any application, model or algorithm designed to intelligently process the face or part of it using input data from facial images or videos. Hupont et al. reported the most commonly used and researched facial processing tasks included the following: face detection, face tracking, facial landmark extraction, face spoofing detection, face identification, face verification, kinship verification, facial expression recognition, action unit (AU) detection, automatic lip reading, facial attribute estimation and facial attribute manipulation.⁴ These facial processing tasks can be used in isolation or built on each other to perform more complex tasks.

The first automated facial AI recognition system dates back to the 1960s in the USA.⁵ Despite still requiring human input and moderation, its efficiency and accuracy were shown to be superior to humans. It was not until the 1990s when this technology was no longer a military-only asset and commercial applications of facial AI were encouraged, greatly increasing the size of databases and thus the accuracy of AI-generated predictions.⁵ The application of facial AI systems is ubiquitous in the 21st century, including identification verification, surveillance and forensics.^6,7 Facial processing tasks have been applied worldwide in different scenarios, both by academic researchers and industry players. In this article, we will discuss the aforementioned facial AI processing tasks and their applications relevant to ophthalmology and other specialties.

Methods

The study design, inclusion and exclusion criteria were defined before the literature search was conducted to identify all eligible records.

The inclusion criteria involved all studies:

- Reporting on patients in ophthalmology and subspecialties, plastic, maxillofacial and cosmetic surgery with ocular lesions whose facial biometrics were processed by AI.

- Written in English

- Reporting on only human subjects

- Published on September 2012–September 2022

Exclusion criteria involved:

- Published in languages other than English

- AI that did not process facial features

- No full-text copy was available.

The following keywords were used: ‘ophthalmology’, ‘facial artificial intelligence’, ‘facial recognition’ and ‘periorbital measurements’.

Records on PubMed and Ovid were identified and screened for eligibility by two reviewers (J.C. and W.L.) independently. Conflicts in the process were solved by discussing with the senior author (P.D.). A quantitative summary of identified records was extracted, including ophthalmological subspecialty or non-ophthalmological specialty, input type and biometric type.

Results

A total of 291 and 513 articles were identified on PubMed and Ovid respectively. Four hundred eighty-three duplicate articles were excluded. One irretrievable article was excluded. Eight articles out of scope were excluded. Four articles with unavailable full manuscripts were excluded. Twenty articles were included for analysis in this literature review. This is visualised as a PRISMA flowchart in Figure 1. A summary of included studies can be found in Table 1.^1,8–26

Figure 1.

PRISMA flowchart.

Table 1.

Summary of included studies.

First author	Year	Title	Specialty	Study type	Type of facial AI
Bahceci Simsek	2021	Analysis of surgical outcome after upper eyelid surgery by computer vision algorithm using face and facial landmark detection	Ophthalmology - Oculoplastic surgery	Study	Facial detection Facial landmark extraction
Bodnar	2015	Automated ptosis measurements from facial photographs	Ophthalmology - Oculoplastic surgery	Study	Facial landmark extraction
Bodnar	2019	Evaluating new ophthalmic digital devices for safety and effectiveness in the context of rapid technological development	Ophthalmology	Review	-
Bouguila	2020	Facial plastic surgery and face recognition algorithms: Interaction and challenges. A scoping review and future directions	Plastic surgery	Review	-
Brahim	2022	Automation of dry eye disease quantitative assessment: a review	Ophthalmology	Review	-
Chen	2021	Smartphone-based artificial intelligence-assisted prediction for eyelid measurements: algorithm development and observational validation study	Ophthalmology - Oculoplastic surgery	Study	Facial landmark extraction
He	2020	Deployment of artificial intelligence in real-world practice: opportunity and challenge	Ophthalmology	Review	-
Jarvis	2020	Artificial intelligence in plastic surgery: current applications, future directions and ethical implications	Plastic surgery	Review	-
Karlin	2022	Ensemble neural network model for detecting thyroid eye disease using external photographs	Ophthalmology - Oculoplastic surgery	Study	Facial attribute estimation
Kaur	2020	Application of tpsDig2 software in nasal angle measurements	Plastic surgery	Study	Facial landmark extraction Facial attribute estimation
Khetpal	2022	Perceived age and attractiveness using facial recognition software in rhinoplasty patients: a proof-of-concept study	Plastic surgery	Study	Facial attribute estimation
Li	2023	Digital technology, telemedicine and artificial intelligence in ophthalmology: a global perspective	Ophthalmology	Review	-
Lou	2019	A novel approach for automated eyelid measurements in blepharoptosis using digital image analysis	Ophthalmology - Oculoplastic surgery	Study	Facial landmark extraction
Parikh	2020	Advances in telemedicine in ophthalmology	Ophthalmology	Review	-
Qu	2022	Effect of multichannel convolutional neural network-based model on the repair and aesthetic effect of eye plastic surgery patients	Ophthalmology - Oculoplastic surgery	Study	Facial attribute estimation
Thomas	2020	An artificial intelligence approach to the assessment of abnormal lid position	Ophthalmology - Oculoplastic surgery	Study	Facial landmark extraction Facial attribute estimation
Ting	2019	Artificial intelligence and deep learning in ophthalmology	Ophthalmology	Review	-
Ting	2019	Deep learning in ophthalmology: the technical and clinical considerations	Ophthalmology	Review	-
Tseng	2021	Considerations for artificial intelligence real-world implementation in ophthalmology: providers’ and patients’ perspectives	Ophthalmology	Review	-
Zuo	2019	Facial recognition technology: a primer for plastic surgeons	Plastic surgery	Review	-

AI, artificial intelligence.

Facial landmark extraction

Facial landmarks, also known as nodal points, are distinctive features that can act as a reference point on the face.^27–29 Common facial landmarks of interest in ophthalmology include pupillary distance, horizontal eyelid fissure, corneal diameter, margin-to-reflex distance (MRD) 1 and MRD 2, etc.

Multiple validation studies have repeatedly demonstrated AI can reliably perform automated eyelid and periorbital measurements in an automated fashion in static photos,^{12,15,19,30–32} as well as dynamically in real-time videos.^19,30,33 However, the images used in these studies were captured by professional cameras and lighting, especially when camera and lighting manoeuvres are required in response to the corneal light reflex, and thus may not be accessible to the general public.

Oculoplastic surgeons recognised readily accessible images, namely on social media, were likely from handheld devices.³⁴ Overall, the respective images lacked strict standardisation in facial position, lighting, shadowing and colour.³⁴ Despite this, there is a trend in validated algorithms developed to take advantage of the high accessibility of handheld devices and applications, where MRD1 and MRD2 can be statically and dynamically measured in smartphones,^23,31 as well as diagnosing blepharoptosis with a tablet application to allow for early referrals and thus treatment.³⁵

Facial attribute estimation

Facial attribute estimation utilises the information taken from facial landmark extraction to discern the presence or absence of certain facial characteristics.²⁷ The collective information includes facial features, gender, age and identity, which can allow the facial processing task to deduce an individual’s demographics.^36–39 Furthermore, AI development means it can now determine the sexual orientation of individuals based on facial features.⁴⁰

Companies utilise facial attribute estimation in marketing and advertising to identify individual’s demographics, for example age or gender, and thus target their products to that specific group.⁴¹ Facial attribute estimation is widely used across different specialties including ophthalmology. Especially in aiding diagnosis where network models are trained to identify facial benign skin lesions, differentiate between benign and malignant eyelid tumours, detect clinical features associated with thyroid eye disease, clinically rate eye closure in blepharospasm, evaluate dry eye disease and fully automate diagnosing ptosis from images.^{22,26,35,42–46} In oculoplastic surgery, Qu et al. were able to demonstrate a facial AI being capable of extracting specific facial attributes of eyelids and compare allocated aesthetic scores based on these outcome measures preoperatively and postoperatively.¹⁴

Neurologists have also used facial attribute estimation, mainly diagnostically, in neurological pathologies that compromise oro-facial function for example stroke detection or facial nerve palsy.^47,48 They analyse facial movements and the extent of facial alignment bias.⁴⁹ Another use of facial attribute estimation by neurology and geneticists includes diagnosing syndromic genetic disorders by identification of dysmorphic craniofacial features.^50,51

Facial attribute estimation, however, does have its limitations and concerns around its use. Regarding demographics, there are ethical problems arising should this be implemented for discriminatory purposes and targeting individuals of particular ethnicities for systemic surveillance.^52–54 Algorithms used in oculoplastic surgery of eyelids mentioned previously were set on Western and Asian Populations. Thus it could risk further enforcement of unattainable beauty standards on different let alone the same cultures.²⁵ The same concern is present in plastic surgery when aesthetic ratings are used in post-rhinoplasty patients which again burdening people with an unattainable beauty standard based on a perceived attractiveness rating determined by AI.^9,10,55

Face detection

Face detection utilises nodal points on the face, typically eyes as they are easily identifiable features, followed by surrounding anatomical landmarks, and then outputs the location and extent of each face present.^27,56 During the COVID-19 pandemic, face detection played a part in public health in occupancy control and social distance monitoring for disease control.⁴ The application of this facial processing task was also transferred to detecting facial masks.⁴

There is a recurring theme for facial AI in identifying facial landmarks and obtaining objective measurements in oculoplastics as discussed previously. Bahçeci Şimşek et al. recognised that the non-standardised nature of photographs adversely affects an algorithm’s ability to extract measurements of ophthalmic landmarks accurately.⁸ Such factors include the heterogeneity of photograph size and facial deformations secondary to head movements and facial expressions. By utilising facial detection, photographs can be calibrated for better accuracy which is useful in mobile platforms. Despite being highly accessible, mobile photographs are more prone to heterogeneity and deformation.

Face tracking

Face tracking is a facial processing task that follows the position of each face that comes into view in a video.⁵⁷ Social media applications such as Snapchat and Instagram utilise face tracking and apply selfie filters with augmented reality to integrate visual elements around the user’s face in real time.⁵⁸ Face tracking technology has shown to be feasible in reliable real-time dynamic capturing of facial landmarks, such as MRD1 and MRD2, from patient videos.³⁰

In developing an algorithm for blink detection, Kitazawa et al. utilised facial detection to identify the upper face and its respective anatomical landmarks to create a three-dimensional model to capture dynamic facial features,⁵⁹in this case, they are the eyes and eyelids for real-time analysis with commercially available cameras. Moriyama et al. also utilised the generation of three-dimensional models from capturing eyelid appearance and eye movements to mathematically describe their movements and contours in explaining how that contributes to the heterogenous appearances of eyes in individuals.⁶⁰

Motion-capturing eye movements are also a topic of interest in neurology, particularly in identifying abnormalities in posterior stroke patients and changes in depression patients from verbal distractions, showing that it can be a viable screening tool in aiding diagnosis.⁶¹

Facial expression recognition

Facial expression recognition aims to detect the emotion expressed on a person’s face in an image or video.⁶² This is used in pain management to aid in detecting pain or malingering.^4,63 It has also found its use in psychiatry to aid children with autism in emotional management.⁴

Plastic surgeons have utilised facial expression recognition in assessing facial palsy patients’ ability to express emotions before and after facial reanimation surgery,^64,65 as well as in the setting of face transplant patients.^66,67 These studies were useful in quantifying the surgical benefits and residual deficits of the procedures by determining facial expression through facial landmarks of the mouth or in conjunction with the eyes in videos or images derived from them.^64–66,68

AU detection

With human facial expressions catalogued into AUs in the Facial Action Coding System,⁶⁸ it is possible to more objectively quantify the changes and extent of facial muscle movement.⁶⁹ The nature of AUs allows for the objective quantification of dynamic assessment of facial muscle function before and after facial surgery.⁷⁰ Boonipat et al. more specifically compared facial muscle activity before and after browplasties and blepharoplasties using AUs to quantify the benefits and deficits of these procedures that correlated to emotions.⁷⁰

AUs have also been used to quantify facial muscle activity in stroke patients to identify specific weaknesses and regions that have been preserved functionally following a neurological insult.⁷¹ Volk et al. demonstrated that specific AUs associated with eyes did not have significant changes when compared to controls as most of the significantly affected AUs were related to the mouth.⁷¹

AUs were also used for identifying symptomatic patients with major depressive disorder in conjunction with face tracking in the absence of clinical diagnostic information.⁶¹ Stolicyn et al. showed that depression patients had lowered maximum intensity for upper lid raiser AU (AU5).⁶¹ Together with other AU findings, they were able to accurately identify depression patients in the absence of clinical diagnostic information. There were attempts to train algorithms to detect and measure pain in individuals with AUs, however a review concluded that this technology is not ready to be clinically applicable yet.⁷²

Facial attribute manipulation

Facial attribute manipulation involves altering features on a face. This involves exaggerating or diminishing a facial feature, altering AUs which alters facial expressions, changing the perceived age of a person, etc. Facial attribute manipulation is used to predict postoperative appearances in oculoplastic surgery, such as ptosis surgery or orbital decompression surgery for thyroid eye disease, provided that the integrity of patients’ consent is ensured.⁷³ This is useful in managing patients’ expectations, reassuring them and aiding clinical decisions.⁷⁴

However, facial attribute manipulation upon predicting the postoperative appearance can significantly impact the individual by highlighting potential imperfections pre-operation compared to post-operation. This can provide motivation for surgery and ethically there are blurred lines due to the monetary gain involvement. However, a major concern of facial attribute manipulation is ‘Deepfakes’, which is when people create images and replace a person’s face with another’s. This could cause ethical concerns as this could be used without the individual’s consent or knowledge. A notable example involved singer Taylor Swift being deepfaked into compromised situations without her consent.⁷⁵ This prompted the White House and the European Union to push bills to punish the creation of such content.^76,77

Face identification and verification

Face identification and verification utilise a database of faceprints.²⁷ Akin to fingerprints, faceprints are composed of data relating to measurements of up to 80–90 unique nodal points on a face that are labelled with an identity and are unique. This database is key for facial AI recognition systems to identify individuals. The AI works by identifying an individual’s face and then comparing their facial biometrics with recorded faceprints in the database for the closest match, which can be either accepted or rejected.^27,78,79

Face identification compares a detected face to a database containing facial profiles of interest.⁸⁰ This is a one-to-many approach. Close circuit video cameras utilise this for security and surveillance purposes by screening faces captured on a screen at an event for individuals of interest.⁸¹ Similarly, Facebook identifies the faces of people in uploaded images and automates tagging them to existing accounts.⁸²

Face authentication differs from face identification in only comparing a detecting face to another faceprint in the database.^27,78,79 This is a one-to-one approach. The AI aims to reach the closest match which can be either accepted or rejected.^27,78,79 This is commonly used in identity verification, such as airport check-ins and concert admittances.⁶ Its application can also be seen in on-site and virtual attendance moderation, especially during the COVID-19 pandemic.⁴ Attention was also given to facial appearance alterations within feasibility, as demonstrated by Apple’s Face ID taking into account factors such as aging, facial hair and makeup.⁸³

Plastic surgeons recognise that patients post-plastic surgery would not be recognised by algorithms and they might even take advantage of the situation to part-take in illegal activity without being able to be identified, as matching pre-surgery faces to post-surgery faces is a contemporary challenge.²¹ A solution is to have plastic surgeons document and report pre-operative and post-operative facial changes to local authorities, given that patient consent has been granted.¹

Training the algorithms

Some studies were regional and therefore had algorithms trained for the local population or main ethnic group,^9,43,46 which pose biases when used to examine a general population and have a high possibility of error, for example, the wrong attribute being labelled to patients due to having darker skin colour.⁸⁴ Multiple studies recommended that a larger training dataset should be used for these algorithms, which should then be validated so that it can be as close as it can to the ground truth.^11,16,20,24 This is particularly important as each algorithm needs to be specifically trained to a specific function since there is no one-fits-all solution.¹⁶

Additionally, the choice of eye pathology for AI algorithms to be trained should be considered carefully. Slowly progressive eye conditions may be more appropriate than rapidly progressive eye conditions for automated algorithms in the context of screening as a missed diagnosis/false negative and thus delayed treatment may lead to significant sight loss in a brief period of time.²⁰

While multiple studies had investigated developing algorithms in clinically grading the condition of interest with severity scales,^14,71,85 studies that used attractiveness as an outcome measure quantified it with a Likert score.^10,55 As the nature of attractiveness can be subjective and is often determined by plastic surgeons, Dusseldorp et al. recommended that laypersons are better suited to evaluating this, as well as other social outcomes such as non-verbal communication, in the context of facial palsy postoperatively.⁸⁶

Privacy and security

With a large database of images or videos taken from patients and volunteers to train the facial AI algorithms, it is important to keep this information safe and confidential, especially when databases are shared to further improve training and cyber security issues may compromise this ^11,16,18,20. Therefore informed consent must be obtained from patients and volunteers, as well as cooperating with data providers and third-party data aggregators to ensure the storage and transfer of these databases are not compromised.^1,25

Implementation

Although fully automated algorithms can share the workload in healthcare systems and relieve strains on manpower, rigorous testing is crucial, followed by the challenges of convincing management bodies that it would work in reality, coupled with the costs of implementing specialised functional hardware and software.^37–39

Furthermore, clinicians must be educated and trained to learn how to use these algorithms, as well as setting and defining standards and guidelines, and monitoring their usage to maintain patient safety.^36,38 This is crucial for obtaining their informed consent, where clinicians have the responsibility to educate and inform them with adequate knowledge of facial AI in a setting where this is yet to be an accepted normal practice by the general population, especially technophobes.^13,36,37,39 A large challenge in doing so is the opaqueness of AI’s decision-making processes.^17,39–41

While patients have a valid concern that it would reduce physical contact with clinicians,³⁷ it is important for clinicians not to replace clinical decision-making with AI alone.⁵²

Discussion

Summary of main findings

This study compiled both everyday use examples and medical applications of facial AI processing tasks which were published between September 2012 and September 2022. The results of this literature search showed a compilation of secondary reviews and primary studies that focused on the conceptualisation and validation of AI algorithms. The identified studies mainly concern the subspecialty of oculoplastic surgery in ophthalmology, followed by plastic surgery, and to some degree otorhinolaryngology.

Amongst the aforementioned facial AI processing tasks, the identified primary studies mainly focus on facial landmark extraction and facial attribute estimation, especially the automation in determining these measurements or features of interest reliably and its use on readily assessable platforms like mobile phones.

Limitations of review and reliability of the status

The findings of this review were skewed towards a selection bias of oculoplastic applications of facial AI compared to other medical specialties. Moreover, this review was prone to reporting bias from irretrievable articles and articles with unavailable full manuscripts.

Although the aim of this study was to explain facial AI concepts and examples in medical applications for laymen, oversimplification may potentially fail to accurately reflect the complexity of actual facial AI processes. This is further complicated by the opaqueness of AI’s decision-making processes when it is impossible to clearly lay out how these algorithms produce their conclusions and whether they would be widely replicable.

Implications to current practice and why this is important

Facial AI is becoming increasingly ubiquitous, as demonstrated by the aforementioned every day and medical applications in this review. While these medical applications were not readily and widely implemented in day-to-day use as of the time of writing, it is within the bounds of reality that clinicians might have to rely on these tools in professional practice. The argument for its use is reinforced by the high workload clinicians have to face in public sectors while pressured into maintaining a good quality of healthcare. With this in mind, it would become imperative for clinicians to be familiar with basic terminology and concepts of facial AI when encountering these tools, communicating with patients, and using them effectively when clinically available.

With a lack of standardisation in the use of facial AI, it will be necessary to create regulatory guidelines for its use in clinical practice. In the short term, this may further increase clinical workload as the field attempts to adopt a new technique and monitor both its use and the predictions produced by AI. Reliable digital infrastructure is necessary for the successful adoption of facial AI in ophthalmology. This may not always be available and would be a challenge to overcome for its widespread application in the field.

Conclusion

The synergy of processing tasks’ machine learning with the already heavily image-based nature of ophthalmology allows it to be one of the most rapidly evolving specialties in medicine. With plans for its use in diagnosis and surgical planning, it has the potential to change current practice in the field. The highly adaptive nature of this technology also ensures it will remain up to date in promoting best practices. AI could help provide measurements for facial reconstruction, show patients post-operative appearance and predict potential diagnoses or make diagnoses quicker to attain thus enabling faster treatment so it can improve outcomes of conditions as they could be diagnosed earlier.

Outside of medicine, as well as inside, AI can be used for surveillance which can help track down people for example in manhunts after a crime. This can help aid professionals in tracking down individuals quicker, resolving solutions faster and ensuring quicker safety for the wider public.

Footnotes

Acknowledgements

None.

Declarations

ORCID iD

Jeremy Jia Hao Chan

References

Zuo

Saun

Forrest

CR.

Facial recognition technology: a primer for plastic surgeons. Plast Reconstr Surg 2019; 143: 1298e–1306e.

Boczar

Avila

Carter

, et al. Using facial recognition tools for health assessment. Plast Surg Nurs 2021; 41: 232–236.

McCarthy

Minsky

Rochester

, et al. A proposal for the dartmouth summer research project on artificial intelligence, August 31, 1955. AI Mag 2006; 27: 12–12.

Hupont

Tolan

Gunes

, et al. The landscape of facial processing applications in the context of the European AI Act and the development of trustworthy systems. Sci Rep 2022; 12: 10688.

Libby

Ehrenfeld

Facial recognition technology in 2021: masks, bias, and the future of healthcare. J Med Syst 2021; 45: 39.

Kortli

Jridi

Al Falou

, et al. Face recognition systems: a survey. Sensors 2020; 20: 342.

Kaur

Krishan

Sharma

, et al. Facial-recognition algorithms: a literature review. Med Sci Law 2020; 60: 131–139.

Bahçeci Şimşek

Şirolu

. Analysis of surgical outcome after upper eyelid surgery by computer vision algorithm using face and facial landmark detection. Graefe’s Arch Clin Exp Ophthalmol 2021; 259: 3119–3125.

Kaur

Krishan

Kaur

, et al. Application of tpsDig2 software in nasal angle measurements. J Craniofac Surg 2020; 31: 319–325.

10.

Khetpal

Peck

Parsaei

, et al. Perceived age and attractiveness using facial recognition software in rhinoplasty patients: a proof-of-concept study. J Craniofac Surg 2022; 33: 1540–1544.

11.

J-PO

Liu

Ting

DSJ

, et al. Digital technology, tele-medicine and artificial intelligence in ophthalmology: a global perspective. Prog Retin Eye Res 2021; 82: 100900.

12.

Lou

Yang

, et al. A novel approach for automated eyelid measurements in blepharoptosis using digital image analysis. Curr Eye Res 2019; 44: 1075–1079.

13.

Parikh

Armstrong

Liou

, et al. Advances in telemedicine in ophthalmology. Semin Ophthalmol 2020; 35: 210–215.

14.

Lin

, et al. Effect of multichannel convolutional neural network-based model on the repair and aesthetic effect of eye plastic surgery patients. Comput Math Methods Med 2022; 2022: 1–8.

15.

Thomas

PBM

Gunasekera

Kang

, et al. An artificial intelligence approach to the assessment of abnormal lid position. Plast Reconstr Surg Glob Open 2020; 8: E3089.

16.

Ting

DSW

Peng

Varadarajan

, et al. Deep learning in ophthalmology: the technical and clinical considerations. Prog Retin Eye Res 2019; 72: 100759.

17.

Ting

DSW

Pasquale

Peng

, et al. Artificial intelligence and deep learning in ophthalmology. Br J Ophthalmol 2019; 103: 167–175.

18.

Tseng

RMWW

Gunasekeran

Tan

SSH

, et al. Considerations for artificial intelligence real-world implementation in ophthalmology: providers’ and patients’ perspectives. Asia-Pacific J Ophthalmol 2021; 10: 299–306.

19.

Bodnar

Neimkin

Holds

JB.

Automated ptosis measurements from facial photographs. JAMA Ophthalmol 2016; 134: 146.

20.

Bodnar

Schuchard

Myung

, et al. Evaluating new ophthalmic digital devices for safety and effectiveness in the context of rapid technological development. JAMA Ophthalmol 2019; 137: 939.

21.

Bouguila

Khochtali

Facial plastic surgery and face recognition algorithms: interaction and challenges. A scoping review and future directions. J Stomatol Oral Maxillofac Surg 2020; 121: 696–703.

22.

Brahim

Lamard

Benyoussef

, et al. Automation of dry eye disease quantitative assessment: a review. Clin Experiment Ophthalmol 2022; 50: 653–666.

23.

Chen

H-C

Tzeng

S-S

Hsiao

Y-C

, et al. Smartphone-based artificial intelligence–assisted prediction for eyelid measurements: algorithm development and observational validation study. JMIR mHealth uHealth 2021; 9: e32444.

24.

Liu

, et al. Deployment of artificial intelligence in real-world practice: opportunity and challenge. Asia-Pacific J Ophthalmol 2020; 9: 299–307.

25.

Jarvis

Thornburg

Rebecca

, et al. Artificial intelligence in plastic surgery: current applications, future directions, and ethical implications. Plast Reconstr Surg Glob Open 2020; 8: e3200.

26.

Karlin

Gai

LaPierre

, et al. Ensemble neural network model for detecting thyroid eye disease using external photographs. Br J Ophthalmol 2023; 107: 1722–1729.

27.

Salama

Almansori

Taha

, et al. A deep facial recognition system using computational intelligent algorithms. PLoS One 2020; 15: e0242269.

28.

Çeliktutan

Ulukaya

Sankur

A comparative study of face landmarking techniques. EURASIP J Image Video Process 2013; 2013: 13.

29.

Iqtait

Mohamad

Mamat

Feature extraction for face recognition via active shape model (ASM) and active appearance model (AAM). In: IOP conference Series: Materials science and engineering, 2018, vol. 332, p. 012032. Bristol, UK: IOP Publishing.

30.

Schulz

Clarke

Makuloluwe

, et al. Automated video assessment of the eyelids. Invest Ophthalmol Vis Sci 2022; 63: 599–A0300.

31.

Van Brummen

Owen

Spaide

, et al. PeriorbitAI: artificial intelligence automation of eyelid and periorbital measurements. Am J Ophthalmol 2021; 230: 285–296.

32.

Choi

Lefebvre

Yoon

MK.

Validation of the facial assessment by computer evaluation (FACE) program for software-aided eyelid measurements. Orbit 2016; 35: 117–120.

33.

Schulz

Clarke

Makuloluwe

, et al. Automated extraction of clinical measures from videos of oculofacial disorders using machine learning: feasibility, validity and reliability. Eye 2023; 37: 2810–2816.

34.

Maamari

Farhood

Holds

, et al. Quantitative analysis of preoperative and postoperative photographs posted on social media by oculoplastic surgeons. Ophthalmic Plast Reconstr Surg 2022; 38: 571–576.

35.

Tabuchi

Nagasato

Masumoto

, et al. Developing an iOS application that uses machine learning for the automated diagnosis of blepharoptosis. Graefe’s Arch Clin Exp Ophthalmol 2022; 260: 1329–1335.

36.

Berry

DS.

Child and adult sensitivity to gender information in patterns of facial motion. Ecol Psychol 1991; 3: 349–366.

37.

Berry

DS.

What can a moving face tell us?

J Pers Soc Psychol 1990; 58: 1004–1014.

38.

Bruce

Valentine

When a nod’s as good as a wink: the role of dynamic information in facial recognition. In: Practical aspects of memory: current research and issues, Vol. 1: Memory in everyday life. Oxford: John Wiley & Sons, 1988, pp.169–174.

39.

Zheng

Guo

Huang

, et al. A survey of deep facial attribute analysis. Int J Comput Vis 2018; 128: 2002–2034.

40.

Wang

Kosinski

Deep neural networks are more accurate than humans at detecting sexual orientation from facial images. J Pers Soc Psychol 2018; 114: 246–257.

41.

Sun

Marketing strategy analysis and evaluation for the adidas. BCP Bus Manag 2022; 35: 317–322.

42.

Yang

Guo

, et al. Development and validation of two artificial intelligence models for diagnosing benign, pigmented facial skin lesions. Skin Res Technol 2021; 27: 74–79.

43.

Qiang

Chen

, et al. Artificial intelligence to detect malignant eyelid tumors from photographic images. NPJ Digit Med 2022; 5: 23.

44.

Huang

, et al. An intelligent diagnostic system for thyroid-associated ophthalmopathy based on facial images. Front Med 2022; 9: 920716.

45.

Hung

J-Y

Perera

Chen

K-W

, et al. A deep learning approach to identify blepharoptosis by convolutional neural networks. Int J Med Inform 2021; 148: 104402.

46.

Hung

J-Y

Chen

K-W

Perera

, et al. An outperforming artificial intelligence model to identify referable blepharoptosis for general practitioners. J Pers Med 2022; 12: 283.

47.

Knappmeyer

Thornton

Etcoff

, et al. Facial motion and the perception of facial attractiveness. J Vis 2010; 2: 612–612.

48.

Wang

Dong

Sun

, et al. Automatic recognition of facial movement for paralyzed face. Biomed Mater Eng 2014; 24: 2751–2760.

49.

Bandini

Rezaei

Guarin

, et al. A new dataset for facial motion analysis in individuals with neurological disorders. IEEE J Biomed Heal Informatics 2021; 25: 1111–1119.

50.

Ferry

Steinberg

Webber

, et al. Diagnostically relevant facial gestalt information from ordinary photos. Elife 2014; 3: e02020.

51.

Hong

Zheng

Y-Y

Xin

, et al. Genetic syndromes screening by facial recognition technology: VGG-16 screening model construction and evaluation. Orphanet J Rare Dis 2021; 16: 344.

52.

Wang

Zhang

Liu

, et al. Expression of concern: facial feature discovery for ethnicity recognition. WIREs Data Min Knowl Discov 2019; 9: e1278.

53.

Van Noorden

. The ethical questions that haunt facial-recognition research. Nature 2020; 587: 354–358.

54.

Daly

Algorithmic oppression with Chinese characteristics : AI against Xinjiang’s Uyghurs. 2019. Glasgow, UK: University of Strathclyde.

55.

Parsaei

Peck

Khetpal

, et al. QS6: improvements in perceived age and attractiveness following aesthetic rhinoplasty using facial recognition and artificial intelligence. Plast Reconstr Surg Glob Open 2021; 9: 46–47.

56.

Yang

Luo

Loy

, et al. WIDER FACE: a face detection benchmark. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, vol. 2016, pp.5525–5533. Las Vegas, NV: IEEE.

57.

Barquero

Hupont

Fernández

Rank-based verification for long-term face tracking in crowded scenes. IEEE Trans Biometrics Behav Identity Sci 2021; 3: 495–505.

58.

Lavrence

Cambre

“Do i look like my selfie?”: filters and the digital-forensic gaze. Soc Media + Soc 2020; 6: 205630512095518.

59.

Kitazawa

Yoshimura

Liang

K-C

, et al. Utilization of facial image analysis technology for blink detection: a validation study. Eye Contact Lens Sci Clin Pract 2018; 44: S297–S301.

60.

Moriyama

Kanade

Jing

Xiao

, et al. Meticulously detailed eye region model and its application to analysis of facial images. IEEE Trans Pattern Anal Mach Intell 2006; 28: 738–752.

61.

Stolicyn

Steele

Seriès

Prediction of depression symptoms in individual subjects with face and eye movement tracking. Psychol Med 2022; 52: 1784–1792.

62.

Hupont

Cerezo

Baldassarri

Sensing facialemotions in a continuous 2D affective space. In: Proceedings of international conference on systems, man, and cybernetics, 2010, 2045–2051. Las Vegas, NV: IEEE.

63.

Cao

Ton-That

Choi

Il . An effective facial expression recognition approach for intelligent game systems. Int J Comput Vis Robot 2016; 6: 223–234.

64.

Boonipat

Asaad

Lin

, et al. Using artificial intelligence to measure facial expression following facial reanimation surgery. Plast Reconstr Surg 2020; 146: 1147–1150.

65.

Dusseldorp

Guarin

van Veen

, et al. Automated spontaneity assessment after smile reanimation: a machine learning approach. Plast Reconstr Surg 2022; 149: 1393–1402.

66.

Bedeloglu

Topcu

Akgul

, et al. Image-based analysis of emotional facial expressions in full face transplants. J Med Syst 2018; 42: 42.

67.

Hebel

Boonipat

Lin

, et al. Correlation of face transplant smile excursion measurements with emotional evaluation by artificial intelligence. Plast Reconstr Surg Glob Open 2021; 9: 32–32.

68.

Harrigan

Rosenthal

Scherer

The New Handbook of Methods in Nonverbal Behavior Research, 2008, pp.1–536. New York, NY: Oxford University Press.

69.

Rosenberg

Ekman

What the face reveals: basic and applied studies of spontaneous expression using the facial action coding system (FACS). 2020. Oxford, UK: Oxford University Press.

70.

Boonipat

Lin

Bite

Detection of baseline emotion in brow lift patients using artificial intelligence. Aesthetic Plast Surg 2021; 45: 2742–2748.

71.

Volk

Steinerstauch

Lorenz

, et al. Facial motor and non-motor disabilities in patients with central facial paresis: a prospective cohort study. J Neurol 2019; 266: 46–56.

72.

Dawes

Eden-Green

Rosten

, et al. Objectively measuring pain using facial expression: is the technology finally ready? Pain Manag 2018; 8: 105–113.

73.

Yoo

Choi

Kim

HK.

A generative adversarial network approach to predicting postoperative appearance after orbital decompression surgery for thyroid eye disease. Comput Biol Med 2020; 118: 103628.

74.

Sun

Huang

Zhang

, et al. A fully automatic postoperative appearance prediction system for blepharoptosis surgery with image-based deep learning. Ophthalmol Sci 2022; 2: 100169.

75.

Kent

. Taylor Swift deepfakes spread online, sparking outrage. CBS News 2024.

76.

Montgomery

Taylor Swift AI images prompt US bill to tackle nonconsensual, sexual deepfakes. The Guardian, https://www.theguardian.com/technology/2024/jan/30/taylor-swift-ai-deepfake-nonconsensual-sexual-images-bill (2024, accessed 4 July 2024).

77.

Goujard

Taylor Swift deepfakes nudge EU to get real about AI. Politico, https://www.politico.eu/article/europe-eye-fix-taylor-swift-nude-deepfake/ (2024, accessed 4 July 2024).

78.

Wang

Alfalou

Brosseau

New perspectives in face correlation research: a tutorial. Adv Opt Photon 2017; 9: 1–78.

79.

Esmaeel

Pasandi

Face, age and gender recognition using local descriptors. University of Ottawa. 2014.

80.

Negri

Cumani

Bottino

Tackling age-invariant face recognition with non-linear PLDA and pairwise SVM. IEEE Access 2022; 9: 40649–40664.

81.

Bennett

Can facial recognition technology be used to fight the new way against terrorism: examining the constitutionality of facial recognition surveilance systems. North Carolina J Law Technol 2001; 3: 151.

82.

Taigman

Yang

Ranzato

, et al. DeepFace: closing the gap to human-level performance in face verification. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit 2014; 1701–1708.

83.

Face ID Security. Apple Inc., https://www.apple.com/business-docs/FaceID_Security_Guide.pdfFace ID Security (2017, accessed 4 July 2024).

84.

Chen

Cheng

, et al. Facial recognition neural networks confirm success of facial feminization surgery. Plast Reconstr Surg 2020; 145: 203–209.

85.

Peterson

Littlewort

Bartlett

, et al. Objective, computerized video-based rating of blepharospasm severity. Neurology 2016; 87: 2146–2153.

86.

Dusseldorp

van Veen

Mohan

, et al. Outcome tracking in facial palsy. Otolaryngol Clin North Am 2018; 51: 1033–1050.

Facial artificial intelligence in ophthalmology and medicine: fundamental and transformative applications

Abstract

Keywords

Introduction

Methods

Results

Facial landmark extraction

Facial attribute estimation

Face detection

Face tracking

Facial expression recognition

AU detection

Facial attribute manipulation

Face identification and verification

Other trends and topics

Training the algorithms

Privacy and security

Implementation

Discussion

Summary of main findings

Limitations of review and reliability of the status

Implications to current practice and why this is important

Conclusion

Footnotes

Acknowledgements

Declarations

ORCID iD

References