Abstract
The integration of artificial intelligence (AI) in healthcare, particularly in the domain of facial processing tasks, has witnessed substantial growth in the 21st century. However, this requires sufficient appraisal for clinicians and researchers to adequately understand nomenclature and key concepts commonly used in this field. This article aims to elucidate the diverse applications of facial processing tasks, such as facial landmark extraction, face detection, face tracking, facial expression recognition and action unit detection, and their relevance to ophthalmology and other medical specialties. The keywords ‘ophthalmology’, ‘facial artificial intelligence’, ‘facial recognition’ and ‘periorbital measurements’ were used on PubMed and Ovid, between September 2012 and September 2022, to identify and screen for eligible articles. Studies reporting on human patients in ophthalmology, plastic, maxillofacial and cosmetic surgery with ocular lesions whose facial biometrics were processed by AI and written in the English language were included. A total of 291 and 513 articles were identified on PubMed and Ovid respectively. Twenty articles were included for analysis in this literature review after duplicates, inaccessible articles and articles without full manuscripts were excluded. Although fully automated algorithms can share the workload in healthcare systems and relieve strains on manpower, rigorous testing is crucial, followed by the challenges of convincing management bodies that it would work in reality, coupled with the costs of implementing specialised functional hardware and software. While patients have a valid concern that it would reduce physical contact with clinicians, it is important for clinicians not to replace clinical decision-making with AI alone.
Plain language summary
Introduction
Medical advancement over the 21st century has been heavily intertwined with technological innovation and computer dependency. The last decade alone has seen artificial intelligence (AI) increasingly applied across different arms of healthcare and medical specialties for a plethora of specific purposes.1,2 However, ubiquitous AI application in the medical context, as with all good practice, requires sufficient appraisal and with this, clinicians and researchers must adequately understand nomenclature and the concepts defining these terms.
The concept of AI was first coined by McCarthy et al. in 1956, who proceeded to define AI as intelligent machines capable of the following aspects to mimic problem-solving competency of the human mind: using language, forming abstractions and concepts, solving problems and self-improvement. 3 Facial AI is an overarching category of AI that encompasses any application, model or algorithm designed to intelligently process the face or part of it using input data from facial images or videos. Hupont et al. reported the most commonly used and researched facial processing tasks included the following: face detection, face tracking, facial landmark extraction, face spoofing detection, face identification, face verification, kinship verification, facial expression recognition, action unit (AU) detection, automatic lip reading, facial attribute estimation and facial attribute manipulation. 4 These facial processing tasks can be used in isolation or built on each other to perform more complex tasks.
The first automated facial AI recognition system dates back to the 1960s in the USA. 5 Despite still requiring human input and moderation, its efficiency and accuracy were shown to be superior to humans. It was not until the 1990s when this technology was no longer a military-only asset and commercial applications of facial AI were encouraged, greatly increasing the size of databases and thus the accuracy of AI-generated predictions. 5 The application of facial AI systems is ubiquitous in the 21st century, including identification verification, surveillance and forensics.6,7 Facial processing tasks have been applied worldwide in different scenarios, both by academic researchers and industry players. In this article, we will discuss the aforementioned facial AI processing tasks and their applications relevant to ophthalmology and other specialties.
Methods
The study design, inclusion and exclusion criteria were defined before the literature search was conducted to identify all eligible records.
The inclusion criteria involved all studies:
- Reporting on patients in ophthalmology and subspecialties, plastic, maxillofacial and cosmetic surgery with ocular lesions whose facial biometrics were processed by AI.
- Written in English
- Reporting on only human subjects
- Published on September 2012–September 2022
Exclusion criteria involved:
- Published in languages other than English
- AI that did not process facial features
- No full-text copy was available.
The following keywords were used: ‘ophthalmology’, ‘facial artificial intelligence’, ‘facial recognition’ and ‘periorbital measurements’.
Records on PubMed and Ovid were identified and screened for eligibility by two reviewers (J.C. and W.L.) independently. Conflicts in the process were solved by discussing with the senior author (P.D.). A quantitative summary of identified records was extracted, including ophthalmological subspecialty or non-ophthalmological specialty, input type and biometric type.
Results
A total of 291 and 513 articles were identified on PubMed and Ovid respectively. Four hundred eighty-three duplicate articles were excluded. One irretrievable article was excluded. Eight articles out of scope were excluded. Four articles with unavailable full manuscripts were excluded. Twenty articles were included for analysis in this literature review. This is visualised as a PRISMA flowchart in Figure 1. A summary of included studies can be found in Table 1.1,8–26
PRISMA flowchart.
Summary of included studies.
AI, artificial intelligence.
Facial landmark extraction
Facial landmarks, also known as nodal points, are distinctive features that can act as a reference point on the face.27–29 Common facial landmarks of interest in ophthalmology include pupillary distance, horizontal eyelid fissure, corneal diameter, margin-to-reflex distance (MRD) 1 and MRD 2, etc.
Multiple validation studies have repeatedly demonstrated AI can reliably perform automated eyelid and periorbital measurements in an automated fashion in static photos,12,15,19,30–32 as well as dynamically in real-time videos.19,30,33 However, the images used in these studies were captured by professional cameras and lighting, especially when camera and lighting manoeuvres are required in response to the corneal light reflex, and thus may not be accessible to the general public.
Oculoplastic surgeons recognised readily accessible images, namely on social media, were likely from handheld devices. 34 Overall, the respective images lacked strict standardisation in facial position, lighting, shadowing and colour. 34 Despite this, there is a trend in validated algorithms developed to take advantage of the high accessibility of handheld devices and applications, where MRD1 and MRD2 can be statically and dynamically measured in smartphones,23,31 as well as diagnosing blepharoptosis with a tablet application to allow for early referrals and thus treatment. 35
Facial attribute estimation
Facial attribute estimation utilises the information taken from facial landmark extraction to discern the presence or absence of certain facial characteristics. 27 The collective information includes facial features, gender, age and identity, which can allow the facial processing task to deduce an individual’s demographics.36–39 Furthermore, AI development means it can now determine the sexual orientation of individuals based on facial features. 40
Companies utilise facial attribute estimation in marketing and advertising to identify individual’s demographics, for example age or gender, and thus target their products to that specific group. 41 Facial attribute estimation is widely used across different specialties including ophthalmology. Especially in aiding diagnosis where network models are trained to identify facial benign skin lesions, differentiate between benign and malignant eyelid tumours, detect clinical features associated with thyroid eye disease, clinically rate eye closure in blepharospasm, evaluate dry eye disease and fully automate diagnosing ptosis from images.22,26,35,42–46 In oculoplastic surgery, Qu et al. were able to demonstrate a facial AI being capable of extracting specific facial attributes of eyelids and compare allocated aesthetic scores based on these outcome measures preoperatively and postoperatively. 14
Neurologists have also used facial attribute estimation, mainly diagnostically, in neurological pathologies that compromise oro-facial function for example stroke detection or facial nerve palsy.47,48 They analyse facial movements and the extent of facial alignment bias. 49 Another use of facial attribute estimation by neurology and geneticists includes diagnosing syndromic genetic disorders by identification of dysmorphic craniofacial features.50,51
Facial attribute estimation, however, does have its limitations and concerns around its use. Regarding demographics, there are ethical problems arising should this be implemented for discriminatory purposes and targeting individuals of particular ethnicities for systemic surveillance.52–54 Algorithms used in oculoplastic surgery of eyelids mentioned previously were set on Western and Asian Populations. Thus it could risk further enforcement of unattainable beauty standards on different let alone the same cultures. 25 The same concern is present in plastic surgery when aesthetic ratings are used in post-rhinoplasty patients which again burdening people with an unattainable beauty standard based on a perceived attractiveness rating determined by AI.9,10,55
Face detection
Face detection utilises nodal points on the face, typically eyes as they are easily identifiable features, followed by surrounding anatomical landmarks, and then outputs the location and extent of each face present.27,56 During the COVID-19 pandemic, face detection played a part in public health in occupancy control and social distance monitoring for disease control. 4 The application of this facial processing task was also transferred to detecting facial masks. 4
There is a recurring theme for facial AI in identifying facial landmarks and obtaining objective measurements in oculoplastics as discussed previously. Bahçeci Şimşek et al. recognised that the non-standardised nature of photographs adversely affects an algorithm’s ability to extract measurements of ophthalmic landmarks accurately. 8 Such factors include the heterogeneity of photograph size and facial deformations secondary to head movements and facial expressions. By utilising facial detection, photographs can be calibrated for better accuracy which is useful in mobile platforms. Despite being highly accessible, mobile photographs are more prone to heterogeneity and deformation.
Face tracking
Face tracking is a facial processing task that follows the position of each face that comes into view in a video. 57 Social media applications such as Snapchat and Instagram utilise face tracking and apply selfie filters with augmented reality to integrate visual elements around the user’s face in real time. 58 Face tracking technology has shown to be feasible in reliable real-time dynamic capturing of facial landmarks, such as MRD1 and MRD2, from patient videos. 30
In developing an algorithm for blink detection, Kitazawa et al. utilised facial detection to identify the upper face and its respective anatomical landmarks to create a three-dimensional model to capture dynamic facial features, 59 in this case, they are the eyes and eyelids for real-time analysis with commercially available cameras. Moriyama et al. also utilised the generation of three-dimensional models from capturing eyelid appearance and eye movements to mathematically describe their movements and contours in explaining how that contributes to the heterogenous appearances of eyes in individuals. 60
Motion-capturing eye movements are also a topic of interest in neurology, particularly in identifying abnormalities in posterior stroke patients and changes in depression patients from verbal distractions, showing that it can be a viable screening tool in aiding diagnosis. 61
Facial expression recognition
Facial expression recognition aims to detect the emotion expressed on a person’s face in an image or video. 62 This is used in pain management to aid in detecting pain or malingering.4,63 It has also found its use in psychiatry to aid children with autism in emotional management. 4
Plastic surgeons have utilised facial expression recognition in assessing facial palsy patients’ ability to express emotions before and after facial reanimation surgery,64,65 as well as in the setting of face transplant patients.66,67 These studies were useful in quantifying the surgical benefits and residual deficits of the procedures by determining facial expression through facial landmarks of the mouth or in conjunction with the eyes in videos or images derived from them.64–66,68
AU detection
With human facial expressions catalogued into AUs in the Facial Action Coding System, 68 it is possible to more objectively quantify the changes and extent of facial muscle movement. 69 The nature of AUs allows for the objective quantification of dynamic assessment of facial muscle function before and after facial surgery. 70 Boonipat et al. more specifically compared facial muscle activity before and after browplasties and blepharoplasties using AUs to quantify the benefits and deficits of these procedures that correlated to emotions. 70
AUs have also been used to quantify facial muscle activity in stroke patients to identify specific weaknesses and regions that have been preserved functionally following a neurological insult. 71 Volk et al. demonstrated that specific AUs associated with eyes did not have significant changes when compared to controls as most of the significantly affected AUs were related to the mouth. 71
AUs were also used for identifying symptomatic patients with major depressive disorder in conjunction with face tracking in the absence of clinical diagnostic information. 61 Stolicyn et al. showed that depression patients had lowered maximum intensity for upper lid raiser AU (AU5). 61 Together with other AU findings, they were able to accurately identify depression patients in the absence of clinical diagnostic information. There were attempts to train algorithms to detect and measure pain in individuals with AUs, however a review concluded that this technology is not ready to be clinically applicable yet. 72
Facial attribute manipulation
Facial attribute manipulation involves altering features on a face. This involves exaggerating or diminishing a facial feature, altering AUs which alters facial expressions, changing the perceived age of a person, etc. Facial attribute manipulation is used to predict postoperative appearances in oculoplastic surgery, such as ptosis surgery or orbital decompression surgery for thyroid eye disease, provided that the integrity of patients’ consent is ensured. 73 This is useful in managing patients’ expectations, reassuring them and aiding clinical decisions. 74
However, facial attribute manipulation upon predicting the postoperative appearance can significantly impact the individual by highlighting potential imperfections pre-operation compared to post-operation. This can provide motivation for surgery and ethically there are blurred lines due to the monetary gain involvement. However, a major concern of facial attribute manipulation is ‘Deepfakes’, which is when people create images and replace a person’s face with another’s. This could cause ethical concerns as this could be used without the individual’s consent or knowledge. A notable example involved singer Taylor Swift being deepfaked into compromised situations without her consent. 75 This prompted the White House and the European Union to push bills to punish the creation of such content.76,77
Face identification and verification
Face identification and verification utilise a database of faceprints. 27 Akin to fingerprints, faceprints are composed of data relating to measurements of up to 80–90 unique nodal points on a face that are labelled with an identity and are unique. This database is key for facial AI recognition systems to identify individuals. The AI works by identifying an individual’s face and then comparing their facial biometrics with recorded faceprints in the database for the closest match, which can be either accepted or rejected.27,78,79
Face identification compares a detected face to a database containing facial profiles of interest. 80 This is a one-to-many approach. Close circuit video cameras utilise this for security and surveillance purposes by screening faces captured on a screen at an event for individuals of interest. 81 Similarly, Facebook identifies the faces of people in uploaded images and automates tagging them to existing accounts. 82
Face authentication differs from face identification in only comparing a detecting face to another faceprint in the database.27,78,79 This is a one-to-one approach. The AI aims to reach the closest match which can be either accepted or rejected.27,78,79 This is commonly used in identity verification, such as airport check-ins and concert admittances. 6 Its application can also be seen in on-site and virtual attendance moderation, especially during the COVID-19 pandemic. 4 Attention was also given to facial appearance alterations within feasibility, as demonstrated by Apple’s Face ID taking into account factors such as aging, facial hair and makeup. 83
Plastic surgeons recognise that patients post-plastic surgery would not be recognised by algorithms and they might even take advantage of the situation to part-take in illegal activity without being able to be identified, as matching pre-surgery faces to post-surgery faces is a contemporary challenge. 21 A solution is to have plastic surgeons document and report pre-operative and post-operative facial changes to local authorities, given that patient consent has been granted. 1
Other trends and topics
Training the algorithms
Some studies were regional and therefore had algorithms trained for the local population or main ethnic group,9,43,46 which pose biases when used to examine a general population and have a high possibility of error, for example, the wrong attribute being labelled to patients due to having darker skin colour. 84 Multiple studies recommended that a larger training dataset should be used for these algorithms, which should then be validated so that it can be as close as it can to the ground truth.11,16,20,24 This is particularly important as each algorithm needs to be specifically trained to a specific function since there is no one-fits-all solution. 16
Additionally, the choice of eye pathology for AI algorithms to be trained should be considered carefully. Slowly progressive eye conditions may be more appropriate than rapidly progressive eye conditions for automated algorithms in the context of screening as a missed diagnosis/false negative and thus delayed treatment may lead to significant sight loss in a brief period of time. 20
While multiple studies had investigated developing algorithms in clinically grading the condition of interest with severity scales,14,71,85 studies that used attractiveness as an outcome measure quantified it with a Likert score.10,55 As the nature of attractiveness can be subjective and is often determined by plastic surgeons, Dusseldorp et al. recommended that laypersons are better suited to evaluating this, as well as other social outcomes such as non-verbal communication, in the context of facial palsy postoperatively. 86
Privacy and security
With a large database of images or videos taken from patients and volunteers to train the facial AI algorithms, it is important to keep this information safe and confidential, especially when databases are shared to further improve training and cyber security issues may compromise this 11,16,18,20. Therefore informed consent must be obtained from patients and volunteers, as well as cooperating with data providers and third-party data aggregators to ensure the storage and transfer of these databases are not compromised.1,25
Implementation
Although fully automated algorithms can share the workload in healthcare systems and relieve strains on manpower, rigorous testing is crucial, followed by the challenges of convincing management bodies that it would work in reality, coupled with the costs of implementing specialised functional hardware and software.37–39
Furthermore, clinicians must be educated and trained to learn how to use these algorithms, as well as setting and defining standards and guidelines, and monitoring their usage to maintain patient safety.36,38 This is crucial for obtaining their informed consent, where clinicians have the responsibility to educate and inform them with adequate knowledge of facial AI in a setting where this is yet to be an accepted normal practice by the general population, especially technophobes.13,36,37,39 A large challenge in doing so is the opaqueness of AI’s decision-making processes.17,39–41
While patients have a valid concern that it would reduce physical contact with clinicians, 37 it is important for clinicians not to replace clinical decision-making with AI alone. 52
Discussion
Summary of main findings
This study compiled both everyday use examples and medical applications of facial AI processing tasks which were published between September 2012 and September 2022. The results of this literature search showed a compilation of secondary reviews and primary studies that focused on the conceptualisation and validation of AI algorithms. The identified studies mainly concern the subspecialty of oculoplastic surgery in ophthalmology, followed by plastic surgery, and to some degree otorhinolaryngology.
Amongst the aforementioned facial AI processing tasks, the identified primary studies mainly focus on facial landmark extraction and facial attribute estimation, especially the automation in determining these measurements or features of interest reliably and its use on readily assessable platforms like mobile phones.
Limitations of review and reliability of the status
The findings of this review were skewed towards a selection bias of oculoplastic applications of facial AI compared to other medical specialties. Moreover, this review was prone to reporting bias from irretrievable articles and articles with unavailable full manuscripts.
Although the aim of this study was to explain facial AI concepts and examples in medical applications for laymen, oversimplification may potentially fail to accurately reflect the complexity of actual facial AI processes. This is further complicated by the opaqueness of AI’s decision-making processes when it is impossible to clearly lay out how these algorithms produce their conclusions and whether they would be widely replicable.
Implications to current practice and why this is important
Facial AI is becoming increasingly ubiquitous, as demonstrated by the aforementioned every day and medical applications in this review. While these medical applications were not readily and widely implemented in day-to-day use as of the time of writing, it is within the bounds of reality that clinicians might have to rely on these tools in professional practice. The argument for its use is reinforced by the high workload clinicians have to face in public sectors while pressured into maintaining a good quality of healthcare. With this in mind, it would become imperative for clinicians to be familiar with basic terminology and concepts of facial AI when encountering these tools, communicating with patients, and using them effectively when clinically available.
With a lack of standardisation in the use of facial AI, it will be necessary to create regulatory guidelines for its use in clinical practice. In the short term, this may further increase clinical workload as the field attempts to adopt a new technique and monitor both its use and the predictions produced by AI. Reliable digital infrastructure is necessary for the successful adoption of facial AI in ophthalmology. This may not always be available and would be a challenge to overcome for its widespread application in the field.
Conclusion
The synergy of processing tasks’ machine learning with the already heavily image-based nature of ophthalmology allows it to be one of the most rapidly evolving specialties in medicine. With plans for its use in diagnosis and surgical planning, it has the potential to change current practice in the field. The highly adaptive nature of this technology also ensures it will remain up to date in promoting best practices. AI could help provide measurements for facial reconstruction, show patients post-operative appearance and predict potential diagnoses or make diagnoses quicker to attain thus enabling faster treatment so it can improve outcomes of conditions as they could be diagnosed earlier.
Outside of medicine, as well as inside, AI can be used for surveillance which can help track down people for example in manhunts after a crime. This can help aid professionals in tracking down individuals quicker, resolving solutions faster and ensuring quicker safety for the wider public.
