Abstract
Background
The Cleft Lip Education with Augmented Reality (CLEAR) project centers around the use of augmented reality (AR) in patient leaflets, as a visual means to overcome the “health literacy” gap. This trial followed Virtual Reality (VR CORE) guidelines for VR Phase 2 (Pilot) trials.
Methods
Participants included families of children treated for Cleft Lip and Palate at the Royal Hospital for Children, Glasgow. Interventions were AR leaflet or Traditional Leaflet. Objectives were to calculate sample sizes, assess outcome instruments, trial design, and acceptability to patients. Primary outcome measure was Mental Effort Rating Scale, and secondary outcomes were Patient Satisfaction (Visual Analogue Scale), Usefulness Scale for Patient Information Material (USE) scale, and Instructional Materials Motivation Survey (IMMS). Randomization was by block randomization. The trial was single blinded with assessors blinded to group assignment.
Results
12 Participants were randomized, with crossover design permitting analysis of 12 per group. Primary outcome with Mental Effort Rating Scale indicated higher mental effort with Traditional compared to AR Leaflet (4.75 vs 2.00, P = .0003). Secondary outcomes for Satisfaction were Traditional 54.50 versus AR 93.50 (P = .0001); USE scale 49.42 versus 74.08 (P = .0011); and IMMS 112.50 versus 161.75 (P = .0003). Subjective interviews noted overwhelmingly positive patient comments regarding the AR leaflet. Outcome instruments and trial design were acceptable to participants. No harms were recorded.
Conclusions
The CLEAR pilot trial provides early evidence of clinical efficacy of AR leaflets in patient education. It is hoped that this will provide a future paradigm shift in the way patient education is delivered.
Keywords
Introduction
Inadequate information, poor health literacy and patient-doctor knowledge disconnect, may critically impact on patient understanding, satisfaction, choice, understanding of surgical complications, and consent. Patient education in this regard relies on patient information leaflets and the doctor-patient consultation. However, patient information leaflets unintentionally neglect a significant proportion of patients, with 16% to 27% in the UK having the lowest level of literacy (Trust, 2021). “Health literacy”—the ability to understand, act on and communicate health information—affects a larger proportion of the population, with approximately 40% unable to engage effectively with health information (Affairs, 1999). This is compounded by inaccurate, inconsistent, and confusing information in patient leaflets, with different hospitals providing their own, often disparate health information (McCartney, 2013). The doctor-patient consultation is highly practitioner dependent, and may therefore unwittingly leave patients inadequately informed and confused (Rubenstein, 2012). With these considerations in mind, our research looked to bypass problems associated with health literacy and the doctor-patient disconnect, with an entirely visual means of communication that leverages augmented reality (AR). This is the innovative use of digital technologies to superimpose information on the world we see, and may allow a more intuitive, 3D understanding of difficult spatial concepts, and as such is perfectly suited to the field of reconstructive surgery (Lo and Chapman, 2020a; Lo and Chapman, 2020b). This randomized crossover pilot trial examined an AR patient leaflet developed for unilateral cleft lip and palate.
One of our key goals in the development of the AR Cleft Lip and Palate Leaflet was to “Leave nobody behind in a digital world.” Therefore, the AR application was intended to be delivered via ubiquitous technology such as smartphones and tablets, rather than the use of inaccessible and costly technologies such as Head Mounted Displays (“Virtual or Augmented Reality Goggles”). Patient participation was placed at the heart of the development process, to include the patient voice early in the development of AR technologies (Birckhead et al., 2019). The international methodological guidelines proposed by the Virtual Reality Clinical Outcomes Research Experts (VR-CORE), subdivides the development of digital or Virtual Reality (VR) trials into 3 phases—VR Phases 1 to 3, which are akin to Clinical Trial Phases 1 to 3. Participatory development phase has been designated VR Phase 1 (VR 1) and has already been completed for the cleft AR leaflet (data unpublished). Here we describe VR Phase 2 (VR 2), which centers around pilot testing with the purpose of providing sample size data for a follow on definitive randomized trial, assessment of the relevance and clarity of the outcome instruments, evaluation of the acceptability of trial design to patients, and measurement of initial clinical efficacy. The follow-on randomized trial—VR Phase 3—aims to provide evidence of its tangible benefits to patients in terms of ease of understanding, satisfaction, and learner motivation, and thereby permit upscaling and implementation at a national level.
Methods
Patient Participatory Involvement—VR Phase 1
Patient participatory development preceded the randomized pilot trials, and is termed VR phase 1 in methodological guidelines on VR and AR trials (Birckhead et al., 2019). In brief, co-development was in collaboration between NHS Greater Glasgow and Clyde and the School of Simulation and Visualisation, The Glasgow School of Art. The development process involved co-development with cleft surgery clinicians, nurse specialists, and parent groups (10 families) over a period of 1 year. The process consisted of expert interview, patient journey mapping, storyboarding, prototyping and focus group assessments (data unpublished). Feedback on the leaflet shaped content, information, functionality, and delivery, prior to commencement of the pilot trial (VR2).
Pilot Randomized Crossover Trial—VR Phase 2
The Cleft Lip Education with Augmented Reality (CLEAR) patient leaflet pilot trial was reported according to the CONSORT extension for Pilot and Feasibility Trials (Eldridge et al., 2016). Trial Design was a randomized crossover trial with 1:1 allocation ratio, with no changes to trial design after commencement of the study. No washout period was used between groups, as the study investigated factors related to user perceptions of the method of information delivery—ease of comprehension, usability, satisfaction and learner motivation, rather than tests of factual knowledge that are more susceptible to carry-over effects. Ethics approvals were obtained from NHS Greater Glasgow and Clyde Research and Development Board.
Participants and Setting
Setting for data collection was The Royal Hospital for Children, Glasgow, and participant identification was sequential patients (both pre and post-operative) attending Cleft Outpatient Clinics or as inpatients for Cleft surgery on the Pediatric Wards, during the study period. Inclusion criteria included patients or parent/guardian of patients with cleft lip or palate; age 16-90; participants with or without dyslexia. Exclusion criteria included the inability to understand or communicate in English; registered blind; severe learning difficulties where participant cannot understand or communicate English. Consent was performed in writing by a member of the research team after participants had read a study information sheet.
Interventions
Crossover study with 1:1 allocation ratio, with groups including an AR Patient Information leaflet and a Traditional Patient Information Leaflet (The treatment of Cleft Lip and Palate. A Parent's guide. The Royal College of Surgeons of England) (Sommerlad, 2003). The Traditional group also included the AR Patient Leaflet without the AR functionality. Patients reviewed each leaflet consecutively on the same day, and given 20 min to review each type of leaflet. As the study focused on surgical aspects of Cleft Lip and Palate, the participants were asked to review pages in the Traditional Leaflet that were relevant to Cleft Lip surgery (pages 20-21) or Cleft Palate Surgery (pages 22-23). The AR leaflet consisted of an actual paper leaflet with AR 3D content automatically triggered by patterns within the leaflet, and viewed using a Samsung S7 plus tablet with custom App (Figures 1–4, Supplemental Figure 1, Supplemental Videos 1 and 2). This was demonstrated to the participants and they were then given the tablet and AR leaflet to use independently.
(a) The cleft lip and palate augmented reality leaflet. This is an actual paper leaflet with a pattern which facilitates tracking of the 3D model. (b) When the camera of the tablet or smartphone is pointed at the AR leaflet, the 3D model rises from the page. The model remains “fixed” to the leaflet and will move if the paper leaflet is moved or rotated.
(a) Demonstration of the markings for a unilateral cleft lip repair. (b) The “skin” goes translucent to enable visualization of underlying structure. Artificial colors are used to highlight specific anatomical areas. (c) The importance of the muscle repair is emphasized, a point which may not be obvious to some parents. (d) Sutures in the cleft lip repair.
As the model is fully 3D the user can go “inside” the model, or around the model and see the back or side of the head. The model has the full 3D anatomy created within the head. As it tracks the leaflet, the leaflet can be moved and the 3D model will remain attached to it.
(a) interactive components. The virtual menu buttons at the bottom of the screen include an AR animation with voiceover (green button), an interactive anatomical model with components that can be swiped and moved around (blue button), and a labelling function will labels the anatomical model and provides additional information in a pop-up box (red button). (b) The anatomical model is fully labelled, and each label can be pressed by the patient for additional information (colour in online).
Outcome Instruments
Outcomes were assessed with validated questionnaires, and subjective interview administered following patient review of both types of leaflet. Primary outcome was Mental Effort Rating Scale (Paas, 1992), a 9 point symmetrical Likert scale which assesses extraneous cognitive load (ease of understanding). Secondary outcomes included Patient Satisfaction (Visual Analogue Scale) (Voutilainen, 2016); Usefulness Scale for Patient Information Material (USE) scale, which consists of 9 questions in 3 subdomains -cognitive, emotional and behavioural (Holzel et al., 2015); and the Instructional Materials Motivation Survey (IMMS) which assesses learner motivation and is frequently used in VR and AR trials (Keller, 2010). Ancillary analyses included patient preference, thematic analysis of subjective interviews, and non-validated questions regarding the leaflets. No changes to outcome instruments occurred after trial commencement.
Sample Size
A pilot sample of 12 per group was chosen as previously specified (Julious, 2005)
Randomization
A University Statistical service provided randomization with 1:1 allocation, block size of 6, generated automatically by computer system. Allocation concealment was with a remote, password protected list held by a person not involved in the research. Enrolment and assignment of participants was performed by a member of the research team.
Blinding and Data Analysis
As the interventions involved patient leaflets, it was not possible to blind the participants, care providers or investigators at time of assessment. Data analysis was performed by investigators blinded to intervention allocation. Statistical analysis was performed by an independent statistical service using SPSS Statistics for Windows, Version 27.0. Armonk, NY (IBM Corp).
Results
Participants
Participant flow is outlined in Figure 5. No exclusions or withdrawals occurred after randomization. The recruitment period was from 22nd April to 27th May, and included 4 outpatient clinics and 3 operating lists. The pilot trial ceased upon pre-defined patient recruitment of 12 patients. Numbers analyzed in each group—as this is a crossover trial—were 12 participants per group. Baseline demographic and clinical data are shown in Table 1.
Participant eligibility, recruitment, and randomization.
Outcomes
The Mental Effort Rating Scale data were checked for normality, showing a normal distribution (P = .466), and therefore difference between groups was assessed with a dependent (paired) t-test, 2 tailed. Primary outcome with Mental Effort Rating Scale indicated Traditional Leaflet mean 4.75 (95%CI, 3.69-5.81) and AR Leaflet 2.00 (1.16-2.84), P = .0003. Secondary outcomes for satisfaction on VAS were Traditional 54.50 (40.72-68.28) and AR 93.50 (90.43-96.58), P = .0001. USE scale was Traditional 49.42 (39.39-59.45) and 74.08 (68.98-79.18), P = .0011. IMMS scores were Traditional 112.50 (95.88-129.12) and AR 161.75 (155.24-168.26), P = .0003 (Table 2).
Clinical and Demographic Variables.
Cross-Over Effects
Cross-over effects were checked against the Mental Effort Rating Scale. The difference between groups was analyzed according to the first system used, and assessed with a 2-sample t-test. There was no evidence of a difference between the order of the trials (P = .455).
Ancillary Analyses
Subdomain analyses of USE or IMMS scales were not performed for the pilot trial. Ancillary analyses of subjective patient interviews were not performed as part of the pilot, but will be explored in more detail in the follow-on trial. In brief, these were extremely positive regarding the AR leaflet. General themes included it making things “easier to understand,” “alleviating anxiety about the operation,” and patients of post-operative children wishing they had access to the AR leaflet before the operation. Comments included (Table 3).
The study also included non-validated questions exploring issues related to consent, understanding, anxiety, leaflet preference, and whether patients would prefer to go to a website (Table 4). These were descriptive in nature and were not analyzed for this pilot, although showed a strong user preference for the AR leaflet (Table 4).
Primary and Secondary Validated Outcome Measures.
Patient Subjective Interview Comments.
Directed Questions Regarding AR Leaflet.
Sample Size Estimates for Follow-on Trial.
Power for Follow-on Trial
The pilot trial primary and secondary outcomes were highly significant, therefore the pilot trial data was considered on statistical review to be definitive in its own right. Nonetheless, sample sizes were calculated for 2 independent means, significance of 0.05, power of 0.80 to 0.90, and with equivalence study design (Table 5).
Acceptability and Clarity of Outcome Instruments
Patients were asked about the acceptability of the length of the questionnaire, clarity of questions and relevance, with no patient raising any concerns. Of note, previous focus groups with parents of post-operative cleft children observed that the USE scale's behavioral subdomain (questions 7-9) were thought to be less relevant to cleft surgery (data not shown). This was not found in this pilot trial, perhaps reflecting the inclusion of pre-operative patients and the use of the trial material in a real-world context (clinics and pre-operatively) rather than in a focus group. As cleft provision in Scotland is provided centrally in Glasgow with many families travelling significant distances, it was therefore noted that any trial requiring scheduled in-person follow up would not be readily acceptable to parents.
No harms or unintended effects were noted in this pilot trial. In particular no parents noted anxiety, fear or upset when viewing the AR leaflets.
Discussion
A Future Paradigm Shift in Patient Information Leaflets
The data presented in this pilot trial relate primarily to establishing data for the follow-on trial, investigating potential sources of bias, assessing outcome instruments, trial design and acceptability to families with cleft children. Nonetheless, the pilot data strongly point towards a future paradigm shift in the way patient information can be delivered and leveraged with AR technologies. Statistical review suggested that given the highly significant outcomes in the pilot trial, the data presented here could be considered definitive and a follow on trial may not be required. The primary outcome measured with the Mental Effort Rating Scale points towards a highly significant benefit in ease of understanding with the AR leaflets. This scale is however not used in isolation, and is supported by significant secondary outcomes including Patient Satisfaction, the USE scale, and IMMS. These strongly point towards perceived benefits from a patient perspective regarding satisfaction, usability and learner motivation, which are all critical aspects in the patient information process.
Outcome Instruments, Trial Design, and Acceptability
There are no validated instruments specifically applicable to cleft patient information leaflets, although the outcome instruments used here are widely used in clinical and non-clinical research. Specifically, the tools chosen look at ease of comprehension (extraneous cognitive load), patient satisfaction, usability, and learner motivation. The Mental Effort Rating Scale was used to evaluate the extraneous cognitive load (or ease of comprehension) of the AR leaflets in comparison to traditional sources of information (Paas, 1992). This is a symmetrical, validated, categorical scale rated from 1 to 9, where 1 is very, very low mental effort, and 9 is very, very high mental effort. The extraneous cognitive load—the mental effort required to understand a topic—varies according to the manner in which information is presented to patients. This element is under the control of educational designers, and can vary according to the method of delivery, for example, AR 3D models versus text based information leaflets only. The inherent difficulty of the topic—known as the intrinsic cognitive load—is fixed and remains the same irrespective of method of delivery. The Mental Effort Rating Scale was chosen as it is simple to use, validated, widely used method of assessment of extraneous cognitive load, and an outcome instrument which we have used in previous studies (Lo and Chapman, 2020; Lo et al., 2020). For patient satisfaction a VAS was used in-lieu of a Likert scale, as this is less prone to confounding and ceiling effects (Voutilainen et al., 2016). The USE scale is a relatively new and not yet widely adopted scale, developed originally for back pain leaflets (Holzel et al., 2016). The scale consists of 9 questions spanning cognitive, emotional, and behavioral subdomains, and although concerns had been raised in focus groups about relevance of the behavioral subdomain, this was not noted to be significant in this pilot trial. The USE scale was chosen as a secondary outcome as it is one of the only validated scales for assessment of patient leaflet usability, with most existing scales focusing on clarity of English and legibility. The IMMS is widely used in the assessment of AR related learning technologies, based on the Attention, Relevance, Confidence and Satisfaction model of learning (Keller, 2010). This looks at learners’ motivation, which is highly relevant to patients as “motivation provides a source of energy that is responsible for why learners decide to make an effort, how long they are willing to sustain an activity, how hard they are going to pursue it, and how connected they feel to the activity.” (Rost, 2006).
Knowledge tests were not included in this trial, although was considered during the design of the study protocol. There are a number of reasons for not including knowledge tests, including lack of validated cleft knowledge questionnaires, the need for an extended washout period, and significant potential for confounding. At present, no validated questionnaires on parental cleft knowledge exist, and if developed for this trial they may be prone to developer bias. Additionally, knowledge tests require a prolonged washout period between crossover testing, which would be a significant burden for parents who may need to travel significant distances for follow-up testing, and would therefore not be practicable in the trial design. Significant confounding would also exist from parental educational level and the degree of parental self-directed learning (for example, the extent to which parents search for information on the internet). No washout period was used between groups in this pilot study, as this trial investigated factors related to user perceptions of the method of information delivery—ease of comprehension, usability, satisfaction and learner motivation—rather than tests of factual knowledge that may be more susceptible to carry-over effects. Indeed, no cross-over effects were noted in this trial irrespective of the first system used by participants.
Generalizability
The pilot trial methods and findings are generalizable to other studies. As the outcome instruments are not cleft lip specific, these can be used in other disease processes. Our research group is utilizing the same protocols for AR leaflet trials in other fields including sarcoma, flap reconstruction, and breast cancer.
Bias and Limitations
The present pilot study compares AR leaflets with the current standard-of-care traditional leaflet from the Royal College of Surgeons of England. Differences may relate therefore in part to content and design, rather than the AR component per se. This was mitigated by giving the Traditional leaflet group both a standard-of-care leaflet but also the AR leaflet without the AR functionality. This reduces differences in content and design between groups, allowing a clearer extrapolation of differences relating specifically to the AR functionality. Furthermore, the IMMS outcome instrument allows sub-analysis of differences in content and design, for example, “Q8 The materials are eye catching. Q23 The content and style of writing in this patient leaflet convey the impression that its content is worth knowing.” Subdomain analysis of the IMMS scale, and thematic analysis of subjective interviews in the follow-on trial, will more clearly define the patient experience of AR and its perceived benefits. Blinding is an issue with any trial using AR, as it cannot be hidden from participants, and can only be single-blinded prior to data analysis.
Considerations for Amendments to Follow on Trial and Sample Sizes
Although no cross-over effects were noted in this trial, to reduce potential methodological criticisms regarding cross-over effects and washout periods, the follow-on randomized crossover trial will include an additional between-subjects comparison of the AR leaflet compared to Traditional leaflet (ie additional analysis of a standard randomized controlled design). The trial will otherwise proceed as planned with the crossover, within-subjects comparison providing the primary outcome. Minimum clinically important differences are not yet known for the outcome instruments in this pilot trial—therefore we will be looking initially at an equivalence trial design rather than superiority, and sample size has been calculated accordingly. An estimated recruitment of 8-10 patients per group will be required based on a primary outcome measured with the Mental Effort Rating Scale. The trial will be powered to allow a between-subjects comparison in addition to the within-subjects crossover comparison, and therefore a total of 20 participants will be recruited.
Future Directions
This pilot trial provides data which appears to be strongly supportive of the benefits of AR in patient leaflets, and given its highly 3-dimensional nature, is of particular relevance to the field of Plastic Surgery. It is hoped that the AR Patient leaflet follow-on trial will provide definitive evidence of the benefits of AR patient leaflets in terms of patient understanding, satisfaction, usability and learner motivation. In doing so, this research may lead to tangible and critically important reductions in the “health literacy” gap worldwide.
Supplemental Material
sj-doc-3-cpc-10.1177_10556656211059709 - Supplemental material for The Cleft Lip Education with Augmented Reality (CLEAR) VR Phase 2 Trial: A Pilot Randomized Crossover Trial of a Novel Patient Information Leaflet
Supplemental material, sj-doc-3-cpc-10.1177_10556656211059709 for The Cleft Lip Education with Augmented Reality (CLEAR) VR Phase 2 Trial: A Pilot Randomized Crossover Trial of a Novel Patient Information Leaflet by Steven John Lo, Paul Chapman, David Young, David Drake, Mark Devlin and Craig Russell in The Cleft Palate Craniofacial Journal
Footnotes
Acknowledgments
We thank Danny Buksh, Sam Ramsay, and Victor Portela at the School of Simulation and Visualisation, Glasgow School of Art, and Jennifer Pettigrew, Cleft Specialist Nurse, The Royal Hospital for Children, for the development of the Augmented Reality leaflet.
Authors Contribution
SL contributed to inception, design, development, data collection, interpretation, and writing manuscript. PC contributed to inception, design, development, and writing manuscript. DY contributed to data analysis, interpretation, and writing manuscript. DD contributed to data collection, interpretation, and writing manuscript. MD contributed to data collection, interpretation, and writing manuscript. CR contributed to development, data collection, interpretation, and writing manuscript.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Chief Scientist Office, Scotland (grant number CGA/19/71) and the Canniesburn Research Trust (no grant number).
Ethics Approval
Ethics approval is obtained from NHS Greater Glasgow and Clyde Research and Development Board.
Trial Registration
Trial Registration number: ClinicalTrial.gov NCT04619030.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
