Protocol for a usability and pilot implementation study of a digital medical device to assess pain in non-verbal people with dementia in Portuguese residential care facilities

Abstract

Objective

People living with moderate to severe dementia (PLWD) are often unable to self-report pain. This matter is of particular concern given that up to 80% experience chronic pain. Mistreated or untreated pain in PLWD is associated with symptoms such as agitation and aggression, and unnecessary use of antipsychotic agents. Further, it can also engender mental burden in formal caregivers. The PainChek^® App, a regulatory cleared class I medical device, enables the assessment and monitoring of pain in people who cannot verbalise it, such as those with moderate to severe dementia. To date there are no data on the real-world use of the PainChek^® App in Portugal. To address this gap, we report the protocol of a pilot study, which combines usability evaluation and implementation research.

Methods

Usability evaluation of the PainChek^® (Portuguese) App will be guided by the ISO framework, focused on effectiveness, efficiency and user satisfaction. Implementation research will combine qualitative interviews to inform the implementation process, a longitudinal study of formal caregivers’ psychological variables, implementation outcomes, plus qualitative interviews to explore the ‘hows’ and ‘whys’ of implementation. The NASSS framework will be used as an implementation framework, together with the COM-B model and the Theoretical Domains Framework.

Results

The usability and implementation studies have received ethics approval from the Egas Moniz Ethics Committee, under numbers 1367 and 64/24, respectively.

Conclusion

This study is expected to inform the scale-up of the PainChek^® (Portuguese) App in real-world settings and establish a foundation for a larger effectiveness and implementation study.

Keywords

Dementia pain assessment usability evaluation implementation science m-health e-health caregivers burden qualitative

Introduction

Context: prevalence and burden of dementia

Dementia is a general term used to designate a range of progressive neurodegenerative conditions that affect memory, other cognitive skills and behaviours. Alzheimer's disease is the most common type of dementia.¹ According to the World Health Organization,¹ there are around 55 million people living with dementia globally. The number of new cases is projected to increase to 153 million by 2050, which represents more than a twofold increase in three decades.² Dementia is the seventh leading cause of death and a leading cause of disability and dependence among older people worldwide.¹

Dementia is one of the diseases with the highest economic burden, leading to significant costs for individuals, families, governments and society. In 2019, the global economic burden of dementia was estimated at USD 1.3 trillion, equivalent to 0.76% of the aggregated global gross domestic product in direct costs.³ By 2030, this figure is expected to reach USD 2.8 trillion.² This magnitude of costs poses a threat to the social and economic development globally and jeopardises the sustainability of health and social services.²

The problem: Unrecognised pain in non-verbal people with dementia

The gold standard for diagnosing and assessing pain is self-report. However, people living with moderate to severe dementia (PLWD) are often unable to explain or self-report to others what is happening to them, due to impaired communication and reasoning skills, rendering them vulnerable to silent pain. This is particularly challenging given that up to 80% of people with dementia suffer from chronic pain.⁴

The problem of unrecognised pain in non-verbal PLWD is compounded by multifaceted barriers. For example, systematic reviews highlighted professionals’ knowledge and training gaps, lack of proactive pain assessment, complexity of pain assessment tools, lack of accessibility to pain assessment guidelines and time constraints.⁵ Further, most pain assessment tools used in dementia lack innovative features to overcome these barriers and continue to depend solely on the assessor's judgement.⁶

Untreated pain is related to symptoms such as agitation and aggression in PLWD, collectively known as behavioural and psychological symptoms of dementia (BPSD).^7,8 These symptoms often lead to the inappropriate use of psychotropic medications, such as atypical antipsychotic agents.⁷ These increase the risk of cardiac and cerebrovascular events and are also linked with increased risks of rapid cognitive decline and mortality.⁹ The increased odds of mortality associated with the off-label use of atypical antipsychotic agents in this population has been confirmed by more recent research.^10,11

Although less explored than for informal caregivers,¹² pain in PLWD also affects the psychological well-being of formal caregivers at a personal and professional level. A study using semi-structured interviews with Chinese nursing assistants is one of the few references that specifically addresses the mental burden experienced by formal carers in managing pain in older adults with dementia. This study suggested that exposure to residents with dementia in pain leads to adverse emotional experiences, including guilt, anxiety and emotional detachment, resulting in diminished enthusiasm for their work and ultimately professional burnout.¹³

As dementia progresses, including the inability to communicate pain verbally, carers’ responsibilities increase, engendering reduced self-esteem, anxiety, exhaustion, depression, sleep problems and social isolation. This emotional overload can, in turn, exacerbate symptoms such as depression and loneliness, as shown by Peavy et al.¹⁴

Emotional exhaustion is a primary manifestation of burnout among carers of PLWD, often aggravated by the negative and demoralising reactions that carers have to the illness. In the study by Shinan-Altman and colleagues,¹⁵ social workers and nurses in residential care facilities and hospitals expressed negative reactions to dementia that they believed contributed to emotional exhaustion. These findings are consistent with other studies demonstrating that higher levels of burnout in healthcare professionals are associated with more pronounced stigmatising beliefs and behaviours.^16,17

The perception of stress is emerging as a significant challenge for both formal and informal carers of people with dementia.¹⁸ Poor mental health among health professionals who work in the long-term care institutions for older adults not only affects their well-being but also compromises the quality of care they provide.¹⁹ Both dissatisfaction and compassion fatigue, resulting from too many tasks, contribute significantly to poor physical health of family caregivers,²⁰ these may be exacerbated in formal carers of PLWD unable to verbally communicate their pain.

Empathy is essential for personalising care and better meeting patients’ needs, serving as a central feature of person-centred practice, as highlighted by a systematic review about empathy training for caregivers of older people.²¹ The benefits of emotional intelligence extend to improving communication with older adults with dementia and reducing anxiety and depression.²² Emotional intelligence and the effective use of skills associated with emotional intelligence in nurses were shown to have a positive effect on the reduction of perceived stress and burnout.²³

Despite the empirical evidence outlined above, further studies are necessary to assess psychological traits among formal carers of PLWD unable to communicate verbally.

Addressing the problem: A novel digital technology to assess pain in non-verbal people with dementia

The PainChek^® system is a software comprised of four components (Figure 1) to assess and monitor pain in people who cannot verbalise it, such as PLWD:

Mobile Application (App);

PainChek^® Portal;

PainChek^® Application Programming Interface (API);

PainChek^® database.

Figure 1.

The PainChek® system. Reprinted with permission from PainChek Ltd.

The PainChek^® point-of-care App is a Class I regulatory-cleared medical device for pain assessment and monitoring in non-verbal adult populations, including individuals with moderate to severe dementia, intellectual disabilities, traumatic brain injury and aphasia. It is approved for use in jurisdictions such as Australia, the United Kingdom, New Zealand, Canada, Singapore and the European Economic Area. The PainChek^® App utilises automated facial detection and analysis technology to identify facial micro-expressions (action units [AUs]), which are validated indicators of the presence of pain (Face Domain). These data are then combined with other non-facial pain indicators inputted by the App user through a series of digital checklists for the remaining five domains (Voice, Movement, Behaviour, Activity and Body). In total, the PainChek^® pain scale is composed of 42 items distributed across these six domains.²⁴ The App automatically computes a numerical pain score and assigns an intensity: no pain (0–6), mild pain, (7–11), moderate pain (12–15), or severe pain (≥16).²⁴

The incorporation of automated facial analysis into the PainChek^® based on the Facial Action Coding System is designed to address the issue of observer bias. This can arise, particularly amongst healthcare providers who can get desensitised to facial expressions of pain.²⁵and the fact that pain assessment tools with anatomically defined facial features have better psychometric properties than those using a vague facial descriptor. Further, the Facial Action Coding System has been shown to be a valid method of assessing pain.²⁶ However, given 10%‒23% of people with significant pain display no facial expressions,^27,28 assessment of facial expressions alone has its limitations. To address this the PainChek^® tool also includes non-facial pain indicators across five domains. These include items from the American Geriatric Society list of behaviours associated in pain in people with cognitive impairment.²⁹ The user is provided with in-App definitions of each of these items to promote consistency and reduce inter-rater variations. Lastly, a binary rather than ordinal scoring system has been adopted to reduce subjectivity. The automated summation of scores and assignment of pain intensities diminishes errors, whilst the multidimensional construct of the PainChek^® tool provides a granular profile of the resident's pain.

The PainChek^® Portal, also known as the Web Admin Portal, is a secure website that allows administrators to manage patient data and activate new users. The portal is currently cloud hosted on Amazon Webs Services using the Amazon Elastic Compute Cloud (Amazon EC2).³⁰

The digital capture of pain assessment results provides evidence of pain assessment practices, whilst integration in care management systems allows reduced data entry time and real-time communication.

PainChek^® has now been used in clinical practice in Australia, New Zealand, Singapore, the United Kingdom and Canada. Over 5 million pain assessments have been conducted using the PainChek^® system.³¹

Identified knowledge gaps

Recently, the PainChek^® App and the App user guide have been translated to European Portuguese, as part of an on-going collaboration between PainChek Ltd and Egas Moniz School of Health & Science. To ensure the accuracy of the Portuguese version of the PainChek^® App, a forward and backward translation cycle was executed by a professional translation service commissioned by PainChek Ltd. This process involved translating the App content from English to Portuguese and then back to English. Any discrepancies that arose between the original and back translated versions were discussed among team members external to PainChek (CR, TA, BS and MPG), with a focus on linguistic nuances in European Portuguese and appropriateness for the national context. Based on these discussions, suggestions for modifications were made and subsequently reviewed by the PainChek Ltd. The PainChek (Portuguese) App was then developed. The interfaces were checked by team members external to PainChek (e.g., BS and IBF) to address potential usability issues.

The usability of the Portuguese version of the App has not yet been evaluated. Further, the PainChek^® system has not yet been introduced in continental Europe. Its use in routine practice is expected to reduce issues related to unrecognised pain in PLWD, such as behavioural problems and poor quality of life. In turn, this is expected to have a positive impact on formal caregivers’ psychological variables, such as burnout, perceived stress and cognitive and affective empathy.

Knowledge on the real-world use of the PainChek^® (Portuguese) App in Portugal and its potential associations with caregivers’ psychological variables is essential for guiding the adoption and scale-up of this technology.

Aim

The research question driving the study is ‘To what extent is the implementation of the PainChek^® (Portuguese) App feasible and acceptable for assessing pain in non-verbal PLWD in Portuguese residential care facilities and what are the key factors influencing adoption and scale-up?’. This research question was converted into the following aim: to evaluate the pilot-scale implementation of the PainChek^® (Portuguese) app to assess pain in non-verbal PLWD in residential care facilities. The aim was subsequently broken down into the following specific objectives (SO):

SO1: To evaluate the usability of the PainChek^® (Portuguese) App;

SO2: To explore individual, organisational and system factors influencing the implementation of PainChek^® (Portuguese) App in non-verbal PLWD in Portuguese residential care facilities;

SO3: To assess the adoption, feasibility, acceptability and fidelity of the PainChek^® (Portuguese) App in non-verbal PLWD in residential care facilities in Portugal;

SO4: To assess psychological factors of formal caregivers involved in pain assessment using PainChek^® (Portuguese) App in residential care facilities in Portugal;

SO5: To explore formal caregivers’ perspectives on using PainChek^® (Portuguese) App in terms of acceptability and experienced barriers and facilitators in this setting.

Methods

To address identified knowledge gaps, we will combine usability evaluation and implementation research.

Usability evaluation of the European Portuguese version of the PainChek^® App will be guided by the ISO framework.³² According to these standards, usability refers to a product or service that can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.³²

The implementation component will resort to a quantitative longitudinal study of formal carers’ psychological variables, implementation outcomes and qualitative semi-structured interviews. The Non-adoption, Abandonment, Scale-up, Spread and Sustainability (NASSS) framework will be used as an implementation framework,³³ combined with the COM-B³⁴ model and the Theoretical Domains Framework (TDF).³⁵ Semi-structured interviews were deemed appropriate as we are seeking to explore people's perceptions and opinions in a topic where there is potentially a low level of awareness, as outlined by Kallio’s framework for qualitative semi-structured interview guides.³⁶

The overall study will be carried out in four sequential stages, depicted in Figure 2.

Figure 2.

Stages of the study.

Setting

Two to four residential care facilities will be purposively sampled, utilising criteria such as contacts within the research team's network and diversity in facility types. We will attempt to include settings from private high-end establishments to facilities operated by non-governmental organisations. The number of study sites will be determined by their respective sizes, which directly impact the staffing levels and the number of formal caregivers available for recruitment.

The list of study sites will be made available in the Open Science Framework (OSF) project repository.

Recruitment

Each selected facility will be contacted to present the study, its objectives and timeline, clarify the expected roles and discuss the potential benefits and requirements of their participation. The commitment of residential care facilities, hereafter referred to as study sites, will be formalised through letters of acceptance, outlining their roles and responsibilities within the project and benefits associated with participation (e.g., training).

Onboarding of residential care facilities and recruitment of formal caregivers will be made in liaison with a designated contact person in each study site. Formal caregivers include healthcare professionals (e.g., nurses, physicians, physiotherapists, psychologists) and health assistants involved in the care of non-verbal PLWD in this setting. The latter are often the main caregivers and interact with residents the most. Since residential care facilities are integrated into the social system, it is uncommon to have an on-site physician.

Recruitment for Stages 1 to 4 will occur separately, allowing formal caregivers the flexibility to choose whether to participate in each study stage. Formal caregivers interested in participating will be provided with the Information for Participants Sheet and will have the opportunity to have all their questions answered. Upon agreeing to participate, they will be asked to provide written informed consent.

The consent form and all related documentation provided to participants will be available in the project OSF repository (https://osf.io/u87rw/).

Ethical considerations

The study will be conducted following the ethical principles outlined in the Declaration of Helsinki.³⁷ The implementation study has received ethics approval from the Egas Moniz Ethics Committee, under number 64/24. The usability study was approved by the same institutional ethics committee, under the internal number 1367.

The study protocol (v.1, 2024.08.10) may be subject to modifications as the study progresses; an audit trail documenting changes with corresponding dates will be maintained. Any protocol amendments will be reported to the Egas Moniz Ethics Committee and detailed in subsequent study reports.

Data management and privacy

Upon obtaining consent, each participant will be assigned a unique identifier, a critical step in the process of pseudo-anonymisation, which ensures privacy while allowing for the integration of participants’ data. This unique identifier will be the cornerstone for creating a central database that collates all participants’ data.

Data will be securely stored in password-protected computers. To reduce any risks of identification, a single researcher will be responsible for assigning unique identifiers to participants and holding the links between these identifiers and the participants’ names.

Research stages

Stage 1: Pre-implementation study (T0)

This stage of the pilot study addresses SO2 and SO4; it will take place in all study sites. It consists of qualitative data collection to design the implementation in the study sites and a baseline measurement of formal carers’ psychological variables.

Sampling and recruitment of participants

For the qualitative interviews, we will purposively sample formal caregivers of PLWD in all study sites to ensure maximum variability in respect to key characteristics (e.g., professional group). Eligibility criteria include being over 18 years old, ability to read, understand and write in the Portuguese language and working with older non-verbal PLWD. Furthermore, residential care facilities managers will be invited to participate.

Drawing from a systematic review on empirical tests for saturation,³⁸ and given the heterogeneous nature of the sample, we estimate conducting around 17 interviews with formal caregivers. Sample size will be confirmed during data collection and analysis, guided by thematic saturation, following the approach suggested by Guest et al.³⁹

Eligibility criteria to participate in the baseline measurement of formal carers’ psychological variables mirror those for the qualitative interviews. Sample size was not estimated given the exploratory nature of the study.

Data collection

Kallio's framework was chosen to guide the development of the interview guides.³⁶ This process includes retrieving and using previous knowledge,^5,40 creating a preliminary guide with main topics and pre-designed follow-up questions in the form of prompts,⁴¹ conducting pilot tests through expert evaluation and presenting the final topic guide in the study report. One of the topic guides will explore experiences and perspectives of formal caregivers in assessing pain in non-verbal PLWD, as well as barriers and facilitators to using the PainChek^® App, informed by the COM-B³⁴ model and the TDF.³⁵ This topic guide will serve as the foundation for the data collection instrument for residential care facility managers, which will have a greater emphasis on organisational aspects.

An online questionnaire via Qualtrics^XM will be used to measure a set of psychological variables (perceived stress, burnout, compassion fatigue, empathy, emotional intelligence), using the validated Portuguese versions of the following instruments: The Maslach Burnout Inventory (MBI),⁴² The Perceived Stress Scale (PSS-10),⁴³ the Professional Quality of Life Scale-Compassion Satisfaction and Compassion Fatigue (ProQOL);⁴⁴ the Questionnaire of Cognitive and Affective Empathy (QCAE)⁴⁵ and the Wong and Law Emotional Intelligence Scale (WLEIS).⁴⁶

Data collection forms will be publicly accessible in the project's OSF repository (https://osf.io/u87rw/).

Remote or in-person interview sessions will be scheduled based on individual availability of participants. If remote, interviews will be conducted through videoconferencing facilities, such as Microsoft Teams or Zoom, and recorded with consent. Following the collection of interview data, participants will be asked to complete an online questionnaire focusing on psychological variables.

Data management and analysis

Interview recordings will be transcribed verbatim, anonymised for dates and places and identified by the participant's unique identifier code. Pseudo-anonymous verbatim transcripts will be thematically analysed with the aid of the MAXQDA software, resorting to the COM-B³⁴ model and the TDF³⁵ as analytical frameworks.

Analysis will be an iterative process starting from the initial interview, conducted by two analysts from the research team, at least one of whom with a proven track record in qualitative data analysis. Firstly, thematic categories (i.e., codes) will be developed based on the topic guide and explicitly defined. Upon repeatedly reading the transcripts, new codes will be added based on data. These codes will be continuously refined as more interviews are conducted, employing constant comparison. This involves comparing textual data across the dataset to establish codes⁴⁷ and scrutinising the analytical idea of each code against the textual data indexed to it, to ensure that the interpretations are solidly grounded in the data.⁴⁸

Safeguards to ensure trustworthiness in data analysis will include discussing the transcript of each interview within the research team, ideally before collecting additional data and independent coding of part of the dataset by a third analyst from the research team with proven track record in qualitative data analysis. Furthermore, an audit trail,⁴⁹ including contextual, methodological and analytical documentation, will be maintained to allow retroactive assessment of the research conduct.

Descriptive analysis will be performed for socio-demographic and psychological variables with the aid of IBM SPSS Statistics, version 29.0.

Stage 2: Usability study

This stage, addressing SO1, will take place in either a subset or all study sites, contingent on achieving the anticipated sample size. As depicted in Figure 2, the usability study is positioned within Stage 2 of the timeline and will proceed once baseline data collection is finished (Stage 1, T0) and the formal caregivers have undergone their training for using the PainChek^® (Portuguese) App.

Regardless of their participation in Stage 1, all formal caregivers in each study site will be invited to receive training on the PainChek^® App (Box 1). Training of formal caregivers will be online and self-paced, with the duration of 30‒45 min.

Box 1.

PainChek^® App training modules.

Introduction to PainChek^®;

Getting started;

The PainChek^® assessment;

Putting PainChek^® into practice;

Manual facial analysis;

Review the participant's learning.

Sampling and recruitment of formal caregivers

We will purposively sample formal caregivers of PLWD in recruited study sites that are within geographical reach of team members, as these data will be collected in-person. A sample of 17 participants was estimated based on the following assumptions: completion rate for the population of at least 70%, 1 failure, 95% confidence interval. A sample of this magnitude is expected to discover 85% of the usability problems, assuming the probability of problem occurrence is 0.1.⁵⁰ Our experience suggests that a sample of this magnitude also maximises variability in sociodemographic characteristics and yields rich suggestions and insights.⁵¹

Formal caregivers will be eligible if they meet the following criteria: they can speak and read Portuguese, are over 18 years old, have access to a smartphone, have been working with non-verbal PLWD for longer than 3 months and have received training on the PainChek^® App.

Data collection

Concurrent think-aloud will be used for evaluating the effectiveness and efficiency dimensions of usability of the PainChek^® App,^52,53 through success in task completion and completion time. A written protocol will be piloted on two individuals, who are not part of the study participants, to ascertain the clarity and comprehensibility of the procedure, gauge time for completion and introduce improvements based on the feedback and data gathered.

The satisfaction dimension of usability will be assessed through an online questionnaire deployed in Qualtrics^XM, comprising the Portuguese version of the Post-Study System Usability Questionnaire (PSSUQ-PT)⁵⁴ and an open question for comments or suggestions. PSSUQ-PT encompasses 19 questions, on system usefulness (SYSUSE, Items 1‒8), information Quality (INFOQUAL, Items 9‒15), interface quality (INTERQUAL, Items 16‒18) and an overall question on satisfaction (Item 19). This instrument uses a 7-point Likert scale, from strongly agree (1) to strongly disagree (7) and provides participants with the option to select ‘Not applicable’ (NA) when relevant. Consequently, a lower score indicates a more favourable evaluation. Reasons for choosing PSSUQ-PT include possessing a significant correlation with widely recognised instruments, such as the System Usability Scale (SUS), the availability of a validated version in European Portuguese and offering a more straightforward interpretation of scores compared to the SUS.

Data collection forms will be publicly accessible in the project's OSF repository (https://osf.io/u87rw/).

Face-to-face sessions with participants will be scheduled based on individual availability. Participants will be asked to perform selected tasks in the App with a preselected ‘training resident’, using vignettes depicting a scenario involving pain in a non-verbal PLWD. This will allow participants to fully browse the PainChek^® App and access its features without involving real residents and their personal and clinical data. Think aloud during task performance will be video recorded.

At the end of the concurrent think aloud protocol, participants will be invited to complete the online questionnaire previously described.

Data management and analysis

Textual data from the think-aloud protocol (recording and notes) plus open questions will be coded into unique usability problems or suggestions within an excel spreadsheet and will be represented in user by problem/suggestion matrices.

Data on effectiveness (tasks completion rate) and efficiency (time to complete the tasks) will be computed from the session recordings and stored in an excel file.

The coding of textual data and calculation of effectiveness and efficiency metrics will undergo independent verification by a second analyst within the research team, with a proven track record in usability evaluation. This verification may involve either a subset or the entirety of the data, contingent upon the agreement of initial findings.

Quantitative data (tasks completion rate, time to complete the tasks and PSSUQ-PT scores) will be subjected to descriptive statistical analysis with the aid of Statistical Package for Social Sciences (IBM SPSS Statistics, version 29.0).

Stage 3: Implementation

This component of the pilot study, addressing SO3, will take place in all study sites. It consists of quantitative data collection regarding the use of the PainChek^® App in study sites.

Sampling and recruitment of participants

Trained formal caregivers will be invited to use the PainChek^® (Portuguese) App in their daily practice to assess pain in non-verbal people living with moderate to severe dementia.

Data collection

Data for calculating the following implementation outcomes, based on Proctors’ taxonomy,⁵⁵ will be gathered over a period of up to 4 months:

Adoption (i.e., intervention uptake): recruitment rate (percentage of formal caregivers accepting to use the PainChek^® App); percentage of formal caregivers who completed the initial training, percentage of formal caregivers using PainChek^® for pain assessment in non-verbal PLWD;

Feasibility (i.e., suitability for everyday use): total number of PainChek^® assessments in non-verbal PLWD, online Feasibility of Intervention Measure;⁵⁶

Acceptability (i.e., satisfaction with the intervention): online Acceptability of Intervention Measure;⁵⁶

Fidelity (i.e., intervention delivered as intended): percentage of formal carers using appropriate functions in the PainChek^® App (e.g., pain assessments in rest and post movement).

Data will be collected from a variety of sources, including residential care facilities’ records by trained and authorised staff and surveys (Feasibility of Intervention Measure and Acceptability of Intervention Measure).

Data collection forms will be publicly accessible in the project's OSF repository (https://osf.io/u87rw/).

Data management and analysis

Administrative data from care facilities records will be delivered for analysis in a fully anonymised and aggregated format. The data will then be cleaned to remove any inconsistencies.

Data will be subjected to descriptive statistical analysis with the aid of IBM SPSS Statistics, version 29.0.

Stage 4: Post-implementation (T1)

This component of the pilot study, addressing SO4 and SO5, will take place in all study sites. It consists of a T1 measurement of formal carers’ psychological variables and a qualitative study, to explore the ‘hows’ and ‘whys’ of implementation.

Sampling and recruitment of participants

Formal caregivers of PLWD in all study sites will be purposively sampled for the qualitive interviews, using the same eligibility criteria as for T0, plus having completed a meaningful number of PainChek^® assessments during Stage 3. Residential care facilities managers will also be invited to participate. Predictably managers and some formal caregivers will be the same individuals previously interviewed at T0.

As presented for the T0 qualitative interviews, we plan to conduct approximately 17 interviews with formal caregivers and adjust the final sample based on thematic saturation during data collection and analysis.

Eligibility criteria for the T1 measurement of formal caregivers’ psychological variables are the same as for the qualitative interviews, with a key distinction: to enable the assessment of trends over time only those who completed the questionnaire at T0 will be invited to participate.

Data collection

The same approach as described for T0 will be used to develop the semi-structured interview guides for formal carers and residential care facility managers. The former will explore experiences of formal caregivers of using the PainChek^® (Portuguese) App in non-verbal PLWD, its acceptability, as well as barriers and facilitators to its use; the latter will have a more organisational focus.

Remote or in-person interviews will be scheduled based on individual availability of participants. As for T0, interviews will be recorded with consent.

Following the collection of interview data, participants will be asked to complete the same questionnaire described in T0 through Qualtrics^XM.

Data collection forms will be publicly accessible in the project's OSF repository (https://osf.io/u87rw/).

Data management and analysis

As outlined in the T0 qualitative data analysis section, our approach will involve thematic analysis of anonymised verbatim transcriptions, with the aid of the MAXQDA software, resorting to the COM-B³⁴ model and the TDF³⁵ as analytical frameworks.

For quantitative data, descriptive analysis will be performed for socio-demographic and psychological variables. If appropriate, we will use paired samples t-test for comparisons between T0 and T1 for each psychological variable, if parametric tests can be used. If the assumptions for data normality are not met, a non-parametric test will be used (Wilcoxon test). Quantitative data will be analysed with statistical software Statistical Package for Social Sciences (IBM SPSS Statistics, version 29.0, IBM Corp, Armonk, NY). All statistical analyses will be performed for a significance level of 0.05 (α ≤ 0.05).

Discussion

To address the research question driving the pilot study—‘To what extent is the implementation of the Portuguese version of the PainChek^® App feasible and acceptable for assessing pain in non-verbal people with dementia in residential care facilities and what are the key factors influencing adoption and scale-up?’—we developed a protocol combining usability evaluation and implementation research. A key strength of our protocol is the robustness of the approach and methods chosen.

By using the ISO framework for usability evaluation, focusing on effectiveness, efficiency and user satisfaction, we are aligned with best practices in technology assessment.^58,59 Further, we will use both inquiry and test methods, resorting to research supported techniques, such as validated questionnaires, think aloud and task performance. For example, think-aloud protocols have proved to be one the most effective techniques to elicit usability issues,⁵⁹ by identifying the cognitive behaviour of performing tasks while using technology and determining how information is used to solve problems. These protocols are generally categorised into concurrent and retrospective.^52,53 In the former, users are asked to think and talk aloud at the same time while performing previously set tasks. In a retrospective think-aloud protocol, users are asked to recall what they were thinking during the process. Both approaches can provide comprehensive insights into the problems that users encounter in their interaction with the technology,^53,60 but immediate verbalisations are believed to describe participants’ thoughts more accurately than retrospective verbalisation.^52,61

We will report the evaluation of usability based on a checklist on the terms and procedures for usability of health-related digital solutions,⁵⁷ contributing to a more transparent and standardised reporting.

Employing both qualitative and quantitative methods enables a detailed understanding of implementation. Initial qualitative interviews with formal caregivers and residential care facility managers, conducted before the use of the digital technology in practice, will inform the development of the implementation process. Subsequent qualitative interviews, following real-world use of the PainChek^® (Portuguese) App, will serve to identify barriers and facilitators and to shed light on quantitative findings, such as those related to feasibility. The incorporation of the NASSS framework, the COM-B model and the TDF in prompts a deeper consideration of aspects affecting implementation that might otherwise be neglected. The NASSS framework is gaining popularity as a framework for implementing health and care technologies; its prospective use will aid the design of the implementation process, as well as data interpretation.⁶² The COM-B model offers a lens for the capabilities, opportunities and motivations influencing formal caregivers’ behaviours in using PainChek^®, whilst the TDF provides a more granular view across a range of domains, such as knowledge, skills, social influences and environmental context and resources. Together, these frameworks offer a structured approach to understanding how various elements interact to either support or hinder the implementation of the PainChek^® App in residential care facilities.

Research on psychological variables in formal caregivers of non-verbal PLWD remains limited. Our research turns to this untapped topic, by examining perceived stress, burnout, compassion fatigue, empathy and emotional intelligence among healthcare professionals and other staff in residential care facilities. The potential positive impact of using the PainChek^® App for pain assessment by formal caregivers can have significant practical implications. For instance, reducing perceived stress, burnout and emotional exhaustion among these caregivers may lead to improved care for non-verbal PLWD.

The current study specifically focuses on residential care facilities, where the users of the PainChek^® App are the staff members responsible for caring for residents. Therefore, they will be enrolled to evaluate usability and implementation processes and to identify barriers and facilitators within this setting. However, we recognise the importance of including the perspectives of other stakeholders. Whilst enrolling people with moderate to severe dementia may be challenging, due to their cognitive limitations and legal considerations for obtaining consent, we will address the perspectives of family members and other informal carers in a separate study, dedicated to a usability evaluation in this group.

This work builds upon previous evaluations of the implementation of PainChek^® both as part of a project funded by the Australian Government to accelerate PainChek^® updake in nursing homes across Australia,⁶³ research evaluating the clinical impact as part of a multifaceted intervention conducted in a UK aged care setting⁶⁴ and research conducted by Pu et al.⁶⁵ in a single Australian nursing home. Unlike those studies, this study will adopt an implementation science approach and will explore the potential effects of introducing the PainChek^® App on staff's psychological variables.

Our study design is not without limitations. One limitation is the limited number of purposively selected residential care facilities. This may affect the generalisability of our findings, as the selected sample is unlikely to represent the broader population of residential care facilities in Portugal. Within the constraints of a pilot study, we will attempt to have a diverse representation of facilities, potentially offering insights applicable to a broader array of environments.

Given the exploratory nature of this study, the sample of formal caregivers is likely not powered to detect statistically significant differences in psychological variables between the two time points (T0 and T1). Instead, the focus is on identifying potential trends. Despite the limitation in statistical power, this approach can still provide valuable preliminary insights to guide future, more definitive research. Moreover, the absence of a control group in our study design means that we are unable to determine whether potential changes are attributable to the adoption of the PainChek^® App or to external factors or variations over time. Our pilot study lays the groundwork for a future study with a larger sample and a more robust design. Such study should include pain management outcomes associated with the adoption of PainChek^®. For example, as part of the Reconnect programme, established by Orchard Care Homes (UK), PainChek^® use resulted in an increase in documented pain assessments, improvements in pain management and reduction in the use and dosage of psychotropic medications.⁶⁴

We plan to disseminate the study results to participants, healthcare professionals, the public and other stakeholders through presentations in scientific events, academic publications and data-sharing via the OSF repository.

Conclusion

The PainChek^® App provides a more objective means of pain assessment for non-verbal PLWD using artificial intelligence and smart automation. Automated facial analysis diminishes user bias, while observer-based digital documentation of the presence of well-defined non-facial features aims to reduce inter-rater variability. Digital collection of pain assessment results and integration with resident care systems contributes to streamlining the care process and humanising care for this vulnerable population. By addressing silent pain, unnecessary suffering is expected to be reduced and care efficiency enhanced.

Benefits associated with the PainChek^® App, which warrant further research, can only be realised through successful adoption of this technology. This entails identifying potential barriers to adoption and outlining strategies to mitigate them. This project uniquely contributes by exploring these aspects in continental Europe for the first time, paving the way for wider adoption in Portugal and other European countries.

Footnotes

Acknowledgements

Then authors thank Beatriz Marques, Mariana Carvalho and Ricardo Carregosa for their contribution to the application of the Angelini University Award 2022.

Contributorship

Mara Pereira Guerreiro, Kreshnik Hoti, Jeff Hughes and Catarina Ramos were involved in the conceptualisation of the study, along with Beatriz Dias and Teresa Andrade. Methodology development was a collaborative effort by all authors. Project administration was managed by Mara Pereira Guerreiro, Isa Brito Félix and Catarina Ramos. Supervision of the research activities was undertaken by Mara Pereira Guerreiro, Isa Brito Félix, Kreshnik Hoti and Jeff Hughes. Mara Pereira Guerreiro, Isa Brito Félix, Morgane Tomé, Kreshnik Hoti, Jeff Hughes and Catarina Ramos contributed to writing the original draft of the manuscript. All authors were involved in the review and editing process. There was no use of professional writers in the preparation of this manuscript and there are no plans to involve professional writers in future work on this project.

Declaration of conflicting interest

JH and KH are shareholders in PainChek Ltd (formerly known as EPAT Technologies Ltd), which is commercialising the PainChek App. They are also named as co-inventors with Mustafa Atee on the patent entitled ‘A pain assessment method and system’. This patent has been granted in the United States, Europe, China and Japan. JH is employed as the Chief Scientific Officer of PainChek Ltd and holds an Emeritus Professor appointment at the Curtin Medical School, Curtin University. KH is employed as a consultant by PainChek Ltd. and he is a Professor at the University of Prishtina. The remaining authors declared no potential conflicts of interest with respect to the research, authorship and publication of this article.

Ethical approval

The implementation study has received ethics approval from the Egas Moniz Ethics Committee, under numbers 1367 and 64/24.

Funding

This work is supported by the Angelini University Award 2022 and a Research Grant 2024 from the Egas Moniz Center for Interdisciplinary Research (CiiEM). Funders were not involved in the study design and will not be involved in the collection, management, analysis and interpretation of data, writing of the study report and the decision to submit the report for publication. Funders will not have ultimate authority over any of these activities. PainChek Ltd covered the article processing charges.

Informed consent

Participants will sign written informed consent.

Data availability

Data collection instruments, anonymised data and related research materials will be made available for public sharing in the OSF repository (https://osf.io/u87rw/), to ensure transparency and facilitate further research.

ORCID iDs

Mara Pereira Guerreiro

Isa Brito Félix

References

WHO. Dementia. https://www.who.int/news-room/fact-sheets/detail/dementia (2023, accessed 4 February 2024).

Long

Benoist

Weidner

. World Alzheimer Report 2023: reducing dementia risk: Never too early, never too late. https://www.alzint.org/resource/world-alzheimer-report-2023/ (2023, accessed 14 December 2023).

Wimo

Seeher

Cataldi

, et al. The worldwide costs of dementia in 2019. Alzheimer’s Dementia 2023; 19: 2865–2873.

Achterberg

Lautenbacher

Husebo

, et al. Pain in dementia. Pain Rep 2019; 5: E803.

Liao

Jao

Berish

, et al. A systematic review of barriers and facilitators of pain management in persons with dementia. J Pain 2023; 24: 730–741.

Deng

. Selection and use of pain assessment tools for patients with Alzheimer’s disease. Int J Clin Exp Med Res 2023; 7: 657–662.

Pergolizzi

Raffa

Paladini

, et al. Treating pain in patients with dementia and the possible concomitant relief of symptoms of agitation. Pain Manag 2019; 9: 569–582.

Atee

Morris

Macfarlane

, et al. Pain in dementia: prevalence and association with neuropsychiatric behaviors. J Pain Symptom Manage 2021; 61: 1215–1226.

Schneider

Dagerman

Insel

. Risk of death with atypical antipsychotic drug treatment for dementia: meta-analysis of randomized placebo-controlled trials. JAMA 2005; 294: 1934–1943.

10.

Yunusa

Rashid

Demos

, et al. Comparative outcomes of commonly used off-label atypical antipsychotics in the treatment of dementia-related psychosis: a network meta-analysis. Adv Ther 2022; 39: 1993–2008.

11.

Phiri

Engelthaler

Carr

, et al. Associated mortality risk of atypical antipsychotic medication in individuals with dementia. World J Psychiatry 2022; 12: 298–307.

12.

Regier

Taylor

Szanton

, et al. Pain in persons with dementia and the direct and indirect impacts on caregiver burden. Geriatr Nurs (Minneap) 2021; 42: 366–371.

13.

Yang

Wang

, et al. Perception, behavior and experience of nursing assistants towards pain of older adults with dementia: a qualitative study. Geriatr Nurs 2024; 56: 100–107.

14.

Peavy

Mayo

Avalos

, et al. Perceived stress in older dementia caregivers: mediation by loneliness and depression. Am J Alzheimers Dis Other Demen 2022; 37: 1–8.

15.

Shinan-Altman

Werner

Cohen

. The connection between illness representations of Alzheimer’s disease and burnout among social workers and nurses in nursing homes and hospitals: a mixed-methods investigation. Aging Ment Health 2016; 20: 352–361.

16.

Kane

Murphy

Kelly

. Assessing implicit and explicit dementia stigma in young adults and care-workers. Dementia 2020; 19: 1692–1711.

17.

Mantzorou

Economou

Kalokerinou

, et al. Exploration of the relationship between stigma and burnout among Greek nurses in dementia care. J Nurs Manag 2020; 28: 615–624.

18.

Pinyopornpanish

Soontornpun

, et al. Perceived stress and depressive symptoms not neuropsychiatric symptoms predict caregiver burden in Alzheimer’s disease: a cross-sectional study. BMC Geriatr 2021; 21: 1–10.

19.

Borges

Julia De Lima Bomfim

Hortes

, et al. Depression and empathy in health professionals who work in the long-term care institutions for older adults. Dement Neuropsychol 2021; 15: 381–386.

20.

Schulz

Sherwood

. Physical and mental health effects of family caregiving. J Soc Work Educ 2008; 44: 105–113.

21.

Maximiano-Barreto

de Morais Fabrício

de Lima Bomfim

, et al. Psychological concerns associated with empathy in paid and unpaid caregivers of older people: a systematic review. Clin Gerontol 2024; 47(5): 716–729. doi:https://doi.org/10.1080/07317115.2022.2090879

22.

Weaving

Orgeta

Orrell

, et al. Predicting anxiety in carers of people with dementia: the role of trait emotional intelligence. Int Psychogeriatr 2014; 26: 1201–1209.

23.

Pérez-Fuentes

MdC

Molero-Jurado

MdM

Gázquez-Linares

, et al. Analysis of burnout predictors in nursing: risk and protective psychological factors. Eur J Psychol Appl Legal Context 2019; 11: 33–40.

24.

Atee

Hoti

Hughes

. A technical note on the PainChek^TM system: a web portal and mobile medical device for assessing pain in people with dementia. Front Aging Neurosci 2018; 10: 352481.

25.

Prkachin

. Facial pain expression. Pain Manag 2011; 1: 367–376.

26.

Erin Browne

Hadjistavropoulos

Prkachin

, et al. Pain expressions in dementia: validity of observers’ pain judgments as a function of angle of observation. J Nonverbal Behav 2019; 43: 309.

27.

Kunz

Prkachin

Solomon

, et al. Faces of clinical pain: inter-individual facial activity patterns in shoulder pain patients. Eur J Pain 2021; 25: 529–540.

28.

Kunz

Lautenbacher

. The faces of pain: a cluster analysis of individual differences in facial activity patterns of pain. Eur J Pain 2014; 18: 813–823.

29.

American geriatrics society panel on the pharmacological management of persistent pain in older persons R with permission from TAGSocietyO article appears in JAGS 2009;57:1331–1346. Pharmacological management of persistent pain in older persons. Pain Medicine 2009; 10: 1062–1083.

30.

AWS. Cloud compute capacity ‒ Amazon EC2 ‒ AWS, https://aws.amazon.com/ec2/ (2017, accessed 11 March 2024).

31.

PainChek Ltd (ASX: PCK). 31 December 2023 Quarterly Update and Appendix 4C. Sydney, 31 December 2023.

32.

ISO 9241-11:2018(en). Ergonomics of human-system interaction—part 11: usability: Definitions and concepts, https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en (accessed 25 April 2024).

33.

Greenhalgh

Wherton

Papoutsi

, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res 2017; 19: e8775.

34.

Michie

van Stralen

West

. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci 2011; 6: 1–12.

35.

Cane

O’Connor

Michie

. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci 2012; 7: 1–17.

36.

Kallio

Pietilä

Johnson

, et al. Systematic methodological review: developing a framework for a qualitative semi-structured interview guide. J Adv Nurs 2016; 72: 2954–2965.

37.

World Medical Association. World Medical Association declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA 2013; 310: 2191–2194.

38.

Hennink

Kaiser

. Sample sizes for saturation in qualitative research: a systematic review of empirical tests. Soc Sci Med 2022; 292: 114523.

39.

Guest

Namey

Chen

. A simple method to assess and report thematic saturation in qualitative research. PLoS One 2020; 15: e0232076.

40.

Smith

Lockey

Johnson

, et al. Pain management for people with dementia: a cross-setting systematic review and meta-ethnography. Br J Pain 2023; 17: 6–22.

41.

Robinson

. Probing in qualitative research interviews: theory and practice. Qual Res Psychol 2023; 20: 382–397.

42.

Maslach

Jackson

Leiter

. Maslach burnout inventory. In: Evaluating stress: a book of resources. 3rd ed. Lanham, Maryland: Scarecrow Education, 1997, pp.191–218.

43.

Cohen

Kamarck

Mermelstein

. A global measure of perceived stress. J Health Soc Behav 1983; 24: 385–396.

44.

Stamm

. Professional quality of life: Compassion satisfaction and Fatigue Version 5 (ProQOL), https://proqol.org/ (2009, accessed 10 March 2024).

45.

Reniers

RLEP

Corcoran

Drake

, et al. The QCAE: a questionnaire of cognitive and affective empathy. J Pers Assess 2011; 93: 84–95.

46.

Wong

Law

. The effects of leader and follower emotional intelligence on performance and attitude: an exploratory study. Leadersh Q 2002; 13: 243–274.

47.

Pope

. Qualitative research in health care: analysing qualitative data. Br Med J 2000; 320: 114–116.

48.

Gibbs

. Qualitative data analysis: Explorations with NVivo. Oxon, UK: Open University Press, 2002.

49.

Rodgers

Cowles K

. The qualitative research audit trail: a complex collection of documentation. Res Nurs Health 1993; 16: 219–226.

50.

Calculators – MeasuringU. https://measuringu.com/calc/ (accessed 13 June 2023).

51.

Balsa

Félix

Cláudio

, et al. Usability of an intelligent virtual assistant for promoting behavior change and self-care in older people with type 2 diabetes. J Med Syst 2020; 44: 1–12.

52.

Fonteyn

Kuipers

Grobe

. A description of think aloud method and protocol analysis. Qual Health Res 1993; 3: 430–441.

53.

Peute

LWP

de Keizer

Jaspers

MWM

. The value of retrospective and concurrent think aloud in formative usability testing of a physician data query tool. J Biomed Inform 2015; 55: 1–10.

54.

Rosa

Martins

Costa

, et al. European Portuguese validation of the post-study system usability questionnaire (PSSUQ). In: 2015 10th Iberian Conference on Information Systems and Technologies, CISTI 2015. June 17–20, 2015, Agueda, PORTUGAL, 2015. doi: https://doi.org/10.1109/CISTI.2015.7170431.

55.

Proctor

Silmere

Raghavan

, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011; 38: 65–76.

56.

Weiner

Lewis

Stanick

, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci 2017; 12: 1–12.

57.

Martins

Santinha

Almeida

, et al. Consensus on the terms and procedures for planning and reporting a usability evaluation of health-related digital solutions: delphi study and a resulting checklist. J Med Internet Res 2023; 25: e44326.

58.

Maramba

Chatterjee

Newman

. Methods of usability testing in the development of eHealth applications: a scoping review. Int J Med Inform 2019; 126: 95–104.

59.

Broekhuis

van Velsen

Hermens

. Assessing usability of eHealth technology: a comparison of usability benchmarking instruments. Int J Med Inform 2019; 128: 24–31.

60.

Georgsson

Staggers

. An evaluation of patients’ experienced usability of a diabetes mHealth system using a multi-method approach. J Biomed Inform 2016; 59: 115–129.

61.

Perez

Nick

Mateo

, et al. Adapting a weight management tool for Latina women: a usability study of the veteran health administration’s MOVE!23 tool. BMC Med Inform Decis Mak 2016; 16: 1–12.

62.

Shin

Hamovitch

Gatov

, et al. The NASSS (Non-adoption, abandonment, scale-up, spread and sustainability) framework use over time: a scoping review. medRxiv 2023; 2023.11.22.23298897.

63.

Daffas

. KPMG Report Finds National PainChek® Rollout Delivers Significant Positive Outcomes for the Australian Aged Care Sector | PainChek. https://www.painchek.com/latest-news/kpmg-report-finds-national-painchek-rollout-delivers-significant-positive-outcomes-for-the-australian-aged-care-sector/ (2023, accessed 30 October 2024).

64.

Baird

Miller

Hoti

, et al. Clinical impact of a multifaceted intervention aimed at decreasing distress in people living with dementia: evaluating the reconnect program. Front Psychiatry 2023; 14: 1191105.

65.

Coppieters

Smalbrugge

, et al. Implementing PainChek and PARO to support pain assessment and management in residents with dementia: a qualitative study. Pain Manag Nurs 2023; 24: 587–594.