Usability testing of an electronic pain monitoring system for palliative cancer patients: A think-aloud study

Abstract

Pain experienced by advanced cancer patients is often poorly controlled due to inadequate assessment. We aimed to test an electronic pain monitoring system (PainCheck) with advanced cancer patients and health professionals. In all, 29 participants (13 patients and 16 health professionals (n = 16)) used PainCheck while thinking their thoughts aloud. After the think aloud, both groups were asked about their experiences. Think-aloud and semi-structured interview data were analysed using framework analysis. Both groups were generally positive about PainCheck and found it easy to understand but they had concerns about clinical integration. Their concerns related to impact on workload, how lack of response may affect patient care and ability to engage with and use the technology. PainCheck has the capacity to be useful in clinical practice. For integration to be successful, patients and health professionals need guidance on how to use PainCheck and the expectations of users need to be clarified.

Introduction

Despite increased attention on cancer pain in recent years, pain is still experienced by 66 per cent of advanced cancer patients. ¹ Over half of all pain reported by advanced cancer patients is described as severe. ¹ Insufficient pain assessment is one of the most common barriers to successful pain control. ² Key recommendations to improve the management of cancer pain are to conduct frequent pain assessments^3,4 and to improve the documentation of these assessments. ⁵ Guidelines are available for the management of pain, ⁶ but guidance on the assessment of pain is limited.

Pain assessments that are conducted are not always well recorded. ⁷ Frequent recording of pain assessments would allow health professionals to focus on changes over time for individual patients and to take the subjective nature of pain into account. There are various pain assessment tools available to aid assessment and documentation but these are rarely used in routine practice. ⁸ Many of the available tools use numerical rating scales to measure pain intensity. Although these scales have been shown to be valid and reliable, ⁹ qualitative characteristics are also important features of pain that can impact on the chosen pain management approach.^10,11

There is growing interest internationally in the development of remote monitoring technologies to enhance pain assessment.^12,13 The use of an electronic pain monitoring system would increase awareness of pain among health professionals and improve documentation and treatment of pain. ¹⁴ In order for electronic monitoring systems to be adopted in practice, patients and health professionals need to be able to engage with the system and feel confident using it. A systematic review of remote monitoring technologies has suggested that clinical relevance and technology functionality are key factors likely to influence whether a system would be adopted in clinical practice. ¹³ Extensive usability testing needs to be carried out in order to ensure the system meets the needs of patients and health professionals and has the optimum impact on care provision and pain management. ¹⁵ The aim of this study is to understand how advanced cancer patients and health professionals might use a patient-reported electronic pain monitoring system (PainCheck) and to consider clinical integration.

Methods

Participants

Patients were recruited from two hospices by research nurses. Eligible patients had to be aged ≥18 years, have advanced cancer (metastatic cancer and/or receiving anti-cancer therapy with palliative intent), have pain (receiving analgesic treatment for cancer symptom-related pain and/or cancer therapy–related pain), have a good level of spoken and written English, be able to provide informed consent and have access to the Internet. Participants were excluded from the study if they were perceived to be unable to understand and complete PainCheck and/or unable to provide informed consent. Participants deemed inappropriate to approach by members of the clinical teams were not approached (e.g. those where death is imminent). A total of 42 patients were approached to take part in the study; 13 completed the think-aloud (TA) and semi-structured interview (Table 1). Three further participants consented and participated, but were excluded from analysis as they did not have basic computer skills (unable to use a mouse/keyboard) or were not able to follow the TA protocol (constantly asked questions and needed guidance throughout). Recruitment flowchart is presented in Figure 1.

Table 1.

Patient and health professional demographics.

Patients
Sex
Female (n)	5
Male (n)	8
Age (median, range)	65 (45–87)
Diagnosis
Prostate (n)	3
Breast (n)	2
Lymphoma (n)	2
Myeloma (n)	2
Lung (n)	1
Oesophagus	1
Oropharyngeal	1
Ovarian	1
Health professionals
Sex
Female (n)	14
Male (n)	2
Age (median, range)	43 (30–56)
Number of years’ experience in current role (median, range)	8.5 (1–25)
Professional group
Palliative care doctor (n)	5
Clinical nurse specialist (n)	4
General practitioner (n)	4
Community/district nurse (n)	3

Figure 1.

Recruitment flowchart for patients and health professionals.

Two methods were used to recruit a diverse sample of health professionals (clinical nurse specialists, general practitioners (GPs), palliative care doctors and community/district nurses). In all, 11 participants were recruited from a pool of health professionals (n = 105) who had taken part in a previous research project. ⁸ A snowballing technique was used to recruit the remaining five participants. Health professionals who had either participated in this study or been approached but were unable to take part recommended others. Health professionals were approached by email and phone; 16 health professionals participated (Figure 1). We recruited health professionals from each of the professional groups with varying levels of experience (Table 1).

Data collection

PainCheck

PainCheck is a web-based system developed with patients and health professionals using a user-centred design (UCD) approach. ¹⁶ PainCheck allows patients to record their pain and gives them access to personalised pain management advice. Figures 2 and 3 present examples of the different screens within PainCheck. Patients are asked to answer various questions about their pain including providing a description of their pain (Figure 2(a)), rating current pain intensity and intensity in the last 12 h (Figure 2(b)), pain control, interference and sleep. Some items were taken from the Brief Pain Inventory (BPI). ¹⁷ Patients are asked about pain management techniques, which of these were helpful and how likely they are to try them in the future (Figure 2(c)). Various question response options are used including multiple choice, numerical (0–10 or 0–6) slider scales and free text. After completion, patients are provided with a summary of their results and suggestions of pain management techniques they may want to try in the future (Figure 2(d)). Health professionals can log in to PainCheck and view all patients registered on PainCheck and see who have completed reports (Figure 3(a)). They are then able to select a patient and view responses to individual questions (Figure 3(b)). Health professionals are presented with a graph that tracks patients’ current pain and pain in the last 12 h over time (Figure 3(c)). Patients are given a ‘red flag’ in the health professional system if they reach certain thresholds for current pain and pain control. After reading the patient report, health professionals can decide what action, if any, they would like to take as a result (Figure 3(d)). Health professionals have the option to contact patients through PainCheck to provide information and advice. All data entered into PainCheck can be exported into Microsoft Excel in comma separated values (CSV) format.

Figure 2.

Screenshots of patient interface: (a) qualitative description of pain. (b) Numerical sliding scales rating pain and interference. (c) Questions exploring what patients have done to control their pain and if they were helpful, the second page included self-help measure such as use of hot/cold, having a bath and exercise. (d) Patient feedback.

Figure 3.

Screenshots of health professional dashboard: (a) Health professional dashboard. (b) Individual patient report. (c) Patient scores over time. (d) Record of health professional action.

The TA technique is often used in the development of new technologies.^18,19 TA testing is the final stage in a rigorous development process which has included a literature review, ²⁰ prototype feasibility testing ²¹ and interviews with patients and health professionals. ⁸ The TA technique involves asking participants to verbalise their thoughts while completing a task. Both groups were given a computer-based task before using PainCheck to allow them to practise thinking aloud. All participants were given minimal explanation of PainCheck and were asked to use it as they would in their role as patient/health professional while verbalising their thoughts. Patients completed the PainCheck assessment and explored their personalised feedback. Health professionals accessed a fully functional PainCheck prototype containing reports from simulated patients. Participants used a desktop computer or a laptop. After the TA, both groups were asked about their experiences in a semi-structured interview. S.T. conducted health professional sessions in a quiet room at the health professional’s place of work. M.J.A. conducted patient sessions at the patient’s home or at the hospice.

Data analysis

The TA task and semi-structured interview were analysed using framework analysis. After initial familiarisation with all transcripts, four health professional transcripts were analysed (one from each professional group). The initial framework was created by S.T. and discussed with B.M.B. The framework was then applied to the remaining health professional transcripts. Any ambiguity about where to code particular issues was discussed. The framework which emerged from and was applied to the health professional data was intended to also be applied to the patient transcripts. However, after examining the first patient transcript, it was decided that patient and health professional utterances were not sufficiently similar for the same framework to be applied. Four patient transcripts were analysed and a patient framework was created and applied to remaining transcripts.

Results

We identified four meta-themes in each dataset. Although, two distinct coding frameworks emerged from the data, the meta-themes identified in the patient and health professional data were strongly aligned and ultimately combined (see Figure 4). Both frameworks showed evidence of a journey of understanding PainCheck through to clinical application. Three central themes focus specifically on PainCheck: how it works, their understanding of it, how useful PainCheck might be and how it might work in practice. Another meta-theme, setting the scene, feeds into the central themes and describes the context that PainCheck will have to fit into.

Figure 4.

Themes and meta-themes from patient and health professional data.

Making sense of PainCheck

Patients and health professionals’ first step during the interview was to familiarise themselves with PainCheck and try to understand the content, how it works and what they were expected to do. Part of the familiarising process involved participants describing or reading aloud the screen content, allowing them to process the information and make sense of it. Participants needed time to familiarise themselves with the technology and the content and to understand what was expected of them. Despite some minor log in and navigation problems, participants generally found PainCheck easy to use:

Yes I think it is, it should be ok once you know people get used to it. (Patient 6, Myeloma, age 65 years)

It’s all self-explanatory really isn’t it? … (Patient 13, prostate, age 75 years)

The problems experienced were largely due to the participants’ unfamiliarity with the system or with computers in general:

I can’t fathom out what I’ve done wrong! … What am I doing wrong here? That box, can’t get an arrow! … (Patient 5, myeloma, age 72 years)

The functionality of PainCheck is similar to other electronic systems. This familiarity made it easier for health professionals to use it. Although health professionals were generally able to work out how the different features worked and what they were expected to do, some health professionals suggested that a little more guidance on screen would have been useful. Health professionals were often unsure exactly what was expected of them in terms of their response and how they were supposed to use PainCheck. Their uncertainties were particularly around how to view patient reports, how reports moved to the reviewed section (Figure 3(a)), the action box at the end of the report (Figure 3(d)) and what the red flags meant (Figure 3(b)):

So I guess I’m unsure what this red flagging right next to their names means … I don’t think I can click on it? It just opens the entry up … You see I’m unsure as to what that coding’s for … (Health professional 3 (HP3), Palliative care doctor)

Participants made a number of suggestions of ways PainCheck could be improved. These suggestions focused on how PainCheck could be made easier to navigate or how functions could be clarified:

it might be an idea to have some sort of information pointers on like you know you get on powerpoint to tell you what you’re looking at. (Patient 9, oropharyngeal, age 56 years)

Considering how useful PainCheck might be

As participants worked through PainCheck, they considered the questions and how they related to their own experiences. The majority of health professionals thought the information provided in PainCheck would be useful in clinical practice. The reports provided an overview of the patients’ situation and would help the health professionals prepare and consider treatment options before patient contact. Some health professionals felt the reports were too basic and did not provide enough information. Similarly, many patients felt that PainCheck only partially represented their experiences. Not all questions were relevant to every patient. For example, some patients had showers because they were unable to have a bath (Figure 2(c)). Some patients wanted to be able to put more information into PainCheck to describe their experiences. Health professionals echoed these feelings and suggested that there need to be more opportunities for patients to enter free text. Health professionals wanted more specific clinical information, more details about medications, contact with health professionals and how pain had changed over time:

I suppose it would be nice to be able to bring up a dashboard or serial results for the pain with dates on, that would make it easier, but I don’t know if that is part of the functionality or not. (HP11, Palliative care doctor)

Patients went through a process of reflection. They reflected on their answers and on the content and purpose of the feedback section. Patients began to think more deeply about PainCheck by considering the meaning of individual questions and trying to understand what it meant to them. There were a number of questions open to interpretation, so patients tried to understand them before deciding how to answer:

I don’t really call tiredness pain. It’s not a pain to me, it’s just something I have to put up with. (Patient 11, prostate, age 67 years (referring to qualitative description of pain; Figure 2(a)))

While responding to PainCheck, patients began to think about its purpose and the goals of clinical application. Patients felt the purpose of PainCheck needed to be agreed and should be made clear to all users:

this isn’t designed to replace your doctor or your you know your nurse or whoever you’re talking to, this is for them to keep them up to date, keep them up to speed and if you want to telephone them it doesn’t, it’s not a replacement it’s in addition, it’s an additional benefit. (Patient 12, prostate, age 72 years)

Health professionals suggested that one of the key benefits of PainCheck was it helped patients to engage in regular pain recording and monitoring which encourages them to take a more active role in pain management. Others felt that asking patients to report pain routinely may be detrimental:

and people who very much are sort of focussed on their medical problems, and you might want to sort of try and get them away from doing that, whereas this would make them worse … (HP9, CNS)

Integration into clinical practice

Once patients and health professionals had familiarised themselves with PainCheck, they started to consider clinical integration. They considered what impact integration would have on the healthcare system and what integration would mean for them. Participants mentioned various practical issues that would need to be considered and may influence successful integration. The accessibility of PainCheck was key. Health professionals felt embedding PainCheck into existing medical records systems was a crucial step to aid integration:

It’s about the practicalities of using this sort of thing in what is, you know, horrendously busy working day (mm) … so it needs to, needs to sit well within whatever system it’s, it’s put into. (HP5, GP)

For integration to be successful, health professionals and patients require easy access to the technology needed to engage with PainCheck. Patients’ and health professionals’ confidence and familiarity with the technology as well as its portability need to be considered:

It’s easy, it’s quite obvious on the computer, you know, cos I am, not computer literate but I’m not very good with them so I did find it quite easy (mm). (Patient 1, breast, age 54 years)

Health professionals had reservations about the ability of this group of patients to be able to use PainCheck. Their concerns related to the patients: cognitive ability, age and health status:

the generation that are going through are very computer based … they’re used to not talking to people, they don’t talk to their friends anymore, they text, so it may well be that in generations to come, this will be an entirely comfortable way for people to feel that they get support and help, I don’t think it’s true of the present generation, particularly older generation who are going through the needs of palliative care at the moment. (HP6, GP)

Despite their concerns, patients and health professionals did think there were patients who would enjoy using the system and would benefit from completing the reports. Health professionals stressed the importance of engaging patients with PainCheck and helping them to understand the benefits. Health professionals and patients felt PainCheck could be used as a monitoring tool. Patients felt PainCheck would help to highlight particular issues they have been experiencing or help them to focus their thinking and put their pain into perspective:

I would think to consolidate my own ideas on what’s going on because you get so wrapped up in yourself sometimes that you can’t think outside the box and you’re just thinking about yourself … it would hone in better on what I was thinking. (Patient 2, breast, age 56 years)

To be implemented successfully, patients and health professionals felt a clear process needed to be established so everyone using PainCheck was clear who was responsible for taking action. Some health professionals were concerned they would not have time to use PainCheck due to existing workloads. Other health professionals could see how PainCheck would fit with their existing practices and potentially save time by helping them prioritise:

because surely then if somebody says that they’re ticking along ok, you might think well, ok we’ll put in a phone call to that person rather than a face to face visit, and then that could save us half an hour of time when they didn’t particularly want a visit and don’t particularly need a visit, but somebody else does. (HP12, District/community nurse)

A proportion of participants were concerned about problems being missed or thought response speed may be affected so they would prefer to communicate by phone. Patients who would opt for telephone communication at the moment felt their views might change in the future if there was a transition to using electronic reporting methods and the healthcare team encouraged it:

If I suddenly got headache I wouldn’t … go and fill it out and send it off to my doctor! But if he said can you keep … a record for three weeks on your pain … I’d happily do that. (Patient 10, Ovarian, age 49 years)

After considering how PainCheck could be integrated into the existing healthcare system, health professionals considered how they would respond on an individual level. In many cases, they wanted more information, whether this was from the patient or another health professional. If pain scores were low and the patient felt their pain was well controlled, some health professionals decided they would take no further action but would continue to monitor them:

There are some that … they’ve just had a bit of pain and … their pain score was fine now and so I’d just sort of acknowledge that I’ve seen it so the patient knew I had but it didn’t really require any action. (HP13, District/community nurse)

Setting the scene

Setting the scene is a descriptive theme where patients and health professionals talk about their current situation in terms of the healthcare they receive or provide, current methods of communication and pain management approaches. Using PainCheck made participants reflect on their experiences which provided context and background to their answers and offered an insight into the system that PainCheck would have to fit into.

One particular challenge of current practice that health professionals described was the complexity of pain management. Pain management was a ‘balancing act’ as it was often difficult to find a middle ground between giving medication time to work and appearing unconcerned about the patient experiencing pain. The subjective nature of pain makes pain management difficult for health professionals. Health professionals were aware that in the busy clinical environment, they may not give patients all the time and input they need or they may focus simply on addressing the pain without considering the other contributing factors:

I think as, certainly as doctors, you get very, sometimes you don’t think as holistically as you might. (HP16, GP)

Patients reflected on many different aspects of pain management. They explained how their pain was currently being managed, their relationships with health professionals and their thoughts and experiences of pain relief. Descriptions of pain management were heavily focused on pain relief medications. Patients had very particular views about pain medication, they avoided taking over the counter medication and relied on prescriptions from health professionals and some had concerns about taking opiates.

Many of the comments within this theme related to current methods of communication. This provided context and helped participants to describe how PainCheck may or may not be beneficial in practice. The current system relies heavily on patients taking the lead and contacting the health professional by phone if they experience problems. The problem with this system, however, is that some patients can be quite stoical and endure a certain level of pain before making contact:

Yes. But I’ve also been told I haven’t to wait till it gets that bad anymore. (Patient 4, lymphoma, age 87 years)

Both patients and health professionals went through a journey of understanding and familiarisation with PainCheck followed by consideration of PainCheck’s application in theory and then finally considering practical implementation. This three stepped process was carried out while also reflecting on the current environment that PainCheck would have to fit into.

Discussion

This is the first study to evaluate the user interface of an electronic pain monitoring system using TA methodology with palliative cancer patients and health professionals. TA methodology provides in-depth understanding of usability and identifies issues that may not be picked up in semi-structured interviews. ²² Usability testing is an essential component of eHealth design and there are various usability testing approaches available. One commonly used method is Nielsen’s Heuristics. ²³ One concern about this method of usability testing, however, is that it is led by experts and excludes end users. ²⁴ Many eHealth systems are developed without addressing the intuitiveness of the interface with participants at an early stage of development. ²⁵ For a system to be effective and applicable in clinical practice, it is important to engage users through every stage of design, development and implementation. ²⁶ A UCD approach was adopted throughout the development of PainCheck, including the evaluation of its usability as reported in this study. A UCD approach is a popular method used in eHealth^27,28 and as the name suggests puts a strong emphasis on engaging with users. Patients and health professionals were quickly able to understand PainCheck and apply it in context with limited instructions. Participants were generally positive about PainCheck and found it easy to understand but they did have some concerns about its clinical application. To be adopted successfully, patients and health professionals felt there need to be clear systems and procedures in place. Participants often forgot about the role of PainCheck as a routine monitoring tool. They were concerned that in a busy working day, health professionals may not have time to respond to PainCheck reports and patient’s needs may be left unaddressed. This concern has been echoed in research exploring chronic pain monitoring using electronic devices. ²⁹ For any kind of remote monitoring to be effective, it is important to ensure that the system promotes reliable and effective communication. ³⁰ Health professionals felt that embedding PainCheck into existing electronic records systems would ease communication problems and encourage regular monitoring. At present, very few electronic monitoring systems have been successfully linked to patient records systems despite industry recommendations. ³⁰

Participants mentioned a number of concerns about the patients’ ability to use PainCheck, particularly, the perceived lack of familiarity of the older generation with technology. Similar findings have been reported in other palliative care studies ³¹ and in the chronic pain context. ²⁹ Ensuring monitoring systems developed are ‘user-friendly’ would help to make the system more appealing to the older generation. ³² Despite initial concerns and lack of experience with computers, the majority of patients were able to navigate the PainCheck system and complete a report.

Similar to concerns raised during development work, ⁸ participants were concerned that PainCheck would replace face-to-face contact. This concern emphasises the need to clarify the purpose of PainCheck. The principle of PainCheck as with other electronic monitoring systems is to enhance existing care and communication rather than replace telephone and face-to-face contact. ³³ In order for PainCheck to be accepted, both groups need to re-evaluate PainCheck and think about how it would help them to maximise the care they provide or receive. Introduction and delivery of PainCheck would need to be considered to ensure it was packaged in a way that encouraged optimal use. PainCheck would not replace existing face-to-face contacts as these contacts are essential to effective palliative care. ²⁶ Instead, it would help healthcare professionals to monitor patients, identify problems and prioritise.

This study was conducted with patients and health professionals from the West Yorkshire region; therefore, the findings may differ regionally or nationally. Future work exploring the use of PainCheck in clinical practice should aim to include a more diverse sample of participants to try to understand how electronic systems may be received in different locations. This may give insight into a broader range of factors that may affect clinical integration. Despite sample limitations, we included health professionals from a number of specialities with varying levels of experience and the patients had a variety of diagnoses.

With minimal explanation, patients and health professionals were able to use PainCheck and consider clinical applicability. Despite some concerns, participants could see how PainCheck may benefit them and potentially lead to improvements in communication and pain management. Lack of involvement of health professionals in the development of eHealth interventions has been documented as one of the key design limitations. ¹² The PainCheck development process has addressed this limitation by involving health professionals and patients at every stage. It is clear from this and other research that health professionals are key to the successful implementation of PainCheck. In order for health professionals to adopt it, they need to have confidence that it works and be convinced of its value to patient care. ³³ Future work needs to explore how PainCheck will be used in clinical practice and what impact implementation has on pain management and patient care.