Development and usability testing of a multi-criteria value clarification methods for patients with localized prostate cancer

Abstract

Current guidelines for the development of decision aids recommend that they have to include a process for helping patients clarify their personal values, for example, by using values clarification methods. In this article, we extensively described the development process of the web-based values clarification method for patients with localized low- to intermediate-risk prostate cancer based on the analytic hierarchy process. With analytic hierarchy process, the relative importance of different attributes of available treatments can be determined through series of pairwise comparisons of potential outcomes. Furthermore, analytic hierarchy process is able to use this information to present respondents with a quantitative overall treatment score and can therefore give actual treatment advice upon patients’ request. The addition of this values clarification method to an existing web-based treatment decision aid for patients with localized prostate cancer is thought to improve the support offered to patients in their decision-making process and their decision quality.

Introduction

For patients with localized low- to intermediate-risk prostate cancer, treatment options vary between active surveillance (AS) and curative treatments (radical prostatectomy (RP), brachytherapy (BT) or external beam radiation (EBRT). As there is no clear benefit of one treatment over the others, shared decision-making (SDM) is highly appropriate for this treatment decision. ¹ In order to support patients and their clinicians in the decision-making process, a web-based treatment decision aid (DA) for localized prostate cancer was evaluated in a cluster randomized controlled trial (RCT) between August 2014 and July 2016 and subsequently implemented in hospitals across the Netherlands. This DA was developed based on the International Patient Decision Aid Standards (IPDAS) criteria and offers patients stepwise guidance through the decision process. ² Moreover, it contains general information about prostate cancer, the available treatment options and a values clarification method (VCM). The VCM allows patients to indicate for a range of statements the strength of their preference towards one of the treatment options on a slider scale without giving a final treatment recommendation.^2,3 Although the DA was positively evaluated, results illustrated that 34% (59/175) of the patients did not indicate a preferred treatment after using it. ⁴ It is unclear whether these patients were undecided or just did not indicate their final treatment preference. For these possible undecided patients, the DA may have been a suboptimal decision-support tool, and therefore, an alternative or additional decision-support may be more suitable for them. Also, 32% (60/186) of the DA users indicated that they preferred to receive an explicit treatment advice from the DA. ⁵

Thus, comparable with other decision-support tools, the DA presents evidence and contains a VCM but does not combine these two aspects in a way that allows patients to see quantitatively which treatment aligns with their statements, that is, what having the desire to avoid erectile dysfunction would mean in terms of treatment choice. Therefore, some patients could benefit more from a VCM based on the analytic hierarchy process (AHP), where the relative importance of different attributes of available treatments can be determined through a series of trade-offs of the (un)desirability of possible outcomes (pairwise comparisons). AHP is one of the most frequently used multi-criteria decision analysis (MCDA) methods, which was originally developed by Saaty ⁶ in 1970s. AHP is a theory of measurement that derives ratio scales from these pairwise comparisons. With the use of AHP, we gain more insight into the individual’s decision by identifying and visualizing the best treatment option based on the user’s responses. ⁷ Literature has suggested that explicitly showing patients the implications of their stated values may be associated with positive outcomes. ⁸ Moreover, with this approach, patients’ are more stimulated to deliberate on the pros and cons of the different treatments.

In addition, multiple VCM design features exist to elicit patients’ well-informed values. Explicit guidelines and best practices for designing and developing VCMs are lacking. According to the systematic review by Witteman et al., ⁸ the most common theory used was expected utility theory followed by conjoint analysis and AHP. However, Witteman et al. noted that the theory behind the design used was often not reported in studies. Previous studies, therefore, suggest that more research is needed regarding VCMs based on specific theories.^8,9 Hence, in this study, we provide a detailed description of the development of an explicit VCM that is able to calculate the preferred treatment option of individual patients by using the AHP method. By involving patients and experts, we intended to develop a tool which patients can easily understand and use in the decision-making process, in addition to the existing DA. Recommendations from the literature are warranted, for example, description of the rationale for the design used and description of the stakeholder input. ⁹

Methods

Patients’ decision-making process requires several steps which are nearly identical to the basic steps in AHP^9–11 (Figure 1). AHP is a decompositional approach and starts with breaking down a complex decision problem into a hierarchical structure of objectives, alternatives (treatment options) and attributes (characteristics of treatment).^10,11 Based on this information, the pairwise comparisons can be developed.

Figure 1.

Required steps in the decision-making process according to the analytic hierarchy process.

Aim of the VCM

The aim of the AHP-based online VCM is to support patients in deciding which treatment for localized low- to intermediate-risk prostate cancer best matches their personal values and preferences and to provide them with an explicit treatment advice based on their responses.

Determination of treatment options

To gain more insight into the decisional context and to identify alternative treatment options, current guidelines and the existing treatment DA with VCM were studied. AS, RP, EBRT and BT were considered as treatment options, as these are the most widely used treatment strategies for localized low- to intermediate-risk prostate cancer. ¹

Determination of most important attributes

After identification of the treatment options, the next step is to identify relevant attributes of these options, such as relevant outcome measures and process of care factors. Within a small working group, including two urologists, a PhD student from the Department of Urology, an epidemiologist and two researchers from the University of Twente, relevant literature^1,3,12–19 and the existing DA with VCM were critically discussed to determine the most important attributes. These include timing of treatment (immediate treatment/postponed treatment), side effects (risk on erectile dysfunction/urinary incontinence/bowel dysfunction), potentially unnecessary treatment, follow-up (e.g. frequency of prostate-specific antigen (PSA) testing), treatment aim (e.g. tumour is removed or not removed from the body), options at recurrence and procedure (e.g. major operation/multiple radiation sessions/small operation). Subsequently, a focus group was used to explore, based on deductive reasoning, whether these attributes were also considered to be relevant by this selected group of patients. To initiate the discussion without influencing patients, a listing exercise from the study of Feldman-Stewart and colleagues was used, where patients were asked to individually rank the 10 most important attributes of a total of 32 attributes. ¹³ Patients were recruited by clinicians from the Elisabeth-TweeSteden Hospital and written informed consent was obtained. We attempted to approach a sample of patients who received different treatment options and who varied in both age and time since diagnosis. The focus group was moderated by two experienced researchers and consisted of eight patients who had been treated for localized prostate cancer, one of whom was a member of the Dutch Prostate Cancer Foundation. Three patients underwent RP, one BT, one EBRT, two were on AS and one patient initially chose RP but received chemotherapy instead due to metastases. With permission of the participants, the session was audio recorded. Two members of the working group (I.B.d.A. and MGMW) read the transcript independently and developed an outline of attributes with illustrative quotes using deductive techniques. This allowed us to explore whether the predetermined set of attributes was sufficient and appropriate. They shared their perspectives on all of the attributes and identified relevant illustrative quotes. After consensus was reached within the working group, a prototype of the VCM was developed. Subsequently, an iterative process of direct feedback from members of the working group was carried out, followed by usability testing between April and June 2017 among all participants of the focus group. ²⁰ For usability testing, all participants first had to complete the DA plus the additional VCM. Next, they had to complete a study-specific questionnaire with open- and closed-ended questions to elicit feedback about the content and design which allowed us to make improvements to the VCM. Descriptive statistics were used for analysis.

Developing the pairwise comparisons

After identification of the treatment options, and the most important attributes of the decision, two of the following main inputs are needed from patients to come up with a preferred treatment: attribute weights and performance scores.

Attribute weights reflect how important each attribute is and express how much each attribute influences a decision. AHP uses the method of pairwise comparisons, in which the patient needs to compare the attributes by considering all possible pairs. In a decision with N attributes, the patient needs to perform (N*(N − 1))/2 pairwise comparisons. Figure 2 displays an example of such a pairwise comparison. The response scale in AHP can be numerical, verbal or graphical, but the verbal mode is recommended when the AHP is used in a social or psychological context. ²¹ The patients’ preference is initially expressed on a verbal scale and then converted into a numerical value to calculate priorities. The level of relative importance varies from equal, moderate, strong, very strong to extreme importance by 1, 3, 5, 7 and 9, respectively (Figure 2). Each comparison reflects the personal judgement of the patient on the importance of one attribute over another in the decision. For instance, an attribute which received the response ‘moderate importance’ is assumed to have a 3 times higher importance than the non-preferred attribute which will receive the reciprocal of this value (1/3). ²² The intermediate values 2, 4, 6 and 8 may be used when it is difficult to choose between two adjacent importance levels, but these were omitted in the proposed VCM to facilitate the ease of application.

Figure 2.

Example of a pairwise comparison to estimate the attribute weights.

The reciprocal matrix can then be formed, which is the basis for identifying attribute weights by use of the geometric mean. For each individual patient, the normalized geometric mean is formed by taking the root of the product matrix of row elements divided by the column sum of row geometric means. ⁶ Results are summarized for each row and finally are normalized to obtain attribute weights. An example of the calculations for one patient can be found in Table 1.

Table 1.

Reciprocal matrix and calculation of the normalized geometric mean.

	Attribute A	Attribute B	Attribute C	Attribute D	Geometric mean	Relative priority of the attribute
Attribute A	1	3	5	9	(1*3*5*9)(1/4)	(3.41/5.80) 0.59
					3.41
Attribute B	1/3	1	3	5	1.50	0.26
	(0.33)
Attribute C	1/5	1/3	1	1	0.51	0.09
	(0.2)	(0.33)
Attribute D	1/9	1/5	1	1	0.39	0.07
	(0.11)	(0.2)
				Sum	5.80

Performance scores are an assessment of how well the alternative treatment options perform on each attribute. ²² This can be done by using expert measurements but in case of preference-sensitive attributes, it is more sensible to ask the individual patient to provide performance scores. Performance scores can be assessed using various methods, such as point allocation, rating scales and pairwise comparisons. Introducing another method in the VCM may potentially lead to confusion and misunderstanding; therefore, it was chosen to let patients answer AHP pairwise comparisons to estimate performance scores for each alternative on each attribute (Figure 3). Similar to the first set of pairwise comparisons, the verbal responses of patients correspond to the underlying numerical performance scores.

Figure 3.

Example of a pairwise comparison to estimate performance scores for the attribute ‘timing of treatment’.

The last step of the AHP is to aggregate the attribute weights and performance scores to come to an overall value of each treatment option. This overall value can then be used to select a most preferred treatment or to rank treatments from most to least appropriate according to the patients’ preference (Table 2).

Table 2.

Example of aggregation of attribute weights and performance scores to estimate the overall value of treatment options.

	Performance scores		Attribute weights	Overall value
	Treatment 1	Treatment 2		Treatment 1	Treatment 2
Attribute A	33	66	0.59	19.7	39.3
Attribute B	0	100	0.26	0	26
Attribute C	0	100	0.09	0	9
Attribute D	80	20	0.07	5.6	1.4
		Overall value		25.3	75.7

Results

Treatment options

The existing treatment DA is designed for patients with localized low- to intermediate-risk prostate cancer. Therefore, a two-stepped approach was chosen. In the first step (VCM-1), patients have to consider AS or curative treatment. In the second step (VCM-2), patients have to make trade-offs between all curative treatment options. Based on patient characteristics, patients have to complete the first and second step or only the second step. For instance, some patients have intermediate-risk prostate cancer which makes them not eligible for AS. Furthermore, BT is only offered in patients without a medical history of transurethral resection of the prostate (TURP), with a good International Prostate Symptom Score (IPSS), and a prostate volume <50 mL. ¹ In case a patient is eligible for all treatment options and prefers AS in VCM-1, he does not have to complete VCM-2.

According to the previous cluster RCT, where the treatment DA for localized prostate cancer was compared with standard care, eight possible combinations of appropriate treatment options appeared to be most common, such as curative treatments only, AS versus robot-assisted laparoscopic prostatectomy (RALP) versus EBRT or AS versus EBRT versus BT. These specific treatment combinations are all integrated into the AHP-based VCM in order to make sure that patients only have to answer questions regarding the attributes which are relevant to their individual situation.

Treatment attributes

An outline of attributes with illustrative quotes is presented in Appendix 1. In the decision between AS and curative treatments (VCM-1), four attributes were selected, subsequently the patient needs to perform (4*(4 – 1)/2) six pairwise comparisons to estimate the attribute weights for VCM-1 (e.g. Figure 2). Performance scores of the attributes ‘timing of treatment’ and ‘follow-up’ are subjective and need to be determined by the individual patient in two additional pairwise comparisons (e.g. Figure 3). For the two other attributes ‘side effects’ and ‘potentially unnecessary treatment’, there is an obvious preference for AS, because it has the best possible outcome on those attributes. Therefore, a score of 100 for AS and 0 for curative treatments was assigned by the working group.

For the decision between RP, EBRT and BT (VCM-2), four attributes were selected, and hence, six pairwise comparisons need to be performed by patients to estimate attribute weights. Performance scores of all attributes in VCM-2 are subjective and need to be determined by patients. The attributes ‘procedure’ and ‘side effects’ both have three possible outcomes, resulting in three pairwise comparisons per attribute. In total, VCM-1 consists of 8 pairwise comparisons and VCM-2 consists of 14 pairwise comparisons.

Attribute weighting

With the provided responses, the reciprocal matrix can be made and attribute weights can be calculated. Attribute weights will be displayed to patients in a simple vertical bar graph in addition to percentages (Figure 4). A vertical bar graph was chosen because they have been shown to be superior to horizontal bar graphs. ²³ The combination of graphical information and numerical information allows communication of the bottom line meaning and the exact size of the attribute weights.^23,24 As well as information on attribute weights, patients receive a copy of the performance table with an additional column displaying patients’ preference on each attribute based on provided performance scores (Figure 5). In theory, the VCM is able to give patients insight into the overall value of treatment options, by explicitly presenting a recommended option or scores that show how well or poorly each treatment option fits with the patients’ responses (Figure 6).

Figure 4.

Patients’ feedback: importance of each attribute.

Figure 5.

Example of performance table with on the left an additional column displaying the patients’ preference on each attribute based on provided performance scores.

Figure 6.

Overall value of treatment options.

Usability testing

Only five participants of the focus group completed all steps and were included for analysis. These participants had a mean age of 63 years (range 57–71 years). Most of the participants (3 out of 5) had a high school education or less and 2 out of 5 participants were retired.

Both the existing DA and the additional VCM are web-based applications. Four out of 5 participants indicated that the online format was desired, and only one participant preferred a paper version of the VCM. For one participant, the questions in the VCM were too complicated. As mentioned previously, the calculated attribute weights are displayed to patients in a bar graph. According to all participants, this bar graph was a clear and useful way of presenting their personal preferences. However, one participant found it hard to interpret the graph. Finally, 4 out of 5 would recommend this VCM to other patients, which gave us a sense of the overall opinion of participants on the use of the VCM. However, one participant stated the following: ‘I cannot estimate whether all patients would correctly understand all questions of this VCM, some words might be too difficult to understand’.

The feedback from this usability testing was used for the final review and revision of the VCM by the working group. Some questions had to be re-written at a grade 8 equivalent level to make sure that all questions can be understood by the majority of the target group.

Discussion and conclusion

Discussion

We described the development of an explicit AHP-based VCM, which is able to give prostate cancer patients insight into the overall value of treatment options, with the possibility of explicitly presenting a recommended treatment option. Based on the literature, the existing DA and a single focus group, the content of the VCM was determined. Since the existing treatment DA is designed for patients with localized low- to intermediate-risk prostate cancer, meaning that not all patients are eligible for AS, a two-stepped approach for the VCM was chosen. This includes step (1) where patients first have to consider AS or curative treatment and step (2) where patients have to make trade-offs between all curative treatment options (RP, EBRT and BT). The feedback from usability testing was used for input for the final review and revision of the VCM by the working group.

According to previous research, the reporting of the development of VCMs is lacking. Therefore, recommendations have been made in the literature for future studies about VCMs to extensively report the rationale for the design of the VCM and report contribution of involved stakeholders. ⁹ In this article, we have implemented these recommendations and described the AHP as a basis for the design of our VCM and for further guidance of the evaluation of the VCM.

Limitations

There are several limitations of this study and VCM. First, results of attribute weights will be displayed to patients in a bar graph, and one participant found it difficult to interpret the graph. Communicating risks to patients are challenging; the use of icon arrays, bar charts and pie charts are often employed to aid this risk communication. According to the literature, they have been found to be effective in improving perception, understanding and interpretation of quantitative information over both textual and numeric formats.^24,25 Up to now, there is no consensus about which form of risk visualization is consistently more effective than the other. The type of visualization format might be dependent on the type of information needed for the particular medical decision. ²⁶ Since the goal of this VCM is not to communicate actual risks, such as event rates, but to communicate patients’ individual view on possible treatment outcomes, we have decided to use both a simple vertical bar graph in addition to percentages. We also present the performance table with an additional column displaying patients’ preferences on each attribute based on the provided performance scores.

Second, for the development of the content of the VCM, a single focus group with prostate cancer patients was conducted. The number of focus groups often depends on data saturation. ²⁷ In this study, the attributes and attribute levels chosen for this VCM were derived from the literature^1,3,12–19 and the existing treatment DA, which was developed using a Delphi study with patient and experts. Therefore, we chose to conduct a single focus group with patients to explore whether the predetermined set of attributes was still sufficient and appropriate, using deductive techniques. However, the limitation remains that the proposed VCM contains predetermined attributes that may not all be relevant to the individual patient or may omit attributes that are relevant to the patient. Furthermore, the algorithm used to calculate the importance score of attributes may not accurately reflect the way in which an individual patient would weigh importance of an attribute and its performance on the options.

Conclusion

Since extensively reporting the rationale for the design of the VCM is recommended, we described a structured method for the development of an AHP-based VCM well accepted by patients, to improve the existing treatment DA for patients with localized prostate cancer. By involving patients and urologists, we aim to facilitate future implementation into clinical practice.

Practical implications

The AHP method is a highly appropriate method to gain more insight into patients’ values and treatment decisions, by structuring and breaking down the decision into smaller elements for analysis. ⁶ It encourages a process of pairwise comparisons of potential outcomes to determine the relative importance of different attributes of available treatment options. ⁷ Furthermore, this VCM is able to explicitly present the calculated treatment option based on the scores that show how well or poorly each treatment option fits with the patients’ responses. This will be appropriate for patients who prefer actual treatment advice from the DA.

Currently, the effectiveness of this specific VCM, in addition to the existing DA, on the outcome decision quality, is being evaluated in a prospective cohort study. In that study, decision quality is defined as congruence between well-informed patients’ values and the calculated and chosen treatment option. In addition, results on disease-specific knowledge and preparation for decision-making will be compared with a historical control group who only used the existing DA. Health literacy and numeracy will also be assessed to evaluate whether the patients with low numeracy and/or low health literacy benefit from the VCM. Furthermore, the VCM will be further evaluated on its usage and patients are able to include additional decision attributes that are not predetermined by the working group but are relevant to them.