Development and usability testing of a multi-criteria value clarification methods for patients with localized prostate cancer

Abstract

Current guidelines for the development of decision aids recommend that they have to include a process for helping patients clarify their personal values, for example, by using values clarification methods. In this article, we extensively described the development process of the web-based values clarification method for patients with localized low- to intermediate-risk prostate cancer based on the analytic hierarchy process. With analytic hierarchy process, the relative importance of different attributes of available treatments can be determined through series of pairwise comparisons of potential outcomes. Furthermore, analytic hierarchy process is able to use this information to present respondents with a quantitative overall treatment score and can therefore give actual treatment advice upon patients’ request. The addition of this values clarification method to an existing web-based treatment decision aid for patients with localized prostate cancer is thought to improve the support offered to patients in their decision-making process and their decision quality.

Keywords

analytic hierarchy process clinical decision-making decision aid localized prostate cancer shared decision-making values clarification method web-based

Introduction

For patients with localized low- to intermediate-risk prostate cancer, treatment options vary between active surveillance (AS) and curative treatments (radical prostatectomy (RP), brachytherapy (BT) or external beam radiation (EBRT). As there is no clear benefit of one treatment over the others, shared decision-making (SDM) is highly appropriate for this treatment decision.¹ In order to support patients and their clinicians in the decision-making process, a web-based treatment decision aid (DA) for localized prostate cancer was evaluated in a cluster randomized controlled trial (RCT) between August 2014 and July 2016 and subsequently implemented in hospitals across the Netherlands. This DA was developed based on the International Patient Decision Aid Standards (IPDAS) criteria and offers patients stepwise guidance through the decision process.² Moreover, it contains general information about prostate cancer, the available treatment options and a values clarification method (VCM). The VCM allows patients to indicate for a range of statements the strength of their preference towards one of the treatment options on a slider scale without giving a final treatment recommendation.^2,3 Although the DA was positively evaluated, results illustrated that 34% (59/175) of the patients did not indicate a preferred treatment after using it.⁴ It is unclear whether these patients were undecided or just did not indicate their final treatment preference. For these possible undecided patients, the DA may have been a suboptimal decision-support tool, and therefore, an alternative or additional decision-support may be more suitable for them. Also, 32% (60/186) of the DA users indicated that they preferred to receive an explicit treatment advice from the DA.⁵

Thus, comparable with other decision-support tools, the DA presents evidence and contains a VCM but does not combine these two aspects in a way that allows patients to see quantitatively which treatment aligns with their statements, that is, what having the desire to avoid erectile dysfunction would mean in terms of treatment choice. Therefore, some patients could benefit more from a VCM based on the analytic hierarchy process (AHP), where the relative importance of different attributes of available treatments can be determined through a series of trade-offs of the (un)desirability of possible outcomes (pairwise comparisons). AHP is one of the most frequently used multi-criteria decision analysis (MCDA) methods, which was originally developed by Saaty⁶ in 1970s. AHP is a theory of measurement that derives ratio scales from these pairwise comparisons. With the use of AHP, we gain more insight into the individual’s decision by identifying and visualizing the best treatment option based on the user’s responses.⁷ Literature has suggested that explicitly showing patients the implications of their stated values may be associated with positive outcomes.⁸ Moreover, with this approach, patients’ are more stimulated to deliberate on the pros and cons of the different treatments.

In addition, multiple VCM design features exist to elicit patients’ well-informed values. Explicit guidelines and best practices for designing and developing VCMs are lacking. According to the systematic review by Witteman et al.,⁸ the most common theory used was expected utility theory followed by conjoint analysis and AHP. However, Witteman et al. noted that the theory behind the design used was often not reported in studies. Previous studies, therefore, suggest that more research is needed regarding VCMs based on specific theories.^8,9 Hence, in this study, we provide a detailed description of the development of an explicit VCM that is able to calculate the preferred treatment option of individual patients by using the AHP method. By involving patients and experts, we intended to develop a tool which patients can easily understand and use in the decision-making process, in addition to the existing DA. Recommendations from the literature are warranted, for example, description of the rationale for the design used and description of the stakeholder input.⁹

Methods

Patients’ decision-making process requires several steps which are nearly identical to the basic steps in AHP^9–11 (Figure 1). AHP is a decompositional approach and starts with breaking down a complex decision problem into a hierarchical structure of objectives, alternatives (treatment options) and attributes (characteristics of treatment).^10,11 Based on this information, the pairwise comparisons can be developed.

Figure 1.

Required steps in the decision-making process according to the analytic hierarchy process.

Aim of the VCM

The aim of the AHP-based online VCM is to support patients in deciding which treatment for localized low- to intermediate-risk prostate cancer best matches their personal values and preferences and to provide them with an explicit treatment advice based on their responses.

Determination of treatment options

To gain more insight into the decisional context and to identify alternative treatment options, current guidelines and the existing treatment DA with VCM were studied. AS, RP, EBRT and BT were considered as treatment options, as these are the most widely used treatment strategies for localized low- to intermediate-risk prostate cancer.¹

Determination of most important attributes

After identification of the treatment options, the next step is to identify relevant attributes of these options, such as relevant outcome measures and process of care factors. Within a small working group, including two urologists, a PhD student from the Department of Urology, an epidemiologist and two researchers from the University of Twente, relevant literature^1,3,12–19 and the existing DA with VCM were critically discussed to determine the most important attributes. These include timing of treatment (immediate treatment/postponed treatment), side effects (risk on erectile dysfunction/urinary incontinence/bowel dysfunction), potentially unnecessary treatment, follow-up (e.g. frequency of prostate-specific antigen (PSA) testing), treatment aim (e.g. tumour is removed or not removed from the body), options at recurrence and procedure (e.g. major operation/multiple radiation sessions/small operation). Subsequently, a focus group was used to explore, based on deductive reasoning, whether these attributes were also considered to be relevant by this selected group of patients. To initiate the discussion without influencing patients, a listing exercise from the study of Feldman-Stewart and colleagues was used, where patients were asked to individually rank the 10 most important attributes of a total of 32 attributes.¹³ Patients were recruited by clinicians from the Elisabeth-TweeSteden Hospital and written informed consent was obtained. We attempted to approach a sample of patients who received different treatment options and who varied in both age and time since diagnosis. The focus group was moderated by two experienced researchers and consisted of eight patients who had been treated for localized prostate cancer, one of whom was a member of the Dutch Prostate Cancer Foundation. Three patients underwent RP, one BT, one EBRT, two were on AS and one patient initially chose RP but received chemotherapy instead due to metastases. With permission of the participants, the session was audio recorded. Two members of the working group (I.B.d.A. and MGMW) read the transcript independently and developed an outline of attributes with illustrative quotes using deductive techniques. This allowed us to explore whether the predetermined set of attributes was sufficient and appropriate. They shared their perspectives on all of the attributes and identified relevant illustrative quotes. After consensus was reached within the working group, a prototype of the VCM was developed. Subsequently, an iterative process of direct feedback from members of the working group was carried out, followed by usability testing between April and June 2017 among all participants of the focus group.²⁰ For usability testing, all participants first had to complete the DA plus the additional VCM. Next, they had to complete a study-specific questionnaire with open- and closed-ended questions to elicit feedback about the content and design which allowed us to make improvements to the VCM. Descriptive statistics were used for analysis.

Developing the pairwise comparisons

After identification of the treatment options, and the most important attributes of the decision, two of the following main inputs are needed from patients to come up with a preferred treatment: attribute weights and performance scores.

Attribute weights reflect how important each attribute is and express how much each attribute influences a decision. AHP uses the method of pairwise comparisons, in which the patient needs to compare the attributes by considering all possible pairs. In a decision with N attributes, the patient needs to perform (N*(N − 1))/2 pairwise comparisons. Figure 2 displays an example of such a pairwise comparison. The response scale in AHP can be numerical, verbal or graphical, but the verbal mode is recommended when the AHP is used in a social or psychological context.²¹ The patients’ preference is initially expressed on a verbal scale and then converted into a numerical value to calculate priorities. The level of relative importance varies from equal, moderate, strong, very strong to extreme importance by 1, 3, 5, 7 and 9, respectively (Figure 2). Each comparison reflects the personal judgement of the patient on the importance of one attribute over another in the decision. For instance, an attribute which received the response ‘moderate importance’ is assumed to have a 3 times higher importance than the non-preferred attribute which will receive the reciprocal of this value (1/3).²² The intermediate values 2, 4, 6 and 8 may be used when it is difficult to choose between two adjacent importance levels, but these were omitted in the proposed VCM to facilitate the ease of application.

Figure 2.

Example of a pairwise comparison to estimate the attribute weights.

The reciprocal matrix can then be formed, which is the basis for identifying attribute weights by use of the geometric mean. For each individual patient, the normalized geometric mean is formed by taking the root of the product matrix of row elements divided by the column sum of row geometric means.⁶ Results are summarized for each row and finally are normalized to obtain attribute weights. An example of the calculations for one patient can be found in Table 1.

Table 1.

Reciprocal matrix and calculation of the normalized geometric mean.

	Attribute A	Attribute B	Attribute C	Attribute D	Geometric mean	Relative priority of the attribute
Attribute A	1	3	5	9	(135*9)^(1/4)	(3.41/5.80) 0.59
Attribute A					3.41
Attribute B	1/3	1	3	5	1.50	0.26
Attribute B	(0.33)
Attribute C	1/5	1/3	1	1	0.51	0.09
Attribute C	(0.2)	(0.33)
Attribute D	1/9	1/5	1	1	0.39	0.07
Attribute D	(0.11)	(0.2)
				Sum	5.80

Performance scores are an assessment of how well the alternative treatment options perform on each attribute.²² This can be done by using expert measurements but in case of preference-sensitive attributes, it is more sensible to ask the individual patient to provide performance scores. Performance scores can be assessed using various methods, such as point allocation, rating scales and pairwise comparisons. Introducing another method in the VCM may potentially lead to confusion and misunderstanding; therefore, it was chosen to let patients answer AHP pairwise comparisons to estimate performance scores for each alternative on each attribute (Figure 3). Similar to the first set of pairwise comparisons, the verbal responses of patients correspond to the underlying numerical performance scores.

Figure 3.

Example of a pairwise comparison to estimate performance scores for the attribute ‘timing of treatment’.

The last step of the AHP is to aggregate the attribute weights and performance scores to come to an overall value of each treatment option. This overall value can then be used to select a most preferred treatment or to rank treatments from most to least appropriate according to the patients’ preference (Table 2).

Table 2.

Example of aggregation of attribute weights and performance scores to estimate the overall value of treatment options.

	Performance scores		Attribute weights	Overall value
	Treatment 1	Treatment 2	Attribute weights	Treatment 1	Treatment 2
Attribute A	33	66	0.59	19.7	39.3
Attribute B	0	100	0.26	0	26
Attribute C	0	100	0.09	0	9
Attribute D	80	20	0.07	5.6	1.4
		Overall value		25.3	75.7

Results

Treatment options

The existing treatment DA is designed for patients with localized low- to intermediate-risk prostate cancer. Therefore, a two-stepped approach was chosen. In the first step (VCM-1), patients have to consider AS or curative treatment. In the second step (VCM-2), patients have to make trade-offs between all curative treatment options. Based on patient characteristics, patients have to complete the first and second step or only the second step. For instance, some patients have intermediate-risk prostate cancer which makes them not eligible for AS. Furthermore, BT is only offered in patients without a medical history of transurethral resection of the prostate (TURP), with a good International Prostate Symptom Score (IPSS), and a prostate volume <50 mL.¹ In case a patient is eligible for all treatment options and prefers AS in VCM-1, he does not have to complete VCM-2.

According to the previous cluster RCT, where the treatment DA for localized prostate cancer was compared with standard care, eight possible combinations of appropriate treatment options appeared to be most common, such as curative treatments only, AS versus robot-assisted laparoscopic prostatectomy (RALP) versus EBRT or AS versus EBRT versus BT. These specific treatment combinations are all integrated into the AHP-based VCM in order to make sure that patients only have to answer questions regarding the attributes which are relevant to their individual situation.

Treatment attributes

An outline of attributes with illustrative quotes is presented in Appendix 1. In the decision between AS and curative treatments (VCM-1), four attributes were selected, subsequently the patient needs to perform (4*(4 – 1)/2) six pairwise comparisons to estimate the attribute weights for VCM-1 (e.g. Figure 2). Performance scores of the attributes ‘timing of treatment’ and ‘follow-up’ are subjective and need to be determined by the individual patient in two additional pairwise comparisons (e.g. Figure 3). For the two other attributes ‘side effects’ and ‘potentially unnecessary treatment’, there is an obvious preference for AS, because it has the best possible outcome on those attributes. Therefore, a score of 100 for AS and 0 for curative treatments was assigned by the working group.

For the decision between RP, EBRT and BT (VCM-2), four attributes were selected, and hence, six pairwise comparisons need to be performed by patients to estimate attribute weights. Performance scores of all attributes in VCM-2 are subjective and need to be determined by patients. The attributes ‘procedure’ and ‘side effects’ both have three possible outcomes, resulting in three pairwise comparisons per attribute. In total, VCM-1 consists of 8 pairwise comparisons and VCM-2 consists of 14 pairwise comparisons.

Attribute weighting

With the provided responses, the reciprocal matrix can be made and attribute weights can be calculated. Attribute weights will be displayed to patients in a simple vertical bar graph in addition to percentages (Figure 4). A vertical bar graph was chosen because they have been shown to be superior to horizontal bar graphs.²³ The combination of graphical information and numerical information allows communication of the bottom line meaning and the exact size of the attribute weights.^23,24 As well as information on attribute weights, patients receive a copy of the performance table with an additional column displaying patients’ preference on each attribute based on provided performance scores (Figure 5). In theory, the VCM is able to give patients insight into the overall value of treatment options, by explicitly presenting a recommended option or scores that show how well or poorly each treatment option fits with the patients’ responses (Figure 6).

Figure 4.

Patients’ feedback: importance of each attribute.

Figure 5.

Example of performance table with on the left an additional column displaying the patients’ preference on each attribute based on provided performance scores.

Figure 6.

Overall value of treatment options.

Usability testing

Only five participants of the focus group completed all steps and were included for analysis. These participants had a mean age of 63 years (range 57–71 years). Most of the participants (3 out of 5) had a high school education or less and 2 out of 5 participants were retired.

Both the existing DA and the additional VCM are web-based applications. Four out of 5 participants indicated that the online format was desired, and only one participant preferred a paper version of the VCM. For one participant, the questions in the VCM were too complicated. As mentioned previously, the calculated attribute weights are displayed to patients in a bar graph. According to all participants, this bar graph was a clear and useful way of presenting their personal preferences. However, one participant found it hard to interpret the graph. Finally, 4 out of 5 would recommend this VCM to other patients, which gave us a sense of the overall opinion of participants on the use of the VCM. However, one participant stated the following: ‘I cannot estimate whether all patients would correctly understand all questions of this VCM, some words might be too difficult to understand’.

The feedback from this usability testing was used for the final review and revision of the VCM by the working group. Some questions had to be re-written at a grade 8 equivalent level to make sure that all questions can be understood by the majority of the target group.

Discussion and conclusion

Discussion

We described the development of an explicit AHP-based VCM, which is able to give prostate cancer patients insight into the overall value of treatment options, with the possibility of explicitly presenting a recommended treatment option. Based on the literature, the existing DA and a single focus group, the content of the VCM was determined. Since the existing treatment DA is designed for patients with localized low- to intermediate-risk prostate cancer, meaning that not all patients are eligible for AS, a two-stepped approach for the VCM was chosen. This includes step (1) where patients first have to consider AS or curative treatment and step (2) where patients have to make trade-offs between all curative treatment options (RP, EBRT and BT). The feedback from usability testing was used for input for the final review and revision of the VCM by the working group.

According to previous research, the reporting of the development of VCMs is lacking. Therefore, recommendations have been made in the literature for future studies about VCMs to extensively report the rationale for the design of the VCM and report contribution of involved stakeholders.⁹ In this article, we have implemented these recommendations and described the AHP as a basis for the design of our VCM and for further guidance of the evaluation of the VCM.

Limitations

There are several limitations of this study and VCM. First, results of attribute weights will be displayed to patients in a bar graph, and one participant found it difficult to interpret the graph. Communicating risks to patients are challenging; the use of icon arrays, bar charts and pie charts are often employed to aid this risk communication. According to the literature, they have been found to be effective in improving perception, understanding and interpretation of quantitative information over both textual and numeric formats.^24,25 Up to now, there is no consensus about which form of risk visualization is consistently more effective than the other. The type of visualization format might be dependent on the type of information needed for the particular medical decision.²⁶ Since the goal of this VCM is not to communicate actual risks, such as event rates, but to communicate patients’ individual view on possible treatment outcomes, we have decided to use both a simple vertical bar graph in addition to percentages. We also present the performance table with an additional column displaying patients’ preferences on each attribute based on the provided performance scores.

Second, for the development of the content of the VCM, a single focus group with prostate cancer patients was conducted. The number of focus groups often depends on data saturation.²⁷ In this study, the attributes and attribute levels chosen for this VCM were derived from the literature^1,3,12–19 and the existing treatment DA, which was developed using a Delphi study with patient and experts. Therefore, we chose to conduct a single focus group with patients to explore whether the predetermined set of attributes was still sufficient and appropriate, using deductive techniques. However, the limitation remains that the proposed VCM contains predetermined attributes that may not all be relevant to the individual patient or may omit attributes that are relevant to the patient. Furthermore, the algorithm used to calculate the importance score of attributes may not accurately reflect the way in which an individual patient would weigh importance of an attribute and its performance on the options.

Conclusion

Since extensively reporting the rationale for the design of the VCM is recommended, we described a structured method for the development of an AHP-based VCM well accepted by patients, to improve the existing treatment DA for patients with localized prostate cancer. By involving patients and urologists, we aim to facilitate future implementation into clinical practice.

Practical implications

The AHP method is a highly appropriate method to gain more insight into patients’ values and treatment decisions, by structuring and breaking down the decision into smaller elements for analysis.⁶ It encourages a process of pairwise comparisons of potential outcomes to determine the relative importance of different attributes of available treatment options.⁷ Furthermore, this VCM is able to explicitly present the calculated treatment option based on the scores that show how well or poorly each treatment option fits with the patients’ responses. This will be appropriate for patients who prefer actual treatment advice from the DA.

Currently, the effectiveness of this specific VCM, in addition to the existing DA, on the outcome decision quality, is being evaluated in a prospective cohort study. In that study, decision quality is defined as congruence between well-informed patients’ values and the calculated and chosen treatment option. In addition, results on disease-specific knowledge and preparation for decision-making will be compared with a historical control group who only used the existing DA. Health literacy and numeracy will also be assessed to evaluate whether the patients with low numeracy and/or low health literacy benefit from the VCM. Furthermore, the VCM will be further evaluated on its usage and patients are able to include additional decision attributes that are not predetermined by the working group but are relevant to them.

Footnotes

Appendix

Appendix 1.

Selected attributes with illustrative quotes from the focus group.

Timing of treatment	• ‘[The doctors] told me about all the side effects and other problems that could come from an operation, and that in my stage it’s better to wait’.
	• ‘If I can live to an old age with the cancer, I might as well wait with treatment for now’.
	• ‘I had no problems [with the cancer itself], it wasn’t a reason to think “there’s something wrong and it has to be removed.” After two or three consultations it was clear to me. I had the choice between nothing of internal or external radiation. And yeah, I thought ‘I can also just do nothing’.
	• ‘We are going to wait with treatment for now, but I wonder what we’re waiting for, I’d prefer like some other patients that the prostate is just removed because I want the cancer to be out of my body’.
	• ‘… but when I heard what kind of problems you can have if there are metastases I keep thinking ‘well just get that terrible thing [the cancer] out of me’.
Side effects	• ‘I heard you can sometimes get an erectile dysfunction from the operation, but after talking it over with my wife I made the conscious choice to go through with the operation’.
	• ‘Unfortunately they could not spare the nerves during my operation and they removed all of them. I will never be able to get an erection again. I knew this going into it, and thought it was more important that I would be alive, and I still feel that way’.
	• ‘With my cancer stage, treatment could give me irritating side effects but removing the prostate wouldn’t have many positive benefits’.
	• ‘I told the doctors [who said I didn’t need an operation] if they think the operation would give me more symptoms, such as cramping after the operation, removal of my bladder and other side effects, that I agree I wouldn’t want to have the operation done’.
	• ‘[I made the choice to wait with treatment] because there are a lot of side effects’.
	• ‘I looked at the [treatment options] on the computer with my wife. We decided not to surgically remove my prostate because it will also have an effect on her [because of possible erection dysfunction] … If you read other stories you sometimes make the choice that you still want to live, so we went through with the treatment anyway’.
	• ‘The doctor advised me that in principle I would have the least side effects and problems with brachytherapy’.
	• ‘The operation has effect on the relationship. Of course you want to treat the disease and that’s also nice for the partner. But on the other hand it will change the sexual relation between you’.
	• ‘[When deciding for treatment using the ranking] I also considered the effect on my bladder’
Unnecessary treatment	• ‘The doctors have told me that in my stage I can better wait with treatment, because it’s possible I become 80 years old without problems’.
	• ‘I’d heard from other patients about someone who didn’t die from the cancer [without an operation] but lived until he was 50 with it. He did not die from his prostate cancer but lived with his prostate cancer until he was old’.
	• ‘Three or four doctors have told me that I don’t necessarily have to be treated, so if I don’t want to be treated I don’t have to be’.
Follow-up scheme	• ‘I do not have an aggressive form. Hopefully, I can stay on active surveillance for a while. Every three months I have my PSA measured. Recently, I had my PSA checked and it was the same as three months before. Yes, I feel reluctant to get my PSA checked every time, but I should not feel that way. You must not lose any sleep over it, otherwise you should get active treatment’.
Follow-up scheme	• ‘Getting the results [of the tests] is stressful, I can’t deny that even though I’m mentally prepared that you cannot change the result [if it is negative]’.
Aim of treatment	• ‘He [other participant] made the choice [to undergo surgical treatment] to ensure the cancer was completely removed from his body’
	• ‘My father also had prostate cancer and died young to a metastasis, I didn’t want to go through that myself to I thought ‘it has to be removed’.
	• ‘I wanted the cancer removed and I thought the best way to do that would be with the Da Vinci robot, in this case in the Maasstad Hospital in Rotterdam’.
	• ‘No matter what the cost, I knew I wanted [the cancer] removed from my body’
Options at recurrence	• ‘… Then we looked up more about treatment and it turns out if you do radiation and it doesn’t work, you can’t be operated anymore. But if you operate and it doesn’t go well, you can still do radiation. That was the biggest eye-opener for me to switch to a surgical treatment’.
Procedure	• ‘[I first chose to get an operation] because I didn’t want to be in the hospital for 2 days, I thought with radiation I just get there at 7 a.m. and can get back home quickly’.

Acknowledgements

We would like to thank the bachelor students Adai Kara, Stijn Kamphuis and Milas Oddo who contributed to the transcription of the data derived from the focus group. Furthermore, we would like to thank Jamie Gilbert for translating selected illustrative quotes of the transcript of the focus group. Finally, we would like to thank Joshua Rosenberg and Stefanie Rosenberg for final textual editing.

I.B.d.A and MGMW contributed equally to concept and design of the VCM. Furthermore, I.B.d.A and MGMW both moderated the focus group and read the transcript to determine whether the predetermined set of attributes was sufficient and appropriate. They also contributed equally to drafting the manuscript.

All authors were members of the working group and contributed to the review, critical revision and final approval of the manuscript. I.B.d.A and MGMW contributed equally to this work.

Availability of data and materials

The transcript of the focus group is not publicly available to protect participants’ confidentiality, but the interview transcripts are available from the corresponding author on reasonable request.

Ethics approval and consent to participate

The regional Medical Ethical Review Board of Brabant, The Netherlands, waived the need for formal ethical approval (reference number NW2017-26), and the study protocol was approved by the research board of the Elisabeth-TweeSteden Hospital in the Netherlands. Participants of the focus group were provided with study information and an informed consent form, which had to be understood and signed prior to any data collection.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

ORCID iD

Marieke GM Weernink

References

Urologie

NVv

. Prostaatcarcinoom. Richtlijnendatabase, 5 June 2014. Utrecht: Urologie NVv.

Cuypers

Lamers

Kil

, et al. A global, incremental development method for a web-based prostate cancer treatment decision aid and usability testing in a Dutch clinical setting. Health Informatics J 2019; 25(3): 701–714.

Cuypers

Lamers

Kil

, et al. Impact of a web-based treatment decision aid for early-stage prostate cancer on shared decision-making and health outcomes: study protocol for a randomized controlled trial. Trials 2015; 16: 231.

Lamers

Cuypers

De Vries

, et al. How do patients choose between active surveillance, radical prostatectomy, and radiotherapy? The effect of a preference-sensitive decision aid on treatment decision making for localized prostate cancer. Urol Oncol 2017; 35: 37.e9–37.e17.

Cuypers

Lamers

RED

Kil

PJM

, et al. Impact of a web-based prostate cancer treatment decision aid on patient-reported decision process parameters: results from the Prostate Cancer Patient Centered Care trial. Support Care Cancer 2018; 26(11): 3739–3748.

Saaty

TL.

What is the analytic hierarchy process? In: Mitra

Greenberg

Lootsma

, et al. (eds) Mathematical models for decision support. Berlin; Heidelberg: Springer, 1988, pp. 209–221.

Cunich

Salkeld

Dowie

, et al. Integrating evidence and individual preferences using a web-based multi-criteria decision analytic tool: an application to prostate cancer screening. Patient 2011; 4: 153–162.

Witteman

Scherer

Gavaruzzi

, et al. Design features of explicit values clarification methods: a systematic review. Med Decis Making 2016; 36: 453–471.

Fagerlin

Pignone

Abhyankar

, et al. Clarifying values: an updated review. BMC Med Inform Decis Mak 2013; 13(Suppl. 2): S8.

10.

Dolan

JG.

Multi-criteria clinical decision support: a primer on the use of multiple criteria decision making methods to promote evidence-based, patient-centered healthcare. Patient 2010; 3: 229–248.

11.

Dolan

JG.

Shared decision-making – transferring research into practice: the analytic hierarchy process (AHP). Patient Educ Couns 2008; 73: 418–425.

12.

Korfage

Essink-Bot

Borsboom

, et al. Five-year follow-up of health-related quality of life after primary treatment of localized prostate cancer. Int J Cancer 2005; 116: 291–296.

13.

Feldman-Stewart

Tong

Siemens

, et al. The impact of explicit values clarification exercises in a patient decision aid emerges after the decision is actually made: evidence from a randomized controlled trial. Med Decis Making 2012; 32: 616–626.

14.

Feldman-Stewart

Capirci

Brennenstuhl

, et al. Information for decision making by patients with early-stage prostate cancer: a comparison across 9 countries. Med Decis Making 2011; 31: 754–766.

15.

de Bekker-Grob

Bliemer

Donkers

, et al. Patients’ and urologists’ preferences for prostate cancer treatment: a discrete choice experiment. Br J Cancer 2013; 109: 633–640.

16.

Robinson

Moritz

Fung

Meta-analysis of rates of erectile function after treatment of localized prostate carcinoma. Int J Radiat Oncol Biol Phys 2002; 54: 1063–1068.

17.

Stanford

Feng

Hamilton

, et al. Urinary and sexual function after radical prostatectomy for clinically localized prostate cancer: the Prostate Cancer Outcomes Study. JAMA 2000; 283: 354–360.

18.

Van den Bergh

Essink-Bot

Roobol

, et al. Do anxiety and distress increase during active surveillance for low risk prostate cancer? J Urol 2010; 183: 1786–1791.

19.

Fitch

Pang

Ouellet

, et al. Canadian Men’s perspectives about active surveillance in prostate cancer: need for guidance and resources. BMC Urol 2017; 17: 98.

20.

Coulter

Stilwell

Kryworuchko

, et al. A systematic development process for patient decision aids. BMC Med Inform Decis Mak 2013; 13(Suppl. 2): S2.

21.

Saaty

TL.

Decision making for leaders: the analytic hierarchy process for decisions in a complex world. Pittsburgh, PA: RWS Publications, 1990.

22.

Belton

Stewart

TJ.

Multiple criteria decision analysis: an integrated approach. Berlin: Springer-Science + Business Media, 2002.

23.

Timmermans

Molewijk

Stiggelbout

, et al. Different formats for communicating surgical risks to patients and the effect on choice of treatment. Patient Educ Couns 2004; 54: 255–263.

24.

Trevena

Zikmund-Fisher

Edwards

, et al. Presenting quantitative information about decision outcomes: a risk communication primer for patient decision aid developers. BMC Med Inform Decis Mak 2013; 13(Suppl. 2): S7.

25.

Lipkus

IM.

Numeric, verbal, and visual formats of conveying health risks: suggested best practices and future recommendations. Med Decis Making 2007; 27: 696–713.

26.

Hawley

Zikmund-Fisher

Ubel

, et al. The impact of the format of graphical presentation on health-related knowledge and treatment choices. Patient Educ Couns 2008; 73: 448–455.

27.

Krueger

RA.

Designing and conducting focus group interviews. Research Report, University of Minnesota, St. Paul, MN, October 2002.