Sage Journals: Discover world-class research

Abstract

Background: With increasing accessibility of the World Wide Web, patients are using it to obtain patient education materials (PEM). With this in mind, our group (surgeons, radiation oncologists, medical oncologists, nursing, allied health professionals and academic researchers) developed a comprehensive information resource for patients with head and neck cancer (HNC), the Head and Neck Cancer Australia, formerly Beyond Five website. The aim of this study was to determine patient usability of the Head and Neck Cancer Australia website. Methods: Usability testing (Cognitive walkthrough & Think-Aloud) in 18 patients treated for HNC was undertaken at a Cancer centre (anonymous for reviewers). Results: The tasks rated easiest by patients were finding information on psychological well-being (mean time spent: 66 s, range: 10–565), health and well-being effecting quality of life (mean time spent: 36 s, range 9–117) and carer information (mean time spent: 10 s, range 3–35). Patients indicated the website contained a lot of information, covers most topics, was a trusted source of information and a springboard to other information. Conclusions: The Head and Neck Cancer Australia website provides a wide range of information and support in multiple formats available to HNC patients. Further refinements in design, navigation and website instructions are needed to allow effective patient interaction.

Keywords

usability head and neck cancer health information on the Web patient education think aloud resources and support

Introduction

The annual incidence of head and neck cancers (HNC) (thyroid, lip, oral cavity, larynx, nasopharynx, oropharynx, hypopharynx and salivary glands) worldwide is more than 1,400,000 with a high burden of morbidity and around 500,000 deaths each year, with more men than women diagnosed with and dying from the disease.¹ Increasing accessibility of the World Wide Web has resulted in patients using it to obtain patient education materials (PEM). Online PEM refers to provision of material catered to the needs of patients who suffer from a disease delivered via the internet.²

There are numerous benefits of online PEM from the patient perspective.³ Several studies have demonstrated improved health outcomes in patients who received tailored online PEM.^4,5 Provision of the right education materials to patients is associated with well-prepared and knowledgeable health consumers, which could improve health outcomes and improve knowledge about their disease and prognosis.^6,7 A patients’ emotional distress can be alleviated by online PEM increasing self-efficacy through controlling their own access to information and reducing anxiety.^8,9 Patients are likely to be more proactive in managing their illness if they are well-informed about their illness.¹⁰

Financial and logistical benefits may be derived through online provision of PEM. Having PEM on the web reduces costs in providing information and increases the currency of information. Benefits of online PEM include time and cost-effectiveness, reduced hospitalisations, easier access to educational material and improved social support.³ It may be cost effective as it reduces patient transport costs, travel time and time spent in a clinical consultation.^11,12 When a patient is diagnosed with a new, potentially life-threatening disease, such as HNC, information may not be absorbed at the time of diagnosis and important quality of life issues such as impact of treatment on swallowing ability may need to revisited.^9,13 Thus, there is a role for healthcare providers to provide online PEM to assist patients in accessing health information at their convenience and when they are more able to process it.

From the clinician perspective, HNC healthcare providers have expressed a need for access to high-quality, current HNC information deliverable via multiple formats including internet-based platforms. They noted particular need for information about the psychological effects of treatment.¹⁴ Producing tailored online HNC PEM is a challenge, as lower levels of health and computer literacy impact patient engagement. There are also challenges with patient preference for traditional written formats, as well as complex user interfaces, navigation and technical designs common across web-based resources.

The definition of usability by the International Organization for Standardization (ISO) is the extent to which users can achieve desired tasks with effectiveness (completeness and accuracy of the task), efficiency (time per task) and satisfaction (user feedback) within a website.¹⁵ Usability testing of PEM allows designers to evaluate how patients interact with information and navigate through the website. Usability testing commonly involves two key approaches: Cognitive Walkthrough and Think-Aloud Analysis.¹⁶ Cognitive Walkthrough refers to a technique requiring participants to complete tasks from a user perspective. Think-Aloud Analysis requires participants verbally expressing their thoughts as they are performing a set of specified tasks, that is, ‘thinking aloud’. This allows insight into the participants’ cognitive processes enabling observers to track how users perceive and use a website.

We developed and implemented the Comprehensive Head and Neck Cancer Patient Education and Support Needs (CHANCES) programme. The core feature of this program is a comprehensive patient education website for HNC patients, Head and Neck Cancer Australia website (http://www.headandneckcancer.org.au). This website is free to access and was determined to be the most effective method of implementing the PEM across patient groups, healthcare institutions, states and nations. The aim of this study was to assess patient usability of the Head and Neck Cancer Australia website.

Methods

We used an exploratory, usability testing design incorporating Cognitive Walkthrough and Think-Aloud Analysis methods. Ethical approval was obtained from Sydney Local Health District Research Ethics Committee – RPAH zone (Protocol No X16-0486 & HREC/16/RPAH/696).

Participants

Participants were recruited between November 2018 and March 2019 at HNC outpatient clinics at a Chris O'Brien Lifehouse. Consecutively selected HNC patients attending the clinic were approached to take part. Participants had to meet the following eligibility criteria: age older than 18 years, post-HNC treatment and able to speak English sufficiently well to complete the study without the aid of an interpreter. Cognitive impairment was an exclusion criterion. Participants provided written consent.

Data collection

Patient age, sex, type of HNC and treatment received were recorded on the demographic form. Prior to user testing, the researcher (JJ) provided patients’ with instructions to access the website and usability testing procedures (including explaining the Think-Aloud method). The Head and Neck Cancer Australia website was loaded onto a laptop computer at the homepage (Figure 1). Participants were assigned nine tasks to complete and reminded they could stop the tasks at any point if they were tired. The tasks participants completed and the features tested are listed in Table 1.

Figure 1.

The Beyond Five website home page. Renamed Head & Neck Cancer Australia. Available from: http://headandneckcancer.org.au [Accessed September 2021].

Table 1.

Think-Aloud website features, content assessed, tasks and participant feedback.

Task No.	Website feature assessed	Content assessed	Task	Feedback
1	Navigation	Psychological well-being information	1. Having a head and neck cancer can be hard on you and your family. You might need to get some help to cope with how you are feeling. Using the Beyond Five website: a) Find a part of the website that tells you about dealing with your feelings b) Download an information sheet c) Was this information easy to find?	‘completely new website, takes time to digest’ (Participant Identifier (PId) 10)
2	Content	Patient support groups information	2. You think going to a patient support group might be useful. Using the Beyond Five website: a) Can you find the contact details of all the Head and Neck patient support groups available in Australia? b) If there is a support group close to where you live, bookmark the contact details for future reference online c) Would you recommend this information to someone else?	‘A page that has all the various branches on one page, Kingsford, Newcastle, Gosford, Liverpool. Rather than having to click one at a time’ (PId 3) ‘Having geographical locations, Southern Sydney. Being able to pick that up’ (PId 4) ‘Patient support groups displayed according to your address’ (PId 15)
3	Navigation and export	Personal stories of cancer patients	3. Each person’s experience of cancer is different. Use the Beyond Five website: a) Find a video or personal story on the cancer journey b) Share this video or personal story online with a carer/family/friend c) Would you visit the website again for this information?	[Moving the personal stories in resources tab from the bottom] ‘more to the forefront then having to scroll all the way down to the bottom’(PId 3) ‘having a link on personal stories as a tab on the home page’ (PId 5) ‘have a go back or home button rather than Beyond Five logo, most people wouldn’t know to click it’ (PId 5)
4	Navigation	Health and well-being help information	4. Head and neck cancer and its treatment can affect different parts of your life, health and how you feel. You want to find out how you might improve your health and well-being. Using the Beyond Five website: a) Locate the section on health and well-being b) Find which type of HCP could help improve your health c) Was this information easy to find?	‘specific heading [and link] for seeking more help from doctors [about a particular health issue] on the webpage’ (PId 5) ‘looking for exercise program, exercise contacts’ (PId 15) ‘bland section on changes in appearance. Highlight heading/s, otherwise lose concentration’ (PId 15) ‘combine headings in health and wellbeing dropdown menu’ (PId 18)
5	Navigation to specific content	Type of HNC information	5. There are many kinds of head and neck cancer. You have been diagnosed with a <insert name> cancer. Using the Beyond Five website: a) Find a picture or video that you could use to show someone where your cancer was on your body b) Print out an information sheet on your cancer c) Would you visit the website again for this information?	‘less steps to take’ (PId 2) ‘not quite sure where my cancer would be with all these medical names. Bring closer to forefront or highlight in the beginning this is where information sheets or pdf sheets are, downloadable sheets’ (PId 3) ‘on the side rather than scrolling to the bottom of the page’(PId 5) ‘have a print icon/symbol on the first page’ (PId 8)
6	Navigation and sharing information	Post-treatment quality of life information	6. Other people in your life are important helpers when you are having cancer treatment and recovering from it. a) Locate the section on being a carer. b) Would you recommend this information to someone else? (Yes or no) c) Was this information hard to find?	‘links to contacts or other services’ (PId 15)
7	Content	Search bar	7. Getting back to your daily activities after head and neck cancer treatment can be affected by feeling exhausted, trouble concentrating, trouble speaking, swallowing or eating. Choose one problem and using the Beyond Five website: a) Find information giving you some ideas about how to cope and overcome these problems b) Would you visit the website again for this information?	‘space [white] on a website where you want to find something like a search bar on other websites’ (PId 2) ‘reduced space between search term then results’ (PId 10) ‘provide they [search bar] link to where you want, search functions are good’ (PId 6).
8	Navigating search function	HNC of interest to the participant	8. Use the Beyond Five website search function to find more information about an unmet need you may have. a) Was this information easy to find?	‘iterative refinements. First layer of stuff very well. The next layer is not so well, for example finding contact details’ (PId 7) ‘expecting answer on upper page, but it’s at the bottom of page’ (PId 10) ‘yes would visit website again as springboard to other information’ (PId 15).
9	User perspective	HNC of interest to the participant	9. You were diagnosed with a head and neck cancer and have been told about the Beyond Five website. a) Use the website to visit 3 different pages that interest you. b) Would you recommend this information to someone else?	Nil

During the task analysis of the programme, participants were instructed to think aloud while they were being recorded using software, Free Cam 8 (https://www.freescreenrecording.com/). The software recorded the screen activity and vocalisations of participants. Once verbalising of thoughts ceased, they were prompted by the researcher to answer the following questions: ‘What do you think about this Web page?’; and, ‘How can we improve this feature(s)?’. Whilst the participants were undertaking the task, a researcher observed their performance.

Data analysis

The following metrics were recorded for each task: keystrokes were counted, time spent per task (seconds), number of clicks, number of errors (clicking the wrong link during the activity – operational error), number of prompts by the researcher, whether task was completed, participant rating of difficulty (verbal response) and feedback on each task. Four academic experts (JC, HD, HS and PS) undertook the nine tasks and rated the difficulty of each task on a 3-point Likert scale: easy (1), moderate (2) and hard (3) (Table 2).

Table 2.

Expert rating of difficulty for each task.

	Task
Expert	1	2	3	4	5	6	7	8	9
JC	1	1	1	1	1	1	2	1	1
HD	1	1	3	3	2	1	2	1	1
HS	2	3	3	3	3	1	2	1	1
PS	1	1	3	1	1	1	2	1	1
Median rating	1	1	3	2	2	1	2	1	1
	Easy	Easy	Hard	Moderate	Moderate	Easy	Moderate	Easy	Easy

1 = Easy, 2 = Moderate, 3 = Hard.

Results

Sociodemographic characteristics

A total of 18 people, treated for HNC, were recruited. Their median age was 56 years (range 25–77) and the majority were male compared to female (13 vs 5). The sociodemographic and clinical characteristics of the participants are summarised in Table 3.

Table 3.

Sociodemographic and clinical characteristics of patients.

Patient	Age	Gender	Type of HNC	Treatment	Stage
1	63	F	Tongue	Surgery + chemoradiotherapy	Post-treatment
2	66	M	Tongue	Surgery + chemoradiotherapy	Post-treatment
3	62	M	Tongue	Surgery	Post-treatment
4	61	F	Tonsil	Surgery + radiotherapy	Post-treatment
5	36	F	Thyroid	Surgery	Post-treatment
6	53	M	Tongue	Surgery + chemoradiotherapy	Post-treatment
7	59	M	Oral cavity	Surgery	Post-treatment
8	48	M	Lip	Surgery	Post-treatment
9	63	M	Tonsil	Surgery + chemoradiotherapy	Post-treatment
10	77	M	Scalp	Surgery + radiotherapy	Post-treatment
11	48	M	Oral cavity	Surgery + chemoradiotherapy	Post-treatment
12	59	M	Tonsil	Surgery + chemoradiotherapy	Post-treatment
13	47	F	Sinus	Surgery + chemoradiotherapy	Post-treatment
14	54	M	Tonsil	Surgery	Post-treatment
15	25	F	Mandible	Surgery + chemoradiotherapy	Post-treatment
16	58	M	Tonsil	Surgery	Post-treatment
17	44	M	Oral cavity	Surgery	Post-treatment
18	46	M	Tongue	Surgery	Post-treatment

Think aloud task results

All participants were able to complete all of the nine assigned tasks, and the number of prompts required was low but varied between zero and five. Examples of prompts included ‘Please scroll back to the top of the page to find menu bar and make a selection’ or ‘Click the Head and Neck Cancer Australia Logo on the top left of the screen to return to home page’ to re-orientate before continuing task. Details of each task and each metric are summarised in Figure 2 and Table 4.

Figure 2.

Mean time spent on each task.

Table 4.

Summary of keystroke measures for task analysis.

	Tasks
Measure	1	2	3	4	5	6	7	8	9
Completion	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Prompts (median)	0	1	2	0	1	0	0	0	0
Time spent	66	163	145	96	167	10	36	62	67
Number of clicks (median)	1	5	5	2	3.5	1	1	2	3
Number of errors (median)	0	0	0	0	0	0	0	0	0

The tasks rated easiest by all participants were the following: finding information on health and well-being, finding three web pages that interested them and finding information on being a carer (Figure 3). The two hardest tasks were using the search bar and accessing videos of personal stories. In some tasks, participants did not rate the difficulty due to choice and thus the total may not be n = 18. In comparison, the experts (radiation oncologists, surgeon and academic researcher) rated the same previous tasks as easy and moderate including finding information about patient support groups (Table 2). All feedback from participants is presented in Table 1.

Figure 3.

Participant rating of difficulty for each task.

Task 1 – Psychological well-being information

All participants rated this task as easy (n = 18). Feedback from participants indicated the need for additional information about other healthcare professionals (HCPs) and the time they required to process new information (Table 1).

Task 2 – Patient support groups information

Participant ratings for this task varied with 10 (77%) indicating it was easy and hard for three (23%). Finding the closest geographical patient support group was difficult on the website due to the layout and appearance of large local health districts rather than individual suburbs (Table 1).

Task 3 – Personal stories of cancer patients

The task was rated as easy by 11 (73%) and hard by four (27%). Navigating to access the personal stories videos was complex with few visible shortcuts on the top of the webpage to guide patients to the multimedia resources (Table 1).

Task 4 – Health and well-being help information

The participants rated this task as easy in 12 (86%) and hard in two (14%). The appearance of headings needed to be clearer to help them navigate to and obtain contact details for further support from a relevant HCP. Currently, there is a long dropdown tab with multiple health and well-being topics without specific contact details for support located nearby (Table 1).

Task 5 – Type of HNC information

The participant rating for this task was easy in six (67%) participants and hard in three (33%). Finding a specific fact sheet on a particular type of HNC was challenging with participants not always knowing the scientific name, for example, throat cancer patients would not recognise this as oropharyngeal carcinoma. Moreover, the layout and navigation to download an exportable fact sheet were complex involving multiple steps and having to scroll to the end of the webpage. Suggestions included cutting the number of clicks and having a symbol/icon of a printer or portable document file (PDF) on the top of the screen rather than the bottom (Table 1).

Task 6 – Carer information

The participant rating for this task was easy for all patients. All patients would recommend this information to a carer.

Task 7 – Post-treatment quality of life information

The participant rating for this task was easy in 11 participants (92%) and hard in one (8%). Participants suggested access to local contacts if they required support with a particular quality of life issue (Table 1).

Task 8 – Search bar

The participant rating for this task was easy in nine participants (69%) and hard in four (31%). Two terms were searched more than once: recurrence (searched three times) and tiredness (searched two times). The remaining terms were ‘swallowing’, ‘financial’, ‘throat cancer’, ‘secondary cancers’, ‘pain in neck’, ‘sleep’, ‘speech’, ‘tonsil cancer’ and ‘osteosarcoma’. The search bar was not obvious and required patients to search for the magnifying glass symbol before entering a term. Moreover, once the search bar was used, search results were not specific to the search term and the majority of the participants did not find their search query:

Task 9 – Interest HNC information

All patients found this task easy. Refinements were suggested on the website to ensure information appears easily and is layered in a succinct manner to allow navigation to the required destination. Shortcuts on the top of the webpage may assist navigation to reach specific content.

Discussion

This study evaluated the usability of the Head and Neck Cancer Australia website (www.beyondfive.org.au), a website designed for patients and HCPs information needs and support. In brief, the participants in this study were able to complete all the tasks assigned with a differing degree of difficulty, although most reported the tasks were easy to complete. They provided insightful comments and suggestions regarding potential refinements to the website to improve navigation and usability. It is important to test HNC websites with end users to make improvements.

Completion of cognitive walkthroughs requires participants to rapidly develop familiarity with material under investigation.¹⁷ In our study, the majority of patients completed the task with minimal prompts and, while completion times varied, most did so relatively quickly. The more users of a website access it, the more familiar they become with it, increasing their knowledge of the navigation and increasing their efficiency over time.^18,19 The most common usability issues in our study were navigation, design and learnability which are consistent with the literature.^20,21

Navigation is important to manoeuvre through multifaceted information on a website with accessing support groups and psychological well-being an easy task for participants. Although, locating further information content such as contact details was challenging. Navigation was challenging in the Head and Neck Cancer Australia website. Some elements of navigation were easy such as the clear menu options and link recognition to effectively complete a task. The challenges could be explained by lack of orientation, as after clicking several links participants may not have recognise where they were on the website, for example, trying to find a personal story video and ending up in the health and well-being section. Moreover, there needs to be a clear ‘home’ icon or ‘back’ button to take patients back to the previous page if they get lost within the site. Support groups offer patients the opportunity to be empowered through connection with others who have had similar experiences and sharing their stories and coping strategies to help improving their outcomes and maintaining an engagement in treatment and recovery.^22,23 In the Head and Neck Cancer Australia website, the navigation to patient support groups or HNC services was highlighted with a ‘Find support’ button on the top right hand corner of the page. While participants could locate the support access point, they were unsure about the geographically closest patient support group due to poor presentation of the geographical locations, this information was provided as a list rather than an interactive map. Having contact details accessible on an interactive map or presented via regions or suburbs may alleviate the confusion and increase accessibility and efficiency of the task.¹⁵

Design of a website allows for easy visibility and recognition of multiple media resources. Videos are important learning tools and assist patients in understanding and processing information.²⁴ However, in our website, the ‘Personal Story’ video task required the most prompting and clicks for participants to access the correct video. The increased prompts and clicking may be due to the architecture and technical design of the website.³ To effectively engage users, a simplistic design and obvious navigation require fewer clicks and greater visibility via buttons or obvious links to the ‘Personal Stories’ section. One suggestion of moving the ‘resources’ tab to the top of the page, above the major headings may make it more visible and thus increase accessibility.¹⁵

The efficiency per task (learnability) was variable with no improvement in subsequent tasks once participants became familiar with the website. This variability in our website over time could be due to 8 heading on the menu bar and multiple options on the drop down menu, for example, Health and Well-being has 16 options in the dropdown menu. However, ensuring simplicity of the Head and Neck Cancer Australia website may compensate for the variable efficiency through the inclusion of simple subject headings, transparency of information, uncluttered layout, consistency of design throughout and removing redundant features.¹⁵

The academic experts rated the majority of tasks easy or moderate when compared to the patient ratings. Navigation to personal story videos was rated as hard both by patients and experts. This may suggest the need to alter the website user interface by having aids for navigation (visible links at the top of page), simple and easily identifiable search bar and specific search results to the query and easy access to pages (multiple pathways with limited number of clicking and backtracking).¹⁵ Increasing patient involvement in their healthcare can increase the efficiency of communication between them and HCP.²⁵ Having an academic expert or HCP introduces or orients the patient to a website may assist in learnability (ability to easily learn how to use a website) and efficiency in obtaining information.^2,26

To increase the use of a PEM Web website, it must be perceived as usable and credible by the target group, people diagnosed with HNC as well as information providers who include general practitioners as well as specialists and allied health professionals.^27,28 Training the trainer workshops are recommended to familiarise healthcare professionals to important aspects of website usability and content so they may educate patients during their clinical interaction. Modifications that may contribute to making a website usable include the following: 1. Simple design and layout; 2. feedback from website to assist patients in completing a task such as ‘please enter your suburb to obtain the closest support group’; (3) consistent format on each page, for example, having an icon for a PDF or print symbol at the top of each page; (4) instructions on how to navigate to a destination such as ‘click here to get factsheets on my cancer’; (5) navigation with clear menus and links, (6) terminology familiar to patients, for example, tonsil cancer rather than oropharyngeal cancer; and, (7) learnability making it easy to learn about the website, for example, a video orientation to the website.²⁷

The main limitations of this study were potential selection bias with the consenting patients’ likely having a bias towards being high information seekers and users, prior information provided to patients potentially contributing to topic search bias and being post-treatment skewing information towards survivorship compared to treatment. However, as few as five people are needed to achieve a representative sample of end users using the cognitive walkthrough approach.²⁹ The perspective of culturally and linguistically diverse people who make up a proportion of the Australian population was excluded from the study as the website is only available in English. Moreover, there was no orientation to the website prior to usability testing which may have affected keystroke measures on each task. Some participants may be uncomfortable sharing their opinions whilst being prompted and recorded, although Think-Aloud has been shown to be a robust method in other studies.^30,31

Future research repeating the current study could assess the pre-and post-participant user experience with education level as covariate. Regardless, incorporating user testing of enhancements to the website architecture, design and content should be a routine part of the website maintenance. A randomised controlled trial evaluating the effect of the Head and Neck Cancer Australia website (once revised) on more clinically relevant outcomes such as adherence with medical advice, severity of side effects and swallowing ability would provide definitive evidence of the value of this approach to patient support. Accessibility may be further enhanced through training of HCP in use of the website and how best to introduce and orient their patients to the website.

Conclusion

The Head and Neck Cancer Australia website is accessible to patients, carers and HCPs for multi-media HNC information on a range of relevant topics including cancer information, health and well-being and support groups. Support and information obtained via the website can be maximised if people are orientated to it by the HCP during the face-to-face consultation. Iterative refinements are needed to improve the website appearance and navigation. Future studies are needed to ensure continuous quality improvement for all aspects of the site.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Joe Jabbour

References

Bray

Ferlay

Soerjomataram

, et al. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 2018; 68(6): 394–424. DOI: 10.3322/caac.21492.

Casebeer

Strasser

Spettell

, et al. Designing tailored Web-based instruction to improve practicing physicians’ preventive practices. J Medical Internet Research 2003; 5(3): e20–e20. DOI: 10.2196/jmir.5.3.e20.

Win

Hassan

Bonney

, et al. Benefits of online health education: perception from consumers and health professionals. J Med Syst 2015; 39(3): 27. DOI: 10.1007/s10916-015-0224-4.

Ellis

Speroff

Dittus

, et al. Diabetes patient education: a meta-analysis and meta-regression. Patient Educ Couns 2004; 52(1): 97–105.

Singh

Armstrong

Lipsky

. Preventing foot ulcers in patients with diabetes. JAMA 2005; 293(2): 217–228. DOI: 10.1001/jama.293.2.217.

Homer

Susskind

Alpert

, et al. An evaluation of an innovative multimedia educational software program for asthma management: report of a randomized, controlled trial. Pediatrics 2000; 106(1 Pt 2): 210–215.

Dodd

Waller

Marlow

. Human papillomavirus and head and neck cancer: psychosocial impact in patients and knowledge of the link - a systematic review. Clin Oncol (R Coll Radiol) 2016; 28(7): 421–439. DOI: 10.1016/j.clon.2016.02.012.

Borowitz

Ritterband

. Using the Internet to teach parents and children about constipation and encopresis. Med Inform Internet Med 2001; 26(4): 283–295.

Fleisher

Buzaglo

Collins

, et al. Using health communication best practices to develop a web-based provider-patient communication aid: the CONNECT study. Patient Educ Couns 2008; 71(3): 378–387. DOI: 10.1016/j.pec.2008.02.017.

10.

Ullrich

Jr Vaccaro

. Patient education on the internet: opportunities and pitfalls. Spine 2002; 27(7): E185–E188.

11.

Levin-Zamir

Peterburg

. Health literacy in health systems: perspectives on patient self-management in Israel. Health Promotion Int 2001; 16(1): 87–94. DOI: 10.1093/heapro/16.1.87.

12.

Azar

Gabbay

. Web-based management of diabetes through glucose uploads: has the time come for telemedicine? Diabetes Res Clin Pract 2009; 83(1): 9–17. DOI: 10.1016/j.diabres.2008.09.055.

13.

Brockbank

Miller

Owen

, et al. Pretreatment information on dysphagia: exploring the views of head and neck cancer patients. J Pain Symptom Manage 2015; 49(1): 89–97. DOI: 10.1016/j.jpainsymman.2014.04.014.

14.

Jabbour

Wykes

Milross

, et al. Examining clinicians’ perceptions of head and neck cancer (HNC) information. Asia Pac J Clin Oncol 2018; 14: e428–e433. DOI: 10.1111/ajco.12858.

15.

Garett

Chiu

Zhang

, et al. A literature review: website design and user engagement. Online J Commun Media Technol 2016; 6(3): 1–14.

16.

Hornbæk

Frøkjær

. Comparison of techniques for matching of usability problem descriptions. Interacting Comput 2008; 20(6): 505–514. DOI: 10.1016/j.intcom.2008.08.005.

17.

McTavish

Gustafson

Owens

, et al. CHESS (comprehensive health enhancement support system): an interactive computer system for women with breast cancer piloted with an underserved population. J Ambul Care Manage 1995; 18(3): 35–41.

18.

Danaher

McKay

Seeley

. The information architecture of behavior change websites. J Med Internet Res 2005; 7(2): e12–e12. DOI: 10.2196/jmir.7.2.e12.

19.

Danielson

McCauley

Gros

, et al. SiHLEWeb.com: development and usability testing of an evidence-based HIV prevention website for female African-American adolescents. Health Informatics J 2014; 22(2): 194–208. DOI: 10.1177/1460458214544048.

20.

Uğraş

Gulsecen

Çubukçu

, et al. Research trends in web site usability: a systematic review. Conference paper. International Conference of Design, User Experience, and Usability. July 2016. DOI: 10.1007/978-3-319-40409-7_49

21.

Taylor

Stables

Matata

, et al. Website design: technical, social and medical issues for self-reporting by elderly patients. Health Informatics J 2013; 20(2): 136–150. DOI: 10.1177/1460458213488382.

22.

. Reflections: the value of patient support groups. Otolaryngol Head Neck Surg 2017; 156(4): 587–588. DOI: 10.1177/0194599817697030.

23.

Checklin

Bain

Bath

, et al. Patients’ perspectives on what makes a better care experience while undergoing treatment for oropharyngeal dysphagia secondary to head and neck cancer. Dysphagia 2020; 35(4): 702–716. DOI: 10.1007/s00455-019-10077-y.

24.

Reen

Muirhead

Langdon

. Usability of health information websites designed for adolescents: systematic review, neurodevelopmental model, and design brief. Review. J Med Internet Res 2019; 21(4): e11584. DOI: 10.2196/11584.

25.

Win

Susilo

. Personal health record systems and their security protection. J Med Syst 2006; 30(4): 309–315.

26.

Nahm

Preece

Resnick

, et al. Usability of health web sites for older adults: a preliminary study. Comput Inform Nurs 2004; 22(6): 326–334, quiz 335-6.

27.

Hinchliffe

Kerry Mummery

Applying usability testing techniques to improve a health promotion website. Health Promot J Austr 2008; 19(1): 29–35.

28.

Di Pietro

Nguyen

Doran

. Usability evaluation: results from “evaluation of mobile information technology to improve nurses’ access to and use of research evidence”. CIN: Comput Inform Nurs 2012; 30(8): 440–448.

29.

Nielsen

. Why you only need to test with 5 users, 2000. Retrieved from, https://www.nngroup.com/articles/why-you-only-need-totest- with-5-users/.

30.

Charters

. The use of think-aloud methods in qualitative research an introduction to think-aloud methods. Brock Educat J 2003; 12: 68–82. DOI: 10.26522/brocked.v12i2.38.

31.

Jaspers

Steen

Bos

, et al. The think aloud method: a guide to user interface design. Int J Med Inform 2004; 73(11): 781–795. DOI: 10.1016/j.ijmedinf.2004.08.003.

A web-based comprehensive head and neck cancer patient education and support needs program: Usability testing