Utility and usability testing of the FrESH (free from E-Cigarettes and smoking for health) web application: A qualitative study

Abstract

Background

Adolescent tobacco and e-cigarette use remains a major public health concern due to early nicotine dependence. Although digital cessation tools exist, only few have been systematically evaluated. The FrESH (Free from E-Cigarette and Smoking for Health) Web App was developed as part of the Stop Nicotine Addiction Among Secondary School Students (SNAASSS©) peer-led school intervention in Malaysia. This study reports a formative evaluation of the web app through expert utility and adolescent usability testing.

Methods

A qualitative formative evaluation was conducted in two phases. Utility testing involved medical and user experience experts using Cognitive Task Analysis (CTA), think-aloud protocols, and semi-structured interviews. Usability testing was subsequently conducted with adolescent cigarette smokers, vape users, and dual users in a secondary school. Recorded interviews were transcribed verbatim and analyzed via thematic content analysis using NVivo v.14 software.

Results

Experts evaluation focused on utility, identifying core strengths in educational content, self-monitoring tools, and motivational feedback mechanisms (e.g., progress tracking, money saved), while recommending simpler language, clearer dashboard outputs, and improved consent readability to enhance value. Usability testing with adolescents found the app learnable, visually appealing, and supportive, with good comprehension when content was structured. However, barriers included text-dense information, unfamiliar terminology, ambiguous tracker values, limited system feedback, navigation uncertainty, alongside concerns regarding data privacy.

Conclusion

The FrESH Web App shows strong potential as a school-integrated digital cessation support tool for Malaysian adolescents. Findings provide actionable guidance to improve content clarity, feedback interpretability, navigation support, and youth-friendly consent processes. Further mixed-method evaluation across multiple schools is warranted to assess engagement and effectiveness.

Introduction

Tobacco use remains one of the most preventable causes of disease and death worldwide. Globally, approximately 38 million adolescents aged 13–15 years are current users of tobacco, representing about 10% of this age group. ¹ In Malaysia, e-cigarette use among secondary school students increased from 9.8% in 2017 to 14.9% in 2022. ² This shift in adolescent nicotine consumption patterns signals the need for interventions that address the evolving landscape of nicotine dependence.

Tobacco products expose users to thousands of toxic substances, including chemicals that cause cancer, cardiovascular disease, and chronic lung conditions. ³ These risks are especially concerning for adolescents, as exposure to tobacco smoke during this developmental period has been shown to impair lung growth and contribute to mild airway obstruction, ⁴ while nicotine disrupts brain maturation and rapidly promotes dependence even with low levels of use. ⁵ The increasing preference for e-cigarettes among adolescents is attributed not only to misperceptions of reduced harm compared to conventional cigarettes but also to the appeal of flavoured products, which have been shown to increase interest in use and complicates cessation efforts. ⁶ In addition to long-term risks, adolescent e-cigarette use has been linked to serious acute lung injuries, including E-cigarette or Vaping-Associated Lung Injury (EVALI), which led to thousands of hospitalizations, many involving adolescents and young adults, in the United States during the 2019–2020 outbreak, highlighting the urgent need for prevention strategies. ⁷

These long-term and emerging acute health risks have prompted global and national action. In response, Malaysia has aligned with the World Health Organization’s Framework Convention on Tobacco Control (WHO FCTC) through initiatives such as pictorial health warnings on cigarette packs, bans on tobacco advertising and promotion, smoke-free zones in public places, and increased excise duties on tobacco products. ⁸ The Generational End Game (GEG) clause in Malaysia’s Control of Smoking Products for Public Health Bill 2022 sought to prohibit the sale of tobacco and vape products to individuals born on or after 1 January 2007, but was excluded from the final 2023 bill following legal and policy debates over Its constitutionality and feasibility. ⁹ Considering these gaps, digital health interventions can serve as valuable tools to complement policy measures and behavioural support in helping adolescents overcome nicotine addiction. This is particularly relevant given that adolescents are highly engaged with smartphones and digital platforms, with over 92% reporting daily internet use and widespread ownership of mobile devices. ¹⁰ Given this high level of digital engagement, adolescents represent a crucial target population for smoking cessation interventions, particularly within the school setting, where peer influence is both prominent and potentially strategic for behaviour change. Middle adolescence, typically defined as ages 14 to 19, is a period marked by increased susceptibility to peer pressure, identity exploration, and risk-taking behaviours, including tobacco and e-cigarette use.^11,12 In Malaysia, secondary school students typically range in age from 13 to 17 years, aligning closely with this critical developmental window. This age range also corresponds with the World Health Organization’s definition of adolescence, which spans from 10 to 19 years. ¹³ By focusing on this subgroup, the intervention not only aligns with international classifications and national school structures but also targets a developmental stage where effective behavioural support may have the greatest impact.

However, current evidence suggests that most smoking cessation apps have been designed for adults and often fail to address the cognitive, social, and developmental needs of younger users, limiting their effectiveness in supporting adolescent smoking cessation. ¹⁴ To address this gap effective interventions must appeal to adolescents, support behaviour change, and provide engaging, age-appropriate tools to help them resist nicotine use. This study responds to that need by developing and evaluating a digital intervention tailored specifically for adolescent smoking and vaping cessation.

In response to these challenges, the Department of Primary Care Medicine at Universiti Teknologi MARA (UiTM) launched the Stop Nicotine Addiction Among Secondary School Students (SNAASSS) © programme in 2020. ¹⁵ The SNAASSS © programme is a school-based, peer-led smoking cessation intervention targeting adolescent cigarette smokers, vape users and dual users in Malaysian secondary schools. “Peer-led” refers to smoking cessation support delivered by trained undergraduate medical students who act as peer counsellors for secondary school students, facilitating group sessions and reinforcing behavioural change strategies within the school setting.

The FrESH (Free from E-cigarettes and Smoking for Health) Web App is a browser-based digital intervention developed to modernize and adapt the Buku Kendiri Pelajar©, a validated behavioral self-help booklet. Originally designed for a four-week structured counseling program delivered by trained medical student peers, the transition to a web-based platform was intended to better resonate with the lifestyle of digital-native adolescents aged 13–17. The application digitizes the booklet’s core curriculum, guiding users through essential modules including nicotine dependence assessment, the 4R model (Relevance, Risks, Rewards, and Roadblocks), and personalized quit planning. By providing tools for habit tracking and withdrawal management in a mobile-responsive format, the app offers universal accessibility across smartphones and computers without the friction of an app store download, requiring only an active internet connection for use.

A rigorous formative evaluation was undertaken to ensure the FrESH Web App meets the needs of its target audience: adolescent smokers, vapers, and dual users in secondary schools. This study employed a qualitative framework to bridge the gap between clinical utility and technical usability. The primary objectives were: (1) to assess the application’s perceived utility among clinical professionals and digital health experts; (2) to evaluate usability and user experience among adolescent end-users through recorded interviews and direct observations; and (3) to generate formative insights to guide refinement and future outcome-focused evaluation of the application.

Methods

Development of FrESH web app

The FrESH (Free from E-Cigarettes and Smoking for Health) Web App is a digital intervention developed to support adolescent smoking and vaping cessation within a school-based programme. The app was designed to complement the Stop Nicotine Addiction Among Secondary School Students (SNAASSS©) programme, a peer-led group intervention implemented in secondary schools in Malaysia. The FrESH Web App was designed to target adolescents who were active smokers, vape users or dual users and who had expressed a willingness to quit. Behavioural interventions are generally more effective among individuals who have reached later stages of readiness to change, as described in the Transtheoretical Model of Behaviour Change. ¹⁶ Rather than functioning as a standalone intervention, the web app serves as a supplemental digital tool that reinforces behavioural goals introduced during peer-led counselling sessions. This hybrid model allows adolescents to engage in self-monitoring, reflection, and motivational activities between face-to-face sessions, while maintaining structured support within the school environment.

The technical design and development of the FrESH Web App guided by the iterative model of the Software Development Life Cycle (SDLC) framework. ¹⁷ In addition, the web app was informed by empirical evidence on adolescent smoking cessation, behaviour change theory, and prior evaluations of youth-focused digital health interventions. ¹⁸ Building on this evidence-informed foundation, specific web app features were intentionally selected to reflect core components of effective adolescent cessation support, including assessment, self-monitoring, motivational reinforcement, and guided quit planning.

Accordingly, the app includes several key modules designed to address adolescents’ readiness to quit and support behaviour change. For example, assessing baseline nicotine dependence is important for tailoring cessation support, as dependence measures help guide treatment intensity and cessation planning.^19,20 Assessments of motivations for quitting are associated with greater engagement and self-directed behaviour change; intrinsic motives such as health concerns and financial savings enhance quit intentions among adolescents. ²¹

The FrESH Web App incorporates standardized measures of nicotine dependence, including the Fagerström Test for Nicotine Dependence (FTND) and its adapted version for vaping (eFTND), because higher dependence levels are linked to lower cessation success and require differentiated support strategies.^19,22 Among adolescents, higher levels of nicotine dependence have been shown to predict lower likelihood of successful cessation, suggesting that dependence severity should be considered when tailoring motivational feedback and cessation planning.^23,24

Self-monitoring features, including daily smoking or vaping diaries and progress dashboards, were incorporated based on strong evidence demonstrating that self-monitoring and feedback are effective behaviour change techniques for smoking cessation and sustained behaviour change among adolescents and young adults.^25,26 Digital self-tracking increases awareness of smoking patterns, reinforces progress, and supports self-regulation, which are important mechanisms for positive health behaviour change. ²⁷

Immediate and tangible feedback, such as visualisation of money saved, was included in recognition of evidence that adolescents are more responsive to short-term, concrete rewards than to distant health outcomes, and such features can enhance motivation and sustained engagement in cessation activities. ²⁸ Evaluations of smoking cessation apps indicate that features such as progress tracking, visual feedback, and gamification are core engagement functions and are associated with higher motivation and self-efficacy, which are mechanisms linked to successful quit attempts and sustained engagement in cessation activities. ²⁹

Readiness-to-quit assessments were incorporated to support motivation and tailoring of cessation support. Assessing readiness for change is grounded in the Transtheoretical Model and has been widely applied in adolescent health interventions to strengthen commitment and guide behavioural strategies appropriate to the user’s stage of change.^16,25 Educational content was intentionally designed to be brief and modular, as evidence suggests that adolescents engage more effectively with concise, visually supported digital health information than with extensive text-heavy content. ³⁰

The design choices at feature and structural levels were guided by evidence indicating that digital smoking cessation interventions are more effective when integrated with face-to-face counselling or school-based programmes, as hybrid models allow digital tools to reinforce behavioural goals introduced during interpersonal interactions and improve continuity of support. ³¹

A storyboard was created to visualize the user journey and simulate interface navigation as shown in Figure 1. This facilitated iterative input from the research team, and UI/UX designers, allowing refinement of flow, logic, and adolescent-appropriate visual design prior to full-scale development.

Figure 1.

Storyboard illustrating the FrESH web app integrated with the SNAASSS© programme.

Content of the FrESH web app

The FrESH Web App is a mobile-friendly, web-based platform designed to support adolescent smoking cessation. It comprises ten core sections, with their content and functions summarized in Table 1. Whereas, Figure 2, details the navigation flow and user journey.

Table 1.

Summary of the FrESH web app sections, content, and functions.

No.	Web app section	Content	Function
1.	Terms and conditions and privacy policy	This section outlines the app’s usage rules, responsibilities, and data privacy practises. Ensures compliance with data protection standards and highlights the need for parental consent for underage users. It promotes transparency, safeguards data, and builds trust with adolescent users and their guardians.	To inform users of their rights, responsibilities, and privacy protections, and ensure parental approval before app use.
2.	Welcome page	This section introduces the app’s purpose and features, in clear, age-appropriate language.	Provides a positive first impression and ensures users have basic understanding about the function of this web app
3.	Tell Us About Yourself	This section collects the user’s smoking or vaping history including number of cigarettes and vape puffs per day, cost of cigarette packs and vape pods, weekly expenditure, duration of smoking or vaping, and previous quit attempts.	To collect baseline smoking history and support individualized quit planning.
4.	Reasons for Quitting	This section collects the user’s personal motivations for quitting smoking or vaping, using predefined options and open-ended input.	To help users reflect on and document their quitting motivations, reinforcing commitment.
5.	My Profile	This section captures the user’s personal information, including name, age, gender and date of birth.	To store and display basic user identity information.
6.	Fagerström Test	This section includes the Fagerstrom Test of Nicotine dependence (FTND) for cigarette users, and a modified version for vape users (eFTND). The eFTND uses the same six questions as the FTND, with adjustments in terminology and context to reflect vaping behaviours. 32 Both versions provide a brief explanation of the test’s purpose, the scoring system, the six-item questionnaire, and test result interpretation.	To help users understands the test’s purpose and assess their level of nicotine dependence for cigarettes or vaping. This enables the web app to provide personalized feedback to support quitting efforts.
7.	Smoking Cessation Readiness Level	This section provides a self-assessment of the user’s readiness to quit smoking or vaping. It includes a simplified explanation of the stages of change based on the Transtheoretical Model of Behavioural Change (TTM) from pre-contemplation to maintenance to support user reflection.	To assess the user’s motivation and preparedness to quit.
8.	Dashboard	This section displays a countdown to the quit smoking date, a daily progress tracker showing cigarette or vape puffs usage, and the amount of money saved. It also presents the user’s previously selected reasons for quitting to sustain motivation.	To provide visual feedback on quit date progress, daily smoking or vaping reduction, promote financial awareness, and reinforce motivation to quit.
9.	My Diary	This section allows users to log their daily cigarette or vape use, amount of money spent, desire to smoke using a slider scale, and any withdrawal symptoms experienced.	To support daily self-monitoring of smoking or vaping behaviours, track financial expenditure, cravings, and raise awareness of cravings and withdrawal symptoms to encourage behavioural reflection.
10.	Information Corner	This section offers educational videos and articles on the harms of smoking and vaping, the benefits of quitting, the S.T.A.R (Set, Tell, Anticipate challenges, Remove) and techniques to manage urges and withdrawal symptoms.	To deliver engaging educational content that enhances knowledge, develops coping skills, and empowers users to successfully quit smoking or vaping.

Figure 2.

User navigation flow in the FrESH web app.

The visual design and user interface of the FrESH Web App enhance accessibility and adolescent engagement through a sequence of intuitive screens. The onboarding experience includes motivational messages, Terms and Conditions, and privacy policy. Following this, users are guided through a welcome page that outlines the web app’s main objectives and sets the tone for self-directed smoking cessation. Key functional areas such as the Dashboard for progress tracking, the Smoking Cessation Readiness Level for self-assessment, and My Diary for daily behaviour logging, reflecting the interactive structure of the app. The Fagerström and modified eFTND modules, used to assess nicotine dependence for cigarette and vape users respectively. Lastly, educational content including multimedia and written information in the Information Corner is organised thematically across several subpages.

Study 1. Utility testing: Evaluations by medical and user experience experts

Within human-centred design and usability evaluation frameworks, utility and usability represent related but distinct constructs. Utility refers to the extent to which a system provides the necessary functions and features to enable users to achieve their intended goals. ³³ In contrast, usability focuses on how effectively, efficiently, and satisfactorily users are able to interact with those functions when performing tasks. In the development of digital health interventions, evaluating both utility and usability is recommended to ensure that the system not only contains appropriate behavioural and functional components, but can also be easily understood and used by the intended users.^33,34 Therefore, this study adopted a two-stage evaluation approach: expert utility testing to assess the relevance and functional appropriateness of the web application, followed by usability testing with adolescent end users to evaluate interaction, navigation, and user experience.

Utility testing refers to the evaluation of whether a system or application provides the essential features and functions that users need to accomplish their goals.^33,35 It focuses on determining whether the necessary features are present, implemented accurately, and capable of supporting user tasks effectively. In this study, utility testing was conducted by medical and user experience experts to ensure that the FrESH Web App provided the necessary functionality to support adolescent smoking cessation.

Study design

The utility testing phase consisted of a qualitative formative evaluation conducted with subject matter experts to assess the functional appropriateness, content relevance, and overall usefulness of the FrESH Web App as a supplementary smoking and vaping cessation tool for adolescents. This phase represented an alpha-level evaluation aimed at identifying early design issues and ensuring that the intervention content and features aligned with behavioural objectives and the needs of the target population.

A qualitative approach was employed using Cognitive Task Analysis (CTA) alongside a concurrent think-aloud protocol. Experts were invited to interact with key components of the web app while verbalising their observations, expectations, and critiques in real time.

Study participants

Utility testing was conducted at a public medical university facility in Klang Valley. A total of six experts participated, comprising family medicine specialists, psychiatrists, respiratory physicians, and user experience experts with professional experience in adolescent health, smoking cessation, app development, or utility testing. Inclusion criteria were medical or user experience experts with professional experience in adolescent health, smoking cessation, digital health, or health informatics. Experts who were members of the FrESH Web App development team were excluded.

Sampling and sample size

Experts were purposively recruited to provide multidisciplinary perspectives on the clinical relevance and functional utility of the FrESH Web App. Sampling was guided by qualitative principles of thematic saturation, defined as the point at which feedback from at least three consecutive interviews did not yield new themes or insights regarding app utility. ³⁶ Data collection continued until saturation was reached. This approach aligns with established guidance for heuristic and utility evaluations, suggesting that expert panels of approximately three to five participants are typically sufficient to identify the majority of usability issues, with additional evaluators contributing diminishing returns.^37,38

Data collection

Data were collected through semi structured in-person in-depth interviews conducted by SAJ. Reflexivity was maintained throughout the study through continuous reflection and the use of reflexive notes to minimise potential interviewer bias. Before each interview, participants received a verbal briefing and written materials, including a user manual for the FrESH Web App (Appendix 1) and task scenarios (Appendix 2). The demonstration version of the FrESH Web App is accessible at: https://fresh-uitm.web.app.

Participants were given approximately 30 minutes to familiarise themselves with the web app before commencing the interview. A semi-structured interview guide was used to facilitate discussion, with probes tailored to participants’ real-time experiences during task completion. Expert utility testing was guided by cognitive task analysis (CTA) principles and comprised 11 structured task scenarios reflecting key functionalities of the FrESH Web App. Participants were encouraged to use a think-aloud approach, verbalising their thoughts, challenges, and reflections while navigating the web app. The summary of topic guides used for this interview are shown in Table 2. Participants were encouraged to verbalize their thoughts as they worked through the tasks, providing insight into their reasoning, challenges, and reflections. Reflexivity was maintained throughout the study through continuous reflection and the use of reflexive notes to minimise potential interviewer bias.

Table 2.

Task-based topic guide components for both the expert utility testing and adolescents’ usability testing.

Task	FrESH web app component	Sample interview guide questions
Task 1 and Task 2	Sign up and sign in page	What are your thoughts on the design of the login page?
Task 3	Tell Us About Yourself	Are the questions easy to understand?
Task 4	Dashboard (before daily diary entry)	What are your thoughts on the data displayed in the ‘Dashboard’ section?
Task 5	Profile	Is it easy to update and save your information?
Task 6	Reasons for Quitting	Is the list of reasons easy to read and relevant to adolescents?
Task 7	Fagerstrom Test (Week One)	What are your thoughts on the test results (e.g., colour scheme, percentage, and description)?
Task 8	Your Smoking Cessation Readiness Level (Week One)	Is it easy to choose a readiness level that matches your willingness to quit smoking?
Task 9	Information Corner	Is the information useful?
Task 10	My Diary	How do you feel about the process of recording your smoking habits and symptoms?
Task 11	Dashboard (after daily diary entry)	Is the information displayed accurate?

All interviews were audio- and video-recorded with participants’ consent. Recordings were securely stored on a password-protected institutional drive, de-identified during transcription, and assigned study IDs prior to analysis. Only the research team had access to the recordings. Identifiable information was removed during transcription, and participants were assigned study IDs. De-identified transcripts were used for analysis.

Data analysis

All audio recordings were transcribed verbatim using Transkriptor AI, then reviewed and corrected by SAJ. To ensure anonymity, pseudonyms were assigned to participants. SAJ, a fluent speaker of both Malay and English, personally performed the transcription and translation of the Malay interviews. Translation accuracy was rigorously verified by back-comparing the English transcripts against the original Malay audio and text to ensure linguistic and conceptual equivalence. Another bilingual member of the research team, SAR reviewed a subset of transcripts to confirm meaning equivalence, and any discrepancies were resolved through discussion. A bilingual member of the research team reviewed a subset of transcripts to confirm meaning equivalence, and any discrepancies were resolved through discussion.

Transcriptions were supplemented with observational field notes documenting non-verbal cues and usability-related behaviours. Data were analysed using inductive thematic analysis and managed using NVivo (version 14). The analysis followed Braun and Clarke’s six-phase thematic analysis framework, which includes familiarisation with the data, generation of initial codes, searching for themes, reviewing themes, defining and naming themes, and producing the final analytic report. ³⁹ Coding was conducted independently by three researchers (SAJ, NMN, SAR) to enhance analytical rigour. The researchers subsequently met to compare coding decisions, discuss discrepancies, and reach consensus through iterative discussion. This process ensured that the final themes accurately represented patterns within the dataset. Data saturation was considered achieved when no new codes or themes emerged from successive interviews, indicating sufficient depth and coverage of the phenomenon under investigation. ³⁶ Themes were reviewed for internal coherence and alignment with the full dataset, then assigned clear interpretive labels. Researcher triangulation, reflexive memoing, and an audit trail were maintained throughout to enhance the credibility and trustworthiness of the analysis.

This qualitative study was reported in accordance with the Consolidated Criteria for Reporting Qualitative Research (COREQ) guideline, which provides a 32-item checklist for transparency in interviews and focus group studies. The COREQ checklist is included as a supplementary file.

Result

A total of six experts participated in the utility testing phase of the FrESH Web App. All participants had prior experience in either, adolescent health, smoking cessation management, or digital intervention development. Their demographic details are presented in Table 3.

Table 3.

Sociodemographic and professional background of experts (N= 6).

Demographic characteristics	Frequency (n, %)
Age (years)
Mean (SD)	46.0 (±7.67)
Range	36–58
Gender
Male	2 (33.3%)
Female	4 (66.7%)
Race
Malay	5 (83.3%)
Indian	1 (16.7%)
Type of Phone
Android	3 (50.0%)
iPhone	3 (50.0%)
Area of expertise
Family medicine specialist	2 (33.3%)
Psychiatrist	1 (16.7%)
Respiratory physician	1 (16.7%)
User experience expert	2 (33.3%)
Years of experience
Family medicine specialist A	20 years
Family medicine specialist B	15 years
Psychiatrist	30 years
Respiratory physician	10 years
User experience expert A	10 years
User experience expert B	15 years
Experience handling adolescent smokers
Family medicine specialist A	20 years
Family medicine specialist B	15 years
Psychiatrist	30 years
Respiratory physician	15 years
User experience expert A	10 years
User experience expert B	15 years
Experience in app development
Family medicine specialist 1	1- 2 years
Family medicine specialist 2	5 years
Psychiatrist	1 - 2 years
Respiratory physician	1 - 2 years
User experience expert 1	5 -10 years
User experience expert 2	> 10 years

Themes of utility testing

Utility testing with expert participants yielded a rich set of themes and categories highlighting key features of the application’s functional alignment and clinical relevance. The experts identified core strengths of the application while providing critical insights into necessary refinements to optimize the platform’s value for end-users. Five themes were identified: (i) Supporting adolescents to understand cessation information, (ii) Enabling self-monitoring and goal tracking, (iii) Guiding adolescents through the quitting journey, (iv) Sustaining motivation and engagement, and (v) Data privacy, confidentiality and consent of the FrESH Web App. These are shown below in Table 4.

Table 4.

Themes and categories identified from expert utility testing.

No.	Themes	Categories
1.	Supporting adolescents to understand cessation information	2. Accessible, adolescent-friendly delivery formats
		3. Age-appropriate content
		4. Clarity of cessation information
		5. Comprehension of key assessments
		6. Information load
		7. Clarity of language and terminology
		8. Interpretation of assessment outputs
		9. Misinterpretation risks
		10. Need for refinement of cessation content
		11. Visual presentation supports content comprehension
2.	Enabling self-monitoring and goal tracking	1. Accuracy and reliability of tracking displays
		2. Clarity of tracking labels
		3. Design supports understanding of tracking features
		4. Recommended refinements to improve tracking inputs
		5. Tool supports efficient tracking
		6. Understanding tracking outputs
		7. Usability of self-monitoring inputs
3.	Guiding adolescents through the quitting journey	1. Clarity of process steps
		2. Clear onboarding requirements and sign-up guidance
		3. Interface supports navigation and task completion
		4. Navigation barriers that disrupt independent use
		5. Features discoverability
		6. Overall usability perceptions
		7. Recommended improvements to support task progression
		8. Smooth task flow
		9. Task-flow challenges
		10. Technical issues affecting smooth use
		11. Visual cues supporting task progression
4.	Sustaining motivation and engagement	1. Engagement boosters and support for long-term use
		2. Motivational features that support quitting effort
		3. Perceived rewards
		4. Personalisation and tailored motivation support
5.	Data privacy, confidentiality and consent	1. Consent content
		2. Consent language clarity
		3. Consent presentation and guidance
		4. Entry of personal data fields

Theme 1: Supporting adolescents to understand cessation information

Expert feedback indicated that FrESH Web App provides cessation information that is largely appropriate and suitable for an adolescent audience, with content perceived as relevant to quitting and presented in a generally supportive tone as shown in Figure 3. Experts judged the informational sections as useful for reinforcing understanding of health effects and quitting-related knowledge, suggesting the app has strong potential as an educational cessation aid.

Figure 3.

Digital cessation modules in the FrESH web app. (3a) Multimedia-driven vaping education: (3b) The S.T.A.R quitting guidance module.

However, experts also identified areas where minor edits could improve clarity and accessibility, including simplifying phrasing and reducing potentially technical or dense wording. This suggests that while the content is fit-for-purpose, small refinements to language and structure would strengthen comprehension and reduce the risk of misunderstanding. Experts also recommended reducing text density in the Information Corner by using more engaging formats such as videos, cartoons, or visual aids, to better match adolescents’ reading preferences and improve engagement.

Theme 2: Enabling self-monitoring and goal tracking

Experts viewed the self-monitoring and progress features as valuable core functions that align with cessation support, particularly the ability to track behaviour and observe change over time. The image in Figure 4 illustrates the app’s “My Progress” dashboard, which enables adolescents to self-monitor key cessation indicators such as days not smoked, cigarettes or vape puffs reduced, and total money saved. Experts identified this progress-tracking function as a central strength of the FrESH Web App, supporting behaviour change by translating user-entered data into meaningful feedback and reinforcing goal achievement over time.

Figure 4.

Self-monitoring and progress-tracking dashboard feature in the FrESH web app.

At the same time, experts flagged risks to utility where tracking outputs may be misinterpreted, especially when values appear negative or when calculation logic is not transparent. They recommended adding clearer explanation (e.g., brief text, tooltips, or labels) so that adolescents can understand why certain outputs occur and trust the feedback provided.

Theme 3: Guiding adolescents through the quitting journey

Experts were largely able to navigate key tasks such as signing up, completing profile questions, and entering quitting-related information, indicating that the app’s core journey is learnable and workable. The interface and dashboard were often described as simple and visually calm, supporting task completion without excessive complexity.

Nevertheless, several moments disrupted smooth task flow, including uncertainty about password requirements, quit-date entry, and what to do after saving profile details. These usability barriers point to the value of stronger on-screen guidance and system feedback (e.g., clearer step-by-step prompts, confirmation messages, “next step” cues), which would reduce reliance on trial-and-error and support adolescents to progress independently through the quitting journey.

Theme 4: Sustaining motivation and engagement

Motivational elements such as progress tracking, money saved, and reminders of reasons for quitting were consistently identified as strengths that can support continued effort and reinforce cessation goals. Figure 5 illustrates the dashboard presents progress indicators (e.g., vape reduction and money saved) alongside a personalised “Reasons for Quitting” reminder list, supporting adolescents’ motivation and continued engagement with cessation goals. Experts viewed these features as capable of helping users stay focused, providing a sense of movement and accountability through visible indicators of change.

Figure 5.

Motivational dashboard feature in FrESH web app.

However, engagement may decline if sections feel repetitive or overly text-heavy. Participants suggested that maintaining interest would be improved through greater variety in content delivery (e.g., more visuals, shorter texts, interactive elements) and increased personalisation. Together, these findings indicate that FrESH has a strong motivational foundation but would benefit from design refinements that better match adolescents’ preferences for quick, visually supported engagement over repeated use.

Theme 5: Data privacy, confidentiality and consent

Experts raised important concerns about whether adolescents can meaningfully understand consent-related information, particularly when Terms and Conditions are presented in dense or formal language (see Figure 6). Experts noted that long, text-heavy content may reduce adolescents’ comprehension and encourage passive agreement rather than informed consent. Although the consent process was generally accessible in the interface, experts feedback suggested that comprehension may be limited, increasing the risk of passive agreement rather than informed consent.

Figure 6.

Data governance and informed consent interfaces within the FrESH web app showing the terms and conditions 6(a) and Privacy policy with user consent confirmation 6(b).

In addition, some experts expressed discomfort around getting users sharing personal information such as phone numbers, highlighting the need for clear justification, minimisation of required fields, and reassurance about how data are used and protected. Improving consent readability (e.g., summaries, plain language, key-point formats) and clarifying the purpose of data collection would strengthen ethical transparency and support trust among adolescent users.

Table 5 presents illustrative quotes from the expert panel that substantiate these themes, offering deeper insight into the perceived clinical relevance and value of the FrESH Web App’s core features.

Table 5.

Illustrative quotes supporting utility testing themes (experts).

Themes	Illustrative quote
Theme 1: Supporting adolescents to understand cessation information	The information is suitable for teenagers and covers important quitting topics.” (P1, Psychiatrist).
Theme 2: Enabling self-monitoring and goal tracking	Tracking progress and money saved can reinforce quitting behaviour.” (P3, Respiratory physician)
Theme 3: Guiding adolescents through the quitting journey	“In general, from the start, everything was structured and followed step by step. Good.” (P4, Digital Expert).
Theme 4: Sustaining motivation and engagement	“It’s good they can actually see things. When they see, ‘Oh, I could have saved this much,’ or ‘I’ve saved this much,’..that’s kind of motivating.” - referring to the dashboard content (P2, Family Medicine Specialist).
Theme 5: Data privacy, confidentiality and consent	“The privacy policy should be written in more youth-friendly language, so students can understand it.” (P5, Digital Expert)

Study 2. Usability testing: Evaluation by end users

Study design

Usability testing is essential during the development of new digital health applications because it evaluates whether the system enables intended users to complete tasks effectively, efficiently, and with satisfaction, which are core dimensions of usability. ³⁵ A qualitative usability evaluation was conducted using CTA and a think-aloud approach. This design enabled real-time exploration of how adolescents interacted with key functions of the FrESH Web App, while capturing their perceptions, challenges, and feedback through semi-structured interviews.

Study participants

Usability testing was conducted among adolescent smokers from a selected secondary school participating in the SNAASSS© programme in Gombak District, Selangor. The school is a public daily secondary school for both boys and girls situated in an urban setting, with an enrolment of approximately 1,200 students. ³⁸ It operates on a dual-session system, with Forms 3 to 5 attending morning session and Forms 1 to 2 attending afternoon session. The student population predominantly comes from middle to lower socioeconomic backgrounds. ³⁸ Participants were eligible for inclusion if they were aged 13–17 years, identified as active smokers by their school counsellor, enrolled in the SNAASSS© programme, and owned or had access to a smartphone capable of accessing the FrESH Web App. Participants were excluded if they were unable to understand or communicate in either Malay or English, lacked reliable internet access, or did not obtain parental consent or provide personal assent. Although the study was open to all eligible adolescent smokers and vapers, only male participants were recruited due to the absence of identified female smokers in the referral list provided by the school counsellor during the study period.

Sampling and sample size

A purposive sampling approach was used to ensure representation of adolescents with varied smoking and or vaping histories. Participants were recruited with the assistance of school counsellor. Information sheets and consent forms were distributed to those interested and eligible participants and their parents or guardians. Those who agreed to participate provided written informed consent prior to involvement in the study. To support recruitment, participants received a RM10 bookshop voucher as a token of appreciation.

The sample size for usability testing was guided by the principle of data saturation, with recruitment continuing until no new usability concerns, navigation issues, or thematic insights emerged from successive interviews. ³⁵ A target of 10 to 15 adolescent smokers was set in line with established recommendations for usability evaluations involving end-user testing, where this range is typically sufficient to identify most major usability issues, particularly in general user interface studies. ³⁹

Data collection

Data were collected through in-person usability testing interviews involving adolescent participants aged 13–17 years. Each usability testing session followed a structured task-based process. Participants were first given a brief orientation to the study and the purpose of the FrESH Web App. They were then asked to complete predefined task scenarios while interacting with the web application using their own mobile devices. During task completion, participants were encouraged to verbalise their thoughts using a think-aloud approach, and the interviewer conducted a semi-structured in-depth interview simultaneously to explore participants’ reasoning, challenges, and reactions while navigating the app. This approach enabled real-time exploration of usability issues as they occurred during interaction with the system. Interviews were conducted by SAJ at the participating schools to maximise convenience and minimise disruption to students’ schedules.

Reflexivity was maintained throughout data collection via reflective journaling to minimise researcher bias during interpretation. Prior to each session, participants received a user manual (Appendix 1) and a set of task scenarios (Appendix 2) to familiarise themselves with the FrESH Web App. The same version of the FrESH Web App was used for both expert utility testing and adolescent usability testing; refinements based on expert feedback were planned for subsequent development cycles and were not implemented prior to adolescent testing. The same set of scenarios was used for both utility and usability testing; however, for usability testing, the scenarios were translated into Malay to enhance comprehension among adolescent participants. They were given approximately 30 minutes to explore the app independently using their personal mobile devices.

The same topic guide used during utility testing, developed based on the app’s user journey and structured using CTA principles, was also employed for usability testing. However, the guide was translated into Malay to accommodate participants’ primary language and enhance fluency during the think-aloud process. Participants were encouraged to verbalize their thoughts throughout the session to provide insights into their understanding, challenges, and user experience. All sessions were audio-, video-, and screen-recorded to support triangulated analysis of verbal and non-verbal cues.

All interviews were audio- and video-recorded with participants’ consent. Recording files were downloaded immediately after each session and stored on a password-protected, access-restricted institutional drive. Only the research team had access to the recordings. Identifiable information was removed during transcription, and participants were assigned study IDs. De-identified transcripts were used for analysis.

Data analysis

Data analysis for the usability testing phase followed the same procedures as those applied during the utility testing. Thematic analysis was employed to identify patterns and themes within the data set, ensuring consistency in the approach across both expert and end-user evaluations.

Results

The usability testing phase involved 14 adolescent participants, all male, with more than two-thirds from upper secondary (15–17 years) and the remainder from lower secondary (13–14 years). The other demographic details are presented in Table 6.

Table 6.

Demographics characteristics of the adolescent’s users (N= 14).

Demographic characteristics	Frequency (n, %)
School level
Lower secondary (13–14 years)	3 (21.4%)
Upper secondary (15–17 years)	11 (78.6%)
Gender
Male	14 (100.0%)
Race
Malay	6 (42.9%)
Indian	8 (57.1%)
Type of Phone
Android	11 (78.6%)
iPhone	3 (21.4%)
Type of smoker
Cigarette only	4 (28.6%)
Vape only	7 (50.0%)
Dual users (cigarette and vape)	3 (21.4%)

Themes of usability testing

Qualitative usability testing identified six overarching themes that shaped participants’ experiences of using the smoking/vaping cessation app (see Table 7): (1) Content comprehension, (2) Dashboard and trackers understanding, (3) Task interaction and navigation clarity, (4) Data privacy and ethical concerns, and (5) Visual design and user satisfaction. Across themes, participants generally perceived the app as clear and supportive, but usability barriers emerged around information density, interpretability of progress metrics, interaction affordances (e.g., sliders and default selections), and pacing/feedback clarity. These findings highlight both strengths that supported engagement and design friction points that may disrupt sustained use, particularly for adolescent users.

Table 7.

Themes and categories identified from usability testing.

No.	Themes	Categories
1.	Content clarity	2. Onboarding message clarity
		3. Educational content is understandable and informative
		4. Content length increases cognitive load
		5. Readibility and visual presentation support comprehension
		6. Technical terms reduce understanding
		7. Video readability and pacing
2.	Dashboard and tracker feedback	1. Misinterpretation of colour-coded negative values
		2. Progress metric meaning and data transparency
		3. Trackers promote motivation and self-monitoring
		4. Labels or categories may confuse users or feel judgemental
		5. Interpretation of assessment outputs
		6. Data reliability and persistence
		7. Dashboard clarity
3.	Task interaction and navigation clarity	1. Task completion and ease of use
		2. Structured inputs improve usability
		3. Sliders require clearer cues to support learnability
		4. Default selections and unclear feedback increase uncertainty
		5. Micro-frictions and error messages interrupt task flow
		6. Discoverability issues
		7. Weak post-action guidance
		8. System pacing and transition control
4.	Data privacy and ethical concerns	1. Privacy concerns and comfort sharing information
		2. Parental consent feasibility and adolescent autonomy
		3. Low engagement with terms and conditions
		4. Terms and conditions comprehension
5.	Visual design and user experience	1. Visual appeal
		2. Readibility and visual comfort
		3. Acceptability and experience of use
		4. Engagement and content preference

Theme 1: Content clarity

Participants’ ability to comprehend the app was shaped by onboarding message clarity, the perceived understandability and informativeness of educational content, and the extent to which readability features supported meaning-making. Overall, adolescents were able to identify the app’s purpose early and interpret key educational modules such as reasons for quitting, withdrawal symptoms, readiness levels, and dependence-related information, suggesting that the app provided accessible health education when content was presented in clear, structured formats. Figure 7 shows that comprehension was strengthened when information was broken into simple sentences, organised in a logical flow, and supported by visual presentation cues such as bullet points and images, indicating that adolescents benefitted from content that reduces effort and supports quick scanning rather than requiring sustained reading.

Figure 7.

Information corner section of the FrESH web app showing educational content on the negative effects of smoking/vaping (a), health benefits of quitting (b), and common withdrawal symtpoms (c).

However, comprehension was not uniform across all content types. Content length increased cognitive load when information was perceived as excessive or text-dense, resulting in tendencies to skim or disengage, which implies that adolescents may value content usefulness but still require brevity and progressive disclosure to sustain attention. Technical terms reduced understanding when unfamiliar health vocabulary or chemical names were presented without contextual explanation, suggesting that adolescent comprehension barriers were often driven by terminology accessibility rather than lack of interest. Video readability and pacing further shaped comprehension, as fast transitions and long on-screen text limited the ability to read and process key messages, even when participants found the video format engaging. Collectively, these findings indicate that content comprehension in adolescent-facing digital health apps depends not only on what information is delivered, but also on how it is packaged, paced, and scaffolded to match adolescents’ reading habits, cognitive load tolerance, and preference for visually supported learning.

Theme 2: Dashboard and trackers feedback

Participants’ understanding of dashboard feedback was influenced by the extent to which the app transparently conveyed progress meaning, assessment outputs, and the calculation logic behind tracked indicators. While many adolescents could recognise the motivational intent of trackers such as the countdown and progress indicators, interpretation became inconsistent when progress metrics were not immediately explainable. Adolescents frequently misinterpreted the color-coded negative metrics - rather than recognizing red text and negative integers as indicators of behavioral consequences (e.g., health risks or financial loss), users often attributed these cues to system errors, penalties, or invalid data entry. This suggests a misalignment between the app’s visual semiotics and the users’ established mental models of digital feedback. This pattern suggests adolescents relied on familiar interface conventions from other apps in which red typically signals “wrong” or “failure,” highlighting the need for explicit signposting when colour is used as a semantic cue.

Understanding improved when participants could link dashboard values to prior diary entries and selections, demonstrating that data transparency and input–output traceability were essential for building trust in self-monitoring features. However, uncertainty about the source of values and the rationale behind certain metrics reduced confidence in dashboard accuracy, indicating that adolescents require dashboards to be not only visually engaging but also interpretable and explainable. In addition, labels and category descriptors related to spending and consumption sometimes created confusion or felt judgemental, implying that feedback systems should use neutral, non-stigmatising language to preserve acceptability. Data reliability and persistence further shaped perceived credibility, as consistent dashboard information after re-sign-in strengthened trust, whereas missing or inconsistent data undermined confidence in the app’s usefulness as a progress-tracking tool.

Theme 3: Task interaction and navigation clarity

Participants demonstrated high task success across primary workflows, including registration and self-assessment, suggesting robust baseline usability. Structured input mechanisms (e.g., dropdowns and calendar selections) were appraised as highly intuitive, effectively reducing interaction friction. By prioritizing recognition over recall, the interface lowered the burden of manual entry and supported a more seamless data-entry process for the adolescent demographic. Examples include the nicotine history module and the interactive calendar tool. Adolescents identified these features as primary drivers of navigational ease, favoring structured recognition tasks (choosing from menus) over open-ended data generation (see Figure 8).

Figure 8.

Structured input interfaces in the FrESH web app for smoking history (a) and quit-date selection (b).

Despite successful task completion, interaction confidence decreased when the interface required guesswork or when system status was unclear. Sliders were a common source of uncertainty due to unclear scale cues and limited visibility of numeric values, and default selections created doubt about whether an answer had been actively chosen or automatically assigned. These issues imply that adolescents depend heavily on visible system state, immediate feedback, and clear affordances to feel confident their input has been recorded correctly, especially during first-time use. Micro-frictions such as repetitive questions, keyboard obstruction during text entry, quit-date entry difficulties, and error messages interrupted task flow and increased cognitive load, suggesting that adolescent-facing digital health tools must prioritise low-friction data entry, error prevention, and recovery support to reduce drop-off and sustain engagement over repeated use.

Navigation clarity was shaped by discoverability issues, the strength of post-action guidance, and the degree of system pacing and transition control. While participants could locate familiar functions such as diary icons or profile-related options, uncertainty emerged when features were less visible or when adolescents relied on expectations from other apps to infer where functions “should” be located. This indicates that adolescents navigate using recognition and familiar patterns, and when signposting is weak, they may hesitate or require external confirmation even if the pathway exists.

Navigation confidence was further influenced by what happened after actions were completed. Participants sometimes reported uncertainty after saving profile information and required manual return to the dashboard or additional reassurance before proceeding, reflecting limited post-action guidance and weak “next step” cues. In addition, rapid system transitions reduced users’ ability to process information and maintain control, particularly when screens changed quickly before adolescents could read or decide what to do next. Collectively, these findings imply that adolescent-facing apps benefit from stronger wayfinding support through clear labels, consistent navigation placement, explicit next-step prompts, and pacing that supports comprehension rather than assuming users will infer the correct flow.

Theme 4: Data privacy and ethical concerns

Ethical usability concerns emerged around privacy comfort, parental consent feasibility and adolescent autonomy, low engagement with Terms and Conditions, and limited comprehension of consent-related content. Adolescents varied in their comfort sharing personal information such as phone number or full name, suggesting that willingness to engage is shaped not only by usability but also by perceived risk and trust. The tension between parental consent feasibility and adolescent autonomy further highlighted practical barriers to adoption, as participants indicated that peers may differ in whether they can or would seek parental approval, which has implications for real-world uptake even when the app is technically usable.

Participants also demonstrated low engagement with Terms and Conditions, reflecting an informed-use gap driven by habitual skipping behaviour and lengthy, complex text. Although some participants understood the general purpose of consent and privacy statements, others expressed confusion about key terms, checkboxes, and account responsibilities, indicating that ethical information was not consistently accessible in practice. These findings imply that ethical usability in adolescent digital health tools requires consent processes that are not only compliant but also developmentally appropriate and usable, including simplified summaries, layered disclosure, and clear explanations of why information is collected and how it is protected.

Theme 5: Visual design and user satisfaction

User satisfaction was shaped by visual appeal, readability and visual comfort, overall acceptability and experience of use, and engagement and content preferences. Adolescents generally responded positively to the app’s colour scheme, bright and attractive welcome interface, and the use of images, graphics, and animations, indicating that visual design contributed to a friendly and approachable first impression. Adolescents’ understanding of assessment feedback was strongly influenced by how clearly the app translated tracked data into simple, visually meaningful outputs, including the use of clear labels, colour cues, and graphic summaries that supported quick interpretation of progress and dependence levels. Figure 9 image displays the Fagerström Test result using a simple score summary and colour-coded circular indicator to communicate nicotine dependence levels in a clear, adolescent-friendly format. Participants reported that such straightforward graphics enhanced readability, reduced cognitive effort, and supported understanding of quitting-related progress through quick visual interpretation.

Figure 9.

Nicotine dependence scoring interface.

Beyond aesthetics, acceptability and experience of use were reflected in overall positive impressions and approval of the app interface, while engagement preferences highlighted the importance of concise content, simple wording, and visually supported learning materials. These findings suggest that adolescent engagement is strengthened when the app feels easy to consume and enjoyable to interact with, but sustained use depends on balancing attractive design with clarity and cognitive simplicity. In this sense, visual design functioned not only as “appearance,” but also as a usability mechanism that reduced resistance, supported comprehension, and enhanced perceived usefulness of the app experience.

Table 8 presents illustrative quotes that substantiate these themes, providing direct evidence of the users’ experiences with the platform’s navigation, visual semiotics, and task interaction. These excerpts highlight both the intuitive strengths of the interface and the specific ‘micro-frictions’ that impacted user confidence.

Table 8.

Illustrative quotes supporting usability testing themes (adolescents).

Theme	Illustrative quote
Theme 1: Content Comprehension	“I understand the reasons for quitting because it’s short and easy to read.” (S2, lower secondary)
Theme 2: Dashboard and Trackers Understanding	“The money saved part is good because I can see how much I can save” (S7, upper secondary).
Theme 3: Task Interaction and navigation clarity	“I liked the calendar...it helped me select the quit date easily”- referring to Tell Us About Yourself section (S10, lower secondary)
Theme 4: Data privacy and ethical concerns	“I think… it explains who can use this app…and how my data are stored” - Referring to T&C contents (S6, upper secondary)
Theme 5: Visual Design and User Satisfaction	“The video has good animations and interesting facts” (S11, upper secondary)

Discussion

This study evaluated a web-based smoking cessation application integrated with the SNAASSS© programme to support Malaysian adolescent smokers. Aligned with Malaysia’s Health White Paper (2023), ⁴⁰ the FrESH Web App addresses the global demand for scalable, user-centered digital health interventions for youth. Unlike standalone applications such as quitSTART or This is Quitting, FrESH is uniquely embedded within a peer-led school curriculum, ensuring cultural and linguistic relevance while bridging the gap between digital tools and real-world implementation. It also responds to the global call for user-centred, scalable digital health interventions targeting youth. ⁴¹ Utility and usability testing provided comprehensive formative insights, identifying both strengths and specific areas requiring refinement to align with best practises in adolescent digital health interventions.

Expert perspectives on clinical utility

Unlike many adolescent smoking cessation apps that focus on narrow functions (e.g., craving management or educational modules), ⁴² the FrESH Web App adopts a multi-faceted approach, incorporating: 1) Educational content on smoking-related harms and quitting benefits; 2) Behavioural tracking tools (e.g., smoking or vape diaries); 3) Motivational features (e.g., money saved, reasons for quitting); and 4) Planning aids (e.g., quit date setting, STAR method guidance). This integrative design reflects evidence that combining knowledge, self-monitoring, and motivational reinforcement is more likely to sustain adolescent engagement and behaviour change.

Current existing adolescent-targeted smoking cessation apps, such as quitSTART, ⁴³ and This is Quitting ⁴⁴ have undergone formal evaluation, but operates as a standalone intervention and is not integrated within a structured school-based programme. These apps also offered minimal interactivity and have not been adapted for local cultural or linguistic settings. Robinson et al. (2020) and Crane et al. (2018) similarly observed that most youth smoking cessation apps offer limited tailoring, theoretical grounding, or real-world implementation. ⁴⁵ In contrast, the FrESH Web App is embedded within the SNAASSS© peer-led school programme in Malaysia, aligning its digital features with peer-led counselling. It was also subjected to two phases of structured evaluation, utility testing by experts and usability testing by adolescent users, supporting its contextual relevance and iterative refinement. While gamified and interactive elements are still limited, they have been prioritised for future development. This locally grounded, theory-informed, and empirically tested design distinguishes FrESH Web App as a promising school-integrated tool for adolescent smoking cessation.

Expert feedback particularly praising the digitization of the S.T.A.R. framework and FrESH Web App as an adolescent cessation aid, specifically highlighting the integration of educational modules with self-monitoring functionality. Experts considered the cessation information to be appropriate in tone and relevant to quitting, suggesting that the app provides a useful foundation for reinforcing cessation knowledge and supporting school-based counselling. ⁴⁶ While content was generally viewed as fit-for-purpose, experts recommended minor edits to improve clarity and accessibility, including simplifying phrasing and reducing technical wording, echoing findings that health information must be tailored for developmental comprehension. ⁴⁷

Experts also identified self-monitoring and goal tracking as a core functional strength, viewing the dashboard as central to reinforcing progress and encouraging quitting effort consistent with behavior change theories.^48,49 However, experts flagged “text-dense” modules as a barrier to health literacy, recommending simplified language to match adolescent developmental stages. Experts also noted that user engagement may diminish if adolescents cannot derive meaning from displayed metrics, particularly when the underlying computational logic is opaque or when values are presented as negative integers. This finding underscores the importance of algorithmic transparency in digital health feedback, which is essential for establishing and maintaining user trust within the adolescent demographic aligning with research showing that transparency in digital feedback strengthens user trust. ⁵⁰

The expert panel underscored the importance of immediate reinforcement in the FrESH app. By visualizing financial gains and progress indicators, the app leverages tangible incentives that resonate more effectively with adolescent reward processing than traditional long-term health warnings. Adolescents are particularly responsive to immediate and tangible rewards compared with distant health outcomes, making features such as money saved highly relevant for maintaining quit motivation. ⁵¹ Digital smoking cessation interventions that incorporate motivational feedback and progress reinforcement are associated with improved engagement and increased self-efficacy. ⁵² However, experts cautioned that engagement may decline if content becomes repetitive or overly text-based. Sustained engagement is one of the most persistent challenges in digital cessation tools, particularly among youth populations. ⁵³

Data privacy and consent emerged as critical utility considerations in this expert evaluation. Experts stressed that adolescents may not meaningfully understand dense Terms and Conditions if presented in formal or technical language. Transparent and youth-friendly privacy communication is essential to ensure informed consent and foster trust in interventions targeting minors. ⁵⁴ Strengthening consent readability through plain-language summaries and emphasising confidentiality safeguards may therefore improve both ethical integrity and user trust.

Usability testing (Adolescents feedback)

Adolescent feedback confirmed that FrESH is both learnable and functional, with high completion rates across key workflows, a hallmark of effectiveness and efficiency in mHealth design. Most participants able to complete key workflows, consistent with usability definitions of effectiveness and efficiency. ⁵⁵ Participants demonstrated a strong conceptual grasp of the app’s purpose, particularly when information was delivered via structured, high-readability formats, supporting prior work that readability and visual structure improve health app comprehension. ⁵⁶

However, comprehension was non-uniform and varied depending on content type and delivery. Adolescents reported reduced understanding when information was overly text-dense or included unfamiliar technical terms, highlighting the need for cognitive accessibility in adolescent design. ⁵⁷ Furthermore, while video content was engaging, rapid pacing and high “on-screen text density” impeded information processing and understanding, consistent with research on multimedia learning that stresses pacing and redundancy avoidance. ⁵⁸

In relation to the dashboard and trackers, adolescents recognised the motivational intent of progress indicators, but misinterpretation became common when values were not immediately explainable particularly when negative values were displayed in red. This aligns with findings that colour cues carry strong semantic associations and must be supported by labels/tooltips to avoid misinterpretation. ⁵⁹ Trust and understanding improved when adolescents could link dashboard outputs to their diary entries, echoing user trust theories in HCI. ⁶⁰

User confidence in task interaction was contingent upon interface transparency. When the system state was ambiguous, users experienced a decrease in confidence, even if the task was ultimately completed, supporting human-computer interaction research indicating that visible system state and immediate feedback are fundamental to usability. ³³ Micro-frictions such as ambiguous sliders and default selections increased cognitive load, which is known to disrupt flow and reduce sustained engagement in digital health contexts. ⁶¹

Navigation clarity was shaped by discoverability and guidance after action, aligning with work demonstrating that weak wayfinding increases cognitive effort and reduces satisfaction. ⁶² Ethical concerns, such as comprehension of consent, reflect broader digital health literature highlighting the “informed use” gap and the need for developmentally appropriate consent processes. ⁶³

Finally, adolescents’ satisfaction with visual design including layout, colours, and animations supports research showing that aesthetics influence perceived ease of use and enjoyment, which can heighten engagement.^64,65 However, sustained engagement requires balancing visual appeal with cognitive simplicity, consistent with the aesthetic-usability effect described in HCI.

Key insights and alignment with global standard

Both expert and adolescent feedback reflected common challenges in adolescent-focused health apps. Findings are consistent with previous evaluations that stress the need for interactive features to sustain motivation (e.g., gamification supports engagement and healthy behaviours in youth digital interventions).^66,67 Although gamification was not directly raised by participants in this formative evaluation, future iterations may explore interactive elements (e.g., quizzes or rewards), as prior research suggests these may enhance adolescent engagement.

Simplified adolescent-friendly language and clear navigation and visual feedback cues have also been highlighted as critical design considerations for youth-centred digital health solutions. ⁶⁸ Minimisation of cognitive and technical burdens aligns with broader evidence emphasising usability and accessibility as foundations for sustained use and engagement in mHealth tools. ⁶⁸ These findings collectively align with national and international frameworks that advocate for adolescent digital health solutions designed to prioritise usability, accessibility, security, and user-centred design. ⁴¹ Although no national technical guidelines exist for adolescent smoking cessation apps, this study adopted global best practices by applying iterative design principles focused on content clarity, task flow, and user engagement to strengthen usability and user experience.^41,68

Planned refinements

Based on the utility and usability findings, several improvements will be implemented in the next development cycle. These include simplifying technical language, enhancing visual cues, standardising data input formats, and adding interactive features such as progress tracking, personalised feedback, and gamified elements (e.g. quizzes and challenges) to improve adolescent engagement and task completion confidence.^67,69 Following these refinements, structured usability evaluations can be conducted using tools such as the Malaysian-adapted mHealth App Usability Questionnaire (M-MAUQ) to quantitatively assess key usability domains such as ease of use, learnability, and user satisfaction. ⁷⁰ The use of M-MAUQ would complement the current formative findings by enabling objective measurement and benchmarking of usability following refinements.

Implications

The findings of this study carry important implications for practise, policy, and future research. The FrESH Web App offers a potentially scalable digital solution to support adolescent smoking cessation within school-based health programmes in Malaysia. Its integration into the existing SNAASSS© programme, a peer-led group intervention, demonstrates how digital tools can potentially complement current interventions and support user engagement among young users. From a policy perspective, the app’s development aligns with Malaysia’s digital health priorities by demonstrating how user-centred and scalable solutions can strengthen adolescent health services. For future development, incorporating gamification, interactive quizzes, and personalised feedback will be critical to optimise engagement and support behavioural change. Further research is recommended to validate the refined web app using standardised usability measures and assess its impact on long-term cessation outcomes and sustained user engagement. Future work should explore the scalability and broader implementation of the FrESH Web App across multiple school settings and regions. The scalability of the FrESH Web App is rooted in its design as a low-friction, browser-based intervention integrated into the existing SNAASSS school programme. Unlike native apps, its web-based format provides universal accessibility across various devices without app store downloads, making it highly suitable for students from diverse socioeconomic backgrounds. To achieve broader implementation, the intervention should be embedded within national school health promotion strategies through collaboration with the Ministry of Education Malaysia and public health authorities.

Strengths and limitations

A key strength of this study was the dual focus on expert utility and adolescent usability testing, enabling a comprehensive formative evaluation of the FrESH Web App. The inclusion of both expert and adolescent perspectives, and the use of CTA methods, provided rich, task-specific feedback that informed planned refinements. ⁷¹ The task-based testing approach generated detailed feedback on core workflows (e.g., onboarding, dashboard interpretation, assessments, and diary entry), enabling identification of both design strengths and usability barriers relevant to adolescent users. In addition, conducting sessions in the school setting supported ecological relevance by reflecting the intended implementation context.

Several limitations should be noted. First, the study was conducted in a single school, which may limit transferability to other settings. The sample included only male adolescents, and experiences may differ among female adolescents or other demographic groups. Interviews were conducted in Malay and translated into English; although translations were checked for consistency and accuracy, minor loss of contextual meaning remains possible. Recruitment was also affected by logistical challenges related to parental consent, which required additional follow-up and may have influenced participation. Finally, this formative study did not include standardised quantitative usability measures (e.g., SUS), limiting benchmarking against other digital interventions. Future work should incorporate mixed-method evaluation across multiple schools to strengthen generalisability and assess longer-term feasibility, acceptability, and potential effectiveness.

Conclusion

This formative evaluation provides evidence that the FrESH Web App is generally usable and perceived as useful by both experts and adolescent users, with strengths in educational content, self-monitoring features, and overall visual design. Usability testing indicated that adolescents could complete key workflows, while also identifying areas where comprehension and interaction confidence were reduced by text-dense content, ambiguous progress indicators, limited system feedback, and consent-related complexity. Expert utility feedback further supported the relevance of the app’s cessation information and progress tracking functions, while highlighting the need for clearer explanations and more engaging content formats.

Overall, these findings offer actionable guidance to refine the FrESH Web App prior to broader implementation. With targeted improvements to content readability, dashboard interpretability, task guidance, navigation support, and ethical transparency, the app is well-positioned for further feasibility and acceptability evaluation in school-based settings, and subsequent research to explore engagement and potential effectiveness in supporting adolescent smoking cessation.

Supplemental material

Supplemental material - Utility and Usability Testing of the FrESH (Free from E-Cigarettes and Smoking for Health) Web Application: A Qualitative Study

Supplemental material for Utility and Usability Testing of the FrESH (Free from E-Cigarettes and Smoking for Health) Web Application: A Qualitative Study by Siti Anisah Jamaludin, Suraya Abdul-Razak, Nik Munirah Nik-Mohd-Nasir, Sazzli Shahlan Kasim, Hayatul Najaa Miptah, Lina Lohshini Kanoo, and Mohd Zuhairi Zainuddin in DIGITAL HEALTH.

Supplemental material

Supplemental material - Utility and Usability Testing of the FrESH (Free from E-Cigarettes and Smoking for Health) Web Application: A Qualitative Study

Supplemental material for Utility and Usability Testing of the FrESH (Free from E-Cigarettes and Smoking for Health) Web Application: A Qualitative Study by Siti Anisah Jamaludin, Suraya Abdul-Razak, Nik Munirah Nik-Mohd-Nasir, Sazzli Shahlan Kasim, Hayatul Najaa Miptah, Lina Lohshini Kanoo, and Mohd Zuhairi Zainuddin in DIGITAL HEALTH.

Supplemental material

Supplemental material - Utility and Usability Testing of the FrESH (Free from E-Cigarettes and Smoking for Health) Web Application: A Qualitative Study

Supplemental material for Utility and Usability Testing of the FrESH (Free from E-Cigarettes and Smoking for Health) Web Application: A Qualitative Study by Siti Anisah Jamaludin, Suraya Abdul-Razak, Nik Munirah Nik-Mohd-Nasir, Sazzli Shahlan Kasim, Hayatul Najaa Miptah, Lina Lohshini Kanoo, and Mohd Zuhairi Zainuddin in DIGITAL HEALTH.