Usability evaluation of the digital anger thermometer app

Abstract

The digital anger thermometer is a prototype for a mobile application (app) for use with adults in anger management treatment. The digital anger thermometer incorporates standards of software development in addition to anger management resources from the Substance Abuse and Mental Health Services Administration. The digital anger thermometer underwent a usability study conducted by five expert reviewers. The results indicate that it is easy to learn, efficient, and ergonomically sound. However, it does not offer support features or user-error tolerance. The digital anger thermometer prototype requires additional usability studies and comparative research in order for it to become an actual mental health app.

The digital anger thermometer (DAT) is a prototype for an anger management application (app). The purpose of this article is to review the features and usability of the DAT. Despite the availability of numerous mental health apps, few appear to consider usability evaluations in their development.^1,2 Such an evaluation is an important precondition for the usefulness and efficiency of apps. ³ It saves time and money in the developmental phases of a project and reduces overall system errors.^1,4 Consequently, the DAT mobile app prototype underwent an expert review to inform design revisions so that it may eventually become a useful tool in anger management.

With advances in mobile device technology, clients now have access to a wide range of therapy apps to supplement their treatment. ⁵ Mobile apps typically include tools for tracking moods, managing daily stressors, and improving overall mental health. ⁶ Apps are also capable of monitoring physical symptoms, providing biofeedback, guiding the user through therapeutic scripts, and offering additional self-help resources. There are an increasing number of individuals using mobile technology, and mental health apps allow them to participate actively in their own care. ⁷ At the same time, there is growing demand for expansive and cost-effective mental health services. ⁸ The portable and cost-effective nature of therapy apps could extend treatment to individuals with limited access to mental health resources.^9,10

Therapy apps address a wide range of issues such as mood disorders, anxiety, post-traumatic stress disorder, schizophrenia, and anger. In a mixed-methods field study using a mood-tracking app, participants reported lower mean scores on a series of single-dimension instruments for anger, anxiety, and sadness scales (p ⩽ .01 for all three) following its use. ¹¹ Apps assist in modulating emotions through self-regulation. Self-regulation reduces the severity of mental health symptoms, improves interpersonal relationship, and helps individuals live and work effectively. ¹² Anger problems affect relationships and physical health, ¹³ and anger management requires self-regulation and awareness.^14,15

Studies show that mood-tracking mobile devices reduced anger and stress levels. Of 78 trait-anxious participants, researchers noted lower scores in both stress reactivity and anxiety following the use of an app addressing cognitive biases.^11,16 Similarly, other researchers ¹⁷ reported an overall decrease in state anxiety while using the Mobile Stress Management app (F(1, 28) = 71.365, p ⩽ .001). Although the DAT requires future trials to assess its potential, it models some of the features of currently studied apps by tracking anger levels, cues, and patterns. The interactive nature of the DAT tools could increase client participation in treatment. ¹⁸ Apps similar to the DAT include the Mobile Stress and Anger Management Tool ¹⁹ (M-SAT) and Mobile Therapy app. ¹¹ The M-SAT is a support suite used conjointly by therapist and client. The M-SAT provides real-time support and cognitive behavioral therapy tools and tracks stress and anger levels through physiological sensors and an electronic anger meter (EAM). The interval scale on the M-SAT meter is the same as the meter within the DAT. However, unlike the M-SAT, the DAT directly links the user to online anger management resources and activities based on selected anger meter levels.

The mood-mapping app, known as Mobile Therapy, shares several qualities with the DAT. For one, it is an adjunctive treatment tool requiring weekly interviews between therapists and clients in order to work through anger levels and cues. The Mobile Therapy app’s real-time anger management resources resemble those found in the DAT. The Mobile Therapy app employs a selectable mood scale similar to the one featured in the DAT. The original version of the DAT was a hand-held device known as the EAM. ¹⁹ The EAM featured an analog scale of illuminated light-emitting diodes (LEDs), each representing a level of anger. Results of testing the EAM with participants revealed that subsequent designs would benefit from digitization to fit the growing trend of portable device usage. ²⁰ While the DAT is largely a digital version of the EAM, it improves upon it by incorporating software development protocol and additional resources on anger management.^21–25

The eventual purpose of the DAT is anger assessment. Individuals experience anger through physical symptoms, cognitions, and behaviors.^26,27 Anger is traditionally assessed through self-report inventories. State (anger episodes in real-time) and trait (pervasive patterns) anger are often included in anger inventories. ²⁸ When attempting to measure anger objectively, it is important to consider spectrums of anger that take into account the intensity, frequency, and manner of the episodes.^28–30 An anger spectrum might also include affective experiences from mild irritability to actual assault. ³¹ In order to measure anger with mobile devices, developers are beginning to incorporate spectrums of affect into their app designs.^11,30 The development of the DAT aligns itself with this practice by including an anger meter informed by sources while undergoing app development guidelines.

While the contribution of this article is to show the initial development of the DAT, completion of the app requires several stages beyond the usability evaluation. These include issuing additional evaluations to target users, field testing, deployment, and maintenance. ³² It is important to use participants that the app intends to serve during early phases of development. Testing for a majority of commercial apps occurs with participants for which the app is intended. ³³ Field testing commences when testers control for as many variables as possible in the user’s environment. ³⁴ After testing, deployment and maintenance involve configuring networks and adjusting security for true mobile use. ³⁵ Prior to any of these steps, basic design and usability must occur. ³⁶

The section “Features of the DAT” outlines the features of the DAT and the body of literature that informs its design. The section “Materials and methods” outlines the usability evaluation. The section “Results” discusses the implications and further work required for developing the DAT.

Features of the DAT

Every app consists of a series of displays and functions known as a graphical user interface (GUI). ³⁷ A GUI is an interfacing program containing a series of controls and displays that allow a user to interact with a computer. ³⁷ The DAT interface was designed for Microsoft Surface Pro tablets and emerging Windows-based mobile systems. An overview of the main DAT form is shown in Figure 1. The following section outlines the tools involved in constructing the prototype, followed by the layout. The layout includes a login frame, a database, main GUI, dialog boxes, and graphing tools.

Figure 1.

Overview of the DAT app.

Development tools

Developing the DAT required an integrated development environment (IDE), which is a set of programming tools used to develop software. ³⁸ Programming occurred on an ASUS X75A powered by an Intel Core i3 processor with a 4-GB RAM. SQLlite open-source software provided the framework for constructing the DAT database.

Login screen and security

The issue of security in mental health apps is ongoing. ³⁹ Security standards for the ubiquitous use of mobile mental health applications are lacking. ⁴⁰ Even now, the Health Insurance Portability and Accountability Act of 1996 (HIPAA) does not adequately address the security issues surrounding telehealth.

Security breaches of programs and pages are a major concern and increasingly make news headlines, such as the case of the Heartbleed bug, which affected mobile applications last year. ⁴¹ The SQLite database that holds user login information includes the SQLCipher encryption extension, a security component that actively defends against hacker intrusions. This extension works concurrently with the database to encrypt information further and is a standard choice for mobile applications. Additionally, the DAT login page allows a user to select a password up to 14 characters in length, offering adequate unpredictability. ⁴² The DAT also requires that the user enter both characters and numbers in the password.

Database

A database is a collection of digitally retrievable information. ⁴³ Mobile app design standards recommend that mobile applications with multiple storage requirements use a shared database. ⁴⁴ The DAT employs the open-source database known as SQLite. ⁴⁵ The SQLite software library uses an SQL database engine capable of lightweight data storage, making it ideal for most mobile apps.

Main form

DAT thermometer

The main form consists of several panels, and the main panel contains a thermometer graphic controlled by user input on a slider button (see Figure 2). Inspiration for the thermometer design came from the 10-scale anger meter featured in the Substance Abuse and Mental Health Services Administration (SAMHSA) anger management manual.^20,24 The thermometer currently provides a means for self-report with the hope that someday it can be used for more direct measurement of anger.

Figure 2.

The user selects an anger level, which populates the explanation and suggestions text fields.

The DAT uses an interactive 0–10 scale on all items and the default meter level is zero, signifying no anger, represented by a blue graphic color of the “mercury” at the bulbous base of the thermometer graphic. This incremental measure of arousal is an important component of assessing anger.^30,46 Starting at one, the information on anger levels begins to appear and the mercury color changes to green. As it progresses, the colors change to warmer yellow and red tones. This color scheme primarily follows temperature thermometers that monitor heat levels, popularly found in graphic anger meters. ⁴⁷ The thermometer connects two other panels, one for explaining the levels and the other for recording anger incidents.

Explanations panel

The explanations panel contains the results of each selected level. It explains the symptoms of anger on various levels of a spectrum.^23,31 The top end of the panel features a combination box containing suggestions for possible interventions tailored to each level of anger. Typical interventions include cognitive behavioral therapy, relaxation scripts, and progressive muscle relaxation. Previous anger management app research revealed the necessity for users to add their own interventions;^11,48 consequently, this panel offers an editable text field prompt for entering a personalized intervention.

After choosing an intervention, clickable hyperlinks (shortcuts between web pages) to online anger management resources appear (see Figure 3). Effective anger management requires a psychoeducational component. ⁴⁹ The sites offer information on cognitive behavioral therapy, ⁵⁰ mindfulness, ⁵¹ deep breathing, ⁵² and general relaxation. ⁵³ Because additional self-help resources reinforce anger intervention effectiveness,^54,55 a button on the bottom of the panel links the user to further reading on anger management.

Figure 3.

The user selects a hyperlink for improving communication skills. With weekly levels filled out, clicking the graph button produces a pie chart of the data.

Weekly levels panel

The weekly levels panel contains tools for monitoring and recording the highest level of anger throughout the week, an important component of anger management.^23,56 The weekly levels panel can record behavioral information across a designated time span for informing treatment, ⁵⁷ and the panel includes spaces known as text fields for recording the highest daily level of anger, along with fields for adding user notes regarding the level. With the ability to review anger levels over time, a user can record possible antecedents to the anger or further describe the anger incident for use in clinical sessions. ²⁵ The weekly levels panel also offers a means for the user to illustrate the data through a pie chart or bar chart, which provides users with quick visual representation of the anger data, although no evidence currently supports the use of such a graphic. The panel also features a save function for the charts so that users can review information over time.

Materials and methods

Usability evaluations help identify strengths and weaknesses of applications while offering feedback for improving their usability. ¹ They also offer a glimpse into actual user experiences prior to further development of software models so that developers can streamline the applications and reduce system errors. ⁵⁸ The DAT underwent a usability evaluation conducted by subject matter experts. The evaluation consisted of a questionnaire issued to the experts in a controlled setting. The scored instruments revealed information pertaining to the design and functions of the DAT.

Questionnaire

This study employed the IsoMetrics questionnaire ⁵⁹ (for the evaluation of DAT GUI). The IsoMetrics inventory is a user-oriented instrument based on part 10 of the ISO 9241. It presents with an appreciable amount of literature backing its use in the evaluation of health-based applications.^1,60–63 Validity studies of the instrument resulted from comparing the scale means of five major software systems. ⁵⁹ The questionnaire consists of 75 items, each ratable on a 5-point Likert-style scale within seven test dimensions: Suitability for the task, Self-descriptiveness, Controllability, Conformity with user expectations, Error tolerance, Suitability for individualization, and Suitability for learning.

Participants and procedure

The expert evaluation occurred between July and August 2014. Probabilistic models show that five raters are sufficient for usability testing. ⁶⁴ The minimum requirements for participation included an advanced degree in a computer field and at least 2 years of software development. Candidates included three app developers, a website designer, and a computer scientist. Mean age of these participants was 32.2 years (standard deviation (SD) = 2.32, range: 29–36).

The IsoMetrics manual ⁶⁵ guided the testing procedure. Each expert received an interview regarding their general experience and familiarity with systems similar to the DAT prior to receiving the test. Any questions that arose were addressed prior to handing out the questionnaire. The experts then received a brief review of the app and a paper version of the IsoMetrics questionnaire, followed by a reading of the text on the instructions statement.

There was a high selection rate on the IsoMetrics test. Scoring involved use of the mean to calculate the response averages. Negatively formulated items underwent a transformation prior to calculation. An analysis of the Suitability for the task subscale further illustrated positive and negative points of the DAT.

Results

The results of the questionnaire are in Table 1. The means and corresponding SD scores were calculated from the responses of the five raters. The standard cutoff score for the IsoMetrics usability evaluation is 3, which denotes a moderate score. ¹ Any rating higher than 3 is therefore considerable, but should be assessed depending on the context of use. ⁶⁵ The highest scores are in the Conformity with user expectations (M = 3.68, SD = 0.47) and Suitability for learning (M = 3.80, SD = 0.57) dimensions. The lowest scores are in the Error tolerance (M = 2.75, SD = 0.26) dimension, of which there is high agreement among the raters.

Table 1.

Usability evaluation scores of the DAT.

Dimension	M	SD
Suitability for the task	3.41	0.46
Self-descriptiveness	3.38	0.41
Controllability	3.28	0.26
Conformity with user expectations	3.68	0.47
Error tolerance	2.75	0.26
Suitability for individualization	3.40	0.75
Suitability for learning	3.80	0.57

All scores from n = 5 raters.

Individual evaluation of the items yielded additional information. The items with the highest scores included S.5, ease of retrieving about a certain entry field (4.00), and E.1, anticipating which screen will appear in next processing sequence (4.20). The lowest scored items pertained to S.3, understanding immediately what is meant by the messages displayed by the software (2.40), and F.6, entries are checked for correctness before further processing is initiated (1.80).

The Suitability for the task dimension covers the overall effectiveness and efficiency of software features. Delineation of this area reveals detailed information about the usability of the DAT. The results are given in Figure 4. Item descriptions with the corresponding means and SDs are as follows.

Figure 4.

Mean values for individual items in the Suitability for the task dimension.

Positively rated items include the following:

• A.1. The software forces me to perform tasks that are not related to my actual work (M = 4.60, SD = 0.89) (reverse-scored for statistical analysis).

• A.8. Too many different steps need to be performed to deal with a given task (M = 4.20, SD = 1.30) (reverse-scored for statistical analysis).

Negatively rated items include the following:

• A.3. The software lets me completely perform entire work routines (M = 2.20, SD = 0.54).

• A.4. The functions implemented in the software support me in performing my work (M = 2.40, SD = 0.89).

The positively rated items show a measure of efficiency in performing the tasks (above a rating of 3), although there appears to be an outlier in at least one of the responses of A.8 where the rating deviated significantly from the other respondents. The negatively rated items indicate incompleteness of the program and lack of support features. Considering these results, the DAT app could be improved as discussed below.

Discussion

A usability evaluation and anger management resources informed the development of the DAT. The results indicate that the overall ergonomic quality of the DAT is appreciable, and it presents with some measure of efficiency, but the expert sample size was low and the program is far from complete. The highest dimension score is in the Suitability for learning subscale. This indicates that it does not require much time to learn the DAT program and that some features of the software assist the user in becoming familiar with the layout. The lowest score is in Error tolerance. This essentially means that the program is not accommodating mistakes made by the user, and small user errors are yielding larger issues within the program. During testing of the DAT, several crashes occurred in the programming when users attempted to correct their information. Analysis of the individual items in the Suitability for the task dimension indicated a lack of support features when these events occurred. This affirms that the DAT requires debugging in this area and that the DAT can benefit from a user support component currently absent in its design. In addition to reworking the software aspect of the DAT, its ability to work as a mental health app requires further exploration.

Despite the growing popularity of mental health apps, there are numerous drawbacks to their use, including effectiveness, device limitations, security, and reimbursement. ⁶⁶ There is a lack of studies in app effectiveness, and newly formed committees are only now beginning to determine standards to ensure measures of reliability and validity. ⁶⁷ Although some practitioners are beginning to use apps as therapeutic tools, no long-term studies on their effectiveness in treatment have been done. ⁵ In a comprehensive 5-year literature search of 5464 abstracts, researchers found little evidence for the effectiveness of mental health apps. ⁹ Although there is some evidence for their effectiveness in use with engaging younger individuals in therapy, gaps in research remain to address app feasibility, cross-cultural considerations, technical literacy, and ethical issues. ⁶⁸

Concerning device limitations, limited memories of many portable devices are unable to hold large amounts of data. ⁶⁹ Small features can deter disabled and elderly individuals from using apps. ⁷⁰ Small device size forces mobile apps to run with reduced graphics and formats and it makes these devices easy to lose. Smaller devices are vulnerable to prying eyes and hackers, which pose a threat to the security of sensitive information contained in the devices. ⁷¹

Cost is a significant barrier to the widespread adoption of mental health apps. The use of mobile technology in treatment sometimes requires dialing costs incurred by clinician and client. ⁷² A large gap exists in reimbursement for healthcare technology: there is no billing code for the use of mobile apps in therapy. ⁴⁶ In time, with growing initiatives for technology and participatory care, this could change and mobile apps could be used for billable expenses. Until then, clients, clinicians, and researchers must cover the costs of app technology.

The DAT, like other mental health apps, requires controlled studies to determine true effectiveness. ¹¹ Subsequent versions of the DAT should incorporate the results of this study during further development. The next stage of usability engineering involves iterative development.^73,74 This includes a process of continuously evaluating and changing features of the DAT in order to identify major bugs in the software. In addition, a second usability study using non-expert users will help to identify additional issues with software ergonomics by offering a population resembling intended users of the DAT.^4,75

After refining the user interface of the DAT, its ability to measure anger requires examination. To assess effectiveness, a follow-up study might include a negative affect measure such as the State-Trait Anger Expression Inventory ²⁷ used in conjunction with trials of the DAT in a pre- and post-study of participant scores. If the DAT app undergoes such refinements, it could become an additional tool for use in anger management treatment.

Conclusion

Development of the DAT app included sources on anger management and a usability evaluation. A digital variation of the SAMHSA anger meter scale is at the core of its user interface. The main body of the DAT contains tools derived from established anger management interventions. Although the DAT includes some standards and sources in its design, the usability evaluation pointed to incomplete areas of the DAT that require further study before becoming a shareable app. Additionally, the DAT requires experimental studies with target users in order to assess its effectiveness.