User-Informed Medical Device Development: A Case Study for Pediatric Malnutrition Assessment

Abstract

Background. Mid-upper arm circumference (MUAC) has been extensively used to classify pediatric malnutrition. Recently, MUAC z score was recommended; however, the logistics of implementation were not addressed. This study examines the usability of a device that provides MUAC and corresponding z score range in a single device. Methods. This was a single-center study of nutrition services providers. The device was applied to children aged 2 months to 18 years admitted as inpatients or seen as outpatients. Surveys incorporated benchmarking questions, assessed the ease with which respondents could perform 6 critical tasks, and provided an open-ended question to elicit feedback. Users were surveyed monthly until saturation was reached. Survey data were analyzed in aggregate and cross-tabulated by the respondents’ experience with the device. Thematic analysis of the open-ended responses followed a structured approach. Results. Sixty device users responded to the survey 280 times. Respondents were female (100%) with a mean age of 45.2 ± 13.2 years and 9.6 ± 8.0 years in practice. Increasing device use was accompanied by significantly shortened measuring times (P < .001) and shifts in ease of performance for 5 of 6 critical tasks (P < .05). Open-ended response themes related largely to design and materials. These were used to iteratively refine the device. Conclusions. The active engagement of end users in the real-world testing of our nutritional assessment device allowed us to refine the innovation with special attention paid to the needs of dietitians. The result is a device the majority of dietitians found easy to use, efficient, convenient, and preferable to alternative measurement options.

Introduction

Mid-upper arm circumference (MUAC) has been widely used internationally to evaluate the nutritional status of children in medically underserved and resource-restricted settings.^1-4 With accumulating data to suggest that MUAC is a sensitive predictor of mortality in malnourished patients, ⁵ 2 nutrition-focused professional organizations in the United States drafted a consensus statement with recommendations for classifying and documenting pediatric malnutrition. ⁶ Included among the indicators were MUAC z score, decline in z score, and negative z score, which reflect how far a child deviates from the mean relative to children of the same age and gender. However, the consensus statement provided no guidance with regard to implementation of MUAC z score screening, which offers challenges beyond the simply measuring the circumference of the upper arm in millimeters.

The critical needs with regard to implementing assessments of MUAC z score are (1) normative data for a matched reference population from which the necessary parameters can be derived in order to estimate the distance of any child from the mean and (2) a device or other mechanism by which to apply these data to a given patient. With the first requirement satisfied via the publication of reference standards for US children 2 months through 18 years of age, ⁷ options for implementation were initially restricted to making a measurement of the MUAC in millimeters followed either by plotting the data on published MUAC growth curves and subsequently determining the approximate z score value from the graph, or performing calculation (manual or automated) of the z score value with the MUAC value and the lambda (L), mu (M), and sigma (S) values corresponding to the age and gender of the patient according to equation: Z = ((X/M)^L) − 1)/(LS), where Z represents the individual z score, X the individual MUAC value, and LMS the lambda, mu, and sigma values, respectively. In both cases, at least 1 device (a traditional tape measure) and 1 set of references (growth charts or LMS data tables) are required, making the process cumbersome for a busy clinician. Automating the calculations, as has been done in some online resources (https://peditools.org/cdcmuac/), can simplify implementation but it still requires a multistep process and real-time access to the electronic resource, which may not be available in all practice settings.

What was needed was a device capable of estimating nutritional status for any child, at any age, across a broad weight range, without regard for concerns related to inaccuracies arising from unmatched reference populations. What was desirable was a device that could determine nutritional status (as defined by z score) in a single step, without the need for ancillary reference charts and calculators. In response, we created a prototype device to facilitate MUAC z score assessment in accordance with the needs of our dietitians. The device (ie, MUAC z score tape) combines a circumferential tape measure, which enables determination of MUAC in centimeters/millimeters, overlaid with color bands that demarcate the z score range into which the patient of a given age falls. This study was designed to examine the usability and performance characteristics of this device in a real-world clinical setting while recording user perceptions and comments to inform device refinement.

Methods

Device Design

The MUAC z score tape was designed for construction using a semidurable, flexible, nonstretchable strip of plastic or plasticized paper with prespecified tolerances (Figure 1). In our first iteration we chose Tyvek, because of its nondeformable, lightweight, tear-resistant properties, and provided our users with the choice of a laminated or nonlaminated version. On the device is printed a traditional measuring scale depicted in metric units from which the MUAC can be determined, linear markings corresponding to selected age-ranges are overlaid with color-coded bands demarcating the z score range into which the child’s MUAC falls (Figure 1). The device has a broader “head” with 2 slits to accept the “tail,” which when threaded through the slits forms a loop. In our first iteration, we created 2 variations (1) a smaller tape providing z score ranges for children 2 to 36 months and (2) a larger tape providing z score ranges for children 4 to 18 years of age. Markings appeared on a single side and genders were pooled to mitigate the need to carry around more than 2 versions of the devices. In our second iteration, we retained 2 variations with pooled gender but printed on 2 sides to eliminate age gaps. The smaller tape ranged from 2 to 59 months and the larger 5 to 18 years of age. After production, the tapes were checked for dimensional accuracy using a National Institute of Standards and Technology certified ruler in compliance with ISO 9000 standards.

Figure 1.

Prototype device for the younger age group (not drawn to scale) with an expanded view to illustrate how the loop is formed.

Device Use

Circumferential measurements of the upper arm are made at the midpoint between the acromion and olecranon. In its linear state, the ruler on the device can be used to measure between the 2 anatomic landmarks and identify the midpoint. When ready to perform the measurement, the tail end is threaded into slit “A” and back out through slit “B” to create a loop (Figure 1). When positioned at the midpoint, the tail end of the tape is pulled until it is snug but does not compress the skin. Finally, the user identifies the color band corresponding to the age of the patient and records the z score range.

Participants

This was a single-center study of inpatient and outpatient providers within the Nutrition Services Department at a large Midwestern pediatric hospital in the United States. Prior to study involvement, all participating nutrition staff were required to complete malnutrition competency training, which included a didactic presentation with accompanying written resource materials, case studies, and practice sessions designed to introduce the staff to the first iteration of the device along with an assessment to ensure that the staff could correctly identify humeral landmarks, locate the midpoint of the upper arm, measure MUAC, and identify z score with the device. Instructions for performing MUAC measurements is provided as a supplemental figure (available online). Satisfactory completion of a competency quiz at the time of training and again 9 months later was required of all staff. The device was used in children 2 months to 18 years of age admitted as inpatients or seen in outpatient settings including, but not limited to adolescent medicine, cardiology, cleft palate, endocrinology, failure to thrive, gastroenterology, general pediatrics, hematology/oncology, nephrology, neurology, pulmonology, and weight management clinics.

Survey Design

Being similar in design to paper-based circumferential measuring tapes that are used for other anthropometric indications (eg, head circumference tapes) our analysis focused primarily on users’ interactions with the device. ⁸ Benchmarking questions included categorical assessments of how many times the respondent had used the device (<10, 10-25, 25-50, >50 times) and approximately how long it took them to perform device measurements (<30 seconds, 30-60 seconds, 1-2 minutes, >2 minutes) at the time of the survey. A 5-point Likert-type scale was used to assess the ease or difficulty with which the respondent could perform the critical tasks associated with the device. These included the following: identifying the midpoint of the upper arm, threading the tape through the openings to construct the loop, positioning the tape at the midpoint of the upper arm, identifying the correct position at which to read the tape output, reading the MUAC value in millimeters, and reading the MUAC Z score range. Likert-type scales were also used to evaluate perceptions related to the overall ease of use, efficiency, convenience, and acceptability of the device. Respondents were asked whether they preferred the new device or traditional methods for obtaining the relevant measures with the option to indicate that they did not have enough information to make this decision. Finally, participants were provided an open-ended comments section to specify challenges they experienced and describe changes they would like to see with the device.

Data Collection

Questionnaires were administered online using Research Electronic Data Capture tools hosted at Children’s Mercy. ⁹ Research Electronic Data Capture is a secure, web-based application designed to support data capture for research studies, providing (1) an intuitive interface for validated data entry, (2) audit trails for tracking data manipulation, and (3) automated export procedures for data downloads. To ensure voluntariness and mitigate concerns that decisions related to participation would affect the clinicians’ standing within the department, no identifiers were tracked and all surveys were submitted anonymously. Emails containing links to the survey were delivered monthly to all providers until saturation was reached and no new “early” users were responding.

Data Analysis

Survey data were analyzed in aggregate and cross-tabulated by the respondents’ experience with the device. Closed-ended questions were analyzed by determining the percent of responses in each category and examined using standard descriptive statistics. Comparisons between user experience groups were performed using a χ² test. Thematic analysis of the open-ended responses followed a structured approach as previously described, ¹⁰ where major and minor themes were identified and coded according to specific human factors and usability concepts. All statistical analyses were conducted in IBM SPSS version 24 (IBM Corp, Armonk, NY).

Ethical Approval and Informed Consent

Participants were enrolled with informed consent under a protocol that was reviewed and approved by the Institutional Review Board at Children’s Mercy Hospital (Institutional Review Board # 15040158).

Results

In total, 60 device users responded to the survey 280 times averaging 4.7 times per user. Our respondent population was 100% female with a mean age of 45.2 ± 13.2 years and a tenure of 9.6 ± 8.0 years in their current position. As previously reported, the population of children being measured included 5004 children ranging from 2 months to 18 years (53% male) with weights and lengths ranging from 2.59 to 206 kg and 36 to 196 cm, respectively. ¹¹ Stratification by device experience revealed 30 responses for participants having used the device <10 times, 45 responses when participants had used the device 10 to 25 times, 51 responses after the device was used 25 to 50 times, and 154 responses where participants had used the device in excess of 50 times.

With increasing device use, respondents reported a significant drop in the amount of time it took to perform the measurements. In early users, the distribution in the duration of time required to perform the measurements was largely skewed beyond 30 seconds with only 3% of respondents indicating they could complete the measurement in <30 seconds versus 50%, 33%, and 13% indicating completion in 30 to 60 seconds, 1 to 2 minutes, and >2 minutes, respectively. By the time the user had acquired more experience, this distribution shifted such that 42% of respondents indicated they could complete the measurement in <30 seconds, 42% indicated that completion of the measurements took between 30 and 60 seconds, and the remaining 26% indicated that it took between 1 and 2 minutes to complete the measurements (P < .001).

When dissected according to the steps involved in using the device, ease of performance for 5 of the 6 critical tasks shifted significantly from very or somewhat difficult to somewhat or very easy with increasing use (Table 1). For only one of the tasks, threading the tape through the openings to construct the loop, where 73% of starting scores were very/somewhat easy and 20% of starting scores were somewhat/very difficult, did we not observe a significant shift in user perceptions (Table 1). We also observed a significant increase in perceptions of overall ease of use (P < .001), efficiency (P = .002), convenience (P < .001), and acceptability (P = .021) with increasing user experience. Accompanying these findings was a significant increase in the overall preference for using this device over traditional methods of obtaining MUAC z score in the clinical practice setting. At the outset, 23% of users favored the device, 17% favored traditional methods, and 60% reported that they did not have enough information to make a determination. After gaining additional experience with the device, 74% of users favored the device, 7% favored traditional methods, and 19% reported that they did not have enough information to make a determination (P < .001). Information garnered from the open-ended questions revealed that individuals reporting they did not have enough information to inform this decision had never been in a position to obtain the measurement using the traditional method.

Table 1.

User Perceptions of Critical Tasks Stratified by Experience With the Device.

Task	Aggregate Scores	<10 Uses	10 to 25 Uses	25 to 50 Uses	>50 Uses	P
Using the tape to identify the midpoint of the upper arm	V/s easy	53%	71%	61%	88%	<.001
	Neutral	10%	13%	18%	4%
	V/s difficult	37%	16%	22%	8%
Threading the tape through the openings to construct the loop	V/s easy	73%	69%	65%	78%	.287
	Neutral	7%	11%	6%	8%
	V/s difficult	20%	20%	29%	14%
Positioning the tape at the midpoint of the upper arm	V/s easy	77%	64%	55%	84%	<.001
	Neutral	7%	7%	10%	9%
	V/s difficult	17%	29%	35%	6%
Identifying the correct position at which to read the tape output	V/s easy	53%	73%	73%	90%	<.001
	Neutral	17%	13%	6%	4%
	V/s difficult	30%	13%	22%	6%
Reading the MUAC value in millimeters	V/s easy	70%	87%	84%	96%	<.001
	Neutral	10%	4%	14%	1%
	V/s difficult	20%	9%	2%	3%
Reading the MUAC z score range	V/s easy	43%	67%	71%	82%	<.001
	Neutral	20%	16%	4%	6%
	V/s difficult	37%	18%	25%	11%

Abbreviation: v/s, very or somewhat.

General themes revealed by the open-ended questions related largely to design and materials. By far the largest criticism centered on the age bands selected for the first iteration of the device. Z scores are derived from lambda (L), mu (M), sigma (S) values defined for each gender at each month of age between 2 months and 18 years. ⁷ With finite space on the semi-disposable device, the 442 discrete LMS sets needed to be condensed and combined into a finite number of age bands. Selected users, depending on the population of children they served, felt these age bands were too broad. Also, under the category of design were comments about the size of the device that was felt to impinge on their ability to use and/or transport the device. A final theme under design was the font size of some printed elements, which impaired legibility and was deemed too small. With regard to materials, the primary themes related to durability of the materials and overlays that were used. Specifically, it was felt that the device was weakened at the slits through which the tail end of the tape was repeatedly threaded and that the process of routinely sanitizing the device disrupted the integrity of the materials, removing the markings from nonlaminated tapes over time, and disrupting the laminate bond for laminated versions.

Discussion

User-centered design bases the development of products, services, or processes on information obtained from the ultimate end user and validates or refines the innovation in a real-world environment. ¹² Failure to accommodate the expectations, behaviors, and workflows of the end user can be a key factor in the slow adoption or early abandonment of a new technology.^13,14 The intent of this study was to provide empirical evidence for the usability of our anthropometric screening tool in a real-world setting and to iteratively refine the device based on user experience.

We observed that ability to use the device and the efficiency with which the critical tasks could be accomplished significantly improved as users gained experience with the device. Correspondingly, user preferences also changed with 23% of users favoring the device early during implementation and 74% favoring the device by the end of the study. Feedback provided by the users reinforced our initial design assumptions that the green, yellow, and red color overlays improved communication with families for who are already primed to interpret these colors as go/good, caution, and stop/bad, respectively. The feedback also validated our expectation that this unfamiliar device would be perceived as less threatening and more acceptable than stepping on a weighing scale by patients who are struggling with weight issues or eating disorders (eg, anorexia, bulimia, morbid obesity).

As a result of the comments received by the users, several changes were made to the subsequent iteration of the device. We sourced a different material for device construction and identified a vendor who could perform double-sided printing. With twice the printing surface, we could expand the number of age groups depicted on the device and eliminate age gaps. At the same time, we modified some of the printed elements on the device to improve legibility and reduce unnecessary markings. A unique observation that remains to be addressed was brought to our attention by a single user who communicated that the device design was awkward for the left-handed user. While we view this as important feedback for design considerations, the cost of redesigning and producing a unique version for a minority of users would offset the goal of creating an inexpensive screening tool.

Based on the comments received, we also felt that it was important to proceed with the fabrication of an electronic prototype (Figure 2). The electronic embodiment requires the user to input gender and age in months. It then reports the MUAC in millimeters and the MUAC z score as a discrete number rather than a range. We incorporated a large LED display into the design that should address the concerns of legibility and confidence in interpreting the values. The device utilizes a retractable measuring strip to improve portability, and both the measuring strip and exterior housing are constructed from materials that can be sanitized. While we see a continued role for the paper-based version in the setting of nutritional screening, the electronic version might find utility in the setting where confirmatory clinical assessments are taking place.

Figure 2.

Prototype of electronic MUAC z-score tape.

There were several limitations of this study. The device was deployed at the same time that recording MUAC z scores was instituted as a practice requirement at our institution. As a result, many users did not have a frame of reference whereby MUAC z scores were obtained with the use of a traditional tape measure and secondary reference materials, which could influence their preferences. Another limitation was the anonymous nature of the survey, which could result in selection bias. Users could have elected not to participate at all and individuals who were not pleased with the device may have declined to respond over time biasing the results toward increasing preference with increasing use. However, we did note that the constructive feedback provided by the users via open-ended questions continued through the end of the study reassuring us that respondents were comfortable providing comments for improvement irrespective of use frequency.

We acknowledge that the execution of this study at a single pediatric health care center could be perceived as a limitation; however, we do not believe this to be the case. More than 5000 measurements were made in newborns, infants, children, and adolescents who spanned the spectrum from severely malnourished to morbidly obese. Even if device use proved problematic in only a subset of children, we were likely to detect that given the breadth of treatment populations and environments in which the device was applied.

The active engagement of end users in the real-world testing of our nutritional assessment device was essential for ensuring that we developed a practical, functional device. Refinement of the innovation with special attention paid to the needs of our dietitians is expected to result in a final device with fewer implementation hurdles than might be expected if the device was developed in a simulation environment. Though repetitive application was required, this study suggests that acceptability rates were high with end users of the MUAC z score tape, which the majority of dietitians found easy to use, efficient, and convenient. Overall, the MUAC z score tape satisfies several of the requirements laid out for a desirable pediatric malnutrition screening tool including the fact that it is inexpensive, noninvasive, easy to implement, and developed specifically for pediatric patients without exclusions. ¹⁵

Supplemental Material

Supplemental_figure – Supplemental material for User-Informed Medical Device Development: A Case Study for Pediatric Malnutrition Assessment

Supplemental material, Supplemental_figure for User-Informed Medical Device Development: A Case Study for Pediatric Malnutrition Assessment by Kristi Thaete, Kasey Rowzer, Karen Stephens and Susan M. Abdel-Rahman in Global Pediatric Health