Abstract
The purpose of this study was to evaluate the acceptability and usability of an online training to prepare Expanded Food and Nutrition Education Program (EFNEP) educators to collect 24-hr dietary recalls using the Automated Self-Administered 24-Hour Dietary Assessment Tool (ASA24). Fifty-eight educators in 17 states were recruited to take the training; 29 completed the online training, assisted two individuals in completing a 24-hr recall using ASA24, and completed a survey about their experiences. The sample included 26 respondents. The majority (n = 16; 61.5%) of the sample was EFNEP educators with college education. The majority of the respondents indicated that they found the readings and videos acceptable for learning (n = 21; 80.7%). Half of the participants (n = 13) felt prepared to collect data using ASA24. The majority (n = 22; 84.6%) had positive feedback about the training content. The training may be acceptable for preparing EFNEP educators with at least some college education to collect 24-hr recalls using ASA24. Because of potential bias due to self-selection and nonresponse, the training has been revised and is now intended to be used to train trainers of EFNEP paraprofessionals.
Introduction
Dietary assessment tools are important for evaluating the effectiveness of nutrition education programs such as the Expanded Food and Nutrition Education Program (EFNEP; Thompson & Subar, 2001). The 24-hr recall has been useful in assessing dietary intake of EFNEP’s participants and evaluating the effectiveness of EFNEP in achieving its goals for improved nutrition behavior among its participants and can be a useful tool for evaluating health promotion or disease prevention programs that include a nutrition intervention (Dietary Assessment Primer, n.d.; Gills, Baker, & Auld, 2017). The 24-hr recall involves obtaining information about food and beverages consumed in the last 24 hr or from midnight to midnight the previous day. It has typically been an interviewer-administered instrument, due to the high degree of literacy required to obtain detailed information about food intake (Thompson & Subar, 2001). EFNEP has used 24-hr recalls as part of their program since its founding in 1969 (Chipman & Kendall, 1990).
Background
Because of cost-effectiveness concerns, there was a shift from one-on-one education strategies to group education in EFNEP (Chipman & Kendall, 1990). With this change, the 24-hr recall transitioned from an interviewer-administered mode, where questions were administered and recorded by the trained paraprofessional using paper and pencil, to a group-led session requiring participants to record their individual data, also using paper and pencil. However, this paper and pencil method is prone to data coding error and missing data because data are manually entered into a national food database system, the Web-Based Nutrition Education Evaluation and Reporting System (WebNEERS), at a later time by other people (Gills et al., 2017). In addition, updates to the food database in WebNEERS are infrequent (Guenther & Luick, 2015), making it difficult to match newly popular foods. Gills et al. (2017) expressed concern for the quality of the dietary data obtained because of inconsistency in protocols used to collect data and variations in training. Finally, although group recalls are considered the preferred assessment method for EFNEP, research on the validity of this method is promising but limited (Kirkpatrick et al., 2019).
Recently, web-based, self-administered interviewing has provided an easier and less expensive alternative to obtain dietary assessment data. The National Cancer Institute developed the Automated Self-Administered 24-Hour Dietary Assessment Tool (ASA24), allowing participants to complete a 24-hr recall without an interviewer. Foods and beverages reported using ASA24 are automatically coded and linked to the U.S. Department of Agriculture’s (USDA’s) Food and Nutrient Database for Dietary Studies (FNDDS) to obtain the nutrient values of the foods reported, and this database is updated frequently (Subar et al., 2012). Research has demonstrated that ASA24 yields high quality data, performing comparably with the industry standard Automated Multiple-Pass Method (Kirkpatrick et al., 2014; Thompson et al., 2015), and this data collection method has been well accepted among adults and older children with Internet access (Kirkpatrick et al., 2017; Thompson et al., 2015).
EFNEP paraprofessionals at both county and state levels see value in conducting dietary recalls as highly important (Wakou, Keim, & Williams, 2003). ASA24 offers a standardized method of data collection and coding that could provide higher quality data for EFNEP’s program evaluation. As with all self-administered systems, participant literacy is an important consideration, and low literacy may be an issue for EFNEP participants. Clients with low incomes typically read at a lower level and find traditional text-only materials difficult to understand (Townsend, Ganthavorn, Neelon, Donohue, & Johns, 2014; Townsend, Sylva, Martin, Metz, & Wooten-Swanson, 2008). The low literacy level of EFNEP participants suggests that effective use of ASA24 may require guidance from paraprofessionals (Benavente, Jayarante, & Jones, 2009; Thompson et al., 2015). If ASA24 is to be used by programs such as EFNEP for program evaluation purposes, a standard training is needed to train paraprofessional educators how to assist participants in completing the task. Currently, paraprofessionals are trained on how to administer the participant-recorded 24-hr recall (Islam, Paddock, & Dollahite, 2015). Several training kits have been developed and are being used in various states, including programs provided by Oklahoma State University Extension, Cornell University, and Louisiana State University (Gills et al., 2017). However, there is concern regarding the quality of data collected due to inconsistencies in data collection protocols and variation in the training provided to personnel (Gills et al., 2017). Therefore, standardization of the training given to nutrition educators across the program in data collection is vital to obtaining data that are fit for use in program evaluation.
This article describes a pilot study examining the acceptability and usability of an online training for EFNEP paraprofessionals who would assist EFNEP participants in using ASA24 to report dietary intake data. The research question that guided this study was, “Is the ASA24 online training, developed at Utah State University, acceptable to and usable by EFNEP paraprofessionals?”
Method
Study Design
This was a mixed-methods cross-sectional pilot study utilizing a convenience sample of EFNEP nutrition educators to assess the acceptability and usability of an online training for ASA24-2016. The training development utilized principles of andragogy, the science of teaching adults. Adult learning experts have established that adult learners learn differently than children or adolescents (Parker, Lenhart, & Moore, 2011); therefore, the training development utilized teaching principles specific to adults. Andragogy theory suggests that adult learners must be involved in their own instruction, have opportunities for experiential learning exercises, and respond best to immediate application of knowledge (Knowles, 1980). These techniques should be used in the classroom and in online or distance education settings (Rossman, 2000). The training implemented these theoretical frameworks by providing readings, videos, and experiential activities by assisting others in completing a 24-hr recall.
The online training consisted of three modules, which included readings, instructional videos, online activities, and quizzes. The training began with emailed instructions for navigating between the course site (Canvas) and the ASA24-2016 website. Module 1 introduced ASA24 and the multiple-pass method for data collection, how to search for food and beverage items in ASA24, and how to make changes to meals and food/beverages already reported. Module 2 addressed handling of problematic foods, such as food items that were unknown to the participant (i.e., “Unknown Food”), multi-ingredient foods, and foods that could not be found. Module 3 introduced techniques for assisting another individual with completing a recall using ASA24-2016. Once they completed the training, the EFNEP nutrition educators were asked to assist two individuals in completing a dietary recall using ASA24-2016. A description of the content of each module and activity can be seen in Table 1. Figure 1 provides an example of an experiential learning activity.
Content of ASA24 Online Training.
Note. ASA24 = Automated Self-administered 24-hour Dietary Recall.
Example of an experiential learning activity from the training manual.
Five nutrition experts with extensive knowledge of ASA24 reviewed a paper copy of the training for content. Three undergraduate nutrition students reviewed the online training to evaluate content and usability. Finally, two nutritionists who had used ASA24 with low-income individuals reviewed the online training for content. Revisions were made based on each round of review.
Participants and Recruitment
The principal investigator sent an email message to all EFNEP program directors inviting them to participate in the study. Program directors from 17 states responded, and then each state’s program invited educators to participate according to what worked best for their program, for example, some invited all educators in the program, whereas others invited a certain number from each district to participate in the training. Of the 58 EFNEP nutrition educators recruited, 29 completed the training, and each assisted two EFNEP participants in completing a 24-hr recall using ASA24-2016 on a mobile device. These recalls were administered individually using the English ASA24-2016 platform. After their training was completed, educators were invited to complete an online Qualtrics survey (Provo, Utah, 2017). Twenty-six participants responded (response rate: 45% [26/58]). The Utah State University Institutional Review Board approved all data collection instruments, procedures, and protocols. Participants were informed of the research study via letter, and they provided informed consent by continuing to the survey.
Instruments
The survey consisted of 28 items. Survey items were informed by work conducted at the Colorado State University Extension (Natker, Auld, Baker, & McGirr, 2015). The survey collected information regarding demographics (age, education, gender, race, ethnicity [Hispanic or not], and years working with EFNEP), and the acceptability and usability of the training (length of training, method used in training, order of the training, and reading level). A full list of survey questions can be seen in Table 2.
Questions Asked in Survey.
Note. EFNEP = Expanded Food and Nutrition Education Program; ASA24 = Automated Self-administered 24-hour Dietary Recall.
Participants also responded to several open-ended questions about their experiences. Participants who responded negatively to any of the questions were asked to indicate their reason/rationale for disagreeing with the statement. Participants identified problems they had in completing activities, gave feedback for improving the online training, and indicated if there were any areas of ASA24 that caused confusion for the individuals they assisted in completing a dietary recall. A list of the open-ended questions asked can be seen in Table 2.
Data Analysis
Descriptive statistics were calculated in SAS 9.4 (SAS Institute, Cary, NC, 2012). The variable assessing length of the online training was collapsed from 10 response categories to five for data analysis. Items in the survey that were measured on a 5-point Likert-type scale (strongly agree, agree, neither agree nor disagree, disagree, and strongly disagree) were collapsed into three categories (agree, neutral, and disagree) for data analysis. Responses to open-ended questions were coded for themes by one of the authors (LAS).
Results
Demographics
The majority of participants in the pilot test were females (92%), between the ages of 50 and 64 years (42%), identified as non-Hispanic White (65%), and had completed a 4-year college degree (50%). Although 35% of participants reported working with EFNEP for more than 7 years, almost one-third of participants had worked with EFNEP for less than 1 year (31%; Table 3).
Characteristics of Study Participants (N = 26).
Note. Percentages may not add to 100 owing to missing data and/or rounding. EFNEP = Expanded Food and Nutrition Education Program; ASA24 = Automated Self-administered 24-hour Dietary Recall; GED = General Education Development certification.
The majority of participants said the training took between 3 and 7 hr to complete, felt prepared to collect ASA24 data at the conclusion of the training, and found that both the videos and readings were acceptable (Table 4). However, 40% of respondents ran into problems completing the training activities; many of these problems were related to difficulty accessing the Canvas website where the online training was built. Educators reported difficulty with the process of logging on to the online training, forgotten usernames and passwords, and other technical difficulties. They also had problems accessing the ASA24 website. Initially, the researchers thought that the demo version of ASA24 would be easier to use than the official website as it does not require an additional username and password to access. Unfortunately, the researchers did not anticipate that the increased traffic on the demo website from the study participants would cause it to crash. Once the problem was identified, the researchers switched to the official ASA24 website and provided each educator with a list of usernames and passwords to use for the training. Because only one recall per day is permitted, multiple accounts per educator were required, so that they could complete more than one exercise in a day. These technical difficulties undoubtedly contributed to the low participant completion rate.
Participants’ Experiences With the ASA24 Training (N = 26).
Note. Percentages may not add to 100 owing to missing data and/or rounding. ASA24 = Automated Self-administered 24-hour Dietary Recall.
Nevertheless, most participants responded favorably to the training content. The majority agreed that topics were presented in a logical order, important points were clearly stated, and activities allowed readers to develop and practice relevant skills (Table 5).
Participants’ Evaluation of the ASA24 Training Manual (N = 26).
Note. Percentages may not add to 100 owing to missing data and/or rounding. ASA24 = Automated Self-administered 24-hour Dietary Recall.
Qualitative Themes
Training/website problems
Twelve respondents (46%) reported problems with the online training website or the ASA24-2016 website. The most common issues were related to logging in. Because two platforms were used, Canvas and ASA24, several respondents were unsure when to use each set of log-in information. Other issues included difficulty with course navigation. As one participant commented, “The layout was a bit confusing and hard to mark where I left off on reading or quizzes.”
Challenges with adding meals/foods not in list
Six respondents (23%) reported that the individuals they helped complete the ASA24 had questions about adding meals and finding foods not found in ASA24. One participant said, “Working alongside the student [an individual they selected to participate in the recall] helped clarify any questions that came up, the questions mainly dealt with locating a certain food.”
Length of the online training
Five respondents (19%) stated the length of the online training was a concern for them. Although some identified the length of the online training as a barrier, several suggested that the training was helpful despite the length. One participant described, “There was a lot of reading required, but it was helpful and informative.”
Other comments
Two individuals suggested that receiving results of the nutrient analysis at the conclusion of the dietary recall would be helpful for both nutrition educators and clients. (Although ASA24 currently provides immediate dietary analysis results for participants, this feature was not available at the time of the study.) As one participant commented, “It would be nice to get a survey results page at the end. Not only would this provide instant feedback about an individual’s one-day-diet, but it would give the instructor a good idea of how to critique their lesson plans.” Another identified the potential benefit of having paraprofessionals complete several individual recalls themselves prior to implementing with clients.
Discussion
Surprisingly, more than half the participants who completed the survey had a college education. Results from this pilot study suggest that the online training may be acceptable for preparing some EFNEP educators to collect 24-hr dietary recalls using ASA24-2016. However, participants ran into technical issues related to logging into both the training and the ASA24 websites. In addition, the learning levels of the training content and the length of the training modules were identified as points of concern among some study participants. In addition, the fact that half of the participants who were able to complete the training were college educated suggests that nonresponse bias may have played a role in these results. For example, those educators who were less educated may have chosen not to participate or may not have finished the training because of difficulties encountered. Therefore, the authors believe that using the online manual to train college-educated program staff, who would then train the paraprofessional educators in person, would be a better approach.
The online training developed for this pilot study provides a consistent protocol for data collection, which has been identified as a critical need in EFNEP programming (Gills et al., 2017). Although EFNEP educators indicated that the readings and videos were acceptable for learning and that they felt prepared to collect data using ASA24 after completing the training, almost half of the study participants reported a desire for additional assistance in completing the training and in assisting others in completing a 24-hr dietary recall using the ASA24. However, participants were generally able to answer questions while assisting other individuals, and all participants were able to complete the recalls with their practice participants. Despite technical issues that may be encountered, online trainings remain an acceptable way of distributing information, particularly, because of their ability to save expenses compared with face-to-face trainings (Christofferson, Christensen, LeBlanc, & Bunch, 2012).
Based on these results and input from collaborating program directors, the researchers have developed an in-person training curriculum that covers the same material and utilizes best practices for training EFNEP paraprofessionals. Instead of being used to train the paraprofessional educators directly, the online training is now intended to be completed by a staff member or educator who is experienced with computer technology and/or online courses. This person can then use the in-person training curriculum to train the paraprofessional educators.
The investigators utilized the feedback from this pilot study to improve the online training, including adding information about interfacing with the training and the ASA24 websites, creating a Frequently Asked Questions page, adding an introductory module, and making the readings optional. When completed, the curriculum will be available at extension.learn.usu.edu for a nominal fee per user. However, continued research is needed to assess the usability of the online training and in-person curriculum among the intended audiences, knowledge gained, and self-efficacy changes in the paraprofessional educators. In addition, further work is needed to determine the feasibility of using ASA24 in the field to collect data from EFNEP participants.
Overall, although more work is needed to smooth the logistics of administration, ASA24 offers many potential benefits to EFNEP and other nutrition education programs and to others interested in assessing the diets of the low-income population. These benefits include standardized data collection and coding of recalls, a continuously updated database of foods, and reduced paperwork for educators and program staff. Depending on current practices in each state, it may also increase the detail and accuracy of dietary recalls. Furthermore, ASA24 is undergoing continual improvement by the National Cancer Institute (NCI; 2018), for example, the most recent release, ASA24-2018, includes a recipe feature, where participants can enter recipes for foods not found in the database.
Limitations
Limitations of the study include a strong potential for self-selection and nonresponse biases, that is, EFNEP nutrition educators who volunteered for the study and completed the survey may differ in significant ways from those who did not participate. Although 58 paraprofessionals were recruited, only 29 were able to complete the training; this may indicate that many of the invited educators did not have the skills necessary to complete an online training. Of note, the majority of participants in this pilot study reported education levels at or beyond a 4-year college degree, which is not representative of the EFNEP nutrition educator workforce. This limits the generalizability of these results but has informed the changes made to the training described above.
Conclusion
Results of this study suggest the online training may be acceptable for preparing EFNEP educators with at least some college education to collect 24-hr dietary recall data using ASA24. Because the results of this study indicate that the online training does not seem to be suitable for the intended EFNEP workforce, the researchers recommend having staff or educators who are comfortable with online courses complete the training and then use the in-person curriculum to spread the knowledge to the rest of the educators.
Footnotes
Authors’ Note
Any opinions, findings, conclusions, or recommendations expressed are those of the authors and do not necessarily reflect the views of the U.S. Department of Agriculture.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This manuscript is based upon work that is supported by the National Institute of Food and Agriculture, U.S. Department of Agriculture under award number 2015-67001-24290.
