The Impact of Formative Computer-Based Testing on Learners’ Anxiety and Performance on the American Registry for Diagnostic Medical Sonography Examination

Abstract

The intent of this research was to evaluate the effects of computerized mock examinations on student performances during the preparatory stage for a national board examination. Three measures were used: web-based surveys, student data from the ExamSim software (treatment group), and American Registry for Diagnostic Medical Sonography (ARDMS) examination scores. Three research questions were posed: (1) What are the differences in ARDMS scores between the treatment and control groups? Mean scores showed that the treatment group scored significantly higher than the control group on the ARDMS examination. (2) To what degree did the scores and time invested on the ExamSim software predict performance on the ARDMS examination? A multiple linear regression was conducted, and the result was significant, with time and software scores accounting for 37.6% of the variance in ARDMS scores. (3) How did utilizing a formative computer-based assessment change perceived testing anxiety levels during the test preparatory phase? The result of the dependent sample t test was significant. Self-perceived scores at posttreatment were lower than self-perceived scores at pretreatment.

Keywords

testing anxiety ARDMS certification examination high-stakes test preparation computerized testing

The demand for clinical sonographers continues to exceed the supply, with some geographic areas being underserved more than others. According to the US Bureau of Labor Statistics, employment is expected to increase by 44% from 2010 through 2020.¹ Couple this with the idea that states are currently passing legislative requirements for the licensure of diagnostic medical sonographers at a growing rate, and the need for more credentialed sonographers becomes apparent. States that currently have established licensure laws have mandated the certification of sonographers by a nationally recognized credentialing body, such as the American Registry for Diagnostic Medical Sonography (ARDMS), as a component of the licensure process.^2,3 In an effort to produce a larger supply of certified or credentialed sonographers for their communities, educational programs continue to focus on successful student pass rates. Although the majority of currently enrolled college students have prior computer experience, there still exists a gap between those being able to use computer technology to complete assignments or examinations and those having the skills necessary to successfully pass a computerized high-stakes test under pressure.^4,5

This study was designed to quantitatively evaluate the effects of providing a computerized environment with a series of practice criterion-referenced tests that allow for testing familiarity, immediate feedback, continuous access, and a personal dynamic analysis of each practice session. These effects can be measured by determining the enhancement of the student experience with the high-stakes examination, in the form of decreased anxiety levels and increased test scores. Longitudinally analyzing the progress of students toward passing or failing their credentialing examination was attempted by assessing their software usage. Overall, this study aimed to increase the depth of understanding of the benefits that self-directed learning with software such as Pegasus Lectures Inc ExamSim provide and to promote the practice of computer testing use in sonography education.

Theoretical Framework

With an impact both direct and tangential on the learning process, assessments typically are at the core of an educational platform. Summative assessments are designed to establish whether course objectives and goals have been attained, whereas formative assessments can provide prescriptive feedback to assist students in reaching their academic goals.^6,7 This study was based on the conceptual framework of the learning-testing cycle (Figure 1).

Figure 1.

The three stages of the learning-testing cycle.

The learning-testing cycle is a method that students often unconsciously utilize as they progress through their educational endeavors. It explains the relationship of three stages of assessment performance: testing preparation, performance, and reflection—all having great influence on one another both directly and indirectly and often dictating future actions based on previous experiences.⁴ These three phases to testing are interrelated in that they all have influence on one another yet are recognized as independent segments in their approach to testing. As perceptions of tests develop over time, they begin to influence test preparation indirectly through past performance. Eventually, past performance begins to directly influence test preparation and current test performance. During the test reflection phase, previous experiences in the testing environment begin to directly influence the types of preparation strategies that will be employed during the test preparation phase.

As educational models continue to experience a conceptual shift from teacher-centered learning to student-centered learning environments, the process of assessments should rely on high levels of student involvement in the testing practice.^8-11 Judgments or people’s ideas of their own levels of self-efficacy and testing confidence levels can affect preparatory and eventually assessment performance efforts in a multitude of ways.^12-14 Anxiety theorists generally believe that when students are faced with difficult challenges, such as preparing for and/or taking difficult examinations, those who have higher self-efficacy will often exert a deeper effort to succeed.^15-18

Test Familiarity

Practice of a skill to the point of overlearning can lead to mastery and automaticity of a subset of skills.^19,20 The mechanism of transfer is thought to be related to conditioning and the application of the newly learned principles. Transfer is more likely to occur when more of the same elements are present in the learning and transfer situations.^21,22 “If the task is identical in both training and transfer, trainees are simply practicing the final task during training and there should be high positive transfer.”²³ Test familiarity or testing exposure similar to that of the examination itself can develop a perception of control as more information and familiarity about a test or a testing situation become unveiled. In the same regard, the less that a student knows about an examination, the more ambiguity she or he will experience about how to prepare, thereby creating a greater potential for test anxiety to prevail. A lack of transparency about the nature or testing environment may cause setbacks related to the clarity of the goals. If the comparison point is unclear, it becomes harder to judge where a student may be (self-assessment) in relation to where one would eventually like to arrive (established goal). Students with lower confidence levels in their testing and preparation abilities may perceive their lack of clarity as a reflection of their inability to manage the entire testing situation. These types of actions may result in the student accruing more emotion-focusing strategies, such as self-blame and wishful thinking during the test-taking period.²⁴

Test-Taking Skills

Test familiarity as a conceptual idea is based on the premise that encoding specificity is always stored with its context. Those contextual cues present when information is encoded for storage and can later be used for retrieval.^21,22 While the importance of practice may be most obvious in psychomotor domains, such as athletics or even simple keystroke movements, it can be just as important for learning in domains such as declarative knowledge, conceptual, procedural, principle, and cognitive strategy.²⁵ Deliberate practice is a highly structured activity designed to improve performance,²⁶ and to the degree that practice is authentic and similar to the testing environment or the test itself, it can make learning more memorable and promote transfer.

There is an abundance of literature regarding the testing effect and the idea that completing an examination can do more than simply assess learning. Employing tests as a form of engaging students and their cognitive processes—their retrieval processes in particular, thereby improving their long-term retention—has been well reported.^27,28

Ubiquitous Learning and Practice

With the advent and adoption of mobile learning, ubiquitous learning—or learning that occurs when and where it is convenient to the learner—has become more commonplace.^29-31 Delivering practice tests online provides additional benefits by allowing students to complete tests conveniently, without environmental distractions that are common during in-class practice tests. Online practice tests have served as effective test preparation strategies for many allied health professions utilizing certification examinations,^19,32-34 with their ubiquitous nature being a positive attribute.

Immediate Feedback

The success of using formative assessments is partially reliant on the manner through which feedback is conveyed to the learner. A desirable feedback approach appears to be immediate postperformance reporting, which provides feedback directly after the entire assessment has been completed.³⁵ Learners need corrective feedback to validate their knowledge.^36,37 As noted previously,³⁸ evidence that feedback aimed at helping students, as opposed to rewarding or punishing them, allowed each student to identify his or her true understanding of course concepts. This practice, in turn, provided students the opportunity to seek remedial action when need be.^38,39

Method

The general purpose of this research study was to evaluate the effect of computerized mock examinations on sonography student performances and anxiety levels during the preparatory stage of a criterion-referenced national board examination. To achieve this purpose, the following questions framed the research:

Research Question 1: What is the difference in ARDMS scaled scores between the treatment and control groups?

Research Question 2: To what degree did the scaled scores and time invested in the Pegasus Lectures Inc ExamSim software predict performance on the ARDMS examination?

Research Question 3: How did utilizing a formative computer-based assessment change perceived testing anxiety levels during the test preparatory phase?

The design of this study included the collection and analysis of data from 3 surveys and the ARDMS examination scores from each participant, as well as the collected evidence of participation time in, and practice scores from, the software used (for the treatment group). An equivalent treatment-control design was attempted, with each participant randomly assigned and processed in a blinded fashion with regard to sex, age, cumulative grade point average, previous computer experience, high-stakes examination history, year in school, and/or employment status. Three measures were used: (1) a web-based survey, which included the Testing Anxiety Inventory (TAI); (2) student input data from the Pegasus Lectures Inc ExamSim software (treatment group); and (3) ARDMS certification examination scores (all participants). Voluntary consent was electronically confirmed with each participant, with a thorough understanding of the study parameters, timelines, and expectations from each group. Each participant was able to utilize 1 Pegasus Lectures Inc ExamSim software product (treatment group—during the time of ARDMS preparation, control group—for future ARDMS board preparation). At the time of this research study, products included Abdomen, Breast, Obstetrics and Gynecology, Adult Cardiac, Vascular Technology, and Sonography Principles and Instrumentation.

Institutions were randomly selected from the publicly available list of diagnostic medical sonography schools in the United States currently accredited by the Commission on Accreditation of Allied Health Education Programs (CAAHEP) that offer either an associate or baccalaureate degree in diagnostic medical sonography. Program size (in terms of number of students in each class), formal education level (associate or baccalaureate degree), and geographic location were not factors in soliciting programs or selecting participants. All participants voluntarily enrolled knowing that this study would have no impact on their current educational situation and that their diagnostic medical sonography program educators had no knowledge of their participation. Program directors were encouraged to not discuss the study in their classrooms, and it was made apparent that this study could in no way be affiliated with a program of study or be used for coursework purposes (as a final in a class or as part of a review course). All control group members were permitted to prepare for their ARDMS examination in any fashion that they so desired, as long as they held to the study stipulation to not use any commercially available computer-based practice tests prior to taking their ARDMS examination. This was deemed an important feature of the study to control for contamination of control group variables. A study limitation was evident with the knowledge that most, if not all, colleges use computerized testing with their learning management systems. However, the degree and extent of formative feedback, didactically and in the form of specific question analysis and test-taking skills, are drastically different when compared with the commercially available products on the market today. The use of a learning management system and its computerized preparation for board examinations was therefore considered a negligible limitation when compared with the robust nature of a dedicated high-stakes examination preparation tool addressing such items as details of areas of weakness, breakdown of score by topic and subtopic, individual test-taking habits, changing of answers detailed with score impact, time per topic relative to key concepts, actual question types that the participants answer incorrectly, and performance relative to cardinal test-taking rules.

Instrumentation

Pegasus Lectures Inc ExamSim Software employs the basic principles of providing a testing format similar to that of the ARDMS. Besides doing the practice examinations, participants can access a tutorial on how to use and navigate the testing software, learn how to develop test-taking strategies, review examinations already taken, and evaluate uniquely generated test reports. All testing modules had 3 full criterion-referenced examinations, each with approximately 170 established questions. Students had 3 hours to complete each test, with no help or hints from the software regarding the correct answers. During the test-taking time, the participant was able to navigate back and forth to answer, review, or change any question in any order. Randomly assigned questions directly correspond with the subject matter content, as specified on the ARDMS website content outline pages. Test takers can pause an examination at any point in time and later return to the same question.

Upon completion of an examination, participant answers are submitted for instant analysis, with the screen navigated to a results page. A patented Intelligent Analysis Package gives a detailed report of each participant’s level of preparedness for his or her specific examination. Test takers’ scores are reported with the same cutoff score that the ARDMS uses for a pass or failed attempt at 555. Additionally, the time spent answering each question is analyzed, allowing test takers to evaluate not only whether they answered the questions correctly but also whether they did so expeditiously. After each examination and review of performance analysis, the program allows for a review session that displays the test questions, the answers selected, the correct answers, and a detailed explanation of why each correct item is the most appropriate. Each treatment group participant had unlimited access for up to 30 days prior to the ARDMS examination. There were no testing mandates or requirements for treatment group members; each decided how much or how little she or he wanted to utilize the formative practice tool and at what times.

Pegasus Lectures Inc did not at any time have knowledge of identifying features of any of the participants, as they were given unique identifiers that could not be traced back to any names or institutions of enrollment.

The TAI is best utilized to measure test anxiety as a personality trait specifically related to examination situations. This survey is one of the most widely used instruments for assessing testing anxiety among high school and college students,^40,41 providing a reliable assessment of acute or state anxiety. The TAI consists of 20 statements based on a 4-point Likert-type scale describing the frequencies of a particular anxiety symptom before, during, and after an assessment. The respondents self-report how often they experience the feelings described in each statement. The TAI results were used to establish a baseline to identify those who were predisposed toward a high, low, or moderate level of debilitating test anxiety, as well as a comparison after the treatment was provided, prior to sitting for an ARDMS examination.

Three surveys were created to assist the researcher with data collection. All surveys were delivered online from the University of Wyoming survey tool. The TAI was included in two of those surveys, with the third being a means to collect participant ARDMS scores and, for the control group, to notify the researcher about which ExamSim software participants wanted to use on future ARDMS examination attempts, as an appreciation of being a part of the study. Control group members were able to select from any one of the Pegasus Lectures Inc ExamSim software products and had unlimited access to the testing software of choice for 30 days after establishing an account. The treatment group was not offered any compensation as its incentive was the use of the testing software prior to sitting for the ARDMS examination.

Results

Data were collected for 102 participants and were assessed for univariate outliers. Outliers were assessed with the creation of z scores. Final data analysis was conducted on 101 participants. Descriptive statistics were conducted to depict the sample, which consisted of 54 (53%) participants in the control group and 47 (47%) in the treatment group.

Research Question 1

What is the difference in ARDMS scaled scores between the treatment and control groups?

To determine whether there were significant differences in ARDMS scaled scores between the treatment and control groups, an independent sample t test was conducted. The result of the t test was significant, t(99) = −2.78, P = .007, Cohen’s d = 0.56, indicating that there were differences in the ARDMS scores by group. Examination of the mean scores showed that the treatment group (mean = 627.55) scored significantly higher on the ARDMS examination than the control group (mean = 605.46). Results of the independent sample t test are presented in Figure 2.

Figure 2.

Differences in American Registry for Diagnostic Medical Sonography (ARDMS) scores by group.

Research Question 2

For the treatment group only: To what degree did the scaled scores and time invested on the Pegasus Lectures Inc ExamSim software predict performance on the ARDMS examination?

To assess and determine to what degree the scores and time invested on the Pegasus Lectures Inc ExamSim software predicted performance on the ARDMS examination, a multiple linear regression was conducted, and the result was significant, F(2, 44) = 13.23, P < .001, R² = 37.6%, with time spent on software and software scores accounting for 37.6% of the variance in ARDMS scores. Upon further examination, time spent on software and software scores each offered a unique significant contribution to the model: as time spent on software increased by one unit, ARDMS scores increased by 0.11 units; additionally, as software scores increased by one unit, ARDMS scores increased by 0.08 units.

Research Question 3

How did utilizing a formative computer-based assessment change perceived testing anxiety levels during the test preparatory phase?

To assess Research Question 3, a dependent sample t test was conducted, the result of which was significant, t(46) = 2.89, P = .006. Examination of the mean scores showed that the scores at posttest (mean = 44.26) were lower than the scores at pretest (mean = 49.38), indicating less testing anxiety at posttest for the treatment group. Figure 3 presents the results of the dependent sample t test.

Figure 3.

Testing Anxiety Inventory (TAI) total scores at pre- and posttreatment for the treatment group.

Ancillary Analysis

As the research was being conducted, it became apparent that there may be value in knowing whether the control group experienced a rise in testing anxiety as its ARDMS examination date encroached. For this reason, a modification to the study was made with Institutional Review Board approval, and control group members began taking a second TAI prior to sitting for their board examinations.

To assess the ancillary analysis, a dependent sample t test was conducted and determined to be significant, t(34) = −2.33, P = .026. Examination of the mean scores showed that the scores at prepreparation (mean = 43.26) were lower than the scores preceding the ARDMS test (mean = 47.43), indicating more testing anxiety at postpreparation, or just prior to taking their ARDMS examination.

Discussion

Sonography students’ self-perceptions of their levels of anxiety were evaluated prior to using a sonography formative assessment software program and again just before sitting for their board examination. ARDMS scores proved to be higher in the treatment group, or participants who were exposed to formative practice testing software, which supported the expectation of the first research question. A second independent variable was examined to determine whether the time spent on the practice testing software and the scores earned on the practice tests were predictive of the ARDMS scores of those in the treatment group. The result of the multiple regression showed that scores on the practice test and time invested with the software accounted for 38% of the variance in ARDMS scores. As time spent on the software increased, ARDMS scores increased as well. Additionally, as scores on the practice tests increased, ARDMS scores increased. It was evident in the treatment group that testing anxiety levels significantly decreased after using the testing software. Furthermore, for the limited number of control group members who took a pre- and post-TAI, testing anxiety levels increased during this same test preparation phase.

Diagnostic medical sonography educators hold the responsibility of educating and preparing sonography students to become credentialed health care professionals capable of performing within their scope of practice and obtaining a state licensure, when applicable. This process should include the appropriate preparation to ensure successful outcomes on the ARDMS examination. Recognizing the stages of the learning-testing cycle and capitalizing on the opportunities that each phase has to offer (preparation, performance, and reflection) may eventually allow students to evolve and ultimately create higher opportunities for credentialing examination success. As students prepare outside the classroom, a testing environment similar to the one that they will experience during their board examination encounter may give them the advantage that they need to allow for success.

By better understanding, respecting, and tackling components to testing anxiety, educators and students alike may be more prepared to defeat its detrimental effects. Reforming students from high test anxiety and low self-efficacy states to low test anxiety and higher self-efficacy stances, simply by allowing the students to better prepare and learn ubiquitously with formative assessments, may prove to be beneficial.

The literature has shown that students find online formative assessment tools (practice quizzes/tests) to be useful in preparing for upcoming examinations. The completion of realistic testing events can promote performance by facilitating the transfer of content information or contextual cues while increasing self-efficacy, cognitive appraisals, and testing abilities while decreasing interfering agents. Allowing students to freely access practice tests and receive immediate corrective feedback provides personal control over their test preparation efforts, thereby lending to greater self-assessment opportunities and possibly decreasing test anxiety.

Study Limitations

There were several limitations inherent to this study, including a restrained time factor, diagnostic medical sonography program involvement, and, to some extent, the actual study parameters themselves. As many diagnostic medical sonography programs in the United States are not currently accredited, the sonography students involved in this study were accepted only if they were enrolled in or had graduated from a diagnostic medical sonography program accredited by the CAAHEP. This stipulation was added in an attempt to ensure rigorous and as consistent as possible standards within the educational programs. Although this was a good effort in theory, participants were still solicited from multiple accredited diagnostic medical sonography programs across the country, bringing with them their unique educational experiences.

The smaller number of participants was an unexpected element in this study. It was an unknown factor that a limited number of students would be eligible and wanting to sit for the ARDMS examination during the data collecting time frame. Many diagnostic medical sonography program directors provided feedback that their students were very interested in participating but that they would not be taking their board examinations until a later date. If this study is to be replicated, it may prove advantageous to increase the data collection time frame to work better with the different scholastic time frames established across the United States, for graduations to occur and credentialing examinations to be completed.

This research study was approved through the University of Wyoming Institutional Review Board and did not include the participation of program directors, deans, or educators within any diagnostic medical sonography program, in an attempt to allow students to volunteer without academic institutional influence. Although this offered many advantages to the student and allowed for one board approval (as opposed to approval from each educational facility involved), this study’s parameters created a situation in which many students were not as involved as they probably could have been, if encouraged to do so by their instructors.

Another limitation is that all students voluntarily self-enrolled in this study. While there are many factors and internal reasons why participants may have been more inclined to take part in the study, students from two opposing ideologies may be involved. One extreme could involve the highly motivated student with low test anxiety who wants to continue to explore all options that could possibly contribute to their obtaining the highest board examination score possible. Alternatively, the diametrically different student may have volunteered; that is, the poorly motivated and highly anxious test taker may have been desperately reaching out for any help she or he could obtain, trying to avoid another possible failed examination. This study did not address this issue but attempted to control for it by randomizing every student from every program into the control or treatment group.

Another limitation involves the self-reporting of multiple variables, such as participant ARDMS examination scores and student test anxiety levels. Attempts were made to decrease these limitations; for instance, names and self-reported ARDMS scores were shared with the ARDMS. However, due to its confidentiality policy, it was unable to provide actual test scores—just the fact that there was a variance between what the ARDMS had on file and what the researcher submitted from self-reports. This allowed the researcher to communicate with these participants in hopes of finding a more accurate score. Unfortunately, the self-reported TAI scores are just that and rely on honest answers to personal questions.

Areas for Further Study

Assisting students in completing the rigorous formal sonography curriculum and helping them to become successful ARDMS candidates has always been a high priority for sonography educators. In hopes of solidifying this theory, specifically for the sonography profession, the literature would benefit from more research in this area utilizing a broad range of commercially available formative testing software. After much research of sonography preparation software on the market at the time of this study, the researcher was able to secure the Pegasus Lectures Inc products free of charge for all participants and with no expectations from the researcher or participants after study completion. As such, there are many robust commercially available sonography preparation products in the market today. A great study for further research may be to compare different products with different students on a mixture of ARDMS examinations. A second area for more research that may lend valuable insight into student high-stakes examination preparation would be to evaluate cohorts of students with similar educational presentations and expectations, such as those from the same diagnostic medical sonography program. Finally, the following question was posed to the control and treatment groups: what types of study materials did they use during their preparation journey? If one could tease out the amount of time dedicated to each preparation method, it may be interesting to find a correlate to examination scores.

Footnotes

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

References

Bureau of Labor Statistics: Occupational outlook handbook. http://www.bls.gov/ooh/Healthcare/Diagnostic-medical-sonographers.htm#tab-6. Published 2014–2015.

Wilson

: What does government regulation really mean? J Diagn Med Sonography 2009;25(2):88–92.

Wilson

: Legislation approaching sonography: J Diagn Med Sonography 2011;27(4):191–197.

Cassady

Gridley

: The effects of online formative and summative assessment on test anxiety and performance. Journal of Technology, Learning, and Assessment 2005;4(1). http://files.eric.ed.gov/fulltext/EJ848518.pdf

Wheeler

: Instructional design in distance education through telematics. Quarterly Review of Distance Education 2000;1(1):31–44.

Birenbaum

: Assessment 2000: towards a pluralistic approach to assessment, in Birenbaum

Dochy

FJRC

(eds): Alternatives in assessment of achievements, learning processes and prior knowledge. Boston, MA, Kluwer Academic Publications, 1996, pp 3–29.

Maddalena

: Assessment: summative and formative—some theoretical reflections. British Journal of Educational Studies 2005;53(4):466–478.

Boud

Lawson

Thompson

: Does student engagement in self-assessment calibrate their judgment over time? Assess Eval High Educ 2013;38(8):941–956.

Huba

Freed

: Learner-Centered Assessment on College Campuses: Shifting the Focus From Teaching to Learning. Boston, MA, Allyn & Bacon, 2000.

10.

Rodgers

: A call for student involvement in the push for assessment. Assessment Update 2011;23(1):4–5.

11.

Sanders

: Improving assessment in university classrooms. College Teaching 2001;49(2):62–64.

12.

Bandura

: Self-efficacy mechanism in human agency. American Psychology 1982;37(2):122–147.

13.

Bresó

Schaufeli

Salanova

: Can a self-efficacy-based intervention decrease burnout, increase engagement, and enhance performance? A quasi-experimental study. Higher Education 2011;61(4):339–355.

14.

Rana

Mahmood

: The relationship between test anxiety and academic achievement. Bulletin of Education and Research 2010;32(2):63–74.

15.

Bandura

Schunk

: Cultivating competence, self-efficacy, and intrinsic interest through proximal self-motivation. J Pers Soc Psychol 1981;41:586–598.

16.

Cruz

: Emotion Perception and Reactions to Tests: Affective Influences on Test Performance [doctoral dissertation]. ProQuest Dissertations and Theses, 2010, UMI 3425201.

17.

Reeve

Bonaccio

: Does test anxiety induce measurement bias in cognitive ability tests? Intelligence 2008;36(6):526–538.

18.

Yusuf

: The impact of self-efficacy, achievement motivation, and self-regulated learning strategies on students’ academic achievement. Procedia-Social and Behavioral Sciences 2011;15:2623–2626.

19.

Dosch

: Practice in Computer-Based Testing and Performance on the National Certification Examination for Nurse Anesthetists [doctoral dissertation]. ProQuest Dissertations and Theses, 2010, UMI 3397328.

20.

Stefanidis

Scerbo

Montero

Acker

Smith

: Simulator training to automaticity leads to improved skill transfer compared with traditional proficiency-based training. Ann Surg 2012;255(1):30–37.

21.

Butterfield

Nelson

: Theory and practice of teaching for transfer. Educ Technol Res Dev 1989;37(3–4):5–38.

22.

Driscoll

: Psychology of Learning for Instruction. Boston, MA, Pearson Allyn & Bacon, 2005.

23.

Yamnill

McLean

: Theories supporting transfer of training. Human Resource Development Quarterly 2001;12(20):195–208.

24.

Chua

Chen

Wong

AFL

: Computer anxiety and its correlates: a meta-analysis. Comput Human Behav 1999;15(5):609–623.

25.

Smith

Ragan

: Instructional Design, 3rd ed. Hoboken, NJ, John Wiley & Sons, 2005.

26.

Ericsson

Krampe

Tesch-Romer

: The role of deliberate practice in the acquisition of expert performance. Psychological Review 1993;100(3–4):363–406.

27.

Agarwal

Karpicke

Kang

Roediger

McDermott

: Examining the testing effect with open-and closed-book tests. Applied Cognitive Psychology 2008;22:861–876.

28.

Rowland

: The effect of testing versus restudy on retention: a meta-analytic review of the testing effect. Psychological Bull 2014;140(6):1432–1463.

29.

Martin

Diaz

Plaza

Ruiz

Castro

Peire

: State of the art of frameworks and middleware for facilitating mobile and ubiquitous learning development. J Syst Softw 2011;84:1883–1891.

30.

Wilson

Aagard

: Exposing the gap between what is possible and what is acceptable: how M-learning can make a contribution to sonography education. J Diagn Med Sonography 2012;28(4):202–206.

31.

Wilson

Bolliger

: Mobile learning: endless possibilities for allied health educators. J Diagn Med Sonography 2013;29(5):220–224.

32.

Cornell

: Test Anxiety and Academic Performance in Nursing: The Effects of a Test Taking Intervention [doctoral dissertation]. ProQuest Dissertations and Theses, 2011, UMI 3468657.

33.

DiBartolo

Seldomridge

: A review of intervention studies to promote NCLEX-RN. Nurse Educ 2005;30(4). doi:10.3928/01484834-20110228-01

34.

Frith

Sewell

Clark

: Best practices in NCLEX-RN readiness preparation for baccalaureate student success. Comput Inform Nurs 2005;23(6):322–329.

35.

King

Behnke

: Technology-based instructional feedback intervention. Educational Technology 1999;39(5):43–49.

36.

Clariana

: A review of multiple-try feedback in traditional and computer based instruction. Journal of Computer Based Instruction 1993;20(3–4):74–76.

37.

Halpern

: Applying the science of learning to the university and beyond: teaching for long term retention and transfer. Change 2003;35(4):36–41.

38.

Peat

Franklin

: Supporting student learning: the use of computer-based formative assessment modules. Br J Educ Technol 2002;33(5):515–523.

39.

Vojdanoska

Cranney

Newell

: The testing effect: the role of feedback and collaboration in a tertiary classroom setting. Applied Cognitive Psychology 2010;24(8):1183–1195.

40.

Chapell

Blanding

Silverstein

et al : Test anxiety and academic performance in undergraduate and graduate students. J Educ Psychol 2005;97(5):268–274.

41.

Sterian

Mocanu

: Test anxiety. Euromentor Journal 2013;4(3):75.