Sage Journals: Discover world-class research

Abstract

Objective:

This research examined the effects of multiple combined competency-based methods on sonography students’ perceptions of adult echocardiography training components. In addition, clinical preceptor evaluation scores were compared with faculty objective structured clinical examination (OSCE) scores.

Materials and Methods:

A quasi-experimental nonequivalent group research design was used to evaluate students enrolled in an adult cardiac Commission on Accreditation of Allied Health Education Programs (CAAHEP) accredited curriculum. Students’ perceptions pre and post multiple competency-based methods (formative assessment, OSCE, & simulation) intervention were recorded via course evaluations. Questions were analyzed individually using descriptive statistics and the Bonferroni correction. Students’ clinical evaluation and OSCE scores were analyzed using Spearman’s rank correlation.

Results:

The majority of students’ perceptions pre- and postintervention of multiple competency-based assessments demonstrated significant differences, or they primarily agreed that their echocardiography knowledge or skill set was enhanced. There was weak correlation between students’ clinical competency evaluation scores and OSCE scores—post implementation of multiple competency-based assessments, r_s(37) = −.01, P = .93.

Conclusion:

These results suggest further evaluation of the credentialing process’s clinical assessment to ensure clinical competency.

Keywords

credentialing education and training cardiac ultrasound simulation training evaluation competency-based education

The creation and maintenance of an effective curriculum to meet sonography accreditation requirements while producing a competent clinical sonographer is an ongoing process. Numerous studies have individually explored different assessments and techniques to optimally learn sonography.^1–3 Methods have ranged from experimental designs encompassing objective structured clinical examinations (OSCEs) to simulation, and formative assessments. Other studies have also analyzed participants’ perceptions of these different techniques;^4–6 however, this line of inquiry is the only attempt at encompassing multiple competency-based assessments including formative assessments, OSCEs, and simulation. This work is an extension of assessing multiple competency-based assessments within a Commission on Accreditation of Allied Health Education Programs (CAAHEP) accredited adult echocardiography program,¹ focused on analyzing students’ perceptions, as well as validating clinical assessments.

Use of Simulations

Sidhu et al² sought to determine whether procedural simulation led to improved sonography competence in a clinical setting via a systematic review. While most studies demonstrated improved performance in the same simulated environment as well as knowledge acquisition, minimal evidence supported the widespread adoption of simulation-based medical education.² Gibbs³ recognized that limited simulated learning research was apparent and explored methods of integrating simulation-based learning into sonography training to enhance clinical preparation. However, despite clinical preceptors’ positive recognition of simulation in that it provides learning in a less stressful environment, a call for continued research to develop simulation strategies to effectively integrate it to support students, educators, and preceptors is needed.³ The call for additional strategies is supported by Al Enazi⁴ who discovered that health care students in programs such as nursing, respiratory therapy, nutrition, and occupational therapy had positive perceptions toward simulation education after exposure and feel that it should be included in their education.⁴ Another study gauged the perceptions of physicians, sonographers, nurses, residents, and medical students. Ninety-eight percent approved simulation for training and teaching, and the authors concluded positive perceptions toward integrating simulation into the curricula.⁵ In addition, medical students exposed to seven pathology-simulation ultrasound training models felt that simulation prepared them to use ultrasound in a clinical setting.⁶ Delving deeper into the use of simulation’s history is helpful and necessary to understand further exploration of its potential application.

Full-body mannequin simulators first originated in the health care field in the 1960s to help the field of anesthesia. The mannequin, known as Sim One, enabled the training of endotracheal intubation and anesthesia induction.⁷ Half a century later, simulation developed and became a common tool to help educate and train allied health and medical professionals.^3,8–10 Unlike practicing on real patients that may result in a patient consequence or self-embarrassment, simulation permits trainees and students to identify anatomy and physiology, and make mistakes while facilitating confidence in their knowledge and skills.^8,11 Specific simulation tools expose students to rare pathology they may not be exposed to in clinical practice—where students mainly acquire and finesse their skills.^3,8,12 In this regard, simulation standardizes education as student clinical exposure (e.g., patient population and pathology) varies from clinical site to clinical site.⁸ Simulation provides students the ability to remediate after receiving negative feedback from clinical sites.⁸ The simulation should supplement clinical experience, not replace it, by enhancing student readiness for the clinical environment.^8,9,13 Platts et al’s¹⁴ research revealed that sonographers found a transesophageal and transthoracic echocardiography simulator process to be realistic, assist in image acquisition, and improve spatial relationship assessment. The study indicated echocardiography simulation to fill an important role in future echocardiographic training.¹⁴

Incorporating simulation into teacher-led instruction (TI) (e.g., didactic methods) may prove useful and helpful to assess competency in a structured and uniform format.^8,9,11 However, CAAHEP does not accept simulation to establish student competency.¹⁵ Hence, simulation is designed to supplement learning. Pessin and Tang-Simmons⁸ examined CAAHEP Diagnostic Medical Sonography DMS) programs’ use of simulation by surveying program directors to assess their use and perception of simulation and its value, respectively. No assessment of educational outcomes and specific simulation activities transpired. Seventy-five percent of program directors who responded use simulation; 89% stated it was a good teaching tool; 9% were not interested; 81% recorded positive student experience; 55% and 64% stated it was most useful for anatomy and transducer manipulation, respectively.⁸ Programs did not report assessment tools. “Seventy-six percent of respondents felt it was necessary to have an instructor during simulation training.”^8(p439) However, only 20% and 30% of schools provide faculty for low- and high-fidelity simulation, respectively.⁸ Low-fidelity simulation mirrors actions, associated with task trainers, and utilized in procedural training, but may omit some real-life experience.⁸ Whereas, high fidelity simulation is intended to be as realistic as possible.⁸ Pessin and Tang-Simmons⁸ state that faculty training requires resources, as noted by participants, and may be a hindrance for programs to include faculty training.¹³ Faculty are crucial to sonography students’ education, but clinical preceptors are essential as well.

CAAHEP requires students to complete their clinical competencies on-site and therefore assessed by clinical preceptors.¹⁵ In addition, credentialing examination prerequisites are aimed to confirm clinical competence, which is assessed by field sonographers or those who are in clinical preceptor positions. Unfortunately, the prerequisites can be subjective and contain nonvalid assessments at clinical sites as clinical preceptors may not be adept at assessments.^1,16 Moreover, in a study that assessed nursing preceptors’ assessment strategies, findings revealed that many preceptors did not apply all recommended assessment strategies when assessing students, as well as not completely understanding the assessment process and lacked experience.¹⁷ Additional studies^8,17,18 assessed students’ and healthcare personnel’s perceptions of simulation as well as other instructional methods - participants perceptions were positive. Therefore, students’ perceptions of multiple competency-based techniques should also be assessed. An OSCE is an example of a competency-based technique that can be used to emulate the clinical experience and should also be assessed to compare the evaluation process of faculty within a CAAHEP accredited program to clinical preceptors.

Competency Methods in Sonography—OSCE

Medical education has utilized OSCEs since 1975.¹⁹ Effective OSCEs should test Bloom’s three main domains.²⁰ Variation exists in the composition of an OSCE as it developed over time.^19,21,22 Typical OSCEs incorporate standardized patients to ensure uniform scenarios,¹⁹ which comes with a high cost—a primary disadvantage.^16,19 Baker et al¹⁶ recommend using students as models to offset the cost. Other advantages of OSCEs are (1) their availability, (2) safety (no danger of injury to patients), (3) no risk of litigation, (4) feedback from actors (if standardized patients are used), (5) stations tailored to the level of skills to be assessed, (6) allows for teaching audit, (7) emergency skill demonstration, and (8) recall.¹⁹ OSCE disadvantages include organizational training, time, reliability, potential variability between the examiner and “patient,” and “textbook” scenarios that may not simulate real-world situations.^16,19,23 Shulruf et al²³ stated OSCE examiners’ biases (i.e., low reliability) are likely to affect students performing at a borderline level based on Brannick et al’s²⁴ comprehensive meta-analysis study for medical students. Implementing an Objective Borderline Method 2 (OBM2) method can address this issue if needed. OBM2 compares the difficulty of an item to student ability; if the ability is less than difficulty, the borderline pass score (one of four) is graded as an F.²³ Investigating how OSCEs affect performance can facilitate educators in designing more effective curricula assessment processes to optimally evaluate students’ instruction.²⁵ In addition, an OSCE can help identify areas of improvement for the student and the curriculum.^16,26 Employing a cogent standard to assess and prove competency is a challenge of OSCEs.²⁷

Vanda Yazbeck et al²⁰ analyzed internal structure validity using multiple psychometric measures compared with a standard passing or cut score. “Gathering consequential and internal structure validity evidence by multiple metrics provides support for or against the quality of an OSCE.”^20(p6) Messick’s framework assessed five validity sources that should be considered to accept an assessment. The sources are content (construct is assessed accurately), response process (evidence of data coherence), internal structure (psychometric properties of the examination), relation with other variables, and consequences (impact on instructors, learners, and the curriculum).²⁰ Different medical specialties and education programs have experimented with OSCEs using various assessment methods, design, and success rates—most revealed a positive experience.^{16,25,28–32}

The following research questions guided this portion of the study:

Research Question 1: What is the relationship between the implementation of multiple competency-based assessments in an adult cardiac accredited CAAHEP DMS program and its students’ perceptions of adult echocardiography training components?

Research Question 2: What is the relationship between students’ clinical competency evaluation scores from clinical preceptors and their OSCE scores that evaluate competency from faculty?

In addition to answering the research questions, the study’s objective was to build on the previously published work¹ that found no relationship between multiple competency-based assessments and clinical evaluation scores administered by clinical preceptors.

Materials and Methods

The quasi-experimental nonequivalent group research design was used to evaluate students enrolled in a CAAHEP accredited adult echocardiographic curriculum since students were not randomly assigned to conditions as intervention was part of the curriculum. Students’ perceptions, clinical evaluation scores, and OSCE scores from multiple cohorts were used for statistical analysis. The study’s control and experimental groups consisted of 42 and 39 students, respectively. The control group was not exposed to Research Question 1’s independent variable, which were the multiple competency-based assessments (cardiac OSCE, cardiac simulation assignment, and formative assessments). The dependent variable for Research Question 2 was students’ Likert scale adult echocardiography course evaluations obtained from the CAAHEP accredited DMS program’s learning management system, which encompassed ordinal data. A Likert scale assesses an individual’s attitude from one extreme to another (e.g., strongly agree, agree, neutral, disagree, and strongly disagree).

The course evaluations administered to the control group consisted of 22 Likert scale questions, three descriptive questions that asked students to describe the strengths and weaknesses of the courses, as well as asking for suggestions to improve the course. The course evaluations administered to the experimental group entailed the same questions as the control group, in addition to the seven Likert scale questions. The seven Likert scale questions were explicitly related to the competency-based assessments (Q23–Q29).

All course evaluation questions, except those specific to assessing the multiple competency-based assessments, were created by the DMS program and administered after each course. The remaining course evaluation questions were created to assess students’ perceptions of the multiple competency-based assessments. For example, individually asking if the OSCE improved a student’s ability to competently perform an echocardiogram or helped prepare them for their multiple-choice final examination. Other evaluation questions were created to assess if the competency-based assessment such as simulation or formative assessments improved their clinical knowledge.

Of the 22 Likert scale questions, only 11 questions that had the potential to be affected by the implementation of competency-based techniques were included. For example, assignment usefulness, a reflection of course content on examinations, and the instructor’s preparedness and presentation of subject matter; the latter is to determine if a more structured design of assignments affected students’ perception of the instructor’s competency. Omitted questions remained relatively constant or were unlikely to affect training components of echocardiography (e.g., classroom atmosphere, proctoring of examinations, and instructor’s availability and respect for students). Diagnostic Medical Imaging (DMI) students have designated classrooms for each cohort and scanned in the same scan lab; meanwhile, classroom atmosphere questions asked about lighting, heating, seating, and conduciveness to learning. Questions related to the instructor such as availability and respect were omitted as the instructor’s office hours remained constant, and the college has an open-door policy.

The 11 Likert scale questions administered to both groups were assessed using the Bonferroni correction method with an alpha level of 0.004 significance. The seven Likert scale questions only administered to the experiment group were analyzed using descriptive statistics. Research Question 1 was designed to assess if students perceived there to be a benefit. Theoretically, competency-based assessment training should lead to higher evaluation scores; however, Dunstatter’s¹ results demonstrated no significant difference between students’ clinical competency evaluation scores from clinical preceptors and multiple competency-based assessments. The data types of the clinical evaluation scores, OSCE, cardiac simulation, and formative assessments are interval, interval, nominal, and nominal, respectively.

A Spearman’s rank correlation test was used to analyze Research Question 2. A clinical evaluation summary score was obtained and is an average of four different categories with multiple measures for each. The categories are the psychomotor, cognitive, and affective learning domains, in addition to professional behavior. Considering Dunstatter’s¹ multiple competency-based methods were implemented and there was deemed to be significant relationship between them and clinical evaluation scores, the goal of Research Question 2 was to assess if there was a correlation between clinical evaluation and OSCE scores.

Results

All 11 common Likert scale questions had a 100% response rate, except for Q3 (response rate = 98%) and Q13 (response rate = 99%) (see Table 1). The seven Likert scale questions were only applied to the experimental group and had a response rate of 95% (see Table 2).

Table 1.

Course Evaluation Questions—Research Question 1’s Dependent Variable & Results Using Bonferroni Correction Method.

Common Questions (Control & Experimental Group)	Results(α = 0.004)
Q1. A syllabus including a clear statement of objectives, assignments, attendance, and grading policies were provided and explained.	P = .049
Q3. Readings, lectures, and other course activities were relevant to stated course objectives.	P = .042
Q4. Assignments were useful for learning subject matter.	P = .083
Q5. Quizzes and examinations reflected the content and emphasis of the course.	P = .002
Q6. Papers and reports were returned with constructive criticism or comments.	P = .001
Q8. Grading on examinations and quizzes was fair.	P = .008
Q9. Tests and assignments were returned within a reasonable time.	P = .024
Q10. The instructor provided adequate time for students to ask questions and receive feedback.	P = .052
Q12. The instructor seemed knowledgeable and well-prepared.	P = .000
Q13. The instructor’s presentation of the subject matter was well-organized.	P = .007
Q18. Audiovisuals were relevant to course objectives and logically integrated.	P = .036

Bolded values are intended to highlight statistical significance.

Table 2.

Course Evaluation Questions—Experimental Group.

Experimental Questions	Likert Scale Results %
Experimental Questions	Strongly Agree	Agree	Neutral	Disagree	Strongly Disagree
Q23. Do you feel the cardiac OSCE improved your ability to competently perform an echocardiogram?	22.22	44.4	11.1	22.22	0
Q24. Do you feel the cardiac OSCE helped prepare you for your multiple-choice final examination?	11.11	16.66	38.88	22.22	11.11
Q25. Did the cardiac simulation project increase your understanding of cardiac pathology?	22.22	33.33	38.88	5.55	0
Q26. Did the cardiac simulation increase your understanding of echocardiographic anatomy and scan planes?	27.77	16.66	38.88	16.66	0
Q27. Did the cardiac simulation assignment increase your understanding of an echocardiography protocol?	16.66	27.77	38.88	16.66	0
Q28. Did the scan lab assignments and feedback improve your knowledge of cardiac measurements that can and should be obtained within an echocardiogram?	38.88	38.88	16.66	0	0
Q29. Did the scan lab assignments and feedback improve your knowledge of cardiac images that can and should be obtained within an echocardiogram?	50	38.88	5.55	0	5.55

Abbreviation: OSCE, objective structured clinical examination.

The second research question was to explore the relationship between students’ clinical competency evaluation scores from clinical preceptors and their OSCE scores that evaluate competency from faculty. A Spearman’s rank correlation test was run to measure the relationship between students’ clinical competency evaluation scores and their OSCE scores. There was a weak correlation between students’ clinical competency evaluation scores and OSCE scores, r_s(37) = −.01, P = .93 (see Figure 1).

Figure 1.

Ranked scatterplot demonstrating OSCE score versus clinical evaluation score correlation. OSCE, objective structured clinical examination.

Discussion

The first research question aimed to assess students’ perceptions of adult echocardiography training components within an adult cardiac CAAHEP accredited DMS program through the acquisition of course evaluations. Including perceptions aimed to demonstrate that no other variable (e.g., teaching style, instructor knowledge, organization, etc.) attributed to any potential statistically significant difference that multiple competency-based assessments may have had. The rationale is that often, health care educators such as pharmacists and nurses demonstrate competence in their professional field. However, professional competence does not translate to effective teaching.^33–35 Also, sonography educators struggle to teach reasoning and critical thinking skills as these capabilities typically develop with increased clinical experience.³⁶ A teacher is a primary factor in the educational process,³⁷ and so the combination of outcomes and course evaluations should be used to assess the instructor and learning environment. Considering the incorporation of multiple competency-based methods affected students’ perceptions in areas that were relatively controlled,¹ the results were somewhat expected and assist in demonstrating relatively consistent teaching methods. Three of the Bonferroni Corrections (Q5, Q6, & Q12) showed a statistically significant difference.

Q1’s results show that the instructor continued to provide clear expectations through the syllabus for all cohorts. For Q3, readings and lectures were constant—only course activities (i.e., competency-based assessments) changed. An instructor may perceive this to be beneficial as prior assignments were also considered relevant by students not exposed to the multiple competency-based assessments. It is important to note that students often ask for more scanning time, which relates to their clinical skills, particularly of the psychomotor and cognitive domains.

Q4’s result is relatively consistent with Q3’s as course activities are part of assignments. Future research may want to encompass questions that are more specific to each assignment.

Q5’s results were ironically statistically significant considering the same final examination was administered. However, quizzes were modified, but questions came from the same question bank. The results could mean that the questions pulled from the question bank were not reflective of the course content or that the competency-based assignments made it more evident what the course emphasized, despite no significant difference in their final examination scores.

Q6’s results were anticipated and verified the weekly formative assessments throughout the course. Formative assessments serve as an ongoing diagnostic tool and are designed to enhance student learning during the instructional process while providing feedback to both the educator and learner.^38–40 Future research or course evaluations may want to evaluate feedback effectiveness of competency-based assignments in echocardiography.

Q8 of the course evaluation assessed whether grading on examinations and quizzes was fair and was not statistically significant. This makes sense considering that the final examination was consistent and quiz questions came from the same question bank (Q5). It is important to note that the instructor reviewed any question that 50% or more of the class got wrong via discrimination scores and if the question’s wording was ambiguous.

Q9 of the course evaluation assessed whether tests and assignments were returned within a reasonable time and was not statistically significant; more students within the experimental group strongly agreed. This question semi-aligns with Q6 and confirms the instructor’s appropriate use of formative assessments as test scores were typically returned within a week. Regular formative assessments with timely results and feedback to students related to identifiable learning outcomes are most effective.^38,40–42 For Q10, the majority of both groups strongly agreed that the instructor provided adequate time to ask questions and receive feedback.

Q12 was statistically significant, but Q13 was not; however, more students within the experimental group strongly agreed for both. The results for Q12 are surprising as the instructor had over 10 years of sonography experience for both groups, and it is unlikely that the instructor’s knowledge changed significantly during this time. Though, student perceptions of the instructor being well-prepared may have changed due to the acquisition of more teaching experience over time. However, this slightly conflicts with Q13’s results, which demonstrate no statistical difference in instructor’s presentation of subject matter and being well-organized; one may surmise that more teaching experience could enhance the instructor’s skills in these areas. In addition, the feedback from the assessments could have affected student perceptions of instruction knowledge as they had a tangible written resource to review for each of the assignments compared with the control group, where it was mostly verbal. Additional research could look into differentiating between the effects of written, verbal, and combined feedback concerning ultrasound echocardiography. Q18’s results were not significant as the same audiovisuals were used.

The seven Likert questions administered only to the experimental group were fascinating relative to Dunstatter’s¹ findings. Dunstatter’s¹ study demonstrated no relationship between multiple competency-based assessments and students’ summative assessment grades and clinical competency evaluation scores given by clinical preceptors, respectively. However, Q23 demonstrated that students primarily agreed that the OSCE improved their ability to perform a protocol, which was not reflected in their clinical evaluation scores. While perceptions do not necessarily translate to outcomes, this finding may further support the difficulty and inaccuracy of clinical assessment.¹ An alternative option for future research could be to evaluate students’ perceptions of clinical rotations affecting their ability to perform an echocardiogram competently.

Q24 sought to assess whether the OSCE helped prepare students for their multiple-choice final examination, which the majority chose neutral or disagree. These results support what is found in the literature in that didactic assessment (i.e., multiple-choice examinations such as a cumulative final or credentialing examination) do not entirely assess competency.^1,16,43–45 In addition, multiple-choice tests and OSCEs combined could provide a more thorough and accurate evaluation of abilities.^25,28,30 For the portion that did agree, it may likely be attributed to the real-world application of the OSCE, where students had to scan and acquire images and measurements that assess a particular pathology. The assessment of pathology applies to the cognitive domain as students must be aware of which findings appear within a particular pathology. In addition, 46% of the Registered Diagnostic Cardiac Sonographer (RDCS) credentialing examination is pathology, and 23% is measurement techniques, maneuvers, and sonographic views.⁴⁶

The results of Questions 25, 26, and 27 correspond to the literature. Pessin and Tang-Simmons⁸ study showed that 81% of CAAHEP directors reported positive student experience with simulation, but the study did not report specific assessment tools. This study can add to the type of assessment tool used in CAAHEP programs. Although one must remember that simulation is designed to supplement learning as CAAHEP does not accept simulation to establish student competency.¹⁵

The literature supports that specific simulation tools expose students to rare pathology they may not be exposed to in clinical practice, which is where they primarily acquire and finesse their skills.^3,8,12 Although no statistically significant outcomes were achieved between multiple competency-based assessments and students’ clinical competency evaluation scores,¹ in the eyes of the students, there was an overall benefit that aligns with Alsalamah et al’s⁴⁷ study where optimal simulation benefit achievement is acquired through goal-oriented objectives and assignments within a curriculum. Furthermore, the incorporation of simulation into didactic methods proved useful in a structured and uniform format.^8,9,11 The findings also aligned with other studies^4–6 (though not as positive) where participants had positive perceptions toward simulation. In this study, the instructor designed an assignment based on the simulation that incorporated the didactic component where students had to work together to understand a particular pathology and a protocol to assess the pathology. The assignment was also designed to help students perform better on their OSCE. However, students were not allowed to scan the same pathology of their assignment. It would be interesting to assess students’ perceptions on whether simulation helped them understand a pathology protocol they were tested on via the OSCE.

The results of Questions 28 and 29 indicated that students perceived the formative assessments to benefit them the most. The questions differ from asking solely about a basic echocardiography protocol—as Q23 did with the OSCE. A protocol is merely a baseline of required images for a routine study. However, echocardiography encompasses dozens of measurements that may not be utilized in a standard protocol. The assignments were designed to teach students to be aware of all measurements in case they acquire a job that requires a thorough understanding of all possible quantitative analytical measurements. These results proved most useful to the researcher moving forward for clinical competency and further incorporating formative assignments into the curriculum. Similar to Q23’s analysis, Q28 and Q29 may support the difficulty of clinical assessment^1,16 as students are also evaluated on the cardiac images and measurements they acquired during their clinical experience. In other words, students perceived formative assignments to improve their clinical skills, but it was not represented through their clinical evaluation scores. In addition to clinical assessment, it may have proven beneficial to ask the students if the formative assignments helped prepare them for their final examination, OSCE, and to perform a protocol.

Overall, the majority of Research Question 1’s results suggest that multiple competency-based assessments have changed students’ perceptions of the course. The seven additional questions were meaningful, though not statistically significant, as there is no other group who were asked these questions or administered different assignments while asked to perform the same end objective (e.g., OSCE). The additional questions appear to incorporate information with the potential to create new assignments or research moving forward. It is essential to note that sometimes, student perceptions may not correlate with teacher effectiveness, and there are other variables to consider, such as students’ grade expectations, subject, gender, race, and others.⁴⁸

The second research question sought to identify the relationship between students’ clinical competency scores from clinical preceptors and their OSCE scores that evaluate competency from faculty using a structured rubric. Due to validation of health care workers’ clinical assessment, and less experienced sonographers awarding higher grades,⁴³ future research could assess the experience of clinical preceptors. The outcomes of Research Question 2 are relevant, considering it is comparing two scores that both assess scanning competency and support clinical assessment issues in health care. The findings potentially could correlate with Dunstatter’s¹ and be a reason why there was a lack of statistical significance between multiple competence-based assessments affecting students’ clinical evaluation scores, despite the students’ overall perceptions in Research Question 1 that the competency-based assessments help them perform clinical skills. Such evidence could warrant credentialing organizations to focus more on assessing clinical competency in addition to CAAHEP and Joint Review Committee on Education in Diagnostic Medical Sonography (JRC-DMS) reevaluating their competency requirements. For example, the JRC-DMS requires that students must complete specific standards,⁴⁹ but they could also assess the reliability of the requirements. Considering that graduating from an accredited ultrasound school is only one pathway for an applicant to sit for their credentialing exam, it would make the most sense for credentialing organizations to reconsider their credentialing process. Unfortunately, limitations do exist and must be addressed before changing the credentialing process. A limitation of Research Question 2’s analysis is the comparison of two different assessments. While the OSCE was assessed by educators (with possibly more assessment experience) in a structured environment, it was not done on an actual patient in a clinical setting. The structured environment, in theory, could be more standardized, and the rubric used could be more thorough than American Registry for Diagnostic Medical Sonography’s (ARDMS) clinical forms. Further research must be conducted to have stricter criteria or, at the least, an official assessment criterion where personnel is specially trained for this role. While this may make the process to become registered more difficult for applicants, it could ensure that those who become registered are competent.

Study Limitations

Due to the study design, these student results are not generalizable due to several threats to validity. However, the study reveals areas that are enlightening to the educational and credentialing process. This study was conducted using one sample of convenience from the host echocardiographic program. Therefore, it restricted the number of cohorts and students analyzed. Another limitation is the implemented OSCE utilized fellow students as suggested by Baker et al,¹⁶ instead of standardized patients, partly due to a limited budget. Although students did not know who they were scanning until the day of the examination, there may have been a bias as the students involved in the research had been completing peer scanning examinations throughout the curriculum. Also, personal bias may exist among evaluators despite multiple evaluators assessing each student’s images.²³ In addition, personal bias may be present from clinical preceptors’ lack of training, knowledge, or education to ensure a completely valid and reliable assessment.¹⁶

Conclusion

Diagnostic Medical Sonography’s credentialing organizations (i.e., ARDMS) have multiple pathways to apply for a credentialing examination. Regardless of pathway, applicants must demonstrate clinical competency, ideally via a reliable and valid assessment. CAAHEP and credentialing organizations have criteria in place to account for learning and assessment. However, this study also reveals potential gaps in the clinical assessment prerequisite process to become credentialed.

Sonography’s clinical competency assessment process could be further evaluated as there was no correlation between educators’ and clinical preceptors’ evaluation of clinical skills despite students perceiving multiple competency-based assignments to help them perform clinical skills. This finding further bolsters the argument of subjective clinical assessment within the literature, and that clinical experience does not translate to reliable and valid evaluation. An option for CAAHEP, JRC-DMS, or credentialing organizations could be to establish or require a more stringent official assessment criterion where personnel are specially trained for this role to ensure that one who becomes registered is clinically competent. This study also contributes to the literature on how CAAHEP programs may want to incorporate multiple competency-based assignments such as simulation, formative assessments, and OSCEs to help students become clinically competent. Of the multiple assignments, students perceived formative assessments to benefit their skills the most.

Footnotes

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Douglas Dunstatter

References

Dunstatter

: Competency-based methods in adult cardiac sonography and its application to credentialing. J Diagn Med Sonogr. 2021;37(4):359–368. doi:10.1177/87564793211002502.

Sidhu

Olubaniyi

Bhatnagar

Shuen

Dubbins

: Role of simulation-based education in ultrasound practice training. J Ultrasound Med. 2012;31(5):785–791. doi:10.7863/jum.2012.31.5.785.

Gibbs

: The role of ultrasound simulators in education: an investigation into sonography student experiences and clinical mentor perceptions. Ultrasound. 2015;23(4):204–211. doi:10.1177/1742271X15604665.

Al Enazi

: Healthcare students’ perceptions of simulation education at an urban university. Georgia State University, 2017. https://scholarworks.gsu.edu/rt_theses/37.

Hani

Chalouhi

Lakissian

Sharara-Chami

: Introduction of ultrasound simulation in medical education: exploratory study. JMIR Med Educ. 2019;5(2):e13568. doi:10.2196/13568.

Kondrashova

Coleman

: Enhancing learning experience using ultrasound simulation in undergraduate medical education: student perception. Med Sci Educ. 2017;27:489–496. doi:10.1007/s40670-017-0416-2.

Lateef

: Simulation-based learning: just like the real thing. J Emerg Trauma Shock. 2010;3(4):348–352. doi:10.4103/0974-2700.70743.

Pessin

Tang-Simmons

: Sonography simulators: use and opportunities in CAAHEP-accredited programs. J Diagn Med Sonogr. 2018;34(6):435–444. doi:10.1177/8756479318799347.

Coerver

Multak

Marquardt

Larson

: The use of simulation in physician assistant programs: a national survey. J Physician Assist Educ. 2017;28(4):175–181. doi:10.1097/JPA.0000000000000173.

10.

Salas

Paige

Rosen

: Creating new realities in healthcare: the status of simulation-based training as a patient safety improvement strategy. BMJ Qual Saf. 2013;22(6):449–452. doi:10.1136/bmjqs-2013-002112.

11.

Okuda

Bryson

DeMaria

Jr , et al: The utility of simulation in medical education: what is the evidence. Mt Sinai J Med. 2009;76(4):330–343. doi:10.1002/msj.20127.

12.

Chalouhi

Bernardi

Ville

: Ultrasound simulators in obstetrics and gynecology: state of the art. Ultrasound Obstet Gynecol. 2015;46(3):255–263. doi:10.1002/uog.14707.

13.

Clau-Terré

Sharma

Cholley

, et al: Can simulation help to answer the demand for echocardiography education. Anesthesiology. 2014;120(1):32–41. doi:10.1097/ALN.0000000000000072.

14.

Platts

Humphries

Burstow

Anderson

Forshaw

Scalia

: The use of computerised simulators for training of transthoracic and transoesophageal echocardiography. The future of echocardiographic training? Heart Lung Circ. 2012;21(5):267–274. doi:10.1016/j.hlc.2012.03.012.

15.

Commission on Accreditation of Allied Health Education Programs: Standards and guidelines for accreditation of educational programs in diagnostic medical sonography, 2011. https://www.caahep.org/CAAHEP/media/CAAHEP-Documents/DMSStandards.pdf.

16.

Baker

Willey

Mitchell

: The attempt to standardize technical and analytic competence in sonography education. J Diagn Med Sonogr. 2011;27(5):203–211. doi:10.1177/8756479311416508.

17.

McCarthy

Murphy

: Assessing undergraduate nursing students in clinical practice: do preceptors use assessment strategies? Nurse Educ Today. 2008;28(3):301–313. doi:10.1016/j.nedt.2007.06.002.

18.

Enskar

Heng

Pua

Wang

: The perspectives of preceptors regarding clinical assessment for undergraduate nursing students. Int Nurs Rev. 2016;63(3):473–481. doi:10.1111/inr.12272.

19.

Zayyan

: Objective structured clinical examination: the assessment of choice. Oman Med J. 2011;26(4):219–222. doi:10.5001/omj.2011.55.

20.

Vanda Yazbeck

Yoon Soo

Ara

Nazih

: Evaluating the validity evidence of an OSCE: results from a new medical school. BMC Med Educ. 2018;18(1):313. doi:10.1186/s12909-018-1421-x.

21.

Hodges

: OSCE! variations on a theme by harden. Med Educ. 2003;37(12):1134–1140. doi:10.1111/j.1365-2923.2003.01717.x.

22.

Jain

DeLisa

Eyles

Nadler

Kirshblum

Smith

: Further experience in development of an objective structured clinical examination for physical medicine and rehabilitation residents. Am J Phys Med Rehabil. 1998;77(4):306–310. doi:10.1097/00002060-199807000-00009.

23.

Shulruf

Adelstein

Damodaran

, et al: Borderline grades in high stakes clinical examinations: resolving examiner uncertainty. BMC Med Educ. 2018;18:272. doi:10.1186/s12909-018-1382-0.

24.

Brannick

Erol-Korkmaz

Prewett

: A systematic review of the reliability of objective structured clinical examination scores. Med Educ. 2011;45:1181–1189. doi:10.1111/j.1365-2923.2011.04075.x.

25.

Dong

Saguil

Artino

Jr , et al: Relationship between OSCE scores and other typical medical school performance indicators: a 5-year cohort study. Mil Med. 2012;177(suppl 9):44–46. doi:10.7205/MILMED-D-12-00237.

26.

Reznick

Smee

Baumber

, et al: Guidelines for estimating the real cost of an objective structured clinical examination. Acad Med. 1993;68(7):513–517.

27.

Khan

Ramachandran

Gaunt

Pushkar

: The objective structured clinical examination (OSCE): AMEE guide no. 81. Part I: an historical and theoretical perspective. Med Teach. 2013;35(9):e1437–e1446. doi:10.3109/0142159X.2013.818634.

28.

Kirton

Kravitz

: Objective structured clinical examinations (OSCEs) compared with traditional assessment methods. Am J Pharm Educ. 2011;75(6):111. doi:10.5688/ajpe756111.

29.

Rebel

Srour

DiLorenzo

, et al: Ultrasound skill and application of knowledge assessment using an innovative OSCE competition-based simulation approach. J Educ Perioper Med. 2016;18(1):E404. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5131258/.

30.

Nair

Siu

Sloggett

Biclar

Sidhu

EHC

: The assessment of technical and interpretative proficiency in echocardiography. J Am Soc Echocardiogr. 2006;19(7):924–931. doi:10.1016/j.echo.2006.01.015.

31.

Nayer

: An overview of the objective structured clinical examination. Physiother Can. 1993;45(3):171–178. https://pubmed.ncbi.nlm.nih.gov/10127817/. Accessed October 10, 2019.

32.

Sloan

Donnelly

Schwartz

Strodel

: The objective structured clinical examination the new gold standard for evaluating postgraduate clinical performance. Ann Surg. 1995;222(6):735–742. doi:10.1097/00000658-199512000-00007.

33.

Mukhalalati

Taylor

: Adult learning theories in context: a quick guide for healthcare professional educators. J Med Educ Curric Dev. 2019;6. doi:10.1177/2382120519840332.

34.

Burton

Boschmans

Hoelson

: Self-perceived professional identity of pharmacy educators in South Africa. Am J Pharm Educ. 2013;77(10):210. doi:10.5688/ajpe7710210.

35.

McAllister

Oprescu

Jones

: N2E: envisioning a process to support transition from nurse to educator. Contemp Nurse. 2014;46(2):242–250. doi:10.5172/conu.2014.46.2.242.

36.

Penny

Zachariason

: The sonographic reasoning method. J Diagn Med Sonogr. 2015;31(2):122–129. doi:10.1177/8756479314567307.

37.

Rahmatullah

: The relationship between learning effectiveness, teacher competence and teachers performance Madrasah Tsanawiyah at Serang, Banten, Indonesia. High Educ Stud. 2016;6(1):169–181. doi:10.5539/hes.v6n1p169.

38.

EMW

: Are students receptive to formative assessment when authoring wiki projects? Ubiquitous Learn. 2018;11(3):1–15. doi:10.18848/1835-9795/CGP/v11i03/1-15.

39.

McTighe

O’Connor

: Seven practices for effective learning. Educ Leadership. 2005;63(3):10–17.

40.

Kulasegaram

Rangachari

. Beyond “formative”: assessments to enrich student learning. Adv Physiol Educ. 2018;42(1):5–14. doi:10.1152/advan.00122.2017.

41.

Hattie

Timperley

: The power of feedback. Rev Educ Res. 2007;77(1):81–112. doi:10.3102/003465430298487.

42.

Nicol

Macfarlane-Dick

: Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Stud High Educ. 2006;31(2):199–218. doi:10.1080/03075070600572090.

43.

Michael

Wampler

Shostrom

Boals

Anderson

: A retrospective analysis of abdominal competency scores to improve clinical education in a sonography program. J Diagn Med Sonogr. 2014;30(4):179–184. doi:10.1177/8756479314529844.

44.

Taylor-Fujikawa

Andrist

: The (updated) history of sonography: education and credentialing. J Diagn Med Sonogr. 2019;355(6):520–524. doi:10.1177/8756479319876233.

45.

Wimmers

Schauer

: Validating OSCE performance: the impact of general intelligence. Health Prof Educ. 2017;3(2):79–84. doi:10.1016/j.hpe.2016.12.002.

46.

American Registry for Diagnostic Medical Sonography: ARDMS general prerequisites, 2019. https://www.ardms.org/wp-content/uploads/pdf/ARDMS-General-Prerequisites.pdf.

47.

Alsalamah

Campo

Tanos

, et al: Face and content validity of the virtual reality simulator “ScanTrainer®.” Gynecol Surg. 2017;14(1):18. doi:10.1186/s10397-017-1020-6.

48.

Lawrence

: Student evaluations of teaching are not valid. American Association of University Professors, 2018. https://www.aaup.org/article/student-evaluations-teaching-are-not-valid#:~:text=But%20a%20preponderance%20of%20evidence,student%20evaluations%20is%20often%20low. Accessed December 20, 2021.

49.

Joint Review Committee on Education in Diagnostic Medical Sonography: Policies & procedures, 2019. https://www.jrcdms.org/policies.htm. Accessed October 29, 2019.