Sage Journals: Discover world-class research

Abstract

Purpose:

To explore knowledge and skill acquisition outcomes related to learning physical examination (PE) through computer-assisted instruction (CAI) compared with a face-to-face (F2F) approach.

Method:

A systematic literature review and meta-analysis published between January 2001 and December 2016 was conducted. Databases searched included Medline, Cochrane, CINAHL, ERIC, Ebsco, Scopus, and Web of Science. Studies were synthesized by study design, intervention, and outcomes. Statistical analyses included DerSimonian-Laird random-effects model.

Results:

In total, 7 studies were included in the review, and 5 in the meta-analysis. There were no statistically significant differences for knowledge (mean difference [MD] = 5.39, 95% confidence interval [CI]: −2.05 to 12.84) or skill acquisition (MD = 0.35, 95% CI: −5.30 to 6.01).

Conclusions:

The evidence does not suggest a strong consistent preference for either CAI or F2F instruction to teach students/trainees PE. Further research is needed to identify conditions which examine knowledge and skill acquisition outcomes that favor one mode of instruction over the other.

Keywords

Physical examination clinical skills computer-assisted instruction e-learning

Introduction

Physical examination (PE) is a fundamental component of the patient assessment conducted by health professionals. It is taught across the health professions using a variety of formats. Traditional health professions education programs offer all courses and training in-person and, teach PE face-to-face (F2F). With advances in technology, most United States universities provide portions of their curricula online.¹ According to the National Center for Education Statistics, close to 5.8 million students are enrolled in online courses at degree-granting institutions, which represents an upward trend in higher education.² George et al³ conducted a systematic review of knowledge, skills, attitudes and satisfaction of dentistry, medicine, nursing, pharmacy, and physiotherapy students regarding effectiveness of online learning and found that health professions students perceived online learning to be equivalent to traditional teaching styles.³

There are many technologies available for online education and training. One method, which has been shown to be both convenient and affordable, is computer-assisted instruction (CAI).^4–6 Computer-assisted instruction is defined as the use of interactive CD-ROMs, multimedia software programs, and audio/visuals to augment instruction including multimedia presentations,⁷ live synchronous virtual sessions offered via a Web-based learning platform, presentations with audio and visuals, and synchronous or asynchronous discussion forums to enhance participation and increase engagement.^4,5,7

Although CAI has been shown to be feasible for use to teach PE to health professions students and trainees,^8–14 studies reporting its effectiveness are lacking. Systematic reviews have been published that explore the use of CAI in health professions students, but these have not focused on PE.^15–20 Hence, we found it essential to examine CAI feasibility and effectiveness as an approach to teach PE to students and trainees. This systematic review explored the published research on PE instruction to health professions students and trainees using computer-based or technology-based approaches in comparison with F2F approaches. It sought to determine whether CAI was equal to or superior to F2F instruction for PE of health professions students and trainees.

Using CAI to provide students and trainees with education and training on PE has many potential benefits. With the increased use of technology in academia, CAI integrates innovative methods of teaching using a consistent approach.^20–23 It allows for added flexibility to accommodate work and school schedules, eliminates the need to travel, which equates to cost savings, and is accessible from any location. Computer-assisted instruction provides the learner with a consistent, reproducible, standardized teaching experience.^20,21 It is conducted at the learner’s convenience, at their pace, and allows for unlimited access and repeated views. When the learning is distributed over time and occurs at the preferred pace of the learner, the repetition of skill practice has been shown to enhance skill acquisition.²⁴

Computer-assisted instruction has possible disadvantages as well. Unlike F2F lectures, the learner has a higher level of responsibility for their own learning; thus, more independence is needed. Computer-assisted instruction requires learners to practice time management and self-motivated with the ability to self-learn.²⁵ Clarification of concepts that are not well-understood may not always occur immediately, as in F2F instruction.²⁶ The capacity to monitor student examinations and assessments is not as well-controlled as in F2F, which may result in increased cheating in some students.²⁵ Perhaps with CAI cheating may be easier unless student assessment processes are designed to minimize potential for cheating.

Our research question was, “How effective is the use of CAI to teach PE among students and trainees in the health sciences professions in comparison with F2F teaching of this topic without CAI in improving knowledge and skill acquisition?” The primary objective was to explore knowledge and skill acquisition outcomes related to learning PE through a computer-assisted approach as opposed to a F2F approach.

Methods

A protocol²⁷ was developed in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P)²⁸ and published on PROSPERO, an international prospective register for systematic reviews.²⁹

Study eligibility criteria

Only studies that explored the use of CAI to teach PE using virtual learning environments were included in this review. Studies were included if participants consisted of health professions students at the baccalaureate level or higher or trainees such as interns, residents, or fellows. Physical examination can be executed differently across the health professions. Therefore, a full head-to-toe PE or any component of the examination was eligible for study inclusion. No exclusion constraints were placed on the CAI in accordance with the duration of the intervention or overall time required for completion of the CAI. A F2F comparison group was necessary to be considered for inclusion. If studies included additional arms such as blended learning groups, the study was still eligible for inclusion, but only data comparing CAI vs F2F instruction were used in the analysis.

Studies were excluded if the design was a systematic review or used a qualitative method. Additional exclusion criteria included lack of a focus on PE, lack of a comparison group, and absence of evaluation of knowledge or skill acquisition outcomes, and inclusion of health care professionals.

Outcome measures

Primary outcome measures were the final evaluation of knowledge and/or skill acquisition scores. Studies objectively measured knowledge by administering short-answer or multiple choice questions (MCQ), exams and skills by objective structured clinical exams (OSCEs), or rubrics administered by trained faculty.

Search strategy

An initial limited search of Medline (Ovid) and CINAHL (Ebsco) was undertaken followed by analysis of the text words contained in the title and abstract and of the index terms used to describe the article. Studies published in English between January 2001and December 2016 were considered for inclusion to highlight technological advancements in online education. Information relevant to the inclusion criteria was imported to a bibliographic software package, EndNote X7.

Electronic search of databases

Before the search process began, a concept map (Supplementary Appendix 1) was created to generate terms for the search process. After examining relevant articles retrieved from the initial search, keywords were adapted and developed to form the basis of the final search strategy (Supplementary Appendix 2). Certain key terms were incompatible with the controlled vocabulary of certain databases. This was controlled through the use of equivalent MeSH searches. Boolean terms such as “OR” and “AND” were used to broaden and refine the search strategy. The combination of keywords that related to CAI, health profession students, assessment, and physical exam skills yielded a limited number of papers. Therefore, the search strategy was reevaluated and expanded to include keywords in relation to health care students and clinical assessments.

Searching for gray literature or unpublished articles

The search of unpublished literature included doctoral dissertations, conference proceedings, and related Web sites. The reference lists of relevant studies were handsearched. A list of sources searched for gray literature is included in Supplementary Appendix 3.

Data selection and extraction

Titles and abstracts were independently screened by 2 review authors. Full-text reviews of the remaining articles were conducted and independently evaluated to determine eligibility for inclusion. In cases where the 2 review authors were not in agreement, a third author was available for discussion to achieve a group consensus. Two review authors independently performed data extraction³⁰ (Supplementary Appendix 4).

Risk of bias assessment

During the data extraction process, researchers independently assessed the risk of bias (ROB) for each study using the Cochrane Collaboration’s ROB tool.³⁰ Evaluation criteria included the following: random sequence generation, allocation concealment, blinding of students and personnel, blinding of outcome assessment, incomplete outcome data, selective reporting, or others which included publication bias. Funnel plots were used to evaluate publication bias. Risk of bias for each criterion was rated as low, high, or unclear according to the Cochrane ROB instructions. No overall within study ROB was assigned. Although a subanalysis of effects based on ROB criteria was planned, this was not possible due to the limited number of studies providing data sufficient for meta-analysis.

Data synthesis and analysis

All studies were synthesized according to study design, CAI intervention, and outcome measures. Individual data from each study was input into OpenMeta[Analyst] for statistical analysis³¹ and confidence intervals (CIs) for the heterogeneity statistic (I²) were computed in Stata 14.³² For continuous outcomes data, a pooled mean difference (MD) using the DerSimonian-Laird random-effects model for meta-analysis was calculated for each outcome. In studies that did not report data using a 100% scale, all knowledge and skill scores were standardized to a 1 to 100 point (100%) scale.^8,9,33 Separate analyses were conducted for knowledge and skill acquisition. For crossover designs, data were analyzed post-CAI prior to the crossover only. In studies that evaluated outcomes separately, data were aggregated to determine a mean and standard deviation on PE outcomes only.³³ Differences were considered significant at P < .05. The degree of heterogeneity was assessed using the I² statistic. I² values were evaluated as 0% to 40% low, 30% to 60% moderate, 50% to 90% substantial, and 75% to 100% considerable variation.³⁰ Because CAI length exposure varied between studies, an a priori hypothesis was formulated to assess whether the outcome effect was influenced by the length of CAI exposure. Differences were considered significant at P < .05. If unable to calculate an effect size using the reported data, additional information was requested from study authors. Results of included studies that could not be combined in the meta-analyses are narratively synthesized with respect to the meta-analysis results.

Grade and quality of evidence

The Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) approach was used to create a quality of evidence assessment table for studies included in the meta-analysis.³⁴ Outcomes included knowledge and skills. Overall quality of evidence for each outcome was downgraded 1 level for serious and 2 levels for very serious assessments.

Results

The systematic search of the literature identified 714 references, and 26 additional records, of which 422 were maintained after removal of duplicates (Figure 1). Following completion of title and abstract screening, 404 records were excluded, which resulted in 18 articles for full-text review. Of those 18 articles, 11 were excluded due to an absence of a F2F comparison group, study designs exclusive to blended learning, no virtual learning platform, or the CAI focus was not PE. This resulted in 7 studies that met inclusion criteria in this systematic review.

Figure 1.

Flowchart of the studies included in the systematic review and meta-analysis.

Included studies

The studies comprised in this systematic review were published between 2004 and 2014 in peer-reviewed journals.^{8–10,22,33,35,36} Of these, 3 investigated CAI with medical,^9,35,36 2 with dental (orthodontic),^8,10 and 2 with physical therapy students.^21,33 In total, 6 of the 7 studies addressed skill acquisition^{8,9,22,33,35,36} and 5 knowledge assessment.^{9,10,22,35,36} Study characteristics are described in Table 1.

Table 1.

Study characteristics: the effectiveness of computer-assisted instruction to teach PE in health professions students/trainees.

Citation	Country	Study design	Population	CAI	Comparison	Assessment	Outcomes
van Duijin et al^33,a	United States	Single blind Crossover Experimental Cohort	53 physical therapy graduate students	Psychomotor examination of the cervical region housed in Angel. Unlimited access for 2 d	F2F 1-h laboratory sessions. Each skill is demonstrated twice	6-section OSCE 2 d postintervention	No differences in skill (P = .09) or knowledge (P = .31). CAI is equally effective as F2F for knowledge and skills
Bains et al^10,b	United Kingdom	Prospective Cluster Randomized design	90 fourth year orthodontic students	Basic cephalometric examination housed in WebCT version 3.8 45-min e-learning videos	45-min F2F session	10 MCQ immediately postintervention	Students in CAI answered 4 MCQ statistically less (P < .05) in person. CAI may be less effective than in person for knowledge
Al-Riyami et al^8,*	United Kingdom	Crossover trial	30 orthodontic graduate students	TMJ module developed for Moodle Available for 2 wk	50-min F2F seminar, with 20-min practical demonstrations. Handouts also provided	4-section skills checklist occurred 3 wk postinstruction	No differences (P = .78) in skill acquisition. CAI is equally effective for skills
Averns et al⁹	Canada	Randomized blinded design	61 first year medical students	Nonemergency rheumatological hand conditions Web-based module with unlimited access to materials for 7 d	F2F session using demonstrations and power points with photos. (Same as online content.) Ability to practice the PE live on each other	28-point OSCE and 10-point short-answer exam occurred 7 d postintervention	No differences (P = .31) in skill acquisition. Average total knowledge scores higher than in person (P = .04). CAI is equally effective for knowledge and skills
Ford et al²²	United States	Prospective Cluster Randomized stratified design	43 first semester physical therapy students	Musculoskeletal special tests housed in Windows Media Player. A 25-min intervention viewable twice	25-min F2F session Handouts also provided	8-section practical exam and 20 MCQ and short-answer exam occurred same day postintervention	All groups demonstrated significant increases in learning (P < .0001) CAI is equally effective for knowledge and skills
Qayumi et al^35,b	Japan	Randomly assigned Pretest post-test design	99 preclinical medical students	Abdominal examination module, Cyberpatient. Interactive history and PE via an online tutorial. Time limit of 4 h	F2F text module of Cyberpatient. Included history and PE. Time limit of 4 h	5 OSCE clinical skill test and 40 MCQ occurred on the same day as the intervention	Significant improvements in knowledge in CAI group (P < .01). No differences in skill acquisition. CAI is more effective than in person to increase knowledge
Kurihara et al³⁶	Japan	Randomly assigned Pretest post-test design	60 preclinical third year medical students	Abdominal examination module, Cyberpatient. Interactive history and PE via an online tutorial. Time limit of 4 h	F2F text module of Cyberpatient. Included history and PE. Time limit of 4 h	5 OSCE clinical skill test and 32 MCQ occurred on the same day as the intervention	Significant improvements in knowledge and skill acquisition in both CAI and F2F groups (P < .001)

Abbreviations: CAI, computer-assisted instruction; F2F, face-to-face; MCQ, multiple choice questions; PE, physical examination; OSCE, objective skills competency examination.

Data from studies with crossover designs or blended learning are representative of pre-F2F intervention data included in meta-analysis.

Not included in meta-analysis. α level (^a P = .05).

Study characteristics

Across the 7 studies, there was a total of 436 combined students: 220 medical,^9,35,36 120 orthodontic,^8,10 and 96 physical therapy students.^21,33 A total of 275 students were included in the 5 papers in the meta-analysis: 148 medical, 30 orthodontic, and 97 physical therapy students. Study populations ranged from 30 to 99 students, with comparable samples for CAI and F2F comparison groups. Students across studies had no formal training in PE, and study samples were at the same point in their curriculum. Of the 7 studies that compared CAI with F2F teaching methods, 2 also made comparisons with blending learning, but these data were not used for analysis. Studies were conducted at universities in Canada,⁹ Japan,^35,36 United Kingdom,^8,10 and the United States.^21,33

Risk of bias

All studies had one or more areas considered to have an unclear ROB (Figure 2).^{8–10,22,33,35,36} Students were randomized in all studies. Random sequence generation was classified as an area for high ROB by Ford et al,²² they performed stratified random sampling based on exam scores from a previous semester. Incomplete outcomes data were considered a high ROB in Averns et al⁹ and Bains et al.¹⁰ Considering that CAI and F2F have distinct features, once initiated, the intervention may become obvious to the participants. Blinding was considered to be a high ROB in 5 studies because the outcome was likely influenced by the lack of blinding.^8–10,22,33 Regarding publication bias, there were too few studies to conduct a formal test of plot symmetry.³⁰ Visual examination of the plots (Figures 3 and 4) does not suggest any obvious asymmetry. The reader is cautioned, however, that this does not rule out publication bias; although without more information, it is difficult to determine how systematic publication bias could affect the results.

Figure 2.

Risk of bias assessment for studies included in the systematic review.

Figure 3.

Knowledge outcomes funnel plot.

Figure 4.

Skill acquisition outcomes funnel plot.

Intervention characteristics

The CAI consisted of Web-based modules with detailed instruction on exam conduct and assessment housed in digital learning platforms in 3 studies.^8,9,22 Computer-assisted instruction of Bains et al¹⁰ was an e-learning video tutorial housed in WebCT. Windows Media Player³⁷ was used to provide an educational video as the CAI on musculoskeletal special tests and exams.²¹ Cyberpatient, an interactive history and physical module with a focus on abdominal examination, was used as the CAI for Qayumi et al³⁵ and Kurihara et al.³⁶ The duration, frequency, and time to administer the CAI varied across the studies in this systematic review. Duration of the CAI ranged from 25 minutes up to 2 weeks, frequency ranged from twice to unlimited access, and assessment measures ranged from the same day to 3 weeks postintervention.

Meta-analysis on primary outcomes measures

Meta-analysis was conducted separately on final knowledge and skill acquisition outcomes. In total, 5 of the 7 studies reported knowledge outcomes^{9,10,21,35,36}; however, the remaining 2 did not include mean values or standard deviations and thus were not included in the meta-analysis.^10,35 In total, 6 of the 7 studies^{8,9,22,33,35,36} reported skill acquisition outcomes, but 1 of these 6³² only included mean change scores, and, we did not have an estimate of the standard deviation of the mean, therefore was not included in the meta-analysis. A meta-analysis was conducted with 3 studies^9,22,36 for knowledge outcomes and 5 studies for skills outcomes.^8,9,22,33,36

Knowledge outcomes

Ford et al,²² Averns et al,⁹ and Kurihara et al³⁶ were included in the meta-analysis on knowledge outcomes. Because only 3 of the included studies reported change in knowledge and skills over the course of the instruction, only the final standards-based scores were available for analysis. Knowledge of PE was evaluated by short-answer questions or MCQ exams and was reported on a 10-point scale (converted to 100% scale for meta-analysis). Among the studies, knowledge questions ranged from 10 to 40 MCQ or short-answer questions.^9–22,36 No statistically significant difference was found between CAI and F2F instructions (MD = 5.39, 95% CI: −2.05 to 12.84; the positive value indicated preference for CAI) (Table 2, Figure 5). However, the reader is cautioned regarding the interpretation of the finding of no difference for 3 reasons. First, the number of studies is very limited, and so, a failure to find a statistically significant difference is likely the result of a lack of power. Second, the direction of effect was not consistent. This could indicate that systematic differences in the design of the arms or in the context of application could affect the preference for either CAI or F2F in any particular setting or study. Third, heterogeneity was high for knowledge (I² index = 87.06%, 95% CI: 49%-94%, P < .001). This also lends support to the notion that a preference for CAI versus F2F may depend on arm design and context characteristics and point to the importance of an examination of these characteristics. Meta-regressions were not conducted to analyze the effect of length of CAI exposure on participant’s knowledge and skill acquisition outcomes due to inadequate power.

Table 2.

Results of meta-analysis of computer-assisted instruction studies on knowledge and skill outcomes.

Outcome	Year	Weight	Effect	LCI	UCI	CAI	F2F
Knowledge
Averns	2009	36.62	10.00	6.80	12.33	26	18
Ford	2005	34.64	−0.80	−5.14	3.54	11	11
Kurihara	2004	28.74	7.00	−0.19	14.19	15	15
Overall (I² = 87.06%, P ⩽ .001)			5.39	−2.05	12.84	52	44
Skills
van Duijin	2014	34.88	−4.18	−8.28	−.0.08	27	26
Al-Riyami	2010	12.09	−1.48	−15.25	12.29	15	15
Averns	2009	22.68	4.20	−03.93	12.33	26	18
Ford	2005	15.44	−3.90	−15.40	7.60	11	11
Kurihara	2004	14.91	11	−0.81	22.81	15	15
Overall (I² = 49.98%, P = .092)			−2.24	−6.11	1.63	94	85

Abbreviations: CAI, computer-assisted instruction; F2F, face-to-face; LCI, lower confidence interval; UCI, upper confidence interval.

Figure 5.

Effect of computer-assisted instruction on knowledge outcomes.

In the 2 studies excluded from the meta-analysis, contrasting results were found,^10,35 which were similar to Averns et al⁹ and Ford et al.²² Bains et al found CAI to be a less effective instructional method for knowledge outcomes (P < .05) in comparison with F2F instruction for 4 of 10 MCQ. In contrast, Qayumi et al³⁵ found significant improvements in total knowledge scores from pre- to post-test on 40 MCQ in the CAI group (P < .01) as compared with in-person group. The results of the meta-analysis (which amount to less than 5% difference in knowledge scores between the CAI and in-person conditions) along with the split between studies in which conditions had the largest effect likely point to the highly localized nature of the comparison. Factors such as instructor skill, quality of the curriculum, quality of the CAI design, and the potential differences based on amount of time allowed to interact with the material may all indicate that there is no universal difference between modalities.

Skill acquisition outcomes

Al-Riyami et al,⁸ Averns et al,⁹ Ford et al,²² van Duijin et al,³³ and Kurihara et al³⁶ were included for skill acquisition outcomes.³⁰ Assessment of skill performance varied; Al-Riyami et al⁸ used a 4-section skills checklist, whereas Ford et al²² used a comparable 8-section practical exam. In the remaining 3 studies, a 5-section, 6-section, and 28-point OSCEs were administered.^9,33,36 There was no statistically significant difference between CAI and F2F instructions (MD = 0.35, 95% CI: −5.30 to 6.01; the negative favoring the F2F group) (Table 2, Figure 6), though, as with the knowledge outcomes, direction of effect was inconsistent among studies. The point estimate for heterogeneity was moderate (I² index = 49.98%, 95% CI: 0%-83%, P = .092), but the CI for the heterogeneity was very wide, again, and indicated a potentially strong effect of intervention design and context characteristics.

Figure 6.

Effect of computer-assisted instruction on skill acquisition outcomes.

Qayumi et al,³⁵ 1 of the 2 studies excluded from the meta-analysis, used a 5-section OSCE and found no significant differences in skill acquisition between the CAI and F2F groups. However, mean OSCE change scores were higher for CAI (8.74 + 2.60) than F2F (6.98 + 2.93). This finding was similar to Averns et al (small, nonsignificant findings in favor of the CAI group) but contrasts with both Ford et al and Al-Riyami et al (both reporting small, nonsignificant differences favoring the in-person group). Only van Duijin et al reported a 4.2% statistically significant difference favoring the F2F group. As with the knowledge outcome, the small (all within ±5% of each other), inconsistent difference between groups likely indicates a sensitivity to local differences in instructor skills and the design of the 2 interventions. Again, there is no evidence for a strong, consistent preference for one approach over the other.

Discussion

Summary of results

This systematic review and meta-analysis analyzed the effectiveness of CAI in comparison with F2F instruction to teach PE to health professions students and trainees. Seven studies met the inclusion criteria for this review^{8–10,22,33,35,36} (Table 1). Meta-analysis was conducted for knowledge and acquisition of skills in 5 of the studies which included students in dentistry, medicine, and physical therapy.^{8–10,22,33,36} No statistically significant differences were found in knowledge or skill acquisition outcomes related to learning PE through a CAI approach as opposed to a F2F approach. This finding must be interpreted with caution. The results are inconclusive regarding a preference for one mode or another. However, our further analyses lead us to suspect that a more nuanced interpretation is needed. In itself, a small sample size only indicates that the meta-analytic findings were likely underpowered. The lack of significance does not tell us whether future comparisons are likely to find a clear statistically significant preference for one mode of instruction over the other. Indeed, based on an examination of (1) the inconsistency of the effects between studies and (2) the high levels of heterogeneity among studies, we suspect that it may be adverse to simply ask whether there is a difference in knowledge or skill acquisition effect between the approaches. Rather, given the weak evidence of inconsistent effects, and high heterogeneity, we believe that a comparative approach that focuses strongly on differences in study and intervention design, sample characteristics, and context or setting characteristics is needed to more fully understand when, for whom, and under what conditions one mode may outperform the other. Because of the limited number of studies available, we could only suggest this type of approach, given the small number of studies available and therefore the low statistical power for meta-analyses. However, the combination of low heterogeneity for both analyses, in addition to the small (±5%), inconsistent direction of effect leads us to conjecture that even had power been greater, and it is unlikely that we would have found a strong, consistent preference for one teaching modality. In short, there is unlikely to be a single universal preference for one method of instruction over the other.

Of those studies which assessed knowledge outcomes, CAI was found to be equally effective as F2F teaching methods and was slightly favored in comparison with F2F instruction.^9,22 The effectiveness of CAI varied in the 2 studies not included in the meta-analysis. Bains et al¹⁰ found that CAI may be less effective than F2F instruction because the F2F group scored statistically higher (P < .05) than the CAI group on 4 questions (out of 10 MCQ). However, limits were placed on the CAI tutorials, which removed the flexibility of the intervention and which may have negatively affected the results for this arm. Providing materials that are not dependent on time and do not have restrictions to access or space can promote student responsibility and motivation for learning.^38,39 In contrast, Qayumi et al³⁵ found that CAI significantly (P < .01) improved knowledge in comparison with traditional learning methods in medical students learning abdominal PE. The investigators encouraged medical students to use the CAI, which may have resulted in the increase in knowledge scores. The design of the CAI curriculum likely has a strong impact on its effectiveness relative to a F2F condition.

Although no differences resulted favoring the effectiveness of F2F instruction versus CAI as teaching methods for knowledge outcomes, there were important methodological and educational design variations between the interventions in each study. Knowledge evaluation was conducted using a varying number of MCQ and short-answer questions per assessment. Similarly, this varying quantity of exam categories also existed among the OSCEs and scoring rubrics for the skill acquisition assessment. Although not statistically significant, there were variations in assessment of skills among disciplines which makes it difficult to generalize the results.

All of the studies had variations in CAI intervention duration, frequency of CAI access, and times until assessment. Ford et al initially assessed knowledge the same day as the intervention and retention 3 weeks post-CAI. Although scores were lower, no significant decreases occurred from immediately post to 3 weeks post-CAI.²² van Duijin et al evaluated knowledge 2 days postintervention. There are no published guidelines on the appropriate time to test knowledge, skill, or even retention after a technology-based intervention.

The use of CAI as an instructional method may be perceived negatively by some students and impact outcomes. The sample of students of Al-Riyami et al reported difficulties accessing the course due to network difficulties with the university’s server such as browser compatibility and Internet band with problems.⁸ The asynchronous features of the discussion boards were not used to the full potential of the course in this study. Both of these problems could have emerged regardless of the online course. In traditional classrooms, students could choose not to engage in discussions, and Internet connectivity issues can happen anywhere. This supports the contention above that local conditions, rather than a general effect, may render one or the other mode of instruction preferable to the other.

Achievement levels and acceptance of CAI can affect student outcomes. Student acceptance of CAI was assessed through student feedback in 4 studies.^8,10,35,36 Students provided ratings using Likert scales in 2 studies.^8,10 Type of student performer was evaluated by Qayumi et al³⁵ and Kurihara et al³⁶ to determine whether achievement levels affected knowledge or skill acquisition. Qayumi³⁵ found that medium achieving students in the CAI group had significantly higher change scores for both knowledge and skill acquisition in comparison with low- or high-achieving students. In contrast, Kurihara et al³⁶ found that low-achieving students performed better with CAI, specifically with the skills assessment. Determining the type of learner or achiever, a student can better prepare faculty on CAI techniques to educate students. Bains et al¹⁰ found that students preferred F2F or blending learning teaching methods over CAI, which may have negatively affected knowledge outcomes. Students in the study of Bains et al perceived CAI as a supplemental addition to F2F learning and not the core component of their education. Contrastingly, students in the study of Averns et al⁹ felt that the online CAI was very comparable with traditional methods and found the online modules practical and easier to use and to navigate. Averns et al⁹ also stated that the students’ scores may have been positively affected due to the planned knowledge of the clinical skills assessment and the desire for high achievement.

Quality of evidence

Quality and reporting of the 7 randomized studies included in this review was assessed according to the GRADE criteria.³⁴ The quality of the evidence was low for CAI in comparison with F2F instruction for teaching PE for both knowledge and skill acquisition outcomes (Table 3). Positive criteria in the quality assessment included randomized study designs and overall low ROB. Limitations noted were due to small sample sizes and wide CIs. However, the inconsistency of the findings may point to high local sensitivity and the possibility that preference for CAI or F2F is dependent on the study characteristics and local conditions.

Table 3.

Quality of evidence assessment: the effectiveness of computer-assisted instruction in comparison with face-to-face instruction.

Quality assessment						Effect			Quality
No. of studies	Study design	Risk of bias	Inconsistency	Indirectness	Imprecision	CAI	F2F	Absolute (95% CI)	Grade
Knowledge3	Randomized trials	Serious^a	Not serious	Not serious	Serious^a	52	44	MD 5.39 higher (2.05 lower to 12.84 higher)	⨁⨁⃝⃝Low
Skills5	Randomized trials	Serious^a	Not serious	Not serious	Serious^a	94	85	MD 0.35 higher (5.30 lower to 6.01 higher)	⨁⨁⃝⃝Low

Abbreviations: CAI, computer-assisted instruction; CI, confidence interval; F2F, face-to-face instruction; MD, mean difference.

Small sample size and wide confidence intervals.

Strengths and limitations

This study has several strengths. The review was based on an in-depth search of the literature following the PRISMA-P guidelines.²⁸ A protocol was developed, published, and followed prior to the literature search.²⁷ Two reviewers screened and extracted the data to ensure data quality and avoid subjective bias. Randomization occurred in all of the studies. Because the number of studies that met the inclusion criteria was limited, this study attempted to combine all health professions into one review. Much of the literature on PE is by individual health profession, and this review includes several professions.^18,19,40 Finally, this is the first published systematic review to our knowledge to evaluate the effectiveness of CAI in comparison with F2F instructional methods of PE for health professions students and/or trainees.

This review also had limitations. The number of studies included is small due to the lack of an F2F comparison group in many of the excluded studies. There was heterogeneity among the study methods regarding CAI intervention type, duration, and assessment procedures, which limits the conclusions that can be drawn about CAI.

However, with only 2 studies available for the meta-regression, the analysis was not adequately powered to determine whether a <2% increase in knowledge score for every additional hour of CAI allowed is likely to be reproduced in other studies. Validity of the tools used to assess learner outcomes was not specified in all of the papers reviewed. Averns et al⁹ used a validated OSCE, and the scoring rubric of van Duijin et al³³ had not stated the reliability but had been adapted from the evaluation tool of Ford that had superior test-retest reliability.²² The knowledge test of Ford et al²² was only evaluated for face validity. Only medical, dental, and physical therapy professions were included in the sample, and thus, not all health professions are represented. The studies cited were conducted in 4 different countries, where different teaching practices and learning styles may be present. Two studies had attrition rates greater than 50%, which were accounted for in the ROB assessment.^9,10 Bains et al¹⁰ attributed the high dropout rate to the students being volunteers. However, it was unclear whether or not recruitment or volunteer bias had occurred in the remainder of the studies.

Implications for practice

The implementation of CAI to teach PE in health professions curricula offers numerous opportunities. However, whether this mode of instruction provides a benefit over F2F appears to depend heavily on educational design and accessibility. Online learning is convenient, flexible and can accommodate both faculty and student schedules. Conceptually, if students access material when it is most convenient, they will be learning when they are most receptive to the material.⁹ Faculty can record lectures and videos, and revise as needed, at times that are convenient according to their workloads and schedules. Using CAI may also afford educators the opportunity to enhance innovative teaching methods. They can take PE from a traditional textbook and create a video demonstration of the PE for students to watch in class or at home. This may enhance student experiences, course evaluations, and overall job satisfaction. There is a potential cost savings related to resources such as travel, physical space, and faculty time. With growing student numbers on university campuses, scheduling adequate lecture hall space is challenging. This challenge can be minimized with virtual classes. The initial creation of CAI resources could be expensive and time-consuming; however, few studies exist that have evaluated the economics of CAI.^41–44

Implications for research

The findings suggest several implications for future research. Because this review only included 3 health professions, future research should encompass a greater number of health professions for which PE is a competency and include a F2F comparison group. Because teaching PE requires development of knowledge and skill acquisition, studies should assess both outcomes and knowledge and skill retention over time. It is crucial that knowledge and skills are maintained for use in the clinical practice setting. Similarly, use of knowledge and skill assessment in clinical practice rotations would be a valuable area of study. Future research should also pay close attention to differences in educational design of the different modes of delivery.

Conclusions

The evidence from this systematic review does not recommend one mode of instruction over the other. Because the results were inconclusive about the direction and size of the effect, a preference for CAI or F2F instruction cannot be suggested. However, it is reasonable to conclude, based on the evidence, that further research is critical to identify the conditions that appear to favor one mode of instruction versus the other method. Additional research is also needed in all health professions students and trainees comparing PE using CAI in comparison with F2F teaching methods.

Footnotes

Peer review:

Four peer reviewers contributed to the peer review report. Reviewers’ reports totaled 1987 words, excluding any confidential comments to the academic editor.

Funding:

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Declaration of conflicting interests:

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Author Contributions

JT and RT-D conceived and designed the experiments. JT and SP analyzed the data. JT wrote the first draft of the manuscript. JT, RT-D, MD, RZ, and SP contributed to the writing of the manuscript; agree with manuscript results and conclusions; jointly developed the structure and arguments for the paper; made critical revisions and approved final version; and reviewed and approved the final manuscript.

Disclosures and Ethics

As a requirement of publication, author(s) have provided to the publisher signed confirmation of compliance with legal and ethical obligations including but not limited to the following: authorship and contributorship, conflicts of interest, privacy and confidentiality, and (where applicable) protection of human and animal research subjects. The authors have read and confirmed their agreement with the ICMJE authorship and conflict of interest criteria. The authors have also confirmed that this article is unique and not under consideration or published in any other publication, and that they have permission from rights holders to reproduce any copyrighted material. Any disclosures are made in this section. The external blind peer reviewers report no conflicts of interest.

References

Hussain

. e-learning 3.0 = e-learning 2.0 + web 3.0 IOSR. J Res Method Educ. 2012;3:39–47.

National Center for Education Statistics. U.S. Department of Education. Digest of Education Statistics, 2015 (NCES 2016-014), Table 311.15. https://nces.ed.gov/fastfacts/display.asp?id=80. Published 2016. Accessed March 15, 2017.

George

Papachristou

Belisario

. Online eLearning for undergraduates in health professions: a systematic review of the impact on knowledge, skills, attitudes and satisfaction. J Glob Health. 2014;4:010406.

Bandla

Franco

Simpson

Brennan

McKanry

Bragg

. Assessing learning outcomes and cost effectiveness of an online sleep curriculum for medical students. J Clin Sleep Med. 2012;8:439–443.

Chudakoff

Obuchowski

Mehta

Reid

. Worldwide utilization of a web-based learning tool for pediatric radiology. Am J Roentgenol. 2013;200:975–979.

Perkins

Kimani

Bullock

. Improving the efficiency of advanced life support training: a randomized, controlled trial. Ann Intern Med. 2014;157:19–28.

Hayward

. Integrating web-enhanced instruction into a research methods course: examination of student experiences and perceived learning. J Phys Ther Educ. 2004;18:54–65.

Al-Riyami

Moles

Lesson

Cunningham

. Comparison of the instructional efficacy of an internet-based temporomandibular joint (TMJ) tutorial with a traditional seminar. Br Dent J. 2010;209:571–576.

Averns

Maraschiello

van Melle

Day

. Evaluation of a web-based teaching module on examination of the hand. J Rheumatol. 2009;36:623–627.

10.

Bains

Reynolds

McDonald

Sherriff

. Effectiveness and acceptability of face-to-face, blended and e-learning: a randomized trial of orthodontic undergraduates. Eur J Dent Educ. 2011;2:110–117.

11.

Buijze

Guitton

van Dijk

Ring

. Training improves interobserver reliability for the diagnosis of scaphoid fracture displacement. Clin Orthop Relat Res. 2012;470:2029–2034.

12.

Kalet

Song

Sarpel

. Just enough, but not too much interactivity leads to better clinical skills performance after a computer assisted learning module. Med Teach. 2012;34:833–839.

13.

Modica

. A136: assessing the efficacy of pediatric rheumatology video-based curriculum. Arthritis Rheum. 2014;66:S179.

14.

Ronn

Smith

Magee

Hahn

Reid

. Can online learning adequately prepare medical students to undertake a first female pelvic examination? J Obstet Gynaecol Can. 2012;34:264–268.

15.

Cook

Levinson

Garside

. Internet-based Learning in the health professions: a meta-analysis. JAMA. 2008;300:1181–1196.

16.

Cook

Erwin

Triola

. Computerized virtual patients in health professions education: a systematic review and meta-analysis. Acad Med. 2010;85:1589–1602.

17.

Cook

Hatala

Brydges

. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA. 2010;306:978–988.

18.

Veneri

. The role and effectiveness of computer-assisted learning in physical therapy education: a systematic review. Physiother Theory Pract. 2011;27:287–298.

19.

McCutcheon

Lohan

Traynor

Martin

. A systematic review evaluating the impact of online or blended learning vs. face-to-face learning of clinical skills in undergraduate nurse education. J Adv Nurs. 2014;71:255–270.

20.

Adams

. A comparison of student outcomes in a therapeutic modalitites course based on mode of delivery: hybrid versus traditional classroom instruction. J Phys Ther Educ. 2013;27:20–34.

21.

El Mansour

Mupinga

. Students’ positive and negative experiences in hybrid and online classes. Coll Stud J. 2007;41:242–248.

22.

Ford

Mazzone

Taylor

. Effect of computer-assisted instruction versus traditional modes of instruction on student learning of musculoskeletal special tests. J Phys Ther Educ. 2005;19:22–30.

23.

Smith

Jones

Cavanaugh

Venn

Wilson

. Effect of interactive multimedia on basic clinical psychomotor skill performance by physical therapist students. J Phys Ther Educ. 2006;20:61–67.

24.

DeBourgh

. Psychomotor skills acquisition of novice learners: a case for contextual learning. Nurse Educ. 2011;36:144–149.

25.

Arkorful

Abaidoo

. The role of e-learning, advantages and disadvantages of its adoption in higher education. Int J Instruct Technol Distance Learn. 2015;12:29–42.

26.

Armstrong

Elliott

Ronald

Paterson

. Comparison of traditional and interactive teaching methods in a UK emergency department. Eur J Emerg Med. 2009;16:327–329.

27.

Tomesko

Touger-Decker

Dreker

Zelig

. How effective is the use of computer assisted instruction to teach physical examination skills among students and trainees in the health sciences professions in comparison to in person teaching of this topic without computer assisted instruction in improving knowledge and skill development? PROSPERO 2016:CRD42016041794. http://www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42016041794.

28.

Shamseer

Moher

Clarke

; PRISMA-P Group. Preferred Reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;349:g7647

29.

PROSPERO International prospective register of systematic reviews. University of York. https://www.crd.york.ac.uk/PROSPERO/. Accessed March 15, 2017.

30.

Higgins

Green

, eds. Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collaboration; 2009. http://www.cochrane-handbook.org.

31.

Open Meta[Analyst]. Brown University. http://www.cebm.brown.edu/openmeta/. Accessed June 12, 2017.

32.

StataCorp. Stata Statistical Software: Release 14. College Station, TX: StataCorp LLC; 2017.

33.

van Duijin

Swanick

Kroog

. Student learning of cervical psychomotor skills via online video instruction versus traditional face-to-face instruction. J Phys Ther Educ. 2014;28:94–102.

34.

Gordon

Oxman

Akl

. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64:383–394.

35.

Qayumi

Kurihara

Imai

. Comparison of computer-assisted instruction (CAI) versus traditional textbook methods for training in abdominal examination (Japanese experience). Med Educ. 2004;38:1080–1088.

36.

Kurihara

Kuramoto

Matsuura

. Academic performance and comparative effectiveness of computer- and textbook-based self-instruction. St Heal T. 2004;107:894–897.

37.

Windows Media Player. c2000. https://support.microsoft.com/en-us/help/18612/windows-media-player. Accessed June 12, 2017.

38.

Barker

Grossman

. The learning impact of a virtual learning environment: students’ views. Teach Educ Adv Netw J. 2013;5:19–38.

39.

Cassidy

. Virtual learning environments as mediating factors in student satisfaction with teaching and learning in higher education. J Curriculum Teach. 2016;5:113–123.

40.

Feng

Chang

Erdley

Lin

Chang

. Systematic review of effectiveness of situated e-learning on medical and nursing education. Worldv Evid Based Nu. 2013;10:174–183.

41.

Phadtare

Bahmani

Shah

Pietrobon

. Scientific writing: a randomized controlled trial comparing standard and on-line instruction. BMC Med Educ. 2009;9:27.

42.

Kandasamy

Fung

. Interactive Internet-based cases for undergraduate otolaryngology education. Otolaryngol Head Neck Surg. 2009;140:398–402.

43.

Kong

Wang

Sun

Zhang

. Effect of digital problem based learning cases on student learning outcomes in ophthalmology courses. Arch Ophthalmol. 2009;127:1211–1214.

44.

Peroz

Beuche

Peroz

. Randomized controlled trial comparing lecture versus self studying by an online tool. Med Teach. 2009;31:508–512.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

The Effectiveness of Computer-Assisted Instruction to Teach Physical Examination to Students and Trainees in the Health Sciences Professions: A Systematic Review and Meta-Analysis

Abstract

Purpose:

Method:

Results:

Conclusions:

Keywords

Introduction

Methods

Study eligibility criteria

Outcome measures

Search strategy

Electronic search of databases

Searching for gray literature or unpublished articles

Data selection and extraction

Risk of bias assessment

Data synthesis and analysis

Grade and quality of evidence

Results

Included studies

Study characteristics

Risk of bias

Intervention characteristics

Meta-analysis on primary outcomes measures

Knowledge outcomes

Skill acquisition outcomes

Discussion

Summary of results

Quality of evidence

Strengths and limitations

Implications for practice

Implications for research

Conclusions

Footnotes

Peer review:

Funding:

Declaration of conflicting interests:

Author Contributions

Disclosures and Ethics

References

Supplementary Material