Practice Still Makes Proficient: A Replication of Kershaw et al. (2018,2023;Kershaw & Gordon,2024) Using an Alternate Form of the Journal Article Comprehension Assessment

Abstract

Background

Recent research suggests that empirical research comprehension can be improved through structured assignments that distribute deliberate practice opportunities over the semester.

Objective

This recent research used identical Journal Article Comprehension (JAC) assessments of this structured practice in a pretest/posttest design. The current study's aim was to develop an alternate form of the JAC assessment.

Method

Across four semesters, 125 students completed one form of the JAC at the pretest and another at the posttest. These different forms were matched on question type and answer options.

Results

All groups improved in empirical research comprehension between the pretest and posttest, as shown through the total JAC score. This improvement occurred regardless of which combination of JAC articles they received at the pretest or posttest.

Conclusion

The findings suggest that both JACs are equivalent measures of empirical research comprehension. The current results also replicate past research showing overall improvement, that improvement is not just a practice effect, and that improvement can occur with only a few practice opportunities.

Teaching Implications

The equivalence of the new form of the JAC gives educators an alternate way to assess empirical research comprehension within the context of a semester or to assess the transfer of skills across the curriculum.

Keywords

reading empirical articles instructional scaffolds alternate forms assessment cognitive psychology

Get full access to this article

View all access options for this article.

References

American Psychological Association. (2017). Project Assessment (PASS). http://pass.apa.org.

American Psychological Association. (2023). APA guidelines for the undergraduate psychology major: Version 3.0. https://www.apa.org/about/policy/undergraduate-psychology-major.pdf.

Appleby

D. C.

(2018). Preparing psychology majors to enter the workforce: Then, now, with whom, and how. Teaching of Psychology, 45(1), 14–23. https://doi.org/10.1177/0098628317744944

Bachiochi

Everton

Evans

Fugere

Escoto

Letterman

Leszczynski

(2011). Using empirical article analysis to assess research methods courses. Teaching of Psychology, 38(1), 5–9. https://doi.org/10.1177/0098628310387787

Beilock

S. L.

Carr

T. H.

(2005). When high-powered people fail: Working memory and “choking under pressure” in math. Psychological Science, 16(2), 101–105. https://doi.org/10.1111/j.0956-7976.2005.00789.x

Breslin

C. W.

Safer

M. A.

(2011). Effects of event valence on long-term memory for two baseball championship games. Psychological Science, 22(11), 1408–1412. https://doi.org/10.1177/0956797611419171

Calamia

Markon

Tranel

(2012). Scoring higher the second time around: Meta-analyses of practice effects in neuropsychological assessment. The Clinical Neuropsychologist, 26(4), 543–570. https://doi.org/10.1080/13854046.2012.680913

Clause

C. S.

Mullins

M. E.

Nee

M. T.

Pulakos

Schmitt

(1998). Parallel test form development: A procedure for alternate predictors and an example. Personnel Psychology, 51(1), 193–208. https://doi.org/10.1111/j.1744-6570.1998.tb00722.x

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.

10.

Coil

Wenderoth

M. P.

Cunningham

Dirks

(2010). Teaching the process of science: Faculty perceptions and an effective methodology. CBE—Life Sciences Education, 9(4), 524–535. https://doi.org/10.1187/cbe.10-01-0005

11.

Dunlosky

Rawson

K. A.

Marsh

E. J.

Nathan

M. J.

Willingham

D. T.

(2013). Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14(1), 4–58. https://doi.org/10.1177/1529100612453266

12.

Faul

Erdhelder

Lang

A.-G.

Buchner

(2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146

13.

Goudsouzian

L. K.

Hsu

J. L.

(2023). Reading primary scientific literature: Approaches for teaching students in the undergraduate STEM classroom. CBE—Life Sciences Education, 22(3), 1–13. https://doi.org/10.1187/cbe.22-10-0211

14.

Halonen

J. S.

Dunn

D. S.

(2018). Embedding career issues in advanced psychology major courses. Teaching of Psychology, 45(1), 41–49. https://doi.org/10.1177/0098628317744967

15.

Hausknecht

J. P.

Halpert

J. A.

Di Paolo

N. T.

Moriarty Gerrard

M. O.

(2007). Retesting in selection: A meta-analysis of coaching and practice effects for tests of cognitive ability. Journal of Applied Psychology, 92(2), 373–385. https://doi.org/10.1037/0021-9010.92.2.373

16.

Kershaw

T. C.

(2024, July 25). Alternate form JAC project. https://doi.org/10.17605/OSF.IO/C67EB

17.

Kershaw

T. C.

Fugate

J. M. B.

O’Hare

A. J.

(2023). Teaching undergraduates to understand published research through structured practice in identifying key research concepts. Scholarship of Teaching and Learning in Psychology, 9(2), 216–233. https://doi.org/10.1037/stl0000239

18.

Kershaw

T. C.

Gordon

L. T.

(2024). Practice assignments improve research comprehension skills and reinforce concept learning. Scholarship of Teaching and Learning in Psychology, 10(4), 536–551. https://doi.org/10.1037/stl0000337

19.

Kershaw

T. C.

Lippman

J. P.

Fugate

J. M. B.

(2018). Practice makes proficient: Teaching undergraduate students to understand published research. Instructional Science, 46(6), 921–946. https://doi.org/10.1007/s11251-018-9456-2

20.

Millis

Halpern

Wiemer

Wallace

(2021). Evaluating research summaries. In Society for the Teaching of Psychology, Resources for teachers of psychology. https://teachpsych.org/page-1603066.

21.

National Association of Colleges and Employers. (2024). Competencies for a career-ready workforce. https://naceweb.org/docs/default-source/default-document-library/2024/resources/nace-career-readiness-competencies-revised-apr-2024.pdf?sfvrsn=1e695024_6.

22.

Sego

S. A.

Stuart

A. E.

(2016). Learning to read empirical articles in general psychology. Teaching of Psychology, 43(1), 38–42. https://doi.org/10.1177/0098628315620875

23.

Society for the Teaching of Psychology. (n.d.). Resources for teachers of psychology. https://teachpsych.org/page-1603066.

24.

Stoa

Chu

T. L.

Gurung

R. A. R.

(2022). Potential potholes: Predicting challenges and learning outcomes in research methods in psychology courses. Teaching of Psychology, 49(1), 21–29. https://doi.org/10.1177/0098628320979881

25.

Van Lacum

E. B.

Ossevoort

M. A.

Goedhart

M. J.

(2014). A teaching strategy with a focus on argumentation to improve undergraduate students’ ability to read research articles. CBE – Life Sciences Education, 13(2), 253–264. https://doi.org/10.1187/cbe.13-06-0110

26.

Varela

M. F.

Lutnesky

M. M. F.

Osgood

M. P.

(2005). Assessment of student skills for critiquing published primary scientific literature using a primary trait analysis scale. Microbiology Education, 6(1), 20–27. httpsss://doi.org/http://doi.org/10.1128/154288105X14285806518972 https://doi.org/10.1128/me.6.1.20-27.2005

27.

Wenk

Tronsky

(2011). First-year students benefit from Reading primary research articles. Journal of College Science Teaching, 40(4), 60–67. https://sites.hampshire.edu/ctl/files/2014/07/JCST_Wenk.pdf .