Research methods and statistics are core courses in the undergraduate psychology major. To assess learning outcomes, it would be useful to have a measure that assesses research methods and statistical literacy beyond course grades. In two studies, we developed and provided initial validation results for a research methods and statistical knowledge concept inventory for eventual use in further scholarship of teaching and learning. In Study 1, we created vignettes and administered open-ended questions to psychology subject pool students. In Study 2, we refined the vignettes and created multiple-choice items using participant responses from Study 1. After administering the measure to psychology subject pool students and a community-based sample of Amazon Mechanical Turk workers, we used item response theory to select 20 items to compose the final Psychological Research Inventory of Concepts.
AllenK. (2006). The Statistics Concept Inventory: Development and analysis of a cognitive assessment instrument in statistics (Unpublished doctoral dissertation). University of Oklahoma, Norman, OK.
AndersonD. L.FisherK. M.NormanG. J. (2002). Development and evaluation of the conceptual inventory of natural selection. Journal of Research in Science Teaching, 39, 952–978. doi:10.1002/tea.10053
4.
BachiochiP.EvertonW.EvansM.FugereM.EscotoC.LettermanM.LeszczynskiJ. (2011). Using empirical article analysis to assess research methods courses. Teaching of Psychology, 38, 5–9. doi:10.1177/0098628310387787
5.
BarronK. E.AppleK. J. (2015). Debating curricular strategies for teaching statistics and research methods: What does the current evidence suggest?Teaching of Psychology, 41, 187–194. doi:10.1177/0098628314537967
BristowM.ErkorkmazK.HiussoonJ. P.JoenS.OwenW. S.WaslanderS. L.StubleyG. D. (2012). A control systems concept inventory test design and assessment. IEEE Transactions on Education, 55, 203–212. doi:10.1109/TE.2011.2160946
8.
BurkleyE.BurkleyM. (2009). Mythbusters: A tool for teaching research methods in psychology. Teaching of Psychology, 36, 179–184. doi:10.1080/00986280902739586
ConnersF. A.MccownS. M.Roskos-EwoldsenB. (1998). Unique challenges in teaching undergraduate statistics. Teaching of Psychology, 25, 40–42. doi:10.1207/s15328023top2501_12
11.
DempsterM.McCorryN. K. (2009). The role of previous experience and attitudes toward statistics in statistics assessment outcomes among undergraduate psychology students. Journal of Statistics Education, 17. Retrieved fromhttp://www.amstat.org/publications/jse/v17n2/dempster.pdf
12.
DuckworthA. L.PetersonC.MatthewsM. D.KellyD. R. (2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 9, 1087–1101. doi:10.1037/0022-3514.92.6.1087
13.
EdelenM. O.ReeveB. B. (2007). Applying item response theory (IRT) modeling to questionnaire development, evaluation, and refinement. Quality of Life Research, 16, 5–18. doi:10.1007/s11136-007-9198-0
14.
EvansD. L.GrayG. L.KrauseS.MartinJ.MidkiffC.WageK. (2003). Progress on concept inventory assessment tools. In Proceedings of the 33rd ASEE/IEEE Frontiers in Education Conference (pp. TG41–T4G8).
15.
FacioneP. A.FacioneN. (1994). The California critical thinking skills test (CCTST): Test manual. Millbrae: California Academic Press.
16.
ForesterM.KahnJ. H.Hesson-McInnisM. S. (2004). Factor structures of three measures of research self-efficacy. Journal of Career Assessment, 12, 3–16. doi:10.1177/1069072703257719
17.
FrederickS. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19, 25–42.
18.
FrengS.WebberD.BlatterJ.WingA.ScottW. D. (2011). The role of statistics and research methods in the academic success of psychology majors: Do performance and enrollment timing matter?Teaching of Psychology, 38, 83–88. doi:10.1177/0098628311401591
19.
GoodmanJ. K.CryderC. E.CheemaA. (2012). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26, 213–224. doi:10.1002/bdm.1753
20.
HakeR. R. (2015). What might psychologists learn from scholarship of teaching and learning in physics?Scholarship of Teaching and Learning in Psychology, 1, 100–106. doi:10.1037/stl0000022
21.
HallounI. A.HestenesD. (1985a). Common sense conepts about motion. American Journal of Physics, 53, 1056–1065. doi:10.1119/1.14031
22.
HallounI. A.HestenesD. (1985b). The initial knowledge state of college physics students. American Journal of Physics, 53, 1043–1055. doi:10.1119/1.14030
23.
HowellJ. L.CollissonB.KingK. M. (2014). Physics envy: Psychologists’ perceptions of psychology and agreement about core concepts. Teaching of Psychology, 41, 330–334. doi:10.1177/0098628314549705
24.
LawsonT. J. (1999). Assessing psychological critical thinking as a learning outcome for psychology majors. Teaching of Psychology, 26, 207–209.
25.
LawsonT. J.Jordan-FlemingM. K.BodleJ. H. (2015). Measuring psychological critical thinking: An update. Teaching of Psychology, 42, 248–253. doi:10.1177/0098628315587624
26.
NorrisP.PaciniR.EpsteinS. (1998). The rational-experiential inventory, short form (Unpublished inventory). University of Massachusetts Amherst, Amherst, MA.
27.
OnwuegbuzieA. J.WilsonV. A. (2003). Statistics anxiety: Nature, etiology, antecedents, effects, and treatments—A comprehensive review of the literature. Teaching in Higher Education, 8, 195–209. doi:10.1080/1356251032000052447
28.
OppenheimerD. M.MeyvisT.DavidenkoN. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872. doi:10.1016/j.jesp.2009.03.009
29.
PintrichP. R. (2003). A motivational science perspective on the role of student motivation in learning and teaching contexts. Journal of Educational Psychology, 95, 667–686.
30.
PliskeR. M.CaldwellT. L.Calin-JagemanR. J.Taylor-RitzlerT. (2015). Demonstrating the effectiveness of an integrated and intensive research methods and statistics course sequence. Teaching of Psychology, 42, 153–156. doi:10.1177/0098628315573139
31.
SmithD. L.BarkerL. (2008). Using yes–no recognition tests to assess student memory for course content. Teaching of Psychology, 35, 319–326. doi:10.1080/00986280802374468
32.
StansburyJ. A.MunroG. D. (2013). Gaming in the classroom: An innovative way to teach factorial designs. Teaching of Psychology, 40, 148–152. doi:10.1177/0098628312475037
33.
StoneC. A.YeF.ZhuX.LaneS. (2009). Providing subscale scores for diagnostic information: A case study when the test is essentially unidimensional. Applied Measurement in Education, 23, 63–86. doi:10.1080/08957340903423651
34.
VeilleuxJ. C.ChapmanK. M. (2017). Validation of the Psychological Research Inventory of Concepts: An Index of Research and Statistical Literacy. Teaching of Psychology, 44, 212–221.
35.
VittenglJ. R.BosleyC. Y.BresciaS. A.EckardtE. A.NeidigJ. M.ShelverK. S.…SapenoffL. A. (2004). Why are some undergraduates more (and others less) interested in psychological research?Teaching of Psychology, 31, 91–97. doi:10.1207/s15328023top3102
