Among the variety of selected response formats used in L2 reading assessment, multiple-choice (MC) is the most commonly adopted, primarily due to its efficiency and objectiveness. Given the impact of assessment results on teaching and learning, it is necessary to investigate the degree to which the MC format reliably measures learners’ L2 reading comprehension in the classroom context. While researchers have claimed that the longer the reading test (i.e., more test items and passages), the higher its overall reliability, few studies have investigated the optimal number of items and passages required for reliable classroom-based L2 reading assessment.
To address this research gap, I adopted generalizability (G) theory to investigate the score reliability of the MC format in classroom-based L2 reading tests. A total of 108 ESL students at an American college completed an English reading test that included four passages, each of which was accompanied by five MC comprehension questions. The results showed that the score reliability of the L2 reading test was critically influenced by the number of items and passages, inasmuch as a different combination of the number of passages and items altered the degree of reliability. Implications for practitioners and educational researchers are discussed.
AldersonJ. C. (2000). Assessing reading. Cambridge University Press.
2.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
3.
AndersonN. J.BachmanL.PerkinsK.CohenA. (1991). An exploratory study into the construct validity of a reading comprehension test: Triangulation of data sources. Language Testing, 8(1), 41–66. https://doi.org/10.1177/026553229100800104
4.
BachmanL. F. (1990). Fundamental considerations in language testing. Oxford University Press.
5.
BachmanL. F.LynchB. K.MasonM. (1995). Investigating variability in tasks and rater judgments in a performance test of foreign language speaking. Language Testing, 12(2), 238–257. https://doi.org/10.1177/026553229501200206
6.
BachmanL. F.PalmerA. S. (1982). The construct validation of some components of communicative proficiency. TESOL Quarterly, 16(4), 449–465. https://doi.org/10.2307/3586464
7.
BachmanL. F.PalmerA. S. (1996). Language testing in practice: Designing and developing useful language tests. Oxford University Press.
BlanchardK. (2004). Get ready to read: Test booklet (1st ed.). Pearson Education.
10.
BlanchardK. (2005). Ready to read now: Test booklet (1st ed.). Pearson Education.
11.
BolusR. E.HinofotisF. B.BaileyK. M. (1982). An introduction to generalizability theory in second language research. Language Learning, 32(2), 245–258. https://doi.org/10.1111/j.1467-1770.1982.tb00970.x
12.
BrennanR. L. (1983). Elements of generalizability theory. American College Testing Program.
13.
BrennanR. L. (2001). Generalizability theory. Springer.
14.
R. L.Brennan (2021). Generalizability theory. In ClauserB. E.BunchM. B. (Eds.), The history of educational measurement: Key advancements in theory, policy, and practice (pp. 206–231). Routledge.
15.
BrennanR. L.GaoX.ColtonD. A. (1995). Generalizability analyses of work keys listening and writing tests. Educational and Psychological Measurement, 55(2), 157–176. https://doi.org/10.1177/0013164495055002001
16.
BrownJ. D. (1984). A norm-referenced engineering reading test. In PughA. K.UlijnJ. M. (Eds.), Reading for professional purposes: Studies and practices in native and foreign languages (pp. 213–222). Heinemann Educational Books.
BrownJ. D. (1993). A comprehensive criterion-referenced testing project. In DouglasD.ChapelleC. (Eds.), A new decade of language testing research (pp. 95–125). TESOL.
19.
BrownJ. D. (1999). The relative importance of persons, items, subtests and languages to TOEFL test variance. Language Testing, 16(2), 217–238. https://doi.org/10.1177/026553229901600205
BuckG. (2001). Assessing listening. Cambridge University Press.
22.
ChapelleC. A. (2008). The TOEFL validity argument. In ChapelleC. A.EnrightM.JamiesonJ. (Eds.), Building a validity argument for the Test of English as a foreign language (pp. 319–352). Routledge.
23.
ClarkJ. L. D. (1978). Interview testing research at Educational Testing Service. In ClarkJ. L. D. (Ed.), Direct testing of speaking proficiency: Theory and application (pp. 211–228). Educational Testing Service.
24.
ClauserB. E.HarikP.MargolisM. J. (2006). A multivariate generalizability analysis of data from a performance assessment of physicians’ clinical skills. Journal of Educational Measurement, 43(3), 173–191. https://doi.org/10.1111/j.1745-3984.2006.00012.x
25.
CohenA. D.UptonT. A. (2007). I want to go back to the text: Response strategies on the reading subtest of the new TOEFL®. Language Testing, 24(2), 209–250. https://doi.org/10.1177/0265532207076364
26.
CrickJ. E.BrennanR. L. (1983). GENOVA: A general purpose analysis of variance system (Version 3.1) [Computer software]. Center for Advanced Studies in Measurement and Assessment (CASMA), University of Iowa. http://www.education.uiowa.edu/casma
27.
DaveyB. (1988). Factors affecting the difficulty of reading comprehension items for successful and unsuccessful readers. The Journal of Experimental Education, 56(2), 67–76. https://doi.org/10.1080/00220973.1988.10806468
28.
DouglasD. (2010). Understanding language testing. Routledge.
29.
DowningS. M. (2006). Selected-response item formats in test development. In DowningS. M.HaladynaT. M. (Eds.), Handbook of test development (pp. 287–301). Lawrence Erlbaum Associates.
30.
EngineerW. (1977). Proficiency in reading English as a second language (Unpublished PhD thesis). University of Edinburgh.
FrederiksenN. (1984). The real test bias: Influences of testing on teaching and learning. American Psychologist, 39(3), 193–202.
33.
GebrilA. (2009). Score generalizability of academic writing tasks: Does one test method fit it all?Language Testing, 26(4), 507–531. https://doi.org/10.1177/0265532209340188
34.
GebrilA. (2010). Bringing reading-to-write and writing-only assessment tasks together: A generalizability analysis. Assessing Writing, 15(2), 100–117. https://doi.org/10.1016/j.asw.2010.05.002
35.
GrabeW. (2009). Reading in a second language: Moving from theory to practice. Cambridge University Press.
36.
GreenA. (2020). Exploring language assessment and testing: Language in action. Routledge.
37.
GrierJ. B. (1975). The number of alternatives for optimum test reliability. Journal of Educational Measurement, 12(2), 109–112. https://www.jstor.org/stable/1434035
38.
GrierJ. B. (1976). The optimal number of alternatives at a choice point with travel time considered. Journal of Mathematical Psychology, 14(1), 91–97. https://doi.org/10.1016/0022-2496(76)90016-X
39.
HabermanS. J.SinharayS. (2010). Reporting of subscores using multidimensional item response theory. Psychometrika, 75(2), 209–227. https://doi.org/10.1007/s11336-010-9158-4
40.
HabermanS. J.SinharayS. (2013). Does subgroup membership information lead to better estimation of true subscores?British Journal of Mathematical and Statistical Psychology, 66(3), 452–469. https://doi.org/10.1111/j.2044-8317.2012.02061.x
41.
HaladynaT. M. (1997). Writing test items to measure higher level thinking. Allyn & Bacon.
42.
HaladynaT. M.DowningS. M. (1993). How many options is enough for a multiple-choice test item?Educational and Psychological Measurement, 53(4), 999–1010. https://doi.org/10.1177/0013164493053004013
43.
HaladynaT. M.RodriguezM. C. (2013). Developing and validating test items. Routledge.
HughesA. (2003). Testing for language teachers. Cambridge University Press.
46.
ImparaJ. C.FosterD. (2006). Item and test development strategies to minimize test fraud. In DowningS. M.HaladynaT. M. (Eds.), Handbook of test development (pp. 91–114). Lawrence Erlbaum Associates.
47.
KunnanA. J. (1992). An investigation of a criterion-referenced test using G-theory, and factor analysis. Language Testing, 9(1), 30–49. https://doi.org/10.1177/026553229200900104
48.
LakinJ. M.LaiE. R. (2012). Multigroup generalizability analysis of verbal, quantitative, and nonverbal ability tests for culturally and linguistically diverse students. Educational and Psychological Measurement, 72(1), 139–158. https://doi.org/10.1177/0013164411408074
49.
LaneS.RaymondM.HaladynaR. (2015). Test development process. In LaneS.RaymondM.HaladynaT. (Eds.), Handbook of test development (2nd ed., pp. 3–18). Routledge.
50.
LeeY. W.KantorR. (2005). Dependability of new ESL writing test scores: Evaluating prototype tasks and alternative rating schemes. ETS Research Report Series, 2005(1), i–76. https://doi.org/10.1002/j.2333-8504.2005.tb01991.x
51.
LiaoR. J. T. (2021). Exploring task-completion processes in L2 reading assessments: Multiple-choice vs. short-answer questions. Reading in a Foreign Language, 33(2), 168–190. http://hdl.handle.net/10125/67399
52.
LinC. K.ZhangJ. (2018). Detecting nonadditivity in single-facet generalizability theory applications: Tukey’s test. Journal of Educational Measurement, 55(1), 78–89. https://doi.org/10.1111/jedm.12164
53.
LinnR. L. (2006). The standards for educational and psychological testing: Guidance in test development. In DowningS. M.HaladynaT. M. (Eds.), Handbook of test development (pp. 27–38). Lawrence Erlbaum Associates.
54.
LiuH. (2021). Does questioning strategy facilitate second language (L2) reading comprehension? The effects of comprehension measures and insights from reader perception. Journal of Research in Reading, 44(2), 339–359. https://doi.org/10.1111/1467-9817.12339
55.
LordF. M. (1944). Reliability of multiple-choice tests as a function of the number of choices per item. Journal of Educational Psychology, 35(3), 175–180. https://doi.org/10.1037/h0061025
56.
LordF. M. (1977). Optimal number of choices per item: A comparison of four approaches. Journal of Educational Measurement, 14(1), 33–38. http://www.jstor.org/stable/1433853
57.
McNamaraT. (2000). Language testing. Oxford University Press.
58.
NewsomR. S.GaiteA. J. H. (1971). Prose learning: Effects of pretesting and reduction of passage length. Psychological Reports, 28(1), 123–129. http://dx.doi.org/10.2466/pr0.1971.28.1.123
59.
OhtaR.PlakansL. M.GebrilA. (2018). Integrated writing scores based on holistic and multi-trait scales: A generalizability analysis. Assessing Writing, 38, 21–36. https://doi.org/10.1016/j.asw.2018.08.001
60.
PlakansL. (2009). The role of reading strategies in integrated L2 writing tasks. Journal of English for Academic Purposes, 8(4), 252–266. https://doi.org/10.1016/j.jeap.2009.05.001
61.
PlakansL.GebrilA. (2015). Assessment myths: Applying second language research to classroom teaching. University of Michigan Press.
62.
RodriguezM. C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2), 3–13. https://doi.org/10.1111/j.1745-3992.2005.00006.x
63.
RuppA. A.FerneT.ChoiH. (2006). How assessing reading comprehension with multiple choice questions shapes the construct: A cognitive processing perspective. Language Testing, 23(4), 441–474. https://doi.org/10.1191/0265532206lt337oa
ShavelsonR. J.WebbN. M. (1991). Generalizability theory: A primer. SAGE.
67.
SinharayS. (2010). How often do subscores have added value? Results from operational and simulated data. Journal of Educational Measurement, 47(2), 150–174. https://doi.org/10.1111/j.1745-3984.2010.00106.x
68.
SinharayS.HabermanS.PuhanG. (2007). Subscores based on classical test theory: To report or not to report. Educational Measurement: Issues and Practice, 26(4), 21–28. https://doi/10.1111/j.1745-3992.2007.00105.x
69.
SireciS. G.ZeniskyA. L. (2006). Innovative item formats in computer-based testing: In pursuit of improved construct representation. In DowningS. M.HaladynaT. M. (Eds.), Handbook of test development (pp. 329–347). Lawrence Erlbaum Associates.
Solano–FloresG.LiM. (2006). The use of generalizability (G) theory in the testing of linguistic minorities. Educational Measurement: Issues and Practice, 25(1), 13–22. https://doi.org/10.1111/j.1745-3992.2006.00048.x
WebbN. M.SchlackmanJ.SugrueB. (2000). The dependability and interchangeability of assessment methods in science. Applied Measurement in Education, 13(3), 277–301. https://doi.org/10.1207/S15324818AME1303_4
75.
WebbN. M.ShavelsonR. J. (2008). Generalizability theory. In SalkindN. J. (Ed.), Encyclopedia of educational psychology (pp. 437–438). SAGE. https://doi.org/10.4135/9781412963848.n118
WeirC. J. (2005). Language testing and validation: An evidence-based approach. Palgrave Macmillan.
78.
ZhangS. (2006). Investigating the relative effects of persons, items, sections, and languages on TOEIC score dependability. Language Testing, 23(3), 351–369. https://doi.org/10.1191/0265532206lt332oa
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
