A meta-analysis was conducted on the effects of multiple-choice and open-ended formats on L1 reading, L2 reading, and L2 listening test performance. Fifty-six data sources located in an extensive search of the literature were the basis for the estimates of the mean effect sizes of test format effects. The results using the mixed effects model of meta-analysis indicate that multiple-choice formats are easier than open-ended formats in L1 reading and L2 listening, with the degree of format effect ranging from small to large in L1 reading and medium to large in L2 listening. Overall, format effects in L2 reading are not found, although multiple-choice formats are found to be easier than open-ended formats when any one of the following four conditions is met: the studies involve between-subjects designs, random assignment, stem-equivalent items, or learners with a high L2 proficiency level. Format effects favoring multiple-choice formats across the three domains are consistently observed when studies employ between-subjects designs, random assignment, or stem-equivalent items.
Alderson, J.C., Clapham, C., & Wall, D. (1995). Language test construction and evaluation. Cambridge, UK: Cambridge University Press.
3.
Anastasi, A., & Urbina, S. (1995). Psychological testing (7th ed.). Upper Saddle River, NJ: Prentice Hall.
4.
Arthur, W. Jr., Edwards, B.D., & Barrett, G.V. (2002). Multiple-choice and constructed response tests of ability: Race-based subgroup performance differences on alternative paper-and-pencil test formats. Personnel Psychology, 55, 985-1008.
5.
Bachman, L.F. (1990). Fundamental considerations in language testing . Oxford, UK: Oxford University Press.
6.
Bachman, L.F., & Palmer, A.S. (1996). Language testing in practice. Oxford, UK: Oxford University Press.
7.
Bennett, R. E., & Ward, W. C. (Eds.). (1993). Construction versus choice in cognitive measurement: Issues in constructed response, performance testing, and portfolio assessment. Hillsdale, NJ: Erlbaum.
8.
Berne, J.E. (1992). The effects of text type, assessment task, and target language experience on foreign language learners'performance on listening comprehension tests. (UMI No. 9236396)
9.
Blok, H. (1999). Reading to young children in educational settings: A meta-analysis of recent research. Language Learning, 49, 343-371.
10.
Borenstein, M., Hedges, L., Higgins, J., & Rothstein, H. (2005). Comprehensive meta-analysis (Version 2.2.023) [Computer software]. Englewood Cliffs, NJ: Biostat.
11.
Brantmeier, C. (2005). Effects of reader's knowledge, text type, and test type on L1 and L2 reading comprehension in Spanish. Modern Language Journal, 89, 37-53.
Brown, H.D. (2006). Principles of language learning and teaching (5th ed.). White Plains, NY: Pearson.
14.
Brown, J.D. (1993). What are the characteristics of natural cloze tests? Language Testing, 10, 93-116.
15.
Brown, J.D. (2005). Testing in language programs: A comprehensive guide to English language assessment. New York: McGraw Hill.
16.
Buck, G. (2001). Assessing listening. Cambridge , UK: Cambridge University Press.
17.
Campbell, J.R. (1999). Cognitive processes elicited by multiple-choice and constructed-response questions on an assessment of reading comprehension . (UMI No. 9938651)
18.
Clapham C., & Corson, D. (Eds.). (1997). Encyclopedia of language and education, Volume 7: Language testing and assessment. Dordrecht, Netherlands: Kluwer.
19.
Cohen, A.D. (1994). Assessing language ability in the classroom (2nd ed.). Boston, MA: Heinle & Heinle.
20.
Cohen, A.D. (1998). Strategies in learning and using a second language . Harlow, Essex, UK: Longman.
21.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed). Hillsdale, NJ: Erlbaum.
22.
Cook, T.D., Cooper, H., Cordray, D.S., Hartmann, H., Hedges, L.V., Light, R.J. et al. (1992). Meta-analysis for explanation: A casebook . New York: Russell Sage Foundation .
23.
Cooper, H., & Hedges, L. V. (Eds.). (1994). The handbook of research synthesis. New York: Russell Sage Foundation.
24.
Cronbach, L.J. (1990). Essentials of psychological testing (5th ed.). New York: HarperCollins.
25.
Davey, B. (1987). Postpassage questions: Task and reader effects on comprehension and metacomprehension processes. Journal of Reading Behavior, 19, 261-283.
26.
Davey, B., & LaSasso, C. (1984). The interaction of reader and task factors in the assessment of reading comprehension. Journal of Experimental Education, 52, 199-206.
27.
Davies, A., Brown, A., Elder, C., Hill, K., Lumley, T., & McNamara, T. (1999). Dictionary of language testing. Cambridge, UK: Cambridge University Press.
28.
Davies, A., & Elder, C. (Eds.). (2004). The handbook of applied linguistics. Malden, MA: Blackwell .
29.
Doughty, C. J., & Long, M. H. (Eds.). (2003). The handbook of second language acquisition. Malden, MA: Blackwell.
30.
Downing, S. M., & Haladyna, T. M. (Eds.). (2006). Handbook of test development . Mahwah, NJ: Erlbaum.
31.
Dunlap, W.P. (1999). A program to compute McGraw and Wong's common language effect size indicator. Behavior Research Methods, Instruments, & Computers, 31, 706-709.
32.
Elinor, S.-H. (1997, May). Reading native and foreign language texts and tests: The case of Arabic and Hebrew native speakers reading L1 and English FL texts and tests. Paper presented at the Language Testing Symposium, Ramat-Gan, Israel. (ERIC Document Reproduction Service No. ED 412746)
33.
Ellis, R. (1994). The study of second language acquisition. Oxford, UK: Oxford University Press.
34.
Flowerdew, J., & Miller, L. (2005). Second language listening: Theory and practice . New York: Cambridge University Press.
35.
Fulcher, G., & Davidson, F. (2007). Language testing and assessment: An advanced resource book. New York: Routledge.
36.
Glass, G.V., McGaw, B., & Smith, M.L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
37.
Grabe, W., & Stoller, F.L. (2002). Teaching and researching reading. Harlow, UK: Pearson.
38.
Grotjahn, R. (Ed.). (2006). The C-test: Theory, empirical research, applications. Frankfurt am Main, Germany: Peter Lang.
39.
Haladyna, T.M. (2004). Developing and validating multiple-choice test items (3rd ed.). Mahwah, NJ: Erlbaum.
40.
Hinkel, E. (Ed.). (2005). Handbook of research in second language teaching and learning. Mahwah, NJ: Erlbaum.
41.
Hunter, J.E., & Schmidt, F.L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.
42.
ILTA Language Testing Bibliography. (1999). Retrieved January 2, 2006, from http://www.iltaonline.com/ILTA_pubs.htm
43.
In'nami, Y. (2006). The effects of task types on listening test performance: A quantitative and qualitative study. Unpublished doctoral dissertation, University of Tsukuba, Japan.
44.
Kaplan, R. B. (Ed.). (2002). The Oxford handbook of applied linguistics. New York: Oxford University Press.
45.
Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge, UK: Cambridge University Press.
46.
Kline, R.B. (2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: American Psychological Association.
47.
Kobayashi, K. (2004). Dokkai test kaito hoho ga jukensha no test tokuten ni ataeru eikyo: Chugokugo bogo washa no baai [The effects of test methods on reading test scores of Chinese students learning Japanese as a foreign language]. Unpublished master's thesis, Ochanomizu University , Japan.
48.
Kobayashi, M. (2002). Method effects on reading comprehension test performance: Text organization and response format. Language Testing, 19, 193-220.
49.
Light, R., & Pillemer, D. (1984). Summing up: The science of reviewing research . Cambridge, MA: Harvard University Press.
Maxwell, S.E., & Delaney, H.D. (1990). Designing experiments and analyzing data: A model comparison perspective. Belmont, CA: Wadsworth.
52.
Morris, S.B., & DeShon, R.P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-group designs. Psychological Methods, 7, 105-125.
53.
Norris, J.M., & Ortega, L. (2000). Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language Learning, 50, 417-528.
54.
Norris, J. M., & Ortega, L. (Eds.). (2006). Synthesizing research on language learning and teaching. Amsterdam: John Benjamins.
55.
Pigott, T.D. (1994). Methods for handling missing data in research synthesis. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 163-175). New York: Russell Sage Foundation.
56.
Pressley, M., Ghatala, E.S., Woloshyn, V., & Pirie, J. (1990). Sometimes adults miss the main ideas and do not realize it: Confidence in responses to short-answer and multiple-choice comprehension questions. Reading Research Quarterly, 25, 232-249.
57.
Raudenbush, S.W. (1994). Random effects models. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 301-321). New York: Russell Sage Foundation.
58.
Richards, J.C., & Schmidt, R. (2002). Longman dictionary of language teaching & applied linguistics (3rd ed.). Harlow, Essex, UK: Longman.
59.
Rodriguez, M.C. (2003). Construct equivalence of multiple-choice and constructed-response items: A random effects synthesis of correlations. Journal of Educational Measurement, 40, 163-184.
60.
Rodriguez, M.C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2), 3-13.
61.
Ross, S. (1998). Self-assessment in second language testing: A meta-analysis and analysis of experiential factors. Language Testing, 15, 1-20.
62.
Rost, M. (2002). Teaching and researching listening. Harlow, UK: Pearson Education.
63.
Shadish, W.R., Cook, T.D., & Campbell. D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
64.
Shadish, W.R., & Haddock, C.K. (1994). Combining estimates of effect size. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 261-281). New York: Russell Sage Foundation.
65.
Shohamy, E. (1984). Does the testing method make a difference? The case of reading comprehension. Language Testing, 1, 147-170.
66.
Teng, H.-C. (1999, March). The effects of question type and preview on EFL listening assessment. Paper presented at the American Association for Applied Linguistics. (ERIC Document Reproduction Service No. ED 432920)
67.
Traub, R.E. (1993). On the equivalence of the traits assessed by multiple-choice and constructed-response tests. In R. E. Bennett & W. C. Ward (Eds.), Construction versus choice in cognitive measurement: Issues in constructed response, performance testing, and portfolio assessment (pp. 29-44). Hillsdale, NJ: Erlbaum.
68.
Trujillo, J.L. (2006). The effect of format and language on the observed scores of secondary-English speakers. (UMI No. 3198256)
69.
Urquhart, A.H., & Weir, C. (1998). Reading in a second language: Process, product and practice. London: Longman.
70.
Weir, C.J. (2005). Language testing and validation: An evidence-based approach. Basingstoke, Hampshire, UK: Palgrave Macmillan.
71.
Wilson, D.B. (2001). METAREG.SPS [SPSS macro]. Retrieved February 13, 2005, from http://mason.gmu.edu/~dwilsonb/downloads/spss_macros.zip
72.
Wolf, D.F. (1991). The effects of task, language of assessment, and target language experience on foreign language learners performance on reading comprehension tests. (UMI No. 9124507)
