The C-test is a type of gap-filling test designed to efficiently measure second language proficiency. The typical C-test consists of several short paragraphs with the second half of every second word deleted. The words with deleted parts are considered as items nested within the corresponding paragraph. Given this testlet structure, it is commonly taken for granted that the C-test design may violate the local independence assumption. However, this assumption has not been fully investigated in the C-test research to date, including the evaluation of alternative psychometric models (i.e., unidimensional and multidimensional) to calibrate and score the C-test. This study addressed each of these issues using a large data set of responses to an English-language C-test. First, we examined the local item independence assumption via multidimensional item response theory (IRT) models, Yen’s Q3, and Jackknife Slope Index. Second, we evaluated several IRT models to determine optimal approaches to scoring the C-test. The results support an interpretation of unidimensionality for the C-test items within a paragraph, with only minor evidence of local item dependence. Furthermore, the two-parameter logistic (2PL) IRT model was found to be the most appropriate model for calibrating and scoring the C-test. Implications for designing, scoring, and analyzing C-tests are discussed.
BabaiiE.Fatahi-MajdM. (2014). Failed restorations in the C-test: Types, sources, and implications for C-test processing. In GrotjahnR. (Ed.), Der C-Test: Aktuelle Tendenzen/The C-Test: Current trends (pp. 263–276). Peter Lang.
2.
BaghaeiP. (2008). The effects of the rhetorical organization of texts on the C-Test construct: A Rasch modeling study. Melbourne Papers in Language Testing, 13(2), 32–51.
BaghaeiP.GrotjahnR. (2014). Establishing the construct validity of conversational C-Tests using a multidimensional item response model. Psychological Test and Assessment Modeling, 56(1), 60–82.
BaghaeiP.TabatabaeeM. (2015). The C-Test: An integrative measure of crystallized intelligence. Journal of Intelligence, 3(2), 46–58. https://doi.org/10.3390/jintelligence3020046
7.
BandalosD. L.FinneyS. J. (2001). Item parceling issues in structural equation modeling. In MarcoulidesG. A.SchumackerR. E. (Eds.), Advanced structural equation modeling: New developments and techniques (pp. 269–296). Lawrence Erlbaum Associates.
8.
BradlowE. T.WainerH.WangX. (1999). A Bayesian random effects model for testlets. Psychometrika, 64(2), 153–168. https://doi.org/10.1007/bf02294533
BraunH.von DavierM. (2017). The use of test scores from large-scale assessment surveys: Psychometric and statistical considerations. Large-Scale Assessments in Education, 5(1), 17. https://doi.org/10.1186/s40536-017-0050-x
11.
BrownJ. D.HudsonT. (1998). The alternatives in language assessment. TESOL Quarterly, 32(4), 653–675. https://doi.org/10.2307/3587999
12.
CaiL.ThissenD.du ToitS. H. C. (2011). IRTPRO for Windows [Computer software]. Scientific Software International.
CastellenoE. K.ShuZ.DonoghueJ.JiaH. (2017). Report for the investigation of the local independence assumption for scenario-based tasks in NAEP. Educational Testing Service.
15.
CaulfieldJ.SmithW. C. (1981). The reduced redundancy test and the cloze procedure as measures of global language proficiency. The Modern Language Journal, 65(1), 54–58. https://doi.org/10.2307/326207
16.
ChalmersR. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06
17.
ChiharaT.ClineW. D.SakuraiT. (1996). If the cloze test is a question, is the C-test the answer. In GrotjahnR. (Ed.), Der C-Test: Theoretische Grundlagen und praktische Anwendungen [The C-test: Theoretical foundations and practical applications] (pp. 183–195). Brockmeyer.
18.
de AyalaR. J. (2009). The theory and practice of item response theory. Guilford Press.
19.
DeMarsC. E. (2006). Application of the bi-factor multidimensional item response theory model to testlet-based tests. Journal of Educational Measurement, 43(2), 145–168. https://doi.org/10.1111/j.1745-3984.2006.00010.x
DrackertA.TimukovaA. (2019). What does the analysis of C-test gaps tell us about the construct of a C-test? A comparison of foreign and heritage language learners’ performance. Language Testing, 37(1), 107–132. https://doi.org/10.1177/0265532219861042
EckesT. (2012). Examinee-centered standard setting for large-scale assessments: The prototype group method. Psychology Science, 54(3), 257–283.
24.
EckesT.BaghaeiP. (2015). Using testlet response theory to examine local dependence in C-tests. Applied Measurement in Education, 28(2), 85–98. https://doi.org/10.1080/08957347.2014.1002919
EdwardsM. C.HoutsC. R.CaiL. (2018). A diagnostic procedure to detect departures from local independence in item response theory models. Psychological Methods, 23(1), 138–149. https://doi.org/10.1037/met0000121
27.
EmbretsonS. E.ReiseS. P. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates.
GlasC. A. W.WainerH.BradlowE. T. (2000). MML and EAP estimation in testlet-based adaptive testing. In van der LindenW. J.GlasC. A. W. (Eds.), Computerized adaptive testing: Theory and practice (pp. 271–287). Kluwer Academic.
30.
GradmanH. L.SpolskyB. (1975). Reduced redundancy testing: A progress report. In JonesR.SpolskyB. (Eds.), Testing language proficiency (pp. 59–70). Center for Applied Linguistics.
31.
GrotjahnR. (1987). How to construct and evaluate a C-Test: A discussion of some problems and some statistical analyses. In GrotjahnR.Klein-BraleyC.StevensonD. K. (Eds.), Taking their measure: The validity and validation of language tests (pp. 219–253). Brockmeyer.
32.
GrotjahnR. (1996). Einleitende Bemerkungen zu Band 3 [Introductory notes on volume 3]. In GrotjahnR. (Ed.), Der C-Test: Theoretische Grundlagen und praktische Anwendungen [The C-test: Theoretical foundations and practical applications] (Vol. 3, pp. 1–22). Brockmeyer.
33.
GrotjahnR.Klein-BraleyC.RaatzU. (1992). C-Tests in der praktischen Anwendung. Erfahrungen beim Bundeswettbewerb Fremdsprachen [C-Tests in practical application. Experiences in the national foreign language competition]. In GrotjahnR. (Ed.), Der C-Test: Theoretische Grundlagen und praktische Anwendungen [The C-test: Theoretical foundations and practical applications] (Vol. 1, pp. 263–296). Brockmeyer.
34.
GrotjahnR.StemmerB. (1985). On the development and evaluation of a C-Test for French. In Klein-BraleyC.RaatzU. (Eds.), Fremdsprachen und Hochschule, 13/14: Thematischer Teil: C-Tests in der Praxis (pp. 101–120). AKS-Verlag.
35.
HallR. J.SnellA. F.FoustM. S. (1999). Item parceling strategies in SEM: Investigating the subtle effects of unmodeled secondary constructs. Organizational Research Methods, 2(3), 233–256. https://doi.org/10.1177/109442819923002
36.
HamdiS.KartowagiranB.HaryantoH. (2018). Developing a testlet model for mathematics at elementary level. International Journal of Instruction, 11(3), 375–390. https://doi.org/10.12973/iji.2018.11326a
37.
HarschC.HartigJ. (2010). Empirische und inhaltliche Analyse lokaler Abhängigkeiten im C-Test [Empirical and substantive analysis of local dependencies in the C-Test]. In GrotjahnR. (Ed.), Der C-Test: Beiträge aus der aktuellen Forschung [The C-Test: Contributions from current research] (pp. 193–204). Peter Lang.
38.
HarschC.HartigJ. (2016). Comparing C-tests and yes/no vocabulary size tests as predictors of receptive language skills. Language Testing, 33(4), 555–575. https://doi.org/10.1177/0265532215594642
39.
HastingsA. J. (2002). Error analysis of an English C-Test: Evidence for integrated processing. In GrotjahnR. (Ed.), Der C-Test: Theoretische Grundlagen und praktische Anwendungen [The C-test: Theoretical foundations and practical applications] (Vol. 4, pp. 53–66). AKS-Verlag.
40.
Hernandez-CamachoA.OleaJ.AbadF. J. (2017). Comparison of uni-and multidimensional models applied in testlet-based tests. Methodology, 13(4), 135–143. https://doi.org/10.1027/1614-2241/a000137
41.
HoutsC. R.EdwardsM. C. (2013). The performance of local dependence measures with psychological data. Applied Psychological Measurement, 37(7), 541–562. https://doi.org/10.1177/0146621613491456
42.
JenkinsJ.LeungC. (2018). From mythical “standard” to standard reality: The need for alternatives to standardized English language tests. Language Teaching, 52(1), 86–110. https://doi.org/10.1017/s0261444818000307
43.
JinY.WuZ.AldersonC.SongW. (2017). Developing the China standards of English: Challenges at macropolitical and micropolitical levels. Language Testing in Asia, 7(1), 1. https://doi.org/10.1186/s40468-017-0032-5
44.
KaneM. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000
45.
KimD.de AyalaR. J.FerdousA. A.NeringM. L. (2011). The comparative performance of conditional independence indices. Applied Psychological Measurement, 35(6), 447–471. https://doi.org/10.1177/0146621611407909
46.
Klein-BraleyC. (1985). A cloze-up on the C-test: A study in the construct validation of authentic tests. Language Testing, 2(1), 76–104. https://doi.org/10.1177/026553228500200108
KohliN.KoranJ.HennL. (2014). Relationships among classical test theory and item response theory frameworks via factor analytic models. Educational and Psychological Measurement, 75(3), 389–405. https://doi.org/10.1177/0013164414559071
50.
KoziolN. A. (2016). Parameter recovery and classification accuracy under conditions of testlet dependency: A comparison of the traditional 2PL, testlet, and bi-factor models. Applied Measurement in Education, 29(3), 184–195. https://doi.org/10.1080/08957347.2016.1171767
51.
Lee-EllisS. (2009). The development and validation of a Korean C-Test using Rasch analysis. Language Testing, 26(2), 245–274. https://doi.org/10.1177/0265532208101007
52.
LiY.BoltD. M.FuJ. (2006). A comparison of alternative models for testlets. Applied Psychological Measurement, 30(1), 3–21. https://doi.org/10.1177/0146621605275414
53.
LordF.NovickM. (1968). Statistical theories of mental test scores. Addison-Wesley.
54.
MinS.HeL. (2014). Applying unidimensional and multidimensional item response theory models in testlet-based reading assessment. Language Testing, 31(4), 453–477. https://doi.org/10.1177/0265532214527277
55.
MozgalinaA.Ryshina-PankovaM. (2015). Meeting the challenges of curriculum construction and change: Revision and validity evaluation of a placement test. The Modern Language Journal, 99(2), 346–370. https://doi.org/10.1111/modl.12217
56.
NorrisJ. M. (2006). Development and evaluation of a curriculum-based German C-test for placement purposes. In GrotjahnR. (Ed.), Der C-Test: Theorie, Empirie, Anwendungen [The C-Test: Theory, empirical research, applications] (pp. 45–83). Peter Lang.
57.
NorrisJ. M. (2008). Validity evaluation in language assessment. Peter Lang.
58.
NorrisJ. M. (2018a). Developing and investigating C-tests in eight languages: Measuring proficiency for research purposes. In NorrisJ. M. (Ed.), Developing C-tests for estimating proficiency in foreign language research (pp. 7–33). Peter Lang.
59.
NorrisJ. M. (Ed.). (2018b). Developing C-tests for estimating proficiency in foreign language research. Peter Lang.
60.
NorrisJ. M. (2018c). Task-based language assessment: Aligning designs with intended uses and consequences. JLTA Journal, 21, 3–20. https://doi.org/10.20622/jltajournal.21.0_3
61.
NorrisJ. M.OrtegaL. (2003). Defining and measuring SLA. In DoughtyC.LongM. H. (Eds.), Handbook of second language acquisition (pp. 716–761). Blackwell.
62.
NorrisJ. M.OrtegaL. (2012). Assessing learner knowledge. In GassS. M.MackeyA. (Eds.), The Routledge handbook of second language acquisition (pp. 573–589). Routledge.
63.
OllerJ. W. (1976). Evidence for a general language proficiency factor: An expectancy grammar. Die neueren Sprachen, 75(2), 165–174.
64.
RaatzU.Klein-BraleyC. (1981). The C-Test—A modification of the cloze procedure. University of Essex Language Centre Occasional Papers, 26, 113–138.
65.
RijmenF. (2010). Formal relations and an empirical comparison among the bi-factor, the testlet, and a second-order multidimensional IRT model. Journal of Educational Measurement, 47(3), 361–372. https://doi.org/10.1111/j.1745-3984.2010.00118.x
SchroedersU.RobitzschA.SchipolowskiS. (2014). A comparison of different psychometric approaches to modeling testlet structures: An example with C-tests. Journal of Educational Measurement, 51(4), 400–418. https://doi.org/10.1111/jedm.12054
68.
SigottG. (2004). Towards identifying the C-Test construct. Peter Lang.
69.
StemlerS. E.NaplesA. (2021). Rasch measurement v. item response theory: Knowing when to cross the line. Practical Assessment, Research and Evaluation, 26, Article 11. https://doi.org/10.7275/v2gd-4441
70.
Ul Hassan MahmoodMU. H.MillerF. (2019). Discrimination with unidimensional and multidimensional item response theory models for educational data. Communications in Statistics—Simulation and Computation. Advance online publication. https://doi.org/10.1080/03610918.2019.1705344
71.
WainerH.BradlowE. T.WangX. (Ed.). (2007). Testlet response theory and its applications. Cambridge University Press.
YenW. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8(2), 125–145. https://doi.org/10.1177/014662168400800201
76.
ZanonC.HutzC. S.YooH.HambletonR. K. (2016). An application of item response theory to psychological test development. Psicologia: Reflexão e Crítica, 29, Article 18. https://doi.org/10.1186/s41155-016-0040-x
