On the basis of a validation study of a new test for assessing low-achieving adolescents’ reading comprehension skills – the SALT-reading – we analyzed two issues relevant to the field of reading test development. Using the test results of 200 seventh graders, we examined the possibility of identifying reading comprehension subskills and the effects of task specificity on test reliability. Regarding the former, we distinguished three subskills indicating different levels of understanding (‘retrieving’, ‘interpreting’, ‘reflecting’). However, confirmatory factor analyses did not support the presence of these subskills. Task specificity refers to the situation that different tasks within a test are not uniformly difficult for individual test takers, which constitutes a form of error negatively influencing test reliability. However, Generalizability Theory analysis showed that such task-specific effects did not occur: the reliability of the SALT-reading was primarily affected by error associated with the score variance within tasks.
BrennanR. L. (2001). Generalizability theory. New York: Springer.
5.
CardinetJ.JohnsonS.PiniG. (2010). Applying Generalizability Theory using EduG. New York/London: Routledge.
6.
CerdánR.Vidal-AbarcaE.MartínezT.GilabertR.GilL. (2009). Impact of question-answering tasks on search processes and reading comprehension. Learning and Instruction, 19(1), 13–27.
7.
DaasR.HavermansR.Van NoortwijkJ. (2009). Een nieuwe test voor het meten van leesbegripsvaardigheden bij leerlingen op het VMBO: Resultaten van een valideringsonderzoek gericht op de inhoudsvaliditeit. [A new test for measuring reading comprehension skills in students in prevocational secondary education: Results of a validition study focused on content validity]. Unpublished manuscript, University of Amsterdam.
8.
DavisF. B. (1968). Research in comprehension in reading. Reading Research Quarterly, 3(4), 499–545.
9.
De Knecht-Van EekelenA.GilleE.Van RijnP. (2007). Resultaten PISA-2006: Praktische kennis en vaardigheden van 15-jarigen [Results PISA-2006: Practical knowledge and skills of 15-year-olds]. Arnhem: Cito.
10.
DunnG.EverittB.PicklesA. (1993). Modelling covariances and latent variables using EQS. London: Chapman & Hall.
11.
GagnonR.CharlinB.LambertC.CarrièreB.Van der VleutenC. (2008). Script concordance testing: More cases or more questions?Advances in Health Sciences Education, 14(3), 367–375.
12.
GoldmanS. R.DuránR. P. (1988). Answering questions from oceanography texts: Learner, task, and text characteristics. Discourse Processes, 11(4), 373–412.
13.
GraesserA. C.FranklinS. P. (1992). QUEST: A model of question answering. Discourse Processes, 13, 279–303.
14.
HuL.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55.
15.
IRDP (2010). EduG.User guide. Neuchatel: IRDP.
16.
KalifaH.WeirC. J. (2009). Examining reading: Research and practice in assessing second language reading. New York: Cambridge University Press.
17.
KintschW. (1998). Comprehension: A paradigm for cognition. New York: Cambridge University Press.
18.
LeeY. W. (2006). Dependability of scores for a new ESL speaking assessment consisting of integrated and independent tasks. Language Testing, 23(2), 131–166.
19.
LeeG.FrisbieD. A. (1999). Estimating reliability under a generalizability theory model for test scores composed of testlets. Applied Measurement in Education, 12(3), 237–255.
20.
LinnR. L. (1993). Assessment. Expanded expectations and challenges. Educational Evaluation and Policy Analysis, 15(1), 1–16.
21.
LinnR. L.BakerE. L.DunbarS. B. (1991). Complex, performance-based assessment: Expectations and validity criteria. Educational Researcher, 20(8), 15–21.
22.
LittleT. D.CunninghamW. A.ShaharG. (2002). To parcel or not to parcel: Exploring the question, weighing the merits. Structural Equation Modeling, 9(2), 151–173.
23.
MeijerJ.Van GelderenA. (2002). Lezen voor het leven: Een empirische vergelijking van een nationale en een internationale leesvaardigheidspeiling [Reading for life: An empirical comparison of a national and an international reading assessment]. Amsterdam: SCO-Kohnstamminstituut.
24.
MessickS. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.
25.
MosenthalP. B. (1996). Understanding the strategies of document literacy and their conditions of use. Journal of Educational Psychology, 88(2), 314–332.
26.
NormanG.BordageG.PageG.KeaneD. (2006). How specific is case specificity?Medical Education, 40(7), 618–623.
27.
NunnallyJ. C.BernsteinI. H. (1994). Psychometric theory (3rdedn). New York: McGraw-Hill.
28.
OECD (2003). The PISA 2003 assessment framework: Mathematics, reading, science and problem solving knowledge and skills. Paris: OECD.
29.
PearsonP. D.HammD. N. (2005). The assessment of reading comprehension: A review of practices – past, present, and future. In ParisS. G.StahlS. A. (Eds.),Children’s reading comprehension and assessment (pp. 13–70). Mahwah, NJ: Lawrence Erlbaum.
30.
RosenshineB. (1980). Skills hierarchies in reading comprehension. In SpiroR. J.BruceB. C.BrewerW. F. (Eds.), Theoretical issues in reading comprehension: Perspectives from cognitive psychology, linguistics, artificial intelligence, and education (pp. 535–554). Hillsdale, NJ: Lawrence Erlbaum.
31.
RostD. H. (1993). Assessing different components of reading comprehension: Fact or fiction?Language Testing, 10(1), 79–92.
32.
RouetJ.-F.Vidal-AbarcaE.ErboulA. B.MillogoV. (2001). Effects of information search tasks on the comprehension of instructional text. Discourse Processes, 31(2), 163–186.
33.
RuppA. A.FerneT.ChoiH. (2006). How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing, 23(4), 441–474.
34.
SawakiY. (2007). Construct validation of analytic rating scales in a speaking assessment: Reporting a score profile and a composite. Language Testing, 24(3), 355–390.
35.
SchedlM.GordonA.CareyP. A.TangK. L. (1996). An analysis of the dimensionality of TOEFL reading comprehension items. Princeton, NJ: Educational Testing Service.
36.
SchoonenR. (2005). Generalizability of writing scores: An application of structural equation modeling. Language Testing, 22(1), 1–30.
37.
ShavelsonR. J.WebbN. M. (1991). Generalizability theory: A primer. Newbury Park, CA: SAGE Publications.
38.
SongM. Y. (2008). Do divisible subskills exist in second language (L2) comprehension? A structural equation modeling approach. Language Testing, 25(4), 435–464.
39.
SpearittD. (1972). Identification of subskills of reading comprehension by maximum likelihood factor analysis. Reading Research Quarterly, 8(1), 92–111.
40.
ThorndikeR. L. (1973–1974). Reading as reasoning. Reading Research Quarterly, 9(1), 135–147.
41.
UllmanJ. B. (2001). Structural Equation Modeling. In TabachnickB. G.FidellL. S. (Eds.), Using multivariate statistics (4th edn., pp. 653–771). Needham Heights, MA: Allyn & Bacon.
42.
Vidal-AbarcaE.GilabertR.RouetJ.-F. (1998). The role of question type on learning from scientific text. Paper presented at Seminario ‘Comprension y produccion de textos cientificos’, Aveiro, Portugal.
43.
WeirC. J.PorterD. (1994). The multi-divisible or unitary nature of reading: The language tester between Scylla and Charybdis. Reading in a Foreign Language, 10, 1–19.
44.
XiX. (2007). Evaluating analytic scoring for the TOEFL® Academic Speaking Test (TAST) for operational use. Language Testing, 24(2), 251–286.
45.
YuanK. H.LambertP. L.FouladiR. T. (2004). Mardia’s multivariate kurtosis with missing data. Multivariate Behavioral Research, 39(3), 413–437.
