In practice, vertical scales have been continually used to measure students’ achievement progress across several grade levels and have been considered very challenging psychometric procedures. Recently, such practices have been drawing many criticisms. The major criticisms focus on dimensionality and construct equivalence of the latent trait or construct of a vertical scale. Using data from a large-scale reading comprehension assessment program with a vertical scale, this study investigated the dimensionality or the factorial structure for each grade and the equivalence of the factorial structure across grades. The empirical results from this study support the unidimensionality assumption of vertical scale across grades. The results provide empirical evidence regarding construct invariance of vertical scale across wide span of grades.
Ackerman, T. ( 1992). A didactic explanation of item bias, item impact, and item validity from a multidimensional perspective. Journal of Educational Measurement, 29(1), 67-91.
2.
Ackerman, T. ( 1996). Graphical representation of multidimensional item response theory. Applied Psychological Measurement, 20, 311-329.
3.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. ( 1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association .
4.
Ansley, T.M., & Forsyth, R.A. ( 1985). An examination of the characteristics of unidimensional IRT parameter estimates derived from two-dimensional data. Applied Psychological Measurement, 9, 39-48.
5.
Arbuckle, J.L., & Wothke, W. ( 1999). Amos 4.0 User’s Guide. Chicago : SmallWaters.
6.
Beauducel, A., & Wittmann, W. ( 2005). Simulation study on fit indices in confirmatory factor analysis based on data with slightly distorted simple structure. Structural Equation Modeling, 12(1), 41-75.
7.
Beguin, A.A. ( 2000). Robustness of equating high-stakes tests. Unpublished doctoral dissertation. Enschede: University of Twente.
8.
Boughton, K., Lorie, W., & Yao, L. ( 2005, April). A multidimensional multi-group IRT model for vertical scales with complex test structure: An empirical evaluation of student growth using real data. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal.
9.
Brogan, E.D., & Yen, W.M. ( 1983). Detecting multi-dimensionality and examining its effects on vertical equating with the three-parameter logistic model. Paper presented at the annual meeting of the American Educational Research Association, Montreal.
10.
Byrne, B.M. ( 2001). Structural equation modeling with AMOS. Mahwah, NJ: Lawrence Erlbaum.
11.
Bryne, B.M., Shavelson, R.J., & Muthén, B. (1989). Testing for equivalence of factor covariance and mean structures: The issues of partial measurement invariance. Psychological Bulletin, 105(3), 456-466.
12.
Camilli, G. ( 1988). Scale shrinkage and the estimation of latent distribution parameters. Journal of Educational Statistics, 40, 237-241.
13.
Camilli, G. ( 1999). Measurement error, multidimensionality, and scale shrinkage: A reply to Yen and Burket. Journal of Educational Measurement , 36, 73-78.
14.
Camilli, G., Yamamoto, K., & Wang, M.M. ( 1993). Scale shrinkage in vertical equating. Applied Psychological Measurement, 17(4), 379-388.
15.
Cheung, G.W., & Rensvold, R.B. (1999). Testing factorial invariance across groups: A reconceptualization and proposed new method. Journal of Management, 25, 1-27.
Cronbach, L. ( 1971). Test validation. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 443-507). Washington DC: American Council on Education.
18.
De Ayala, R.J. ( 1992). The influence of dimensionality on CAT ability estimation . Educational and Psychological Measurement, 52, 513-528.
19.
Deborah, J.H. ( 2007). Practical issues in vertical scaling. In N. J. Dorans, M. Pommerich , & P. W. Holland (Eds.), Linking and aligning scores and scales (pp. 233-251). New York: Springer.
20.
Doran, H.C., & Cohen, J. ( 2005). The confounding effect of linking bias on gains estimated from value-added models. In R. W. Lissitz (Ed.), Value-added models in educations: Theory and applications (pp. 80-104). Maple Grove, MN: JAM Press.
21.
Doran, H.C., & Jiang, T. ( 2006). The impact of linking error in longitudinal analysis: An empirical demonstration. In R. W. Lissitz (Ed.), Value-added models in education: Theory and applications (pp. 210-229). Maple Grove, MN: JAM Press.
22.
Drasgow, F., & Parsons, C.K. ( 1983). Application of unidimensional item response theory models to multidimensional data. Applied Psychological Measurement , 7, 189-199.
23.
Fan, X., & Sivo, S.A. ( 2005). Sensitivity of fit indices to misspecified structural or measurement model components: rationale of two-index strategy revisited . Structural Equation Modeling, 12(3), 343-367.
24.
Feldt, L.S., & Brennan, R.L. ( 1989). Reliability. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 105-146). New York: American Council on Education and Macmillan.
25.
Feuer, M.J., Holland, P.W., Green, B.F., Bertenthal, M.W., & Hemphill, F.C. (1999). Uncommon measures: Equivalence and linkage among educational tests. Washington, DC: National Academy Press.
26.
Finkelman, M., Hooker, G., Boughton, K., Yao, L. ( 2006). Estimation irregularities in compensatory MIRT models . Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco .
27.
Folk, V.G., & Green, B.F. ( 1989). Adaptive estimation when the unidimensionality assumption of IRT is violated. Applied Psychological Measurement, 13, 371-389.
28.
Haertel, E.H. ( 1991). Report on TRP analyses of issues concerning within-age versus cross-age scales for the National Assessment of Educational Progress . Washington, DC: National Center for Educational Statistics.
29.
Holmes, S.E. ( 1982). Unidimensionality and vertical equating with the Rasch model. Journal of Educational Measurement, 19, 139-147.
30.
Hu, L., & Bentler, P.M. ( 1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
31.
Huynh, H., & Schneider, C. ( 2005). Vertically moderated standards: Background, assumptions, and practices. Applied Measurement in Education, 18(1), 99-113.
32.
Ito, K., & Yao, L. ( 2005, April). Differences in ability and item parameter estimates by applying unidimensional versus multidimensional models to across-grade data. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal .
33.
Jöreskog, K.G. ( 1971). Statistical analysis of sets of congeneric tests. Psychometrika, 36, 109-133.
34.
Jöreskog, K.G., & Sörbom, D. (1979). Advances in factor analysis and structural equation models. Cambridge, MA: Abt Books.
35.
Kirisci, L., Hsu, T.C., & Yu, L. ( 2000). Robustness of selected calibration programs to violations of assumptions of unidimensionality of test items and normality of ability distribution. Applied Psychological Measurement, 25(2), 146-162.
36.
Kline, R.B. ( 2005). Principles and practices of structural equation modeling (2nd ed.). New York: Guilford.
37.
Kolen, M.J., & Brennan, R.L. ( 2004). Test equating: methods and practices (2nd ed.). New York: Springer.
38.
Linn, R.L. ( 1993). Linking results of distinct assessments. Applied Measurement in Education, 6, 83-102.
39.
Lissitz, R.W., & Huynh H. ( 2003). Vertical equating for state assessments: issues and solutions in determination of adequate yearly progress and school accountability. Practical Assessment, Research & Evaluation, 8(10). Retrieved February 16, 2006 , from http://PAREonline.net/getvn.asp?v=8n=10
40.
Loehlin, J.C. ( 1997). A test of J. R. Harris’s theory of peer influences on personality. Journal of Personality and Social Psychology , 72, 1197-1201.
41.
Loehlin, J.C. ( 2004). Latent variable models: An introduction to factor, path, and structural equation analysis. Mahwah, NJ: Lawrence Erlbaum.
42.
Lord, F.M. ( 1957). A significance test for the hypothesis that two variables measure same trait except for error of measurement. Psychometrika , 22, 20.
43.
Loyd, B.H., & Plake, B.S. ( 1987, April). Vertical equating: Effects of model, method, and content domain. Paper presented at the annual meeting of American Educational Research Association, Washington, DC.
44.
Marsh, H.W., Hau, K.-T., & Wen, Z. ( 2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings. Structural Equation Modeling, 11(3), 320-341.
45.
Martineau, J. ( 2006). Distorting value-added: The use of longitudinal, vertically scaled student achievement data for growth-based, value-added accountability . Journal of Educational and Psychological Statistics, 31, 35-62.
46.
McDonald, R.P. ( 1999). Test theory. A unified treatment. Mahwah, NJ: Lawrence Erlbaum.
47.
McDonald, R.P. ( 2008). A basis for multidimensional item response theory. Applied Psychological Measurement, 24, 99-114.
48.
McDonald, R.P., & Ho, R.M. ( 2002). Principles and practice in reporting structural equation analyses. Psychological Methods, 7, 64-82.
49.
Meredith, W. ( 1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525-543.
50.
Messick, S. ( 1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). New York: American Council on Education and Macmillan.
51.
Mislevy, R.J. ( 1992). Linking educational assessments: Concepts, issues, methods, and prospects. Princeton, NJ: ETS Policy Information Center.
52.
Nandakumar, R. ( 1991). Traditional dimensionality vs. essential dimensionality . Journal of Educational Measurement, 28, 99-117.
53.
No Child Left Behind Act of 2001, Pub. L. No. 107-110, 115 Stat. 1425 (2002).
54.
Patz, R., Yao, L., Chia, M., Lewis, D., & Hoskens, M. ( 2003, April). Hierarchical and multidimensional models for vertical scaling. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago .
55.
Raykov, T. ( 1997). Scale reliability, Cronbach’s coefficient alpha, and violations of essential tauequivalence with fixed congeneric components . Multivariate Behavioral Research, 32, 329-353.
56.
Reckase, M.D. ( 1979). Unifactor latent trait models applied to multifactor tests: Results and implications. Journal of Educational Statistics , 4, 207-230.
57.
Reckase, M.D. ( 1987, April). A comparison of the results of applying several different unidimensional IRT estimation procedures to multidimensional item response data. Paper presented at the annual meeting of the American Educational Research Association, Washington, DC.
58.
Reckase, M.D. ( 1997). A linear logistic multidimensional item response model for dichotomous item response data. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 271-286). New York: Springer-Verlag.
59.
Reckase, M.D., & Martineau, J.A. (2004, October). Growth as a Multidimensional Process. Paper presented at the Annual Meeting of the Society for Multivariate Experimental Psychology, Naples, FL.
60.
Schafer, W.D. ( 2006). Growth scales as an alternative to vertical scales. Practical Assessment Research & Evaluation, 11(4). Available from http://pareonline.net/getvn.asp?v=11&
61.
Schulz, E.M., & Nicewander, W.A. (1997). Grade equivalent and IRT representations of growth. Journal of Educational Measurement , 34, 315-331.
62.
Stanford Achievement Test Series, Tenth Edition (Technical Data Rep.). (2004). San Antonio, TX: Harcourt Assessment.
63.
Stenner, A.J. ( 1996). Measuring reading comprehension with the Lexile framework . Paper Presented at the Fourth North American Conference on Adolescent/Adult Literacy, Washington, D.C
64.
Wang, S., & Jiao, H. ( 2007, April). Evidences of dimensionality for large-scale assessment and its effect on vertical scale: Practical study with empirical and simulation approaches. Paper presented at the annual meeting of the American Educational Research Association, Chicago .
65.
Wang, S., Jiao, H., Brooks, T., & Young, M.J. ( 2004, August). Construct equivalence between customized and original Stanford Achievement Reading Comprehension Tests (10th ed.) (Research Rep.). San Antonio, TX: Harcourt Assessment.
66.
Wang, S., Jiao, H., & Severance, N. ( 2005, April). An investigation of growth patterns of student achievement using unidimensional and multidimensional vertical scale methods . Paper presented at the annual meeting of the National Council on Educational Measurement, Montreal.
67.
Way, W.D., Ansley, T.N., & Forsyth, R.A. ( 1986, April). The effects of two-dimensional data on unidimensional IRT parameter estimates. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
68.
Widaman, K.F., & Thompson, J.S. (2003). On specifying the null model for incremental fit indices in structural equation modeling. Psychological Methods, 8, 16-37
69.
Yen, W.M. ( 1985). Increasing item complexity: A possible cause of scale shrinkage for unidimensional item response theory. Psychometrika , 50, 399-410.
70.
Yen, W.M. ( 2007). Vertical scaling and No Child Left Behind. In N. J. Dorans, M. Pommerich , & P. W. Holland (Eds.), Linking and aligning scores and scales (pp. 233-251). New York: Springer.
71.
Yen, W.M., & Burket, G.R. ( 1997). Comparison of item response theory and Thurstone method of vertical scaling. Journal of Educational Measurement, 34, 293-313.
72.
Yovanoff, P., Duesbery, L., Alonzo, J., & Tindal, G. ( 2005). Grade-level invariance of a theoretical causal structure predicting reading comprehension with vocabulary and oral reading fluency . Educational Measurement: Issues and Practice, 24(3), 4-10.
73.
Yuan, K.H. ( 2005). Fit indices versus test statistics. Multivariate Behavioral Research, 40(1), 115-148.
