To address whether or not modifications in test administration influence item functioning for students with disabilities on a high-stakes statewide problem-solving assessment, a sample of 868 students (with and without disabilities) from 74 Georgia schools were randomly assigned to one of three testing conditions (resource guide, calculator, or standard administration). The authors describe a conceptual framework for investigating measurement invariance, specifically differential item functioning within the context of assessing students with disabilities. Specifically, we illustrate how the hierarchical generalized linear model, an explanatory model that incorporates item response models into hierarchical models in multilevel settings, can be used to explore issues of measurement invariance traditionally addressed using descriptive item response models, that is, the many-facet Rasch model. Results obtained from the two approaches are reported and discussed.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. ( 1999). Standards for educational and psychological testing. Washington, DC: AERA.
2.
Bennett, R., Rock, D., & Novatkoski, I. (1989). Differential item functioning on the SAT-M Braille edition. Journal of Educational Measurement , 26, 67-79.
3.
Bielinski, J., Thurlow, M., Ysseldyke, J., Freidebach, J., & Freidebach, M. (2001). Read-aloud accommodation: Effects on multiple-choice reading and math items (Technical Report 31). Minneapolis: University of Minnesota, National Center on Educational Outcomes.
4.
Bolt, S., & Thurlow, M. ( 2006). Item level effects of the read aloud accommodation for students with disabilities (Synthesis Report 65). Minneapolis : University of Minnesota, National Center on Educational Outcomes.
5.
Bolt, S., & Ysseldyke, J. ( 2006). Comparing DIF across math and reading/language arts tests for students receiving a read-aloud accommodation. Applied Measurement in Education, 19, 329-355.
6.
Cheong, Y.F. ( 2006). Analysis of school context effects on differential item functioning using hierarchical generalized linear models. International Journal of Testing, 6, 57-79.
7.
Cheong, Y.F., & Raudenbush, S.W. (2000). Measurement and structural models for children’s problem behaviors. Psychological Methods, 5, 477-495.
8.
Cohen, A., Gregg, N., & Deng, M. ( 2005). The role of extended time and item content on a high-stakes mathematics test. Learning Disabilities Research and Practice , 20, 225-233.
9.
Draba, R.E. ( 1977). The identification and interpretation of item bias ( Research Memorandum No. 25). Chicago: Statistical Laboratory , Department of Education, University of Chicago.
10.
Elbaum, B. ( 2007). Effects of an oral testing accommodation on the mathematics performance of secondary students with and without learning disabilities . Journal of Special Education, 40, 218-229.
11.
Elbaum, B., Arguelles, M., Campbell, Y., & Saleh, M. ( 2004). Effects of a student-reads-aloud accommodation on the performance of students with and without learning disabilities on a test of reading comprehension. Exceptionality, 12(2), 71-87.
12.
Engelhard, G. ( 2009). Using Item Response Theory and model data fit to conceptualize differential item and person functioning for students with disabilities. Educational and Psychological Measurement , 69, 585-602.
13.
Engelhard, G., Fincher, M., & Domaleski, C.S. (2006). Examining the reading and mathematics performance of students with disabilities under modified conditions: The Georgia Department of Education modification research study. Atlanta: Georgia Department of Education .
14.
Fuchs, L. ( 2000, July). The validity of test accommodations for students with disabilities: Differential item performance on mathematics tests as a function of test accommodations and disability status (Final Report) . Washington, DC: U.S. Department of Education through the Delaware Department of Education.
15.
Huesman, R., & Frisbie, D. ( 2000, April). The validity of ITBS reading comprehension test scores for learning disabled and non learning disabled students under extended time conditions. Paper presented at the meeting of National Council on Measurement in Education, New Orleans, LA.
16.
Huynh, H., & Barton, K. ( 2006). Performance of students with disabilities under regular and oral administrations of a high-stakes reading examination. Applied Measurement in Education, 19, 21-39.
17.
Huynh, H., Meyer, J., & Gallant, D. ( 2004). Comparability of student performance between regular and oral administrations for a high stakes mathematics test. Applied Measurement in Education, 17, 39-57.
18.
IES/What Works ClearingHouse. (2009). WWC Evidence Review Protocol for Middle School Math Interventions, Version 1.1. Washington, DC : U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, What Works ClearingHouse.
19.
Kamata, A. ( 2001). Item analysis by the hierarchical generalized linear model. Journal of Educational Measurement, 38, 79-93.
Linacre, J.M. ( 2007). A user’s guide to FACETS: Rasch-model computer programs. Retrieved from www.winsteps.com
22.
Linn, R.L., & Harnisch, D.L. (1981). Interactions between item content and group membership on achievement test items. Journal of Educational Measurement, 18, 109-118.
23.
Little, R.J., & Rubin, D.B. ( 2002). Statistical analysis with missing data (2nd ed.). Hoboken, NJ: Wiley.
24.
Meloy, L., Deville, C., & Frisbie, D. ( 2000, April). The effect of a reading accommodation on standardized test scores of learning disabled students. Paper presented at the annual meeting of the American Educational Research Association , New Orleans, LA.
25.
Munger, G., & Loyd, B. ( 1991). Effect of speededness on test performance of handicapped and nonhandicapped examinees. Journal of Educational Research , 85, 53-57.
26.
Randall, J., & Engelhard, G. ( 2010). Using confirmatory factor analysis and Rasch Measurement Theory to assess measurement invariance in a high stakes reading assessment . Applied Measurement in Education, 23, 286-306.
27.
Randall, J., Engelhard, G., Fincher, M., & Domaleski, C. ( 2010). Performance of students with and without disabilities under modified conditions: Using resource guides and read aloud test modifications on a high stakes reading test. Journal of Special Education , 44, 79-93.
28.
Raudenbush, S.W., Bryk, A., Cheong, Y.F., Congdon, R., & Du Toit, M. ( 2004). HLM6: Hierarchical linear and nonlinear modeling . Chicago, IL: Scientific Software .
29.
Raudenbush, S.W., & Bryk, A.S. ( 2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.).Thousand Oaks, CA: Sage.
30.
Raudenbush, S.W., Johnson, C., & Sampson, R.J. ( 2003). A multivariate, multilevel Rasch model for self-reported criminal behavior. Sociological Methodology, 33, 169-211.
31.
Raudenbush, S.W., & Sampson, R.J. ( 1999). Ecometrics: Toward a science of assessing ecological settings, with application to the systematic social observation of neighborhoods . Sociological Methodology, 29, 1-41.
32.
Rock, D., Bennett, R., & Kaplan, B. ( 1985). The internal construct validity of the SAT across handicapped and nonhandicapped populations (ETS Research Report RR-85-50) . Prince-ton, NJ: Educational Testing Service.
33.
Runyan, M. ( 1991). The effect of extra time on reading comprehension scores for university students with and without learning disabilities. Journal of Learning Disabilities, 24, 104-108.
34.
U.S. Department of Education. (2007). Title I: Improving the academic achievement of the disadvantaged; Individuals with Disabilities Act (IDEA); Final Rule. Fed. Reg., Vol. 72, No. 67, Monday, April 9, 2007.
