The Effects of Small Sample Size on Identifying Polytomous DIF Using the Liu–Agresti Estimator of the Cumulative Common Odds Ratio

Abstract

This study is an evaluation of the behavior of the Liu—Agresti estimator of the cumulative common odds ratio when identifying differential item functioning (DIF) with polytomously scored test items using small samples. The Liu—Agresti estimator has been proposed by Penfield and Algina as a promising approach for the study of polytomous DIF but no simulation study focusing on small samples has analyzed this estimator regarding effect size, Type I error, and power rates. The article begins with a description of this estimator in the context of polytomous DIF. Then it presents the methods and results for a simulation study in which three factors are manipulated: between-group difference in ability distribution, form of DIF introduced into the polytomous item, and sample size. The results of this study indicate that for samples smaller than 200, very little power was observed for statistical significance testing; however, Type I error rates were close to nominal levels, and the recovery of the log odds ratio as an effect size was relatively unaffected by sample size for high discriminating items. Implications for practice are discussed.

Keywords

polytomous DIF Liu—Agresti estimator small samples

Get full access to this article

View all access options for this article.

References

Carvajal, J. , & Poggio, A. ( 2006, April). Studying equivalence of Spanish language versions of a large scale assessment: Differential item functioning in the cognitive and affective domain. Paper presented at the meeting of the National Council on Measurement in Education, San Francisco, CA.

Carvajal, J. , & Skorupski, W.P. (2009). LIUA [Computer software] . Lawrence: University of Kansas .

Fidalgo, A.M. , Ferreres, D. , & Muiz, J. ( 2004). Utility of the Mantel-Haenszel procedure for detecting differential item functioning in small samples. Educational and Psychological Measurement, 64, 925-936.

Han, K.T. ( 2007). WinGen2 [Computer software]. Amherst : Center for Educational Assessment, University of Massachusetts at Amherst.

Liu, I.-M. , & Agresti, A. ( 1996). Mantel-Haenszel-type inference for cumulative odds ratios with a stratified ordinal response. Biometrics, 52, 1223-1234.

Parshall, C.G. , & Miller, T.R. ( 1995). Exact versus asymptotic Mantel-Haenszel DIF statistics: A comparison of the performance under small-sample conditions. Journal of Educational Measurement, 32, 302-316.

Penfield, R.D. ( 2005). Differential item functioning analysis system. Applied Psychological Measurement, 29, 150-151.

Penfield, R.D. ( 2007). An approach for categorizing DIF in polytomous items . Applied Measurement in Education, 20, 335-355.

Penfield, R.D. , & Algina, J. ( 2003). Applying the Liu-Agresti estimator of the cumulative common odds ratio to DIF detection in polytomous items. Journal of Educational Measurement, 40, 343-370.