The purpose of this study was to apply two types of Differential Item Functioning (DIF), net and global DIF, as well as the framework of Differential Step Functioning (DSF) to real testing data to investigate measurement invariance related to test language. Data from the Program for International Student Assessment (PISA)–2006 polytomously scored science items for four countries with different test languages were used, where French and English represented the reference languages. The findings showed that many items exhibited both types of DIF, although, in most cases, the results were inconsistent for the two source languages. In addition, net and global DIF tests did not always yield the same results depending on the DSF effect pattern. Furthermore, the DSF analysis provided valuable information over and above that provided by the net DIF analysis concerning the nature and the location of the DIF effect.
ArimR. G.ErcikanK. (2005, April). Comparability between the US and Turkish versions of the Third International Mathematics and Science study’s (TIMSS) mathematics test results. Paper presented at the meeting of the National Council on Measurement in Education, Montreal, Quebec, Canada.
2.
ChangH. H.MazzeoJ.RoussosL. (1996). Detecting DIF for polytomously scored items: An adaptation of the SIBTEST procedure. Journal of Educational Measurement, 33, 333-353. doi:10.1111/j.1745-3984.1996.tb00496.x
3.
CohenA. S.KimS. H.BakerF. B. (1993). Detection of differential item functioning in the graded response model. Applied Psychological Measurement, 17, 335-350. doi:10.1177/014662169301700402
4.
DoransN. J.SchmittA. P. (1993). Constructed response and differential item functioning: A pragmatic approach. In BennettR. E.WardW. C. (Eds.), Construction versus choice in cognitive measurement: Issues in constructed response, performance testing, and portfolio assessment (pp. 135-165). Hillsdale, NJ: Lawrence Erlbaum.
5.
ErcikanK. (1999, April). Translation DIF on TIMSS. Paper presented at the meeting of the National Council on Measurement in Education, Montreal, Quebec, Canada.
6.
FrenchA. W.MillerT. R. (1996). Logistic regression and its use in detecting differential item functioning in polytomous items. Journal of Educational Measurement, 33, 315-332. doi:10.1111/j.1745-3984.1996.tb00495.x
7.
GattamortaK. A.PenfieldR. D. (2012). A comparison of adjacent categories and cumulative differential step functioning effect estimators. Applied Measurement in Education, 25, 142-161. doi:10.1080/08957347.2012.660387
8.
GrisayA. (2003). Translation procedures in OECD/PISA 2000 international assessment. Language Testing, 20, 225-240. doi:10.1191/0265532203lt254oa
9.
KimS. H.CohenA. S. (1998). Detection of differential item functioning under the graded response model with the likelihood ratio test. Applied Psychological Measurement, 22, 345-355. doi:10.1177/014662169802200403
10.
LeL. T. (2006, April). Analysis of differential item functioning. Paper presented at the annual meeting of American Educational Research Association, San Francisco, CA.
11.
LeL. T. (2009). Investigating gender differential item functioning across countries and test languages for PISA science items. International Journal of Testing, 9, 122-133. doi:10.1080/15305050902880769
12.
MagisD.FaconB. (2013). Item purification does not always improve DIF detection: A counterexample with Angoff’s delta plot. Educational and Psychological Measurement, 73, 293-311. doi:10.1177/0013164412451903
13.
MantelN. (1963). Chi-square tests with one degree of freedom: Extensions of the Mantel-Haenszel procedure. Journal of the American Statistical Association, 58, 690-700. doi:10.1080/01621459.1963.10500879
PenfieldR. D. (2007). Assessing differential step functioning in polytomous items using a common odds ratio estimator. Journal of Educational Measurement, 44, 187-210. doi:10.1111/j.1745-3984.2007.00034.x
17.
PenfieldR. D. (2010). Distinguishing between net and global DIF in polytomous items. Journal of Educational Measurement, 47, 129-149. doi:10.1111/j.1745-3984.2010.00105.x
18.
PenfieldR. D.AlginaJ. (2003). Applying the Liu–Agresti estimator of the cumulative common odds ratio to DIF detection in polytomous items. Journal of Educational Measurement, 40, 353-370. doi:10.1111/j.1745-3984.2003.tb01151.x
19.
PenfieldR. D.AlvarezK.LeeO. (2009). Using a taxonomy of differential step functioning to improve the interpretation of DIF in polytomous items: An illustration. Applied Measurement in Education, 22, 61-78. doi:10.1080/08957340802558367
20.
PenfieldR. D.CamilliG. (2007). Differential item functioning and item bias. In SinharayS.RaoC. R. (Eds.), Handbook of statistics, Psychometrics (Vol. 26, pp. 125-167). New York: Elsevier.
21.
PenfieldR. D.GattamortaK.ChildsR. A. (2009). An NCME instructional module on using differential step functioning to refine the analysis of DIF in polytomous items. Educational Measurement: Issues and Practice, 28, 38-49. doi:10.1111/j.1745-3992.2009.01135.x
22.
PotenzaM. T.DoransN. J. (1995). DIF assessment for polytomously scored items: A framework for classification and evaluation. Applied Psychological Measurement, 19, 23-37. doi:10.1177/014662-169501900104
23.
SireciS. G.BerberogluG. (2000). Using bilingual respondents to evaluate translated–adapted items. Applied Measurement in Education, 13, 229-248. doi:10.1207/S15324818AME1303_1
24.
SomesG. W. (1986). The generalized Mantel–Haenszel Statistic. The American Statistician, 40, 106-108. doi:10.1080/00031305.1986.10475369
25.
vandeVijverF.TanzerN. K. (2004). Bias and equivalence in cross-cultural assessment: An overview. Revue Européenne de Psychologie Appliquée/European Review of Applied Psychology, 54, 119-135. doi:10.1016/j.erap.2003.12.004
26.
YildirimH. H.BerberoĝluG. (2009). Judgmental and statistical DIF analyses of the PISA-2003 mathematics literacy items. International Journal of Testing, 9, 108-121. doi:10.1080/153050509-02880736
