Differential item functioning (DIF) can undermine the validity of cross-lingual comparisons. While a lot of efficient statistics for detecting DIF are available, few general findings have been found to explain DIF results. The objective of the article was to study DIF sources by using a mixed method design. The design involves a quantitative phase in which DIF was analyzed followed by a qualitative phase conducting cognitive interviews. To illustrate the proposal, polytomous DIF was analyzed in the scales from the PISA (Programme for International Student Assessment) Student Questionnaire (Organisation for Economic Co-operation and Development). Evidence obtained allowed DIF to be connected with differences in the interpretation patterns of participants from the different linguistic groups. Finally, benefits of mixed methods design for analyzing equivalence in cross-lingual assessments are discussed.
AllaloufA.HambletonR. K.SireciS. (1999). Identifying the causes of DIF in translated verbal items. Journal of Educational Measurement, 36, 185-198.
2.
BeattyP.WillisG. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71, 287-311.
3.
BrymanA. (1988). Quantity and quality in social research. London, England: Unwin Hyman.
4.
Castillo-DíazM.PadillaJ-L. (2012). How cognitive interviewing can provide validity evidence of the response processes to scale items. Social Indicator Research. Advance online publication. doi:10.1007/s11205-012-0184-8
5.
ClarkeP. N.YarosP. S. (1988). Research blenders: Commentary and response. Nursing Science Quarterly, 1, 147-149.
6.
CreswellJ. W. (1995). Research design: Qualitative and quantitative approaches. Thousand Oaks. CA: Sage.
7.
EdmeadesJ.NybladeL.MalhotraA.MacQuarrieK.ParasuramanS.WaliaS. (2010). Methodological innovation in studying abortion in developing countries: A narrative quantitative survey in Madhya Pradesh, India. Journal of Mixed Methods Research, 4(3), 176-198.
8.
ElosuaP.López-JaúreguiA. (2007). Potential sources of differential item functioning in the adaptation of tests. International Journal of Testing, 7, 39-52.
9.
ErcikanK.ArimR.LawD.DomeneJ.GagnonF.LacroixS. (2010). Application of think aloud protocols for examining and confirming sources of differential item functioning identified by expert reviews. Educational Measurement: Issues and Practice, 29(2), 24-35.
10.
FerneT.RuppA. A. (2007). A synthesis of 15 years of research on DIF in language testing: Methodological advances, challenges, and recommendations. Language Assessment Quarterly, 4, 113-148.
11.
GadermannA. M.GuhnM.ZumboB. D. (2011). Investigating the substantive aspect of construct validity for the satisfaction with life scale adapted for children: A focus on cognitive processes. Social Indicator Research, 100, 37-60.
12.
GierlM. J.KhaliqS. N. (2001). Identifying sources of differential item and bundle functioning on translated achievement tests: A confirmatory analysis. Journal of Educational Measurement, 38, 164-187.
13.
HambletonR. K. (2006). Good practices for identifying differential item functioning. Medical Care, 44, S182-S188.
14.
HarknessJ. A. (2003). Questionnaire translation. In HarknessJ. A.van de VijverF.MohlerP. Ph (Eds.), Cross-cultural survey methods (pp. 19-34). New York, NY: Wiley.
15.
HidalgoM. D.Gómez-BenitoJ. (2010). Education measurement: Differential item functioning. In PetersonP.BakerE.McGawB. (Eds.), International encyclopedia of education (3rd ed., pp. 36-44). New York, NY: Elsevier Science.
16.
International Association for the Evaluation of Educational Achievement. (2011). Progress in International Reading Literacy Study (PIRLS). Alexandria, VA: National Center for Education Statistics.
17.
International Test Commission. (2010). International test commission guidelines for translating and adapting tests.Retrieved from http://www.intestcom.org
18.
JohnsonT. P. (2006). Methods and frameworks for crosscultural measurement. Medical Care, 44(11 Suppl. 3), S17-S20.
19.
LinnR. L.HarnischD. L. (1981). Interactions between item content and group measurement on achievement test items. Journal of Educational Measurement, 18, 109-118.
20.
MillerK. (2007, June). Design and analysis of cognitive interviews for cross-national testing. Paper presented at the European Survey Research Association annual meeting, Prague, Czech Republic.
21.
MillerT. R.SprayJ. A. (1993). Logistic discriminant function analysis for DIF identification of polytomously scored items. Journal of Educational Measurement, 30, 107-122.
22.
MillsapR. E.EversonH. T. (1993). Methodology review: Statistical approaches for assessing measurement bias. Applied Psychological Measurement, 17, 297-334.
Organisation for Economic Co-operation and Development. (2009). Programme for International Student Assessment (PISA). Paris, France: Author.
26.
Organisation for Economic Co-operation and Development. (2011). Programme for the International Assessment of Adult Competencies (PIAAC). Paris, France: Author.
PenfieldR. D. (2010). Distinguishing between net and global DIF in polytomous items. Journal of Educational Measurement, 47, 129-149.
29.
PenfieldR. D.AlvarezK.LeeO. (2009). Using a taxonomy of differential step functioning to improve the interpretation of dif in polytomous items: An illustration. Applied Measurement in Education, 22, 61-78.
30.
PenfieldR. D.GattamortaK.ChildsR. A. (2009). An NCME instructional module on using differential step functioning to refine the analysis of DIF in polytomous items. Educational Measurement: Issues and Practice, 28, 38-49.
31.
PresserS.CouperM. P.LesslerJ. T.MartinE.MartinJ.RothgebJ. M.SingerE. (2004). Methods for testing and evaluating survey questions. Public Opinion Quarterly, 68, 109-130.
32.
RoussosL. A.StoutW. F. (1996). A multidimensionality-based DIF analysis paradigm. Applied Psychological Measurement, 20, 355-371.
33.
SchmeiserC. B. (1982). Use of experimental design in statistical item bias studies. In BerkR. A. (Ed.), Handbook of methods for detecting test bias (pp. 64-96). Baltimore, MD: Johns Hopkins University Press.
34.
ShealyR.StoutW. F. (1993). An item response theory model for test bias. In HollandP. W.WainerH. (Eds.), Differential item functioning (pp. 197-239). Hillsdale, NJ: Lawrence Erlbaum.
35.
SireciS. G. (1997). Problems and issues in linking tests across languages. Educational Measurement: Issues and Practice, 16, 12-19.
36.
SireciS. G.PatsulaL.HambletonR. K. (2005). Statistical methods for identifying flaws in the test adaptation process. In HambletonR. K.MerendaP. F.SpielbergerC. D. (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 93-115). Hillsdale, NJ: Lawrence Erlbaum.
37.
SPSS Inc. (2007). SPSS-16 user’s guide. Chicago, IL: Author.
38.
SwansonD. B.ClauserB. E.CaseS. M.NungesterR. J.FeathermanC. (2002). Analysis of differential item functioning (DIF) using hierarchical logistic regression models. Journal of Educational and Behavioral Statistics, 27, 53-75.
vandeVijverF. J. R.PoortingaY. H. (2005). Conceptual and methodological issues in adapting tests. In HambletonR. K.MerendaP. F.SpielbergerC. D. (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 39-63). Hillsdale, NJ: Lawrence Erlbaum.
41.
WillisG. B. (2005). Cognitive interviewing. Thousand Oaks: Sage.
42.
ZiekyM. (1993). Practical questions in the use of DIF statistics in test development. In HollandP. W.WainerH. (Eds.), Differential item functioning (pp. 337-347). Hillsdale, NJ: Lawrence Erlbaum.
43.
ZumboB. D. (1999). A handbook on the theory and methods of differential item functioning (DIF): Logistic regression modeling as a unitary framework for binary and Likert-type (ordinal) item scores. Ottawa, Ontario, Canada: Directorate of Human Resources Research and Evaluation, Department of National Defense.
44.
ZumboB. D. (2009). Validity as contextualized and pragmatic explanation, and Its implications for validation practice. In LissitzR. W. (Ed.), The concept of validity: Revisions, new directions and applications (pp. 65-82). Charlotte, NC: Information Age.
