Evaluating construct validity of the second version of the Copenhagen Psychosocial Questionnaire through analysis of differential item functioning and differential item effect

Abstract

Aims: To evaluate the construct validity of the Copenhagen Psychosocial Questionnaire II (COPSOQ II) by means of tests for differential item functioning (DIF) and differential item effect (DIE). Methods: We used a Danish general population postal survey (n = 4,732 with 3,517 wage earners) with a one-year register based follow up for long-term sickness absence. DIF was evaluated against age, gender, education, social class, public/private sector employment, and job type using ordinal logistic regression. DIE was evaluated against job satisfaction and self-rated health (using ordinal logistic regression), against depressive symptoms, burnout, and stress (using multiple linear regression), and against long-term sick leave (using a proportional hazards model). We used a cross-validation approach to counter the risk of significant results due to multiple testing. Results: Out of 1,052 tests, we found 599 significant instances of DIF/DIE, 69 of which showed both practical and statistical significance across two independent samples. Most DIF occurred for job type (in 20 cases), while we found little DIF for age, gender, education, social class and sector. DIE seemed to pertain to particular items, which showed DIE in the same direction for several outcome variables. Discussion: The results allowed a preliminary identification of items that have a positive impact on construct validity and items that have negative impact on construct validity. These results can be used to develop better shortform measures and to improve the conceptual framework, items and scales of the COPSOQ II. Conclusions: We conclude that tests of DIF and DIE are useful for evaluating construct validity.

Keywords

Differentiel item functioning differentiel item effect psychosocial work environment questionnaire construct validity

Get full access to this article

View all access options for this article.

References

Cronbach LJ

Meehl PE

Construct validity in psychological testing . Psychol Bull1955;52:281-302.

Rugulies R.

Aust B.

Pejtersen JH

Do psychosocial work environment factors measured with scales from the COPSOQ predict register-based sickness absence of three weeks or more in Denmark ? Scand J Public Health2010; 38(Suppl 3):42-50.

Pejtersen JH

Kristensen TS

Borg V.

, Bjorner JB. The second version of the Copenhagen Psychosocial Questionnaire (COPSOQ II). Scand J Public Health2010;38(Suppl 3): 8-24.

Bollen KA

Multiple indicators: internal consistency or no necessary relationship?

Qual Quant1984;18:377-85.

Bollen KA

Lennox R.

Conventional wisdom on measurement: a structural equation perspective . Psychol Bull1991; 110(2):305-14.

Cronbach LJ

Essentials of psychological testing, 5th edn. New York: Harper & Row; 1990.

Messick S.

Validity. In:

Linn RL

, editor. Educational measurement 3rd, edn. New York: Macmillan; 1989. pp. 13-103.

Carver CS

How should multifaceted personality constructs be tested? Issues illustrated by self-monitoring, attributional style, and hardiness. J Pers Soc Psychol1989;56(4):577-85.

Kreiner S.

Validation of index scales for analysis of survey data - the Symptom Index . In:

Dean K

, editor. Population Health research - linking theory and methods. Beverly Hills: Sage; 1993. pp. 116-44.

10.

Kreiner S.

Validity and objectivity: reflections on the role and nature of Rasch models. Nordic Psychol2007;59(3):268-98.

11.

Kristensen TS

Bjorner JB

Christensen KB

Borg V.

The distinction between work pace and working hours in the measurement of quantitative demands at work. Work & Stress2004;18(4):305-22.

12.

Choi B.

Kawakami N.

Chang S.

Koh S.

Bjorner JB

Punnett L.

, et al. A cross-national study on the multidimensional characteristics of the five-item psychological demands scale of the Job Content Questionnaire . Int J Behav Med2008;15(2):120-32.

13.

Rose D.

Pevalin DJ

Elias P.

Martin J.

Towards a European socio-economic classification: final report to Eurostat of the Expert Group. London and Colchester: ONS and ISER, University of Essex; 2001.

14.

Moncada S.

Pejtersen JH

Navarro A.

Llorens C.

Burr H.

Hasle P.

, et al. Psychosocial work environment and its association with socioeconomic status. A comparison of Spain and Denmark2010;38(Suppl 3):42-50.

15.

The Directorate of Labor.

The Danish job classification [in Danish]. Copenhagen: The Directorate of Labor; 1986.

16.

Ortega A.

Hogh A.

Pejtersen JH

Olsen O.

Prevalence of workplace bullying and risk groups: a representative population study. Int Arch Occup Environ Health2009;82(3):417-26.

17.

Groenvold M.

Petersen MA

The role and use of differential item functioning analysis in clinical trials . In:

Fayers PM

Hay RD

, editors. Assessing quality of life in clinical trials: methods and practice, 2nd edn. Oxford: Oxford University Press; 2005. pp. 195-208.

18.

Teresi JA

Fleishman JA

Differential item functioning and health assessment . Qual Life Res2007;16(Suppl 1):33-42.

19.

Zumbo BD

A handbook on the theory and methods of differential item functioning (DIF): logistic regression modeling as a unitary framework for binary and likert-type (ordinal) item scores. Ottawa, ON: Directorate of Human Resources Research and Evaluation, Department of National Defense; 1999.

20.

Bjorner JB

Kosinski M.

Ware Jr, JE

Calibration of an item pool for assessing the burden of headaches: an application of item response theory to the headache impact test (HIT). Qual Life Res2003;12(8):913-33.

21.

Nagelkerke NJD

A note on a general definition of the coefficient of determination. Biometrika1991;78:691-2.

22.

Kristensen TS

Borg V.

Hannerz H.

Socioeconomic status and psychosocial work environment: results from a Danish national study. Scand J Public Health2002;(Suppl 59):41-8.

23.

Choi B.

Bjorner JB

Ostergren PO

Clays E.

Houtman I.

Punett L.

, et al. Cross-language differential item functioning of the Job Content Questionnaire among European countries: the JACE study. Int J Behavioral Med, 16(2):136-147, 2009.

24.

Orhede E.

Kreiner S.

Item bias in indices measuring psychosocial work environment and health. Scand J Work Environ Health2000;26(3):263-72.

25.

Thorsen SV

Bjorner JB

Reliability of the Copenhagen Psychosocial Questionnaire. Scand J Public Health2010; 38(Suppl 3):25-32.

26.

Brenninkmeijer V.

VanYperen N.

How to conduct research on burnout: advantages and disadvantages of a unidimensional approach in burnout research . Occup Environ Med2003;60(Suppl 1):i16-i20.

27.

Cordes CL

Dougherty TW

Blum M.

Patterns of burnout among managers and professionals: a comparison of models. J Organ Beh1997;18(6):685-701.

28.

Fay D.

Frese M.

The concept of personal initiative: an overview of validity studies. Human Performance2001 ;14(1):97-124.

29.

Goldberger AS

Multicollinearity. A course in econometrics . Cambridge, MA: Harvard University Press; 2009. pp. 245-53.

30.

Jobe JB

Cognitive psychology and self-reports: models and methods. Qual Life Res2003;12(3):219-27.

31.

Maldonado G.

Greenland S.

Simulation study of confounderselection strategies . Am J Epidemiol1993;138(11):923-36.