Criterion-validity-based assessment of four scale constructs

Abstract

Background: The aim of this study was to discuss, in general, selected crucial aspects of the appropriate methodology for the development and validation of scales indicating health status, and to illustrate this empirically by within-material comparison of the accuracy of four different scale constructs based on identical raw data. Methods: The empirical example was based on information from the parents of 99 refugee children, aged 3-15 years, from the Middle East, who participated in a structured interview on their children's mental health. Following this, they were exposed to a blinded semi-structured psychological interview. Four anxiety scales were constructed based on answers to 12 anxiety symptom questions in the structured interview: Scale 1, by cumulation of original item scores, each ranging from 0 to 3; Scale 2, by counting the number of symptoms being present; Scale 3, by counting the number of frequent or intense anxiety symptoms; Scale 4, by estimation of the multivariate probability of the child being anxious, as assessed by the psychological interview. The scales were compared for their accuracy in the identification of children assessed as anxious by psychological interview. Results: The four scales correlated mutually, and each of them was significantly associated with anxiety, as assessed by psychological interview. The weighted scale, however, performed significantly better than the unweighted scales for sensitivity but not for specificity. In the present data set the overall amount of misclassification was, however, significantly less than in the unweighted scales. Conclusion: As expected from theory, the weighted scale was found to be superior to the unweighted scales, in identifying the anxious children of the empirical example. In the presence of a blinded criterion measurement, empirical regression-based weighting of scale items thus constitutes an accessible and valid alternative to traditional methods of health and social scaling.

Keywords

anxiety assessment children symptom scales validity.

Get full access to this article

View all access options for this article.

References

Bech P. Rating scales for psychopathology, health status and quality of life Berlin: Springer-Verlag, 1993.

Nunnally JC , Bernstein IH Psychometric theory. New York: McGraw-Hill Inc, 1994.

Staquet MJ , Hays RD , Fayers PM , editors. Quality of life assessment in clinical trials. Methods and practice. Oxford: Oxford University Press, 1998.

Rust J. , Golombok S. Modern psychometrics: the science of psychological assessment London: Routledge, 1989 . 1.

Fayers PM , Machin D , Staquet MJ , Hays RD , Fayers PM , editors. Quality of life assessment in clinical trials. Methods and practice. Factor analysis. Oxford: Oxford University Press, 1998.

Edelbrock C. , Costello AJ Convergence between statistically derived behavior Problem syndromes and child Psychiatric diagnoses. J Abnorm Child Psychol. 1987; 16: 219-31.

Jensen PS , Salzberg AD , Richters JE , Watanabe HK Scales, diagnoses, and child psychopathology: I. CBCL and DISC relationships . J Am Acad Child Adolesc Psychiatry. 1993 ; 32: 397-406.

Verhulst FC , Akkerhuis GW , Althaus M. Mental health in Dutch children: I. A cross-cultural comparison. Acta Psychiatr Scand. 1985; 72: 1-108.

Achenbach TM , Edelbrock CS Behavioral problems and competencies reported by parents of normal and disturbed children aged four through sixteen. Chicago : Monogr Soc Res Child Dev, 1981 .

10.

Rasch G. An item analysis which takes individual differences into account. Br J Math Stat Psychol. 1966; 19: 49-57.

11.

Ipsen J. Appropriate scores in bio-assays using death-times and survivor symptoms . Biometrics. 1955; 11: 465 - 80.

12.

Breslow NE , Day NE Statistical methods in cancer research. Volume 1 - the analysis of case-control studies. Lyon: International Agency for Research on Cancer, 1980.

13.

Altman DG Practical statistics for medical research. London : Chapman & Hall, 1991.

14.

Conners CK Symptom patterns in hyperkinetic, neurotic, and normal children. Child Dev. 1970; 41: 667-82.

15.

Costello EJ , Edelbrock CS , Costello AJ Validity of the NIMH diagnostic interview schedule for children: a comparison between psychiatric and pediatric referrals . J Abnorm Child Psychol. 1985; 13: 579-95.

16.

Brunshaw JM , Szatmari P. The agreement between behaviour checklists and structured psychiatric interviews for children. Can J Psychiat. 1988; 33: 474-81.

17.

Bird HR , Gould MS , Rubio-Stipec M. , Staghezza BM , Canino G. Screening for childhood psychopathology in the community using the child behavior checklist. J Am Acad Child Adolesc Psychiatry . 1991; 30: 116-23.

18.

Montgomery E.

Refugee children from the Middle East

Scand J Soc Med (1998); (Suppl). ; 54: 1-152.