Cognitive pretesting (CP) is an interview methodology for pretesting the validity of items during the development of self-report instruments. This article reports on the development and evaluation of a systematic method to rate self-report item validity performance utilizing CP interview text data. Five raters were trained in the application of that system, resulting in acceptable to substantial levels of interrater agreement. Results from this study suggest that excellent reliability can be achieved in the analysis of CP data. Guidelines for systematically rating the qualitative data collected using CP methods are provided. Future research should focus on empirical demonstrations of how such rating procedures can lead to improvements in the validity of self-report instruments.
Arthur, M. W., Hawkins, J. D., Pollard, J. A., Catalano, R. F., & Baglioni, A. J. (2002). Measuring risk and protective factors for substance abuse, delinquency, and other adolescent problem behaviors: The Communities That Care Youth Survey.Evaluation Review, 26, 575-601.
2.
Bowen, N. K., Bowen, G. L., & Woolley, M. E. (2004). Constructing and validating assessment tools for school-based practitioners: The Elementary School Success Profile. In A. R. Roberts & K. Y. Yeager (Eds.),Evidence-based practice manual: Research and outcome measures in health and human services(pp. 509-517). New York: Oxford University Press.
3.
Cronbach, L. J. (1990).Essentials of psychological testing (5th ed.). New York: HarperCollins.
4.
DeMaio, T. J., & Rothgeb, J. M. (1996). Cognitive interviewing techniques: In the lab and in the field. In N. Schwartz & S. Sudman (Eds.),Answering questions: Methodology for cognitive and communicative processes in survey research (pp. 177-196). San Francisco: Jossey-Bass.
5.
DeVellis, R. F. (2003).Scale development: Theory and applications. Thousand Oaks, CA: Sage.
6.
Dunn, G. (1989).The design and analysis of reliability studies. New York: John Wiley.
7.
Dyrborg, J., Warborg-Larsen, F., Nielsen, S., Byman, J., Buhl-Nielsen, B., & Gautre-Delay, F. (2000). The Children's Global Assessment Scale (CGAS) and Global Assessment of Psychosocial Disability (GAPD) in clinical practice—Substance and reliability as judged by intraclass correlations.European Child and Adolescent Psychiatry, 9, 195-201.
8.
Foddy, W. (1998). An empirical evaluation of in-depth probes used to pretest survey questions.Sociological Methods Research, 27, 103-133.
9.
Forsyth, B. H., & Lessler, J. T. (1991). Cognitive laboratory methods: A taxonomy. In P. P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowietz, & S. Sudman (Eds.),Measurement errors in surveys (pp. 393-418). New York: John Wiley.
10.
Fox, S., Bizman, A., Hoffman, M., & Oren, L. (1995). The impact of variability in candidate profiles on rater confidence and judgments regarding stability and job suitability.Journal of Occupational and Organizational Psychology, 68, 13-23.
11.
Jabine, T. B., Straf, M. L., Tanur, J. M., & Tourangeau, R. (1984).Cognitive aspects of survey methodology: Building a bridge between disciplines. Washington, DC: National Academy Press.
12.
Jobe, J. B., & Mingay, D. J. (1989).Cognitive research improves questionnaires.American Journalof Public Health, 79, 1053-1055.
13.
Kavanaugh,M. J. (1989).Performance rating accuracy improvement throughchanges in individualand system characteristics. San Antonio, TX: State University of New York at Albany, Research Foundation; Sponsored by the Air Force Human Resources Lab.
14.
McKay, R. B., & de la Puente, M. (1996).Cognitive testing of racial and ethnic questions for the CPS supplement.Monthly Labor Review, 119, 8-11.
15.
Shrout, P. E. (1998). Measurement reliability and agreement in psychiatry.Statistical Methods in Medical Research, 7, 301-317.
16.
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability.Psychological Bulletin, 86, 420-428.
17.
Sirken, M. G., Herrmann, D. J., Schecter, S., Schwartz, N., Tanur, J. M., & Tourangeau, R. (Eds.). (1999). Cognition and survey research. New York: Wiley-Interscience.
18.
SPSS. (2000). SPSS for Windows (Version 10.0.5) [Computer Software]. Chicago, IL: Author.
19.
Weigle, S. C. (1998). Using FACETS to model rater training effects.Language and Testing, 15, 263-287.
20.
Willis, G. B., DeMaio, T. J., & Harris-Kojetin, B. (1999). Is the bandwagon headed for the promised land? Evaluating the validity of cognitive interviewing techniques. In M. G. Sirken, D. J. Herrmann, S. Schecter, & N. Schwartz (Eds.),Cognition and survey research (pp. 133-154). New York: Wiley-Interscience.
21.
Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative review.Journal of Occupational and Organizational Psychology, 67, 189-205.
22.
Woolley, M. E., Bowen, G. L., & Bowen, N. K. (2004). Cognitive pretesting and the developmental validity of child self-report instruments: Theory and applications.Research on Social Work Practice, 14, 191-200.
