Despite evidence that the choice of dependent measures can significantly influence design sensitivity, many evaluators default to traditional measures that may be insensitive to intervention effects. This paper describes an innovative set of test development guidelines designed to select items and create aggregate scales that are better able to detect program effects. The application of these Intervention Item Selection Rules (IISRs) is illustrated during the initial development of an outcome measure, completed by teachers, for elementary age children receiving psychosocial services from community mental health agencies. The major scale formed with these change-sensitive items displayed a larger effect size and an adequate reliability estimate.
Get full access to this article
View all access options for this article.
References
1.
Botcheva, L., White, C. R., & Huffman, L. C. (2002). Learning culture and outcomes measurement practices in community agencies. American Journal of Evaluation, 23, 421-434.
2.
Brown, R. E., & Reed, C. S. (2002). An integral approach to evaluating outcome evaluation training. American Journal of Evaluation, 23, 1-17.
3.
Carver, R. P. (1974). Two dimensions of tests: Psychometric and edumetric. American Psychologist, 29, 512-518.
4.
Collins, L. M. (1991). Measurement in longitudinal research. In L. M. Collins & J. L. Horn (Eds.), Best methods for the analysis of change (pp. 137-148). Washington, DC: American Psychological Association.
5.
Constantine, M. G., & Ladany, N. (2000). Self-report multicultural counseling competence scales: Their relation to social desirability attitudes and multicultural case conceptualization ability. Journal of Counseling Psychology, 47, 155-164.
6.
Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation. Chicago, IL: Rand McNally.
7.
Dadds, M. R., Perrin, S., & Yule, W. (1998). Social desirability and self-reported anxiety in children: An analysis of the RCMAS Lie Scale. Journal of Abnormal Child Psychology, 26, 311-317.
8.
Danziger, K. (1990). Constructing the subject. Cambridge: Cambridge University Press.
9.
Dawis, R. V. (1992). The individual differences tradition in counseling psychology. Journal of Counseling Psychology, 39, 7-19.
10.
Dillman, D. A. (1978). Mail and telephone surveys: The total design method. New York: Wiley.
11.
Epstein, S. (1979). The stability of behavior: I. On predicting most of the people much of the time. Journal of Personality and Social Psychology, 37, 1097-1126.
12.
Epstein, S. (1980). The stability of behavior. II. Implications for psychological research. American Psychologist, 35, 790-806.
13.
Gibbs, D., Napp, D., Jolly, D., Westover, B., & Uhl, G. (2002). Increasing evaluation capacity within community-based HIV prevention programs. Evaluation and Program Planning, 25, 216-269.
14.
Guyatt, G., Walter, S., & Norman, G. (1987). Measuring change over time: Assessing the usefulness of evaluative instruments. Journal of Chronic Disease, 40, 171-178.
15.
Heppner, P. P., Kivlighan, D. M., & Wampold, B. E. (1999). Research design in counseling (2nd ed.). Pacific Grove, CA: Brooks/Cole.
16.
Kazdin, A. E. (2000). Psychotherapy for children and adolescents: Directions for research and practice. New York: Oxford Press.
17.
Lambert, M. J. (1994). Use of psychological tests for outcome assessment. In M. E. Maruish (Ed.), The use of psychological testing for treatment planning and outcome assessment (pp. 75-97). Hillsdale, NJ: Lawrence Erlbaum.
18.
Lambert, M. J., Hatch, D. R., Kingston, M. D., & Edwards, B. C. (1986). Zung, Beck, and Hamilton Rating Scales as measures of treatment outcome: A meta-analytic comparison. Journal of Consulting and Clinical Psychology, 54, 54-59.
19.
Lewis, J. D., & Magoon, T. M. (1987). Survey of college counseling centers’ followup practices with former clients. Professional Psychology: Research and Practice, 18(2), 128-133.
20.
Lipsey, M. (1983). A scheme for assessing measurement sensitivity in program evaluation and other applied research. Psychological Bulletin, 94, 152-165.
21.
Lipsey, M. (1990). Design sensitivity. Newbury Park, CA: Sage.
22.
Lipsey, M. (1998). Design sensitivity: Statistical power for applied experimental research. In L. Bickman & D. J. Rog (Eds.), Handbook of applied social research methods (pp. 39-68). Thousand Oaks, CA: Sage.
23.
Meier, S. (1994). The chronic crisis in psychological measurement and assessment. New York: Academic Press.
24.
Meier, S. (1997). Nomothetic item selection rules for tests of psychological interventions. Psychotherapy Research, 7, 419-427.
25.
Meier, S. (1998). Evaluating change-based item selection rules. Measurement and Evaluation in Counseling and Development, 31, 15-27.
26.
Meier, S. (2000). Treatment sensitivity of the PE Form of the Social Skills Rating Scales: Implications for test construction procedures. Measurement and Evaluation in Counseling and Development, 33, 144-156.
27.
Meier, S., & Letsch, E. (2000). Data collection issues in an urban community mental health center: What is necessary and sufficient information for outcome assessment?Professional Psychology: Research and Practice, 31, 409-411.
28.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). Washington, DC: American Council on Education and National Council on Measurement in Education.
29.
Posavac, E. J., & Carey, R. G. (2003). Program evaluation: Methods and case studies (6th ed.). Upper Saddle River, NJ: Prentice Hall.
30.
Rounds, J. B., & Tinsley, H. E. (1984). Diagnosis and treatment of vocational problems. In S. D. Brown & R. W. Lent (Eds.), Handbook of counseling psychology (pp. 137-177). New York: Wiley.
31.
Stiffman, A. R., Orme, J. G., Evans, D. A., Feldman, R. A., & Keeney, P. A. (1984). A brief measure of children’s behavior problems: The Behavior Rating Index for Children. Measurement and Evaluation in Counseling and Development, 16, 83-90.
32.
Thayer, C. E., & Fine, A. H. (2001). Evaluation and outcome measurement in the non-profit sector: Stakeholder participation. Evaluation & Program Planning, 24, 103-108.
33.
Tobler, N. S. (1992). Drug prevention programs can work: Research findings. Journal of Addictive Diseases, 11, 1-28.
34.
Tobler, N. S. (1986). Meta-analysis of 143 adolescent drug prevention programs: Quantitative outcome results of program participants compared to a control or comparison group. Journal of Drug Issues, 16, 537-567.
35.
Tryon, W. W. (1991). Activity measurement in psychology and medicine. New York: Plenum.
36.
Vermeersch, D. A., Lambert, M. J., & Burlingame, G. M. (2000). Outcome Questionnaire: Item sensitivity to change. Journal of Personality Assessment, 74, 242-261.
37.
Vermeersch, D. A., Whipple, J. L., Lambert, M. J., Hawkins, E. J., Burchfield, C. M., & Okiishi, J. C. (2004). Outcome Questionnaire: Is it sensitive to changes in counseling center clients?Journal of Counseling Psychology, 51, 38-49.
38.
Weinstock, M., & Meier, S. T. (2003). A comparison of two item selection methodologies for measuring change in university counseling center clients. Measurement and Evaluation in Counseling & Development, 36, 66-75.
39.
Weisz, J. R., Huey, S. J., & Weersing, V. R. (1998). Psychotherapy outcome research with children and adolescents: The state of the art. In T. H. Ollendick & R. J. Prinz (Eds.), Advances in Clinical Child Psychology (Vol. 20, pp. 49-91). New York: Plenum.
40.
Weisz, J. R., Weiss, B., & Donenberg, G. R. (1992). The lab versus the clinic: Effects of child and adolescent psychotherapy. American Psychologist, 47, 1578-2585.
41.
Yeaton, W. H., & Sechrest, L. (1981). Meaningful measures of effect. Journal of Consulting & Clinical Psychology, 49, 766-767.
42.
Yeaton, W. H., & Sechrest, L. (1986). Use and misuse of no-difference findings in eliminating threats to validity. Evaluation Review, 10, 836-852.
