Randomized field experiments can provide trustworthy evidence about the effects of interventions. But, investigators have limited control over important features of the field experiment (e.g., program implementation, receipt of intended and unintended services, retention of participants in the study). Unchecked, these factors can limit the technical adequacy and utility of the study. In the absence of direct control over these factors, efforts should be directed at developing valid indicators so their influence can be taken into account using advanced analytical techniques. Coupled with logical and statistical analyses, studies that cast a wider measurement net should provide more useful explanations for the pattern of observed effects.
Get full access to this article
View all access options for this article.
References
1.
Aiken, L. S.
, West, S. G., Schwalm, D. E., Carroll, J. L., & Hsiung, S. (1998). Comparison of a randomized and two quasi-experimental designs in a single outcome evaluation: Efficacy of a university-level remedial writing program. Evaluation Reviews, 22, 207-244.
2.
Babcock, J. C.
, & Steiner, R. (1999). The relationship between treatment, incarceration, and recidivism of battering: A program evaluation of Seattle's coordinated community response to domestic violence. Journal of Family Psychology, 13, 45-59.
3.
Boring, E. G.
(1954). The nature and history of experimental control. American Journal of Psychology, 67, 573-589.
4.
Boruch, R. F.
(1997). Randomized experiments for planning and evaluation: A practical guide. Thousand Oaks, CA: Sage.
5.
Brekke, J. S.
(1987). The model-guided method for monitoring implementation. Evaluation Reviews, 11, 281-299.
6.
Brekke, J. S.
, & Test, M. A. (1992). A model for measuring implementation of community support programs: Results from three sites. Community Mental Health Journal, 28, 227-247.
7.
Brown, R. S.
, Bergeron, J. W., Clement, D. G., Hill, J. W., & Retchin, S. M. (1993). The Medicare risk program for HMS—Final summary report on findings from the evaluation. Princeton, NJ: Mathematica Policy Research.
8.
Campbell, D. T.
, & McCormick, T. H. (1957). Military experience and attitudes toward authority. American Journal of Sociology, 62, 482-490.
9.
Campbell, D. T.
, & Stanley, J. C. (1966). Experimental and quasi-experimental designs for research. Boston: Houghton Mifflin.
10.
Carroll, K. M.
, Connors, G. J., Cooney, N. L., DiClemente, C. C., Donovan, D. M., Kadden, R. R., Longabaugh, R. L., Rounsaville, B. J., Wirtz, P. W., & Zweben, A. (1998). Internal validity of project MATCH treatments: Discriminability and integrity. Journal of Consulting and Clinical Psychology, 66, 290-303.
11.
Chalk, R.
, & King, P. (Eds.). (1998). Violence in families: Assessing prevention and treatment programs. Washington, DC: National Academy Press.
12.
Cook, T. D.
, & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis for field settings. Boston: Houghton Mifflin.
13.
Cook, T. D.
, Campbell, D. T., & Peracchio, L. (1990). Quasi-experimentation. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 1, pp. 491-576). Palo Alto, CA: Consulting Psychologists Press.
14.
Cordray, D. S.
(1986). Quasi-experimental analysis: A mixture of methods and judgment. New Directions for Program Evaluation, 31, 9-27.
15.
Cordray, D. S.
, & Pion, G. M. (1993). Psychosocial rehabilitation assessment: A broader perspective. In R. Glueckauf, G. Bond, L. Sechrest, & B. McDonel (Eds.), Improving assessment in rehabilitation and health (pp. 215-240). Newbury Park, CA: Sage.
16.
Cordray, D. S.
, & Pion, G. M. (1998). A multi-site evaluation of NIAAA's Homeless R&D Program. Nashville, TN: Vanderbilt Institute for Public Policy Studies (VIPPS) Working Paper.
17.
CSR Inc.
(1997). Process evaluation of the Comprehensive Child Development Program. Washington, DC: CSR Inc.
18.
Dennis, M. L.
(1990). Assessing the validity of randomized field experiments: An example from drug abuse treatment research. Evaluation Reviews, 14, 347-373.
19.
Einhorn, H. J.
, & Hogarth, R. M. (1986). Judging probable cause. Psychological Bulletin, 99(1), 3-19.
20.
Gueron, J.
(1997). Learning about welfare reform: Lessons from state-based evaluations. New Directions for Evaluation, 76, 79-94.
21.
Jekel, J. F.
, & Lam, J. A. (1994). Research on services for homeless substance abusers. Unpublished report.
22.
Julian, D. A.
(1997). The utilization of the logic model as a system level planning and evaluation device. Evaluation and Program Planning, 20, 251-257.
23.
Kazdin, A. E.
(1986). Comparative outcome studies of psychotherapy: Methodological issues and strategies. Journal of Consulting and Clinical Psychology, 54, 95-105.
24.
Kazdin, A. E.
, & Mazurick, J. L. (1994). Dropping out of child psychotherapy: Distinguishing early and late dropouts over the course of treatment. Journal of Consulting and Clinical Psychology, 62, 1069-1074.
25.
Kutchinsky, B.
(1973). The effects of easy availability of pornography on the incidence of sex crimes: The Danish experience. Journal of Social Issues, 29, 163-181.
26.
Lipsey, M. W.
(1997). What can you build with thousands of bricks? Musings on the cumulation of knowledge in program evaluation. New Directions for Evaluation, 76, 7-24.
27.
Lipsey, M. W.
, & Cordray, D. S. (2000). Evaluation methods for intervention assessment. Annual Review of Psychology, 51, 345-375.
28.
Lipsey, M. W.
, Cordray, D. S., & Berger, D. E. (1981). Evaluation of a juvenile diversion program: Using multiple lines of evidence. Evaluation Reviews, 5, 283-306.
29.
Longabaugh, R.
, Wirtz, P. W., Beattie, M. C., Noel, N., & Stout, R. (1995). Matching treatment focus to patient social investment and support: 18-month follow-up results. Journal of Consulting and Clinical Psychology, 63, 296-307.
30.
Maude-Griffin, P. M.
, Hohenstein, J. M., Humfleet, G. L., Reilly, P. M., Tusel, D. J., & Hall, S. M. (1998). Superior efficacy of cognitive-behavioral therapy for urban crack cocaine abusers: Main and matching effects. Journal of Consulting and Clinical Psychology, 66, 832-837.
31.
McGrew, J. H.
, Bond, G. R., Dietzen, L., & Salyers, M. (1994). Measuring the fidelity of implementation of a mental health program model. Journal of Consulting and Clinical Psychology, 62, 670-678.
32.
Muthen, B. O.
, & Curran, P. J. (1997). General longitudinal modeling of individual differences in experimental designs: A latent variable framework for analysis and power estimation. Psychological Methods, 2, 371-402.
33.
Newhouse, J. P.
, & McClellan, M. (1998). Econometrics in outcomes research: The use of instrumental variables. Annual Review of Public Health, 19, 17-34.
34.
Norman, J.
, Vlahov, D., & Moses, L. E. (Eds.). (1995). Preventing HIV transmission: The role of sterile needles and bleach. Washington, DC: National Academy Press.
35.
Orwin, R. G.
, Sonnefeld, L. J., Cordray, D. S., Pion, G. M., & Perl, H. I. (1998). Constructing quantitative implementation scales from categorical services data: Examples from a multisite evaluation. Evaluation Reviews, 22, 245-288.
36.
Osgood, D. W.
, & Smith, G. L. (1995). Applying hierarchical linear modeling to extended longitudinal evaluations: The Boys Town follow-up study. Evaluation Reviews, 19, 3-38.
37.
Platt, J.
(1964). Strong inference. Science, 146, 347-353.
38.
Prochaska, J. O.
, DiClemente, C. C., & Norcross, J. C. (1992). In search of how people change: Applications to addictive behaviors. American Psychologist, 47, 1102-1114.
39.
Reichardt, C. S., & Rallis, S. E. (Eds.). (1994). The qualitative-quantitative debate: New perspectives [Entire issue]. New Directions for Program Evaluation, 61.
40.
Reynolds, A. J.
, & Temple, J. A. (1998). Extended early childhood intervention and school achievement: Age thirteen findings from the Chicago Longitudinal Study. Child Development, 69, 231-246.
41.
Rog, D. J.
, & Huebner, R. B. (1991). Using research and theory in developing innovative programs for homeless families. In H-T. Chen & P. H. Rossi (Eds.), Using theory to improve program and policy evaluations (pp. 129-144). New York: Greenwood.
42.
Rosenbaum, P. R.
(1995). Observational studies. New York: Springer-Verlag.
43.
Scheirer, M. A.
(1996). A template for assessing the organizational base for program implementation. New Directions for Evaluation, 72, 61-80.
44.
Shortell, S.
, & Kaluzny, A. (1988). Health care management: A text in organization theory and behavior (2nd ed.). New York: Wiley.
45.
Smith, B.
, & Sechrest, L. (1991). Treatment of aptitude X treatment interactions. Journal of Consulting and Clinical Psychology, 59, 233-244.
46.
Smith, J. E.
, Meyers, R. J., & Delaney, H. D. (1998). The community reinforcement approach with homeless alcohol-dependent individuals. Journal of Consulting and Clinical Psychology, 66, 541-548.
47.
St. Pierre, R. G.
, Layzer, J. I., Goodson, B. D., & Berstein, L. S. (1997). National impact evaluation of the Comprehensive Child Development Program: Final report. Cambridge, MA: Abt Associates.
48.
Stecher, B. M.
, Andrews, C. A., McDonald, L., Morton, S., McGlynn, E. A., Petersen, L. P., Burnam, M. A., Hayes, C., & Vaccaro, J. V. (1994). Implementation of residential and non-residential treatment for dually diagnosed homeless. Evaluation Reviews, 18, 689-717.
49.
Weiss, C. H.
(1997). Theory-based evaluation: Past present and future. New Directions for Evaluation, 76, 41-56.
50.
Westat, Inc., Chapin Hall Center for Children, & James Bell Associates
. (1998). Evaluation of the New York City Homebuilders demonstration: Final report. Rockville, MD: Westat, Inc.
51.
Willett, J. B.
, & Sayer, A. G. (1994). Using covariance structure analysis to detect correlates and predictors of individual change over time. Psychological Bulletin, 116, 363-381.
52.
Winship, C.
, & Mare, R. D. (1992). Models for sample selection bias. Annual Review of Sociology, 18, 327-350.
53.
Winship, C.
, & Morgan, S. L. (1999). The estimation of causal effects from observational data. Annual Review of Sociology, 25, 659-707.
54.
Yeaton, W. H.
(1994). The development and assessment of valid measures of service delivery to enhance inference in outcome-based research: Measuring attendance at self-help group meetings. Journal of Consulting and Clinical Psychology, 62, 686-694.
55.
Yin, R. K.
(1994). Discovering the future of the case study method in evaluation research. Evaluation Practice, 15, 283-290.
56.
Yin, R. K.
(1997). Case study evaluations: A decade of progress?New Directions for Evaluation, 76, 69-78.
57.
Zanis, D. A
, McLellan, A. T., Belding, M. A., & Moyer, G. (1997). A comparison of three methods of measuring the type and quantity of services provided during substance abuse treatment. Drug and Alcohol Dependence, 49, 25-32.
