Evaluation studies vary in methodological quality. It is essential to develop methodological quality standards for evaluation research that can be understood and easily used by scholars, practitioners, policy makers, the mass media, and systematic reviewers. This article proposes that such standards should be based on statistical conclusion validity, internal validity, construct validity, external validity, and descriptive validity. Methodological quality scales are reviewed, and it is argued that efforts should be made to improve them. Pawson and Tilley's challenge to the Campbell evaluation tradition is also assessed. It is concluded that this challenge does not have any implications for methodological quality standards, because the Campbell tradition already emphasizes the need to study moderators and mediators in evaluation research.
Baron, Reuben M. , and David A. Kenny. 1986. The moderator-mediator variable distinction in social psychology research: Conceptual, strategic and statistical considerations. Journal of Personality and Social Psychology51:1173-1182.
2.
Bennett, Trevor H.1996. What’s new in evaluation research: A note on the Pawson and Tilley article. British Journal of Criminology36:567-578.
3.
Boruch, Robert F.1997. Randomized experiments for planning and evaluation: A practical guide. Thousand Oaks, CA: Sage.
4.
Brounstein, Paul J. , James G. Emshoff, Gary A. Hill, and Michael J. Stoil. 1997. Assessment of methodological practices in the evaluation of alcohol and other drug (AOD) abuse prevention. Journal of Health and Social Policy9:1-19.
5.
Campbell, Donald T. , and Julian C. Stanley. 1966. Experimental and quasi-experimental designs for research. Chicago: Rand McNally.
6.
Centre for Reviews and Dissemination , U.K. National Health Service. 2001. Undertaking systematic reviews of research on effectiveness: CRD’s guidance for those carrying out or commissioning reviews. 2d ed.York, UK: York Publishing Services.
7.
Chalmers, Thomas C. , Harry Smith, Bradley Blackburn, Bernard Silverman, Biruta Schroeder, Dinah Reitman, and Alexander Ambroz. 1981. A method for assessing the quality of a randomized control trial. Controlled Clinical Trials2:31-49.
8.
Clarke, Ronald V. , and David Weisburd. 1994. Diffusion of crime control benefits: Observations on the reverse of displacement. In Crime prevention studies, Vol. 2, edited by Ronald V. Clarke. Monsey, NY: Criminal Justice Press.
9.
Consortium for Longitudinal Studies. 1983. As the twig is bent… Lasting effects of preschool programs. Hillsdale, NJ: Lawrence Erlbaum.
10.
Cook, Thomas D. , and Donald T. Campbell. 1979. Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally.
11.
Farrington, David P.1983. Randomized experiments on crime and justice. In Crime and justice, Vol. 4, edited by Michael Tonry and Norval Morris. Chicago: University of Chicago Press.
12.
Farrington, David P. . 1987. Evaluating area-based changes in policing strategies and laws. Police Studies10:67-71.
13.
Farrington, David P. . 1994. Interactions between individual and contextual factors in the development of offending. In Adolescence in context: The interplay of family, school, peers and work in adjustment, edited by Rainer K. Silbereisen and Eberhard Todt. New York: Springer-Verlag.
14.
Farrington, David P. . 1998. Evaluating “communities that care”: Realistic scientific considerations. Evaluation4:204-210.
15.
Farrington, David P. , Denise C. Gottfredson, Lawrence W. Sherman, and Brandon C. Welsh. 2002. The Maryland Scientific Methods Scale. In Evidence-based crime prevention, edited by Lawrence W. Sherman, David P. Farrington, Brandon C. Welsh, and Doris L. MacKenzie. London: Routledge.
16.
Farrington, David P. , and Rolf Loeber. 1999. Transatlantic replicability of risk factors in the development of delinquency. In Historical and geographical influences on psychopathology, edited by Patricia Cohen, Cheryl Slomkowski, and Lee N. Robins. Mahwah, NJ: Lawrence Erlbaum.
17.
Farrington, David P. , and Anthony Petrosino. 2000. Systematic reviews of criminological interventions: The Campbell Collaboration Crime and Justice Group. International Annals of Criminology38:49-66.
18.
Farrington, David P. . 2001. The Campbell Collaboration Crime and Justice Group. Annals of the American Academy of Political and Social Science578:35-49.
19.
Farrington, David P. , and Brandon C. Welsh. 1999. Delinquency prevention using family-based interventions. Children and Society13:287-303.
20.
Farrington, David P. . 2002. Improved street lighting and crime prevention. Justice Quarterly19:313-342.
21.
Forrester, David H. , Michael R. Chatterton, and Ken Pease. 1988. The Kirkholt Burglary Prevention Project, Rochdale. Crime Prevention Unit paper 13. London: Home Office.
22.
Forrester, David H. , Samantha Frenz, Martin O’Connell, and Ken Pease. 1990. The Kirkholt Burglary Prevention Project: Phase II. Crime Prevention Unit paper 23. London: Home Office.
23.
Gibbs, Leonard E.1989. Quality of study rating form: An instrument for synthesizing evaluation studies. Journal of Social Work Education25:55-66.
24.
Juni, Peter , Anne Witschi, Ralph Bloch, and Matthias Egger. 1999. The hazards of scoring the quality of clinical trials for meta-analysis. Journal of the American Medical Association282:1054-1060.
25.
Lipsey, Mark W. , and David B. Wilson. 1998. Effective intervention for serious juvenile offenders: A synthesis of research. In Serious and violent juvenile offenders: Risk factors and successful interventions, edited by Rolf Loeber and David P. Farrington. Thousand Oaks, CA: Sage.
26.
Loeber, Rolf , David P. Farrington, Magda Stouthamer-Loeber, and Welmoet van Kammen. 1998. Antisocial behavior and mental health problems: Explanatory factors in childhood and adolescence. Mahwah, NJ: Lawrence Erlbaum.
27.
Lösel, Friedrich , and Peter Koferl. 1989. Evaluation research on correctional treatment in West Germany: A meta-analysis. In Criminal behavior and the justice system: Psychological perspectives, edited by Hermann Wegener, Friedrich Lösel, and Jochen Haisch. New York: Springer-Verlag.
28.
Martinson, Robert M.1974. What works? Questions and answers about prison reform. Public Interest35:22-54.
29.
Moher, D. , A. R. Jadad, G. Nichol, M. Penman, P. Tugwell, and S. Walsh. 1995. Assessing the quality of randomized controlled trials. Controlled Clinical Trials16:62-73.
30.
Pawson, Ray , and Nick Tilley. 1994. What works in evaluation research?British Journal of Criminology34:291-306.
31.
Pawson, Ray . 1997. Realistic evaluation. London: Sage.
Petrosino, Anthony , Carolyn Turpin-Petrosino, and James O. Finckenauer. 2000. Well-meaning programs can have harmful effects! Lessons from experiments of programs such as Scared Straight. Crime and Delinquency46:354-379.
34.
Ross, H. Laurence , Donald T. Campbell, and Gene V. Glass. 1970. Determining the social effects of a legal reform: The British “breathalyzer” crackdown of 1967. American Behavioral Scientist13:493-509.
35.
Shadish, William R. , Thomas D. Cook, and Donald T. Campbell. 2002. Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton-Mifflin.
36.
Sherman, Lawrence W. , David P. Farrington, Brandon C. Welsh, and Doris L. MacKenzie, eds. 2002. Evidencebased crime prevention. London: Routledge.
37.
Sherman, Lawrence W. , Denise C. Gottfredson, Doris L. MacKenzie, John Eck, Peter Reuter, and Shawn Bushway. 1997. Preventing crime: What works, what doesn’t, what’s promising. Washington, DC: U.S. Office of Justice Programs.
38.
Sherman, Lawrence W. . 1998. Preventing crime: What works, what doesn’t, what’s promising. Research in brief. Washington, DC: U.S. National Institute of Justice.
39.
Weisburd, David . 2000. Randomized experiments in criminal justice policy: Prospects and problems. Crime and Delinquency46:181-193.
40.
Weisburd, David , Cynthia M. Lum, and Anthony Petrosino. 2001. Does research design affect study outcomes in criminal justice?Annals of the American Academy of Political and Social Science578:50-70.
41.
Welsh, Brandon C. , and David P. Farrington. 2003. Effects of Closed-Circuit Television on crime. Annals of the American Academy of Political and Social Science587:110-135.
