Increasing numbers of evaluators are cognizant of the need to observe treatment systemat
ically. To date, however, little headway has been made toward operationalizing and
executing the required effort. In the context of an experiment in criminal justice, a number of "real world" constraints on documenting, monitoring, and measuring treatment imple
mentation are discussed. The contract researcher who is hired to conduct an impact
evaluation af a black box intervention cannot readily divert the study'sfocus andfunds to an examination of process. By incorporating provisions for gathering implementation
data into theirfundingcriteria, evaluationfunders would help to advance the meaningful-
ness and social value of outcome studies.
Get full access to this article
View all access options for this article.
References
1.
Adams, S. (1976) "Evaluation: a way out of rhetoric," in Rehabilitation, Recidivism, and Research. Hackensack, NJ: National Council on Crime and Delinquency.
2.
———( 1975) Evaluative Research in Corrections: A Practical Guide. Law Enforcement AssistanceAdministration: National Institute for Law Enforcement and Criminal Justice.
3.
Berger, R.J., J.E. Crowley, M. Gold, and J. GRAY withM.S. Arnold (1975) Experiment in a Juvenile Court: A Study of a Program of Volunteers Working with Juvenile Probationers. Ann Arbor : Institute for Social Research, University of Michigan .
4.
Berstein, I.N., and J. Cardascia (1975) "Strategies and designs for criminal justice evaluation . Presented at the meeting of the American Sociological Association, San Francisco.
5.
——— and H.E. Freeman (1975) Academic and Entrepreneurial Research. New York: Russell Sage.
6.
Boruch, R.F., and H. Gomez' (1977) "Sensitivity, bias, and theory in impact evaluations ." Professional Psychology8: 411-434.
7.
Cook, T.J., and L.D. Dobson (1982) "Reaction to reexamination: more on Type III error in program evaluation." Evaluation and Program Planning5: 119-121.
8.
Cook, T.J., L.D. Dobson, and E.L. Rezmovic (1980) Working with Ex-Offenders : The Challenge Experiment . Research Triangle Park, NC: Research Triangle Institute.
9.
Cook, T.J., and F.P. Scioli (1975) The Effectiveness of Volunteer Programs in Courts and Corrections: An Evaluation of Policy Related Research. Washington, DC: National Science Foundation.
10.
Covner, B.J. (1944) "Studies in phonographic recordings of verbal material: III. The completeness and accuracy of counseling interview reports." J. of General Psychology30: 181-203.
11.
Cressey, D.R. (1958) "The nature and effectiveness of correctional techniques ." Law and Contemporary Problems23: 754-771.
12.
Dobson, L.D. and T.J. Cook ( 1980) "Avoiding Type III error in program evaluation: results from a field experiment." Evaluation and Program Planning3: 269-276.
13.
- - - (1979) "Implementing random assignment: a computer-based approach in a field experimental setting." Evaluation Q.3: 472-489.
14.
Fairweather, G. and L. Tornatzky (1977) Experimental Methods for Social Policy Research . New York: Pergamon.
15.
Fullan, M. and A. Pomfret (1977) "Research on curriculum and instruction implementation ." Rev. of Educ. Research47: 335-397.
16.
Glaser, D. (1974a) "Remedies for the key deficiency in criminal justice evaluation research." J. of Research in Crime and Delinquency11: 144-154.
17.
———( 1974b) "The state of the art of criminal justice evaluation ." Presented at the Second Annual Meeting of the Association for Criminal Justice Research, Los Angeles .
18.
———( 1973) Routinizing Evaluation: Getting Feedback on effectiveness of Crime and Delinquency Programs. Rockville, MD: Department of Health, Education, and Welfare.
19.
Gottfredson, D.M. (1972) "Five challenges." J. of Research in Crime and Delinquency9: 68-86.
20.
Hall, G.E. and S.F. Loucks (1977) "A developmental model for determining whether the treatment is actually implemented." Amer. Educ. Research J.14: 263-276.
21.
Harper, R.A. and J.W. Hudson (1952) "The use of recordings in marriage counseling: a preliminary empirical investigation." Marriage and Family Living14: 332-334.
22.
Kassebaum, G., D.A. Ward, and D.M. Wilner (1971) Prison Treatment and Parole Survival. New York: John Wiley.
23.
Leithwood, K.A. and D.J. Montgomery (1980) "Evaluating program implementation ." Evaluation Rev.4: 193-214.
24.
Lejins, P.P. (1971) "Methodologies in the evaluation of correctional programs," in Proceedings of the 101st Congress of Correction of the American Correctional Association. College Park, MD: American Correctional Association .
25.
——— and T.F. Courtless (1973) "A general model for justification and evaluation of correctional programs," in Proceedings of the 103rd Congress of Correction of the American Correctional Association. College Park, MD: American Correctional Association .
26.
Lipton, D., R. Martinson, and J. Wilks (1975) The Effectiveness of Correctional Treatment: A Survey of Treatment Evaluation Studies. New York: Praeger.
27.
Mann, J. (1971) "Technical and social difficulties in the conduct of evaluative research," in F. G. Caro (ed.) Readings in Evaluation Research. New York : Russell Sage.
28.
Martinson, R. (1974) "What works-questions and answers about prison reform." The Public Interest35: 22-54.
29.
Nelson, E.K., JR. and F. Richardson (1971) "Perennial problems in criminological research." Crime and Delinquency17: 23-31.
30.
Patton, M.Q. (1978) Utilization-Focused Evaluation. Beverly Hills, CA: Sage.
31.
Quay, H.C. (1979) "The three faces of evaluation: what can be expected to work," in L. Sechrest, S. G. West, M. A. Phillips , R. Redner, and W. Yeaton (eds.) Evaluation Studies Review Annual, Vol. 4. Beverly Hills, CA: Sage.
32.
Reckless, W. and S. Dinitz (1972) The Prevention of Juvenile Delinquency: An Experiment . Columbus, OH: State Univ. Press .
33.
Redlich, F.C., J. Dollard, and R. Newman (1950) "High fidelity recordings of psychotherapeutic interviews." Amer. J. of Psychiatry107: 42-47.
34.
Reed, J.A. (1974) "Program evaluation research." Federal Probation38: 37-42.
35.
Rezmovic, E.L. (1982a) "Evaluation practice: determinants of quality in social experimentation." Unpublished Ph.D. dissertation, Northwestern University.
36.
--( 1982b) "Program implementation and evaluation results: a reexamination of Type III error in a field experiment." Evaluation and Program Planning5: 111-118.
37.
---T.J. Cook, and L.D. Dobson (1981) "Beyond random assignment: factors affecting evaluation integrity." Evaluation Rev.5: 51-67.
38.
Riecken, H. W. and R. F. Boruch [eds.] (1974) Social Experimentation: A Method for Planning and Evaluating Social Intervention. New York: Academic.
39.
Rossi, P.H. (1979) "Critical decisions in evaluation studies," in W. B. Schrader (ed.) Measurement and Educational Policy Proceedings of the 1978 ETS Invitational Conference. Washington, DC: Jossey-Bass .
40.
———( 1978) "Issues in the evaluation of human services delivery." Evaluation Q.2: 573-599.
41.
Scheirer, M.A. (1980) "An organizational framework for implementation analysis." Presented at the Special National Workshop on Research Methodology and Criminal Justice Program Evaluation, Baltimore, Maryland.
42.
Schulman, H.M. (1961) Juvenile Delinquency in American Society. New York: Harper.
43.
Sechrest, L. and R. Redner (1979) "Strength and integrity of treatments in evaluation studies," in How Well Does It Work? Review of Criminal Justice Evaluation, 1978. Washington, DC: National Institute of Law Enforcement and Criminal Justice, Law Enforcement Assistance Administration, U.S. Department of Justice.
44.
Sechrest, L., S.O. White, and E.D. Brown (1979) The Rehabilitation of Criminal Offenders: Problems and Prospects. Washington, DC: National Academy of Sciences.
45.
Slaikeu, K.A. (1973) "Evaluation studies on group treatment of juvenile and adult offenders in correctional institutions: a review of the literature ." J. of Research in Crime and Delinquency10: 87-100.
46.
Symonds, P.M. and D.H. Dietrich (1941) "The effect of variations in the time interval between an interview and its recording." J. of Abnormal and Social Psychology36: 593-598.
47.
Thomas, S. (1980) "Numerical indicators of program implementation in field tests: Education" (working paper). Division of Methodology and Evaluation Research, Northwestern University.
48.
Ward, D.A. and G.G. Kassebaum (1976) "On biting the hand that feeds: some implications of sociological evaluations of correctional effectiveness," in C. H. Weiss (ed.) Evaluating Action Programs: Readings in Social Action and Education. Boston: Allyn and Bacon.
49.
Wilkins, L.T. (1964) Social Deviance. London: Tavistock.
50.
Yeaton, W. (1980) "Strength and integrity of treatments in evaluations . Presented at the Meeting of the Evaluation Research Society, Washington, DC.
