Lessons in Contracting for Evaluations

Abstract

A 1995 evaluation of the U.S. Marine Corps’ personnel evaluation system and a 1991 evaluation of the National Assessment Governing Board procedure for setting cut scores on the mathematics section of the National Assessment of Educational Progress provided valuable lessons in how to minimize the risks of misunderstanding what an evaluation will involve, subversion of the evaluation, controversy, and animosity. Both evaluations were nationally significant, had to be conducted quickly, were politically volatile, were keyed to professional standards, and had substantial impacts. However, the latter evaluation went sour, whereas the former received an official commendation. Evaluators should do all they can to demonstrate to clients that sound evaluations are not to be feared, but should be valued and used. An investigation of what went right and wrong in these evaluations identified contracting as a key variable. This article advises evaluators and clients to regularly negotiate clear, sound contracts before proceeding with an evaluation, and presents a checklist to assist in the contracting process.

Get full access to this article

View all access options for this article.

References

Finn, C. E. , Stevens, F. L. , Stufflebeam, D. L. , & Walberg, H. J. (1997). The New York City public schools integrated learning systems project: Evaluation and meta-evaluation. International Journal of Educational Research, 27, 159-174.

Gates, B. M. , Horn, J. G. , & Stufflebeam, D. L. (1996). Developing a personnel evaluation system for the U.S.M.C.: Parallels and contrasts with developing a teacher evaluation process. Presented at the Fifth National Evaluation Institute, Bethesda, MD.

Hambleton, R. , & Dirir, M. (1991). Setting achievement levels on the 1990 NAEP mathematics assessment: Validity/replication results (final report). Amherst, MA: University of Massachusetts.

Joint Committee on Standards for Educational Evaluation . (1981). Standards for educational programs, projects, and materials. New York: McGraw-Hill.

Joint Committee on Standards for Educational Evaluation . (1988). The personnel evaluation standards. Newbury Park, CA: Sage.

Joint Committee on Standards for Educational Evaluation . (1994). The program evaluation standards. Thousand Oaks, CA: Sage.

Linn, R. , Koretz, D. , Barker, E. , & Burstein, L. (1991). The validity and credibility of the achievement levels for the 1990 National Assessment of Educational Progress in Mathematics. (Center for the Study of Evaluation Report No. 330). Los Angeles: Center for Research on Evaluation, Standards, and Student Testing, University of California at Los Angeles.

National Assessment Governing Board . (1990, May 10). NAEP policy framework and technical procedures. Washington, DC: National Assessment Governing Board.

National Assessment Governing Board . (1991, August 14). Response to the draft summative evaluation report on the National Assessment Governing Board’s inaugural effort to set achievement levels on the National Assessment of Educational Progress. Washington, DC: Author.

10.

Pellegrino, J. W. , Jones, L. R. , & Mitchell, K. J. (1999). Grading the nation’s report card. Washington, DC: National Academy Press.

11.

Scriven, M. S. (1969). Evaluation skills [Audiotape No. 6B]. Washington, DC: American Educational Research Association.

12.

Shepard, L. , Glaser, R. , Linn, R. , & Bornstedt, G. (1993). Setting performance standards for student achievement: A report of the National Academy of Education Panel on the evaluation of the NAEP trial state assessment: An evaluation of the 1992 achievement levels. Stanford, CA: National Academy of Education, Stanford University.

13.

Stufflebeam, D. L. , Jaeger, R. , & Scriven, M. (1991). Summative evaluation of the National Assessment Governing Board’s inaugural 1990-91 effort to set achievement levels on the National Assessment of Educational Progress. Kalamazoo, MI: Western Michigan University Evaluation Center.

14.

Stufflebeam, D. L. , Jaeger, R. , & Scriven, M. (1992, April 21). A retrospective analysis of a summative evaluation of NAGB’s pilot project to set achievement levels on the National Assessment of Educational Progress. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.

15.

Stufflebeam, D. L. , Madaus, G. , & Kellaghan, T. (Eds.) (2000). Evaluation models. Boston: Kluwer Academic Publishing.

16.

U.S. General Accounting Office . (1993). Educational achievement standards: NAGB’s approach yields misleading interpretations. Washington, DC: United States General Accounting Office.

17.

Vinovskis, M. (1998). Overseeing the nation’s report card: The creation and evolution of the National Assessment Governing Board (NAGB). Washington, DC: National Assessment Governing Board.