American Journal of Evaluation

Non-partisan, as defined here, refers to an evaluation that is and is regarded by partisans of all persuasions as balanced, fair, and faithful, so that if methodological quality is high, debates focus on the implications of the findings for practice or policy, not on the credibility of the findings themselves. This paper examines, from an historical perspective, what strategies have been used to achieve non-partisan evaluation, how well these have worked, and whether, however hard we try, there is a limit after which our virtue is unrewarded.

The drive for accountability in human services puts pressure on nonprofit agencies to develop performance measurement systems. But efforts to build capacity in this area have been hindered by the lack of instruments to evaluate the quality of proposed performance measurement systems. The Performance Accountability Quality Scale (PAQS) attempts to fill this gap. The instrument was field-tested on 191 program performance measurement systems developed by nonprofit agencies in Central Florida. Preliminary findings indicate that PAQS provides a structure for obtaining expert opinions based on a theory-driven model about the quality of a proposed measurement system in a not-for-profit agency. The instrument also is useful for assessing agency needs for technical assistance and for evaluating progress in the development of performance measurement systems. Further study is needed to test PAQS in other settings and to explore new areas of research in outcome evaluation.

In this paper I argue that research, development, and evaluation of social and educational programs should routinely be integrated through a planned variation approach. This approach typically involves the design, implementation, and evaluation of an enhanced, as well as standard, variation of a program. In contrast to conventional black box evaluations, I argue that a planned variation approach is better suited to the needs of program operators, offers methodological advantages, tests program theory, helps to build the base of knowledge for constructing better programs, and would foster greater utilization of evaluation findings. I discuss implications regarding the role and preparation of evaluators and issues in conducting planned variation evaluations. I compare the planned variation approach to other types of theory-based evaluation and discuss when and why it may be preferable.

In evaluation as it has been practiced, requirements for specialized training have caused a trade-off between rigor of method and the extent of program staff’s involvement in the evaluation process. There is a way to remove the trade-off between “rigor” and “use,” by applying industrial engineering (IE) tools that, though sophisticated and powerful, are packaged to facilitate use by people with relatively little specialized training. One level of IE use in evaluation simply adds tools to the evaluators’ toolbox. The second level employs the methodological abilities of evaluators to adapt and improve IE tools. Several problems must be solved in order to achieve the necessary integration. These include inserting the new evaluation technology into program settings, training and education, and developing a different kind of relationship between professional evaluators and those who work in settings where evaluation takes place.

The empowerment evaluation approach is applied and examined within the context of a case study involving three comprehensive communities’ initiatives. Although considerable literature has been published concerning the efficacy and legitimacy of the empowerment evaluation approach, little documentation exists regarding the empirical application of this approach. In this article we discuss, from both theoretical and empirical perspectives, issues related to the evaluator’s role, differences relative to other inclusive evaluation approaches, stakeholder involvement, evaluation context variables, and the concept and practice of empowering program participants via an evaluation approach. Data were gathered to assess how the approach was viewed through the eyes of both the evaluation consumers and the evaluators. Recommendations are discussed regarding the types of circumstances that might most facilitate and support the empowerment evaluation approach.

Portfolios and other open-ended assessments are increasingly incorporated into evaluations and testing programs. However, questions about the reliability of such assessments continue to be raised. After reviewing forces that may be leading to increased interest in and use of portfolio assessment, we investigate the interrater reliability of a portfolio assessment used in a small-scale program evaluation. Three types of portfolio scores were investigated—analytic, combined analytic (formed by summing across analytic scores), and holistic. The interrater reliability coefficient was highest for summed analytic scores (r 5 .86). Results indicate that at least three raters are required to obtain acceptable levels of reliability for holistic and individual analytic scores.

State-level school counselor certification requirements are discussed with respect to program evaluation expectations. Certification offices of all 50 states and Washington, DC, were asked to supply current school counselor knowledge and skill requirements. Nineteen states and Washington, DC, require some form of program evaluation knowledge and skills. Only Colorado and Washington specifically require program evaluation standards recommended by the Council for Accreditation of Counseling and Related Educational Programs for their school counseling programs. These findings suggest that the nation may be producing school counselors deficient in the program evaluation skills needed to meet their professional responsibilities. Several recommendations are made for ameliorating the current training deficiencies of pre-service school counselors and for using in-service training to remediate the program evaluation knowledge and skill gap of practicing school counselors. Implications for the larger evaluation community are discussed.

International program evaluation is a booming business, with important and challenging evaluations of development programs being conducted in almost every country in the developing world. However, many U.S. domestic evaluation practitioners are not yet familiar with this field. Evaluators of international development programs normally must operate in a very different environment than one would expect to find when evaluating U.S. programs. These differences are discussed and a number of promising developments and methodological approaches are described here. I conclude by suggesting a number of areas in which a closer exchange of experiences between U.S. evaluation practitioners and their colleagues from developing countries could be mutually beneficial.