American Journal of Evaluation

In this paper, we object to Michael Scriven's claim that the basic logic of evaluation is criterial and standards-based. We note that valuing is an integral part of perception and that valuing within perception, repeatedly refined, is an even more basic logic of evaluation. We find unpersuasive his claim that making the final synthesis “governed” will diminish bias, noting that bias will find its way into the required statements of need, function, standards and weighting. We offer our alternative for disciplining the synthesis process, by urging more systematic and demanding critiques of emerging interpretations and values, and by more deliberately using competing conceptual organizers (e.g., goals, issues, decisions and elements of the rules Scriven advocates) as temporary and dialectical grounds for reconsidering the evolving meanings of the program, including its merit and shortcoming.

This study investigated internal and external evaluators' practices and concerns about communicating and reporting evaluation findings. Approximately three-quarters (72%) of a random sample of American Evaluation Association members responded to a survey on this topic. Most of those responding: (1) adhere to traditional reporting formats; (2) are only moderately satisfied with their communicating and reporting efforts; (3) found that insufficient time and political/organizational complexity impedes success in communicating and reporting; and (4) describe effective practice as typically entailing high stakeholder involvement. Internal evaluation was found to be not only equally as prevalent as external evaluation, but different in relation to certain communication and reporting practices.

Differential item functioning (DIF) is not often seen in the literature on attitude assessment. A brief discussion of DIF and methods of implementation is followed by an illustrative example from a program evaluation, using an attitude-towards-science scale with 1550 children in grades one through six. An item exhibiting substantial DIF with respect to gender was detected using the Mantel-Haenszel procedure. In a second example, data from workshop evaluations with 1682 adults were recoded to a binary format, and it was found that an item suspected of functioning differentially with respect to age groups was, in fact, not doing so. Implications for evaluation practice are discussed.

Our recent work with grassroots organizations raises several questions about empowerment evaluation. Critics have focused on concerns such as lack of rigor and objectivity, but few have addressed deeper philosophical issues. One of these issues centers on evaluation standards. Empowered program stakeholders who set the research agenda in the empowerment model may have different ideas from those of the professional evaluator about the structure of inquiry and the use of evaluation. In some cases, the evaluator may question the ethics underlying their choices. One critical issue is whether overarching standards for empowerment evaluation are needed, and if so, who defines them. A second issue focuses on whether there is harmony between the model's twin goals of evaluation ownership and group advocacy, which we found to be incompatible in some situations. Many grassroots organizations who were concerned with group advocacy (e.g., receiving recognition, attracting funding) found it pragmatic to surrender evaluation ownership to the institutionally-based, professional evaluators, particularly when they anticipated favorable evaluation findings.

Fostering self-determination is the defining focus of empowerment evaluation and the heart of its explicit political and social change agenda. However, empowerment evaluation overlaps participatory, collaborative, stakeholder-involving, and utilization-focused approaches to evaluation in its concern for such issues as ownership, relevance, understandability, access, involvement, improvement, and capacity-building. A critical question becomes how to distinguish empowerment evaluation from these other approaches. Making such distinctions has become critical as the field debates the boundaries and implications of empowerment evaluation.

It is suggested that there are some serious problems with defining the concept of empowerment evaluation, with its underlying assumptions, and with its proposed justification.