Assessing and Redressing Comprehension Artifacts in Social Intervention Research

Abstract

The failure of many complex social interventions can be attributed to a lack of comprehension on the part of respondents with respect to the appropriate behavioral response. This research details the means by which such "comprehension artifacts" can be detected and offset. The study employed data obtained from a large-scale electric rate demonstration experiment, in which treatment group subjects'rates were tied to the time of day at which they used electricity. Standard analytic approaches suggested no treatment influence on respondents' usage behavior. However, reanalysis disclosed that the rate variation manipulation had a considerable impact on those who could be identified as having understood the complex features of the treatment. This article details a methodology for discriminating knowledgeable from nonknowledgeable respondents, and outlines the manner in which this categorization can be validated in secondary analysis settings. In addition, the use of the empirically derived categorization as a "blocking "variable is discussed, along with a consideration of alternative methodologies. In research involving complex social interventions, the suggested approach can provide information of considerably greater precision than that obtained through more traditional analytic techniques.