Abstract
In this secondary analysis of a previously conducted systematic review, we analyze social validity assessments in intervention research for transition-age autistic youth. Social validity is concerned with the acceptability of the intervention goals, the acceptability and feasibility of the intervention procedures, and the perceived importance of the intervention outcomes. We found that although just over half of intervention studies assessed some aspect of social validity, only 43% of those studies examined all three dimensions. There were several shortcomings of the social validation procedures, including a lack of psychometric validation for quantitatively scored questionnaires and a failure to describe qualitative procedures for analyzing open-ended questions. These shortcomings likely explain why interpretations of social validity assessments were nearly universally positive.
In the United States, when autistic youth who receive special education services reach transition age (between 14 and 16 years, depending on the state), they are entitled to educational services that are specially designed to prepare them for the transition from school to postsecondary education, employment, or other appropriate postschool contexts (Individuals With Disabilities Education Act, 2004, § 300.43).1 To better understand research geared toward developing services for transition-age autistic youth, we (Bottema-Beutel et al., 2023) recently undertook a systematic review and evaluation of this literature. In this review, we found that 193 intervention studies with transition-age autistic youth were published over a 50-year period. The bulk of this research used single-case research designs (SCDs) and examined behaviorally based interventions. Research guidance on SCDs stipulates that social validity should be examined alongside intervention efficacy to ensure the intervention is acceptable and of value to relevant stakeholders (Horner et al., 2005).
The purpose of this study is to determine how social validity was defined, assessed, and interpreted in the studies included in our previous review. Our motivation for analyzing social validation procedures is at least twofold. First, research quality is increasingly being called into question in autism research (Sandbank et al., 2019). It is possible that in addition to research designs characterized by threats to internal validity, social validity may also be inadequately assessed in this body of research (Ledford et al., 2016). Second, social validity may be particularly important to examine in intervention studies that involve autistic youth, because autistic youth and adults have expressed their ambivalence, and in some cases dissatisfaction, toward the services available to them (Bottema-Beutel et al., 2016, 2020), especially in regard to applied behavior analysis (ABA; Devita-Raeburn & Spectrum, 2016; Sequenzia, 2016). For example, in a qualitative study examining autistic adults’ recollections of participating in ABA, participants describe it as coercive, humiliating, and unethical (Anderson, 2022). Autistic youth have also challenged the expertise of professionals who provide services to them (Bottema-Beutel et al., 2016).
When recipients of intervention are so dissatisfied with services that they engage in activism to discourage their use, services are considered socially invalid (Schwartz & Baer, 1991). Social validity assessments, if conducted appropriately, could be taken as an opportunity to explore this growing sentiment among autistic people. Critically examining social validation procedures can uncover shortcomings that could be addressed in future research to ensure services are developed that are embraced by the populations they are meant to support.
A Historical Look at Social Validity in Intervention Research
Conceptualization
According to Kennedy (1992), “The focus of social validity assessments rests with the question: ‘Is this change in behavior, and/or the process used to change it, of social value?’” (p. 147). Research on social validity has its roots in consumer and employee satisfaction in business, client satisfaction in psychotherapy, and patient satisfaction in medicine (summarized in Carter & Wheeler, 2019, and Kennedy, 1992). In the late 1970s, the concept of social validity was taken up in ABA research (Kazdin, 1977; Wolf, 1978). In this research tradition, social validity is defined along three dimensions: (a) the acceptability of the goals, (b) the acceptability or feasibility of the procedures, and (c) the perceived importance of the outcomes (Kazdin, 1977; Wolf, 1978). The most important determiners of whether or not an intervention is socially valid are those directly or indirectly associated with the intervention in real-world settings, such as the people receiving services, their parents, peers, or school professionals who are trained to implement the intervention (Ledford & Gast, 2018).
Establishing the social validity of interventions is thought to be important for ensuring that interventions examined in research contexts will actually be adopted and sustained in practice (Leko, 2014; Snodgrass et al., 2018). Kazdin (1977), an early proponent of social validation in behavioral research, connected the importance of social validity to ethical and legal issues arising at the time, specifically in regard to the rights of individuals undergoing behaviorally based treatments. In his seminal paper on this topic, Kazdin references a volume exploring court cases that challenged the use of behavior modification techniques in schools, correctional facilities, and mental health institutions (Martin, 1975), implying that social validation procedures may be a means to address these issues and avoid future litigation. These concerns, along with the realization that behavior-analytic experiments were unable to capture the social contexts in which behavior was embedded (Kennedy, 1992), led to the inclusion of social validation procedures as a key quality indicator for establishing evidence-based practices based on SCDs (Horner et al., 2005). SCDs are the method of choice for many behavioral researchers, but intervention researchers with other orientations also sometimes use SCDs. Therefore, social validation is now used in a variety of theoretical approaches to intervention.
Assessment Procedures
Social validation assessments were initially designed to align with the methods already used in behavioral research (Kazdin, 1977). Because social validity was defined in part as meaningful change that can be readily perceived by users, and because SCDs require visual analysis in lieu of significance tests, statistical procedures were discouraged as a means to investigate social validity (Kazdin, 1977). The logic was that behavior changes that are clinically important should be obvious enough to render statistical tests unnecessary. From the outset, guidance on assessing social validity has distinguished between “subjective” assessments of social validity—those that capture stakeholder perspectives on the intervention goals, procedures, and outcomes—and “objective” assessments of social validity—those that involve raters who are masked to experimental condition or group, or that involve ratings of comparison groups of individuals not undergoing the interventions (i.e., social comparison; Kazdin, 1977; Kennedy, 1992; Ledford & Gast, 2018). Behavioral researchers’ purported commitments to objectivity meant that subjective perceptions of social validity were a departure from the methods normally espoused in this field, and there was some uncertainty about including subjective measures alongside experimental analysis (Wolf, 1978). Subjective evaluations were ultimately considered compatible with behavioral research, as expressed by Wolf (1978): “It seems that if we aspire to social importance, then we must develop systems that allow our consumers to provide us feedback about how our applications relate to their values, to their reinforcers” (p. 213).
However, we view this proposed dichotomy between subjective and objective procedures as problematic. Establishing whether interventions or the outcomes they produce are socially valid cannot be examined objectively, because it inherently involves subjective determinations of what is and is not of social value. Take for example social comparison measures, proposed to be an “objective” means of measuring social validity (Kazdin, 1977; Ledford & Gast, 2018). In autism intervention research, social comparison involves contrasting an autistic person's performance on some outcome as they undergo therapy with a nonautistic person's performance on the same outcome while in a natural setting. Selecting this procedure as a means for social validation places value on the behavioral capacities of nonautistic people and sets their conduct as the standard by which autistic people's conduct should be judged. Further, proponents of objective assessment assume that researchers, unlike consumers or their families, provide an unbiased view of the intervention and the resulting behavior. However, this is not the case because researchers—especially those who study interventions they themselves designed and in many cases provide as clinicians—are not free from their own perspectives and biases (Bottema-Beutel & Crowley, 2021), even if they are masked to condition or group when conducting assessments. We point out what we consider to be a false distinction between subjective and objective evaluations of social validity to note that social validation, especially within ABA research, has internal inconsistencies that may prevent researchers from adequately discerning the social implications of the interventions they study.
Previous Reviews of Social Validation Procedures in Intervention Studies
There are several previous reviews that have examined the use and design of social validation procedures that have focused on research with specific populations (e.g., autistic children or children with intellectual disabilities), publication outlets (e.g., special education journals, ABA journals), types of interventions (e.g., peer-mediated behavioral interventions), and time periods (e.g., before and after social validity was popularized among ABA researchers). To offer context to our study, we summarize these reviews and discuss findings related to the prevalence of social validation, research and measurement design, and interpretation of findings.
Among these reviews, the proportion of intervention studies that involve social validation varies between 12% and 44%, with an apparent increase and then leveling off soon after the introduction of the concept to behavioral research (Carr et al., 1999; Ferguson et al., 2019; Kennedy, 1992; Ledford et al., 2016; Snodgrass et al., 2018). However, primary researchers are inconsistent in the dimensions of social validity they assess. In Kennedy's (1992) review of studies published between 1968 and 1990 in Journal of Applied Behavior Analysis and between 1977 and 1990 published in Behavior Modification, he found that most validation procedures focused on discerning the importance of the outcomes, followed by the acceptability of the procedures and goals. Extending Kennedy's work, Ferguson et al. (2019) examined social validation procedures in studies published in the Journal of Applied Behavior Analysis from 1999 to 2016. They found that acceptability of the procedures and the importance of the effects were most often measured; acceptability of the goals was relatively rare. In a review of SCD studies published over an 11-year period in six special education journals, Snodgrass et al. (2018) found that fewer than 7% of studies that assessed social validity assessed the total construct, which includes all three dimensions.
Social validity is most often assessed with brief questionnaires or interviews, with open-ended or Likert-type responses to questions, and is usually administered after the completion of the intervention (Ferguson et al., 2019; Ledford et al., 2016; Snodgrass et al., 2018). Two reviews examined the prevalence of objective versus subjective measures. Kennedy (1992) reported that subjective assessments were far more common than objective assessments, such as normative comparisons. In their review of SCDs examining social skills interventions for young autistic children published between 1994 and 2013, Ledford et al. (2016) corroborated this finding and showed that although none of the studies reported negative social validity findings, objective measures were less positive in regard to social validity than subjective measures.
One review (Snodgrass et al., 2018) examined whether the scientific method was applied to research questions related to social validity among the articles that measured all three social validity dimensions. They found that none of the articles applied all six steps (pose a research question, review literature, state a hypothesis, test the hypothesis, report analysis procedures, and report results). Specific analysis procedures were the least frequently reported, suggesting that there may be a more pervasive lack of attention to how social validity data are analyzed and interpreted as compared with how it is gathered. They also found that social validity was not mentioned in Discussion sections 25% of the time.
Research Questions
Taken together, these reviews suggest that social validation procedures are not consistently applied in intervention research and that there are quality issues in regard to how they are designed, implemented, and interpreted. The current project aims to explore this issue further, in the context of research on transition-age autistic youth. As part of an overall assessment of any body of intervention research, it is of interest to also assess social validation procedures to determine the utility of that research. Because transition-age autistic youth receive different types of intervention and have different communication profiles than other age groups of autistic people or disability populations (which could impact whether feedback is solicited from them, for example), it is worth specifically assessing social validation practices used with this population. At present, no other studies have done this. To understand the robustness of social validation practices, we posed the following research questions:
Frequency of use: What proportion of intervention studies for transition-age autistic youth assessed social validity, and how has this changed over time? Construct dimensions: What dimensions of social validity were measured, including the acceptability of the goals, acceptability and feasibility of the procedures, and perceived importance of the outcome? Assessment procedures: What were the features of the social validation procedures, in terms of data collection methods and measures, timing of the assessment (e.g., before, during, or after the intervention), scoring and analysis procedures, and selection of informants? Interpretation: Were findings mentioned in the Discussion? Did researchers prespecify the type of findings (such as benchmarks on Likert scale assessments or other interpretive frameworks for qualitative procedures) that would allow them to determine the extent to which the intervention was socially valid? What conclusions were drawn by the researchers in regard to social validity, and were social validity findings clearly separated from efficacy results?
Method
A detailed description of the search strategy, a coding manual, a spreadsheet containing raw data, and a list of included studies are available on the Open Science Framework (https://osf.io/ghpeb). Search and selection procedures were preregistered on Prospero (registration number: 231764).
Eligibility Criteria
For the larger systematic review, articles were considered for inclusion if they were published in a peer-reviewed, English-language journal between 1970 and November 2020. To be selected, the study had to (a) use SCD or group design that included a control group, (b) implement a nonpharmacological intervention with transition-age autistic youth ages 14 to 22, and (c) report outcomes for autistic participants separately, or more than 70% of participants had to have a diagnosis of autism, pervasive developmental disorder, or Asperger's syndrome. Studies that included autistic youth who were enrolled in a university (and therefore already completed the transition process) were excluded.
Search Strategy
We searched 11 databases separately, including Academic Search Complete, CINAHL Plus, Sociological Abstracts, Social Work Abstracts, Education Source, ERIC, PsycARTICLES, SocINDEX with Full Text, Web of Science, PsycINFO, and Medline. Each database was searched using terms that reflected the population (autis* OR ASD), age (transition* OR adolescen* OR school age OR high school OR secondary), and the provision of an intervention (intervention OR therapy OR teach* OR treat* OR program OR practice* OR strateg*). A PRISMA diagram depicting the flow of retrieved articles through the search-and-selection process is available in Bottema-Beutel et al. (2023).
Coding
Each article was coded for features of social validation procedures described next.
Presence or absence of social validity assessment
We determined whether the study included a self-described social validity assessment, even if it was described as informal. We did not include studies that used other descriptors (e.g., if they used instead “consumer satisfaction survey”) as we were interested in studies that drew on social validation concepts developed in intervention research relevant to autistic children. We also coded whether social validity was measured before, during, or after the intervention or some combination of these time points.
Social validity dimensions included in the assessment
We examined the social validity assessment system in each study to determine if it included assessment of (a) the acceptability of the goals, (b) the acceptability or feasibility of the procedures, and (c) the perceived importance of the outcomes. We considered assessments of stakeholder perceptions regarding whether the intervention had an effect as a component of dimension (c). If the social validity assessment system did not assess any of these three dimensions but measured something about the intervention that the authors considered social validity, we coded this as “other.”
Features of the social validity assessment system
We examined four features of social validity assessment systems: (a) the type of measure (e.g., standardized instruments, adapted standardized instruments, observations, researcher-created questionnaires, interviews, or some other type of assessment); (b) the scoring and coding procedures (e.g., Likert scale scoring, quantitative observational coding, thematic or content coding, a checklist, some other procedure, or if the scoring or coding system was not clear); (c) assessor or informant (i.e., who completed scoring, coding, or responded on any aspect of the assessment system, such as researchers or clinicians, caregivers, school professionals, peers, autistic participants in the intervention, employers, or some other assessor or informant); (d) whether all of the features of the social validity assessment were available (which could be described in the report or made available through some other source), partially available, or not available; and (e) reliability for quantitative analysis or procedures to establish trustworthiness in qualitative analysis. This latter feature was coded as either (a) yes, reliability or trustworthiness was confirmed; (b) no, but the procedures traditionally require reliability or verification of trustworthiness; (c) no, and the assessment procedures were not described clearly enough to determine if reliability or trustworthiness verification was required; or (d) not applicable, for instances when the assessment protocol did not require reliability or trustworthiness verification, such as a caregiver survey.
Interpretation of social validity findings
This dimension refers to whether the Discussion sections mentioned social validity assessment results, whether there were benchmarks or other appropriate frameworks described for interpreting social validity results anywhere in the manuscript, and whether the authors interpreted that the intervention was socially valid, partially socially valid (if only one or two dimensions of social validity were mentioned), or not socially valid or did not draw conclusions. In order to be considered an interpretation of findings, researchers had to go beyond restating results (e.g., scores on stakeholder Likert scale responses). Finally, we coded whether the researchers used the social validity results to make or bolster claims about the efficacy of the intervention. For instance, if a researcher stated, “Parent interviews on the acceptability of the intervention offer additional evidence that the treatment was effective,” this would have been coded as “yes” for this item.
Intercoder Agreement
Studies were independently coded by all six authors, and 100% of studies were double coded to calculate intercoder agreement. Agreement in regard to whether the study assessed social validity was 95%. After researchers were trained to 80% agreement on an initial set of seven studies, the remaining studies were coded in sets of 20 to 30. Discrepancy discussions were held after each set to prevent coding drift and resolve discrepant codes. After the first round of coding, all but four coding categories met our preset criterion of 80% agreement. These four coding categories were subject to an additional round of agreement; three sets with particularly high rates of disagreement (60 studies) were selected for recoding, and agreement was then recalculated by combining agreement totals on these recoded studies with the full set of studies (102 studies). After this round, two of the four codes were not yet at the 80% threshold and so were subject to a third and final round of coding (again by selecting 60 studies from sets with high disagreement), and agreement was recalculated. After this round, agreement across all coding categories ranged from 80% to 100% and averaged 92%. As a final check on the accuracy of our coding, we examined categories for contradictory codes and made any necessary adjustments after referring to the indicated manuscript (e.g., if all of the categories for scoring procedures were marked as “no,” the unclear scoring category should be coded as “yes”).
Results
We refer readers to the main review (Bottema-Beutel et al., 2023) for participant demographics and intervention characteristics.
Frequency of Use and Construct Dimensions
Of the 193 studies included in our review, 102 assessed social validity (53%). Only one group-design study assessed social validity of the 12 that met inclusion criteria for the larger review. Over time, the percentage of studies that included social validation procedures has steadily increased, ranging from 0% in the 1990s to 69% in the 2020s. The percentage of studies assessing all three dimensions of social validity (i.e., total construct) rose slightly between the 2000s and 2010s, but then decreased in the 2020s. See Figure 1 for an illustration of social validation over time. Considering only the studies that included social validity, most assessed the acceptability of the intervention or the perceived importance of the outcomes, whereas just over half assessed the acceptability of the goals. The total construct was measured in 41% of studies that examined social validity. See Table 1 for numbers and percentages of each dimension.
Percentage of studies with social validity assessments by decade.
Dimensions of Social Validity Measured.
Dimensions are not mutually exclusive categories.
Denominator for percentage is the number of studies that assessed social validity (n = 102).
Studies with social validation procedures coded as “other” assessed peer comparison and participant task choice. Note that “other” was coded only when one of the three dimensions of interest was not a component of the social validity procedures.
Data Collection and Analysis
The most common type of social validity assessment was researcher-created questionnaires with Likert scale rating systems. In comparison, validated instruments or adaptations of validated instruments were infrequent, appearing in only one quarter of studies that examined social validity. However, most social validation procedures included at least one facet for which we could not clearly determine how the data were coded or scored; the most common reason for this was failure to specify how open-ended questions were coded. Autistic youth and school professionals were the most common informants, and most measures were administered after the completion of the intervention. Reliability and trustworthiness were never assessed, even though at least some scoring procedures should have reported reliability (e.g., researcher-coded video observations or coding open-ended questions). Full descriptions of social validation practices including both data collection and analysis strategies were rarely provided in manuscript text, supplementary files, or elsewhere, but almost half included at least some procedures (e.g., sample questions). Table 2 provides details on the features of the social validation procedures, including the type of assessment, the scoring and analysis procedures, whether or not reliability was assessed, whether or not the procedures were available, who completed the measure, and when during the study social validity was assessed.
Features of Social Validation Procedures.
Denotes mutually exclusive categories. Denominator is the number of studies in which social validity was assessed (n = 102).
Interpretation
Researchers almost never specified in advance the kind or extent of evidence that would lead them to conclude that their intervention was socially valid. The results of social validation procedures were mentioned in the Discussion in 65% of studies, but these results were linked to a determination about the extent to which the intervention was socially valid in only half of the studies. The most frequent conclusion was that the intervention was socially valid, followed by conclusions that the intervention was socially valid on at least one dimension (this was relevant to studies that did not measure all dimensions of social validity). None of the studies concluded that the intervention was not sufficiently socially valid. In a quarter of studies that measured social validity, the results of the social validity assessments were inappropriately conflated with evidence for intervention effectiveness. See Table 3 for numbers of percentages of studies in regard to interpretation of social validity findings.
Interpretation of Social Validity Findings.
Note. Denominator is the number of studies in which social validity was assessed (n = 102).
Discussion
Strengths Relative to Previous Findings
A potentially positive finding in our study is that researchers more frequently used social validation procedures, and more frequently measured all three dimensions of social validity, than has been reported in previous systematic reviews (Carr et al., 1999; Ferguson et al., 2019; Kennedy, 1992; Ledford et al., 2016; Snodgrass et al., 2018). It is also promising that the proportion of studies that include social validation is increasing over time. Another potential strength of the studies in our review is that autistic youth were one of the most common informants, which indicates many researchers consider their perspectives to be important sources for social validation. We also found more frequent mention of social validity results in Discussion sections than has been previously reported (e.g., Snodgrass et al., 2018), suggesting that researchers in this area more often consider social validity to have at least some bearing on determining the utility of a given intervention.
Concerns
There were several significant areas of concern in our review of social validation procedures that temper our enthusiasm for increased frequency of social validation. If social validation procedures are performed inadequately, studies may draw misleading conclusions about the social validity of interventions, which is perhaps worse than omitting this process altogether. We describe each of these issues next, including the frequency of use, data collection and analysis, and broader conceptual issues.
Frequency of use and construct dimensions
Although social validation procedures in our study were relatively common compared with other reviews, our findings indicate that there is inconsistent use of these procedures and that the majority of intervention studies designed for autistic youth are socially validated with only partial assessment of the construct. Further, the use of total-construct social validation procedures appears to be declining in the current decade relative to the previous decade. The relative infrequency of socially validating intervention goals is concerning, as autistic people have expressed dissatisfaction with goals that are overly focused on compliance or reducing autistic characteristics, such as stereotypic behavior (Lynch, 2019). Central to these critiques are that many intervention goals aim to teach autistic youth to pass as nonautistic, which can have negative mental health implications (Cage & Troxell-Whitman, 2019). Further, autistic youth view some adult-derived goals as intrusive to their social lives and express dissatisfaction that they are not developed with an understanding of the social contexts in which autistic youth actually live (Bottema-Beutel et al., 2016).
Data collection and analysis
There was a concerning lack of transparency in terms of the methods used to gather stakeholder feedback and, consistent with Snodgrass et al. (2018), in terms of the methods used to analyze that feedback. This issue is most glaring in the relatively common failure to provide the full set of questions posed to participants, and the near-total absence of describing analysis procedures for responses to open-ended questions. Without the full set of questions researchers pose to participants, it is unclear the extent to which researchers directly ask participants whether they perceived any aspects of the intervention to be problematic. Nearly three decades ago, Schwartz and Baer (1991) asserted that as part of social validation procedures, intervention recipients “should be urged to complain, and complain early” (p. 191), as opposed to encouraging “false praise from consumers” (p. 191). This advice was not readily apparent in the studies that comprised our review.
Only three studies reported qualitative procedures for analyzing open-ended questions. In one study, the authors indicated that they “coded for themes” but did not reference any qualitative methods texts or specify any further how they did this (DaWalt et al., 2018). The remaining two studies offered much clearer procedures, cited relevant qualitative methods guidelines, and were written by the same first author (Kellems et al., 2018; Kellems & Morningstar, 2012). Despite this overall lack of transparency, it was quite common for researchers to present direct quotations from participants—nearly all of which cast the interventions in a favorable light. Without systematic procedures for extracting data from open-ended questions, researchers are free to cherry-pick positive quotes and omit mention of stakeholder concerns about the intervention.
When researchers used quantitative methods (e.g., questionnaires with Likert scale responses), most of the data collection instruments were not subject to psychometric validation, despite the availability of such measures (e.g., Briesch et al., 2013). It is therefore unclear if measures adequately assess perspectives about the acceptability of the intervention goals and procedures or the importance of intervention outcomes. Further, when brief quantitative measures are used with small sample sizes in the context of SCDs, and administered only once, the information they provide is quite limited.
Another concern relevant to both qualitative and quantitative approaches is that there were no considerations for counteracting desirability bias, that is, the tendency for participants to provide responses consistent with researcher expectations (Bergen & Labonté, 2020). This is important in work that relies on small sample sizes, as there is little possibility of responding anonymously. Desirability bias was identified as a caveat of social validation procedures soon after their incorporation into behavioral methods (Kazdin, 1977) and was noted as a reason that user perceptions of intervention effectiveness did not always match experimental evidence of efficacy. However, procedures for gathering or analyzing social validation data have not been substantially adjusted to address this issue over the past four decades.
In our opinion, the combination of issues relevant to qualitative and quantitative approaches currently used is a “worst of both worlds” scenario. Quantitative instruments are hastily designed and give too little information when used in very small samples, and qualitative data that are not analyzed with any degree of rigor can yield misleading results.
Interpretation of results
It is concerning that studies that examined social validity offered an interpretation of their findings only half the time. In studies that did interpret data, there was universal determination that the intervention was socially valid on whatever dimension of social validity the authors measured; none of the studies reported insufficient social validity. This is notable given that researchers almost never set a priori benchmarks or frameworks for determining the type of results that would allow for such an interpretation of the data they collected. Our conclusions are consistent with previous observations that social validity procedures are substantially biased toward affirmative findings (e.g., Ledford et al., 2016; Ledford & Gast, 2018; Snodgrass et al., 2018). It defies belief that procedures that either are not significant enough to yield findings that warrant interpretation or result in universally positive conclusions are actually uncovering anything meaningful about the intervention being studied (Schwartz & Baer, 1991).
Kazdin (1977) clarified that social validation presupposes treatment effectiveness; social validity results are not particularly helpful if they are presented alongside interventions that do not result in behavior change. Inherent to this perspective is that social validation procedures are separate from evaluations of intervention efficacy (Schwartz & Baer, 1991). However, a quarter of studies that assessed social validity used their findings to bolster conclusions about efficacy. That is, they construed participants’ perceptions that the intervention was successful as further evidence that there was a functional relation between the intervention and the outcome. This is problematic for multiple reasons, including that social validation procedures are not subject to experimental manipulations that would be needed to determine functional relations, and they are subject to biases that make them inappropriate for determining intervention efficacy, such as placebo effects, placebo-by-proxy effects, and response bias (Jones et al., 2017; Schwartz & Baer, 1991). Such mixing of evidence decreases the soundness of the conclusions compared with instances where social validity is not measured at all.
Broader Conceptual Issues
Neither in the initial days of social validity development in intervention research nor in current practice has social validation involved examination of the processes whereby particular outcomes or behaviors become meaningful or socially valued, the ableism (or other forms of systemic oppression) that likely impact such processes, or how social values change over time.
Another example is punishment procedures. Hanley (2010) reports “counterintuitive findings” (p. 18) in the social validation of interventions that include punishment components, drawn from preference assessments conducted with autistic children. Preference assessments involve presenting participants with a choice between two or more intervention approaches, and their selection is taken as an endorsement of the intervention procedures. In several studies, autistic children chose interventions that involved physical punishment over interventions that did not (e.g., Hanely et al., 2005). However, this work has not sufficiently considered the social contexts in which autistic children are offered these choices, including whether they are systematically deprived of experiences such as social attention, access to activities, or other necessities in the absence of interventions that involve punishment. Failure to consider these important contextual details, and what they mean for autistic people's experience in interventions, can lead to the erroneous conclusion that autistic children find physical punishment acceptable.
These two examples show how social validation procedures as they are both historically and currently used are ill equipped to understand the complex social processes they aim to document. Our analysis, together with previously conducted reviews, indicates this is in part due to the lack of rigor in both the data collection methods used to gather stakeholder perspectives and the analytic procedures used to understand the collected data. In addition, however, is the lack of depth or critical perspectives employed in the conceptualization of the construct. This has limited analysis to the very immediate sphere of the intervention itself, instead of posing broader questions about the types of people and behavior that are thought to require intervention in the first place. Additionally, there is insufficient examination of how broader social contexts in which interventions are situated structure the ways informants respond to social validity assessments—including institutional and societal oppression. Because of this, many significant concerns around ethical practice, human rights, and harm seem to elude researchers even after a half century of research that includes social validation procedures.
Limitations
There are some limitations to note when interpreting the results of this study. First, the study is somewhat narrow in scope, as we examined social validation procedures only for a particular disability population and age group. It is possible that researchers who conduct intervention studies on other populations assess social validity in ways that differ from what we report here. Second, our primary interest was in discerning the utility of social validity assessment procedures in this literature, which we found to be lacking. Therefore, we are unable to comment on the extent to which the interventions included in our review were socially valid. Finally, we conceptualized social validity as involving only three dimensions, which correspond to what Snodgrass et al. (2018) consider to comprise total-construct validity. However, there are other conceptualizations of social validity proposed by other researchers that we did not examine, such as maintenance of behavior change over time (Kennedy, 2002).
Recommendations
Evidence-based practice guidelines for SCDs should be updated to specify much more rigorous methods of social validation, which should include in-depth explorations of stakeholder experiences and perspectives. Brief researcher-designed questionnaires or interviews continue to be insufficient at best, and misleading at worst, for assessing social validity and should have no bearing on whether a practice is considered evidence based. We are certainly not the first to point out the many issues with social validity assessments as they are currently implemented, but our recommendations depart from those of many previous reviewers who pose that only increasing the use of what are considered objective assessments (e.g., preference assessments, social comparison assessments) will lead to more robust social validation (e.g., Kennedy, 1992; Ledford & Gast, 2018; Schwartz & Baer, 1991). We have already provided some critique of objective assessments, including that they are not in fact objective and that they do not capture what are perhaps the most important aspects of the social context of interventions. Instead, we propose more robust qualitative approaches to social validation. These approaches could include extensive, longitudinal investigations using semistructured interviews, extensive focus groups, ethnographic observations, and artifact analyses coupled with established data analysis procedures that attend to issues of trustworthiness (Kornbluh, 2015; see Kellems & Morningstar, 2012, for a description of qualitative methods in the context of SCD research). Researchers should also be clear about their positionality in this work, including whether they are strongly invested in the development and dissemination of the intervention. Stronger social validation will likely be possible only when it is conducted by researchers who are not intervention designers or practitioners who provide the intervention being examined.
There are few examples of robust designs in the research that comprised our review. However, there are more rigorous examinations of social validity in other research on school-based practices for children with disabilities (including autistic youth) that are not in the context of efficacy studies but are still informative. For example, there are examinations of stakeholder experiences and perceptions of paraprofessional support, social-focused school-based interventions, collaborative teaching arrangements, and reading curricula (e.g., Boer et al., 2005; Bottema-Beutel et al., 2016, 2020; Copeland et al., 2004; Gerber & Popp, 1999; Leko, 2014; Lyst et al., 2005). In these studies, methods include extensive interviews and focus groups with a variety of stakeholders, including the people for whom the practice was designed, teachers, and caregivers, and they use established methods for analyzing qualitative data, such as content analysis, thematic analysis, and deductive coding.
There are two things to note about this small body of research. First, none of the interventions represented in these examples were traditional behaviorally based strategies, suggesting that researchers with more general orientations to supporting youth with disabilities may have access to training in rigorous qualitative methods in ways that behavioral researchers generally do not. Second, these examples of social validation research reported much more critical evaluations of the interventions or practices of interest. This affirms our aforementioned conclusion that traditional social validity methods are biased toward showing favorable perceptions (Ledford et al., 2016), and more rigorous methods will likely uncover aspects of interventions that are problematic. Finally, these examples are stand-alone studies, in contrast to studies in which social validity is only a small component of an efficacy study. If researchers wish to understand the social contexts of the interventions they examine, social validation will likely need to be given much greater attention, and perhaps published in dedicated reports, to allow for sufficient analysis and description. Importantly, such studies should be conducted prior to recommendations as to whether interventions should be adopted in practice.
Quantitative instruments are more appropriate for group-design studies compared with small-N SCDs (although they could be used as a means to triangulate robust qualitative analyses). There are examples of intervention-specific social validation procedures that have been subject to at least initial psychometric validation (e.g., Berger et al., 2016). Importantly, researchers should retool instruments that produce ceiling effects (i.e., if respondents provide only the highest ratings across intervention studies). Benchmarks for interpreting psychometrically validated social validity questionnaires should also be empirically determined to offer researchers a framework for interpreting scores on these measures. These instruments could be administered to participants in the context of randomized controlled trials. Additionally, social validity could be compared across interventions to make decisions about which interventions are most appropriate for practice. Finally, psychometrically validated questionnaires could be administered to large, representative samples of autistic people to determine general sentiments about intervention practices and to see how perspectives differ by cultural, racial, ethnic, and geographic groupings.
Conclusion
At present, social validation procedures appear to be largely a box-checking exercise used to meet minimal quality standards that, in our opinion, do not include robust guidelines for establishing social validity.
Footnotes
Notes
Authors’ Note
Kristen Bottema-Beutel has previously received fees for consulting with school districts on intervention practices for autistic children and teaches courses on autism interventions in her role as an associate professor of special education. She has also accepted speaker fees in the amount of $750 to discuss her work on research quality, adverse events, and researcher conflicts of interest as they pertain to autism intervention research. She also receives royalties for a coedited book titled Clinical Guide to Early Interventions for Children With Autism, published by Springer. At the time of publication, the total amount of royalties received for this work was $261. Shannon Crowley La Point was formerly affiliated with an entity that trained students to become Board Certified Behavior Analysts and provided early intensive behavioral intervention. She is also affiliated at University of North Carolina - Chapel Hill. Rachael McKinnon was previously employed as a special educator in a school that provided transition supports to autistic students.
