Abstract
Program theory reconstruction is an often-overlooked aspect of social intervention research. In this paper, we argue that intervention research benefits if the research design is informed by the specific intervention's program theory (i.e., the idea of how the intervention is supposed to lead to the intended outcomes). The purpose of this paper is to offer a comprehensive and accessible guide to program theory reconstruction in research on social interventions and to provide arguments as to how program theory reconstruction can be used to benefit intervention studies. First, we summarize what program theory is and its role in intervention research. Second, we provide a direct “how-to” for researchers, practitioners, and students who may be unfamiliar with the methods of program theory reconstruction but are interested in undertaking a program theory reconstruction. Finally, we conclude with how program theory reconstruction can benefit intervention research.
The science of social intervention is concerned with the exploration, development, testing, and dissemination of intentional change strategies in order to serve various groups across the lifespan (Sundell & Olsson, 2017). Ideally, program developers design such change strategies, or interventions, based on a problem theory (Bickman, 1987; Fraser et al., 2009). A problem theory is grown out of what is known from basic and applied research about the problem at hand, its associated risk and protective factors, as well as the populations that are most vulnerable to the problem (Fraser & Galinsky, 2010; Fraser et al., 2009; Mrazek & Haggerty, 1994; O'Connell et al., 2009). From this problem theory, developers can build their interventions by specifying the ways in which the planned intervention is expected to impact the mediating processes identified in the problem theory through specific program targets, activities and agents and thus achieve intended outcomes (i.e., how and why change is expected to occur). This description of how an intervention is intended to work to bring about change and achieve intended outcomes is called a program theory (Fraser et al., 2009).
Despite the literature describing the theoretically ideal circumstances and processes through which interventions should be developed, investigations and reviews of the effectiveness of interventions still point to shortcomings in the extent to which interventions are developed and described (e.g., Glasziou et al., 2008; Maden et al., 2017). This complicates our ability to gain sound theoretical and empirical insight into how intentional change strategies work in practice settings. Reconstructing the program theory is a solution to this problem. Program theory reconstruction refers to the systematic articulation and documentation of the assumptions underlying interventions already in use in practice settings (Leeuw, 2003). In essence, reconstructing the program theory is done to understand the underpinning ideas and workings of an intervention, or to understand what program components there are and why these components are perceived as being important, when a systematic description of these does not exist. The term “reconstruction” is not used to suggest that the program theory is damaged or needs to be changed in any way. Instead, “reconstruction” is used to denote that information from many different sources is gathered and analyzed to get a complete and accurate picture of the program's “whats,” “whys,” and “hows” after the intervention has already been implemented in practice. The purpose of this commentary is to offer a comprehensive and accessible guide to program theory reconstruction in research on social interventions and to provide arguments as to how program theory reconstruction can be used to benefit intervention studies. Thus, we provide social intervention researchers, practitioners and students who are unfamiliar with program theory reconstruction with a brief but thorough guide to the method by outlining its theory, method, and importance to the field. Our hope is that it appeals to a broad audience of researchers, practitioners and students interested in social intervention in practice settings.
In this commentary, we argue that program theory reconstruction offers social intervention researchers an important tool in their intervention development and research efforts. We summarize what program theory is and provide methodological advice on the steps necessary to reconstruct a program theory using our experiences from two projects as examples. Finally, we discuss how the reconstructed program theories can be used to inform social intervention research. Here, we weave in guidance from the literature, lessons learned from our own research projects and decisions that need to be made when undertaking a program theory reconstruction within the context of social intervention research. Our belief is that more attention to program theory reconstruction in intervention development and research attempts, will help move the entire field of intervention science forward as we could gain more knowledge regarding the activities driving the interventions studied and the mechanisms driving their outcomes. With this knowledge, intervention developers are likely to be able to design and develop more efficient and effective interventions and researchers are likely to be able to design the most appropriate studies of such interventions. A glossary of terms (Table 1) that provides further guidance to the reader is provided at the end of this commentary.
Glossary.
Program Theory Reconstruction's Importance for the Science of Social Intervention
Researchers are common developers of interventions that they later implement and study in their own, controlled research (Gorman & Conde, 2007). In such cases, it is natural for researchers to design interventions based on social science theory and empirical research. In other words, they have a clear theoretical idea about how the intervention is supposed to bring about change and lead to desired outcomes. Based on this idea, or program theory, researchers can design their studies.
At the same time, it is not uncommon for researchers to study interventions that have already been developed, are in use in policy or practice, and are integrated into organizational and contextual settings. In these cases, community-based interventions have been developed by practitioners because of a perceived local need and are grounded and used in practice. In contrast to researcher-driven approaches to intervention development, in practitioner-driven approaches (Chen & Garbe, 2011) there might not be an explicit underlying program theory that informed the program development. Instead, this underlying theory is implicit in the work being done between professionals and clients. Consequently, when researchers have the opportunity to study an intervention developed in practice, a first important step is to explore, identify, describe and specify—that is, reconstruct—the program theory (Pawson, 2013). If this step is overlooked, researchers might not have all information needed to know what type of intervention they are studying, what theoretical grounding the intervention has, and how to best design subsequent research on the intervention at hand (e.g., develop data collection tools, choose appropriate research designs, choose appropriate measurement instruments). Ultimately, this could hamper much-needed scientific progress in the field of social intervention research.
Understanding how and why a social intervention is intended to work is needed to study the program adequately (Rey et al., 2011). In addition, guidance regarding program theory development is almost exclusively confined to the program evaluation literature (Chen, 1990; Funnell & Rogers, 2011; Rossi et al., 2004), with little mention outside of this field. Importantly, researchers may not report their program theory reconstruction work in their publications (Shearn et al., 2017). Consequently, there is a lack of published methodological discussions of program theory reconstruction in the social science literature (Flynn et al., 2020). This situation may leave scientists, practitioners, and students unfamiliar with the methods of program theory reconstruction, the benefits of program theory development to intervention science, and the choices one may face in the reconstruction of program theories.
This commentary is born out of our experience in conducting intervention research over the past 20 years. During this time we have witnessed an increased interest in the science of social intervention (Sundell & Olsson, 2017) through for example a growth in an interest in how we might use research to inform intervention efforts (Mrazek & Haggerty, 1994; O'Connell et al., 2009), how we might systematically develop interventions (Fraser & Galinsky, 2010; Fraser et al., 2009; O'Cathain et al., 2019) and how we might assess the outcomes of interventions (Flay et al., 2005; Gottfredson et al., 2015). Recently, we were involved in two projects that are used in this commentary as examples. The first was a researcher-driven project which aimed to study the theories underlying several active labor market programs (ALMPs) developed in practice or imported and implemented in new contexts. An ALMP is an intervention that aims to move individuals who are unemployed into employment (i.e., work training, subsidized work, job-search assistance, etc.; Brown & Koettl, 2015; Card et al., 2018). We refer to this project in the text as Pathways (Olsson et al., 2020; Starke & Hollertz, 2022). Briefly, Pathways is a quasi- (natural) experiment, which aims to study the effects of a group of municipally provided ALMPs with ALMPs provided by non-governmental organizations (NGOs). Although these interventions are actively recruiting and working with unemployed participants, none of the interventions had a specified program theory at the start of this project. The second project was practice-driven, and practitioners engaged researchers to reconstruct the program theory for a practitioner-developed intervention called Self Assured Parenting (Skoog et al., 2022). We refer to this project as SAP in the text. Briefly, SAP targets immigrant parents living in deprived areas, who are worried that their children (age 12–18) engage in harmful behavior or will be exposed to harmful environments (i.e., substance use, crime, and extremism).
With these two projects as examples along with the theoretical and methodological guidance derived from the current literature, we aim to fill what we as social intervention researchers have experienced as a gap in the literature—the absence of a paper which adequately outlines the theory and application of program theory reconstruction, and one which does so in a way accessible to those not particularly familiar with the approach. There are several papers on program theory development in the literature (Shearn et al., 2017), but to our knowledge none of these provide an accessible guide to program theory reconstruction in research on social interventions developed in practice (see however Leeuw, 2003 for examples of approaches to program theory reconstruction).
Program Theory
Program theories (e.g., Chen, 1989, 1990; Rogers et al., 2000; Weiss, 1972, 1995, 1997) are broad attempts to organize and explain interventions. They are conceptual frameworks consisting of symbolic representations of the mechanisms, structures, and causal processes presumed to underlie the relationships between them (Marx & Godson, 1976; Worthen, 1996). Thus, program theories are made up of a consistent group of statements that present a systematic view of the intervention under consideration. Here, the statements identify, define, and describe the phenomena involved in each social intervention, and specify the nature of their interrelationships. It should be noted that program theories do not attempt to be, nor should they be considered, formal social science theories. That is, they are not explanatory frameworks for the social phenomena of social intervention drawn from social science (Donaldson & Lipsey, 2006). Instead, program theories (Rogers et al., 2000) specify the causal processes underlying an intervention's expected, intended, or unintended effects. Closely related concepts within the program evaluation literature include program specification (Savas et al., 1998; Soloman, 2002), logic modeling (Renger & Titcomb, 2002), causal modeling (Pawson & Tilley, 1997), causal mapping (e.g., Huff, 1990), program modeling (e.g., Kirkpatrick, 1994), pattern matching (Trochim, 1985, 1989), program impact theory (Rossi et al., 2004), theory of change (Rossi et al., 2004), action theory (Rossi et al., 2004), outcome line (Rossi et al., 2004), and outcome chain (Funnell & Rogers, 2011). Although there is general confusion about the terminology associated with program theory (Shearn et al., 2017), one common assumption is that the processes involved in a social intervention are guided by explicit or implicit theories about how and why the program will work and how change will occur (Davidoff et al., 2015). Often, program theories are presented as logic models (see Figure 1). In its basic form, a logic model includes four components: an intervention's inputs, activities or processes, outputs and outcomes (e.g., Funnell & Rogers, 2011; Rossi et al., 2004). As can be seen in this basic structure, a program theory is necessarily a simplified description of the intervention, its most salient components and their relationships. Therefore, choices need to be made throughout the reconstruction process regarding which components to include and for what purpose. The following sections describe, and present examples of these choices based on our own work with program theory reconstruction.
Four-component logic model. Note. Figure 1 presents a logic model in its most basic form. Often called a four-component logic model. The four-component logic model is a conceptual framework and details the inputs necessary to support the activities which are undertaken within the context of a given intervention, the outputs the activities are intended to generate, and the outcomes related to these outputs. A logic model can include a longer chain of outcomes (e.g., proximal, intermediate, and distal) such as those included in Figure 4. A logic model can also include other important aspects of an intervention such as external or contextual factors.
Planning for the Approach to Program Theory Reconstruction
Program theory reconstruction, when undertaken within the context of social intervention research, must be seen as a scientific activity and, therefore, it is imperative that basic principles of scientific method are adhered to (e.g., adherence to a systematic, transparent, and replicable method). As such, program theory reconstruction involves several choices which should be made explicit. These choices include questions regarding the specific aspects of the program to be investigated and highlighted in the program theory; the choice of stakeholders to include in the reconstruction process; the scientific approach to program theory reconstruction; and the specific focus on identifying the underlying mechanisms driving change within the program. In practice, researchers should consider these questions before collection and analysis of data, and there needs to be an ongoing reflexive dialogue on the part of the researcher or researchers with regards to these issues, throughout the analytic process.
What Aspects of the Program Theory Should be Investigated and Highlighted?
At the core of social intervention research is the idea that through the development and implementation of individual, group, organizational, local, national or societal strategies, social intervention practice can make a difference (Fraser et al., 2009). An intervention's program theory should highlight the process through which intervention activities contribute to a chain of events which result in the intended or observed outcomes of the intervention. That is how an intervention works. The aspects of the program theory that should be investigated and highlighted are those which set in motion this chain of events, and which contribute to the change process or sequential chain of events which result in the intervention's outcomes. These aspects are also referred to as an intervention's core components. For example, Figure 2 depicts a section of a program theory from the Pathways project and highlights the core component of participation in subsidized public employment training and how that core component impacts a chain of events for the participant through to the intervention's intended outcome of obtaining gainful employment. Program theories can be simple as in the example given here or more complex as in the example given in Figure 3. Ultimately what guides the choice of which aspects of the program theory to investigate and highlight will rest on the purpose of the reconstruction process. For example, program theories may be reconstructed in order to aid in the design or refinement of interventions (Fraser et al., 2009; Rothman & Thomas, 1994), aid in conceptualizing, planning, managing, and communicating with others about an intervention (c.f., Newcomer et al., 2015), or aid in guiding, designing, and interpreting results from program evaluations (e.g., Newcomer et al., 2015; Shaw et al., 2006).
Simplified program theory from the Pathways project. Note. Figure 2 presents a simplified section of a program theory from the Pathways project. Here a core component of the intervention in this example is the receipt of and participation in subsidized public employment training. It is theorized in this example, that through this receipt and participation, participants will develop an increased understanding of the work environment and the demands of working as well as develop more positive relationships with employers. This in turn will result in obtaining employment in the open labor market.
Logic model of Self-Assured Parenting program. Note. Figure 3 presents a logic model of Self-Assured Parenting program. The model shows a chain of outputs, inputs, and preliminary outcomes through which an expected distal outcome is reached. Grounded in context and theory, the model shows initial input (e.g., relaxed setting) critical for the staff to be able to provide activities (e.g., role play) included in the program. Further, the models describe the number of outputs provided (e.g., nine meetings) for the primary or proximal outcomes to be reached (e.g., parents recognizing triggers for conflicts). In the next step, mediators (e.g., parents having more confidence in their parenting skills) that are driving the effect from the previous steps to the final or distal outcome (e.g., showing indicators of stronger parent-child relationship) are presented.
Which Stakeholders Should be Consulted?
Successful program theory reconstruction is a collaborative effort and depending on which aspects of the intervention are in focus for the reconstruction, individuals both internal and external to organizations delivering the intervention should be involved in the process. This is because the delivery of interventions is impacted not only by processes within a given organization (e.g., referral processes between organizational departments) but also by the processes between the organization providing the intervention and actors outside of the organization (e.g., referral processes between organizations) or between actors external to the organization (e.g., government funding processes). Program stakeholders can be categorized into different groups (Guise et al., 2013). Service user groups (e.g., clients, patients, consumers, caregivers) refer to groups that are representing individuals who have a specific problem or need, and/or use services (e.g., social services). Professionals (e.g., clinicians, social workers, teachers) are individuals that interact directly with service users to deliver an intervention. Other groups of stakeholders include for example, policy makers (e.g., governmental organizations), program managers, researchers, program or research funders, and other community members. It is important to keep in mind that different stakeholders may have different, and maybe, even conflicting perspectives regarding the intervention under consideration (e.g., Olofsson et al., 2016). Therefore, choosing which stakeholders to involve in the reconstruction process depends on which aspects of the program theory are to be investigated and highlighted as well as the purpose and aim of the reconstruction process. Regardless of this choice, one should have an a priori understanding of what each stakeholder group will contribute to the process (e.g., component identification, prioritization), how many stakeholders from each group will contribute, and which specifically identified individuals will be approached for contribution.
In the Pathways project for example, we chose to involve direct service staff and frontline program managers in the reconstruction process. This was due to our four primary goals with the reconstruction process. First, we wanted to be able to compare the ALMPs included in our study with each other to assess the extent to which they were unique or similar interventions in specific regard to program components such as goals, program targets, and activities. Second, we were interested in identifying common and unique implementation components across programs. Third, we were interested in assessing the extent to which the included programs were based, implicitly or explicitly, on current scholarly theories of behavior change in social work generally (e.g., Michie et al., 2014; Payne, 2014) and employment programs specifically (e.g., Hong, 2009, 2013; Hong et al., 2019). Fourth, we wanted to uncover the resources necessary to provide the programs (e.g., staff characteristics, materials, and equipment) as a basis for economic analysis. As such, we were not primarily interested in contextual aspects of the program theory but instead in the perspectives of stakeholders most familiar with direct client work and frontline implementation issues related to the interventions under investigation.
Inductive, Deductive, or Combined Approach?
The inductive approach to program theory reconstruction involves constructing the program theory by making observations of the program in everyday operation. This approach is based on an understanding that when implemented in everyday practice, interventions may deviate from developers descriptions of the intervention (Funnell & Rogers, 2011). In addition, one may use several methods to supplement observation of the program such as interviews with staff, clients, or other stakeholders. Here, the goal is to elicit what actually happens in the course of delivering an intervention (Funnell & Rogers, 2011). This approach relies heavily on field work (Oosthuizen & Louw, 2013) and might be especially useful when one is interested in studying mediating mechanisms or developing scientific theory. For example, as a starting point for program theory reconstruction in the SAP project, investigators conducted an observation of one session of the program. This was necessary to gain an understanding of the program and to uncover the social reality of the program. The observation took two and a half hours, including an informal conversation with clients and staff before and after the session. Observing the activities, discussions and roleplays that took place during the observation session helped investigators see the program through the eyes of both participants and providers of the program. As a complement to the observation, investigators also conducted interviews with stakeholders from the community, staff, and clients of SAP.
In contrast to inductive approaches to program theory reconstruction, a deductive approach infers program theory from informal and formal documentation, wider research literature, and professional experience regarding the problem at hand, its causes and consequences, and effective practices related to the intervention (Funnell & Rogers, 2011). Even if deductive approaches are most often connected to the development of a new intervention, this approach can be especially useful for identifying an intervention's explicit program theory and as such may help inform research that intends to investigate intervention or implementation processes. As program documentation can omit information and thus program theory reconstruction from documentation is dependent upon the inferences or assumptions of investigators, stakeholders should always be consulted to validate these inferences and assumptions so as not to incorporate inaccuracies in the reconstruction process. Another important source of information when taking a deductive approach to program theory reconstruction is by drawing upon current scholarly theories in the specific content area of the program under study. For example, in the area of labor market programming, the theory of Psychological Self Sufficiency (e.g., Hong, 2009, 2013; Hong et al., 2019) provides a clear theoretical account of how individuals who have experience of long-term unemployment move closer to the labor market. This process is achieved according to this scholarly theory through processes of building employment-related hope and removal of employment-related barriers (e.g., Hong, 2009, 2013; Hong et al., 2018; Hong et al., 2019). In development of the program theories in the Pathways project, investigators were particularly interested in how these processes were attended to in the program theories of the individual programs under study.
It should be noted that the above approaches can be combined (Funnell & Rogers, 2011). Although the literature on this approach is scant, it has been suggested that the inductive and deductive approach may be combined sequentially, iteratively, or interactively and may be useful when information is missing or divergent or when a richer more nuanced program theory is desired.
What Counts as a Mechanism in Program Theory Reconstruction?
A mechanism in program theory reconstruction is a process by which something takes place or is brought about. Such a process could be critical in terms of explaining the anticipated change process impacted by the intervention. The understanding and importance of a mechanism changes depending on the discipline and perspective within which one works. At their foundation, mechanisms link cause and effect and are critical in the search for and understanding of deeper and refined explanations of social phenomena, including social interventions. As such, it is not enough for a program theory to describe activities that are expected to lead to outcomes. Instead, the theory must link these activities to outcomes through their causal processes (Astbury & Leeuw, 2000). Mechanisms can be understood as mediating and moderating processes (Chen, 2005) or ideas and opportunities (Pawson & Tilley, 1997) which, if and when appropriately introduced to individuals and groups in the appropriate social and contextual conditions, lead to successful interventions. Kazdin (2014) describes a mechanism as more specific than a mediating process in that a mechanism explains how an intervention translates into precise events that lead to changes in outcome. Put simply, mechanisms answer the questions of how and why change is supposed to occur.
Steps in Program Theory Reconstruction
Program theory reconstruction has been described in the literature as involving several steps. Although current descriptions of program theory reconstruction present a varied number of steps in the process (Funnell & Rogers, 2011; Leeuw, 2003) we have chosen to present the reconstruction process in four distinct steps based on our own experiences with reconstructing program theories in practice settings.
Step 1: Familiarize Oneself with the Program
The first step in the program theory reconstruction process is to familiarize oneself with the program and engage stakeholders. This can be done through informal discussion and relationship building with program stakeholders, and through a review of program materials and documents.
Informal Discussion and Relationship Building with Program Stakeholders
Informal discussion and relationship building is essential for not only clarifying and agreeing on roles and responsibilities in the program theory reconstruction process, but also as a means for developing buy-in and trust and maintaining credibility. Factors that encourage successful stakeholder engagement include: including stakeholders early in the process and clearly detailing roles, responsibilities, and expectations (e.g., time commitment, types of activities), maintaining an ongoing relationship throughout the reconstruction process by for example keeping stakeholders informed, providing opportunities for stakeholders to ask questions throughout the process and providing stakeholders with information which may increase their knowledge, ease, or comfort with a given topic (Guise et al., 2013). If stakeholder needs are not met early in the process, barriers to engaging stakeholders and gaining access to information may develop later in the process. Informal discussion with stakeholders, therefore, is a means to engage and maintain stakeholder relationships throughout the program reconstruction process as well as a means for gaining informal information about the intervention being studied.
Review of Program Materials and Organizational Documents
Reviewing written program materials and program documents is an unobtrusive means of gaining insight into the intervention under study (Rossi et al., 2004) and is not only a good source of background information, but also gives insight into program factors that may not be directly observable (Bowen, 2009). The review should not be strictly limited to formal materials such as manuals and published articles. Rather, the review, if possible, should cover a diverse set of written documents and sources, including information leaflets, chapters, webpages related to the program, etc. This is the best way to help develop an understanding of the history, philosophy, and context within which the intervention is operating.
In most organizations, governmental and non-governmental, documents are produced which contain important information about organizational and programmatic goals, target groups and activities of interventions that take place within the organizations. Yearly reports, evaluations, manuals, strategic plans, budgets, minutes, are all examples of documents that can inform the understanding and reconstruction of the program theory. It can be time consuming to review such documents. Careful planning is therefore advisable. In addition, access to such documents often requires the support and assistance of relevant stakeholders, such as program managers, controllers, program staff or other stakeholders (Bowen, 2009). The stakeholders could in many cases be regarded as gatekeepers to the information that is necessary to reconstruct the program theory (Davies & Peters, 2014; Singh & Wassenaar, 2016). Therefore, it might take some effort to get the necessary information. The stakeholders are likely busy and helping researchers to find documents might not be a priority. Despite these obstacles, reviewing organization documents can prove to be highly important to understand the ideas behind the program. Before proceeding to the next steps in the program theory reconstruction process, it is important to summarize the information gained and to try to identify remaining gaps of knowledge and information in relation to understanding the program theory. Specifically, one should examine if there are answers to the questions of what (does the intervention do), how (does the intervention do it), and why (does the intervention do it)? The unanswered questions can guide the collection of information in the forthcoming steps of the processes.
Step 2: Collect Program Specific Data
Following Step 1, the investigator should have a general understanding of the program as well as a general understanding of the types of information that are necessary to reconstruct the program theory. Step 2 of the reconstruction process is to collect more in-depth and complete information that will allow the investigator to expand the information already received. Here we highlight three methods available for the collection of program-specific data: document analysis, stakeholder interviews, and program observation.
Document Analysis
Document analysis is a systematic procedure for reviewing documents to gain understanding and develop empirical knowledge. Whereas the review of documents described in Step 1 above can be understood as a superficial examination (e.g., skimming), document analysis in Step 2 involves a thorough examination and interpretation of the documents available. Here, care must be taken to assess the added value of a document analysis following the examination conducted in Step 1. The methodologic procedure for document analysis entails identifying, finding, selecting, appraising, and synthesizing data contained in identified documents. Through these systematic activities, the analysis yields data (e.g., excerpts) that are then categorized into the program theory through what has been described as a combination of elements of content analysis (Hsieh & Shannon, 2018, 2005) and thematic analysis (Braun & Clarke, 2008; Fereday & Muir-Cochrane, 2006). Documents can provide data on the context within which interventions operate, aid in developing an understanding of the historical roots of a given intervention and highlight conditions that may impinge on the intervention. In addition, document analysis can be a beneficial approach to tracking change and development within a given intervention. Documents can also be used as a source for uncovering areas that need additional investigation, verify or corroborate findings from other sources (Bowen, 2009). In any document analysis it is important to keep in mind that documents are likely incomplete and, depending on the level of detail one is interested in their program theory reconstruction efforts, supplemental data collection may be required.
The analysis of documents can form the basis for developing data collection tools by informing the types of questions which need to be asked in individual or group interviews as well as highlighting the types of situations which need to be observed in order to gain a full picture of the intervention being studied (Bowen, 2009). In some cases, the program theory might be explicitly described in program materials and documents. In such cases, and particularly if resources are limited, there might not be a need to collect additional information in order to further develop the program theory.
Stakeholder Interviews
Stakeholder interviews can be conducted individually or in groups. Common to both approaches is the development of a series of questions that can be asked in order to elicit stakeholders’ perspectives and understanding of the problems addressed by the program, the causes and consequences of the identified problems, how the program intends to address the problem, and any other pertinent information (Funnell & Rogers, 2011). An interview guide should be developed and used to structure the interviews and to make sure all relevant aspects of the program are uncovered. The interview guide can vary in degree of structure; from open to semi-structured to structured. Important in choosing which approach to use and which questions to ask is the goal of eliciting an if-then story (Funnell & Rogers, 2011; Leeuw, 2003) of what, why and how the program and specific program components lead to outcomes for clients and others. By drawing out information about the if-then relationships within and between program components, investigators learn important information regarding the program's change mechanisms, stakeholder assumptions regarding program operations, and other factors (e.g., external conditions) that are expected to impact program operations.
Program Observation
Observing program operations in action is an approach to collecting data on what happens in practice as opposed to data regarding how a program is intended to work or how stakeholders would like a program to work. As such, observation is an important method for eliciting similarities and differences between data obtained from other methods. Observation may be direct or indirect (Ciesielska et al., 2018). Direct observation is when the investigator watches events happening in real time. Indirect observation is remote and relies on recordings of past events. As it is not possible to observe everything at once it is important to decide a priori what the main goal of the observation session(s) will be. In addition, taking regular notes helps avoid problems that may be connected to memory loss or reinterpretation of events. It is important to carefully consider possible (research) ethical aspects of observing intervention leaders and participants or performing any of the other steps in the program theory reconstruction process.
Step 3: Translate Collected Data into Program Theories
Phase 3 is the most important and central of the four proposed phases. It takes effort to identify all aspects of the program theory. It is important to do so accurately as the drivers of change might lie in any one of the components of the program, or even in a combination of components. Outcome chains, program matrices, and logic models are helpful tools for sorting and presenting information collected during steps 1 and 2.
Outcome Chains
Using the information gained from the activities described thus far, investigators can begin to sort their collected information into preliminary outcome chains (Figure 4). An outcome chain presents the intervention investigated as a linear process by which a series of results is presented chronologically (Funnell & Rogers, 2011). This is done by focusing specifically on what the program attempts to change (i.e., intended, observable characteristics of the target population or social condition) and is thus not primarily concerned with the program activities (Rossi et al., 2004). Funnell and Rogers (2011) suggest four distinct steps (five with validation, which is described in more detail below) in the development of an outcome chain. The first step in developing an outcome chain is to prepare a list of possible outcomes. This can be done by analyzing the data collected in Phase 2 above and searching specifically for those factors which the program attempts to change. The second step is to cluster the identified outcomes into groups to reduce the amount of information. This clustering can be organized in several different ways depending on the approach taken to the program theory reconstruction and the specific content area within which the program operates. For example, one might want to cluster outcomes based on the scholarly theory in the subject area (e.g., self-determination theory, social cognitive theory, resilience theory) or on more general ideas about social intervention such as by clustering changes in participant knowledge, skills, abilities, experiences, and behaviors. The goal here is to sort the outcomes in a way that is meaningful and easy to understand while not losing any substantive information on the program at hand. A common approach to placing outcomes in temporal space is to order outcomes into a longitudinal chain. This is done in the third step by arranging the clusters through chains of if—then statements (Funnell & Rogers, 2011; Leeuw, 2003). If-then statements are conditional statements which connect outcomes in a logical order. For example, in the Pathways project (see Figure 2) the program theory posits: if participants gain work related knowledge from a specific workplace, then participants will have an increased ability to gain employment in the open labor market. Similarly, the SAP program theory (see Figure 3) posits: if parents recognize triggers for conflicts, then parents can adapt their level of control and the demands they place on their child. After organizing outcomes in logical if—then statements, feedback loops (Step 4) can be identified and minimized. Feedback loops are paths in the process that strengthen or weaken earlier processes in the model. A feedback loop may provide a more accurate picture of the change process being described but an abundance of feedback loops in any given program theory can impact the usefulness of the theory and therefore attempts should be made to minimize feedback loops within the program theory.
Example of a simplified outcome chain from the pathways to employment project. Note. Figure 4 presents an outcome chain developed through interviews with program stakeholders in the Pathways to Employment Pathways project. The outcome chain has been simplified here for demonstration purposes. This project took an inductive approach to program theory development and the outcome chain presented above describes how participants are expected to move through proximal and intermediate outcomes to achieve the intervention's more distal outcome of an increased ability of participants to gain employment in the open labor market. Consequent research design decisions can leverage this newfound understanding of this process and the highlighted variables can be isolated and measured to investigate mediating relationships and/or be compared to current social science theory and thus be used to develop our (1) understanding of social intervention specifically and (2) social science theory more generally.
Program Theory Matrices
Table 2 shows a section of a program theory matrix from the Pathways project. A program theory matrix as described in the literature consists of seven components displayed in matrix form (Funnell, 2000; Funnell & Rogers, 2011). The first column of the matrix consists of the sequenced hierarchy of intended outcomes, that is the outcome chain. The outcomes in the chain are placed in ascending order with the most proximal outcome placed in the bottom row of the matrix and the final (or most distal) outcome in the top row. Each outcome in the outcome chain is then linked to the following program characteristics (adapted from Funnell & Rogers, 2011) which are contained in the columns to the right of the outcome chain (see the top row in Table 2):
Success criteria for the outcome which includes attributes, standards or other comparisons. Assumptions about program factors that affect the outcome which are largely within the control of the program. Assumptions about nonprogram factors that affect the outcomes that are largely outside of the control of the program. Outputs and materials produced by the program. Activities or processes undertaken and used by the program; and Resources or inputs including those from outside of the program.
Example of a Program Theory Matrix From the Pathways Project.
Note. Table 2 provides a section of a program theory matrix from the Pathways project. The table illustrates how information from stakeholder interviews was sorted, in line with Funnel and Rogers (2011). Success criteria merely states what is needed to be able to deliver the program at all (i.e., participants who are in specific life-situations who have a referral to the program). The success criteria are thus not directly linked to the outcome chain; but are rather to be understood as a prerequisite to be able to deliver the program at all. Program factors form a basis of the program; indicating on a more general level what the program consists of. Non-program factors are external factors affecting the outcome of the program, but that cannot be controlled by the program (or the organization delivering the program). Outputs are here understood as the general undertakings carried out within the program. In activities, processes, and principles the activities that are carried out in the program are specified, including aspects of intensity and quality. (It should be stated however, that there is no clear-cut line between on the one hand outputs, and on the other hand activities/processes/principles. As a guiding principle however, the latter describes more in depth the activities and the guiding principles framing the activities carried out within the program). Resources and inputs refer to the resources that are available for the delivery of the program, such as staff with certain qualifications but can refer to technical assets and localities where the program is delivered.
The program theory matrix is a tool for helping the investigator and stakeholders think about the various program characteristics that can be involved in achieving program goals and is typically not seen as the final product in a program theory reconstruction exercise.
Logic Models
Researchers often present the identified cause-and-effect relationships in logic models (see Figures 1, 3, and 4). Figure 1 presents a logic model that through different components illustrates processes in a hypothetical program theory. For example, the initial input component presented in Figure 1 describes the success criteria for the outcome which includes the attributes as well as the resources (such as facility, staff) used by the program to provide its activities. The activities are the processes undertaken and used by the program which include services provided by the program, such as meetings and coaching. The output describes, in quantifiable terms, the number of services the program provides, such as the number of coaching sessions in an intervention. Although the number of outcome columns may vary, it is common to include the proximal outcome, the intermediate outcome, and the distal outcome. The proximal outcome is the immediate outcome that happens because of the activities provided in the program. For example, the proximal outcome from several coaching sessions may include individuals having higher self-confidence. The intermediate outcome is a behavioral change resulting from reaching a proximal outcome in the program, which in turn works as a mechanism that links the proximal and distal outcome. For example, having higher self-confidence may lead to individuals searching more job opportunities, which in turn would make reaching the higher employment among individuals possible. Finally, the distal outcome is the final stage of the logic model representing the final goal with the program theory, for example increasing employment among individuals. Thus, logic models, as part of the program theory, help to illustrate the causal processes underlying an intervention's effects.
Step 4: Validation and Calibration of Program Theories
An important step before finalizing any program theory reconstruction exercise is validation and calibration of the proposed program theories. At the heart of a program theory is the definition and causal ordering of the mechanisms that mediate between the delivery and receipt of program activities and the chain of outcomes that these activities set in motion (Weiss, 1998). As such, the final reconstructed program theory should be assessed as being plausible (Funnell & Rogers, 2011) and valid (Brousselle & Champagne, 2011). In addition, both the reconstructed program theory's internal and external validity should be assessed, and calibration of the program theory should be undertaken before the reconstruction process is finalized. This can be done by assessing the reconstruction against evidence regarding effective practices for addressing the problem the program is trying to ameliorate, assessing the reconstruction considering accepted scholarly theories of change, and/or by engaging stakeholders in critique and review of the reconstructed theories. These three activities then provide the basis for calibration and finalization of the reconstructed theories
Using Program Theory to Inform Social Intervention Research
Social intervention research is concerned with how we might explore, develop, implement, test and fine tune interventions to support defined populations across the life span. This work includes both preventive and promotive interventions which may be implemented in a range of contexts (e.g., school, social services) and target problems at the individual, group, or societal level. Understanding the theoretical foundation of such efforts is imperative to the science of social intervention and can aid social intervention researchers in several ways. First, intervention research centers on both program outcomes and hypothesized change processes. A major thrust of program theory reconstruction is to highlight and describe these processes. Results from controlled research can then be assessed considering the reconstructed program theory in order to point to areas where components may need to be adjusted or changed in order to achieve intended outcomes. Second, from an intervention research planning perspective, reconstructing the program theory can aid the researcher in designing their research as well as choosing appropriate measures as it systematically highlights links between program content and program outcomes. Third, program theory reconstruction can help illuminate hypothesized links between program activities and mediating mechanisms which can be assessed in controlled research. Fourth, interventions can fail due to either program theory failure or implementation failure. Reconstructing an intervention's program theory prior to engaging in intervention research can help the investigator plan for including both aspects in their controlled research to gain a better understanding of which aspects of the intervention have worked well and which have not worked so that any subsequent development activities can be focused on the appropriate components.
Conclusions and Future Directions
Program theory reconstruction is a step-by-step approach for students, practitioners, and researchers to better understand interventions used in everyday practice. Program theory reconstruction provides a foundation for understanding how interventions are intended to work to bring about desired outcomes in specific populations in specific contexts and can therefore be used to design systematic research on these well-defined interventions. Future work regarding program theory reconstruction should attempt to develop an understanding of how different approaches to the reconstruction process may lead to a more refined program theory and better-designed research. This would be accomplished through the systematic empirical investigation of how program theory reconstruction impacts the quality and design of intervention research.
Footnotes
Acknowledgments
We would like to thank members of the Intervention and Implementation Research Group (IIF) at the University of Gothenburg's Department of Social Work for thoughtful comments on earlier versions of this manuscript.
Author Note
This paper was submitted as a contribution to the special issue of Research on Social Work Practice “Program Theory in Social Interventions Developed in Practice Settings,” guest edited by Tina M. Olsson and Therése Skoog.
Authors’ Contributions
TO: conceptualization, formulation and evolution of commentary goals and aims, writing original draft, review and editing, supervision/oversight, management, and coordination; SK: evolution of commentary goals and aims, writing original draft and review; KH: writing original draft and review; MS: literature review, writing original draft and review; TS: formulation and evolution of commentary goals and aims, writing original draft and review. All authors have approved the final version of this manuscript.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Pathways to Employment is funded by the Swedish Research Council for Health, Working Life and Welfare (FORTE). Grant number 2018–01324. Self-Assured Parenting is funded by FORTE. Grant number 2020–05–20. The funding body was not in any way involved in the design, collection, analysis, or interpretation of data, nor in writing the current manuscript.
