Abstract
The increasing ubiquity of individualized digital devices expands the scope of Information Systems (IS) research and offers novel opportunities for investigating and influencing individuals’ intentions, cognitions, and behavior. Based on these technological advancements, behavioral science researchers have started to develop experimental methods that allow for rigorously investigating dynamic phenomena in our digitalized world. We introduce an experimental design, called a micro-randomized trial (MRT), and propose that it is widely applicable in IS research for examining complex and dynamic IS research phenomena. MRTs allow for analyzing causalities and testing theories with dynamic components by considering how time-varying personal and contextual factors influence an experimental treatment’s efficacy over time. The resulting insights can be leveraged to refine theories, improve IS design, influence users’ cognitions and intentions, and steer user behavior. This paper motivates the relevance of the MRT experimental design for IS research, equips interested IS researchers with the required knowledge to conduct MRTs, and demonstrates how MRTs can be deployed to advance IS knowledge by achieving research goals that until now, given the standard IS method assortment, could not be properly addressed.
Introduction
The constant increase in digital products provided through individualized digital devices expands the scope of Information Systems (IS) research and practice. Research streams from various fields focus on how the ubiquity of individualized digital devices can be leveraged to influence individuals’ intentions, cognition, and behavior (Hekler et al., 2016; Ho et al., 2011; Schneider et al., 2018; Söllner et al., 2022). This ubiquity of digital devices offers novel opportunities to capture individuals’ dynamic internal and contextual states, thereby offering the potential to advance our understanding of how individuals’ intentions, cognitions, and behavior can be explained, shaped, and changed. To accomplish this potential, researchers need to be equipped with appropriate methods that can be used to generate reliable theoretical knowledge that captures the dynamism of human behavior in our digitalized world (Blohm et al., 2022).
Neighboring disciplines of IS, such as psychology, economics, and marketing, which specifically study human behavior, place significant emphasis on experiments for rigorous empirical inquiry (Falk and Heckman, 2009; Gupta et al., 2018; Viglia et al., 2021). Experiments provide the strongest foundation for analyzing causalities and testing theories (Karahanna et al., 2018; Webster and Sell, 2014). Hence, researchers from behavioral science have started to develop experimental methods that build on ubiquitous technology and allow dynamic phenomena in our digitalized world to be experimentally investigated (Klasnja et al., 2015; Nahum-Shani et al., 2018). Yet, in the IS discipline, although experimental research has become significantly more popular in recent years, experimental designs that can capture dynamic relationships remain underutilized (Cahenzli et al., 2021; Fink, 2022).
In recognizing the necessity of acknowledging the complex and dynamic nature of IS phenomena, we present a novel experimental design that stems from behavioral science, called micro-randomized trials (MRTs). The MRT experimental design was developed specifically to design and test just-in-time adaptive interventions (JITAIs) (Klasnja et al., 2015).
JITAIs are instant messages sent to users via technologies that aim to send the right type and amount of support at the right time, that is, when a user is most receptive to the content of the message (Nahum-Shani et al., 2018). Because user receptivity can vary depending on personal and contextual circumstances, JITAIs are inherently dynamic. Consequently, the goal of MRTs is to examine causal relationships in dynamic contexts. For example, with an MRT we can experimentally examine whether a specific JITAI is more effective if it is sent while a user is at home or at work. Similarly, we can examine the JITAI’s effectiveness depending on a user’s affective state (e.g., relaxed or stressed). We can also assess how multiple JITAIs dynamically influence a selected behavioral outcome over time. Thus, in an MRT, Information Technology (IT) allows researchers to both capture dynamic user contexts and states (e.g., physiological measures, location, and behavior) and to intervene to change users’ intentions, cognitions, and behaviors (e.g., through adaptive interventions).
Accordingly, the distinguishing factor of this design is that it allows for the dynamic evaluation of treatment effects by considering how both contextual factors and the individual’s state influence the treatment effect over time. This is achieved through sequential randomization to study conditions, in which users repeatedly receive pre-specified treatments while their context and responses are simultaneously observed and recorded. This experimental setup facilitates mixed-subject comparisons, which enable the dynamic longitudinal investigation of individuals’ receptivity to various treatments under specific conditions over time. Thus, MRTs allow us to test which treatments generally support a desired outcome, while also identifying which specific treatments work best under specific contextual circumstances (Klasnja et al., 2015). Such treatments can, for example, be messages with varying content sent to users’ smartphones, that is, JITAIs (Nahum-Shani et al., 2018).
In contrast to MRTs, prevailing study designs, such as randomized controlled trials (RCTs), which randomly allocate participants to a treatment group once for the entire duration of a study, cannot sufficiently integrate dynamic contextual changes or varying states of the individual. Insights from RCTs can only evaluate overall treatment effects at a single distal point in time; thus, they facilitate only static investigations (Cahenzli et al., 2021; Chalmers et al., 1981). Accordingly, when examining JITAIs with RCTs, researchers can only determine which JITAI combination works best on average by examining the outcome at a distal point in time. However, an RCT does not allow an assessment of how each JITAI’s effectiveness is influenced by contextual factors or how this effectiveness varies over time.
Given the advantages MRTs provide, we propose and encourage the use of MRTs in IS research because they are able to account for the dynamic and complex contexts of an increasingly digitalized world. Based on the MRT experimental design, we can generate granular data that provides specific insights into chains of causality. Thus, we can identify causal effects between factors and covariates that influence the intentions, cognitions, and behavior of individuals over time. Extending our knowledge about these influential factors will enable us to theorize more thoroughly about the intentions, cognition, and behaviors of individuals for IS-relevant research purposes. We posit that the MRT experimental design is applicable to multiple IS research settings.
The research objectives of this article are threefold. First, we aim to motivate the relevance of the MRT experimental design by highlighting research questions from selected IS research domains that can be answered by conducting MRTs. Second, we aim to equip interested IS researchers with the required knowledge to conduct MRTs by explaining the MRT design requirements and MRT components. We further illustrate each phase of the MRT research process with one concrete illustrative example from the IS field. And third, we aim to show how the MRT experimental design can be deployed to expand IS knowledge by highlighting the methodological advantages of the MRT experimental design, outlining specific future research opportunities, and discussing the limitations of MRTs for IS research.
Micro-randomized trials
The MRT is an experimental design that originated in behavioral science and was specifically developed to test JITAIs (Klasnja et al., 2015). A JITAI is tailored to or triggered by the contextual and personal factors that influence an individual’s receptivity (Nahum-Shani et al., 2018). To identify and measure these dynamic contextual and personal factors, behavioral scientists leverage ubiquitously available individual digital devices, such as smartphones, tablets, and smartwatches, as mobile sensing technologies. This technology is used to both record contextual and personal factors (e.g., a user’s location, physiological, affective, or behavioral state) and to interact with users of JITAI applications (i.e., through the user interface via messages or pop-ups) (Klasnja et al., 2015). Thus, essentially, MRTs are designed to investigate how the prevalence of IT can be leveraged to purposefully influence human cognitions, attitudes, and behaviors.
A closer look at MRT design requirements and components reveals that MRTs are not limited to the development of JITAIs. Beyond the ideas of MRTs developed in behavioral science (Klasnja et al., 2015; Nahum-Shani et al., 2018), we demonstrate that MRTs can be applied to any IS that relies on user interactions. For example, they can be applied to quantify and analyze the time- and context-varying impact of different IS feature designs and IS user interface components over time. Further, if the experimental components of the MRTs are informed by relevant theories or frameworks, they can be deployed to apply, test, and extend the chosen theory to understand the role of time-varying contextual and personal factors based on the chosen theory (Klasnja et al., 2015). Accordingly, a multitude of IS-specific research questions from varying IS research domains can be answered by conducting MRTs.
Information Systems research questions for micro-randomized trials
Generally, research questions that can be answered with MRTs should be composed of specific components. These include different treatment options, an expected direct (proximal) outcome to these treatment options, time-varying contextual factors that may influence the treatments’ efficacy, and a desired long-term outcome of the treatments and their direct (proximal) outcome. Generalized MRT research questions may, for example, ask “how do contextual factors dynamically influence the impact of different treatment options on the direct (proximal) outcome? And how does this effect on the direct (proximal) outcome impact the desired long-term outcome at a distal point in time?”. These and other relevant MRT components are explained in detail in the following section on “Micro-Randomized Trial Components.”
Potential MRT research questions for IS research domains.
Neuro-IS
The domain of Neuro-Information-Systems (Neuro-IS) relies on neuroscience and neurophysiological knowledge and tools to better understand the development, use, and impact of information and communication technologies (Dimoka et al., 2010).
To better understand the use of IS, researchers may ask how the framing (or wording) of a task influences which features of an IS are used to complete a specific task. They would therefore be interested in understanding the impact of different task framings (treatment options) on IS feature use (proximal outcome). This relationship is likely to be influenced by users’ affective state (time-varying personal factor) when they receive the task. Hence, Neuro-IS researchers may be specifically interested in how the different framings’ effects on IS feature use are influenced by users’ affective state at the moment of task communication. To capture users’ affective states in real time, neurophysiological tools can be leveraged, for example, by measuring heart rate variability, galvanic skin response, or respiration and skin temperature. These measures can then be translated into users’ affective states. This real-time data can be used to understand how varying affective states dynamically impact the effect of task framing on IS use behavior. Consequently, it is necessary to collect longitudinal experimental data while capturing users’ time-varying affective states when conducting a corresponding MRT. Details about these and other “MRT Design Requirements” are explained in the next chapter.
Overall, Neuro-IS researchers are likely to be interested in supporting the outcome of satisfactory task completion (long-term outcome) and could therefore ask how the task design and the resulting feature use affect overall task completion. These types of Neuro-IS research questions, which investigate the dynamic influence of neurophysiological factors on IS use and user behavior, can be investigated and answered with MRT experiments.
Cybersecurity
In the domain of Cybersecurity, many IS researchers look at how warning messages (alerts) affect individuals’ online privacy behavior, often to ensure the overall cybersecurity of their employers (Cram et al., 2024). They may thus be interested in investigating how adaptive alerts (treatment options) can be used to influence users’ online data sharing (proximal outcome) to increase cybersecurity (long-term outcome). The effectiveness of such alerts is likely to vary as users access multiple websites with different demands for data sharing (varying contextual factor) over time. For example, an employee may be interested in receiving an informative newsletter for which they must provide their name, email address, age, and function. Similarly, an employee might want to sign up for an ad-free service but must provide personal data (such as their IP-address) to gain access to specific content. These different data sharing demands of multiple websites vary over time for each user and influence the effectiveness of alerts with different designs. Similarly, the repeated exposure of users to warning messages may influence their effectiveness. Accordingly, warning pop-ups should be personalized based on users’ previous data sharing behavior and the data sharing demands they face in the moment of message delivery. Hence, different design options need to be tested in varying contexts creating multiple different experimental groups, if these alerts were to be tested with traditional experiments (e.g., RCTs). With the MRT experimental design, such granular investigations for the personalization and contextualization of alerts to influence users’ data sharing behavior can be robustly conducted even with smaller sample sizes due to the MRT’s unique micro-randomized treatment assignment.
Hence, cybersecurity and privacy researchers can investigate questions such as “how do adaptive alerts with varying content affect users’ data sharing over time? How effective are these messages for sensitizing users to better protect private data on different websites with varying data sharing demands? And how does this ultimately influence an organization’s overall level of cybersecurity?”. These types of research questions, asking how users’ online behavior can be influenced and steered, can be answered with MRT experiments.
AI Support
A large stream of IS research examines the impact of AI systems on human and organizational decision-making (Berente et al., 2021; Jain et al., 2021). Within this domain, IS researchers investigate how joint human–AI decision-making affects humans and organizations and how this joint decision-making can be improved (Allen and Choudhury, 2022; Bauer et al., 2021; Oberste and Heinzl, 2023). This research can be supported by conducting MRT experiments, for example, by investigating how AI Support with varying explainability features (treatment options) influences a user’s decision-making confidence (proximal outcome) over time. Over time, as users repeatedly deploy an AI tool for decision support, their increasing experience with the AI tool (time-varying personal factor) likely moderates the relationship between explainability features and users’ decision confidence. The level of experience with the respective AI tool can be measured by the number of decisions previously supported by the AI tool or by observing how users navigate through the tool, for example, by recording mouse movements or eye tracking. An IS researcher in the domain of AI Support may then be interested in answering questions such as “how does AI Support with varying explainability features influence users’ decision confidence over time?” and “how is this effect influenced by a user’s level of AI experience over time?”. The overall desirable outcome of joint human–AI decision-making is an improved decision quality (long-term outcome). Accordingly, researchers could ask whether AI Support, explainability features, and users’ decision-making confidence lead to improved decision quality over time.
These dynamic relationships, which include several context- and time-varying factors, such as multiple treatment options, an influential endogenous covariate (AI experience) as well as different short-term (decision confidence) and long-term outcomes (decision quality), cannot be tested with traditional factorial experiments such as RCTs. Instead, they require a longitudinal experimental approach that allows for capturing dynamic effects and considers endogenous time-varying covariates in the effect estimation. Traditional longitudinal experiments that employ stages as an experimental factor are also inadequate. Such predefined stages cannot capture the contextual dynamics of individually and continuously changing covariates as these likely vary between participants at each experimental stage. These covariates need to be measured objectively rather than assuming changes at each experimental stage. Investigating phenomena with endogenous time-varying covariates calls for a more flexible experimental approach such as the MRT experimental design, which, through its unique randomization procedure, captures these contextual dynamics to then incorporate their impact in the effect estimations.
Therefore, the types of research questions that take various time-varying personal and contextual covariates (e.g., state of knowledge or skill-level of the user) into account can be effectively addressed by analyzing data generated with MRT experiments.
In general, MRT research questions focus on the dynamic effects of multiple treatment options on a proximal outcome which may be influenced by contextual factors or characteristics of the individuals. These questions are typically asked in light of a desired distal outcome that is caused by the immediate (proximal) outcome and its previous treatment options. The above conceivable research questions in Neuro-IS, Cybersecurity, and AI Support illustrate potential treatment options, proximal outcomes, and distal outcomes that would be meaningful and relevant for IS research. They further highlight conceivable time-varying contextual and personal factors that can dynamically influence these relationships.
To answer these types of research questions, researchers from various IS domains may be interested to know how to set up MRT experiments. Therefore, we subsequently present the experimental design requirements and the components required to set up MRT experiments, before explaining the MRT research process with the help of an illustrative but more in-depth example.
Micro-randomized trial design requirements
The MRT experimental design was originally constructed as an online field experiment (Nahum-Shani et al., 2018). Although it is most intuitively applicable to field experiments, it can also be applied for laboratory (lab) experiments and quasi-experiments (for a classification of the different types of experiments, see Appendix A). The below sections explain four types of experimental design choices and the corresponding requirements that should be implemented when conducting an MRT in IS research.
Online treatment delivery
Generally, experimental treatments can be delivered and conducted either offline or online in both lab and field experiments. In MRTs, treatments are typically delivered online, especially for investigating IS-relevant research phenomena that usually include a digital or online component. This online delivery of treatments often comes with the benefit that the timing of treatment delivery can be steered digitally, and that the treatment’s effects can be recorded digitally (Fink, 2022). Together the digital records of treatments and treatment reactions comprise digital traces that store and capture the temporality of the experimental data. However, even if the treatments are delivered online, resulting reactions can take place offline and then require additional means for data collection (e.g., with sensors or observations).
If possible, conducting experiments on online platforms (e.g., Amazon Mechanical Turk, 2018) provides access to larger and potentially more relevant sample populations than conducting them locally or in a lab. Yet, this also limits the nature of treatment tasks to those that can be delivered and performed online and limits the researchers’ control over participants’ external environment (Karahanna et al., 2018). MRTs are usually conducted as online field experiments with a digital component or online lab experiments, where users’ reactions can be observed and recorded easily.
Sequential, mixed-subject randomization procedure
In experimental research, different procedures of randomizing participants to treatment conditions can be applied. Commonly used randomization procedures include, for example, RCTs, factorial designs as well as sequential or conditional assignment of participants to study conditions (Alferes, 2012).
The MRT experimental design employs a unique type of randomization procedure that deviates from the aforementioned ones. As the eponym “micro-randomized” indicates, an essential feature of the design is its sequential randomization of participants to study conditions in a longitudinal within-subject setup. The sequential randomization further enables a between-subject comparison of participants who have received a treatment in one sequence with participants who have randomly not received this treatment during the same sequence. Accordingly, a mixed-subject setup is created, which allows for both within- and between-subject comparisons (Klasnja et al., 2015). These sequences can be applied in field and lab experiments. They can also be used in quasi-experiments for which the chosen personal or contextual characteristics should then be considered as covariates in the data analysis.
Intent-to-treat and per-protocol effect
In experimental research, after the delivery of a treatment, a control mechanism can be incorporated to ensure that the participants received the treatment in the desired way, which is then referred to as the per-protocol effect. Researchers can also investigate the intent-to-treat effect. This refers to the possibility that participants may choose not to receive a treatment that has been delivered to them. If it is important for the experimental setting that the treatment is received in the desired way, the per-protocol effect should be tested by implementing an additional control mechanism. However, sometimes delivering the treatment and testing the delivery’s effects may be sufficient or desirable, while leaving the choice of truly receiving the treatment or not to the participants (McCoy, 2017). This more closely reflects reality and can thus increase an experiment’s external validity. Yet, considering both the intent-to-treat and the per-protocol effect allows for more robust data analyses (Dettori and Norvell, 2020; Gupta, 2011).
For MRTs, both per-protocol and intent-to-treat effects may be of interest. Originally, the MRT was designed to test push notifications delivered to users on a smartphone screen. Such messages might not always be fully read and are often swiped away. Conceivable control mechanisms in this setup would be either a rating button or recording whether the message was clicked on to be fully read within a limited timeframe after the message’s delivery (Klasnja et al., 2019). Conversely, the message could simply be intended as a reminder or nudge for which reading the message content is considered less relevant, in which case it might be desirable to focus only on the intent-to-treat effect (Nahum-Shani et al., 2018). However, if possible, considering both effects in the data analysis might provide additional insights and strengthen the experimental findings by investigating the treatment’s boundary conditions (Gupta, 2011). Accordingly, the online implementation of unobtrusive control mechanisms and full control over the treatment design are appropriate for MRT experiments. This can be achieved, for example, by collecting reliable real-time digital trace data (in field settings), employing sensors to record contextual or physiological reactions (in lab and field settings), or by closely monitoring and observing participants with audio-visual recordings (in lab settings).
Measurement of contextual variables
For most experiments, the additional measurement and recording of potentially confounding variables and covariates enables a more robust data analysis, especially if true randomization and full control of the treatment and the environment cannot be achieved with the experimental setup (Shadish et al., 2002). It is often easier to collect and record potentially confounding variables in online rather than offline settings, because there the data is automatically generated and can be stored as digital trace data (Fink, 2022). Collecting potentially confounding variables and covariates further allows researchers to define boundary conditions for generalizing the experiment’s findings to other contexts and settings.
For the MRT design, it is a requirement to simultaneously measure contextual variables that are theorized or expected to influence the treatment efficacy (Qian et al., 2022). While these variables do not necessarily need to be time- or context-varying, it is one of the unique characteristics of the MRT experiment that the influence of endogenous covariates can be quantified and considered for causal effect estimations. Hence, static as well as dynamic variables, for example, personal factors, such as the level of attentiveness or affective state of the participant, or contextual factors, such as the time and place of treatment delivery, should be measured in real time.
Generally, the recording of such variables is assumed to be more easily performed in online settings in which technologies can be applied for automatic recording (e.g., GPS and calendar functions in mobile applications) or for manual digital recording with little user burden (e.g., ecological momentary assessment with short digital questionnaires) (Walton et al., 2018). For IS field experiments, the interaction with an IS in the field is likely to be of interest to the investigator. The corresponding system should then be able to directly collect and record potentially relevant covariates and, thus, would require a certain degree of context awareness. An alternative option would be to employ additional means of contextual data collection, for example, by using sensor technologies (Battalio et al., 2021). In a lab setting, relevant contextual factors are more likely to focus on the participants themselves, since the environment is already controlled for. Contextual variables can include physiological measurements, such as blood pressure, eye movement, or brain waves (especially in Neuro-IS experiments), but also emotions and intentions (relevant in behavioral economics), which can either be verbalized by participants or observed through audio-visual and sensory recording of participants (Kirwan et al., 2023; vom Brocke and Liang, 2014).
If these four design requirements are met: (1) Online or digital setting, (2) Sequential randomization procedure, (3) Unobtrusive data collection, and (4) Measurement of time-varying contextual variables, IS researchers can set up MRT experiments consisting of the components presented and explained in the next section.
Micro-Randomized Trial Components
The following descriptions are primarily based on Klasnja et al. (2015) and also draw on Nahum-Shani et al. (2018), Klasnja et al. (2019), and Qian et al. (2020, 2022). Figure 1 gives an overview of the components included in a typical MRT design. The conceptual framework and general structure of an MRT design.
In an MRT, the experimental treatment is composed of multiple Treatment Options for achieving the unique mixed-subject MRT experimental design. Treatment options are randomized at each Decision Point according to pre-specified Randomization Probabilities. Every time participants arrive at a specific decision point, they are, according to the predefined probabilities, randomly assigned to the study conditions specified in the chosen treatment option. Decision Rules define under which contextual conditions a decision point occurs, for example, at a predefined timepoint on a specific day of a week, in a specific location, or when a specific IS feature is accessed.
The repeated and iterative exposure to varying treatment options at recurring decision points results in the sequential randomization and within-subject design. This longitudinal, sequential randomization procedure, by which participants receive various treatment options multiple times, is visualized by the circular gray arrows in the background of Figure 1. Further, at each decision point, a Baseline Randomization or “do-nothing” option is required to create a control group within the sequence. This enables both a between-subject and a within-subjects comparison and thus the unique mixed-subject, that is, micro-randomized, design of MRTs.
The overall goal that the user, researcher, or IS developer desires to achieve with the IS under investigation is called the Distal Outcome. This distal outcome is theorized to be achieved by the accumulation or repeated occurrence of one or multiple immediate or proximal outcomes which act as mediators between treatment options and the distal outcome. Thus, Proximal Outcomes are the intended short-term goals of the treatment options. In the original behavioral science context, proximal outcomes are typically defined as offline behaviors (e.g., step counts or medication adherence metrics). In IS research, however, proximal outcomes can also encompass treatment options’ effects on participants’ IS use (often a mediator for offline behavior as the distal outcome), their intentions or attitudes (to technology and its use), or their offline behaviors and cognition (such as decisions and tasks participants perform, or digital literacy and knowledge). Commonly, other research contexts would use the term Dependent Variable. However, for MRTs, this general term does not accurately reflect the distinction between proximal and distal outcomes. Considering this ambiguity, we refrain from using the commonly applied notion of (in)dependent variables. 1
To investigate how potentially time-varying contextual and personal factors affect treatment options and proximal and distal outcomes, they are measured via the Observation of Context. This serves, first, to determine decision rules and restrict the treatment to feasible time points and, second, to identify potential moderators and covariates that can be used to identify which option performs best in different contexts.
Because of the sequential randomization, a treatment’s effect can occur later than initially intended, and the repeated exposure to treatment options can have arbitrary effects on proximal outcomes. Thus, spillover or Lagged Effects must be considered in the MRT data analysis. This means that the effect estimation should not be attributed only to the latest treatment but should incorporate previous treatments as well. Further, because we assume that covariates are likely to be endogenous, and since we are interested in the dynamic effects when we conduct MRTs, the main data analysis focuses on the Causal Excursion Effect. The causal excursion effect describes the effect of treatment options on proximal outcomes while considering lagged effects and the influence of selected likely endogenous covariates. We explain how the causal excursion effect can be quantified and computed when discussing the details of the MRT data analysis in analogy to an illustrative example in the next section.
The MRT components explained above are depicted in the Conceptual Framework in Figure 1. The conceptual framework should be informed by relevant theories or theoretical frameworks, literature streams, and practical knowledge. Based on this combination of knowledge, the links between contextual observations, treatment options, proximal outcomes, and distal outcomes can be hypothesized. We illustrate these MRT design requirements and MRT components by guiding the reader through a step-by-step illustrative example of the MRT research process in the following section.
An Illustrative Example of a Micro-Randomized Trial in Information Systems Research
In this section, we demonstrate how an MRT can be conducted in IS research by giving a step-by-step description of one specific illustrative example. We aim to provide a blueprint which IS researchers can refer to when setting up an MRT study. To do so, we reflect on the important aspects to be considered at each phase of the MRT research process.
We relate our explanations to a study (Fallon et al., 2021), which, to the best of our knowledge, is the first example of an MRT published in the IS field. Considering the use of a mobile health application (mHealth), the study specifically examines the impact of various feedback messages (treatment options) on IS use (proximal outcome) and the resulting physical activity behavior (distal outcome). It probes the time-varying effects of the feedback messages as individuals pursue their goals over time (observation of context) (Fallon et al., 2021).
Overview of MRT research process.
Phase 1: MRT design requirements
The chosen example investigates how push notifications with social or goal-related feedback can trigger the use of specific mHealth features and physical activity behavior. It relies on social cognitive theory (SCT), as well as literature on goal pursuit, to theorize about the effect of feedback messages on mHealth use and real-world behavior that accompanies such use. Further, the study reflects on the dynamic relationships that occur as mHealth users progress toward their physical activity goals. The study’s aim is to “determine how the user’s stage of goal pursuit can differentially impact the effectiveness of messages with content regarding (1) goal feedback and (2) social feedback,” and it focuses “on the impact of both immediate mHealth use and subsequent behavior change” (Fallon et al., 2021 p. 2). Thus, the study asks the following research questions: - RQ1: How do goal feedback messages dynamically impact feedback feature use at different stages of goal pursuit (beginning, middle, and end)? - RQ2: How do social feedback messages dynamically impact social feature use at different stages of goal pursuit (beginning, middle, and end)? - RQ3: How does mHealth feature use subsequently impact behavior change?
These research questions are suited to be answered with an MRT experimental design. It is up to the researcher to first develop assumptions on how X (feedback messages) impacts Y (mHealth feature use) that can be investigated experimentally. Besides testing theoretical postulations, the problem has high practical relevance and should be generalizable to the broader mHealth and IS use context. Importantly, the study aims to understand the time-varying effect of feedback messages on mHealth use as users progress toward their physical activity goals. Understanding this time-varying effect is especially suited to being studied with an MRT.
Specifically, the first two RQs ask how various treatment options (goal feedback and social feedback) dynamically impact proximal outcomes (use of specific mHealth features). RQ3 then asks how the distal outcome, behavior change, is impacted by the proximal outcomes of mHealth use. The study employs a more specific IS use concept by considering use of mHealth features, instead of relying on a general binary IS use concept (Burton-Jones and Grange, 2013). RQ1 and RQ2 further emphasize an important contextual variable that is expected to impact the treatment efficacy, namely, the stage of goal pursuit. The RQs emphasize the expectation that the effects of social and goal feedback messages will dynamically change as individuals progress toward their goals. Accordingly, they evaluate the time-varying effect of the various feedback messages on mHealth use when users are at the beginning, middle, and end stage of goal pursuit. Thus, the stage of goal pursuit is identified as an endogenous covariate that is expected to influence the effect of feedback messages on mHealth feature use. Consequently, the study aims to measure the causal excursion effect to explore how mHealth feature use and the resulting physical activity behavior change develop as the study progresses. Given these characteristics, the research goal can be achieved by conducting an MRT.
Phase 2: MRT components
Informed by SCT and the literature on goal pursuit, a research model and experimental framework for answering the RQs were derived (Bandura, 1991; Koo and Fishbach, 2010, Koo and Fishbach, 2012). To better understand how the dynamics of mHealth use facilitate behavior change, SCT was applied because it acknowledges ongoing, dynamic feedback loops of behavior in response to evolving social, personal, and contextual circumstances (Bandura, 1991; Fallon et al., 2021: 5).
The theory defines self-regulation as a dynamic within-person process influenced by internal and external cues. These cues can be instantiated with goal and social feedback messages. Goal feedback messages provide information regarding the extent to which users have progressed toward their goals (or not), triggering internal cues. Social feedback messages provide information on how social others are performing, triggering external cues. According to SCT, these internal and external cues can influence an individual’s behavior (e.g., motivate physical activity). However, SCT does not indicate how the effect of feedback dynamically changes as individuals progress toward their goals over time (Bandura, 1991; Fallon et al., 2021).
Thus, the researchers consulted literature on goal pursuit to define the stages at which different forms of feedback messages were expected to be more or less effective at various stages of goal pursuit. Figure 2 summarizes this argumentation. Feedback messages and psychological processes during stages of goal pursuit (taken from Fallon et al., 2021).
Given the theoretical background, a research model which included the following six hypotheses was developed: - H1: Feedback feature use will be higher immediately after receiving a goal feedback message (subgoal or overall goal) compared to receiving no goal feedback message. - H2a: Goal feedback messages emphasizing daily subgoals will be most effective at impacting feedback feature use in the beginning stage of goal pursuit. - H2b: Goal feedback messages emphasizing overall weekly goals will be most effective at impacting feedback feature use at the end stage of goal pursuit. - H3: Social feature use will be higher immediately after receiving a social feedback message compared to receiving no social feedback message. - H4: Social feedback messages will be most effective at impacting social feature use in the middle stage of goal pursuit. - H5: An increase in mHealth use leads users to be more physically active.
These hypotheses can be tested with the MRT design illustrated in Figure 3. MRT design of the illustrative example.
With an experiment duration of 28 days, this study generated 28 decision points per participant for randomizing goal feedback messages and 84 decision points per participant for randomizing social feedback messages. This is achieved as each participant follows the data collection loop multiple times given the time of day on each day of the study.
This setup consulted prior literature on goal pursuit and relied on mHealth and IS use domain knowledge to enable a feasible experiment. Thus, it explores how different feedback messages specifically and on average influence mHealth feature use across a period of 28 days while considering the contextual factor of goal progression. Concurrently, given that these dynamic relationships indicated by SCT have not been validated empirically before, the experiment promised insights that contribute to the refinement of SCT by considering the time-varying effect of feedback messages.
Phase 3: MRT feasibility assessment
Given the setting described above, the research team decided that being in full control of a given mHealth app’s design and data collection process was required. Thus, an app, which we will call Research App (REA), was developed. This allowed the research team to design and implement app features corresponding to goal and social feedback (goal progression, leaderboard, and social feed, respectively). The app provided access to objective trace data and allowed full control of push notifications’ delivery and the data collection process.
At this stage of the experimental research process, researchers must assess how much they need to control the experimental environment, treatment delivery, and treatment design and must decide how to translate these considerations into concrete design choices as well as the operationalization of MRT components. In the example’s context, it had already been determined that a field experiment was suitable and that the researchers needed minimal control of the experimental environment, since insights can better be generalized if they result from testing in the participants’ regular, unmanipulated surroundings.
For MRTs specifically, participants’ availability is considered to ensure that they can receive the treatment when it is delivered. In the given example, this was ensured by requiring participants to pre-select three timepoints during the day and one timepoint in the evening (all at least 3 hours apart) when they would usually be able to look at their phone if a message popped up (Decision Rules in Figure 3). The treatment options were operationalized as push notifications with carefully designed content such as “You have reached X% of your daily goal with Y% to go!” to deliver subgoal feedback.
To collect physical activity data as unobtrusively as possible, REA was integrated with Apple Health and Google Fit to automatically record minutes of physical activity. Users could further input physical activity data manually. The proximal outcomes of feedback and social feature use were measured as minutes of respective feature use within the 60 minutes after each treatment delivery. The distal outcome was measured by assessing the total duration of physical activity and whether users had reached their pre-specified goals or not. Stages of goal pursuit (beginning, middle, and end) were directly computed from the app data given a user-specified goal and the user’s progression toward that goal. Accordingly, having reached 0–33%, 34–66%, or 67–100% of the specified weekly goal was defined as the beginning, middle, or end stage of goal pursuit, respectively, for each user at each decision point. Overall, the use of REA resembled the use of commercially available fitness apps without requiring participants to actively provide additional continuous data input during the trial.
This unobtrusive, commercially resembling app design required a reliable logging function in the app’s backend and data storage, which was ensured via the app development platform Firebase (Google, nd). Before and after the experimental period, participants filled out a survey to provide demographic data and additional information that could be used later in robustness checks. The app’s functionality and MRT components’ operationalizations were refined iteratively before an experimental pre-test was conducted.
Phase 4: MRT pre-test and sample size considerations
To ensure that the MRT setup, operationalization of goal stages, and technical requirements worked as intended, a pre-test was conducted, after which phases 2 and 3 were re-evaluated. This confirmed the experimental framework and operationalization of MRT components; however, the outcome of phase 3 indicated some technical bug fixes were necessary, which were implemented, while the activity tracking was also improved.
Thus, as a general guideline, phases 2, 3, and 4 can be performed iteratively until a final MRT setup is determined. Additionally, during this phase of the MRT research process, researchers are expected to apply for ethical approval and pre-register 2 their experimental study. This is especially important if the experimental setup carries ethical concerns such as psychological user burden or unequal sampling strategies of potentially (dis)advantageous treatments. Pre-registration and a pre-analysis plan also ensure transparency and trustworthiness of the study (Burton-Jones et al., 2021). These best practices hold for all kinds of experiments, thus also for MRT designs.
Based on the insights gained from the pre-test, such as user responsiveness to messages and expected treatment effects, a power analysis and corresponding sample size for the planned trial can be calculated. This ensures that later a robust and significant effect estimation can be performed with the experimental data. In MRTs, different primary and secondary research questions could require different numbers of decision points, different treatment options, and different participant availabilities. Thus, the sample sizes required to achieve a desired effect size and power to answer these questions can vary considerably. In such cases, to determine the sample size, the primary research questions should be in focus (Qian et al., 2022). For further information about sample size calculations, see Liao et al. (2020).
In general, MRTs require fewer participants than traditional full factorial study designs because of the MRT’s mixed-subject design, due to which a higher number of effect estimations is achieved by a smaller number of participants, compared to experiments where only between-subject or only within-subject comparisons are feasible (Klasnja et al., 2015). Seewald et al. (2020) developed a convenient sample size calculator 3 that considers all factors relevant to achieving statistical power for a specific research question. To compute either the required sample size or the expected power of the analysis, the calculator requires the duration of the study, the number of daily decision points, the randomization probability, the expected availability of participants, the expected effects of the treatment option on the proximal outcome, and either the desired power or the available sample size as input variables.
Sample size calculations summary.
Since participants were required to self-select decision points at which they would be available to receive a treatment, an availability of 100% was assumed for goal feedback messages, which were delivered once per day in the evening. Because social feedback messages were sent three times per day, a more conservative availability of 80% was assumed. The expected treatment effects were estimated to linearly decrease based on knowledge from previous studies and on the conducted pre-test. The assumed parameter resulted in sample sizes of 64 and 50 participants, respectively. For the final experiment, 67 participants were invited, of which 61 completed the experiment in its entirety.
Phase 5: MRT data collection
During the data collection, the researcher either waits until the data has been collected or might be required to accompany the procedure, for example, with observation, giving instructions to participants, or being available in case unexpected technical problems occur. Generally, it is important to avoid any data missingness and to be prepared for technical issues.
REA’s deployment on Android and iOS smartphones and its synchronization with Apple Health and Google Fit were meticulously tested beforehand. Then, all participants underwent the treatment during the same 28-day period.
It might also be possible to enroll participants on an ongoing basis over the course of multiple months. Ensuring the same experimental period for all participants mitigates the threat of structural differences between participants caused by changes in the uncontrolled external environment. Ongoing enrollment, however, provides access to a larger participant pool and can achieve larger sample sizes. For MRTs, both options are feasible and should be evaluated considering each specific MRT setup, context, and technical feasibility.
In addition to the main data collection, participants can be asked to answer questionnaires that collect demographic data, as well as other important control variables. In our illustrative example, two surveys were sent to participants, one before and one after the completion of the experiment, to collect user data and assess users’ usual physical activity behavior, self-efficacy, goal commitment, and overall personal health perception. The pre-survey included instructions on how to download the app. The post-survey additionally assessed participants’ satisfaction with the app. Further, participants were able to reach out to the research team at any time during the study to provide feedback or manage possible technical difficulties. Generally, at the end of phase 5, all experimental data is collected to be analyzed, reported, and discussed in the following phases of the MRT research process.
Phase 6: MRT data analysis approaches
To better convey the logic of MRT data analysis approaches, we now denote the above-introduced MRT components in mathematical terms. At every decision point
Accordingly, the data generated by an MRT is intensive longitudinal data to which a variety of modeling approaches can be applied, such as multilevel models (MLMs) or generalized estimating equation models (GEEs) (Fitzmaurice et al., 2008). These methods, however, do not sufficiently account for the causal excursion effect, which is caused by endogenous time-varying covariates (Qian et al., 2022). Because the main purpose of an MRT is to investigate the time-varying effect of treatment options, the dynamism of this effect, and whether and how contextual factors impact it, needs to be considered. We explain how this can be achieved, also for GEE and MLM, below.
The main effect we are interested in is the effect of delivering a treatment option versus not delivering it on the next proximal outcome. Correspondingly, the proximal outcome at decision point
Since potential covariates are typically endogenous, which means that they may depend on previous treatments or their outcomes, the treatment schedule, called history
If possible, participants’ availability should also be considered. For example, the illustrative example considers participants to be available for only 80% of the social messages sent to them throughout the day, because their device might be switched off. Thus, 
By including the history and relevant covariates, lagged effects of previous treatments are captured and the endogeneity caused by known and unknown confounding variables is accounted for. Including and controlling for endogenous covariates in the data analysis provides insight into their influence and is further useful for reducing the variance when assessing the effects of treatment options (Boruvka et al., 2018).
To compute this causal excursion effect 
The components of the equation are defined as introduced above:
Further
WCLS data analysis model for the illustrative MRT design.
The proximal outcomes were measured as time deltas; thus, the duration users spent attending to the respective feature one hour after treatment delivery. As the distribution of these values was highly skewed, a log transformed version was computed for the estimation procedure.
Because the WCLS estimation method focuses on calculating causal excursion effects on proximal outcomes, it does not provide an evaluation of the effect of proximal outcomes on the distal outcome. Thus, H5 cannot be tested within the main data analysis. Instead, whether mHealth use positively impacted physical activity behavior was computed. Other possible approaches include conducting t-tests or regression analyses with pre- and post-experimental data to determine the degree to which proximal outcomes and treatments affect the distal outcome. Further, the link between proximal and distal outcomes could be established based on previous studies and existing literature.
Phase 7: MRT results’ presentation and evaluation
Summary of results from Fallon et al. (2021).
Based on these results, the study shows that the impact of feedback messages on mHealth use is influenced by the message content and users’ goal progression. The results indicate that goal feedback impacts feedback feature use (H1) and social feedback impacts social feature use (H3). In more detail, support for the increased effectiveness of daily subgoal messages at the beginning stage of goal pursuit could not be detected (H2a). But social feedback messages were most effective at the middle stage of goal pursuit (H4) and overall goal feedback messages were most effective at the end stage of goal pursuit (H2b). Lastly, mHealth use increased users’ physical activity and goal accomplishments (H5).
To verify the robustness of the obtained results, randomization checks or testing for alternative proximal outcomes can be performed. In a randomization check, the obtained dataset is resampled before the analysis is performed again. If the same results are obtained, they cannot be attributed to the experimental treatment. If, however, different or no effects are found, it is unlikely that the results were caused by statistical coincidence; thus, the results are robust.
The results presented above were shown to be robust after rerunning the analysis 5000 times with reshuffled datasets, where treatment delivery
An important aspect in evaluating the robustness of the results is the reliability of the estimation method itself. The WCLS estimator is a non-parametric method that does not require a specific model that correctly generates proximal outcomes. To estimate the causal excursion effect consistently, only the causal excursion effect model needs to be correct (i.e.,
Phase 8: MRT insights and contributions
After presenting and evaluating the results, the findings should be discussed in the broader context of the study. In the illustrative example, different types of messages were tested dynamically at different stages of goal pursuit (dynamic contextual factor). Accordingly, the empirical findings contribute to SCT, IS use, and mHealth research.
Social cognitive theory (SCT) specifies bidirectional feedback loops between the environment and human behavior (Bandura, 1991) but does not explicate how these feedback loops behave over time. Based on the conducted MRT, the temporal specificity within these regulatory feedback loops was investigated. Drawing on goal pursuit literature, the example demonstrates how the effects of goal-related and social feedback messages vary, depending on whether users are at the beginning, middle, or end stage of goal pursuit. This dynamic challenges the notion that environmental effects on behavior are constant. Instead, feedback influences behavior, and behavior, in turn, influences users’ reaction to the provided feedback. This aligns with SCT’s bidirectional self-regulatory relationships. However, the MRT adds a temporal dimension to these relationships by explicating that the effects of feedback messages vary as users progress toward their goals, while this goal progression is simultaneously influenced by the respective feedback messages. These insights that foreground the underlying causal mechanisms and the influence of an endogenous time-varying covariate could not have been gained without conducting an MRT and leveraging its dynamic properties for causal inference.
The example also contributes to research on IS use by conceptualizing it as a dynamic interaction with specific mHealth features. Prior mHealth research often considered mHealth use as either binary (present or absent) or based it on the extent of use (time spent in app). These simplistic concepts fail to capture interactions with specific features and how feature use changes over time. The MRT conducted in this illustrative example advances this research stream by examining how individuals’ motivational states evolve as users progress toward their goal. The findings indicate that goal feedback messages directly affect the use of feedback features, while social feedback directly affects the use of social features. These effects are not constant but vary with users’ goal progression, highlighting that a dynamic perspective on IS feature use offers valuable insights into how, why, and when feedback messages influence feature use.
Based on these insights gained with the MRT study, practical contributions for the design of mHealth are offered. The example reveals that the use of feedback and social features significantly influences physical activity behavior, suggesting that mHealth developers should focus on how to activate the use of these features. This can be achieved by either sending goal feedback or social feedback messages, each at a specific point in time. More concretely social feedback messages should be sent at the middle stage of users’ goal pursuit, while overall goal progress messages should be sent at the end stage of users’ goal pursuit. Therefore, app providers should consider both the content and the timing of messages to influence mHealth use and subsequent behavior.
These novel insights into the dynamics of how social and goal feedback messages impact mHealth feature use could be generalized to other contexts, such as commercial fitness apps or further IS that are employed to track goal progression and send feedback messages, such as learning and study apps or IS for promoting sustainable behavior. After implementing the corresponding changes, A/B testing or RCTs can be conducted to test a new version of the IS that applies message delivery rules based on the derived insights and compares it with the previous version that sends reminders to its users without considering goal progress. Thus, the granular insights derived from an MRT should be confirmed with a practically oriented RCT. However, determining the dynamic decision rules according to which varying treatment options should be delivered based on progressing stages of goal pursuit cannot be accomplished with an RCT.
Discussion
The above example illustrates how MRTs can be deployed to provide theoretical and practical contributions for mHealth research. Following the outlined eight-phase research process, MRT experiments can also be leveraged to expand knowledge in other IS research domains.
To generally motivate the application of MRTs in IS research, we presented potential research questions from three IS research domains that can be answered by conducting MRT experiments. We then explained the MRT design requirements and specific MRT components, before instantiating these requirements and components with an in-depth illustrative example that has been conducted and published in the research domain of Healthcare IS (cf. Fallon et al., 2021). Based on this detailed illustration, the MRT experimental design can be applied to various other research contexts, investigating dynamic IS phenomena and advancing the methodological toolkit available to IS researchers.
Methodological advantages of micro-randomized trials for IS research
In this section, we discuss the main methodological advantages of MRTs. Specifically, we address the benefits regarding purposeful sample recruitment and the estimation of dynamic treatment effects that vary over time due to endogenous covariates. Additionally, we briefly present two conceptual examples that further illustrate these methodological advantages, supplementing the in-depth illustrative example discussed earlier.
Based on the presented MRT methodology, IS researchers are now able to experimentally investigate IS-relevant phenomena and corresponding research questions that involve dynamic relationships. As highlighted in the section “IS Research Questions for MRTs,” these dynamic relationships can, for example, be caused by the influence of time-varying neurophysiological factors on IS use and user behavior or by other time-varying personal and contextual factors (such as the skill level or knowledge of a user) on decision-making and online behavior.
Capturing such dynamic relationships necessitates longitudinal observations. To examine causal effects with such observations, experimental setups are required. However, attempting to test these dynamics with well-known factorial experimental designs would require multiple experimental groups and very large sample sizes. Further, such factorial setups would not be able to quantify the dynamic impact of time-varying covariates. MRTs, on the other hand, allow the causal mechanisms of dynamic interactions between treatment options, covariates, and proximal outcomes to be unveiled with considerably smaller sample sizes than comparable experimental designs that allow only for between- or within-subject comparisons.
These two main advantages of the MRT experimental design are achieved with its name-giving, unique micro-randomization procedure. According to this procedure, participants sequentially follow the treatment delivery and data collection multiple times under varying contextual conditions. As a result, decision points can be compared not only between different participants but also within a single participant over time. The resulting mixed-subject setup requires smaller sample sizes for robust effect estimations than comparable between- or within-subject comparisons. Further, the online design requirement of MRTs typically provides access to a larger and potentially more relevant sample base. Together, smaller sample size requirements and improved access to relevant samples ease the purposeful sample recruitment according to well-defined sample selection criteria.
During the data collection phase of MRTs, the treatment delivery, its outcomes, and important contextual and personal factors are recorded in real time. These contextual observations are then incorporated into the data analysis. To quantify the dynamisms of the observed effects, the inherent endogeneity of time-varying contextual covariates must be acknowledged in the effect estimation. As explained with the illustrative example, certain analysis methods, which are particularly tailored to MRT data, such as the WCLS estimation method (Boruvka et al., 2018), provide robust estimates of treatment effects that take known and unknown endogenous factors into account.
Further, due to the repeated treatment assignment of the micro-randomization procedure, lagged, delayed, or spillover effects need to be considered when analyzing MRT data. The effect estimation should therefore be attributed to not only the latest treatment but previous treatments as well (Qian et al., 2020). In the MRT data analysis, such lagged effects are acknowledged by incorporating the treatment history of participants when computing the causal excursion effect of treatment options on proximal outcomes. We have provided an example of how this effect estimation is computed.
Based on these advantages of the MRT experimental design, novel theoretical and practical insights can be generated by following the outlined eight-phase MRT research methodology. Next to the detailed discussion of the contributions of our in-depth illustrative example, we present two conceptual examples to further exemplify the advantages of MRTs for extant IS research (for details, see Appendixes C and D).
The first example (Appendix C) is based on a study by Loock et al. (2013) that explores the potential of sustainability-promoting IS in encouraging energy efficient behavior in private households. The authors draw on the extended model of goal-directed behavior and goal-setting theory to develop hypotheses that were tested in a five-group factorial experiment considering only between-subject comparisons. An MRT could combine these insights with social normative and social incentive theories to scrutinize the granular effects of a feedback-providing IS with dynamic goal-setting functionalities on users’ electricity-conserving behavior over time. This example illustrates how theoretical contributions can be achieved if the MRT framework is developed based on appropriate theories and models that are currently lacking a consideration of time-varying contextual or personal factors (i.e., the extended model of goal-directed behavior, as well as social normative and social incentive theories). This example further highlights the general applicability of MRTs for studying and influencing human behavior in human–IT interaction research.
The second example (Appendix D) is concerned with improving the design of online dating platforms inspired by a study by Jung et al. (2022). The original study investigates how choice capacity (i.e., the number of potential candidates a user can view and select) impacts the number of matches achieved on the platform, while considering gender as a static moderating variable in a factorial experiment. Inspired by this experimental setup, the conceptual MRT example in Appendix D suggests that MRTs can be deployed to evaluate different machine learning (ML)–based approaches for improving user engagement on matching platforms. This example illustrates how the design of online dating platforms can be improved on the basis of MRT results. Such results may, for example, provide insights for improved profile presentation and adaptive choice capacity (treatment options) informed by the influence of previous matches and currently active matches (time-varying contextual factors). Overall, the impact of platform design and functionality on user engagement (proximal outcome) and matching success (distal outcome) can be improved based on the MRT results. Corresponding empirical findings could then also be generalized to inform the design strategies of other matching markets and to elicit boundary conditions for utilizing different platform presentation modes in different contexts. Accordingly, this example highlights the practical applicability of MRTs for improving the design of IS for different application contexts.
Similarly, conducting MRTs can also contribute to IS research in the domains of Neuro-IS, Cybersecurity, and AI Support. In general, MRTs are beneficial for researching dynamic IS phenomena concerned with user interactions, users’ online and offline behavior, or the design of IS components.
Limitations of micro-randomized trials for IS research
While the MRT experimental design advances experimental IS research by enabling the investigation of dynamic phenomena through causal effect estimations of dynamic relationships, as with any experimental design, this method entails some limitations. The advantages of the micro-randomization procedure are accompanied by challenges generally associated with longitudinal within-subject designs, such as learning effects (White and Arzi, 2005), as well as with challenges inherent to MRTs, such as the granularity of MRT results, technical feasibility, and potential user burden. We discuss these constraints subsequently.
First, learning effects occur when participants internalize knowledge from one treatment and apply their new knowledge when receiving the next treatment. This happens if, for example, later experimental tasks are performed more easily, because participants have become used to the task design and remember their earlier answers. Further, participants might respond differently to subsequent treatment tasks because they have become familiar with the content. If, however, the participant’s state of knowledge is included as a contextual variable in the MRT, examining its influence on the treatment’s impact can be part of the investigation. It should then be considered as an endogenous covariate in the data analysis (Qian et al., 2022). For some research settings, however, including the participant’s state of knowledge as a contextual variable may be impractical or even inadvertently affect the experimental treatment. For settings in which learning effects are to be minimized, MRTs are not applicable because learning effects cannot be avoided with the sequential randomization procedure of MRTs.
Second, while the WCLS method enables a robust estimation of the causal excursion effect of treatment options on proximal outcomes, it cannot be applied to compute the effects of treatment options and proximal outcomes on the distal outcome. Hence, while MRTs provide granular insights into treatment options’ effects on proximal outcomes, they cannot compare the effectiveness of the whole treatment package to other overall treatment packages. Therefore, MRTs are not able to assess which different approaches for achieving a desired distal outcome are most effective. For such comparisons, additional experiments, for example, large-scale RCTs, are required. It may also be feasible to rely on previous studies and domain knowledge to justify the link between proximal outcomes and the distal outcome instead of comparing it with other approaches experimentally.
Third, researchers conducting MRTs should possess full control over the treatment design to test either the per-protocol, the intent-to-treat, or both effects on the proximal outcome. To improve the robustness of results, the technical setup ideally allows for testing both effects by implementing a control mechanism after the treatment delivery. Yet, the need to constantly monitor, record, and store treatment outcomes together with contextual variables increases technological demands. Fulfilling these requirements is essential for MRT experiments, as the measurement of contextual variables is one of the core requirements of the MRT design. This recording and storing of real-time contextual data can pose technical and cost challenges in some research settings and may thus be impractical for some specific researcher endeavors. Further, depending on the data collection method, the recording of real-time data can also be an additional burden for participants which may confound the experimental treatment (Nahum-Shani et al., 2018). For example, daily reminders for participants to synchronize their phones could remind them that they are participating in an experiment, which is an undesirable effect since this could inadvertently affect their behavior and, thus, trigger effect-confounding reactions. Therefore, the data collection should be executed as unobtrusively as possible to avoid effect-confounding disturbances and reduce user burden during data collection, sometimes at the expense of high technical demands and costs.
Overall, MRTs are not a panacea to commonly encountered difficulties of experimental setups. The longitudinal design and its within-subject composition entails challenges such as spillover and learning effects due to which treatments from one sequence affect the treatment efficacy in following sequences. This might further cause endogeneity and unwanted interaction effects between different treatments. Some of these limitations can be addressed by applying suitable data analysis methods, such as the WCLS estimation method, and by including relevant control and confounding variables in the data analysis. Most importantly, however, such issues must be considered before setting up and running the MRT, thereby preemptively avoiding potential threats in the design. As such, it is important to assess the treatment design, data measurement approaches, and user burden carefully. The data collection should be performed as unobtrusively as possible. Also, treatments should be defined not only for a specific theoretical background and research model but also by incorporating practical knowledge and domain considerations. Further, while the MRT can provide granular and detailed insights and examines underlying causal chains of evidence, it cannot claim the effectiveness of the treatment package (or final IS artifact) as a whole. To compare the optimized component compilation with a control group or a previous compilation, an RCT and between-subject investigation should be conducted. Despite these limitations, MRTs can provide valuable contributions to the IS research community.
Contributions
With this article, we contribute to IS research in three key areas. First, we contribute to both IS literature and behavioral science literature by introducing MRTs as an experimental method for IS research. MRTs were originally developed in behavioral science to understand, test, and improve JITAIs (Nahum-Shani et al., 2018). For this application purpose, MRTs are typically performed by using mobile apps on smartphones, sometimes combined with additional sensors for data collection (Battalio et al., 2021). We propose that MRTs can essentially be conducted with any IS that interacts with its users and thus broaden the scope of MRTs to also be deployed for IS research purposes. We do so by outlining exemplary MRT research questions in three IS research domains (Neuro-IS, Cybersecurity, and AI Support), explaining the details of an MRT experiment conducted in the Health IS domain, and providing two additional conceptual examples (IS for Sustainability and Digital Platforms). We explicitly state for which research settings MRTs are not suitable and should thus be replaced by or complemented with other approaches and methods. Accordingly, we contribute to experimental IS literature (cf. Cahenzli et al., 2021; Fink, 2022; Gupta et al., 2018; Karahanna et al., 2018) by introducing a novel experimental method for IS research. We further contribute to behavioral science MRT literature (cf. Klasnja et al., 2015, 2019; Liao et al., 2020; Nahum-Shani et al., 2018; Qian et al., 2022; Walton et al., 2018) by expanding the scope of MRTs to the technology-driven IS research context.
Second, our paper is important for IS researchers who are interested in conducting MRTs. We provide a blueprint and step-by-step illustrative example that can be followed to set up, run, analyze, and report on an MRT experiment. This illustrative example is based on an IS research study in the Health IS domain (cf. Fallon et al., 2021). While the original goal of the presented study was to provide theoretical and practical contributions based on the outcomes of the conducted MRT, the current paper adds to this initial publication with a methodological contribution by structuring and subdividing the IS research process into eight distinct phases. When conducting MRTs, IS researchers should follow these eight phases: (1) set up the MRT research requirements; (2) define the instantiation of MRT components; (3) assess the MRT’s feasibility; (4) conduct a pre-test to compute the required MRT sample size; (5) collect the MRT data; (6) analyze the MRT data; (7) evaluate the robustness of MRT results; and (8) communicate the insights and discuss the contributions based on the MRT results.
For each of the eight phases, we have presented the steps undertaken in the respective example. Additionally, we elaborate on important considerations that are relevant to MRT research in general. Such considerations include, for example, the necessary components of the research context and questions, or the specifications for interpreting MRT results and insights. In addition to these considerations, we also provide auxiliary information that was not included with the initial publication but is relevant for comprehending the underlying MRT research process. This includes, for instance, details on the conducted pre-test and resulting sample size calculations, background knowledge on the logic of the WCLS data analysis method, as well as the presentation and discussion of robustness checks that were conducted for but not presented with the initial publication of the study. Based on this illustration and the detailed explanations of MRT requirements and components, we are confident that IS researchers are now better equipped to conduct MRTs in their research domains.
Third, with the illustrative example, we also demonstrate how novel theoretical insights can be gained by employing MRTs because they allow for consideration of the dynamic impact of contextual and personal factors over time. By discussing the advantages of the MRT experimental design, we elaborate that MRTs can be employed to refine and extend theories by incorporating important dynamic contextual and personal factors in the MRT data analysis. We explicitly highlight that if the MRT conceptual framework is based on and constructed with the help of a chosen theory or theoretical framework, then this MRT experiment allows for testing and potentially extending or refining this theory, often with dynamic components that could previously not be considered in experimental research due to the limited toolbox available for IS researchers. Accordingly, our paper provides a methodology that allows IS researchers to refine and extend theories by considering dynamic phenomena in our digitalized world.
Conclusion
Currently, the most common experimental designs in IS research are individual level confirmatory lab experiments with between-subject compositions (Cahenzli et al., 2021). Yet, researchers have called for an increased use of different experimental designs that better account for the complexities of our digitalized world (Fink, 2022; Gupta et al., 2018; Karahanna et al., 2018). We answer this call and demonstrate that IS researchers can leverage the MRT experimental design to investigate dynamic IS phenomena and refine existing theories.
The MRT experimental design is considerably novel because it relies on technological advancements, specifically ubiquitous computing and individualized digital devices, which have only become widely available in the past decade. The unique, micro-randomized, mixed-subject design of MRTs delivers reliable outcomes with smaller sample sizes than comparable between-subject or factorial designs. The online treatment delivery and digital recording of experimental data of MRTs further provides access to larger sample populations for improved sample selection procedures and an easier definition of experimental boundary conditions. Thus, the MRT experimental design can be applied in a variety of IS research domains to investigate human–IT interaction, online behavior, or the influence of IS use on human behavior and vice versa. By following the eight phases of the MRT research process, IS researchers are enabled to contribute to IS theory, practice, and research by acknowledging the dynamic impact of time-varying contextual and personal factors of many IS research phenomena. Despite some constraints, such as the inherent granularity of MRT results, MRTs allow us to achieve research goals that until now, considering the standard IS method assortment, could not be properly addressed. Supposing that MRTs are applicable to an array of research problems in our discipline, we hope to encourage IS researchers to apply the MRT experimental design in a variety of contexts and settings.
Supplemental Material
Supplemental Material - Micro-randomized trials in Information Systems research: An experimental method for advancing knowledge about our dynamic and digitalized world
Supplemental Material for Micro-randomized trials in Information Systems research: An experimental method for advancing knowledge about our dynamic and digitalized world by Mechthild Pieper, Monica Fallon, and Armin Heinzl in Journal of Information Technology.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is supported by Deutsche Forschungsgemeinschaft (492693936).
Supplemental Material
Supplemental material for this article is available online.
Notes
Author biographies
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
