A Multidimensional Array Representation of State-Transition Model Dynamics

Abstract

Cost-effectiveness analyses often rely on cohort state-transition models (cSTMs). The cohort trace is the primary outcome of cSTMs, which captures the proportion of the cohort in each health state over time (state occupancy). However, the cohort trace is an aggregated measure that does not capture information about the specific transitions among health states (transition dynamics). In practice, these transition dynamics are crucial in many applications, such as incorporating transition rewards or computing various epidemiological outcomes that could be used for model calibration and validation (e.g., disease incidence and lifetime risk). In this article, we propose an alternative approach to compute and store cSTMs outcomes that capture both state occupancy and transition dynamics. This approach produces a multidimensional array from which both the state occupancy and the transition dynamics can be recovered. We highlight the advantages of the multidimensional array over the traditional cohort trace and provide potential applications of the proposed approach with an example coded in R to facilitate the implementation of our method.

State-transition models (STM) are decision models commonly used in cost-effectiveness analysis (CEA) to estimate economic and health outcomes of different strategies over time in discrete time cycles.^1,2 In a cohort state-transition model (cSTM), the disease dynamics are captured by distributing a closed cohort among a mutually exclusive and collectively exhaustive set of health states.^2–4 The cohort trace is the primary outcome of cSTMs, which comprises the proportion of the cohort in each health state over time (i.e., it summarizes state occupancy).^1,5 A limitation of the cohort trace is that it does not keep track of the transitions among health states over time (i.e., the transition dynamics of the cohort). As a consequence, it can only be used to capture outcomes that result from residing in a state for a full cycle by applying the so-called state rewards and does not contain a mechanism to assign transition rewards, which are applied only when specific transitions occur. It also limits the type of epidemiological outcomes that can be obtained from cSTMs. For example, obtaining incidence of a disease requires knowledge of the proportion of the population transitioning from a subset of states without disease to the state(s) representing the disease of interest. ⁶

To overcome the limitations of the cohort trace, we propose a multidimensional array–based approach that serves as a full summary of cSTM dynamics that complements the already useful cohort trace. The proposed approach, called the dynamics-array approach, allows modelers to efficiently calculate all measures of interest that rely on transition dynamics and at the same time to aggregate all model dynamics into a standard cohort trace.

We start by providing a formal definition of cSTM components and the cohort trace. We complement this standard notation with a description of the detailed transition dynamics. Then, we introduce the multidimensional array structure and show how it can be easily generated. In addition, we illustrate its use to compute a measure of interest that depends on transitions among health states. Finally, we demonstrate the dynamics-array approach with an illustrative example of a cSTM programmed in R^7,8 and compare this implementation with the traditional cohort trace approach in a simulation study. The R code is provided in the Supplementary Material and in GitHub (https://github.com/DARTH-git/state-transition-model-dynamics).

Traditional Cohort Trace Approach

We denote the distribution of the cohort across $n_s$ health states in a cSTM at the beginning of cycle $t$ for all $t$ in $0,\dots ,n_t$ as the state vector ${\mathbf{m}}_t$ of dimensions $1\times n_s$ . That is, each element in ${\mathbf{m}}_t$ represents the proportion of the cohort in health state $i=1,\dots ,n_s$ at time $t$ . Thus, ${\mathbf{m}}_t$ is written as follows:

$${\mathbf{m}}_t=\left[\begin{array}{cccc}{m}_{[t,1]}\hfill & m_{[t,2]}\hfill & \cdots \hfill & m_{[t,n_s]}\hfill \end{array}\right],$$

(1)

where the initial state vector ${\mathbf{m}}_0$ contains the distribution of the cohort across all $n_s$ health states at the start of the simulation. The simulation begins at cycle 0 and all transitions are assumed to happen at the end of each cycle. The probability of transitioning from health state $i$ to health state $j$ at the end of cycle $t$ is denoted as $p_{[i,j,t]}$ . This means that $p_{[i,j,t]}$ determines how the population will be distributed in cycle $t+1$ . The collection of transition probabilities across the model states over the time horizon forms the time-dependent state-transition probability matrix, $P_t$ of dimensions $n_s\times n_s$ :

$$P_t=\left[\begin{array}{cccc}{p}_{[1,1,t]}\hfill & p_{[1,2,t]}\hfill & \cdots \hfill & p_{[1,n_s,t]}\hfill \\ \multicolumn{1}{c}{p_{[2,1,t]}}& p_{[2,2,t]}\hfill & \cdots \hfill & p_{[2,n_s,t]}\hfill \\ \multicolumn{1}{c}{⋮}& ⋮\hfill & \ddots \hfill & ⋮\hfill \\ \multicolumn{1}{c}{p_{[n_s,1,t]}}& p_{[n_s,2,t]}\hfill & \cdots \hfill & p_{[n_s,n_s,t]}\hfill \end{array}\right].$$

(2)

For any $t$ , all rows of $P_t$ must sum to 1. Note that if $P_t$ is equal for all $t$ times, equation (2) becomes a time-homogeneous transition probability matrix, where $P_t=P$ .

The state vector at cycle $t+1$ , ${\mathbf{m}}_{t+1}$ , is then obtained by the inner product between the state vector at cycle $t$ , ${\mathbf{m}}_t$ , and the corresponding transition probability matrix $P_t$ , such that

$${\mathbf{m}}_{\mathbf{t}+1}={\mathbf{m}}_{\mathbf{t}}{P}_t\mathrm{for}t=0,\dots ,(n_t-1).$$

(3)

Stacking the state vectors by rows for all $t=0,\dots ,n_t$ results in the full cohort trace matrix, $M$ , of dimensions $(n_t+1)\times n_s$ , where each row is a state vector $(-{\mathbf{m}}_t-)$ , resulting in

$$M=\left[\begin{array}{c}-{\mathbf{m}}_0-\hfill \\ \multicolumn{1}{c}{-{\mathbf{m}}_1-}\\ \multicolumn{1}{c}{⋮}\\ \multicolumn{1}{c}{-{\mathbf{m}}_{n_t}-}\end{array}\right].$$

(4)

Together, the state vectors ${\mathbf{m}}_t$ , the transition probability matrices $P_t$ , and the cohort trace $M$ in equations (1) to (4), respectively, represent the 3 main components of a cSTM.

Dynamics-Array Approach

The trace matrix $M$ aggregates transitions from all the states to a specific state, thus loses details of the transition dynamics. We propose using a multidimensional array, $\mathbf{A}$ , of dimensions $n_s\times n_s\times (n_t+1)$ to store the proportion of the cohort that transitions between any 2 health states in each cycle over the time horizon. This array can be thought of as a set of 2-dimensional matrices stacked along a third dimension that represents time. Below, we illustrate how to compute $\mathbf{A}$ from the 3 main components, the state vector ${\mathbf{m}}_t$ , the transition probability matrices $P_t$ , and the cohort trace $M$ , of cSTMs.

$A_0$ represents the first “slice” of $\mathbf{A}$ . We compute $A_0$ as a matrix containing the initial state vector ${\mathbf{m}}_0$ in its diagonal and 0s in the off-diagonal, such that

$$A_0=\mathrm{diag}({\mathbf{m}}_0)=\left[\begin{array}{cccc}{m}_{[0,1]}\hfill & 0\hfill & \cdots \hfill & 0\hfill \\ \multicolumn{1}{c}{0}& m_{[0,2]}\hfill & \cdots \hfill & 0\hfill \\ \multicolumn{1}{c}{⋮}& ⋮\hfill & \ddots \hfill & ⋮\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & \cdots \hfill & m_{[0,n_s]}\hfill \end{array}\right].$$

(5)

Each subsequent $(t+1{)}^{\mathrm{th}}$ “slice” of $\mathbf{A}$ is obtained by multiplying a diagonal matrix of ${\mathbf{m}}_t$ , denoted as diag( ${\mathbf{m}}_t$ ), by $P_t$ , such that

$${\mathbf{A}}_{t+1}=\mathrm{diag}({\mathbf{m}}_t)·P_t\mathrm{for}t=0,\dots ,(n_t-1).$$

(6)

The resulting elements of the t^th slice of $\mathbf{A}$ , $A_t$ for $t>0$ , are

$$A_t=\left[\begin{array}{cccc}{a}_{[1,1,t]}\hfill & a_{[1,2,t]}\hfill & \cdots \hfill & a_{[1,n_s,t]}\hfill \\ \multicolumn{1}{c}{a_{[2,1,t]}}& a_{[2,2,t]}\hfill & \cdots \hfill & a_{[2,n_s,t]}\hfill \\ \multicolumn{1}{c}{⋮}& ⋮\hfill & \ddots \hfill & ⋮\hfill \\ \multicolumn{1}{c}{a_{[n_s,1,t]}}& a_{[n_s,2,t]}\hfill & \cdots \hfill & a_{[n_s,n_s,t]}\hfill \end{array}\right],$$

(7)

where $a_{[i,j,t]}$ is the proportion of the cohort that transitions from state $i$ to state $j$ between cycles $t-1$ and $t$ , generated via

$$a_{[i,j,t]}=m_{[t-1,i]}{p}_{[i,j,t-1]}\mathrm{for}t=1,\dots ,(n_t-1),$$

(8)

where $m_{[t-1,i]}$ is the proportion of the cohort in state $i$ at cycle $t-1$ and $p_{[i,j,t-1]}$ the corresponding transition probability of transitioning from state $i$ to state $j$ at the end of cycle $t-1$ . In other words, $\mathbf{A}$ stores the transition dynamics of a simulated cohort in a cSTM.

Figure 1 presents graphically the computation involved in both (a) the traditional cohort trace approach and (b) the dynamics-array approach and shows the structures of (c) the resulting cohort trace $M$ and dynamics-array A.

Figure 1

(a) The cohort trace approach computes each row vector ${\mathbf{m}}_{t+1}$ of the cohort trace $M$ , where ${\mathbf{m}}_{t+1}$ describes the distribution of the simulated cohort among different health states at time $t+1$ . ${\mathbf{m}}_{t+1}$ results from multiplying the state vector ${\mathbf{m}}_t$ (gray) by the transition probability matrix $P_t$ . (b) The dynamics-array approach computes matrix $A_{t+1}$ containing information regarding the transition dynamics of the simulated cohort at time $t+1$ . The state vector ${\mathbf{m}}_t$ is highlighted (gray) to emphasize that the information in both approaches is identical. (c) The resulting matrix $M$ and array $\mathbf{A}$ of the approaches (a) and (b), respectively.

In R, it takes only a few lines of code to generate $\mathbf{A}$ complementary to M (Box 1).

Box 1

R code to iteratively generate the cohort trace $M$ and the dynamics array A

The cohort trace $M$ can be computed from A by obtaining the t^th row of $M$ , ${\mathbf{m}}_t$ , summing each of the columns of ${\mathbf{A}}_t$ as follows:

$${\mathbf{m}}_t={\mathbf{1}}^T{\mathbf{A}}_t=[\sum _{i=1}^{n_s}{a}_{[i,1,t]},\sum _{i=1}^{n_s}{a}_{[i,2,t]},\dots ,\sum _{i=1}^{n_s}{a}_{[i,n_s,t]}],$$

(9)

where $1$ is a vector of ones of dimension $n_s\times 1$ . Although $M$ can be obtained from $\mathbf{A}$ , we prefer to compute both $M$ and $\mathbf{A}$ simultaneously. Once generated, both $M$ and $\mathbf{A}$ can be exported as data objects that contain all the information about the cSTM dynamics, which can then be used in future calculation of outcomes of interest.

Applying State and Transition Rewards

One of the main advantages of $\mathbf{A}$ over $M$ is the ability to incorporate transition rewards. Here, we demonstrate how to apply both state and transition rewards (e.g., utilities or cost) to the cSTM by using the dynamics array, $\mathbf{A}$ . Let $R_t$ be a reward matrix of dimensions $n_s\times n_s$ that contains both state and transition rewards:

$$R_t=\left[\begin{array}{cccc}{r}_{[1,1,t]}\hfill & r_{[1,2,t]}\hfill & \cdots \hfill & r_{[1,n_s,t]}\hfill \\ \multicolumn{1}{c}{r_{[2,1,t]}}& r_{[2,2,t]}\hfill & \cdots \hfill & r_{[2,n_s,t]}\hfill \\ \multicolumn{1}{c}{⋮}& ⋮\hfill & \ddots \hfill & ⋮\hfill \\ \multicolumn{1}{c}{r_{[n_s,1,t]}}& r_{[n_s,2,t]}\hfill & \cdots \hfill & r_{[n_s,n_s,t]}\hfill \end{array}\right],$$

(10)

where $r_{[i,j,t]}$ is the reward associated with transitioning from state $i$ to state $j$ at the end of cycle $t$ . When $j=i$ , $r_{[i,i,t]}$ is the reward associated with staying in the i^th health state at cycle $t$ . That is, the off-diagonal entries of $R_t$ store the transition rewards and the diagonal of $R_t$ stores the state rewards for cycle $t$ . Note that if $R_t$ is equal for all $t$ times (i.e., neither state nor transition rewards vary over time), equation (10) becomes a time-homogeneous rewards matrix, where $R_t=R$ .

The state and transition rewards can be applied to the model dynamics by element-wise multiplication between $A_t$ and $R_t$ , indicated by the ⊙ sign, which produces the matrix of outputs at cycle $t$ , $Y_t$ . Formally,

$$Y_t=A_t\odot R_t.$$

(11)

With this approach, the state rewards are accounted for at the beginning of the cycle and the transition rewards—assumed to happen at the end of the cycle—are accounted for at the next cycle.

In R, applying these rewards required 1 additional line of code compared with Box 1, as shown in line 5 of Box 2.

Box 2

R code to apply time-invariant state and transition rewards to the model dynamics stored in array A

The total rewards for each health state at cycle $t$ , ${\mathbf{r}}_t$ , is obtained by summing the rewards across all $j=1,...,n_s$ health states:

$${\mathbf{r}}_t=1^T{Y}_t=[\sum _{i=1}^{n_s}{Y}_{[i,1,t]},\sum _{i=1}^{n_s}{Y}_{[i,2,t]},\dots ,\sum _{i=1}^{n_s}{Y}_{[i,n_s,t]}].$$

(12)

Implementation in R Using an Illustrative Example

To facilitate the implementation of the dynamics-array approach, we demonstrate its use with a stylistic healthy-sick-dead 3-state time-homogeneous cSTM example coded in R. ⁸ The model is used to simulate a cohort of 70-year-old individuals to compute their expected costs and quality-adjusted life years (QALYs) accrued over their remaining lifetime accounting for several transition rewards. Accounting for transition rewards with the traditional cohort trace approach is possible; however, this requires creating additional temporary health states that keep track of the transitions. For our simple 3-state model, this already requires 2 additional temporary states. In more complex models, accounting for transition rewards will result in state explosion, and consequently, it is more likely to make errors while coding these models. The explanation of the state expansion and the R code for the traditional cohort trace approaches for our stylistic model can be found in the Supplementary Material and on GitHub (https://github.com/DARTH-git/state-transition-model-dynamics). GitHub also provides some code that shows that both approaches give identical results.

Comparison of Methods Using a Simulation Study

We conducted a simulation study to compare the computation time and memory requirements of the dynamics-array and traditional cohort trace approaches. We created a full factorial $(13\times 110)$ design of experiment with the number of health states (from 2 to 62 with increments of 5 states) and the number of cycles (from 12 to 1320 with increments of 12 cycles) in a cSTM as the factors of the simulation study. We ran this full factorial experiment 10 times and took the average of the required time and memory to smooth out the variations in the computation time of R. Correcting for variation in R computation time is important, because the total required time is small (<50 seconds) and even a small variation (e.g., 3–5 seconds) could affect our results. Figure 2 shows that the dynamics-array approach is both faster (140-times with 62 health states; Figure 2, top) and requires less memory than the traditional cohort trace approach as the number of states increases (Figure 2, bottom). A more detailed description of the output of the simulation study is included in the Supplementary Material and the code is available on GitHub.

Figure 2

Computation time and memory storage of the 2 approaches as a function of the number of states when running the model for 1320 cycles. The top left panel shows the absolute computation time in seconds of both approaches. The top right panel shows the relative speedup of the dynamics-array approach compared with the traditional cohort trace approach. The horizontal line at y-axis equals 0 indicating that the 2 approaches are equally fast. The bottom left panel shows the absolute memory storage in megabytes (MB) of the 2 approaches, while the bottom right panel shows the relative required memory of the dynamics-array approach compared with the traditional cohort trace approach. The horizontal line at y-axis equals 1 indicates that both approaches required the same memory storage. Above the line the traditional cohort trace requires less memory, while below the line the dynamics-array approach requires less memory. All results are based on the average of 10 simulations. This was done to smooth out the variations caused by the computation time of R.

Estimation of Epidemiological Measures

By obtaining $\mathbf{A}$ , it is possible to compute epidemiological outcomes that otherwise would not have been easily derived from $M$ . For example, obtaining incidence and lifetime risk from $M$ would require creating additional steps, variables, or health states. Epidemiological outcomes could be used as outputs of simulation models for calibration or validation purposes. A full exposition of computing epidemiological measures from $\mathbf{A}$ is case-specific and is beyond the scope of this brief report. However, we illustrate the potential application of our approach with an example below.

Consider a cSTM with $n_s>3$ health states. We are interested in calculating a ratio $e_t$ of those that transition from health state 2 to health state $n_s$ at cycle $t$ to those that make the transition to health state $n_s$ from health states 1, 2, and 3. Using the dynamics-array approach, the ratio $e_t$ can be computed as follows:

$$e_t=\frac{a_{[2,n_s,t]}}{a_{[1,n_s,t]}+a_{[2,n_s,t]}+a_{[3,n_s,t]}}\mathrm{for}t=1,\dots ,n_t.$$

(13)

With the traditional cohort trace approach, calculating this ratio would require adding 3 temporary health states to distinguish those that transition to $n_s$ from the health states 1, 2, and 3.

Discussion

We propose a multidimensional array approach to overcome a limitation of the cohort trace produced by cSTM in not being able to store transition dynamics. The practical application of our approach involves adding a simple step to the traditional cohort trace approach that stores all transitions among health states over time in multidimensional array $\mathbf{A}$ . We described the multidimensional array approach for a general cSTM where transitions are allowed from any state to others within a cycle, but our approach can also be applied to models that only allow one-state transition in a cycle (i.e., where the $p_{i,j,t}$ are 0 for those transitions that are not allowed).

Traditionally, researchers have dealt with this limitation of the cSTM cohort trace by creating temporary health states that collect the state-to-state transition information. However, as we showed in our “Comparison of Methods Using a Simulation Study” section, this solution can quickly complicate a model and result in an explosion of the number of health states. Using an individual-based microsimulation STM is another alternative, ¹ with considerable implications on computational time. ⁹

Another method that explicitly keeps track of state-to-state transitions is through a discretely integrated condition-event (discretely integrated condition event [DICE]) simulation.^10,11 DICE is a modeling technique that can free up some of the Markov restrictions that makes it possible to explicitly include many events occurring at various times. Although DICE simulation is a well-structured method, and the authors of the DICE papers provided very useful supplementary files to apply the method, we see the dynamics-array approach as a relatively simpler method to compute than DICE to overcome the limitation of the cohort trace on applying transition rewards and generating all the epidemiological outcomes of interest.

A potential limitation of the use of $\mathbf{A}$ is the additional computation needed when building the model. However, for many applications this may be a minor limitation given the matrix-based computational efficiency of current computers. Another potential limitation is the additional storage memory required to store $\mathbf{A}$ , which could become a limitation in systems with limited memory. This could be an issue for computationally complex models with multiple states. However, the benefit of using $\mathbf{A}$ for large models is that all the complexity in the model dynamics is summarized into a compact structure which makes it relatively simple to extract information or to apply new rewards without re-running the model.

In conclusion, structuring the output of cSTMs using the dynamics-array approach is an efficient, simple, and convenient method of summarizing the model dynamics. This simple structure allows applying state and transition rewards and obtaining epidemiological measures while still being able to obtain and display the conventional cohort trace.

Supplemental Material

2D_time_memory_relative – Supplemental material for A Multidimensional Array Representation of State-Transition Model Dynamics

Supplemental material, 2D_time_memory_relative for A Multidimensional Array Representation of State-Transition Model Dynamics by Eline M. Krijkamp, Fernando Alarid-Escudero, Eva A. Enns, Petros Pechlivanoglou, M.G. Myriam Hunink, Alan Yang and Hawre J. Jalal in Medical Decision Making

Supplemental Material

3D_memory – Supplemental material for A Multidimensional Array Representation of State-Transition Model Dynamics

Supplemental material, 3D_memory for A Multidimensional Array Representation of State-Transition Model Dynamics by Eline M. Krijkamp, Fernando Alarid-Escudero, Eva A. Enns, Petros Pechlivanoglou, M.G. Myriam Hunink, Alan Yang and Hawre J. Jalal in Medical Decision Making

Supplemental Material

3D_time – Supplemental material for A Multidimensional Array Representation of State-Transition Model Dynamics

Supplemental material, 3D_time for A Multidimensional Array Representation of State-Transition Model Dynamics by Eline M. Krijkamp, Fernando Alarid-Escudero, Eva A. Enns, Petros Pechlivanoglou, M.G. Myriam Hunink, Alan Yang and Hawre J. Jalal in Medical Decision Making

Supplemental Material

arxiv – Supplemental material for A Multidimensional Array Representation of State-Transition Model Dynamics

Supplemental material, arxiv for A Multidimensional Array Representation of State-Transition Model Dynamics by Eline M. Krijkamp, Fernando Alarid-Escudero, Eva A. Enns, Petros Pechlivanoglou, M.G. Myriam Hunink, Alan Yang and Hawre J. Jalal in Medical Decision Making

Supplemental Material

STM_diagram_array – Supplemental material for A Multidimensional Array Representation of State-Transition Model Dynamics

Supplemental material, STM_diagram_array for A Multidimensional Array Representation of State-Transition Model Dynamics by Eline M. Krijkamp, Fernando Alarid-Escudero, Eva A. Enns, Petros Pechlivanoglou, M.G. Myriam Hunink, Alan Yang and Hawre J. Jalal in Medical Decision Making

Supplemental Material

STM_diagram_cohort_approach – Supplemental material for A Multidimensional Array Representation of State-Transition Model Dynamics

Supplemental material, STM_diagram_cohort_approach for A Multidimensional Array Representation of State-Transition Model Dynamics by Eline M. Krijkamp, Fernando Alarid-Escudero, Eva A. Enns, Petros Pechlivanoglou, M.G. Myriam Hunink, Alan Yang and Hawre J. Jalal in Medical Decision Making

Supplemental Material

STM_diagram_traditional_approach – Supplemental material for A Multidimensional Array Representation of State-Transition Model Dynamics

Supplemental material, STM_diagram_traditional_approach for A Multidimensional Array Representation of State-Transition Model Dynamics by Eline M. Krijkamp, Fernando Alarid-Escudero, Eva A. Enns, Petros Pechlivanoglou, M.G. Myriam Hunink, Alan Yang and Hawre J. Jalal in Medical Decision Making