Reinforcement Learning-Based Control of Epidemics on Networks of Communities and Correctional Facilities

Abstract

Background

Correctional facilities can act as amplifiers of infectious disease outbreaks. Small community outbreaks can cause larger prison outbreaks, which can in turn exacerbate the community outbreaks. However, strategies for epidemic control in communities and correctional facilities are generally not closely coordinated. We sought to evaluate different strategies for coordinated control.

Methods

We developed a stochastic simulation model of an epidemic spreading across a network of communities and correctional facilities. We parameterized it for the initial phases of the COVID-19 epidemic for 1) California communities and prisons based on community data from covidestim, prison data from the California Department of Corrections and Rehabilitation, and mobility data from SafeGraph, and 2) a small, illustrative network of communities and prisons. For each community or prison, control measures were defined by the intensity of 2 activities: 1) screening to detect and isolate cases and 2) nonpharmaceutical interventions (e.g., masking and social distancing) to reduce transmission. We compared the performance of different control strategies including heuristic and reinforcement learning (RL) strategies using a reward function, which accounted for both the benefit of averted infections and nonlinear cost of the control measures. Finally, we performed analyses to interpret the optimal strategy and examine its robustness.

Results

The RL control strategy robustly outperformed other strategies including heuristic approaches such as those that were largely used during the COVID-19 epidemic. The RL strategy prioritized different characteristics of communities versus prisons when allocating control resources and exhibited geo-temporal patterns consistent with mitigating prison amplification dynamics.

Conclusion

RL is a promising method to find efficient policies for controlling epidemic spread on networks of communities and correctional facilities, providing insights that can help guide policy.

Highlights

• For modelers, we developed a stochastic simulation model of an epidemic spreading across a network of communities and correctional facilities, and we parameterized it for the initial phases of the COVID-19 epidemic for California communities and prisons in addition to an illustrative network.

• We compared different control strategies using a reward function that accounted for both the benefit of averted infections and cost of the control measures; we found that reinforcement learning robustly outperformed the other strategies including heuristic approaches such as those that were largely used during the COVID-19 epidemic.

• For policy makers, our work suggests that they should consider investing in the further development of such methods and using them for the control of future epidemics.

• We offer qualitative insights into different factors that might inform resource allocation to communities versus prisons during future epidemics.