Sage Journals: Discover world-class research

Abstract

This paper advocates for causal models of the emergency management cycle (EMC) for decision support in epidemic or pandemic scenarios. The model capability is demonstrated for the case of the COVID-19 attack at the NATO flagship USS Theodore Roosevelt in early 2020. Computational intelligence is a reasonable approach for dealing with uncertainties such as low reliability of information and source credibility. The proposed EMC causal models enable the development of countermeasures for epidemiological attacks using the notion of gaps in the four EMC phases: mitigation, preparedness, response, and recovery. In particular, the EMC problem can be formulated and formalized as bridging the identified technology–society gap, e.g., mitigation of risks and biases; and machine reasoning can be incorporated at any level of the EMC decision-making. Using available real-world data on the USS Theodore Roosevelt outbreak, we show how machine reasoning mechanisms can help the captain to make more reliable decisions in critical epidemiological situations.

Keywords

Emergency management cycle epidemiological surveillance decision support machine reasoning causal networks

1. Introduction

An epidemic attack in a military unit is rather a “closed set” scenario. On the other hand, it is a part of a global epidemiological situation characterized by a rapid spread of disease to a large number of people in a given population, or region, within a short period of time.

The recent COVID-19 attack on the NATO flagship, USS Theodore Roosevelt (USS TR), in early 2020 prompted the discussion about the decisions made under uncertainty such as unknown infection fatality and prevalence rates. One fact is certain: no situational awareness support based on reliable scientific data analysis was available to the captain.

This paper aims at answering the question “How can computational intelligence (CI) help the leadership team of military units such as the USS TR make reliable decisions under critical pandemic attacks?”

Logistics and infrastructure in military units, unlike civil communities, are closed but robust. This makes it simpler to implement various countermeasure technologies, such as tracing crew members and their social contacts. These are unique possibilities for monitoring and control of the epidemiological situation. In this paper, we make use of those constraints in our approach to building an intelligent decision support system (DSS) that would assist the captains in making reliable decisions in critical epidemiological situations.

The COVID-19 pandemic highlighted many technologies and social gaps in the emergency management cycle (EMC),¹ a mechanism to respond to disasters such as pandemics. Those gaps are yet to be addressed by designing the DSS for various stages of the EMC.

Our results are introduced as follows: (1) we provide a background and offer a brief overview of EMC in fighting epidemics and pandemics; (2) we identify gaps in the existing EMC and explain the core of decision support for EMC; and (3) we introduce the DSS model for the case of the USS TR outbreak and provide experimental results.

2. Background

2.1. Epidemiological management doctrine

Figure 1 illustrates the contemporary epidemiological management doctrine. Decision at any phase of the EMC is based on information provided by the following technologies:

current epidemiological situation (state) profiling technologies called epidemiological surveillance (ES);

proactive epidemiological situation profiling technologies called epidemiological intelligence (EI).

Figure 1.

Epidemic attack detection, recognition, and countermeasure according to the contemporary epidemiological doctrine: joint landscape is provided (1) by proactive profiling using EI resources and (2) current state profiling using ES resources through the EMC phases. The source of the USS TR image is Wikipedia.

Given a military unit such as a USS TR under an epidemic attack, the captain has to make a decision based on two kinds of reported data: (1) on the current state of the crew, e.g., how many cases of deceases related epidemic are confirmed,² and (2) on early warnings of public health emergencies and potential threats.³

2.1.1. ES models

The COVID-19 outbreak provided valuable lessons and unveiled critical disadvantages of the existing ES models. The ES model’s outcomes need to be further translated to become usable for human decision-makers. There is a technology gap between the existing models and the decision-making process, and it can be filled in by CI support in the decision-making process. The CI tool needs to be based on causal models that account for uncertainties, as well as perform fusion and forecasting on those uncertainties.

In this study, we identify the gaps in the EMC in both technical and conceptual domains (Figure 2). This paper addresses the key research question: How can we bridge these gaps using the DSS concept? We follow a well-identified trend in academic discussion on the future generation DSS.⁴

Figure 2.

Technological gaps in the EMC between all causal phases: (1) mitigation (preventing steps), (2) preparedness (precaution measures), (3) response (immediate actions), and (4) recovery (actions to return to normal mode). Source of the USS Theodore Roosevelt image is Wikipedia.

The answers to the above challenges lie in the usage of the CI approach called machine reasoning, namely, causal models such as causal networks. These causal models operate using probabilities, thus accounting for uncertainties, and enable knowledge inference based on priors and evidence.⁵ This approach has been applied to risk assessment in multiple areas of engineering and business,⁶ risk profiling in identity management,^7,8 medical diagnostics,⁹ and very recently to the analysis of COVID-19 risks such as fatality and disease prevalence rates.¹⁰

The causal models shall be the core component of the system that will support human decisions made during an infectious disease outbreak by assessing the different risks associated with the outbreak.

The system concept was once known as an “expert system.” Such a system provides certain automation of reasoning (though mainly based on deterministic rules rather than causal network approach) and interpretation strategies to extend experts’ abilities to apply their strategies.¹¹ Examples of contemporary DSS include personal health monitoring systems¹² and e-coaching for health.¹³

2.1.2. EI models

EI is a systematic event detection aimed to detect early warning signs of public health emergencies.²

Epidemiological models^14,15 are based on mathematical models that utilize both the biological knowledge and mechanism of the epidemic or pandemic and the available statistical data that represent the phenomenological part of the model. They aim at estimating the outbreak risks and transmission routes, incubation and infectious periods and their statistical distributions, length of latency, inferred transmission chains, etc. They are generally divided into individual-based models (based on individual characteristics such as age, spatial location, gender, and immune status), and compartmental models (based on the states of subjects being susceptible, infectious, immune, and recovered).

2.2. Introduction to the USS TR outbreak

2.2.1. Epidemiological measures

The infectious disease risk known as infection prevalence rate (IPR), is defined as the proportion of the number of current cases at a specific point of time to the population at that time¹⁶:

IPR = \frac{Number of cases at a specific time}{Population at that time}

(1)

A metric for measuring the seriousness of the disease is the case fatality rate (CFR). It is defined as follows:

CFR = \frac{Number of fatalities due to disease}{Number of people with disease}

(2)

The infectious disease risk called an attack rate, also known as incidence rate, is defined as the proportion of the new cases at a certain point in time to the population at that time¹⁶:

\begin{matrix} Attack rate = \\ \frac{Number of new cases at a specific time}{Population at risk at that time} \end{matrix}

(3)

Note that the population at risk is a fraction of the population, i.e., those who are not yet infected.

2.2.2. Epidemiological situation and commander’s response

The first three subjects have self-identified themselves; they have lost the sense of smell, but were sent back to work; three others tested positive and were quarantined. On March 25, a total of four infected sailors were taken away by a helicopter. We assume that only infected sailors were quarantined in the following few days, as indicated in the denominator of the attack rates in Table 1. On March 29, the captain lifted the quarantine. As of May 5, 1156 crew members have tested positive. This is about 24% of the initial population; taking into account evacuations, this number might be higher.

Table 1.

Observed infection risks within the first six days of the first identified cases at the USS TR.

Date	New cases	IPR	Attack rate
03/24	3	3/4800×100=0.06%	3/4800×100=0.06%
03/25	5	8/4800×100=0.17%	5/4797×100=0.10%
03/26	25	33/4800×100=0.69%	25/4789×100=0.52%
03/27	3	36/4800×100=0.75%	3/4756×100=0.06%
03/28	10	46/4800×100=0.96%	10/4720×100=0.21%
03/29	7	53/4800×100=1.10%	7/4674×100=0.15%
05/5		1156/4800×100=24.08%

2.2.3. Critical analysis

Critical analysis of epidemiological attack (Table 1) and the Commander’s response led to the following conclusions.

The commander made decisions under deep uncertainties such as unknown infection, unknown spreading characteristics, unknown fatality rate, and unknown period of contagion.

The commander was in need of reliable decision-making support when given uncertain, incomplete, imprecise, and possibly conflicting information, both qualitative and quantitative. Thus, the modeling and simulation (M&S) approach is the only way to help the commanders better manage situations under uncertainty.

2.2.4. Vision of horizon

Our work makes a step forward to prepare for future epidemic attacks and proposes an intelligent DSS with M&S of various epidemiological scenarios. We project the EMC onto the DSS life cycle; we state that such DSS must be operational at the mitigation, preparedness, response, and recovery modes distinguished in the EMC doctrine. According to the DARPA’s program,¹⁷ intelligent tools such as the DSS, must satisfy the requirements such as interpretability and explainability, i.e., the recommended decisions should be understandable to the user. This DSS has been created using the data of the current COVID-19 attack of the USS TR, and can potentially be instrumental in assessing the epidemiological risks for both military and civil “closed” communities such as ships or isolated city clusters. We experimentally show how a commander can be supported in a few typical scenarios such as assessing the effectiveness of the preventive measures and other analyses of the epidemiological landscape.

3. Problem formulation and contribution

The struggle with the COVID-19 outbreak worldwide revealed many technology–society gaps in the EMC. This paper contributes to bridging the gaps, and consider a particular pandemic case in a closed-set scenario such as the case of the NATO flagship USS TR. The management decisions have been widely discussed in the media and some data have been made available.

Central research question: How can intelligent DSS help management teams make reliable decisions in the case of epidemic attacks?

We propose the following strategy to answers the above question (Figure 3):

Step 1: Problem formulation as the EMC causality landscape.

Step 2: Problem specification as gap identification.

Step 3: Problem description using the causal networks.

Step 4: Solution using machine reasoning.

Figure 3.

The strategy for the solution is a four-step process, starting with a causal description of the EMC landscape (Step 1) and proceeding to design a DSS with a machine reasoning mechanism (Step 4). Intermediate Steps 2 and 3 identify the gaps and build a causal network as the DSS core.

3.1. Limitations

The data used in this case study is obtained after the fact of the epidemic outbreak, the reasoning is evidential, and, thus, reactive rather than proactive. It stands to reason that when using these data as the situation develops, new knowledge is accumulated and can be used for predicting the effect of future outbreaks.

3.2. Contribution

The key contribution is the novel causal landscape for the EMC and its translation into a DSS built on machine reasoning.

4. EMC phases, causality, M&S

The EMC of a civilian community or a military unit includes the following epidemiological phases: mitigation, preparedness, response, and recovery.¹ The countermeasure applied at any of these phases can be a success or failure, caused by the existing technology and societal gaps.

To illustrate these gaps, we provide a graphical notion of the standard EMC concept in Figure 2. This representation reflects the causal-and-effect relationships between the EMC phases.

4.1. EMC as a system of systems model

The causal landscape of the EMC is represented by the following hierarchy of networks as shown in Figure 4. Each EMC phase is represented by its corresponding modeling networks and simulation principles. The mitigation phase consists of the states such as reliability of assessment resources, losses, and planning; the preparedness phase includes risks of various strategies, the trust of precautionary measures, as well as the risk of biases in data processing and decision-making; the response phase is represented by the risk decision states; the recovery phase is a causal network of recovery actions. Hence, Figure 4 represents the system of systems (SoS) architecture that is characterized by operational and managerial independence of each EMC phase, evolutionary development, geographic distribution, and emergent behavior.¹⁸ Specifically, the EMC of the DSS is characterized by the following properties.

Complexity: Interactions between the components are complex. This requires a well-structured M&S, e.g., an arbitrary state of the response phase is in a causal relationship with the selected states from mitigation and preparedness phases.

Concurrency: The number of the EMC states is large; the states may be concurrent, e.g., in modeling, a given state of the response phase may be concurrent with the R&D processes from both the mitigation and preparedness phases.

Specificity: The number of application scenarios is large, and each of them requires a particular simulation, e.g., each epidemiological scenario is a subject of particular consideration.

Adaptivity: The EMC is adaptive by nature, that is, the structure of the EMC and its configuration can be changed in order to reach better performance, e.g., using the parameters such as vaccination and virus spread characteristics.

Figure 4.

Each EMC phase (mitigation, preparedness, response, and recovery) can be represented by a causal network. The EMC phases are related as shown by the red lines.

A system with these properties, i.e., the EMC, is a typical discrete event system that requires a modular and hierarchical formalism for the modeling. A proven universal modeling platform for a such system is DEVS, a Discrete Event system Specification.¹⁹ In this work, SoS includes the intelligent components with adaptability properties that are combined on the DEVS platform. Zeigler et al.¹⁹ formulated the requirements to the development of the healthcare M&S infrastructure: most of them are acceptable for the EMC such as organizational structure and ontology, macro behaviors, a large spectrum of models, and high quality of data.

4.2. Epidemiological profiling of military units

From an epidemiological point of view, military ships such as the USS TR are unique environments that demand specific technological and societal solutions and countermeasures:

− they are closed-set infrastructure that includes purpose, specific, isolated spaces, as well as heat, ventilation, and air conditioning system;

− any contacts between crew members can be traced efficiently due to the specified daily routines;

− the population (crew) statistics such as age, race, gender, and health status are known.

These unique characteristics make it easy to implement advanced tracing technologies, including wireless sensing, continuous health status monitoring, and reliable isolation for quarantine. This is also one of the reasons to consider the handling of the COVID-19 outbreak as a motivational example in our work.

5. DSS reasoning mechanism

The core of the proposed DSS is a causal network that allows us to perform reasoning. The reasoning operations are defined as follows.

(1) Prior data representations and assessments, such as statistics and distribution of data after an outbreak that has already happened, as well as the pre-existing conditions. In causal modeling, the priors are represented by a corresponding probability distribution function.^5,20

(2) Causal analysis is based on the “cause–effect” paradigm.^5,20 Another advanced tool is Granger causality analysis, usually used to analyze time series and to determine whether one can forecast the other.²¹

(3) Reasoning is the ability to form an intelligent conclusion or judgment using the evidence. Causal reasoning is a judgment under uncertainty performed on a causal network.^5,20

(4) Prediction. In complex systems, meta-learning and meta-analysis can be used to predict the overall success or failure of the predictor. The most valuable information is in the “tails” of the probabilistic distributions.^22,23

5.1. Classes of causal networks

A causal network is a directed acyclic graph where each node denotes a unique random variable. A directed edge from node $X_{1}$ to node $X_{2}$ indicates that the value of $X_{1}$ has a direct causal influence on the value of $X_{2}$ . Uncertainty in causal networks is represented as conditional uncertainty tables (CUTs). A CUT is assigned to each node in the causal network and is represented as a table that indexes all possible value assignments to the parents of this node. Thus, each entry of the CUT represents a conditional “uncertainty” that varies according to the choice of the uncertainty metric.

A recent review²⁴ described the various types of causal networks that are deployed in machine reasoning, e.g. Bayesian, imprecise, interval, credal, fuzzy, and subjective networks. The type of causal network is chosen based on the given DSS model and a specific scenario. The choice depends on the scenario as well as the CUT as a carrier of primary knowledge. Various causal computational platforms for modeling several systems were compared, in particular, Dempster–Shafer versus credal networks²⁵ and Bayesian versus interval versus Dempster–Shafer versus fuzzy networks.^7,8

In our study, among various causal networks, we have chosen to use Bayesian causal networks. Our motivation for this choice is driven by the fact that the Bayesian (probabilistic) interpretation of uncertainty provides acceptable reliability in decision-making.

In a Bayesian network, the nodes of a graph represent random variables $X = {X_{1}, \dots, X_{m}}$ and the edges between the nodes represent direct causal dependencies. To construct a Bayesian network, factoring techniques are generally applied. Thus, such a network is based on a factored representation of joint probability distribution. While the Bayesian network structure reflects the causal relationships, its probability reflects the strengths of the relationships.

Probabilistic reasoning on causal (Bayesian) networks enables knowledge inference based on priors and evidence has been applied to diagnostics for precision medicine.⁹ Recently, COVID-19 risks analysis was performed using the Bayesian inference to learn the proportion of the population with or without symptoms from observations of those tested, along with observations about testing accuracy.¹⁰

5.2. Risk assessment

In ES, the risk is simply a rate such as the infection incidence rate, also known as the attack rate, and the IPR.¹⁶ Another component for risk assessment is the CFR. In this study, CFR is treated as a hidden variable in the DSS due to the lack of information regarding the fatalities on the USS TR. The risk measures are used in ES in simple forms such as “high-risk group,”“risk factor,” and “systematic difference in the enrollment of participants.”²⁶

In DSSs, risk and other “precaution” measures such as bias and trust are often used to evaluate a cognition-related performance.²⁷ The DSS is expected to provide the experts with detailed assessments of epidemic scenarios and make the decision process more transparent and explainable. For example, syndrome surveillance consists of real-time indicators for a disease allowing for early detection.²⁶

The experts seek advice from the DSS in answering the following questions.

What are the risks given the state of the disease outbreak and the health care resources?

How reliable are the surveyed or collected data?

What kind of biases are present or expected in data collection, algorithms, and CI decision making?

The risk in the decision-making context is defined by the US National Institute of Standards and Technology (NIST)²⁸ as “a measure of the extent to which an entity is threatened by a potential circumstance or event, and typically is a function of: (i) the adverse impact, also called cost or magnitude of the harm, that would arise if the circumstance or event occurs, and (ii) the likelihood of event occurrence.”

In particular, risk of a decision upon a certain action is a function $F$ of an impact, or cost, $I$ , of such action given the likelihood, $P$ , of the decision error²⁷:

Risk = F (I, P)

(4)

Acceptance of the cognitive DSS technology by human decision-makers is determined by the combination of bias, trust, and risk factors.^29,30 Other contributing factors include belief, confidence, experience, certainty, reliability, availability, competence, credibility, completeness, and cooperation.^31,32 In our approach, the causal inference platform calculates various uncertainty measures²⁷ in risk and bias assessment scenarios.

5.3. NATO standards for the decision support

Risk, trust, and bias manifest themselves in various ways, such as the reliability of information sources (e.g., infection survey data) and the credibility of the information (e.g., testing procedure accuracy):

\begin{matrix} \underset{Quality}{\underset{︸}{Data Reliability}} & \Leftrightarrow {\begin{matrix} Risk \\ Trust \\ Bias \end{matrix}} \Leftrightarrow \underset{Reputation}{\underset{︸}{Test Credibility}} \end{matrix}

(4)

This relationship can be represented as follows.

(a) Source reliability as the quality of being reliable, or trustworthy, is related to (1) risk as a function of potential adverse impact and the likelihood of occurrence, (2) confidence in the quality, and (3) bias as a systematic over- or under-assessment of the parameter of interest.

(b) Information credibility is the reputation impacting one’s ability to be believed. In ES and EI, in particular, it can be associated with the credibility of infection testing technology.

The above metrics of uncertainty closely resemble the ones known as the Admiralty Code defined in the NATO Standardization Agreements such as STANAG 2022 and STANAG 2511³³ (Figure 5). NATO uses the Admiralty Code to resolve conflicting scenarios in human–human, human–machine, and machine–machine interactions. The reliability of the recommended decisions can be increased by using more reliable sources and creditable information, or it can be diminished due to a lowered reliability of the source and/or credibility of the information. For example, scenario $C 4$ in Figure 5 is composed of the source reliability C≡ <Fairly Reliable> and information credibility 4≡ <Doubtful true>. In this context, risk can be expressed in terms of the fair reliability of data sources while being doubtful about the credibility of the information. Details can be found in Lai et al.²⁷

Figure 5.

The manifestation of risk, trust, and bias via assessments of the reliability of the source and the credibility of information using the admiralty code.

6. EMC causal model for USS TR

During the time of deployment of the USS TR around March 2020, an outbreak of COVID-19 occurred that affected marines (younger healthy adults). Approximately a quarter of the service members were determined to be infected with COVID-19. An investigation conducted by the US Navy and CDC during April 20–24 includes a study on 382 voluntary service members.³⁴

6.1. Three-level causal hierarchy interpretability

A causal network is a directed acyclic graph in which each node denotes a unique random variable. A directed edge from node $A$ to node $B$ indicates that the value attained by $A$ has a direct causal influence on the value attained by $B$ . Uncertainty inference requires a specific type of data structure referred to as CUTs. A CUT is assigned to each node in the causal network. Bayesian network is defined as a causal network with the CUT being a conditional probability table (CPT) using point probability measures. The fundamental properties of causal networks which are most useful in the DSS development are as follows.

Partitionable: any causal network can be represented by a set of smaller networks, i.e., decomposed.³⁵

Updatable: this is an inherent property of causal network.²⁰

Tractable: efficient learning in scenarios of partially observed data.³⁶

Robust: interpretable uncertainty caused by different ways of formalization.^7,24

Interpretable: three-level causal hierarchy ensures this property.⁵

Concept compatible: deep learning can support causal reasoning in order to achieve a synergy of various CI concepts.³⁷

The formalism of the interpretable DSA is represented by Pearl’s three-level causal hierarchy interpretability.⁵

− Low level:“What is? How would seeing $X$ change my belief in $Y$ ?”– this is associational (statistical) interpretability.

− Intermediate level:“What if? What if I do $X$ ?”– this is interventional interpretability.

− Top level:“Why? Was it $X$ that caused $Y$ ? What if I had acted differently?”– this is counterfactual interpretability.

Three-level causal hierarchy interpretability is an inherent property of the DSA. Note that interpretability and explainability are the central requirements to all intelligent tools.¹⁷

In our study, we created a fragment of a causal network based on the available data (Figure 6). The main risk assessed in the DSS using the causal network is the IPR. The other risks are shown as hidden variables (gray color), since there was not enough data reported on the case to include them in the numerical analysis. Those hidden risks include the reliability of testing related to errors in conducted serological tests. We combine the variables in the network according to the EMC phases as per Figure 4, although not all variables of influence are included in this study. In particular, the iterative process between the EMC response and recovery phases in Figure 6 is simplified and introduced only by factor “infection rate.” In reality, as we observe the unfolding pandemic, this process is represented by a set of harmonized epidemiological measures including the infection and vaccination rates, the cost of restrictions, and societal factors.

Figure 6.

Causal network for the case of the USS TR data. The variables/nodes are grouped according to the EMC cycles (mitigation, preparedness, response, and recovery), whereas gray color indicates hidden nodes that are not included in the numerical calculations. The gray arrows indicate hidden connections that exist between the different nodes.

6.2. Data collection and causal network design

Table 2 lists all the nodes in the network and their corresponding states and probabilities. The probabilities for the prior nodes in Table 2 are captured based on the statistics collected by Payne et al.³⁴ For example, in Payne et al.³⁴ there is a total of 382 volunteers, of which 351 volunteers reported washing their hands as a prevention measure. This results in the probabilities of 8.12% (31/382) of volunteers who were not washing their hands, and 91.88% (351/382) of those washing their hands. In this paper, we assume a uniform distribution for the node “Infection Rate” as no value was given by Payne et al.³⁴ It should be noted that this value is approximately 70% (1000/1417) for the USS TR, based on the reported results.³⁴

Table 2.

Prior nodes and states are defined based on the reported data.

Node	States (proportions)
“Hand Wash”	no (8.12%); yes (91.88%)
“Hand Sanitizer”	no (6.81%); yes (93.19%)
“Avoid Common Areas”	no (62.04%); yes (37.96%)
“Face Cover”	no (25.92%); yes (74.08%)
“Workspace Cleaning”	no (19.63%); yes (80.37%)
“Berth Cleaning”	no (34.03%); yes (65.97%)
“Keeping Distance”	no (49.74%); yes (50.26%)
“Infection Rate”	0 to 10 (Uniform Distribution)
“Gender”	male (75.65%); female (24.35%)
“Age”	18-24 (29.58%); 25-29 (20.42%);
	30-39 (38.74%); 40-59 (11.26%)

6.3. Infection rate and prevention index

We use the definition of IPR as stated in Equation 1, as the “Infection Rate.”

The chance of infection is influenced by a variable called a prevention index (PI).

Definition 1. The PI of the ith preventive action is defined as follows:

P I_{i} = 1 + \frac{β_{i} - α_{i}}{β_{i}}

(5)

where subscript $i$ represents one of the preventive measures, $α_{i}$ represents the probability of having COVID-19 given that preventive action $i$ is performed, and $β_{i}$ represents the probability of having COVID-19 given no preventive measure $i$ implemented.

In other words, the $P I_{i}$ captures the effect of the $i$ th preventive measure on the chance of being infected. The probability values, $α$ and $β$ , are calculated based on the statistics from Payne et al.,³⁴ and are summarized in Table 3. Note that the reported statistics do not have any information on how frequently a person performs a preventive action. If such statistics were provided, it would be beneficial to analyze the relationship between the PI and the frequency of action such as hand washing. Another key interest could be the analysis of the effect of vaccination as a preventive action. However, owing to the time the USS TR report³⁴ was created, vaccines had not yet been developed and, therefore, could not be used as a preventive action.

Table 3.

$P I_{i}$ for various preventive measures.

	Doing	Not doing	Prevention
	( $α$ )	( $β$ )	Index
“Hand Wash”	62.11	64.52	1.0373
“Hand Sanitizer”	61.52	73.08	1.1582
“Avoid Common Areas”	53.79	67.51	1.2032
“Face Cover”	55.83	80.81	1.3091
“Workspace Cleaning”	63.52	57.33	0.8921
“Berth Cleaning”	61.90	63.08	1.0186
“Keeping Distance”	54.69	70.00	1.2188

Example 1. It was reported that:

283 volunteers used face covers as a preventive measure;

158 of these volunteers reported having COVID-19;

238 total volunteers reported having COVID-19.

Combining the above data, we get 55.83% (158/283) and 80.81% (80/99) of having COVID-19 for the subjects who used face cover or not, respectively. Using Equation (5), a $PI$ of $1.3091$ ( $1 + (80.81 - 55.83) / 80.81$ ) is obtained for using a face cover. This represents a “positive” PI, therefore, reducing the overall probability of infection.

The CPT for the variable “Prevention Index” represents a distribution of the cumulative PI. The cumulative PI is calculated based on the product of the individual preventive indices:

P I_{cumulative} = P I_{0} Π_{i = 1}^{N} γ_{i} P I_{i} + (1 - γ_{i})

(6)

where $N$ is the number of preventive measures (seven in this paper), $γ_{i}$ is a binary value indicating whether the $i$ th measure is taken, and $P I_{0}$ represents the index for no preventive measure taken.

Example 2. The PI for no preventive action taken is equal to 1. Using Equation (6), and assuming the“Hand Wash” is applied, the index is $1.0373$ ( $1 \times 1.0373$ ). This can be further increased to $1.2014$ ( $1 \times 1.0373 \times 1.1582$ ) if “ Hand Sanitizer ” is used with “ Hand Wash .” Note that the “ Workspace Cleaning ” action happens to reduce the PI, counter-intuitively. The PI value ranges from $1.8921$ (only “ Workspace Cleaning ”) to $2.3491$ (all actions except “ Workspace Cleaning ”) depending on the number of preventive actions taken.

The variable “Vulnerable” (V) corresponds to the degree of being susceptible to the infection depending on the variables such as age and gender. The CPTs for the node “Vulnerable” (V) uses the normalized values of the product of the probabilities for “Gender” (G) and “Age” (A):

V = \frac{A_{i} \times G_{j}}{(A_{i} \times G_{j}) + ((1 - A_{i}) \times (1 - G_{j}))}

(7)

where $A_{i}$ represents the probability of having COVID-19 for the $i$ th age category (we consider four such age categories: 18–24, 25–29, 30–39, and 40–59 years old), and $G_{i}$ represents the probability of having COVID-19 for gender category $j$ (male or female). We defined such probability distributions based on the evidence reported in the literature for the limited sample of the USS TR population. Nevertheless, we accepted those distributions for the purpose of this example.

Example 3. Given that the probability of having COVID-19 for the age group 18–24 is 68.1% and the probability of having COVID-19 for gender group male is 65.7%, the degree of vulnerability is computed as follows according to Equation (7):

V = \frac{0.681 \times 0.657}{0.681 \times 0.657 + 0.319 \times 0.343} = 0.8035

(7)

The two CPT tables, for the nodes “Symptoms” and “Vulnerable” are listed in Tables 4 and 5, respectively.

Table 4.

CPT for the node Symptoms.

	Symptoms
Has COVID	0	1–3	4–5	6–8	>8
No	0.3750	0.3403	0.0903	0.1111	0.0833
Yes	0.1849	0.2143	0.1555	0.2101	0.2353

Table 5.

CPT for the node Vulnerable.

Gender	Age	Vulnerable
		No	Yes
Male	18–24	0.1965	0.8035
	25–29	0.2262	0.7738
	30–39	0.2678	0.7322
	40–59	0.2926	0.7074
Female	18–24	0.3053	0.6947
	25–29	0.3444	0.6556
	30–39	0.3966	0.6034
	40–59	0.4263	0.5737

The CPT for the variable “Has COVID” is estimated based on the relationship of “Prevention Index” (PI), “Vulnerable” (V), and “Infection Rate” (IR), defined as follows:

P (Has COVID) = \frac{IR}{PI} \times (V + 1)

(8)

In Equation (8), the IR is reduced based on the PI and then multiplied by the value ( $V + 1$ ). The value $V = 0$ (False) means that the person has less than 50% chance of being vulnerable, and $V = 1$ (True) indicates that the subject’s chance of being vulnerable is higher than 50%. Thus, the vulnerable subject’s chance of having COVID-19 is multiplied by a factor of $V + 1 = 2$ .

The remaining nodes “Symptoms” and “Test” have CPTs created directly based on the data from Payne et al.³⁴

Example 4. Conditional probabilities that a subject who has or does not have COVID-19 is asymptomatic are estimated as follows:

\begin{matrix} P (No Symptoms | COVID = Yes) = 11.52 %; \\ P (No Symptoms | COVID = No) = 14.14 % \end{matrix}

(8)

The list of the COVID-19 symptoms reported by Payne et al.³⁴ included loss of taste, smell, or both, palpitations, fever, chills, myalgia, cough, nausea, fatigue, shortness of breath/difficult breathing, chest pain, abdominal pain, runny nose, diarrhea, headache, vomiting, and sore throat. Note that, in this paper, we considered the number of symptoms and not their type.

6.4. Serological test errors and reliability

This subsection discusses the reliability of tests which is a hidden variable shown in Figure 6. Although this theme is not included in the investigated scenarios, the test reliability is related to overall decision credibility.

Given the reported test results based on two types, enzyme-linked immunosorbent assay (ELISA) and real-time reverse transcription polymerase chain reaction (RT-PCR), error rates such as the false positive rate (FPR) and false negative rate (FNR) can be estimated. The designed Bayesian network includes a “Test” node representing the results of ELISA and the previous RT-PCR test results. Given:

true negative ( $TN$ ) represents a healthy subject reported (by testing) as healthy;

true positive ( $TP$ ) represents an infected subject reported (by testing) as infected;

false positive ( $FP$ ) represents a healthy subject reported (by testing) as infected; and

false negative ( $FN$ ) represents an infected subject reported (by testing) as healthy.

Definition 2. False positive rate (FPR) and false negative rate (FNR) are defined as follows:

FPR = \frac{FP}{FP + TN}, FNR = \frac{FN}{FN + TP}

(9)

The risk of error depends on the rates such as FPR and true positive rate (TPR) and FNR and true negative rate (TNR). The overall risks of such errors are defined as follows:

Ris k_{p} = Impac t_{u} \times FNR + Impac t_{k} \times TPR

(10)

Ris k_{n} = Impac t_{q} \times FPR + Impac t_{c} \times TNR

(11)

where $TPR = 1 - FNR$ , and $TNR = 1 - FPR$ and FPR and FNR are defined by Equation (9).

Example 5. Assume that RT-PCR test results are ground truth and ELISA test results are predicted cases. Based on this assumption, FPR and FNR are calculated to be 10.88% (16/147) and 9.79% (23/235), respectively. With the given effect, or cost of such errors (very high = 4, high = 3, low = 2, very low = 1), the overall risk is computed as follows:

\begin{matrix} Ris k_{p} = 4 \times 0.0979 + 2 \times 0.9021 = 2.1958 \\ Ris k_{n} = 3 \times 0.1088 + 1 \times 0.8912 = 1.2176 \end{matrix}

(11)

The maximum value of positive risk is 4 (FNR=100%) and the minimum value is 3 (FNR=0%). These two cases represent the extreme cases where either all infected subjects are misdiagnosed or correctly diagnosed. Similarly, the maximum value of negative risk is 3 (FPR=100%) and the minimum value is 1 (FPR=0%). The two cases represent the extreme cases where either all healthy subjects are misdiagnosed or all of them cleared.

7. DSS application scenarios

The causal reasoning using the Bayesian network shown in this section was implemented using the open-source Python library pyAgrum³⁸.

The following scenarios were considered in our experiments.

DSS Scenario 1: The effects of increasing the “Prevention Index” with a constant “Vulnerability” and “Infection Rate” on “Has COVID.”

DSS Scenario 2: The influence of “Symptoms” on “Has COVID.”

DSS Scenario 3: The effect of “Has COVID” on “Vulnerable.”

DSS Scenario 4: The predicted “Infection Rate” given “Symptoms” and “Prevention Index.”

7.1. Scenario 1: Preventive measures

Scenario 1 explores how different preventive measures influence the chance of the sailors being infected with COVID-19. The DSS data to the captain is presented as shown in the following report.

Captain: Report the epidemic status.

Medical officer: The first 2 cases and their contacts (estimated as 20 sailors) are quarantined. The current preventive measures are medium (the prevention index is 1.5).

Captain: What are the chances of infection among the contacts?

Medical officer: Based on their age (all under 24 years old) and gender (all males), they are 80% vulnerable. According to the DSS, their chance to contract COVID-19 is 68%.

Captain: What are the recommendations for the rest of the crew?

Medical officer: Increase prevention measures to the maximum; pay attention to face cover, common area avoidance, and keeping distance. This will reduce the infection chance of the vulnerable group from 68% to 55%, and from 39% to 31% for the non-vulnerable group.

DSS Report #1 February 2020USS Theodore RooseveltWarning level: HighSpecification: High risk of infectionEMC status: 18 sailors are vulnerableDSS Reasoning:
	Vulnerable	Non-vulnerable
Prevention Index	Sailor has COVID	Sailor has COVID
0.9	82%	49%
1.0	79%	47%
$1.5$	$68 %$	$39 %$
2.0	59%	33%
2.3	55%	31%
No Evidence	67%	39%

7.2. Scenario 2: COVID symptoms

For scenario 2, the causal relationship between the number of symptoms and the fact of having COVID-19 is analyzed. As the number of symptoms increases, the chance of the individual having COVID-19 increases proportionally. For asymptomatic sailors, the chance of having COVID-19 is 29.67%. It increases to 59.57% for the subjects having 4-5 symptoms.

Captain: In the quarantined group of 20 sailors, while the tests are not yet administered, what are the chances that sailors have COVID-19 based on the exhibited symptoms?

Medical officer: 12 of them have four and more symptoms. According to the DSS, they have COVID-19 with 60% certainty, 6 are asymptomatic (19% chance they have COVID-19), and 2 have less than 4 symptoms (21% chance they have COVID-19).

DSS Report #2 February 2020USS Theodore RooseveltWarning level: HighSpecification: High risk of infectionEMC status: 12 sailors four or more symptomsDSS Reasoning:
Symptoms	Sailor has COVID
0	30%
1–3	35%
$4-5$	$60 %$
6–8	62%
>8	71%
No Evidence	46%

7.3. Scenario 3: Vulnerability and COVID

Scenario 3 considers how to infer the chance of a subject being vulnerable to the infection given the evidence of having COVID-19.

Medical officer: Yesterday we had 20 confirmed cases of COVID-19.

Captain: If we were to have 20 new cases tomorrow, what age and gender groups will they most likely belong to?

Medical officer: According to the DSS, there is an 84% chance that they belong to the vulnerable group. It means that the majority of these 20 people will be male (75%, or 15 of them), and 40% (8 people) will be 30–39 years old.

DSS Report #3 February 2020USS Theodore RooseveltWarning level: HighSpecification: High risk of infectionEMC status: Expect 20 new infections; 84% likelihood that 75% of them will be male, and 40% will be 30–39 years old.DSS Reasoning:
Has COVID	Vulnerable sailor
Yes	84%
No	67%

7.4. Scenario 4: IPR

Scenario 4 estimates the IPR, or Infection rate, given various evidence for both the number of symptoms and preventive behaviors.

Captain: Report on the progress in containing the COVID-19 among 2000 sailors remaining on board.

Medical officer: We increased the prevention index to the maximum (PI=2.3) using all available resources. According to the DSS, the workspace cleaning only as a measure of cleaning (PI=0.9) will not stop the infection among the sailors, and all of the preventive measures (PI=2.3%) reduce it slightly. The infections could be of various degrees of severity in terms of the number of symptoms.

Captain: What are the expected numbers of infected in a month, the expected end of the outbreak?

Medical officer: With workspace cleaning only, the infection rate among the 2000 un-quarantined sailors will be between 1089 ( $\frac{49 %}{0.9} * 2000$ , asymptomatic cases) and 1333 ( $\frac{60 %}{0.9} * 2000$ , cases with severe symptoms). If we implement all the preventive measures (PI=2.3), there will be between 418 ( $\frac{48 %}{2.3} * 2000$ ) and 530 ( $\frac{61 %}{2.3} * 2000$ ) sailors infected.

This scenario shows that there is a minimal relationship between the PI and IPR because they are independent of each other; however, they both affect the chance of contracting COVID-19 which, in turn, causes various amount of symptoms.

This indicates that inaction (no preventive measures, $Prevention Index = 1$ ) and an infection rate of 70% results in a 71.71% chance of a sailor being infected. If all beneficial prevention behavior is taken ( $Prevention Index = 2.3$ ), the chance for a sailor to be infected is reduced to 48.92%.

DSS Report #4 February 2020USS Theodore Roosevelt2000 sailors on boardWarning level: MediumSpecification: High risk of infectionEMC status: Up to 1333 (67% of crew) sailors are at risk to be infected with no prevention measure taken except for the workplace cleaning; up to 530 (26.5% of crew) sailors are at risk of being infected when all preventive measure are in place.DSS Reasoning:
Symptoms	AverageInfection Ratefor PI=0.9	AverageInfection Ratefor PI=2.3
$0$	$49 %$	$48 %$
1–3	50%	50%
4–5	57%	58%
6–8	57%	58%
$8$	$60 %$	$61 %$

7.5. A software platform for experiments

PyAgrum is a C++ aGrUM library (using SWIG interface generator). It provides an interface to the part of aGrUM allowing us to create, model, learn, use, and embed Bayesian networks and other probabilistic graphical models.³⁸ Figure 7 illustrates the proposed Bayesian network (Figure 6) with the populated conditional probability values. In this scenario, the tables in white illustrate the probability when no evidence is given, that is, for example, the “hand_wash” node shows an 8.17% probability of “no hand-washing” and a 91.81% probability of “hand washing.” These probability values represent on average a chance of a sailor performing this action. The tables highlighted in orange represent the cases where evidence is provided. For example, for the age node, evidence shows the subject’s age is between 40 and 59, therefore, there is a 100% probability of being in that age group and 0% probability otherwise. Based on these three pieces of evidence, the network infers that there is a 52.02% chance that the sailor has COVID. In addition, the prevention index value of an average sailor is 1.17 with 69.82% of him/her being vulnerable.

Figure 7.

A screenshot of the Bayesian network example generated using PyAgrum. The marginalized probabilities (CPT) are shown for each node. Three pieces of evidence are set (highlighted in orange): not using face cover, not keeping distance, and the age between 40 and 59.

8. Conclusions, limitations, and future work

8.1. Conclusion

We argue that the intelligent DSS for epidemiology profiling shall become an integrated part of EMC for military units such as the aircraft carrier USS TR. This profiling is based on the causal network approach that allows for the analysis of the variables of influence. The proposed causal approach work can be useful for other military units and civil communities. However, the efficiency of the proposed EMC model and performance of intelligent DSS depends on logistics, infrastructure, and conditions, i.e., epidemic scenarios.

8.2. Limitations of the EMC causal model

Given the proposed EMC causal model, there are several limitation/assumptions including:

insufficient amount of data to capture the true causal relationship;

the CPT of the causal network nodes are populated based on simplified equations and approximations/assumptions;

there is also an assumption that each prevention measure is independent, while in reality they might be related; and

there are various biases in the EMC model that are not taken into attention.

8.2.1. Insufficient amount of data

As indicated earlier, some causal relationships are inferred based on the data. Therefore, in the case of insufficient data, some relationships can be misleading and/or missing. For example, data regarding spatial location and congestion of the crew is currently missing. Selected mariner may be required to walk through the ship to the targeted areas due to their duty, and this required movement may result in an increased chance of infection.

8.2.2. Approximations and assumptions

As well, the nodes such as “Has COVID,”“Vulnerable,” and “Prevention Index,” are populated based on the proposed equations. These equations only illustrate the general, not necessarily the exact relationship, and were derived for the given scenario, and may require modification when transferred to another study. For example, age, gender, and preventive behaviors can greatly increase or decrease the chance of infection, but this relationship cannot be captured by deterministic equations.

8.2.3. Independence

In this study, we assume that the seven preventive behaviors reported by the volunteers in Payne et al.³⁴ are independent. This assumption is not sufficient, as sailors can generally be classified as risk-averse or risk-tolerant. Risk-averse individuals are much more likely to take preventive measures, that is, the sailors that use face masks are also those who maintain a social distance.

8.2.4. Biases

Bias in the sampling of data can severely affect the causal network model, specifically the creation of the CPTs in the EMC model. Bias refers to the tendency of an assessment process to systematically over- or under-estimate the value of a population parameter. In the USS TR case, there is a significant bias regarding the crew composition of younger males. Based on the collected data,³⁴ the age group 18–24 contains the most members but also contributes to the largest percentage of having COVID-19 (68.1%), whereas the age group 40–59 contains the least amount of infected members (55.8%). This contradicts the belief that older people are more vulnerable. Possible reasons for this contradiction may be that younger people are prone to more interactions, whereas older people take more precautionary measures, as they are more likely to be of higher rank on the ship and have different duties requiring less contact. Lastly, all the data collected by Payne et al.³⁴ were collected on a volunteer basis, and, therefore, represent only a fraction of the ship population.

8.2.5. Trustworthiness

The reliability and trustworthiness of the CI-based tools such as the described DSS are related to the credibility of the information source and, first, the reliability of testing. The latter depends on the maturity of the testing technology, and other factors. It may also be biased due to flu seasons as many of the symptoms are similar to the flu.

8.3. Future work

8.3.1. Risks, trust, and biases

Key research questions of future work address risks, trust, and biases of intelligent DSS. Given a recommended decision, we have the following questions.

How reliable is this DSS recommendation?

Can the captain trust the DSS recommendation?

What kind of biases can be expected from intelligent DSS?

There are several related studies on the risk–trust–bias causal landscape, e.g., in biometrics⁴ and military medicine.³⁹

8.3.2. Alternatives to the EMC causal model

There are several alternatives to the causal models used in EMC modeling at both the global and local levels. In Section 4, the global level of the EMC modeling is described. An alternative to the EMC global causal model is a universal M&S platform, e.g., DEVS. This M&S alternative is useful due to the following perspectives.

The efficiency of causal networks is limited by their complexity, i.e., the required size of the CUTs or CPTs in the Bayesian networks.

Decomposition (partitioning) of the causal networks is possible with some restrictions.

In contrast, the large size problems are addressed efficiently using the DEVS formalism based on the state transition tables. Integration of some causal indicators into the DEVS platform⁴ provides a new horizon in solving the large-size M&S problem, in particular, in healthcare.¹⁹

At the local M&S level, such as the DSS design, there are various alternatives to Bayesian networks.

In summary, the Bayesian network that has been chosen in this paper as a DSS core, provides only one of many possible interpretations of uncertainty using point probabilities. There are many alternative approaches that operate with other probabilistic representations of uncertainty,²⁴ e.g., credal causal networks,³⁵ fuzzy causal networks,³⁷ Dempster–Shafer causal networks,⁴⁰ and subjective causal networks.³⁶ The choice of a specific causal model is heavily dependent on the data that is available for creating the CUTs. Other alternatives include, in particular, copula-based techniques⁴¹ and Granger analysis for causal exploration of data.⁴²

Footnotes

Acknowledgements

The authors are grateful to Ivan Hu for helping to collect data on the COVID-19 outbreak case on the USS Theodore Roosevelt, and Eur Ing Phil Phillips, CEng (UK), for useful suggestions.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was partially supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) through grant “Biometric-enabled Identity Management and Risk Assessment for Smart Cities”, Department of National Defence’s Innovation for Defence Excellence and Security (IDEaS) program, Canada, and NATO linkage grant EAP.CLG 984 “Intelligent Assistance Systems”.

ORCID iDs

Kenneth Lai

Svetlana N Yanushkevich

Notes

Author biographies

Kenneth Lai received the BSc and MSc degrees from the University of Calgary, Calgary, AB, Canada, in 2012 and 2015, respectively, where he is currently pursuing the PhD degree with the Department of Electrical and Software Engineering. His areas of interest include biometrics and its application to security and health care systems.

Svetlana N Yanushkevich received the Dr.Tech.Sc. (Dr. Habilitated) degree from the Warsaw University of Technology, Poland, in 1999. She is currently a professor with the Department of Electrical and Software Engineering, University of Calgary. She is directing the Biometric Technologies Laboratory and conducting research in the area of biometric-based authentication technologies.

Vlad P Shmerko is a developer of authentication and risk-assessment machines for security applications. He received the Dr.Tech.Sc. (Dr. Habilitated) degree from Latvian Academy of Sciences, Riga, Latvia, in 1990. Currently, he is a researcher in the Biometric Technology Laboratory and adjunct professor at the Electrical and Software Engineering Department, University of Calgary, Canada. He is fellow and Chartered Engineer of the Institution of Engineering and Technology (UK).

References

World Health Organization. Emergencies preparedness, response, http://www.who.int/csr/alertresponse/epidemicintelligence/en/ (2020, accessed 28 September 2020).

Joshi

Karimi

Sparks

, et al. Survey of text-based epidemic intelligence: A computational linguistics perspective. ACM Comput Surv 2019; 52(6): 119.

Beard

Wentz

Scotch

. A systematic review of spatial decision support systems in public health informatics supporting the identification of high risk areas for zoonotic disease outbreaks. Int J Health Geogr 2018; 17(38): 1–19.

Lai

Oliveira

HCR

Hou

, et al. Risk, trust and bias: Causal regulators of biometric-enabled decision support. IEEE Intell Biomet Syst 2020; 8: 148779–148792.

Pearl

. The seven tools of causal inference, with reflections on machine learning. Commun ACM 2019; 62(3): 54–60.

Fenton

Neil

. Risk Assessment and Decision Analysis With Bayesian Networks. Boca Raton, FL: CRC Press; 2012.

Yanushkevich

Eastwood

Drahansky

, et al. Understanding and taxonomy of uncertainty in modeling, simulation, and risk profiling for border control automation. J Defense Model Sim 2018; 15(1): 95–109.

Yanushkevich

Howells

Crockett

, et al. Cognitive identity management: Risks, trust and decisions using heterogeneous sources. In: Proceedings IEEE International Conference on Cognitive Machine Intelligence, Los Angeles, CA, 12–14 December 2019; pp. 33–42.

Vinarti

Hederman

. A personalized infectious disease risk prediction system. Expert Syst Appl 2019; 131: 266–274.

10.

Neil

Fenton

Osman

, et al. Bayesian network analysis of COVID-19 data reveals higher infection prevalence rates and lower fatality rates than widely reported. medRxiv 2020.

11.

Zachary

Ryder

. Decision support systems: Integrating decision aiding and decision training. In: Helander

Landauer

Prabhu

(eds), Handbook of Human–Computer Interaction. Amsterdam: Elsevier, 1997; pp. 1235–1258.

12.

Andreu

Chiarugi

Colantonio

, et al. Wize mirror – a smart, multisensory cardio-metabolic risk monitoring system. Comput Vis Image Understand 2016; 148: 3–22.

13.

Ochoa

Gutierrez

. Architecting e-coaching systems: A first step for dealing with their intrinsic design complexity. Computer 2018; 51(3): 16–23.

14.

Marathe

Ramakrishnan

. Recent advances in computational epidemiology. IEEE Intell Syst 2013; 28(4): 96–101.

15.

Pionti

Perra

Rossi

, et al. Charting the Next Pandemic: Modeling Infections Disease Spreading in the Data Science Age. Berlin: Springer, 2019.

16.

Pettygrove

. Attack rate. In: Boslaugh

(ed.) Encyclopedia of Epidemiology. London: SAGE, 2008.

17.

Gunning

Aha

. DARPA’s Explainable Artificial Intelligence (XAI) program. AI Magazine 2019; 40(2): 44–58.

18.

Maier

. Architecting principles for systems-of-systems. Syst Eng 1998; 1(4): 267–284.

19.

Zeigler

Mittal

Traore

. Formal requirements and DEVS approach for modeling and simulation of complex adaptive system of systems: Healthcare reform. In: Proceedings of the Symposium on Modeling and Simulation of Complexity in Intelligent, Adaptive and Autonomous Systems, Baltimore, MD, April 2018; pp. 1–12.

20.

Pearl

. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. San Mateo, CA: Morgan Kauffmann, 1988.

21.

Spirtes

Zhang

. Causal discovery and inference: Concepts and recent methodological advances. Appl Informat 2016; 3(1): 3.

22.

Davison

Huser

. Statistics of extremes. Annu Rev Statist Applicat 2015; 2: 203–235.

23.

Stehlík

Potocky

Fabián

. On the favorable estimation for fitting heavy tailed data. Comput Statist 2010; 25(3): 485–503.

24.

Rohmer

. Uncertainties in conditional probability tables of discrete Bayesian belief networks: A comprehensive review. Eng Applicat Artif Intell 2020; 88: 103384.

25.

Misuri

Khakzad

Reniers

, et al. Tackling uncertainty in security assessment of critical infrastructures: Dempster-Shafer theory vs. Credal sets theory. Safety Sci 2018; 107: 62–76.

26.

Dicker

Coronado

Koo

, et al. Principles of Epidemiology in Public Health Practice; An Introduction to Applied Epidemiology and Biostatistics. Centers for Disease Control and Prevention, 2006.

27.

Lai

Oliveira

HCR

Hou

, et al. Assessing risks of biases in cognitive decision support systems. In: European Signal Processing Conference, Special Session “Bias in Biometrics”, Amsterdam, Netherlands, January 2021; pp. 1–5.

28.

Joint Task Force. Security and Privacy Controls for Information Systems and Organizations. Report 800-53, National Institute of Standards and Technology, 2017.

29.

Anand

Bharadwaj

. Pruning trust–distrust network via reliability and risk estimates for quality recommendations. Social Netw Anal Mining 2013; 3(1): 65–84.

30.

Feng

Wang

. A security risk analysis model for information systems: Causal relationships of risk factors and vulnerability propagation analysis. Inform Sci 2014; 256: 57–73.

31.

Zhang

Mao

. Trust prediction via belief propagation. ACM Trans Inform Syst 2014; 32(3): 1–27.

32.

Cho

Chan

Adali

. A survey on trust modeling. ACM Comput Surv 2015; 48(2): 1–40.

33.

Blash

Laskey

Jousselme

, et al. URREF reliability versus credibility in information fusion (STANAG 2511). In: Proceedings of the 16th International Conference on Information Fusion, Istanbul, Turkey, July, pp. 1600–1607.

34.

Payne

Smith-Jeffcoat

Nowak

, et al. SARS-COV-2 infections and serologic responses from a sample of US Navy service members – USS Theodore Roosevelt, April 2020. Morbid Mortal Weekly Rep 2020; 69(23): 714.

35.

Cozman

. Credal networks. Artif Intell 2000; 120(2): 199–233.

36.

Ivanovska

Jøsang

Kaplan

, et al. Subjective networks: Perspectives and challenges. In: Proceedings of the International Workshop on Graph Structures for Knowledge Representation and Reasoning, Buenos Aires, Argentina, July 2015; pp. 107–124.

37.

Baldwin

Tomaso

. Inference and learning in fuzzy Bayesian networks. In: Proceedings IEEE International Conference Fuzzy Systems, St. Louis, MO, May 2003; pp. 630–635.

38.

Gonzales

Torti

Wuillemin

. ARGUM: A graphical universal model framework. In: Proceedings of the International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, Arras, France, June 2017; pp. 171–177.

39.

Crump

Schlachta-Fairchild

. Achieving a trusted, reliable, AI-ready infrastructure for military medicine and civilian care. In: Proceedings of SPIE: Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, April-May 2020.

40.

Simon

Weber

Evsukoff

. Bayesian networks inference algorithm to implement Dempster-Shafer theory in reliability analysis. Reliab Eng Syst Safety 2008; 93(7): 950–963.

41.

Iyengar

Varshney

Damarla

. A parametric copula-based framework for hypothesis testing using heterogeneous data. IEEE Trans Signal Process 2012; 59(5): 2308–2319.

42.

Benozzo

Olivetti

Avesani

. Supervised estimation of Granger-based causality between time series. Front Neuroinform 2017; 11: 68.