Is eliciting dependency worth the effort? A study for the multivariate Poisson-Gamma probability model

Abstract

We examine whether it is worthwhile eliciting subjective judgements to account for dependency in a multivariate Poisson-Gamma probability model. The challenge of estimating reliability during product design motivated the choice of model class. For the multivariate Poisson-Gamma model we adopt an empirical Bayes methodology to present an estimator with improved accuracy. A simulation study investigates the estimation error of this estimator for different degrees of dependency and examines the impact of dependency being mis-specified when assessed by subjective judgement. Our theoretical and simulation findings give analysts insights about the value of eliciting dependency.

Keywords

Multivariate Poisson-Gamma model dependency empirical Bayes inference expert judgement product design reliability

Introduction

Probability modelling is an established means of representing and analysing uncertainties associated with risk and reliability problems. Dependencies between uncertain variables expressed probabilistically require appropriate modelling to provide meaningful results. Consider a problem that can be characterised by multiple uncertain variables, then dependency will arise if information for one variable provides information about other variables. That is, the conditional expectation of a variable differs from its unconditional expectation. Some modelling classes explicitly capture dependency within the probabilistic structure. For example, Bayesian belief networks (BBN) reported in a range of contexts^1–3 as well as multivariate distributions for risk and reliability problems.^4–7

Quantifying probability models with dependent variables means that the joint, or conditional, probability distributions need to be expressed in addition to the marginal probability distribution. Data might not be available to support such quantification for various reasons. For example, if the purpose of the model is to support analysis of the reliability of new designs or during product development, or if the model is to analyse risk in a context with rare events, then in such contexts observed events might be unavailable in whole or part given the nature of the data generating processes. We could also encounter situations, say for operational systems, where data might be too expensive to collect. An alternative to data can involve assessing dependency by eliciting subjective probabilities.

Many methods and processes for the structured elicitation of subjective probabilities exist.^8–10 Specifically, there has been consideration of the particular challenges of assessing dependency using subjective judgement.^11–13 A comprehensive literature review of issues associated with assessing dependency via elicitation is given in Werner et al.¹⁴ This review emphasises the need for a structured approach to support the assessment of informed subjective judgements and to explore particular issues, such as cognitive fallacies, that might affect the accuracy of the assessments made. Methods for elicitation of probabilistic dependency are summarised and classed as direct models (this includes many approaches used in risk and reliability such as BBN, multivariate distributions) and indirect models where auxiliary methods are used (such as regression models). While the majority of methods considered by Werner et al.¹⁴ express judgements as probabilities, they also consider methods that express dependencies using moments, referencing¹⁵ who evaluated this approach for modelling the reliability of new system designs to inform assurance decisions. Building on their review in Werner et al.¹⁴ an elicitation process for dependent events is proposed in Werner et al.¹⁶ This process is designed to mitigate the particular cognitive challenges associated with eliciting dependency.

We argue that representing dependencies appropriately is important if we are to build good models for risk and reliability problems. However drawing on the literature¹⁴ and references therein, as well as, for example,^2,8,17,18 we also recognise that eliciting dependency for real model building is challenging and resource intensive. This leads us to ask whether eliciting dependency is worth the effort?

We do not seek to provide a universal answer to this question. Rather, we investigate this issue for a particular multivariate probability model, a Poisson-Gamma model. This model has underpinned analysis for real industry problems. For example, recent modelling (involving two of the authors) to support decisions about the reliability of a one-shot system during new product development. This was a new generation of a product design family for which data from earlier design generations was deemed relevant for some elements of the new system together with test data generated for the new design throughout its development project. A correlation parameter represents the dependency in the multivariate Poisson-Gamma probability model used for this reliability estimation problem for the new design. The dependencies were elicited from suitably qualified engineers using a structured process based on the method described in Quigley and Walls.¹⁹ This elicitation methodology maps the model parameters to the engineering expertise then uses relevant data (say from related past design elements and/or test) to quantify the dependency in view of the elicited judgements. The elicitation approach, grounded in a specifically designed defensible protocol, was resource intensive. It was also cognitively demanding for those expressing their subjective assessments despite having adopted an approach which asked engineers to express their engineering, rather than probabilistic, expertise.

This research aims to investigate, in the context of the multivariate Poisson-Gamma probability model, whether accounting for correlation in the analysis is worth the effort. To address this aim we state two objectives which examine the statistical value of modelling dependency, thereby enabling an analyst to trade-off the wider benefits and the costs of elicitation. Our first objective is to examine the benefits towards error reduction, and hence estimation accuracy, when we explicitly account for the correlation in the model. Our second objective is to explore the consequences of the correlation being mis-specified when assessed via subjective judgement elicitation. Our findings contribute new insights for this particular probability model by providing a formula to express the mis-specification error in the dependency parameter. Although our results are limited by both the chosen probability model and its parameter sets examined, we provide analysts with an approach to considering modelling choices applicable to a wider classes of probability models with dependency.

The paper is structured as follows. First we present an estimator for the multivariate Poisson-Gamma model that pools data from correlated processes and should result in reduced model estimation error. We develop this estimator through a comparative argument based on alternative inference approaches. We describe a simulation study to investigate the accuracy of the proposed inference approach given the degree of dependency (controlled through the correlation parameter) and the amount of data (controlled by the number of processes in the pool). This study is extended to further examine the impact of subjective mis-specification of the correlation parameter. We conclude by reflecting on the limitations of our study, the implications of our findings and provide suggestions for further work.

Model and inference framework

Our first objective is to develop an understanding of how pooling data from similar processes can reduce estimation error. Specifically, we consider Homogeneous Poisson Processes (HPP) and we adopt an empirical Bayes framework to support the inference under the assumption of Gamma marginal prior distributions. These are conjugate to the Poisson and so are mathematically convenient as well as flexible. To motivate the value of the proposed inferential framework for our multivariate Poisson-Gamma model, we develop our reasoning through comparisons with standard methodological approaches. After describing a classical inference approach, which provides a benchmark for later assessing estimation error, we present the Bayesian approach and show the theoretical error reduction resulting by incorporating prior information. Since expressing subjective judgement in the form of a prior distribution can be challenging and resource intensive to elicit,²⁰ we are motivated to present an empirical Bayes approach where the Bayes mechanism is used but the prior distribution is estimated by pooling data on similar processes. Finally in this section we consider the pooling of data from multiple processes which are measurably correlated in their underlying mean rates, such that data from other processes can be explicitly incorporated into the inferential updating to reduce estimation error.

Classical inference

Under classical inference we assume a probability model that measures the variation in the data as a function of a parameter. We consider a Poisson distribution with parameter $λ$ which corresponds to the mean value of the distribution:

P (N = n; λ) = \frac{λ^{n} e^{- λ}}{n!}, n = 0, 1, 2, \dots

(1)

A typical classical approach to estimation would be either to estimate $λ$ through a moment matching approach or a maximum likelihood approach. Assuming we have $t$ observations from the same stochastic process, where the observations are denoted by $n_{j}$ , then the estimator is given by:

\overset{⌢}{λ} = \frac{\sum_{j = 1}^{t} n_{j}}{t}

(2)

To assess the accuracy of such an estimator we treat the data as random variables from the probability distribution and evaluate the Mean Square Error (MSE) which is given by:

MS E_{C} = E [{(λ - \frac{\sum_{j = 1}^{t} N_{j}}{t})}^{2}] = \frac{λ}{t}

(3)

The $MS E_{C}$ for the classical estimator approaches 0 as the sample size increases.

Bayesian inference

Under Bayesian inference we again assume the Poisson distribution but now consider it as a conditional probability distribution assuming the true mean, denoted by $λ$ , is known. This mean is then modelled as a random variable where the uncertainty is described by a prior distribution. Here we assume the prior distribution $π (λ)$ belongs to the Gamma distribution family. This Poisson-Gamma model is given by:

\begin{matrix} P (N = n | Λ = λ) = \frac{λ^{n} e^{- λ}}{n!}, n = 0, 1, 2, \dots \\ π (λ) = \frac{β^{α} λ^{α - 1} e^{- β λ}}{Γ (α)}, α > 0, β > 0, λ > 0 \end{matrix}

(4)

Combining the Poisson and Gamma models, we obtain the predictive distribution which is essentially a weighted average of Poisson distributions where the weights are provided by the prior distribution. For this combination of Poisson and Gamma, the predictive distribution is in the form of a Negative Binomial distribution:

\begin{matrix} P (N = n) = \int_{0}^{\infty} \frac{λ^{n} e^{- λ}}{n!} \frac{β^{α} λ^{α - 1} e^{- β λ}}{Γ (α)} d λ \\ = \frac{Γ (α + n)}{Γ (α) n!} {(\frac{β}{β + 1})}^{α} {(\frac{1}{β + 1})}^{n} \end{matrix}

(5)

Since Bayesian inference incorporates prior information on the process then the mean prior, denoted by $E [Λ]$ , should be specified before observing any data, where:

E [Λ] = \frac{α}{β} .

(6)

Once data have been observed on the process (such as the aforementioned $t$ observations), the prior distribution can be updated using Bayes’ Theorem to give the following posterior distribution:

π (λ | \underset{~}{n}) = \frac{{(β + t)}^{α + \sum_{j = 1}^{t} n_{j}} λ^{α + \sum_{j = 1}^{t} n_{j} - 1} e^{- (β + t) λ}}{Γ (α + \sum_{j = 1}^{t} n_{j})}

(7)

The associated posterior mean is:

E [Λ | \underset{~}{n}] = \frac{α + \sum_{j = 1}^{t} n_{j}}{β + t}

(8)

To facilitate comparison with the classical inference for the same model, we calculate the MSE assuming a Bayesian framework by first averaging over the mean and subsequently over the data that will be realised to obtain:

\begin{matrix} MS E_{B} = E_{N} E_{Λ} [(Λ - E [Λ | \underset{~}{N} {])}^{2} | N] \\ = E_{\underset{~}{N}} [\frac{α + \sum_{j = 1}^{t} N_{j}}{(β + t)^{2}}] = \frac{α}{β (β + t)} \end{matrix}

(9)

The $MS E_{B}$ for the Bayesian estimator also approaches 0 as the sample size $t$ increases. Further, inspection of $MS E_{B}$ shows that it is less than $E [Λ]$ , which is to say that prior to observing any data we anticipate that the $MS E_{B}$ will be smaller than the expected value of the mean. Moreover, we anticipate that $MS E_{B} < MS E_{C}$ since the denominator of the former is $β + t$ rather than just $t$ as for the latter. This insight is not surprising given more data are being introduced to the analysis in the form of prior information. As $β$ increases, the smaller the variance of the prior distribution (consistent with more precise judgement) and hence a smaller $MS E_{B}$ .

Empirical Bayes inference

An empirical Bayes inference approach presumes we have a pool of Poisson processes, each with their own $λ$ all of which have been realised from the same probability distribution, namely the prior distribution. Thus, by pooling the data associated with the rates allows estimation of the prior distribution. Then Bayes’ Theorem can be applied (as in a traditional Bayes approach) to provide a tailored posterior estimator for the process of interest.

The empirical Bayes estimator for the Poisson-Gamma model is given by:

E [Λ_{i} | N_{i} = n_{i}] = \frac{\overset{⌢}{α} + n_{i}}{\overset{⌢}{β} + t}

(10)

where the estimators of the prior distribution are denoted by $(\overset{⌢}{α}, \overset{⌢}{β})$ and we index the mean values with subscript i to correspond to the ith process in a pool of m Poisson processes. The corresponding mean square error is given by:

\begin{matrix} MS E_{EB} = E_{N_{i}} [E_{Λ_{i}} [{(Λ_{i} - \frac{α + \sum_{j = 1}^{t} N_{j}}{β + t})}^{2} | \underset{~}{N_{i}}]] \\ + E_{N} [E [{(\frac{α + \sum_{j = 1}^{t} N_{j}}{β + t} - \frac{\overset{⌢}{α} + \sum_{j = 1}^{t} N_{j}}{\overset{⌢}{β} + t})}^{2} | \underset{~}{N}]] \\ = MS E_{B} + MS E_{PE} \end{matrix}

(11)

The $MS E_{EB}$ can be decomposed into two terms. The first term is the $MS E_{B}$ (equation (9)) and the second term is the $MS E_{PE}$ , which is the pool parameter estimation error. This implies that $MS E_{EB}$ is affected by both the number of processes in the pool and the number of observations in each process since $MS E_{B}$ only decreases as more data are observed for process i and $MS E_{PE}$ decreases as the number of process in the pool increases. Thus, an empirical Bayes estimator provides a means of reducing estimation error since it allows the error to become closer to that of a Bayes estimator without the need for a prior distribution. However, we note that the role of increased pool size is to reduce $MS E_{PE}$ only and not $MS E_{B}$ .

Dependency between processes

We now consider the situation of primary interest where we wish to include data that is correlated with a process of interest with the aim of reducing $MS E_{B}$ . To accommodate this we require a multivariate prior distribution to model the correlation between the data generating processes. This multivariate prior model can be used within a Bayesian approach if obtained by subjective judgement or within an empirical Bayes approach if the parameters have been estimated from observations across a pool of processes.

As a motivating example, we could consider estimating the rate of occurrence of major accidents at a specified location. By pooling data on major accidents across multiple locations then an empirical Bayes estimator could be derived to improve the accuracy of the estimators. However, we can also include data on minor accidents at each location where the rates are likely to be correlated with each other but not necessarily perfectly. As such, the data from the minor accidents for the same location can have a direct impact on reducing the $MS E_{B}$ due to its correlation with the major accidents at that location.

Here we propose a framework that could be operationalised with a Bayesian or an empirical Bayes approach to inference depending upon how the prior parameters have been obtained. Let

ρ = corr (Λ_{1}, Λ_{2})

(12)

For $ρ < 1$ we assume the following bivariate Gamma distribution developed by Minhajuddin et al.²¹ for which the marginal distributions for each process have a Gamma prior:

\begin{matrix} π (λ_{1}, λ_{2}) = \frac{β^{α} λ_{1}^{α - 1} e^{- β λ_{1}}}{Γ (α)} \frac{β^{α} λ_{2}^{α - 1} e^{- β λ_{2}}}{Γ (α)} \\ \times {(1 - ρ)}^{α - 2}_{0} F_{1} [[], [α]; {(\frac{β}{1 - ρ})}^{2} ρ λ_{1} λ_{2}] \end{matrix}

(13)

where

\begin{matrix} {{}_{0}F}_{1} [[], [r]; {(\frac{β}{1 - ρ})}^{2} ρ λ_{1} λ_{2}] \\ = \sum_{k = 0}^{\infty} \frac{{({(\frac{β}{1 - ρ})}^{2} ρ λ_{1} λ_{2})}^{k}}{Γ (α + k) k!} \end{matrix}

(14)

For $ρ = 1$ , we assume the Gamma prior distribution $π (λ)$ , given in equation (4).

The bivariate Gamma distribution in equation (13) was first proposed as a multivariate prior for an HPP by Quigley et al.⁴ where many of the results we require are derived. Here we state only those results which are key for our research. The posterior mean for this model which is given by:

E [Λ_{i} | n_{1}, n_{2}] = \frac{n_{i} + α + E [K]}{t + \frac{β}{1 - ρ}}

(15)

where $K$ is a discrete random variable whose probability distribution belongs to generalised hypergeometric family of distributions.²² This distribution is expressed as:

\begin{matrix} P (K = k) = \\ \frac{\frac{Γ (α) Γ (n_{1} + α + k) Γ (n_{2} + α + k)}{Γ (α + k) Γ (n_{1} + α) Γ (n_{2} + α)} \frac{{(ρ {(\frac{β}{t (1 - ρ) + β})}^{2})}^{k}}{k!}}{{{}_{2}F}_{1} [[n_{i} + α], [α]; ρ {(\frac{β}{t (1 - ρ) + β})}^{2}]} \\ , k = 0, 1, 2, \dots \end{matrix}

(16)

Further Quigley et al.⁴ show that:

lim_{ρ \to 0} E [Λ_{i} | n_{1}, n_{2}] = \frac{n_{i} + α}{t + β}

(17)

lim_{ρ \to 1} E [Λ_{i} | n_{1}, n_{2}] = \frac{n_{1} + n_{2} + α}{2 t + β}

(18)

These results indicate that as the correlation approaches 0, we obtain the Bayes estimate for the multivariate Poisson-Gamma model. Also, as correlation approaches 1, we obtain the same estimate as we would derive if all $2 t$ observations were observed from the same Poisson process. While we can reason the effect of dependency under perfect and no correlation, we are interested to understand the effects for varying degrees of dependency.

Hence now that we have presented an estimator that, by pooling data from correlated processes, should reduce estimation error, we investigate the accuracy of this method for changes in the degree of correlation and the size of the pool of processes.

Simulation study for estimation error

We conduct a simulation study to investigate the MSE of the estimator obtained from pool dependent data, where the parameters of the marginal distribution are estimated from observations using an empirical Bayes approach but the correlation parameter is specified through subjective judgement. This reflects the general modelling situation where engineering experts identify relevant data sets and provide a measure of their similarity between these data sets. Specifically this is the case for our motivating industry problem when we estimated the reliability of a new variant engineering system design.

After describing the simulation study design, we discuss the conditions that lead to under dispersed data being generated in our simulations under dispersion occurs when the variance of the Negative Binomial distribution used to model the distribution of the data in the pool is less that the corresponding mean. Then we present the results from the simulation study and provide an expression that relates the MSE to correlation and pool size.

Study design

We assume the correlation between two processes has been specified by subjective judgement but that the marginal Gamma distributions have been estimated with observed data, hence an empirical Bayes inference approach is adopted (e.g. following Quigley and Walls¹⁹). We assume a pool of HPPs each with a pair of correlated observations. The purpose of the study is to assess the impact of the correlation ( $ρ$ ) and pool size ( $m$ ) upon the MSE of the estimator. As in the previous section, the rates are assumed realised from a Gamma distribution. But, without loss of generalisation, we set the scale parameter to be $β = 1$ . Moreover, we assume the data are realised from the HPP given their rates. To estimate the parameters of the marginal distribution ( $α, β$ ), following,²³ we use a moment based approach to obtain the following estimators:

\hat{α} = \frac{U^{2}}{W - U^{2}}

(19)

\hat{β} = \frac{U}{W - U^{2}}

(20)

where

U = \frac{\sum_{i = 1}^{m} n_{i}}{m} and W = \frac{\sum_{i = 1}^{m} n_{i}^{2}}{m} - U

(21)

A range of values are specified for the three parameters we wish to control in the simulation study correlation, pool size, shape parameter of the marginal distribution as shown in Table 1.

Table 1.

Parameter values controlled in the simulation study.

Input parameter	Specified values
$m$	$5, 10, 20, 30, 40, 50, 60$
$α$	$0.5, 1, 5, 10, 20, 30, 40, 50$
$ρ$	$0, 0.1, 0.2, 0.3, \dots, 0.7, 0.8, 0.9, 1$

Parameter values controlled in the simulation study.

Algorithm 1. Simplified simulation steps in code inner loop:
1: Set: $α, m$ and $ρ$ 2: for $j = 1 : m$ do 3: Simulate: $λ_{1 j}, λ_{2 j} ~ Γ (α, 1; ρ)$ 4: Simulate: $[N_{1 j}, N_{2 j}] ~ Poisson (λ_{1 j} t, λ_{2 j} t)$ 5: end for 6: Calculate: $[\hat{α}, \hat{β}] \overset{moment estimator}{\to} [\underset{~}{N_{1 \cdot}}, \underset{~}{N_{2 \cdot}}]$ 7: if $\hat{α} \leq 0$ or $\hat{β} \leq 0$ then ▹ underdispersion 8: $E [Λ_{1} \| \cdot] \leftarrow \frac{\sum_{i = 1}^{2} \sum_{j = 1}^{m} N_{ij}}{2 m}$ 9: else 10: for $j = 1 : m$ do 11: Calculate: $E [Λ_{1 j} \| N_{1 j}, N_{2 j}, \hat{α}, \hat{β}, ρ]$ 12: end for 13: end if 14: for $j = 1 : m$ do 15: Calculate: $E [Λ_{1 j} \| N_{1 j}, N_{2 j}, \hat{α}, \hat{β}, ρ] - λ_{1} = e_{1 j}$ 16: end for 17: Calculate: $MSE = (\frac{\sum_{j = 1}^{m} e_{1 j}^{2}}{m})$

Algorithm 1. Simplified simulation steps in code inner loop:

1: Set:

α, m

and

ρ

2: for

j = 1 : m

do
3: Simulate:

λ_{1 j}, λ_{2 j} ~ Γ (α, 1; ρ)

4: Simulate:

[N_{1 j}, N_{2 j}] ~ Poisson (λ_{1 j} t, λ_{2 j} t)

5: end for
6: Calculate:

[\hat{α}, \hat{β}] \overset{moment estimator}{\to} [\underset{~}{N_{1 \cdot}}, \underset{~}{N_{2 \cdot}}]

7: if

\hat{α} \leq 0

\hat{β} \leq 0

then ▹ underdispersion
8:

E [Λ_{1} | \cdot] \leftarrow \frac{\sum_{i = 1}^{2} \sum_{j = 1}^{m} N_{ij}}{2 m}

9: else
10: for

j = 1 : m

do
11: Calculate:

E [Λ_{1 j} | N_{1 j}, N_{2 j}, \hat{α}, \hat{β}, ρ]

12: end for
13: end if
14: for

j = 1 : m

do
15: Calculate:

E [Λ_{1 j} | N_{1 j}, N_{2 j}, \hat{α}, \hat{β}, ρ] - λ_{1} = e_{1 j}

16: end for
17: Calculate:

MSE = (\frac{\sum_{j = 1}^{m} e_{1 j}^{2}}{m})

The algorithm for the study is as follows.

Treatment of underdispersion

Since we use a Negative Binomial distribution for the data in the pool when sampling from Poisson-Gamma model we risk encountering the problem of underdispersed data, as discussed in Kokonendji et al.²⁴ Underdispersion occurs if the variance of the Negative Binomial distribution is smaller than its mean due to sampling and compromises the moment estimator proposed by Quigley et al.²³

Figure 1 shows the probability of underdispersion given the choice of pool size and shape parameter across all simulation combinations in our study. The plot shows that the highest chance of underdispersion ocurring is for our smallest combination of shape parameter and pool size ( $α = 0.5$ , pool size = 5). We find an area of low underdispersion for $α \geq 5$ , pool size $\geq 30$ and our plot suggests that a smaller pool size matters more than a small value of $α$ . If underdispersion occurs then we have two options to address it; either to discard the samples or to take the mean and use it as our best estimate. We chose the latter option.

Figure 1.

Probability of underdispersion for simulation combinations of shape parameter and pool size.

MSE results

We now consider the findings of the simulation study to examine the impact of the shape parameter ( $α$ ), pool size ( $m$ ) and correlation ( $ρ$ ) upon the MSE of the estimator. First we investigate the relationship between the MSE and the shape parameter ( $α$ ). We find this relationship to be linear. For example, Figure 2 shows the MSE as a function of $α$ when the pool size is $m = 20$ and for six settings of the true correlation ( $ρ$ ) between 0 and 1. Although not shown here, similar patterns are found for other input combinations. Evidence of a linear relationship between the MSE and $α$ is not surprising given the analytical results shown in the Appendix which allow comparison to the simulation study.

Figure 2.

Relationship between MSE and the prior shape parameter for pool size = 20 and selected values of the true correlation.

Next we examine the relationship between the ratio of $MSE / α$ with the correlation and the pool size. Figure 3 shows the MSE values computed for simulation combinations together with a model fitted to this relationship. After investigating a variety of transformations to this relationship, we obtain the following expression for the best fitting model through a regression analysis:

MSE = α \cdot (0.71 - 0.05 \ln (m) - 0.17 ρ^{2})

(22)

Figure 3.

MSE at different values of shape parameter α for settings of pool size and true correlation parameter with fitted model of the form $MSE = α \cdot (c_{0} + c_{1} \ln (m) + c_{2} ρ^{2})$ .

Interestingly we find that the interaction terms between $ρ$ and $m$ do not contribute to the model. Moreover, the impact of $m$ and $ρ$ are proportional to the value of the shape parameter $α$ .

While this regression model is defensible only for the range of parameter values used in the simulation study, we can build upon our earlier consideration of the inference approaches to develop analytical results to provide the limit of the MSE as $m$ tends to infinity and as $ρ$ tends to one. Since an infinite pool size corresponds to the Bayesian estimator, and if the observations are perfectly correlated, then this implies the sample size is doubled in relation to the case of no correlation when processes are statistically independent. Thus we find:

lim_{ρ \to 1, m \to \infty} MSE = \frac{α}{3}

(23)

lim_{ρ \to 0, m \to \infty} MSE = \frac{α}{2}

(24)

The difference between these two limits, $0.17 α$ , is consistent with the coefficient for the correlation in the regression model (equation (22)). However, the regression model does not have these limits. Consider the situation where pool size is $m = 1$ , which is outside the range investigated in the simulation study where the minimum is $m = 5$ . When $m = 1$ an empirical Bayes approach is not appropriate because there is no pool of processes from which to estimate the pool variability. Under such circumstances where we have prior information, then we would apply a classical approach as described earlier, which would have the following limits:

lim_{ρ \to 1, m \to 1} MSE = \frac{α}{2}

(25)

lim_{ρ \to 0, m \to 1} MSE = α

(26)

Therefore, extrapolating our regression model would underestimate the MSE when processes are independent and overestimate the MSE when processes are perfectly dependent.

Dependency mis-specification

Let us now investigate the implications of mis-specifying the correlation parameter. We extend the simulation study to explore situations where we assume a correlation $ρ = ρ_{A}$ has been specified, say by an engineering expert’s subjective assessment, when the true correlation is actually $ρ = ρ_{T}$ . In the study we simulate data under the case $ρ = ρ_{T}$ then analyse it as if it was specified as $ρ = ρ_{A}$ to mimic the parameter mis-specification. We share a selection of results to illustrate key findings.

Figure 4 shows the relationship between the MSE and $ρ_{A}$ for situations where $α = 50$ , $m = 60$ and $ρ_{T}$ = 0, 0.5 and 1. Regardless of the true correlation, we find the same MSE when no dependency is specified ( $ρ_{A} = 0$ ). However when $ρ_{T} = 0$ and $ρ_{A}$ increases towards 1 then there is a nonlinear growth in the MSE. Whereas for $ρ_{T} = 1$ there is an almost linear decrease in the MSE and for $ρ_{T} = 0.5$ there is relatively little change in the MSE as $ρ_{A}$ increases to 1. Similar patterns are found for other combinations of pool size and shape parameter.

Figure 4.

MSE when correlation is mis-specified, where $ρ_{A}$ is assessed correlation, under selected true correlations $ρ_{T}$ and for case of α = 50, m = 60.

By assuming a true $α$ value, we can examine the effects of mis-specifying the correlation under either Bayesian inference or for the empirical Bayes asymptotic case when the pool size $m$ tends to infinity. Figure 5 shows the relationship between the ratio $MSE / α$ and $ρ_{A}$ for $ρ_{T} =$ 0, 0.5 and 1 for three settings of the prior shape parameter $α =$ 0.5, 10 and 50. We find that the limits agree with the findings discussed earlier (equations (23) and (24)). That is, for $ρ_{A} = 0$ then $MSE / α = 0.5$ and for $ρ_{A} = ρ_{T} = 1$ then $MSE / α = 1 / 3$ . Figure 5 also shows the effects of varying $α$ on $MSE / α$ for $ρ_{T} = 0$ . Further analysis (see Appendix) reveals the following:

MS E_{ρ_{T} = 0, ρ_{A} = 1} = \frac{7 α}{9}

(27)

Figure 5.

$MSE / α$ for assessed correlations, $ρ_{A}$ under selected values for the true correlations, $ρ_{T}$ , and prior shape and scale parameters.

Therefore, incorporating data under the assumption it is realised from the same HPP (i.e. $ρ_{A} = 1$ ) when in fact there is no correlation ( $ρ_{T} = 0$ ) can introduce considerable estimation error depending on the variability of the pool of processes.

Conclusion and further work

Our study has investigated the effect of incorporating data that is correlated with the event process of interest to reduce estimation error. This requires the correlation between processes to be assessed by subjective judgement, which under many circumstances can require a resource intensive elicitation exercise. Through our study we have explicated the relationship between correlation, pool size and MSE within the context of a particular probability model to provide insight as to whether gains from MSE reduction are worth the cost of elicitation.

Empirical Bayes is a rich methodology offering the opportunity to gain the benefits of error reduction enjoyed by the Bayesian methodology but without the same elicitation burden for subjective assessment with its recognised associated biases. Empirical Bayes relies on pooling relevant data together. It is well known that the more homogeneous this data pool, then the stronger the inference in the sense that the estimation error will be smaller. One way of homogenising the pool is to assess correlations between processes and so discriminate between degrees of similarity for the events of interest. For example, in our motivating industry case, the events related to failures, the candidate pools were formed by data on events experienced by earlier design generations or on test for the new system design, and the correlation was assessed by engineering experts via a structured elicitation. However, regardless of how well this elicitation was constructed and managed there still lurks the possibility that the dependency is mis-specified. This will be the case more generally too when we develop risk and reliability models with probabilistic dependencies.

Our study has explored the impact of corrupting the inference through mis-specifying the correlation. We have derived a formula to explicate the MSE in relation to the parameters of the marginal distribution under cases of assumed assessed and true correlations. A more general derivation is shown in the Appendix. Such formulae can inform analytical choices about the incorporation of data from perceived correlated processes by aiding assessment of the consequences. These findings can guide practical choices about the elicitation method selected to support inference about the reliability of a new system design or other applications where a multivariate Poisson-Gamma model is appropriate. Further, the methodological approach we have adopted to assess an understanding of mis-specification could be applicable to examine the implications for estimation errors for a wider class of probability models.

Is elicitation worth the effort? Rather anti-climatically, our answer is it depends. It depends on how accurate the results need to be so that the value of elicitation can be assessed in relation to a fuller consideration of the costs and benefits. Costs include not only the time and effort to plan and conduct an elicitation but also the cognitive burden to those providing subjective assessments. For the particular context of our study, the value depends on the potential of the candidate correlated processes. This potential is determined by both the correlation as well as the characteristics of the marginal distribution, since the benefits of eliciting the dependency are found to be proportional to the shape parameter in our study for a multivariate Poisson-Gamma model.

There are a number of limitations of our study even within the context of HPPs. First, we had a specific form to our multivariate prior distribution. It would be interesting to investigate the sensitivity to the form of this distribution to both the dependency structure and the marginal distributions. Second, it would be interesting to explore the impact of using data to assess the correlation between processes to allow a full empirical Bayes solution to be found. Thirdly, it would be of value to develop a value of information framework to coherently assess the decision as to whether more precise assessments of correlation is worth learning. This latter point opens up questions in relation to probability models with dependencies more generally and need not be constrained to the particular model considered here.

Footnotes

Appendix

The general derivation of the Mean Squared Error (MSE) is as follows:

(28)

\begin{array}{l} M S E = E [{(E_{ρ_{A}} [Λ | N_{1}, N_{2}] - Λ)}^{2}] \\ = E_{N_{1}, N_{2}} [E_{ρ_{T}} [E_{ρ_{A}} {[Λ | N_{1}, N_{2}]}^{2} \\ - 2_{ρ_{A}} [Λ | N_{1}, N_{2}] + Λ^{2} | N_{1}, N_{2}]] \\ = E_{N_{1}, N_{2}} [E_{ρ_{A}} {[Λ | N_{1}, N_{2}]}^{2}] \\ - 2 E_{N_{1}, N_{2}} [E_{ρ_{A}} [Λ | N_{1}, N_{2}] E_{ρ_{T}} [Λ | N_{1}, N_{2}]]; \\ + E_{N_{1}, N_{2}} [E_{ρ_{T}} [Λ^{2} | N_{1}, N_{2}]] \end{array}

The moments under different assumptions are given by:

(29)

E_{ρ_{A} = 0} [Λ | N_{1}, N_{2}] = \frac{α + N_{1}}{β + 1}

(30)

E_{ρ_{A} = 1} [Λ | N_{1}, N_{2}] = \frac{α + N_{1} + N_{2}}{β + 2}

(31)

E_{ρ_{A} = 0} [Λ^{2} | N_{1}, N_{2}] = \frac{α + N_{1} + (α + N_{1})}{{(β + 1)}^{2}}

(32)

\begin{array}{l} E_{ρ_{A} = 1} [Λ^{2} | N_{1}, N_{2}] \\ = \frac{α + N_{1} + N_{2} + {(α + N_{1} + N_{2})}^{2}}{{(β + 2)}^{2}} \end{array}

(33)

E_{ρ_{T} = 0} [N_{1}] = E_{ρ_{T} = 0} [N_{2}] = \frac{α}{β}

(34)

E_{ρ_{T} = 1} [N_{1}] = E_{ρ_{T} = 1} [N_{2}] = \frac{α}{β}

(35)

E_{ρ_{T} = 0} [N_{1}^{2}] = E_{ρ_{T} = 0} [N_{2}^{2}] = \frac{α}{β} + \frac{α + α^{2}}{β^{2}}

(36)

E_{ρ_{T} = 1} [N_{1}^{2}] = E_{ρ_{T} = 1} [N_{2}^{2}] = \frac{α}{β} + \frac{α + α^{2}}{β^{2}}

(37)

E_{ρ_{T} = 0} [N_{1} N_{2}] = \frac{α^{2}}{β^{2}}

(38)

E_{ρ_{T} = 1} [N_{1} N_{2}] = \frac{α + α^{2}}{β^{2}}

Substituting the moments into the expression for the MSE results in the following:

(39)

\begin{matrix} MS E_{ρ_{A} = 0, ρ_{T} = 1} = \frac{α}{β (β + 1)} \\ = MS E_{ρ_{A} = 0, ρ_{T} = 0} \\ = MS E_{B} \end{matrix}

This form is as we would expect because we are ignoring the information from $N_{2}$ as we are assuming independence.

Assuming dependence when the rates are independent results in the following:

(40)

MS E_{ρ_{A} = 1, ρ_{T} = 0} = \frac{(β^{2} + 4 β + 2) α}{β^{2} {(β + 2)}^{2}}

Assuming dependence when the rates are dependent results in the following:

(41)

\begin{matrix} MS E_{ρ_{A} = 1, ρ_{T} = 1} = \frac{α}{β (β + 2)} \\ = MS E_{B} \end{matrix}

Again this is as we would expect because we are correctly using a Bayesian approach with two observations.

Setting $β = 1$ as in the main paper we obtain:

(42)

MS E_{ρ_{A} = 0, ρ_{T} = 1, β = 1} = \frac{α}{2}

(43)

MS E_{ρ_{A} = 1, ρ_{T} = 0, β = 1} = \frac{7 α}{9}

(44)

MS E_{ρ_{A} = 1, ρ_{T} = 1, β = 1} = \frac{α}{3}

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Rafael Schwarzenegger

Lesley Walls

References

Lee

Pan

. A nonparametric Bayesian network approach to assessing system reliability at early design stages. Reliab Eng Syst Saf 2018; 171: 57–66.

Mkrtchyan

Podofillini

Dang

. Bayesian belief networks for human reliability analysis: a review of applications and gaps. Reliab Eng Syst Saf 2015; 139: 1–16.

Mahadevan

. Sensitivity analysis of a Bayesian network. ASCE-ASME J Risk Uncert Eng Syst Part B Mech Eng 2018; 4(1): 011003.

Quigley

Wilson

Walls

, et al. A Bayes linear Bayes method for estimation of correlated event rates. Risk Anal 2013; 33(12): 2209–2224.

Olatubosun

Zhang

. Conditional subjective probability density approach to dependency consideration in passive system reliability. Nucl Eng Des 2018; 337: 108–115.

Wang

Chang

, et al. Reliability and sensitivity analysis for bearings considering the correlation of multiple failure modes by mixed Copula function. Proc IMechE, Part O: J Risk and Reliability 2020; 234: 15–26.

Khosravi

Müller

Glaß

, et al. Simulation-based uncertainty correlation modeling in reliability analysis. Proc IMechE, Part O: J Risk and Reliability 2018; 232(6): 725–737.

Dias

Morton

Quigley

. Elicitation: the science and art of structuring judgement. International series in operations research & management science. New York: Springer International Publishing, 2017.

Cooke

. Experts in uncertainty: opinion and subjective probability in science. New York: Oxford University Press on Demand, 1991.

10.

O’Hagan

Buck

Daneshkhah

, et al. Uncertain judgements: eliciting experts’ probabilities. New York: John Wiley & Sons, 2006.

11.

Daneshkhah

Oakley

. Eliciting multivariate probability distributions. In: Böcker

Klaus

(ed.) Rethinking risk measurement and reporting, London: Risk Books, 2010, vol. 1, p.23.

12.

Werner

Hanea

Morales-Nápoles

. Eliciting multivariate uncertainty from experts: considerations and approaches along the expert judgement process. In: Dias

Luis C.

Morton

Alec

Quigley

John

(eds.) Elicitation. New York: Springer, 2018, pp.171–210.

13.

Morales-Napoles

Hanea

Worm

DTH

. Experimental results about the assessments of conditional rank correlations by experts: example with air pollution estimates. In: 22nd annual conference on European safety and reliability (ESREL), Wroclaw, 2014.

14.

Werner

Bedford

Cooke

, et al. Expert judgement for dependence in probabilistic modelling: a systematic literature review and future research directions. Eur J Oper Res 2017; 258(3): 801–819.

15.

Revie

Bedford

Walls

. Supporting reliability decisions during defense procurement using a Bayes linear methodology. IEEE Trans Eng Manag 2011; 58(4): 662–673.

16.

Werner

Bedford

Quigley

. Sequential refined partitioning for probabilistic dependence assessment. Risk Anal 2018; 38(12): 2683–2702.

17.

Hosack

Hayes

Barry

. Prior elicitation for Bayesian generalised linear models with application to risk control option assessment. Reliab Eng Syst Saf 2017; 167: 351–361.

18.

Morales-Nápoles

Paprotny

Worm

, et al. Characterization of precipitation through copulas and expert judgement for risk assessment of infrastructure. ASCE-ASME J Risk Uncert Eng Syst Part A Civil Eng 2017; 3(4): 04017012.

19.

Quigley

Walls

. A methodology for constructing subjective probability distributions with data. In: Dias

Luis C.

Morton

Alec

Quigley

John

(eds.) Elicitation. New York: Springer, 2018, pp.141–170.

20.

Quigley

Walls

. Characteristics of a process for subjective probability elicitation. In: Dias

Luis C.

Morton

Alec

Quigley

John

(eds.) Expert judgement in risk and decision analysis. Cham: Springer International Publishing, 2021, pp.287–318.

21.

Minhajuddin

ATM

Harris

Schucany

. Simulating multivariate distributions with specific correlations. J Stat Comput Simul 2004; 74(8): 599–607.

22.

Patil

Joshi

. Dictionary and bibliography of discrete distributions, Edinburgh: Oliver and Boyd, 1968.

23.

Quigley

Bedford

Walls

. Estimating rate of occurrence of rare events with empirical Bayes: a railway application. Reliab Eng Syst Saf 2007; 92(5): 619–627.

24.

Kokonendji

Mizère

Balakrishnan

. Connections of the Poisson weight function to overdispersion and underdispersion. J Stat Plan Inference 2008; 138(5): 1287–1296.