SHM-informed life-cycle intelligent maintenance of fatigue-sensitive detail using Bayesian forecasting and Markov decision process

Abstract

Civil and maritime engineering systems must be efficiently managed to control the failure risk at an acceptable level as their performance is gradually degraded throughout the operational life, caused by fatigue and corrosion. Structural health monitoring develops a timely capability to assess the structural condition and performance metrics. However, using actual long-term monitoring data to guide the life-cycle management under stochastic environments has not been sufficiently studied. To realize an optimal maintenance strategy within the service life, an integrated monitoring-based optimal management framework is developed on the basis of the partially observable Markov decision processes (POMDPs) and Bayesian forecasting. In the proposed framework, the stochastic fatigue processes are quantified by the state transition matrix. The Bayesian dynamic linear model is embedded in POMDPs as a continuous observation part to forecast the cycling impacts and estimate the deterioration rate using long-term dynamic strain responses. In addition, making use of the special features of the problem considered in this paper, an adaptive discretization strategy is proposed to alleviate the complexity of large discrete observed spaces in the POMDP. The applicability and feasibility of the framework are evaluated by intelligent maintenance of fatigue-sensitive components with real-world monitoring data. After solving the POMDP by an efficient offline solver, the results obtained in this paper demonstrate that structural interventions are uneconomical to extend the life when a welded detail is approaching its end of life due to the normal service. Furthermore, if multiple interventions are available, the framework can find optimal maintenance actions based on the trade-off between long-term utility and the corresponding cost. This framework as the prototype could also be adjusted to aid life-cycle intelligent maintenance of other types of components under different deterioration scenarios.

Introduction

Within the service life of an engineered structure, it could be subjected to different deterioration scenarios (e.g., corrosion, fatigue), which could affect the serviceability and functionality of the structure. For instance, Wardhana and Hadipriono ¹ summarized failures of bridge structures in the United States between 1989 and 2000 and found 11 bridges failed due to fatigue. In addition, fatigue of ship critical details has always been the main concern for ship operation and is one of the deterioration mechanisms that can affect structural performance. Under these circumstances, decision-makers are faced with fatigue failure risk and must distribute limited budgets so that the component can be operated safely during the life cycle. These facts highlight the importance of an effective decision-making system for the vulnerable infrastructure in the years to come, and underscore the imperative need for developing versatile and intelligent decision-support frameworks that can serve multi-purpose real-time and life-cycle objectives. Motivated by the evolution of cutting-edge monitoring technologies, reliable transducers, and effective information processing algorithms, structural health monitoring (SHM) systems develop a real-time data-driven performance-based assessment ability which is the precondition of rational maintenance. Accordingly, developing decision-support systems for infrastructure management based on real-world monitoring data becomes possible and necessary.

In this paper, fatigue assessment based on long-term monitoring data is conducted. Through reviews of the successful cases in SHM, the fatigue of the steel components is usually assessed by the rain-flow algorithm and linear cumulative damage theory. Ni et al. ² proposed a fatigue reliability model which integrates the distribution of the T-joint stress range with Miner’s damage cumulative rule to estimate the fatigue life with long-term monitoring data. At the structural level, Ye et al. ³ proposed a monitoring-based method for the fatigue assessment of Tsing Ma bridges by using the continuously measured dynamic strain responses. Furthermore, Farreras-Alcover et al. ⁴ incorporated monitoring data with the S–N model to estimate the residual service life of welded joints in the Great Belt Bridge. Herein, to enhance the compatibility associated with different types of data, the Markov chain model is proposed to assess the fatigue process. This model can express a non-stationary state deterioration based on inspection data, dynamic fatigue degradation rested on monitoring data, and is rooted in the stochastic process.

SHM is recognized as a feasible field facilitating to secure the operational safety of structures and prognosis of the damage in time to prevent additional maintenance costs or catastrophic failure. ⁵ This is attributed to the capability of the SHM system in monitoring the impacts actually acting on the vulnerable components. SHM provides an insight into the structures undergoing gradual changes in condition. For fatigue component management, one of the underlying requirements is to rationally forecast cycling impacts based on the long-term strain responses. However, it poses a challenge in the practical application of prediction when facing the uncertainty and non-stationary features inherent in the monitoring data. To solve this aspect, different forecasting algorithms have been widely studied and attempted, for instance, the statistical models, ⁶ time series models, ⁷ gray system theory, ⁸ and singular spectrum model. ⁹ More recently, the Bayesian forecasting model has been applied in real-world data prediction due to its merits. Bayesian dynamic linear model (BDLM) can automatically accommodate non-stationary time series data through time-varying model parameters, ¹⁰ which increases the robustness to tackle data missing. In addition, BDLM can adjust the model parameters through newly obtained observations and provide ahead forecasts without reconstructing models, and it can consider the uncertainty in monitoring data and quantifying the probabilistic distribution in forecasting. ¹¹ These advantages motivate the BDLM favorably for forecasting the performance of an in-service structure based on long-term monitoring data. For instance, Ni et al. ¹² employed this model in expansion joints assessment and damage alarm on the basis of the long-term displacement and temperature data. Wang and Ni ¹³ also used BDLMs to predict the structural strain response. Although establishing a BDLM using SHM data is achievable, life-cycle maintenance of fatigue-sensitive components incorporating the long-term SHM data and predicted dynamic strain responses resulting from BDLM has not been well investigated. In this paper, BDLM is constructed to estimate the daily cumulative damage based on real-world dynamic strains and delineate the fatigue development by using different hidden blocks, such as overall trend, seasonal, and auto-regressive (AR). Moreover, predictions made available by BDLM are essential for the decision-maker to make more informed and rational maintenance actions on a time scale.

Another key step in infrastructure management is how to exploit prior information and long-term monitoring data to guide optimal maintenance. Some difficulties are associated with (i) incomplete information about the structural condition, (ii) state stochastic degradation and random outcome of maintenance, and (iii) long-term objectives. The majority of the models^14–17 in infrastructure management cannot simultaneously account for the above three paramount factors. The frameworks that satisfy three requirements^18–20 are solely feasible in simulation models without being compatible with the actual data (e.g., inspection, monitoring data). More realistic decision models are needed to develop in infrastructure management facing real data and uncertainty. These sequential decision-making models can be classified as genetic algorithm (GA), decision tree, and dynamic programming. The GA in optimal maintenance has been systematically studied.^21–23 Through setting genetic operators and mutation, the GA can search for the optimal strategy under multi-objectives. However, heavier computation leads to the low convergence speed of GA in complex decision problems. The decision tree ²⁴ reduces the computation cost by setting a series of criteria to determine maintenance actions. This method is realized in high efficiency but may result in a feasible scheme rather than the optimal solution. In recent studies, dynamic programming methods in the robotic control field have been applied in infrastructure management. An approach referred to as the partially observable Markov decision processes (POMDPs) in dynamic programming provides a well-suited framework for maintenance in structural stochastic degradation under imperfect observation. Due to the sound life-cycle optimality guarantees and good convergence property, ²⁵ POMDPs are employed in highway pavement maintenance, ¹⁹ deck structure, ²⁶ and bridge component management. ²⁷ It is also employed to consider the value of information for the SHM system.^28–30 However, the aforementioned studies are restricted to numerical studies and existing research lacks sufficient attention to how to verify and update some assumptions, such as discrete observation value and stationary state transition matrix in traditional POMDPs, ³¹ by incorporating the long-term health monitoring data. To accommodate the dynamic deterioration process and actual monitoring data, the solver for non-stationary ³² and continuous observation POMDP ³³ should be utilized. However, most online solvers use limiting steps lookahead exploration methods or artificially discretize the observation space to reduce the computation to a tractable level. ³⁴ This simplification may induce the policy to fall into the sub-optimal solution. ³⁵ Lim et al. ³⁶ pointed out that there are no theoretically guaranteed online solvers that can find optimal solutions with the limit of finite computational resources. Until now, solving POMDPs with continuous observation spaces in an uncertain environment remains challenging. ³⁷ In summary, to address the optimal decision-making problem for welded details, the model should address two issues: (1) establish a quantified model for structural informatics, probabilistic performance prediction, risk management, and control by incorporating real-world long-term health monitoring data. (2) Use a highly efficient algorithm to compute optimal or near-optimal solutions.

In this paper, an intelligent decision-support framework is proposed for welded details management based on long-term real-world monitoring data and POMDPs. The primary functions of the framework include a database from SHM, structural condition assessment, degradation prediction, life-cycle cost, and decision-making. The fatigue deterioration model is developed by using long-term health monitoring data and quantified by the Markov chain. The dynamic strain response of welded details is embedded into the observation part of POMDPs which successively update the prior information on structural conditions using the monitoring data to guide the decision-making. Furthermore, this study develops an efficient discretization strategy on the basis of the features of the state transition matrix and reward matrix in infrastructure management to tackle continuous observation spaces. The remainder of the paper is organized as follows: Methodology presents the Markov chain-based approach for fatigue assessment, the BDLM, and the continuous observation POMDPs. Subsequently, Illustrative example describes the application of the proposed model in life-cycle optimal maintenance of welded details under partially observable stochastic environments. The maintenance results are discussed in Results and discussions, before drawing conclusions. In summary, this study aims to: (i) use the Markov chain to describe the probabilistic fatigue damage process; (ii) establish the BDLMs with different regressive blocks incorporating the long-term monitoring data; (iii) design an adaptive discretization strategy for POMDPs with continuous observation spaces based on the features of infrastructure management; and (iv) establish an optimal maintenance policy-making framework using long-term monitoring data. It can (1) avoid theoretical inconsistencies of existing studies, (2) be compatible with various statistical models and advanced sequential decision-making algorithms, and (3) give optimal maintenance policy to the decision-makers.

Methodology

In this section, the developed computational flowchart by incorporating the algorithms in the framework is illustrated in Figure 1, which includes the Markov chain in fatigue assessment, and BDLMs in fatigue deterioration prediction. The continuous POMDPs are explained in how it formulates infrastructure management. The flow chart is shown in Figure 1.

Figure 1.

Optimal life-cycle management framework to consider stochastic deterioration components based on long-term monitoring data. The bold words denote the algorithms introduced in this paper.

Markov-chain-based fatigue assessment

This part introduces the assessment of the cumulative fatigue damage via the Markov chain. The model uses discrete states and stochastic state-transition probabilities to describe the fatigue process, which is identical to the concept of state transition in POMDP. Generally, there exist two conventional methods for fatigue assessment. The crack growth model describes the possible crack propagation during the stress cycles. S–N curve model focuses on failure probability after the structure suffers stress cycles. The Markov chain model retains the properties of the above two models, which can define different physical states to express the crack expansion or the failure probability.

Fatigue model

Fatigue describes the blunting and re-sharpening advancing progress of a crack front with increasing and decreasing stresses respectively during load cycles. One characteristic of fatigue is the uncertainty of crack development as depicted in Figure 2 which uses different colors to distinguish confidence intervals. The expected crack growth function (black line in Figure 2) in the welded details thickness direction is given as ³⁸ :

$$\begin{array}{c}N=\frac{1}{C}\underset{a_0}{\overset{a_c}{\int }}\frac{\mathrm{d}a}{{(\mathrm{\Delta }K{M}_{k}F)}^m}\\ =\frac{1}{C}\underset{a_0}{\overset{a_c}{\int }}\frac{\mathrm{d}a}{{(\mathrm{\Delta }\sigma \sqrt{\pi a}{M}_{k}F)}^m}\end{array}$$

(1)

Figure 2.

Crack development curve. The crack in fatigue presents slow growth in early stage and rapid propagation close to failure.

Equation (1) indicates the expected cycling number N of the crack growth from the lower limit $a_0$ to the upper limit $a_c$ of integral. C and m are experimentally obtained constants. $\mathrm{\Delta }K$ denotes the stress intensity factor associated with the stress cycle $\mathrm{\Delta }\sigma$ and crack depth a. Geometry modification coefficient F is introduced for the geometrical deviations from the central through-thickness crack in an infinite plate to a semi-elliptical shape crack at the plate surface on the weld side. ³⁹ In addition, a semi-empirical formula is proposed by Bowness and Lee ⁴⁰ to estimate non-uniform stress effects $M_k$ at the weld notch. As there is uncertainty within the fatigue assessment process as Figure 2 depicts, the probabilistic model should be embedded in the crack growth process. In this study, the logarithmic normal distribution is employed in considering the uncertainty of fatigue, in which the standard deviation $S_{\log N}$ is regarded as constant during crack development. ⁴¹

To simplify Equation (1), the S–N curve model is introduced to present a bilinear relation between the stress amplitude $\mathrm{\Delta }\sigma$ and the expected number of cycles ${\mu }_{\log N}$ to failure on the double-logarithmic scale. The bilinear inflection point is the fatigue limit $\mathrm{\Delta }{K}_{\mathrm{th}}$ . The Equation of the S–N curve is given as:

$${\mu }_{\log N_0}=\log A_0-m\log \mathrm{\Delta }\sigma$$

(2)

where m is the slope of a linear function, and $A_0$ is the material property determined by the corresponding fatigue experimental results. The subscript number of ${\mu }_{\log N_0}$ and $A_0$ refer to the reliability. For variable amplitude stress spectrum $\mathrm{\Delta }{\sigma }_i$ , the corresponding failure cycling number $N_0^i$ is calculated by Equation (2). The damage is assumed as a linear cumulative process $n_i/N_0^i$ , where $n_i$ is cumulative cycles in stress range $\mathrm{\Delta }{\sigma }_i$ .

For fatigue analysis, corrosion is an essential parameter since the fatigue life of welded details in the marine environment is significantly lower than in dry air. The corrosion of the welded details can be approximately divided into three phases. ⁴² In the first phase, the corrosion happens at the welded area due to the stress concentration which could trigger coating disabling protection. In phase 2, moderate corrosion weakens the cross-section and induces the principal stress increase. In phase 3, the welded toe loses mass which induces severe damage to the details. To quantify the corrosion in the component life-cycle, an approximate estimation method is proposed ⁴³ :

$$M=\nu t^{\lambda }$$

(3)

where M refers to the mass loss. In welded details, as the size of the thickness direction is much less than the other two dimensions, M is equivalent to the thickness reduction. $\nu$ denotes the mass loss after exposure of the first year, while $\lambda$ indicates the corrosion diffusion rate. Notably, Equation (3) is a general statistical model which does not consider nonuniform corrosion influences and coupling effects of erosion and stress. But applications^42,44,45 in structural details indicate that the above approximations are acceptable. Due to the reduction of cross-section, the equivalent cycling impacted $N_{\mathrm{e}}$ after corrosion is derived from Equation (2):

$$N_{\mathrm{e}}={\mathrm{\Lambda }}^{m}N=\frac{N}{{(1-2\nu t^{\lambda }/T)}^m}$$

(4)

where $\mathrm{\Lambda }$ is the stress amplification factor, and T is the thickness of details. According to the erosion experiments ⁴⁶ and practical statistics, ⁴⁷ the coating is conservatively assumed that it can protect welded details without corrosion for 4 years.

Markov Chain model

The Markov chain model is employed because it not only represents the stochastic degradation process but simplifies complex mathematical operations, such as integration in Equation (1). The Markov chain is related to the state transition. It uses the probability transition matrix to describe a component state $s_t$ degradation to state $s_{t+1}$ after cycling impacts, as indicated in Equation (5). Herein, the duty cycle (DC) is introduced to quantify cumulative cycling during the fixed time intervals (e.g., year). To connect fatigue with the Markov chain model, the discrete states s need to be defined as specific physical quantities, such as crack depth or failure probability. Then, the extent of fatigue damage is approximately characterized by finite discrete states. Because of uncertainty in the fatigue process, the component condition at any moment is not a certain state, but represented by a d-sized vector ${\mathit{b}}_{\mathit{t}}$ (belief state) in a probabilistic manner $(p_{s_1}(t),p_{s_2}(t),\dots ,p_{s_d}(t))$ , where the subscript d denotes the number of discrete states. Afterwards, the expected state $E(s_t)$ is obtained by multiplying probability with the corresponding state, and variance $\sigma (s_t{)}^2$ is calculated through $E(s_t^2)-E(s_t{)}^2$ . These two parameters $E(s_t)$ and $\sigma (s_t)$ correspond to the N and $S_{\log N}$ in the S–N model. In summary, the Markov chain model uses an alternative way to approximately present the uncertainty in fatigue by a finite discrete state $s_j$ and its probability $p_{s_j}$ , shown bottom right in Figure 2.

The following step is to build the numerical expression between failure probability and DC in the Markov chain model. Different from the S–N model, the fatigue process in the Markov chain is expressed by state vector transform in the matrix, where the state is defined as reliability. The unit-jump matrix is assumed to denote the state transition in a DC based on the statistical inspection results, ⁴⁸ defined as ⁴⁹ :

$${\mathit{P}}_{\mathbf{DC}}={\left(\begin{array}{ccccc}\frac{r_1}{r_1+1}\hfill & \frac{1}{r_1+1}\hfill & \dots \hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& \frac{r_2}{r_2+1}\hfill & \frac{1}{r_2+1}\hfill & \dots \hfill & 0\hfill \\ \multicolumn{1}{c}{⋮}& ⋮\hfill & ⋮\hfill & \ddots \hfill & ⋮\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & \dots \hfill & \frac{r_{d-1}}{r_{d-1}+1}\hfill & \frac{1}{r_{d-1}+1}\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & 0\hfill & 1\hfill \end{array}\right)}_{(d\times d)}$$

(5)

The unit-jump assumes that the current state can only degenerate to an adjacent higher state in a DC. In Equation (5), the argument $r_j(j\in [1,d])$ in each row is defined as state transition resistance (STR) which represents the expected number of DC to deteriorate a unit state, as indicated in Figure 3. Using STR, the multiple-slope $(\mathrm{\Delta }{S}_j/(r_j+1))$ lines are plotted to approximate the black curve in Figure 2. Once the component state is defined, the expected DCs $(r_j+1)$ are obtained based on the curve. Another parameter that needs to be determined is the size of matrix d. In fact, the variance of fatigue failure distribution, the size of discrete states d, and the expected number of DCs before failure $N_{\mathrm{f}}$ have followed the relationship ⁵⁰ :

$$\sigma (s_t)=\sqrt{N_{\mathrm{f}}(\frac{N_{\mathrm{f}}}{d-1}-1)}$$

(6)

The parameters’ standard deviation $\sigma (s_t)$ and expected failure cycles $N_{\mathrm{f}}$ can be obtained from fatigue experimental statistics. Finally, the probabilistic distribution of the belief state ${\mathit{b}}_{\mathit{t}}$ from ${\mathit{b}}_{\mathit{t}-1}$ after bearing the number $N_t$ of DCs is expressed by the Markov chain:

$${\mathit{b}}_{\mathit{t}}={\mathit{b}}_{\mathit{t}-1}({\mathit{P}}_{\mathbf{DC}}{)}^{N_t}={\mathit{b}}_0\underset{i=1}{\overset{t}{\Pi }}({\mathit{P}}_{\mathbf{DC}}{)}^{N_i}$$

(7)

It should be noted that the latter part of Equation (7) assumes ${\mathit{P}}_{\mathbf{DC}}$ is constant in the life cycle. To consider the non-homogeneous deterioration processes induced by the erosion and material properties changing, the equivalent cycling number $N_t$ needs to be adjusted by Equation (4). In summary, the merits of the Markov chain model in fatigue are:

• - Initial damage can be incorporated into the model via setting the initial condition vector ${\mathit{b}}_0$ .

• - The evolution of damage can be traced at any moment.

• - The accelerated fatigue damage process is approximated by different STRs $r_j$ .

• - Uncertainty is quantified as $E(s_t)$ and $\sigma (s_t{)}^2$ in discrete states.

The Markov chain model plays as an important role in connecting different algorithms in the integrated management framework. Through Equation (7), the monitoring data is transferred into the probabilistic distribution of the belief state ${\mathit{b}}_{\mathit{t}}$ , which is used as the observation vector to update the prior state vector in the POMDP model.

Figure 3.

The fatigue process is described in the Markov chain model. $r_j$ as the expected number of DC in a state jump.

Bayesian dynamic linear models

An appropriate maintenance strategy should consider the current component condition and the deterioration rate in the future. Thus, a prediction method named BDLM is employed in this study. The fundamental background theories of AR and Kalman filtering (KF) which are embedded within the BDLM are not discussed for brevity. The BDLM has shown promising merits in the field of SHM. The flexibility in BDLM allows it to incorporate various regressive blocks to reflect the characters of data. In addition, it also considers the relationship between different blocks through covariance. Since KF is rooted in the updating process, BDLM has robustness to deal the cases with distortion or incomplete data.

BDLM contains two parts, the dynamic linear model (DLM) and the Bayesian method to update the model parameters. For DLM, the observation equation and system evolution equation are implemented to describe the dynamic changing of monitoring data. Define Y as the dynamic response of structure based on SHM sampling. $\mathit{F}$ is the regressor and superscript represents transpose, X is the system variable, and v is the noise term obeying normal distribution. Then, the time-dependent observation equation is written as ⁵¹ :

$$Y_t={\mathit{F}}_{\mathit{t}}^{\mathrm{T}}{\mathit{X}}_{\mathit{t}}+v_t,v_t~N[0,V_t]$$

(8)

Since the $X_t$ is evolved over time, it develops from a static value into a dynamic variable with an evolution function ⁵¹ :

$${\mathit{X}}_{\mathit{t}}={\mathit{G}}_{\mathit{t}}{\mathit{X}}_{\mathit{t}-1}+{\mathbf{\omega }}_{\mathit{t}},{\mathbf{\omega }}_{\mathit{t}}~N[0,{\mathit{W}}_{\mathit{t}}]$$

(9)

where ${\mathit{G}}_{\mathit{t}}$ indicates the evolution matrix which correlates system variables from ${\mathit{X}}_{\mathit{t}-1}$ to ${\mathit{X}}_{\mathit{t}}$ . ${\mathbf{\omega }}_{\mathit{t}}$ denotes a stochastic error vector that quantifies the information loss over systematic variables evolution. It is expressed by multivariate Gaussian distributions with zero mean and covariance matrix ${\mathit{W}}_{\mathit{t}}$ . It should be noted that the observation noise $v_t$ and variable evolution error ${\mathbf{\omega }}_{\mathit{t}}$ are assumed independent in DLM.

The recursive process in DLM indicates the initial variable ${\mathit{X}}_0$ can affect subsequent variables. To consider the uncertainty of initial value, assuming ${\mathit{X}}_0$ is formed by Gaussian distributions $N[{\mathit{z}}_0,{\mathit{C}}_0]$ , where ${\mathit{z}}_0$ and ${\mathit{C}}_0$ represent the mean value and covariance matrix. Starting with the initial variable ${\mathit{X}}_0$ , the evolution of ${\mathit{X}}_{\mathit{t}}$ is obtained via Equation (9). Meanwhile, the corresponding response $Y_t$ is estimated by Equation (8). If the prediction $Y_t$ is identical with observation $Y{\prime }_t$ , it illustrates that the current model parameters match the monitoring data. Nonetheless, Bayesian updating is necessary to modify the model variable when a discrepancy between prediction and observation cannot be negligible. Before detailing the updating procedures, the following part briefly reviews the regressive blocks in BDLM.

Block component

Theoretically, BDLM can combine any mathematical functions which are utilized to describe a changing rule, such as linear growth, accelerated increasing, periodic fluctuation in Fourier form, periodic oscillation in Kernel regression form, and AR. For stress cycling prediction, three-block components are employed: (a) a trend component, (b) a seasonal component, and (c) an AR component.

The trend is the most fundamental component in the forecast model which describes a steady variation in time series. This component can be expressed using polynomial functions. Normally, second-order polynomials are recommended to avoid the overfitting problem when using higher-order polynomials, the matrix pattern is given as ¹¹ :

$$\begin{array}{cc}{\mathit{F}}_{\mathit{T}}^{\mathrm{T}}=(1,0);{\mathit{X}}_{{\mathit{T}}_{\mathit{t}}}=(X_L,X_T{)}_t\hfill & {\mathit{G}}_{\mathit{T}}=\left(\begin{array}{cc}1\hfill & \mathrm{\Delta }t\hfill \\ \multicolumn{1}{c}{0}& 1\hfill \end{array}\right);{\mathit{W}}_{\mathit{T}}={\sigma }_T^2\left(\begin{array}{cc}\mathrm{\Delta }{t}^4/4\hfill & \mathrm{\Delta }{t}^3/2\hfill \\ \multicolumn{1}{c}{\mathrm{\Delta }{t}^3/2}& \mathrm{\Delta }{t}^2\hfill \end{array}\right)\hfill \end{array}$$

(10)

where subscript T denotes that the above parameters are related to the trend component. Additionally, any arguments evolving with the time series are marked with subscript t. To avoid repetitiveness, the observation noise $v_t$ is individually included in the final observation equation.

Periodicity is another feature of structural monitoring data. The periodic fluctuation of strain is affected by freight transport, monsoon, and temperature. To consider these factors, the seasonal component is introduced in DLM. By contrasting the likelihood value between the Fourier form and the Kernel form, the Fourier form is proved more suitable for fatigue cycling monitoring data. The seasonal block is expressed as ¹¹ :

$$\begin{array}{cc}\multicolumn{1}{c}{{{\mathit{F}}_{\mathit{S}}}^{\mathrm{T}}=}& (1,0,1,0,\dots ,1,0)\hfill \\ \multicolumn{1}{c}{{\mathit{X}}_{{\mathit{S}}_{\mathit{t}}}=}& (X_{S1}^1,X_{S1}^2,X_{S2}^1,X_{S2}^2,\dots ,X_{\mathrm{Sn}}^1,X_{\mathrm{Sn}}^2{)}_t^{\mathrm{T}}\hfill \\ \multicolumn{1}{c}{{\mathit{G}}_{\mathit{S}}=}& \mathrm{diag}(G_{S1},G_{S2},\dots ,G_{\mathrm{Sn}});G_{\mathrm{Si}}=\left(\begin{array}{cc}\cos p_i\hfill & \sin p_i\hfill \\ \multicolumn{1}{c}{-\sin p_i}& \cos p_i\hfill \end{array}\right)\hfill \\ \multicolumn{1}{c}{{\mathit{W}}_{\mathit{S}}=}& \mathrm{diag}({\omega }_{S1},{\omega }_{S2},\dots ,{\omega }_{\mathrm{Sn}});{\omega }_{\mathrm{Si}}={\sigma }_{\mathrm{Si}}^2\left(\begin{array}{cc}1\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 1\hfill \end{array}\right)\hfill \end{array}$$

(11)

In Equation (11), the subscript i indicates the number of periods considered. An oscillation is decided by amplitude $\sqrt{X_{\mathrm{Si}}^1+X_{\mathrm{Si}}^2}$ and frequency $p_i$ .

The AR component describes the relationship between the current output and the n-order historical outputs. The previous n-order $X_{{\mathrm{AR}}_{t-j}}$ and coefficient ${\varphi }_j$ are used to estimate the current variable $X_{\mathrm{A}{\mathrm{R}}_t}$ . The matrix pattern is written as ¹¹

$$\begin{array}{cc}\multicolumn{1}{c}{{\mathit{F}}_{\mathbf{AR}}^{\mathrm{T}}=}& (1,0,\dots ,0);{\mathit{X}}_{\mathbf{A}{\mathbf{R}}_{\mathit{t}}}=(X_{\mathrm{AR}1},X_{\mathrm{AR}2},\dots ,X_{\mathrm{AR}n}{)}_t\hfill \\ {\mathit{G}}_{\mathbf{AR}}=& \left(\begin{array}{ccccc}{\varphi }_1\hfill & {\varphi }_2\hfill & \dots \hfill & {\varphi }_{n-1}\hfill & {\varphi }_n\hfill \\ \multicolumn{1}{c}{1}& 0\hfill & \dots \hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 1\hfill & \dots \hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{⋮}& ⋮\hfill & \ddots \hfill & ⋮\hfill & ⋮\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & \dots \hfill & 1\hfill & 0\hfill \end{array}\right);\hfill \\ \multicolumn{1}{c}{{\mathit{W}}_{\mathbf{AR}}=}& \mathrm{diag}({\sigma }_{\mathrm{AR}}^2,0,0,\dots ,0)\hfill \end{array}$$

(12)

In Equation (12), the subscript AR refers to the auto-regressive feature. Once three-block components are integrated, the synthetical observation function is obtained as:

$$Y_t=Y_T+Y_S+Y_{\mathrm{AR}}+v_t,v_t~N[0,V_t]$$

(13)

Sequential updating

In this part, the Bayesian updating process for system variables ${\mathit{X}}_{\mathit{t}}$ is introduced. The essential model arguments in the Bayesian process are listed: regressor ${\mathit{F}}_{\mathit{t}}$ , evolution matrix ${\mathit{G}}_{\mathit{t}}$ , noise item $v_t$ , and error item ${\mathbf{\omega }}_{\mathit{t}}$ , which remain constant in the updating process. The values of those arguments will be calculated in the next section by the optimization algorithm. Starting with initial variables ${\mathit{X}}_0$ , the Bayesian updating process is illustrated in Figure 4. When the posterior distribution of variables ${\mathit{X}}_{\mathit{t}}$ at time t is obtained as $P({\mathit{X}}_{\mathit{t}}|D_t)~N[{\mathit{z}}_{\mathit{t}},{\mathit{C}}_{\mathit{t}}]$ , where $D_t$ is the information from the initial distribution $N[{\mathit{z}}_0,{\mathit{C}}_0]$ and observation sequence $[Y{\prime }_1,Y{\prime }_2,\dots ,Y{\prime }_t]$ . Using Equation (9), the prior distribution of $P({\mathit{X}}_{\mathit{t}+1}|D_t)$ at time step $t+1$ is estimated as:

$$\begin{array}{c}\multicolumn{1}{c}{E[{\mathit{X}}_{\mathit{t}+1}|D_t]={\mathit{G}}_{\mathit{t}+1}E[{\mathit{X}}_{\mathit{t}}|D_t]+E{\mathbf{\omega }}_{\mathit{t}+1}]}\\ \multicolumn{1}{c}{={\mathit{G}}_{\mathit{t}+1}{\mathit{z}}_{\mathit{t}}={\mathit{\mu }}_{\mathit{t}+1}}\end{array}$$

(14)

$$\begin{array}{c}\multicolumn{1}{c}{\mathrm{Var}[{\mathit{X}}_{\mathit{t}+1}|D_t]={\mathit{G}}_{\mathit{t}+1}\mathrm{Var}[{\mathit{X}}_{\mathit{t}}|D_t]{\mathit{G}}_{\mathit{t}+1}^{\mathrm{T}}+\mathrm{Var}[{\mathbf{\omega }}_{\mathit{t}+1}]}\\ \multicolumn{1}{c}{={\mathit{G}}_{\mathbf{t}+1}{\mathit{C}}_{\mathit{t}}{\mathit{G}}_{\mathit{t}+1}^{\mathrm{T}}+{\mathit{W}}_{\mathit{t}+1}={\mathit{R}}_{\mathit{t}+1}}\end{array}$$

(15)

Once the variables ${\mathit{X}}_{\mathit{t}+1}$ are obtained, the prior distribution of response $P({\mathit{Y}}_{\mathit{t}+1}|D_t)$ can be calculated by Equation (8). This process is referred to as the 1 step-ahead forecast in BDLM:

$$\begin{array}{c}\multicolumn{1}{c}{E[Y_{t+1}|D_t]={\mathit{F}}_{\mathit{t}+1}E[{\mathit{X}}_{\mathit{t}+1}|D_t]+E[v_{t+1}]}\\ \multicolumn{1}{c}{={\mathit{F}}_{\mathit{t}+1}{\mathit{\mu }}_{\mathit{t}+1}=f_{t+1}}\end{array}$$

(16)

$$\begin{array}{c}\multicolumn{1}{c}{\mathrm{Var}[Y_{t+1}|D_t]={\mathit{F}}_{\mathit{t}+1}\mathrm{Var}[{\mathit{X}}_{\mathit{t}+1}|D_t]{\mathit{F}}_{\mathit{t}+1}^{\mathrm{T}}+\mathrm{Var}[v_t]}\\ \multicolumn{1}{c}{=}& {\mathit{F}}_{\mathit{t}+1}{\mathit{R}}_{\mathit{t}+1}{\mathit{F}}_{\mathit{t}+1}^{\mathrm{T}}+V_{t+1}=Q_{t+1}\hfill \end{array}$$

(17)

Another key step of BDLM is to update the variables ${\mathit{X}}_{\mathit{t}+1}$ with the actual observation $Y_{t+1}^{\prime }$ . The expected error is $e_{t+1}=Y_{t+1}^{\prime }-f_{t+1}$ . According to the KF algorithm, two distributions can be combined through Kalman gain ${\mathit{K}}_{\mathit{t}+1}={\mathit{R}}_{\mathit{t}+1}{\mathit{F}}_{\mathit{t}+1}/Q_{t+1}$ . The posterior distribution $P({\mathit{X}}_{\mathit{t}+1}|D_{t+1})~N[{\mathit{z}}_{\mathit{t}+1},{\mathit{C}}_{\mathit{t}+1}]$ is given as:

$${\mathit{z}}_{\mathit{t}+1}={\mathit{\mu }}_{\mathit{t}+1}+{\mathit{K}}_{\mathit{t}+1}{e}_{t+1}$$

(18)

$${\mathit{C}}_{\mathit{t}+1}={\mathit{R}}_{\mathit{t}+1}-{\mathit{K}}_{\mathit{t}+1}{{\mathit{K}}_{\mathit{t}+1}}^{Q_{t+1}}$$

(19)

The entire process is shown in Figure 4. If current information $D_t$ is used to repeat 1 step-ahead forecast procedure without the Bayesian updating (Equations (18) to (19)), then the k step-ahead prediction is obtained.

Figure 4.

BDLMs updating flow chart.

Model parameters estimation

The final problem is how to calculate the arguments $\mathbf{\Omega }=\{{\mathit{F}}_{\mathit{t}}^{\mathrm{T}},V_t,{\mathit{G}}_{\mathit{t}},{\mathit{W}}_{\mathit{t}}\}$ in BDLM which satisfies the maximum log-likelihood estimation:

$${\mathbf{\Omega }}^{*}=\underset{\Omega }{\arg max}[\log (P(Y_1^{\prime }~Y_t^{\prime }|\mathbf{\Omega }))]$$

(20)

$$P(Y_1^{\prime }~Y_t^{\prime }|\mathbf{\Omega })=\underset{i=1}{\overset{t}{\Pi }}P(Y_i^{\prime }|Y_1^{\prime }~Y_{i-1}^{\prime },\mathbf{\Omega })$$

(21)

Equation (20) is the objective function, which aims to maximize the joint probability density of observations (Equation (21)). Any optimization algorithm, ⁵² such as batch gradient descent, can be implemented to search for the optimal solution. The termination criterion is based on the log-likelihood value as follows:

$$\mathcal{L}(\mathit{n})>\mathcal{L}(\mathit{n}-1)\cap |\frac{\mathcal{L}(\mathit{n})-\mathcal{L}(\mathit{n}-1)}{\mathcal{L}(\mathit{n}-1)}|<\tau$$

(22)

where $\tau$ refers to the termination tolerance, which is $1e-6$ . n corresponds to the nth iterative model parameter of ${\mathbf{\Omega }}_{\mathbf{n}}$ . Once arguments of BDLM are determined, the Kalman smoother algorithm can be employed to recalculate the variables ${\mathit{X}}_{\mathit{t}}~{\mathit{X}}_0$ to maximize the expected log-likelihood value. Motivated by the advantages of BDLM, the prediction model is established to estimate daily load cycles based on dynamic strain data.

Partially observable Markov decision processes

This section aims to establish a sound decision framework to plan optimal maintenance actions based on the information from the SHM. The POMDP model is employed, which is a well-suited model for sequential decision-making problems operating under partial observation and an uncertain environment.

Discrete model

A traditional POMDP can be formulated by 7-tuple arguments $〈\mathbb{S},\mathbb{A},\mathbb{O},\mathit{T},\mathit{O},\mathit{R},\gamma 〉$ , where the first three hollow capital letters indicate the total set of structural states $\mathbb{S}$ , available maintenance actions space $\mathbb{A}$ , and observable values $\mathbb{O}$ . The corresponding minuscule is used to represent discrete elements in these sets: $s_i\in \mathbb{S}$ , $a_m\in \mathbb{A}$ and $o_n\in \mathbb{O}$ . The scalar symbols $|\mathbb{S}|$ , $|\mathbb{A}|$ , $|\mathbb{O}|$ denote the number of elements in those sets. The middle three bold capital letters indicate three matrices in POMDP, which construct fundamental operators. The state transition matrix T with three-dimensions $|\mathbb{S}|\times |\mathbb{S}|\times |\mathbb{A}|$ characterizes a state evolution process in a conditional probability $p(s_{t+1}|s_t,a_t)$ . The observation probability matrix $\mathit{O}(|\mathbb{S}|\times |\mathbb{O}|\times |\mathbb{A}|)$ quantifies the probability of observation $p(o_{t+1}|s_{t+1},a_t)$ after the action $a_t$ is executed and the state transfers to $s_{t+1}$ . The reward matrix $\mathit{R}(|\mathbb{S}|\times |\mathbb{A}|)$ defines the immediate cost $r(a_t,s_t)$ to the agent after conducting an action. The last parameter is the reward discounting factor $(\gamma \in [0,1))$ which guarantees the convergence of POMDPs in infinite-horizon time series. Summarizing the aforementioned framework, an episode in infrastructure management is defined as: when a structure starts with the initial state $s_0$ and follows the stochastic degradation, the agent implements a series of maintenance actions based on the observed system (SHM). This episode will terminate once the component fails or finishes its life service. The goal of the manager is to establish an optimal policy ${\pi }^{*}$ to minimize the cost by considering the long-term utility of maintenance action in the light of observations, formulated as:

$${\pi }^{*}=\underset{\pi }{\arg max}\mathbb{E}[\sum _{i=0}^t{\gamma }^{i}r(s_i,a_i)|\pi (a_0,o_1,\dots ,a_t,o_{t+1})]$$

(23)

The challenge in POMDP is that the structural state is hidden. To handle this, we need to speculate the state distribution conforming to the Bayesian theory. A pertinence parameter referred to as the belief state vector b is introduced to fuse the historical information into a probabilistic distribution over a set of all possible discrete states. The hidden state in POMDP is evolved to a belief-MDP, as shown in Figure 5. The posterior belief state ${\mathit{b}}_{\mathit{t}+1}$ is derived from a Bayesian update based on the prior state transition matrix T and observation matrix O , given as ⁵³ :

$$\begin{array}{c}\multicolumn{1}{c}{b_{t+1}(s_{t+1})=p(s_{t+1}|o_{t+1},a_t,{\mathit{b}}_{\mathbf{t}})}\\ \multicolumn{1}{c}{=\frac{p(o_{t+1}|s_{t+1},a_t)}{p(o_{t+1}|a_t,{\mathit{b}}_{\mathit{t}})}\sum _{s_t\in \mathbb{S}}p(s_{t+1}|s_t,a_t)b(s_t)}\end{array}$$

(24)

where the denominator $p(o_{t+1}|a_t,{\mathit{b}}_{\mathit{t}})$ is the normalization coefficient which integrates all possible cases, given as:

$$p(o_{t+1}|a_t,{\mathit{b}}_{\mathit{t}})=\sum _{s_{t+1}\in \mathbb{S}}p(o_{t+1}|a_t,s_{t+1})\sum _{s_t\in \mathbb{S}}p(s_{t+1}|s_t,a_t)b(s_t)$$

(25)

Additionally, the hidden state $s_i$ in POMDPs needs to be substituted by the belief state $b(s_i)$ , such as reward $\sum _{s_i=1}^{\mathbb{S}}r(a_t,s_i)b(s_i)$ . In MDP, the policy decides the optimal action based on the current state and observation $\pi (s,o)\Rightarrow a$ . Since the belief state b contains all the historical information, the action-decision can rely on the belief state mapping $\pi (\mathit{b})\Rightarrow a$ as the dashed line in Figure 5 shows. Then, the expected maximum cumulative reward (Equation (23)) in an episode is evolved to:

$$V^{{\pi }^{*}}({\mathit{b}}_0)={\mathbb{E}}^{{\pi }^{*}}[\sum _{t=0}^n{\gamma }^{t}r(\pi ({\mathit{b}}_{\mathit{t}}),{\mathit{b}}_{\mathit{t}})|{\mathit{b}}_0=\mathit{b}]$$

(26)

where Equation (26) is defined as the value function of belief point b under optimal policy ${\pi }^{*}$ . The objective of the POMDP model is to determine the optimal action in any belief point to maximize the value function. In dynamic programming, Equation (26) can be rewritten compactly by a one-step forward value function, named the Bellman Equation ⁵⁴ , based on the transition between two belief states $({\mathit{b}}_{\mathit{t}}\to {{\mathit{b}}_{\mathit{t}}}_{+1})$ :

$$\begin{array}{c}\multicolumn{1}{c}{V^{{\pi }^{*}}({\mathit{b}}_{\mathit{t}})=\underset{a_t\in \mathbb{A}}{max}[r(a_t,{\mathit{b}}_{\mathit{t}})}\\ +\gamma \sum _{o_{t+1\in \mathbb{O}}}p(o_{t+1}|a_t,{\mathit{b}}_{\mathit{t}})V^{*}({\mathit{b}}_{\mathit{t}+1}|o_{t+1},a_t,{\mathit{b}}_{\mathit{t}})]\hfill \end{array}$$

(27)

Equation (27) contains two parts, instantaneous reward and expected belief point value after being transferred $({\mathit{b}}_{\mathit{t}}\to {\mathit{b}}_{\mathit{t}+1})$ . An important feature of the POMDP model is that the optimal value functions are piecewise linear and convex, thus they can be approximated by a linear polynomial function, ⁵⁵ expressed as:

$$V(\mathit{b})=\underset{\mathit{\alpha }\in \Gamma }{max}\sum _{s_j=1}^{\mathbb{S}}\alpha (s_j)b(s_j)$$

(28)

Through Equation (28), the belief point value is simplified as the probability of state $b(s)$ times the value coefficient $\alpha (s)$ . To visualize the physical meaning of Equation (28), a geometric model is constructed. Assuming the structural state number equals 3, the belief state vector is written as $(b(s_1),b(s_2),1-b(s_1)-b(s_2))$ . If the $b(s_1)$ , $b(s_2)$ , and belief value $V(\mathit{b})$ are the coordinate values in Cartesian space, as shown in Figure 6, the adjacent belief points which share an identical value coefficient can be represented by a plane in belief domain. Afterwards, using the family of value coefficients $\Gamma =({\mathbf{\alpha }}_1,{\mathbf{\alpha }}_2,\dots ,{\mathbf{\alpha }}_{\mathit{n}})$ , multiple planes can be constructed in the belief domain. Finally, the upper boundary of these multiple planes represents the maximum value of Equation (28), which is the geometrical solution of POMDP guided by a policy. The optimal policy ${\pi }^{*}$ of POMDP is equivalent to finding the value coefficient $\Gamma =({\mathbf{\alpha }}_1,{\mathbf{\alpha }}_2,\dots ,{\mathbf{\alpha }}_{\mathit{n}})$ satisfying Equation (27) in the entire belief domain.

Figure 5.

A schematic illustrating the POMDPs transference to belief-MDPs. The additional consideration is the probability of observation in each possible state.

Figure 6.

The solution process for three-dimensional POMDPs. The policy is represented by the multiple planes. The belief point value change from $V({\mathit{b}}_{\mathit{t}})$ to $V({\mathit{b}}_{\mathit{t}+1})$ after the belief state stochastic transition.

Importing Equation (28) into Equation (27), the belief value updating function (29) is derived as:

$$\begin{array}{cc}\multicolumn{1}{c}{V({\mathit{b}}_{\mathit{t}})=}& \underset{a_t\in \mathbb{A}}{max}[\sum _{s_t\in \mathbb{S}}r(s_t,a_t)b(s_t)+\gamma \underset{\mathit{\alpha }\in \Gamma }{max}\sum _{s_t\in \mathbb{S}}b(s_t)\sum _{o_{t+1}\in \mathbb{O}}\sum _{s_{t+1}\in \mathbb{S}}p(o_{t+1}|s_{t+1},a_t)p(s_{t+1}|s_t,a_t){\alpha }_{t+1}(s_{t+1})]\hfill \end{array}$$

(29)

In optimization, the above equation is named the BACKUP operator. An alternative pattern that facilitates programming in computer language is written as:

$$\begin{array}{c}\multicolumn{1}{c}{\underset{\mathbf{\alpha }\in \Gamma }{\arg max}\mathbf{\alpha }{\mathit{b}}_{\mathit{t}}=\underset{a\in \mathbb{A}}{\arg max}[r(a_t,{\mathit{b}}_{\mathit{t}})}\\ +\gamma \sum _{o_{t+1}\in \mathbb{O}}p(o_{t+1}|a_t,{\mathit{b}}_{\mathit{t}})\underset{\mathbf{\alpha }\in \Gamma }{max}\mathbf{\alpha }{\mathit{b}}_{\mathit{t}+1}]\hfill \end{array}$$

(30)

Notably, Equations (29) and (30) imply that each value coefficient $\mathbf{\alpha }$ is associated with a specific action that can maximize the current belief value. Hence, a mapping can be established from value vector $\mathbf{\alpha }$ to action a once the entire optimal vectors ${\Gamma }^{*}$ are obtained. In summary, computing an optimal policy ${\pi }^{*}$ in the POMDPs is to construct the maximum value boundary of multiple hyperplanes in the belief domain. However, there are still existing limitations before continuous POMDPs can be implemented in bridge management incorporating the SHM system. Since the distributions generated by BDLMs are arbitrary, they are needed to be expressed as the probability density function (PDF) as the observation part in POMDPs. As a result, a tailored Gaussian-based methodology is introduced to overcome this problem.

Continuous observation model

To embed BDLMs into the continuous observation POMDPs, the Gaussian mixture model (GMM) is employed to express the distribution associated with the prediction. It is well known that an arbitrary distribution can be divided into finite Gaussian distributions $g(o|\mu ,\sigma )$ with corresponding weight factors w through the expectation-maximization algorithm. ⁵⁶ Thus, the PDF of monitoring data can be written as:

$$f_p(o)=\sum _{i=1}^K{w}_{i}g(o|{\mu }_i,{\sigma }_i)$$

(31)

where K is the number of distributions in GMM. Once the PDF of the fatigue cycles $f_p(o)$ is established from the monitoring data, the number of DC $\left(N_t\right)$ is obtained during the time interval $t\to t+1$ . The component condition after DC is given by Equation (7):

$$p({\mathit{b}\prime }_{\mathit{t}+1}|o_{t+1},a_t,{\mathit{b}}_{\mathit{t}})=p({\mathit{b}}_{\mathit{t}})({\mathit{P}}_{\mathbf{DC}}{)}^{N_t}$$

(32)

Then, the observation item $p(o_{t+1}|s_{t+1},a_t)$ in Equation (29) for continuous observation is derived by Bayes’ theorem:

$$\begin{array}{c}\multicolumn{1}{c}{p(o_{t+1}|s_i,a_t)=\frac{p(s_i|o_{t+1},a_t)f_p(o_{t+1})}{\underset{+\infty }{\overset{-\infty }{\int }}p(s_i|o_{t+1},a_t)f_p(o_{t+1})\mathrm{d}o}}\\ i\in (1,2,\dots ,d)\hfill \end{array}$$

(33)

Another discrepancy between POMDP and the model in this study is the hidden state transition. Generally, POMDP assumes that the state degradation only relies on the prior state probability transition matrix. The observation is utilized to detect the current hidden state since the measurements will not affect the results. However, these two assumptions are not consistent with the practical component-level management incorporating the SHM system due to the following aspects. As the fatigue problem intrinsically contains high uncertainty, the condition of the component estimated by monitoring data is a probabilistic distribution rather than confidence about the state. In addition, the prior state transition matrix established by the inspected database reflects the generally deteriorated behavior of the similar component. For structural-level management, this roughly statistical model may be acceptable. But in component-level management, this approximation could induce large discrepancies with specific component deterioration. Otherwise, all the components in the bridge can share the same maintenance strategy, which is contradictory to the component maintenance in different locations of the bridge, load cases, and environment. Thus, the concept that the state transition depends on the posterior matrix is proposed. The confidence factor $\eta \in [0,1]$ is introduced to combine the prior state transition matrix with observation. This coefficient reflects the agent’s confidence about the structural state decided by the prior transition matrix or observation. Zero indicates that the agent only believes prior state transition. One represents the updated belief state only depending on observation. The posterior belief distribution over the defining state 1–d is given:

$$\begin{array}{c}\multicolumn{1}{c}{p({\mathit{b}}_{\mathit{t}+1}|o_{t+1},a_t,{\mathit{b}}_{\mathit{t}})=p({\mathit{b}}_{\mathit{t}})\mathit{T}(a_t)}\\ +\eta p({\mathit{b}}_{\mathit{t}})(({\mathit{P}}_{\mathbf{DC}}{)}^{N_t}-\mathit{T}(a_t))\hfill \end{array}$$

(34)

Point-based algorithms in solving POMDP

To enhance the calculation efficiency, the point-based algorithm is proposed to alleviate the complexity in POMDP by avoiding the exponential increase of $\alpha$ vectors in every time step. The core of this algorithm is to restrict value iteration to a meaningful collection of discrete belief points, rather than calculate the maximum value $V(b)$ in every belief point. In infrastructure management, there is only a finite number of belief states that are actually visited. For instance, a component belief state has specific transferred paths based on the transition parameters T and Equation (34), rather than an arbitrary transition in the belief domain. These visited points are defined as the reachable belief point B. To further cut down the set of reachable points, the subset of reachable belief points $B^{*}$ under optimal sequences of actions is proposed. However, the method which can directly decide the optimal action in the arbitrary belief point does not exist. The recommended sampling approach uses successive approximations of the reachable space under optimal policies (SARSOP). This solver can avoid redundant searching in the suboptimal solution through heuristic exploration and information gathered from earlier samples. The detailed algorithm is introduced by Kurniawati et al. ⁵⁷

However, the continuous observation space could cause other issues when utilizing point-based algorithms to sample the representative belief points. The lossless partitioning of the observation space could induce large observation spaces that need to be considered in POMDPs. To alleviate the complexity of observation spaces, various online solvers^33,37,38 are developed, which use heuristic search in conjunction with branch-and-bound pruning to construct a look-ahead decision-making tree. Through interleave planning and plan execution, online solvers use the forward sampled scenarios to estimate the current belief value and decide the single best action for the current state. Although the online solver can dramatically decrease the computation, the solution achieved often falls sub-optimal. To obtain the optimal or near-optimal solution, the off-line solver needs to be employed, in conjunction with the algorithm which can “slim” the representative belief points. Therefore, by the consideration of the POMDPs features in infrastructure management, an efficient discretization strategy is proposed in this paper. Two characteristics in structural maintenance are utilized, (1) Generally, the consequence of failure is unacceptable in structure. Hence, the cost of failure state has high orders of magnitude compared with other states. (2) All the state transition matrix is unit-jump upper triangular matrix as indicated in Equation (5), except the replacing action. The mathematical property that the arbitrary value function $V(\mathit{b})$ is concave and would monotonically decrease with the belief state transition is proved in Appendix A. Motivated by this property, the large discrete observation space that selects the same action can be aggregated. The Kernel of the proposed method is to find the belief area in which different observations lead to choosing different actions, as Figure 7 shows the belief state ${\mathit{b}}_{\mathit{i}}~{{\mathit{b}}_{\mathit{i}}}_{+1}$ , and ${{\mathit{b}}_{\mathit{i}}}_{+3}~{{\mathit{b}}_{\mathit{i}}}_{+4}$ . Then, the observation in these areas needs to be fine-discretized which in turn can define the boundary. The detailed derivation and computational process are introduced in Appendix A steps 1–7.

Figure 7.

Adaptive discretization strategy in representative belief points sampling.

For the welded details, the observation in SHM is the number of stress cycles. Since the stress cycles are quantified as the unit-jump upper triangular matrix, the belief state is also monotonically increasing with the number of cycles. Hence, once it defines the belief state range corresponding to the same maintenance action in the optimal policy, the continuous observation that affects the belief state distribution in this range can be aggregated.

Illustrative example

The fatigue details in bridge are studied to demonstrate how to plan the optimal maintenance strategies based on long-term monitoring data. To possess the real-time condition of the bridge, a dynamic SHM system was installed which includes weldable foil-type strain gauges. Major strain transducers are attached to components prone to fatigue. In this paper, since the methodology is concentrated on individual component management, the most vulnerable fatigue joint is selected which experiences the maximum stress range. Figure 8 marks the site of sensors which is fixed under the top flange of the box girder. This detail connects to the top flange plate through the U-welded joint. In this paper, the parameters of welded detail are listed in Table 1. The material arguments (F2 class steel) C and m are determined by the BS-7910 code and fatigue experimental results. ³⁸

Figure 8.

Welded detail monitored in the box girder. (a) Marks the welded joint location and traffic and (b) Shows the geometry of welded details.

Table 1.

Parameters of welded detail.

Geometry parameters								Material parameters
$a_0$	$a_c$	T	$a/c$	$\theta$	$L/T$	W	$\phi$	C	$\mathrm{\Delta }{\sigma }_0$	m ( $\ge \mathrm{\Delta }{\sigma }_0$ )	m ( $<\mathrm{\Delta }{\sigma }_0$ )
0 mm	9 mm	10 mm	0.2	$\pi /4$	2	$\infty$	$\pi /2$	$3.60e-13$	35MPa	3	5

It should be noted here that the rate of crack propagation approaches infinity at the end of the failure phase. Since fragile behavior is unacceptable in the serviceability limit stage, conservative state thresholds need to be set. Herein, the states related to reliability are defined as infinite, 4, 3, 2, 1, based on the standard ⁴⁹ proposed. State 1 refers to the intact state whose no cracks are possible to detect, while state 5 implies the potentially hazardous state at the onset of fracture. The fatigue properties of F2 class steel $A_i$ in Table 2 are proposed by standard. ⁵⁸ Afterwards, the expected number of cycles $N_i$ corresponding to different reliability can be calculated by Equations (1) and (2). If DC is defined as 100 stress cycles (35 MPa as stress range), the STR $r_i$ in the unit jump matrix is obtained via Equations (2) and (5). All the parameters of the Markov chain model are given in Table 2. As mentioned before, corrosion reduces the section area of the component and increases the cycling impacts according to Equation (4). In the marine atmosphere, the proposed parameters ⁴³ in Equation (3) are listed in Table 3.

Table 2.

Markov chain model parameters.

Reliability (state)	4	3	2	1	Fatigue parameters in Equation (6)
Material fatigue property $A_i$	$0.15e12$	$0.25e12$	$0.43e12$	$0.64e12$	Variance $V(s_t)$	$2.58e+14$
Expected cycles number $N_i$	$3.5e6$	$5.9e6$	$1.0e7$	$1.5e7$	Failure cycles $N_{\mathrm{f}}$	$2.87e+07$
STR $r_i$	$3.5e4$	$2.4e4$	$4.1e4$	$4.8e4$	Matrix size d	5

Table 3.

Corrosion parameters.

Atmosphere	Site	Time of wetness (h/year)	$\nu (\mu \mathrm{m})$	$\lambda$
Light	Barcelona	3200	49.9	0.6–0.9
Severe	Alicante	4300	92.6	0.6–0.9
Severe	Hong Kong	4152	92.6	0.75

Data processing

The intensive sampling reproduces the original waveforms of the measurands, especially for extreme points. The representative measurements are shown in Figure 9(a), which has three characteristics: First, the strain pulses have a significant decrement from 1:30 am to 5:30 am as the train ceases its service during this period. It can infer that the strain pulses caused by the train are much larger than the vehicles by contrasting the amplitude of strain in daily strain time history. Moreover, the overall drift of strain is strongly synchronized with the temperature variation. However, the temperature strain does not produce stress in this welded detail because no constraints are in the strain direction. To separate temperature strain from the original signal, the maximal overlap discrete wavelet transform is used. This algorithm can decompose the signal into kth different frequency bands. Each band only contains the frequency that belongs to $[1/(2^{k+1}\mathrm{\Delta }t),1/(2^k\mathrm{\Delta }t)]$ , where the $\mathrm{\Delta }t$ is the sampling interval, and k is the integer from 0 to 17. After filtering the signal, a smooth curve is extracted under the frequency band $[0,1/(2^{18}\mathrm{\Delta }t)]$ , which is represented by the dark blue line in Figure 9(a). Afterwards, the strain response to the live load is obtained. The effective stress range $\mathrm{\Delta }\sigma$ and the number of cycles N are calculated by the rain-flow cycle counting technique. $\mathrm{\Delta }\sigma$ less than 2 MPa is not considered because their effects on fatigue life can be neglected. The statistical results of stress cycles are shown in histograms of Figure 10. Different stress amplitudes are adopted in Figure 10 according to the BS5400, the low-stress ranges should use a small interval to reduce the conservatism in fatigue assessment. An insight into Figure 10 reveals the daily traffic conditions. The motor vehicles induce high-frequency and low-stress (2–16 MPa) cycles, while the low-frequency and high-stress cycles (16–36 MPa) represent the railway impacts. Due to the combined effect of highway traffic, railway traffic, and wind loads, occasional impacts can reach the range of 36–50 MPa. Finally, the daily cumulative cycles N are calculated by linear superposition principle $n_i/N_0^i$ , where $n_i$ is the cycles in different stress range $\mathrm{\Delta }{\sigma }_i$ . Figure 11 displays 2017-year cycling impacts, which will be utilized to build the BDLMs to predict the fatigue cycles. If missed or distorted data occupy 30% in daily sampling, then those dynamic response data will not be used in the fatigue analysis. The data has a 20% missing due to the temporary electricity interruption or periodic maintenance for the SHM system, as Figure 11 shows.

Figure 9.

Typical daily strain spectra. The upper figure displays the temperature stain hidden in data. The lower picture is the live load strain.

Figure 10.

Stress spectra histogram. The different intervals are implemented based on the BS5400 recommendation.

Figure 11.

Whole year cumulative daily cycle. The fluctuation means the external impacts have randomness.

BDLM establishing

BDLM is constructed based on the daily stress cycling data in 2017, as Figure 11 shows. As mentioned before, the BDLM contains various block components where each block reflects one characteristic of data. The predicted model establishment is to attempt the different combinations of blocks to find a model that matches the data in Figure 11 with maximum likelihood value and acceptable error. Four representative BDLMs are assessed and listed in Table 4: (a) the AR model ( $\varphi$ and ${\sigma }_A$ in Equation (12)) with the local level block (normal distribution ${\mu }_L$ and ${\sigma }_L$ ), (b) the periodic model (p and ${\sigma }_S$ in Equation (11)) with local level blocks, (c) the AR model with the periodic and local level blocks, and (d) the AR model with periodic and trend blocks ( ${\mu }_T$ and ${\sigma }_T$ in Equation (10)). For brevity, the above models are abbreviated as LAR, LP, LPAR, and TPAR, respectively. The model parameters are calculated using a batch gradient descent algorithm with the objective function (20) and the terminated criterion (22). Because the autocorrelation function (ACF) exponentially reduces to zero with the lag increases and the partial ACF cuts off after lag 1, as Figure 12 shows. Three BDLMs choose the 1-order AR block according to the joint analysis of the ACF and partial ACF. The results of the four model parameters are summarized in Table 4. The last two items in Table 4 are used as quantifiable indicators of the fitting degree of BDLMs. The log-likelihood value indicates the probability which the monitoring data fall into the area defined by BDLM. The residual error $V~[0,{\sigma }_v]$ quantifies the features of data not reflected in BDLM. Accordingly, the periodic block and AR block are essential parts of the BDLMs. As indicated, the residual error decreases from 9.1 in LAR to 3.2 in LPAR and 19.9 in LP to 3.2 in LPAR. Although the TPAR model extracts a linearly decreasing function, the discrepancy induced by this function between LPAR and TPAR is negligible. To demonstrate whether this decrease is conformed to real situations, the statistics of actual traffic volume are analyzed. According to the Transport Department of Hong Kong SAR Government report, ⁵⁹ the railway volume is stationary based on the schedule in daily operation. In addition, the number of heavy trucks (>24 tonnes) presents fluctuation which has a minute increase from $9.4e5$ in 2017 to $9.5e5$ in 2018, but it decreases to $7.6e5$ in 2019. In general, the increment of heavy traffic volume reaches stabilization after bridge operation for several years. The small decrease in the TPAR model may represent the localized trend in long-term fluctuation. The daily fatigue impacts are assumed to obey the LPAR model. Using the parameters in LPAR, Figure 13 is drawn with Kalman smoother (fill the missing data) and KF (prediction). The black dots in Figure 13 is the prediction by executing a Monte Carlo simulation. If Monte Carlo simulations are carried out enough times, then the distribution of the cumulative number of cycles N will be obtained. This distribution is finally parameterized as the GMD pattern, defined in Equation (31). In this case, the accumulative number of cycles in a year can be represented using the Gaussian distribution $N~[75,823,{1149}^2]$ .

Figure 12.

ACF and PACF in the AR blocks. The upper plot demonstrates lag reduced to zero. The lower plot shows a lag decrease to zero after AR(1).

Table 4.

BDLMs model parameters.

Models	${\mu }_L$	${\sigma }_L$	$\varphi$	${\sigma }_A$	p	${\sigma }_S$	${\mu }_T$	${\sigma }_T$	Log-likelihood	Residual error ${\sigma }_v$
LAR	209.7	0	0.61	19.1	—	—	—	—	−1.204e+03	9.1
LP	207.7	0	—	—	365.2	0	—	—	−1.185e+03	19.9
LPAR	207.6	0	0.27	19.2	365.2	0	—	—	−1.178e+03	3.2
TPAR	215.9	—	0.26	19.0	365.2	0	−0.045	7.81e-6	−1.178e+03	3.8

BDLM: Bayesian dynamic linear model.

Figure 13.

BDLMs for cycling forecasting. The light blue area is the two standard deviations intervals. The black dash line indicates the expected value development.

Management model establishing

This section describes how the continuous POMDP is established for bridge welded details management. The aforementioned methods, such as the Markov chain model for fatigue, BDLM for prediction, and GMM for observation are integrated into continuous observation POMDP to develop a comprehensive management system based on monitoring data. POMDP includes 7-tuple arguments $〈\mathbb{S},\mathbb{A},\mathbb{O},\mathit{T},\mathit{O},\mathit{R},\gamma 〉$ . To conform to the fatigue failure distributions of F2 class steel, 5 states are defined based on Table 2 results. Those five states respectively correspond to reliability $\mathbb{S}=[+\infty ,4,3,2,1]$ . Four routine maintenance treatments recommended by the manual^56,60 are employed, which are “Do Nothing,”“Descaling and Painting,”“Rehabilitation,” and “Replacing” $(|\mathbb{A}|=4)$ . It should be noted that the “Rehabilitation” action is post-weld maintenance techniques, such as burr grinding or hammer peening. This action aims to maintain the component condition and avoid further deterioration, while it does not fundamentally enhance the performance of the component. The set of actions $\mathbb{A}$ determine different prior state transition matrices T . For the “Do Nothing” action, if the coating is valid for corrosion resistance, the prior state transition matrix per year ${\mathit{T}}_1^0$ is given according to the statistic of welded details degradation.^49,61 The superscript of ${\mathit{T}}_1^0$ donates the corrosion phase.

$${\mathit{T}}_1^0=\left(\begin{array}{ccccc}0.964\hfill & 0.036\hfill & 0\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0.949\hfill & 0.051\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0.974\hfill & 0.026\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & 0.978\hfill & 0.022\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & 0\hfill & 1\hfill \end{array}\right)$$

The corrosion will accelerate the deterioration of welded joints at three different rates. In phase 1, the accumulative exposure time is assumed less than 15 years and the corrosion accelerates the degradation 1.8 times. In POMDP, this acceleration coefficient is quantified by discounting the parameter of STR $r_i$ 1.8 times in the transition matrix ${\mathit{T}}_1^0$ . The prior state transition matrix changes to:

$${\mathit{T}}_1^1=\left(\begin{array}{ccccc}0.937\hfill & 0.063\hfill & 0\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0.912\hfill & 0.088\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0.954\hfill & 0.046\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & 0.961\hfill & 0.039\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & 0\hfill & 1\hfill \end{array}\right)$$

For phase 2, the joint is cumulatively exposed to an erosive environment for 15–30 years, and the accelerated coefficient is assumed as 2.6. For phase 3 (>30 years), this value increases to 3.4. The “Descaling and Painting” action shares the state transition matrix $({\mathit{T}}_2={\mathit{T}}_1)$ as the “Do nothing” action because the coating will not affect the cross-section. The utility of this action is to avoid welded joint erosion in the next 4 years. But the cumulative exposure time will not reset to 0 since the corrosion process is irreversible. The “Rehabilitation” can effectively prevent further degradation. This property is quantified by the large value of STR $(r_i=99)$ in the state transition matrix ${\mathit{T}}_3$ which indicates the component can maintain the current state for a long time. This action also recovers the coating to its initial condition.

$${\mathit{T}}_3=\left(\begin{array}{ccccc}0.99\hfill & 0.01\hfill & 0\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0.99\hfill & 0.01\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0.99\hfill & 0.01\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & 0.99\hfill & 0.01\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & 0\hfill & 1\hfill \end{array}\right)$$

The state transition matrix ${\mathit{T}}_4$ in the “Replacing” action is assumed the component state has a 90% probability to restore the initial state and 10% to state 2. Additionally, the accumulative exposure time in a corrosive environment will be reset to 0.

$${\mathit{T}}_4=\left(\begin{array}{ccccc}0.9\hfill & 0.1\hfill & 0\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0.9}& 0.1\hfill & 0\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0.9}& 0.1\hfill & 0\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0.9}& 0.1\hfill & 0\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0.9}& 0.1\hfill & 0\hfill & 0\hfill & 0\hfill \end{array}\right)$$

The observation parameters in continuous POMDPs are evolved from the probability matrix to the probability density functions. After the BDLMs are established, a stochastic observation value $o_{t+1}$ is obtained based on GMM in Equation (31). The observed belief state is updated by Equation (32), where parameter ${\mathit{P}}_{\mathbf{DC}}$ in Equation (32) is given via Equation (5) and STRs in Table 2. For instance, if an observed value $o=74,000$ $(N=740)$ from the SHM system is predicted and the previous belief state is ${\mathit{b}}_{\mathbf{t}}=[0.7,0.3,0,0,0]$ , then the observed belief vector based on observation is given by Equation (32):

$${\mathit{b}}_{\mathit{t}+1}^{\prime }={\mathit{b}}_{\mathit{t}}({\mathit{P}}_{\mathbf{DC}}{)}^N=[0.6855,0.3052,0.0093,0.0001,0]$$

It should be mentioned that the different maintenance actions also affect the scope of observation results. When the welded joint has been cumulatively exposed to the erosive atmosphere for t years, the possible distribution cycling impacts during a time step are amplified as $[{\mathrm{\Lambda }}^m\mu ,{\mathrm{\Lambda }}^m\sigma ]$ based on Equation (4). For the “Rehabilitation” action, the $({\mathit{P}}_{\mathbf{DC}}{)}^N$ is assumed identical with the ${\mathit{T}}_3$ , which maintains the component current state. When the manager executes the ’Replacing’ action, the $({\mathit{P}}_{\mathbf{DC}}{)}^N$ equals ${\mathit{T}}_4$ and the cumulative exposure time resets to zero. Finally, the posterior belief state ${\mathit{b}}_{\mathit{t}+1}$ is updated by Equations (34) and (24). For the value updating $V(\mathit{b})$ , the probability density of $p(o_{t+1}|s_i,a_t)$ in Equation (29) is calculated through Bayes’ theorem (Equation (33)). The last two arguments $〈\mathit{R},\gamma 〉$ are associated with the economic indicators. Herein, two categories of rewards are assumed based on the reasonable cost of maintenance and the consequence of failure. The details of the reward R are listed in Table 5. In this paper, the reward discounting factor γ equals 0.95, which represents the decreasing value of future rewards.

Table 5.

Reward matrix table ($).

Conditional level	$s_1$	$s_2$	$s_3$	$s_4$	$s_5$
Do Nothing	0	−2k	−4k	−8k	−400k
Descaling and Painting	−2k	−4k	−6k	−10k	−400k
Rehabilitation	−10k	−12k	−14k	−18k	−400k
Replacing	−200k	−200k	−200k	−200k	−400k

Results and discussions

Before gaining further insight into the maintenance behavior in optimal policy, the characteristics of modified POMDP will be emphasized again because these concepts are commonly overlooked and significant. It is worth noting that the conventional POMDP assumes that the hidden state transmission only depends on the prior state transition matrix T . The observation is utilized to reduce the uncertainty of the belief state, which can avoid unnecessary maintenance actions. A concept named value of information is proposed as a metric to quantify the saving when the manager makes decisions based on observation. The different POMDPs can effectively quantify the benefit of different “accuracy level” SHM systems. Nevertheless, the accuracy of a real-world SHM system is difficult to evaluate because it is affected by several factors, such as sensor accuracy, sampling frequency, and damage evaluation algorithms. Additionally, the prior state transition matrix representing the deteriorated mechanism of a specific welded detail is difficult to obtain because most of the inspection database does not classify the component according to external factors. Consequently, the observation system in continuous POMDP is an intermediate to consider the external stochastic influences. The prior state transition matrix T summarizes the general deteriorated regular from historical statistics of similar components. The observation matrix $({\mathit{P}}_{\mathbf{DC}}{)}^N$ is used to update the prior state transition matrix T in the specific ambient. The behavior of hidden state transformation is more in line with the law expressed by the posterior state transition matrix. The confidence factor $\eta$ in Equation (34) quantifies how much information from observation is used in updating. In this paper, this factor is assumed as $\eta =0.5$ considering the error from three aspects: the fatigue failure is prominent in metal welded detail but not unique, the cumulative forecast error from BDLMs, and the empirical formula.

To illustrate how the welded joint degradation, the expected belief state deteriorated curves based on the prior state transition matrix and stress cycles from observation are separately plotted in Figure 9. Afterwards, the belief state evolution is obtained based on the posterior transition matrix $(\mathit{T}+{{\mathit{P}}_{\mathbf{DC}}}^N)/2$ , as depicted by the black lines in Figure 14. The blue area between corrosion and protection is the reachable belief state domain. In bridge management, the manager has a risk threshold which is quantified by the likelihood times consequence. The optimal scheme pursues an appropriate trade-off between failure risk and the cost of the replacement component. Therefore, the optimal reachable belief state domain will be restricted in the gray area to avoid excessive risk. The size of this area is controlled by the punishment $r(a,s_5)$ value (failure consequence).

Figure 14.

A schematic of component expected deteriorated curve. Notice that the dash-dot line refers to steel with perfect painting protection. The solid line is the deterioration in natural corrosion. The blue area B is the belief reachable space. The dark gray area $B^{*}$ shows the reachable belief points in the optimal policy.

According to the reward parameters in Table 6, the comparison between the SARSOP algorithm and proposed discretization strategy in Appendix A is listed in Table 6. Since the maintenance fees will dramatically affect the optimal management behaviors, the sensitivity of price is studied by altering the reward parameter R in POMDPs. The optimal welded detail management is displayed in Figures 15 and 16. Three maintenance actions are marked with different symbols, “Descaling and painting” marked with a triangle, “Rehabilitation” with a square, and “Replacing” with a diamond. The background of Figures 15 and16 is the belief state cumulative histogram, which reflects the agent’s confidence about the current component condition. The dashed line denotes the hidden state transition which is not accessible to the agent in practical engineering. The hidden state plotted in figures is regarded as a reference to check the reasonable maintenance actions adopted. For instance, the “Replacing” action should be implemented when a component is working in state 3 for a long period or deteriorates to state 4. Figure 15 unfolds the optimal intervened behaviors of the welded joint based on reward Table 5. Three phenomenons can be observed: (1) The “Descaling and painting” action is periodically executed per 4 years to avoid the cross-section reduction of welded joints by corrosion. Although the influence of cross-section reduction is slight in the early stage, the additional cumulative cycling will significantly affect the useful lifespan, according to Equation (4). (2) It is interesting information to know that preventive maintenance is not adopted when the component approaches its end life, because the cost of action is larger than the expected return. (3) “Rehabilitation” is not adopted since the low-cost performance is not competitive among other maintenance.

Table 6.

Solver comparison table.

Problem size: $|\mathbb{S}|=5^4$ , $|\mathbb{A}|=4$ , $|\mathbb{O}|=7840$
Solvers	Reward	Times (h)
SARSOP	$-1.84e3\pm 0.06e3$	5.7
Adaptive discretization	$-1.87e3\pm 0.08e3$	1

SARSOP: successive approximations of the reachable space under optimal policies.

Figure 15.

Policy realization of welded detail management. “Rehabilitation” is not adopted as an uneconomical action (10 k$).

Figure 16.

Policy realization of welded detail management with economical “Rehabilitation” action. With the decrement of “Rehabilitation” cost, the adopted frequency of this action has significant growth: (a) ``Rehabilitation'' cost is –5, (b) ``Rehabilitation'' cost is –4, (c) ``Rehabilitation'' cost is –3, and (d) ``Rehabilitation'' cost is –2.

The effects of cost performance in infrastructure management are investigated. The change in optimal maintenance strategy is observed when the cost of “Rehabilitation” is decreased. For the case that the cost of “Rehabilitation” decreases to −5, it will first be adopted in the mid-life time of the component, as shown in Figure 16(a). This behavior can be explained by the feature of the prior state transition matrices. In a welded detail naturally degradation (“Do nothing”), the capability of the component to maintain state 2 is the weakest ( $r_2=18.61$ in ${\mathit{T}}_1^0$ ) compared with other states. Meanwhile, the “Rehabilitation” action has the identical capability to maintain every state ( $r_i=99$ in T ₃ ), which indicates this action achieves the maximum utility in state 2 when it substitutes the “Do nothing” action. Therefore, once the “Rehabilitation” has a higher cost performance than other maintenance actions, it will be adopted. For case II, the expenditure of “Rehabilitation” decrease to −4, and it will gradually substitute the “Descaling and painting” action, as shown in Figure 16(b). For the last case, the cost of the “Rehabilitation” action decreases to −2 and becomes the most cost-effective action in maintenance. As indicated in Figure 16(c) and (d), this action will be frequently used. It is also worth noting that the deterioration rate will fluctuate due to the uncertainty in the prior state transition matrix and prediction of BDLM. This randomness induces a slight difference of sequential actions in the same POMDP, as shown in Figures 15 and 16. In summary, the reward matrix is a key parameter that could significantly affect the optimal policy ${\pi }^{*}$ when the prior state transition matrices, maintenance utility, and the prediction of observation are established.

The different maintenance actions present a competitive and complementary relationship based on their cost performance. To illustrate the effect of the cost-performance on the optimal maintenance policy, the concept of the “barrel effect” is utilized to present the competitive and complementary relationship among different interventions (Figure 17). When the maintenance utility ( $\mathit{T},\mathit{O},\mathit{A}$ ) is determined in all POMDPs, the cost performance of action is inversely proportional to the reward parameter ( R ). The cost of action determines the length of the barrel board, and the capacity of the barrel is the average maintenance fee per year. Four barrels correspond to study cases in Figures 15 and 16(a) to (c), and the number on the board is the action reward from state 1 (bottom) to state 5 (top). Then, uneconomical action as the longboard would not affect the capacity of the barrel. With the decrement of the “Rehabilitation” action cost, the “Rehabilitation” action changed from a longboard to a short one, and the expected maintenance fee per year decreased from 9.2k to 6.8k.

Figure 17.

Barrel effect in decision-making. The value in the board denotes the cost of action in the different states according to Table 5.

However, the continuous POMDP has some limitations in application. The computation requirements considerably increase with solving large discretization observation spaces. For a welded joint, a computer equipped with an AMD 3900X and 64 GB memory still needs 1 hour to converge to the optimal solution. For large multi-component systems, individual component failure may not significantly affect the reliability of the entire structure. It requires the model to consider the synergy effect of component degradation act on structural safety. The offline solver cannot be applied to this scenario as the state and action spaces dramatically increase for the system-level management. The online solver, such as Actor-Critic, can alleviate the computation for optimal policy searching, but it cannot tackle the time-varying problem. Future work in sequential decision-making for complex engineering systems includes using dedicated deep reinforcement learning methods to solve the non-homogeneous deterioration problem and explorative training algorithms to ensure the model convergences.

Conclusions

A comprehensive decision-making framework for addressing the intelligent life-cycle management based on a real-world SHM system is presented in this paper. To realize this framework, the continuous observation POMDP is chosen as the theoretical foundation on the basis of its capability of formulating optimal sequential decision-making under stochastic environments with uncertain action outcomes and noisy observations. Through the Markov chain, the fatigue process is quantified as the discrete state transition in POMDP, which provides an alternative way to assess the condition of welded details based on long-term monitoring data. Further, to forecast the cumulative stress cycles of welded details in the life-cycle, the BDLM is used on the basis of its mathematical properties for a flexible combination of different blocks to characterize the data features and uncertainty in dynamic responses. Finally, based on the characteristic of infrastructure management, this study demonstrates the value function is concave and monotone decreasing with the expected belief state increasing. Using this property, an adaptive discretization strategy is designed to efficiently compute the optimal or near-optimal policy for continuous observation POMDPs.

A constructive concept about state transition in POMDP is proposed in this paper. The hidden state degradation is not merely reliant on the prior state transition matrix because stationary matrices cannot reflect the dynamic environmental evolution and uncertainty in external impacts. The state transition is determined by the posterior matrix which simultaneously considers the prior state transition matrix (general law) and observations (actual situation). According to the statistical results of maintenance behavior on a minimum budget constraint, the following specific conclusions can be drawn: (i) In infrastructure management, three factors will significantly affect the maintenance policy in the life-cycle: maintenance utility, deterioration rate, and cost of action. The optimal solution is to find the trade-off between long-short-term utility and cost. (ii) The available maintenance actions present a competitive and complementary relationship based on their cost and utility. The decision-making follows the “barrel effect,” which means that maintenance action with low cost-performance is the longboard and it would not affect the barrel capacity (cost). (iii) When a welded detail approaches the end of its lifespan, interventions to extend the lifetime are uneconomical. The “Do nothing” action is executed until replacing the detail to save the cost as much as possible.