A novel concept and assessment method for trustworthiness of prognostics

Abstract

The existence of the unavoidable uncertainties has a great effect on the prognostics accuracy and performance. However, their influences are rarely taken into account in the current prognostics performance evaluation. Consequently, the validity and/or trustiness degree of prognostics and health management system has not been comprehensively evaluated or measured. Considering the definition of prognostics performance and the characteristics of uncertainties, four prognostics performance metrics and two kinds of quantitative evaluation methods are proposed. On the basis of prognostics performance definition, a novel trustworthiness of prognostics concept is presented. Then, three kinds of assessment methods for trustworthiness of prognostics are provided. Finally, a case study has been carried out on the aviation hydraulic pump for verification purposes, in which the oil-return flow is monitored for predicting the degradation of the pump. The systematic application of trustworthiness of prognostics assessment is implemented based on the back propagation neural network prognostics algorithm. The results show that the proposed trustworthiness of prognostics concept and assessment methods can contribute to the systematic and comprehensive understanding of the performance of prognostics and health management system.

Introduction

Prognostics refers to the process of predicting the future reliability of a product by assessing the extent of deviation or degradation from expected normal operating conditions. ¹ In the process of prognostics, various types of data are comprehensively utilized, such as monitoring parameters (e.g. environmental conditions and physical loads), field or test data, and historical experience. The remaining useful life (RUL) of products or systems can be evaluated and the future health or reliability state can be predicted by utilizing various prognostic methods and models.

In recent years, researchers have carried out a number of studies on prognostics methods. However, whatever method is adopted, the unavoidable uncertainties exist in the process of data acquisition, data transmission, and data processing. These uncertainties have a significant effect on the accuracy of RUL prediction and the applicability of prognostics methods. When the state estimates, future loading conditions, and operating conditions are uncertain, the future states and the RUL also become uncertain. ² Besides, the existence of uncertainties also increases the difficulty of verification and validation (V&V). Essentially, it is necessary to understand and manage the uncertainties in the process of prognostics.

The main contents of uncertainty management includes (1) analyzing the uncertainty sources, (2) assigning a probability density function (PDF) around the RUL estimate, and (3) calculating the confidence bounds for the estimate of a significant system reliability measure, such as failure probability distribution and failure rate. Furthermore, regarding the advanced uncertainty management framework, the algorithms or methods used to reduce the uncertainties are also involved. The uncertainty management is crucial and helpful for decision makers to know the time to take action offline or online on the basis of the current risk tolerance. ³

As one of the intrinsic properties of prognostics and health management (PHM) systems, prognostics performance is similar to the concept of product performance, which refers to the nature and ability of a prognostics method to perform required functions and obtain expected results by making use of specific resources. Prognostics performance has a direct impact on the development of maintenance strategies for end users. Therefore, effective performance evaluation methods are necessary for building and qualifying a real and mature PHM system. Nevertheless, current researches concerning performance evaluation demonstrate that there is no general agreement on an appropriate and acceptable set of metrics which can be effectively employed to evaluate the technical performance of prognostics.^4–6 In addition, the influences of uncertainties are rarely considered in the existing performance evaluation methods and processes, which result in an unsatisfactory result of requirements.^7–10 Consequently, the validity and/or trustiness degree of PHM system has not been comprehensively evaluated.

This article proposes four prognostics performance metrics and two kinds of quantitative evaluation methods for prognostics performance. On the basis of prognostics performance definition, a novel trustworthiness of prognostics (ToP) concept is presented. Then, three kinds of assessment methods for ToP are provided. Finally, a case study has been carried out on the aviation hydraulic pump for verification purposes, in which the oil-return flow (ORF) is monitored for predicting the degradation of the pump. The systematic application of ToP assessment is implemented based on the back propagation (BP) neural network prognostics algorithm.

Prognostics performance and evaluation

Concept of prognostics performance

A clear and systematic definition of prognostics performance has not been found in the published literatures underlining the prognostics performance and evaluations. Aiming to promote the unified cognition, a concept is proposed as follows, which is based on what has been put forward by Liu and Sun. ¹¹

Definition 1

Prognostics performance refers to the nature and ability of a prognostics method or PHM system to perform a required function and obtain expected results by making use of specified resources.

Among above definition, a required function means the adaptive capacity of a prognostics method or system. The expected results refer to the precision and accuracy of the predictive results. The specified resources include computing time (CT), computer’s memory space, and other resources which are required in the implementation process of prognostics methods.

Prognostics performance metrics

In order to fully describe prognostics performance, the following four prognostics evaluation metrics are proposed, comprehensively taking the characteristics of uncertainties into consideration.

Quality of prognostics

The quality of prognostics results mainly involves prediction accuracy and precision. The prediction accuracy is usually adopted to measure the degree of closeness between actual values and estimated values, while the prediction precision is employed to measure the aggregation extent of estimated values.

In Figure 1, E{P} is the mean of the probability distribution of predicted value P₁ (or P₂), and E{A} is the mean of the probability distribution of actual value A. With the lower limit a and the upper limit b, the acceptable zone can be expressed as follows

$$\mathrm{a}=\mathrm{E}\left\{\mathrm{A}\right\}-{\mathrm{\alpha }}^{-};\mathrm{b}=\mathrm{E}\left\{\mathrm{A}\right\}+{\mathrm{\alpha }}^{+}$$

(1)

where α means the acceptable risk level, which is determined by the end users according to the actual failure severity and criticality.

Figure 1.

Schematic explanation of prognostics quality: (a) continuous situations and (b) discrete situations.

As can be seen from Figure 1(a), the predicted value P₁ is as accurate as P₂ according to the above definitions. However, taking uncertainties into account, it is believed that P₁ is more accurate than P₂ because the probability range of P₁ is larger than that of P₂, which falls into the acceptable zone. Similarly, P₂ is considered to be more precise than P₁ because compared with P₁, the variance of P₂ is smaller than that of actual value distribution A. For simultaneously describing the above aspects, two novel metrics with new conceptual meaning are proposed for prognostics performance evaluation.

Relative accuracy

Saxena et al. ¹² presented relative accuracy (RA) as a measure of error in RUL prediction relative to the actual RUL at a specific time index. However, the function and its parameters of this metric are difficult to use in the actual application. In this article, RA is defined as the probability of all the predicted values falling into the acceptable zone. As shown in Figure 1(a), Area(P₁) and Area(P₂) are for continuous situations, and Area(P) is for discrete situations as shown in Figure 1(b). The RA can be expressed as follows

$$\mathrm{RA}=\{\begin{array}{c}\hfill \underset{a}{\overset{b}{\int }}{\phi }_P\left(x\right)\mathrm{dx}\mathrm{Continuous\; distribution}\hfill \\ \multicolumn{1}{c}{\sum _a^b\phi \left(x\right)\mathrm{Discrete\; distribution}}\end{array}$$

(2)

where ${\phi }_P(x)$ is the PDF of the predicted value and $\phi (x)$ is the probability histogram of the predicted value.

Relative precision

Relative precision (RP) is adopted to measure the consistency level between confidence intervals of predicted value and actual value at the same confidence level. The RP can be expressed as follows

$$\mathrm{RP}=\{\begin{array}{cc}\hfill 1-\frac{\mathrm{C}{\mathrm{I}}_{\mathrm{A}}-\mathrm{C}{\mathrm{I}}_{\mathrm{P}}}{\mathrm{C}{\mathrm{I}}_{\mathrm{A}}}\hfill & \hfill \mathrm{if\; C}{\mathrm{I}}_{\mathrm{A}}\ge \mathrm{C}{\mathrm{I}}_{\mathrm{P}}\hfill \\ \multicolumn{1}{c}{1-\frac{\mathrm{C}{\mathrm{I}}_{\mathrm{P}}-\mathrm{C}{\mathrm{I}}_{\mathrm{A}}}{\mathrm{C}{\mathrm{I}}_{max}-\mathrm{C}{\mathrm{I}}_{\mathrm{A}}}}& \hfill \mathrm{if\; C}{\mathrm{I}}_{\mathrm{A}}\le \mathrm{C}{\mathrm{I}}_{\mathrm{P}}<\mathrm{C}{\mathrm{I}}_{max}\hfill \\ \multicolumn{1}{c}{0}& \hfill \mathrm{if\; C}{\mathrm{I}}_{\mathrm{P}}\ge \mathrm{C}{\mathrm{I}}_{max}\hfill \end{array}$$

(3)

where CI_A and CI_P are, respectively, confidence intervals of actual value and predicted value at a given confidence level, which are recommended to determined according to historical or field data or relevant reliability test data. CI_max is the maximum tolerance interval of the predicted values defined by end users according to the actual acceptable risk level.

Timeliness of prognostics

As shown in Figure 2, when the RUL predicted value fall into the acceptable zone at a certain moment t_i for the first time, the RUL predicted values after t_i will all fall into the acceptable zone. The time span between any moment t_j and the actual time at which the system failure occurred is defined as prognostic horizon (PH), which can be used to measure the timeliness of prognostics ¹²

$$\mathrm{PH}=t_{\mathrm{Eo}{\mathrm{L}}^{-t_j}}$$

(4)

where t_EoL is the time for the end of life (EoL) and t_j represents any moment.

Figure 2.

Illustration of prognostics horizon.

Whether RUL fall into the acceptable zone or not, the criteria are RA ≥ α% and RP ≥ β%, where α% and β% are the judgment thresholds for specific requirement end users.

Comprehensive prognostics quality and algorithm convergence

Based on all the evaluation metrics within the PH scope, the comprehensive measure of the quality of prognostics results and convergence of algorithm can be obtained. Similar to the comprehensive relative accuracy (CRA) concept presented in Saxena et al., ¹² the CRA and comprehensive relative precision (CRP) are also proposed in this article. The formula can be expressed as follows

$$\mathrm{CRA}=\sum _{i=1}^n{w}_i\times \mathrm{R}{\mathrm{A}}_i$$

(5)

$$\mathrm{CRP}=\sum _{i=1}^n{w}_i\times \mathrm{R}{\mathrm{P}}_i$$

(6)

where n is the evaluation times within PH scope and w_i is the weight of the prognostics result at a certain moment. The range of w_i is [0, 1].

When calculating CRA and CRP, more attention will be paid to predicted values near the failure point, and a higher weight will be given. The weight equation can be expressed as follows

$$w_i=\frac{p_i}{\sum _{i=1}^n{p}_i};p_i=\frac{1}{t_{\mathrm{EoL}}-t_i}$$

(7)

Resource occupancy

The CT synthetically reflects the complexity of the prognostics algorithms and the requirements for the performance of hardware. In consequence, the mean of CTs can be employed to comprehensively measure the resource occupancy (RO) of the prognostics algorithms. The formula can be expressed as follows

$$\mathrm{RO}=\mathrm{E}\left\{\mathrm{CT}\right\}$$

(8)

Methods for prognostics performance evaluation

In this article, two kinds of systems are discussed. One is in-service system in working status. In this case, the objective of prognostics is to predict RUL of these systems. Another is broken system with known failure information, which is also called verification system. According to the different types of objects or systems, the prognostics performance evaluation methods can be divided into offline method and online method. The offline method focuses on comparison of performance of different PHM systems or prognostics algorithm or models. It is usually applied for verification system at the design stage of PHM systems. The online method emphasizes on the evaluation of prediction results of prognostics algorithm or models for the in-service system. It is generally adopted for the in-service system at the practical application stage of PHM systems.

Offline prognostics performance evaluation

In ideal situation, the ideal RUL ( ${\mathrm{RUL}}_{\mathrm{ideal}}^i$ ) of verification system at any time t_i can be calculated using the following equation

$${\mathrm{RUL}}_{\mathrm{ideal}}^i=t_{\mathrm{EoL}}-t_i$$

(9)

As shown in Figure 3, the ideal RUL curve can be obtained. The reference system is called the base of the offline prognostics performance evaluation.

Figure 3.

The base of the offline evaluation.

From normal working moment, the curve in Figure 3 represents the actual RUL of the verification system at any time on the condition that the verification system has been doomed to fail at a unique failure time. However, the RUL of the verification system will become uncertain due to load environment uncertainties experienced by systems. Hence, before the system fails, the actual failure time cannot be judged theoretically. When the actual load environment of systems rarely changes or the failure process of systems is not sensitive to load environment changes, the actual RUL of systems can usually be replaced by the ideal RUL curve. On the basis of the above explanation, the procedure of the offline performance evaluation method is shown in Figure 4.

Figure 4.

Flowchart of the offline evaluation methods for prognostics performance.

Online prognostics performance evaluation

With the change in the prediction time, the predictions and results of the in-service system are constantly updated. When the prediction time is closer to $t_{\mathrm{EoL}}$ , the prognostics quality will become higher. Nevertheless, the moment that is infinitely close to $t_{\mathrm{EoL}}$ cannot be chosen to obtain the highest quality in practical applications. Moreover, the prognostics timeliness outputs need to be considered as well. Due to the limitation of the time consumption for PHM system running and corresponding maintenance activities, there is an end time of useful predictions $t_{\mathrm{EoUP}}$ . ¹² The period of time between $t_{\mathrm{EoUP}}$ and $t_{\mathrm{EoL}}$ is called minimum effective prediction horizon (MEPH). ¹¹ At the moment $t_{\mathrm{EoUP}}$ , the most effective prediction of failure time for the in-service system can be realized. If the results of prognostics quality cannot satisfy the end user’s requirements at this moment, the PHM system will be invalid for this in-service system.

According to the definition of MEPH, the relationship among $t_{\mathrm{EoUP}}$ , $t_{\mathrm{EoL}}$ , CT, and the repair time (RT) can be obtained using the following equation

$$t_{\mathrm{EoUP}}=t_{\mathrm{EoL}}-(\mathrm{CT}+\mathrm{RT})$$

(10)

where $\mathrm{MEPH}=\mathrm{CT}+\mathrm{RT}$ , then

$$t_{\mathrm{EoUP}}=t_{\mathrm{EoL}}-\mathrm{MEPH}$$

(11)

In order to predict the system state before the moment $t_{\mathrm{EoL}}$ , the $t_{\mathrm{EoUP}}$ can be regarded as the continuation of the system state prediction within the period of time MEPH. As shown in Figure 5, the system state at the moment $t_{k+i+\mathrm{MEPH}}$ is predicted at the moment $t_{k+i}$ , which is associated with triangle lines. As long as $t_{k+i+\mathrm{MEPH}}\le t_{\mathrm{EoUP}}$ , the state parameter of the system at moment $t_{k+i+\mathrm{MEPH}}$ can be adopted to evaluate the prediction at the moment $t_{k+i}$ . By integrating these evaluation results, the online prognostics performance evaluation can be realized within the period of time MEPH, and the prognostics quality at the moment $t_{\mathrm{EoL}}$ can be determined at the moment $t_{\mathrm{EoUP}}$ .

Figure 5.

Schematic explanation of the online prognostics performance evaluation.

With the change in the loads, the actual characteristic parameters of the system may be changed significantly. On account of the lag of the algorithm, a number of prediction results that significantly deviate from the actual state of characteristic parameter distribution may occur, such as prediction results at $t_{k+i}$ (i = 4, 5, and 6), as shown in Figure 5. Therefore, the triangle lines ③, ④, and ⑤ should be ignored in the process of evaluation. The procedure of the online evaluation method for prognostics performance is shown in Figure 6.

Figure 6.

Flowchart of the online evaluation method for prognostics performance.

Trustworthiness of prognostics and assessment

Concept of ToP

Trustworthiness usually refers to the degree of trustiness that a simulation system (as the alternative system of the prototype system) can repeat the prototype system with a particular purpose and means level of modeling and simulation on the overall structure and behavior. As a special simulation system, the prognostics or PHM system is provided with a similar connotation of trustworthiness to that of the simulation model. The definition of prognostics trustworthiness is proposed as follows.

Definition 2

ToP refers to the comprehensive measure of the degree of trustiness that the prognostics or PHM system can obtain in a specific application under the requirements of end users.

Essentially, the ToP is the remeasurement of prognostics performance. If a prognostics system is deemed to be trustworthy, it means that the prognostics system should be able to conform to the actual degradation process of an actual product or system regarding the overall behavior. However, the changes of external loads and environmental conditions will bring great uncertainty in the actual performance degradation of the actual system. The application object and their load environment shall be made clear in the definition of ToP. ToP is a metric used to assess the effectiveness of a prognostics method or model. Correspondingly, ToP assessment is a subjective recognition process employed to determine whether the prognostics results are effective enough to be accepted or not. As a result, the ToP is constrained by the specific end user’s requirements.

Methods for ToP assessment

A high-performance PHM system does not necessarily obtain a higher ToP in actual application. One of the main reasons is that the prognostics performance evaluation is not combined with the specific in-service systems and application conditions. According to the definition of ToP, the prognostics performance evaluation can be involved in the ToP assessment for a specific application by considering the requirements of end users. Similarly, the ToP assessment methods can also be divided into offline method and online method.

Offline ToP assessment

Similar to the offline prognostics performance evaluation, the offline ToP assessment method is mainly adopted for verification system. The procedure of the offline ToP assessment method is shown in Figure 7. The $\mathrm{To}{\mathrm{P}}_{\mathrm{offline}}$ can be expressed using the following equation

$$\mathrm{To}{\mathrm{P}}_{\mathrm{offline}}=\mathrm{I}{\mathrm{F}}_{\mathrm{PH}}\times (w_{\mathrm{CRA}}\times \mathrm{CRA}+w_{\mathrm{CRP}}\times \mathrm{CRP})$$

(12)

where $\mathrm{I}{\mathrm{F}}_{\mathrm{PH}}$ is the impact factor of PH. As for PH < CT + RT, $\mathrm{I}{\mathrm{F}}_{\mathrm{PH}}=0$ or else $\mathrm{I}{\mathrm{F}}_{\mathrm{PH}}=1$ . $w_{\mathrm{CRA}}$ and $w_{\mathrm{CRP}}$ , respectively, represent the weights of CRA and CRP evaluation accounting the percentage of total ToP, and hence

$$w_{\mathrm{CRA}}=\frac{\frac{1}{\mathrm{CRA}}}{(\frac{1}{\mathrm{CRA}}+\frac{1}{\mathrm{CRP}})}$$

(13)

$$w_{\mathrm{CRP}}=\frac{\frac{1}{\mathrm{CRP}}}{(\frac{1}{\mathrm{CRA}}+\frac{1}{\mathrm{CRP}})}$$

(14)

Figure 7.

Flowchart of the offline ToP assessment method.

Online ToP assessment

The online ToP assessment is employed for the in-service system. Based on the online prognostics performance evaluation method, the procedure of the online ToP assessment method is shown in Figure 8.

Figure 8.

Flowchart of the online ToP assessment method.

The $\mathrm{To}{\mathrm{P}}_{\mathrm{online}}$ be expressed using the following equation

$$\mathrm{To}{\mathrm{P}}_{\mathrm{online}}=w_{\mathrm{RA}}\times \mathrm{RA}+w_{\mathrm{RP}}\times \mathrm{RP}$$

(15)

where $w_{\mathrm{RA}}$ and $w_{\mathrm{RP}}$ represent the weights of RA and RP evaluation accounting the percentage of total ToP, which can be calculated according to the similar formulae (13) and (14) with RA and RP instead, respectively.

In addition to the prognostics subsystem, the data acquisition subsystem, the data processing subsystem, the condition monitoring subsystem, the health assessment subsystem, and other subsystems are also involved in total PHM system. Each of these subsystems works independently without affecting each other and possesses its own ToP. Therefore, the ToP of each subsystem needs to be comprehensively considered in the process of online assessment. The ToP of total PHM system can be gained as follows

$$\mathrm{ToP}\left(\mathrm{S}\right)=\mathrm{ToP}\left({\mathrm{S}}_1\right)\times \mathrm{ToP}\left({\mathrm{S}}_2\right)\times ···\times \mathrm{ToP}\left({\mathrm{S}}_{\mathrm{r}}\right)$$

(16)

where ToP(S₁), ToP(S₂), …, and ToP(S_r), respectively, represent the ToP of each subsystem of PHM system.

Comprehensive ToP assessment

The schematic diagram of comprehensive ToP assessment is shown in Figure 9. Furthermore, the weighted average method is adopted for comprehensive ToP assessment using the following equation

$$\mathrm{ToP}\left(\mathrm{S}\right)=w_1\times \mathrm{To}{\mathrm{P}}_{\mathrm{offline}}\left(\mathrm{S}\right)+w_2\times \mathrm{To}{\mathrm{P}}_{\mathrm{online}}\left(\mathrm{S}\right)$$

(17)

where $w_1$ and $w_2$ , respectively, represent the weights of online and offline ToP assessments accounting the percentage of total, and hence, $w_1+w_2=1,w_1\ge 0,\mathrm{and}{w}_2\ge 0$ .

Figure 9.

Schematic diagram of comprehensive ToP assessment.

Case study and results

Overview of aircraft hydraulic pump prognostics

A hydraulic pump used in the aircraft hydraulic system is studied for verification purposes. The ORF is selected as the indicator of wear degradation, and the failure criterion is 2.8 L/min. ¹³ The ORF has been monitored and recorded every 5 h. There were 228 points recorded before the hydraulic pump failure occurred. That is to say, the $t_{\mathrm{EoL}}$ of the hydraulic pump is 1135 h. As shown in Figure 10, the envelope of time sequence for the ORF change is determined by integrating the original ORF time sequence (assuming that the ORF approximately obeys a normal distribution at any time). The standard deviation of the hydraulic pump service life is approximately 17.5 h.

Figure 10.

RUL prediction of the hydraulic pump.

Assuming that the ORF change can reflect the failure or health state of the hydraulic pump by making use of a high-precision data acquisition subsystem. Without considering the influence of data acquisition and data processing subsystem, the ToP of prognostics subsystem can be regarded as the ToP of total PHM system.

In this study, the BP neural network prognostics algorithm is utilized for RUL prediction. The ORF of the hydraulic pump has uncertain characteristics due to the uncertainties of load conditions. The Monte Carlo method is adopted to propagate the uncertainties in the process of prognostics. The RUL prediction results of the hydraulic pump are shown in Figure 10.

Offline ToP assessment results

First, the quality of prognostics, the timeliness of prognostics, and other performance metrics should be provided by end users according to the actual situation of the object and application. For this case, the $\mathrm{C}{\mathrm{I}}_{\mathrm{A}}$ of the actual service life of the hydraulic pump is about 105 h, which is within the acceptable range for end users. Assuming that the maximum tolerance interval is $\mathrm{C}{\mathrm{I}}_{\max }=2\mathrm{C}{\mathrm{I}}_{\mathrm{A}}=210\mathrm{h}$ in the actual cases. Because the hydraulic pump has tremendous influence on the safety of the entire aviation equipment, the acceptable zone should be ${\alpha }^{-}>{\alpha }^{+}$ . Let

$$\begin{array}{c}{\alpha }_i^{-}=0.8\times {\mathrm{CI}}_A^i\hfill \\ {\alpha }_i^{+}=0.5\times {\mathrm{CI}}_A^i\hfill \end{array}$$

where ${\mathrm{CI}}_A^i=6\times \mathrm{std}(A(t_i))$ , $i=1,2,\dots ,10$ , and $\mathrm{std}(A(t_i))$ is the standard deviation of the actual life of the hydraulic pump.

When RA ≥ 0.5 and RP ≥ 0.5, the predicted value is regarded in the acceptable zone. According to the above parameter sets, the base of the offline prognostics performance evaluation can be obtained, and the RUL at different moments can be calculated as shown in Figure 11.

Figure 11.

Offline ToP assessment of the hydraulic pump.

As shown in Figure 11, the start of PH is at the 171th monitoring point. Considering the largest variance of the actual life of the hydraulic pump, the end of PH is at the 224th monitoring point. That is PH = 53 under the condition that the acceptable degree of inherent risk is 95%. Assuming that the RT is 100 h, then CT can be neglected relative to RT. When the prognostics timeliness is satisfied by PH > RT + CT, it is assumed that IF_PH = 1. In the next step, the prognostics quality and the comprehensive prognostics quality can be evaluated at different moments. The assessment results are presented in Table 1. According to formulae (13) and (14), the weights of comprehensive prognostics quality can be obtained as follows

$${\mathrm{W}}_{\mathrm{CRA}}=0.477,{\mathrm{W}}_{\mathrm{CRP}}=0.523$$

Table 1.

Quality of prognostics results.

Point	176	180	184	188	192
RA	0.980	0.999	0.982	0.985	0.990
RP	0.690	0.796	0.869	0.915	0.930
Point	196	200	204	CRA/CRP
RA	1	0.998	1	0.993
RP	0.952	0.949	0.963	0.904

RA: relative accuracy; RP: relative precision; CRA: comprehensive relative accuracy; CRP: comprehensive relative precision.

Then, the offline ToP can be calculated $\mathrm{To}{\mathrm{P}}_{\mathrm{offline}}=0.946$ by employing formula (12).

Online and comprehensive ToP assessment results

The ToP is finally determined by comparing the predicted ORF values with the actual monitoring ORF values. Due to the linear degradation trend and reliability characteristic of the hydraulic pump, the parameters involved in the online ToP assessment can be set in accordance with the offline assessment. Consequently, MEPH = 20 and nine assessment values of the prognostics quality are obtained, which are close to $t_{\mathrm{EoUP}}$ . Furthermore, the weights of CRA and CRP are confirmed by formulae (13) and (14) and used to assess the ToP. The ToP assessment results are shown in Figure 12 and Table 2.

Figure 12.

Online ToP assessment of the hydraulic pump.

Table 2.

Online ToP assessment results.

Point	171	172	173	174	175
RP	0.993	0.990	0.995	0.997	0.998
RA	0.938	0.946	0.961	0.954	0.934
w RA	0.486	0.489	0.491	0.489	0.483
w RP	0.514	0.511	0.509	0.511	0.517
ToP	0.965	0.968	0.978	0.975	0.965
Point	176	177	178	179	–
RP	0.996	1	0.994	1	–
RA	0.940	0.954	0.921	0.943	–
w RA	0.486	0.488	0.481	0.485	–
w RP	0.514	0.512	0.519	0.515	–
ToP	0.967	0.976	0.956	0.971	0.968

ToP: trustworthiness of prognostics; RA: relative accuracy; RP: relative precision.

By combining the online and offline ToP assessment results with formula (16), the comprehensive ToP assessment can be conducted. As the same system is adopted in both offline and online assessment processes, the weight can be set as $w_1=w_2=0.5$ . Then, by solving formula (17), the comprehensive ToP is calculated as 0.957.

Conclusion

Prognostics performance evaluation is a necessary engineering activity in industry for applying prognostics techniques. This article has attempted to enable a comprehensive understanding of the performance of PHM system. The case study results show that proposed ToP concept and assessment methods can contribute to the measurement of the performance of the total PHM system. These also establish a necessary foundation for PHM design, prognostics methods selection, cost–benefit analysis, V&V of PHM system, and maintenance decision. Prognostics performance evaluation and ToP assessment methods will become the enabler for the future development of high-performance and low-cost PHM systems.