Age-dependent production and replacement strategies in failure-prone manufacturing systems

Abstract

A failure-prone manufacturing system that consists of one machine producing one type of product is studied. The random phenomena examined are machine breakdowns and repairs. We assume that the machine undergoes a progressive deterioration while in operation and that the machine failure rate is a function of its age. The aging of the machine (the dynamics of the machine age) is assumed to be an increasing function of its production rate. Corrective maintenance activities are imperfect and restore the age of the machine to as-bad-as-old conditions. When a failure occurs, the machine can be repaired, and during production, the machine can be replaced, depending on its age. When the replacement action is selected, the machine is replaced by a new and identical one. The decision variables are the production rate and the replacement policy. The objective of this article is to address the simultaneous production and replacement policy optimization problem in the context of manufacturing with deterioration and imperfect repairs satisfying the customer demand and minimizing the total cost, which includes costs associated with inventory, backlog, production, repair and replacement, over an infinite planning horizon. We thoroughly explore the impact of the machine aging on the production and replacement policies. Particular attention is paid to the verification of underlying mathematical results that guarantee the existence of optimal solutions and the convergence of numerical methods. Due to imperfect repairs, the dynamics of the system is affected by the system history, and semi-Markov processes have to be used for modeling. Optimality conditions in the form of the Hamilton–Jacobi–Bellman equations are developed, and numerical methods are used to obtain the optimal control policies (production (rate) and replacement policies). A numerical example is given to illustrate the proposed approach, and an extensive sensitivity analysis is presented to confirm the structure of the obtained control policies.

Introduction

In the manufacturing environment, the availability of the machine often decreases due to its age and imperfect maintenance activities. In general, corrective or preventive maintenance brings the state of the machine to a level which is not new, and it may not be able to meet the demand rate for the commodity produced. For this reason, the machine has to be replaced. We consider a machine that is subject to random breakdowns and repairs. It undergoes deterioration while in operation, and the failure rate increases with its age. The aging of the machine is an increasing function of its production rate. The corrective maintenance activities performed are imperfect and restore the machine to as-bad-as-old (ABAO) conditions. Replacement activities for their part renew the machine, which is similar to restoring the machine to as-good-as-new conditions (resetting its age to zero).

The first objective of this article is to simultaneously determine production and replacement policies in a manufacturing environment under deterioration and imperfect repairs. We enhance existing mathematical models by including the production cost in the objective function. Given that the dynamics of the machine aging process depends on the production rate, penalizing the latter helps to amplify the impact of aging and push the system (optimal control policies) toward an appropriate solution. The proposed model appears to be better at addressing industrial reality and is yet to be used in the literature in analyzing age-dependent production and replacement strategies. The obtained solution provides the simultaneous optimal control of production and replacement of the machine (assuming that one replacement is performed). The decision variables are the production rate and the replacement policy. The dynamics of the machine is described by a semi-Markov decision model due to the machine’s deterioration and imperfect repairs (ABAO). The optimal control policies are determined in order to satisfy a deterministic customer demand and minimize inventory, backlog, production, repair and replacement costs over an infinite planning horizon.

The second objective is to consider machine aging as a unique factor affecting the deterioration of the machine in order to get a better understanding of the phenomena specifically relating to machine aging without interference from other deterioration levels. Finally, the third objective is to provide a rigorous formulation of the underlying mathematical results and verification of all conditions in order to guarantee the existence of optimal solutions and convergence of numerical methods used to find these solutions.

The rest of this article is organized as follows: We provide the overview of the existing body of works in the “Literature review” section. In the “Assumptions and problem statement” section, we describe the assumptions regarding the model and formulate the optimal control problem. The “Properties of the value function and optimality conditions” section addresses optimality conditions in the form of Hamilton–Jacobi–Bellman (HJB) equations. A numerical example is presented in the “Numerical example” section, followed in the “Sensitivity analysis” section by the sensitivity analysis illustrating the robustness and effectiveness of the obtained control policies. Discussions of the results are presented in “Discussions” section, and finally, this article is summarized in the “Conclusion” section.

Literature review

To describe the behavior of a repairable system subject to failures, various types of repairs are used in conventional models: the minimal repair, the perfect repair and the imperfect repair. The first one brings the system to its functioning condition just prior to a failure (e.g. ABAO); the second one is as-good-as-new, while the third one can be represented by its so-called virtual age, which is smaller than the real age. Pham and Wang ¹ suggest the faulty part being only partially repaired, human errors such as further damage during maintenance, replacement with faulty parts and so on as reasons behind an imperfect repair. Chiu et al. ² address system failures followed by imperfect rework using economic manufacturing quantity model with Poisson’s failure and abort or resume control policies. Many authors have studied the optimization problems of maintenance, repair and replacement in the context of a progressive deterioration but without addressing production planning. Progressive deterioration is usually characterized either by the repair time increasing with the number of failures or by age accumulation when the machine is in operational mode. Phelps ³ developed a semi-Markov decision model in which a minimal repair is performed on a system subject to random failures. He suggested three optimal replacement policies: in the first policy, the system is replaced at a fixed time T; in the second, the system is replaced after a fixed number of failures N; and in the third, the replacement takes place at the first failure after a fixed time T Kijima et al. ⁴ studied a problem of periodic replacement with a general repair: the system is replaced only at a regular time interval kT (k = 0, 1, …) and is repaired following a failure. They assumed that the costs of repair and replacement are constant. This general repair transfers the system state to a certain “better” state. A stochastic model describing this situation was proposed to find the optimal replacement period. Makis and Jardine ⁵ described a system with deterioration subject to random failures modeled as a semi-Markov process and demonstrated under appropriate conditions that a stationary optimal replacement policy exists. They assumed that the replacement cost is constant and that the repair cost depends on the number of failures and the real machine age. Love et al.^6,7 proposed a discrete semi-Markov model: they determined the optimal policies of a machine subject to breakdowns and which can be replaced or can undergo an imperfect repair. The virtual machine age has been introduced in the decision process, and it has been shown that the optimal repair or replacement policy is of threshold type. Zhang ⁸ developed a bivariate replacement policy (T, N) for a repairable system. The replacement is performed at the first failure either after the cumulative age reaches T or the number of failures reaches N. It was assumed that the system after repair is not refurbished and that the optimal solution can be obtained analytically or numerically under certain conditions.

Many researchers have studied the production planning problem for unreliable manufacturing systems. Based on the formalism of Rishel, ⁹ Olsder and Suri ¹⁰ determined the optimum conditions of a planning problem in the production of a manufacturing system subject to random breakdowns and repairs, with its dynamics described by homogeneous Markov processes. The optimum conditions are described by the HJB equations. The pioneering work of Kimemia and Gershwin ¹¹ similarly showed that the obtained control policy for this problem is of threshold type (Hedging point policy). This allows the optimal production rate to be found in order to meet a constant demand rate and reduce the total cost, which is the sum of inventory and shortage costs. Boukas and Haurie ¹² determined a policy that combines the production and preventive maintenance for a manufacturing system consisting of two machines. They consider that the probability of failures of the machines increases with the age. Since the repair or preventive maintenance activities restore the cumulative age to zero, the dynamics of the system (machines) is modeled in Boukas and Haurie ¹² by a nonhomogeneous Markov process. The set of dynamic programming equations was solved numerically based on Kushner and Dupuis’s ¹³ approach, and the obtained control policy is of modified threshold type. Yan^14,15 discussed a hierarchical stochastic production planning problem of flexible manufacturing systems (FMSs). Imposing some conditions on uncertain demand, the author formulated a stochastic nonlinear programming problem and proposed the algorithms for its solution. By applying some approximations, a production plan with the lowest possible cost is obtained.

In the context of imperfect repairs, the optimal production and repair or replacement joint policy is of great importance for practitioners. It helps to better manage the manufacturing system performance and to intervene with the acquisition of a new machine in a timely fashion. This situation was investigated in Nodem et al., ¹⁶ where the authors proposed a hierarchical decision-making approach to first determine the hybrid repair and replacement policy (T, N). Second, the optimal production rate is determined. The age and the number of failures are combined to make the decision about the repair or replacement of the machine. The repair activities are imperfect due to a number of failures affecting the system behavior, and the replacement activities restore the age of the machine to initial conditions (AGAN). With respect to the modeling, the decision to repair or replace the machine is made when the failure occurs. In Gomez et al., ¹⁷ the simultaneous production, repair or overhaul (equivalent to replacement) and preventive maintenance control policy are obtained under the effect of deteriorations resulting in product quality degradation: The authors combined the impact of two factors, namely, the wear of the machine and human interventions. When the machine is in operational mode, three types of actions can be taken: the machine produces, is sent to overhaul or is sent for preventive maintenance. Upon failure, the machine has to be repaired. A semi-Markov process is used in both,^16,17 due to the imperfect maintenance activities of the machine that depend on the history of breakdowns and repairs. Nodem et al. ¹⁶ and Gomez et al. ¹⁷ developed the optimality conditions in the form of HJB equations, which allow the resolution of the problem of stochastic optimal control based on stochastic dynamic programming and numerical methods, in order to optimize the total cost (e.g. including the inventory, backlog, repair or replacement, preventive maintenance and defectives costs) and satisfy a demand assumed to be deterministic and known. However, due to the mixture of various deterioration mechanisms involved in the proposed models, it remained unclear which mechanism was the most responsible for the phenomena observed.

As mentioned in the introduction, a model that differs from the existing ones is developed in this article. We explore the impact of deterioration mechanism related to the machine aging only, in order to eliminate interference from other sources of deterioration, as in Nodem et al. ¹⁶ and Gomez et al., ¹⁷ and obtain the insight to age-induced deterioration phenomena. We also incorporate the production cost in the objective function. The rationale behind changing the cost function in such a way, given that the dynamics of the machine aging depends on the parts produced, penalizing the production rate with production cost will amplify the phenomenon of aging and push the system (optimal control) to enhance the obtained solution in manufacturing environment. The optimality conditions are developed in the form of HJB equations using the optimal control theory based on stochastic dynamic programming. The practical implication of the proposed model is verified through numerical simulations.

Assumptions and problem statement

This section defines the assumptions used throughout this article, as well as the optimal control problem statement.

Assumptions

The following assumptions are considered in this article:

1. The raw materials are always available and unlimited;

2. The customer demand is known and described by a constant rate over time;

3. The maximum production rate of each machine is known;

4. The repair or replacement costs of the manufacturing machine are assumed constant (($/repair); ($/replacement));

5. The machine deteriorates in operational mode, in which the machine age is continuously growing;

6. The failure rate of the machine increases with its age;

7. The corrective maintenance activities are imperfect and associated with a minimal repair that restores the machine to ABAO conditions;

8. The machine is replaced by a new identical one;

9. The production is more penalized when the machine increases its production rate.

Problem formulation

The manufacturing system under consideration consists of one machine producing one type of product. The machine is unreliable and is subject to random phenomena, such as breakdowns and repairs. The machine deteriorates while in operation, and its failure rate increases with its age. In failure, the machine undergoes a minimal repair, and while it is in operation, a decision can be made to replace the machine with a new and identical one due to its deterioration. A service inventory is built at the end of the production process, which is used to satisfy a constant demand rate. This situation is illustrated in Figure 1.

Figure 1.

Structure of the production system.

The system behavior can be mathematically modeled by a stochastic control system in continuous time (not in multi-period time, as is normally used in the literature) characterized by a hybrid state, which is comprised of two continuous state variables (inventory level $x(t)$ and machine age $a(t)$ ) and one discrete state variable (mode of the machine $\xi (t)$ ) at time t.

The dynamics of the stock level is described by a one-dimensional ordinary differential equation

$$\stackrel{·}{x}\left(t\right)=u\left(t\right)-d\mathrm{with}t\in {\mathbb{R}}_{+}\mathrm{and}x\left(0\right)=x^0$$

(1)

where $x^0$ is the given initial inventory or backlog level. When $x(t)⩾0$ , the system has an inventory and a backlog, otherwise.

The cumulative age of the machine at time t is the solution of the following differential equation

$$\stackrel{·}{a}\left(t\right)=f\left(u\left(t\right)\right)$$

where $f(.)$ is a positive real-valued function.

In the case considered herein, the machine age $a(t)$ is the number of produced parts, and the aging of the machine $\stackrel{·}{a}(t)$ will always be an increasing function of the machine production rate. By referring to the literature (see Boukas and Haurie ¹² and Kenné and Boukas ¹⁸ ), we can use the linear model that describes the relationship between the aging of the machine $\stackrel{·}{a}(t)$ and its production rate $u(t)$ . That is

$$\begin{array}{cc}\multicolumn{1}{c}{\stackrel{·}{a}\left(t\right)}& =\mathrm{ku}\left(t\right)\mathrm{with}t\in {\mathbb{R}}_{+}\mathrm{and}\hfill \\ \multicolumn{1}{c}{a\left(0\right)}& =a^0,a\left(T^{+}\right)=a\left(T^{-}\right)\mathrm{and}a\left(T\right)=0\hfill \end{array}$$

(2)

where k is a given positive constant and $a^0$ , $T^{+}$ , $T^{-}$ and T stand for the initial age, the last times of repair, operation and replacement of the machine, respectively. These values $(T^{+},T^{-},T)$ imply that the repair is ABAO, and the replacement is as-good-as-new.

The production rate $u(t)$ must satisfy the capacity constraint

$$0⩽u(.)⩽u_{\max }$$

(3)

where $u_{\max }$ is the maximum production rate of the manufacturing machine.

The machine’s mode can be classified as operational, denoted by $\xi (t)=1$ ; under repair, denoted by $\xi (t)=2$ ; and under replacement, denoted by $\xi (t)=3$ . The mode of the machine at time t is given by the random process $\{\xi (t)⩾0\}$ with $\xi (t)\in B=\{1,2,3\}$ such that

$$\xi \left(t\right)=\{\begin{array}{cc}1\hfill & \mathrm{operational}\hfill \\ \multicolumn{1}{c}{2}& \mathrm{under\; failure}\hfill \\ \multicolumn{1}{c}{3}& \mathrm{under\; replacement}\hfill \end{array}$$

(4)

Given that the machine deteriorates with age and the fact that it is not new after repair activities (ABAO), its dynamics is modeled as a semi-Markov process in continuous time discrete state over an infinite horizon $\{\xi (t)⩾0\}$ . The transition diagram of such a process is illustrated in Figure 2. We introduce a control variable $w(t)\in \{0,1\}$ initially as a binary variable; it is set to a value 1 if the replacement of the machine is performed and to 0 otherwise. The transition rate $q_{13}(.)$ is defined as a linear function of $w(t)$ : $q_{13}(.)=q.w(t)$ , where q is a given constant.

Figure 2.

State transition diagram of the considered stochastic process.

The failure rate $q_{12}(.)$ is an increasing function of a machine’s age $a(t)$ and is given by

$$q_{12}\left(a\left(t\right)\right)=A_0+A_1^{\infty }(1-e^{-(A_{2}a{(t)}^3)})$$

(5)

where the parameters $A_0$ , $A_1^{\infty }$ and $A_2$ are given constants. The expression given by equation (5) describes the impact of a machine age on its dynamics, as in Kenné and Nkeungoue. ¹⁹ The inverse of the transition rate $q_{12}(a)$ represents the mean time to failure (MTTF) as a function of the age denoted by $\mathrm{MTTF}(a)$ . We use the notation $\mathrm{Ind}\{\mathrm{\Theta }(.)\}$ for the indicator function of a condition $\mathrm{\Theta }(.)$ defined as follows

$$\mathrm{Ind}\left\{\mathrm{\Theta }(.)\right\}=\{\begin{array}{c}1,\hfill \\ \multicolumn{1}{c}{0,}\end{array}\begin{array}{c}\mathrm{if}\Theta (.)\mathrm{is}\mathrm{true}\hfill \\ \multicolumn{1}{c}{\mathrm{otherwise}}\end{array}$$

The machine’s mode switches from operational to replacement with a transition rate denoted by $q_{13}(.)$ , the inverse of $q_{13}(.)$ represents the expected delay between the preparation of the order of the new machine, its installation kit and its arrival on the site. In order to allow this delay to vary continuously, we reintroduce a control variable $w(t)$ allowing it to take its values within the domain $]0,1]$ . We also assume that the decision regarding the machine replacement can be taken only in operational mode. Thus, the delay corresponding to the inverse of $q_{13}(.)=q.w(t)$ is the transition time between the decision to perform the replacement and the effective switch from the operational mode to the replacement mode. The replacement is now allowed for all possible values of $w(t)\mathrm{within}]0,1]$ . When the value of $w(t)$ is close to zero (e.g. 10⁻⁵), the delay before replacement is very large (but finite), thus describing the situation when replacements do not actually take place; as for machine repairs, they can occur at any failure instant. The transition rates $q_{21}$ to repair the machine in mode 2 and $q_{31}$ to replace the machine in mode 3 are assumed to be known constants (their inverses represent the mean time to repair (MTTR) and the mean time to replace (MTTRP), respectively). Other transition rates of the manufacturing system are equal to 0.

In the case of minimal repair, the repair rate is usually considered to be constant. In manufacturing systems, many repairs (such as ABAO) can be considered as minimal repairs. Such a repair brings the age of the machine in the state which is basically the same as it was just before the failure occurred. In other words, a minimal repair means that the age of the machine is not affected by failures: the machine age after each minimal repair is restored, and the failure rate gets the same value as it was before the failure. Thus, after minimal repair, both failure rate and repair rate are restored to their values before the last failure; $q_{12}^{\mathrm{new}}=q_{12}^{\mathrm{old}}$ , and $q_{21}^{\mathrm{new}}=q_{21}^{\mathrm{old}}$ . The ratio of repair duration ${\tau }_r$ and the MTTF $1/q_{12}$ are then kept constant in the case of minimal repair and then

$${\tau }_r^{\mathrm{new}}={\tau }_r^{\mathrm{old}}.\frac{q_{12}^{\mathrm{old}}}{q_{12}^{\mathrm{new}}}={\tau }_r^{\mathrm{old}}.\frac{q_{12}^{\mathrm{old}}}{q_{12}^{\mathrm{old}}}\Rightarrow {\tau }_r^{\mathrm{new}}={\tau }_r^{\mathrm{old}}$$

As a result, the repair duration ${\tau }_r$ is represented by the constant repair rate $q_{21}$ . More detailed description of the minimal repair models can be found in Zio. ²⁰ For example, once the machine is repaired after the first failure, the age continues to accumulate when the machine begins to operate again, and the failure rate also continues to increase with the age. When the next failure occurs, and a minimal repair is executed, the failure rate obtains the same value as before the last failure and according to the equality above $({\tau }_r^{\mathrm{new}}={\tau }_r^{\mathrm{old}})$ , the repair duration does not change; it is the same as before the failure.

Figure 3 shows the jump times occurring within our model in different modes $e_i$ . It should be recalled that the mode of the manufacturing system at time t is given by the finite-state semi-Markov process $\xi (t)\in \mathrm{B}=\{1,2,3\}$ and can be characterized by the matrix $Q(w)=[q_{\alpha \beta }(.)]$ . Its entries $q_{\alpha \beta }$ are real numbers and depend on the decision variable $w(.)$ , with $\alpha ,\beta \in \mathrm{B}$ . If $\alpha \ne \beta$ , we have

$$q_{\alpha \beta }(x,a,u,w)⩾0$$

(6)

$$\sum _{\beta }{q}_{\alpha \beta }(x,a,u,w)=0$$

(7)

and if $\alpha =\beta$ , we have

$$q_{\alpha \alpha }(x,a,u,w)=-\sum _{\alpha \ne \beta }{q}_{\alpha \beta }(x,a,u,w)$$

(8)

Figure 3.

Jump times of different modes $e_i$ $(i=1,2\mathrm{and}3)$ .

The transition probabilities of the manufacturing system from mode $\alpha$ to mode $\beta$ after time t are given by

$$\begin{array}{c}\multicolumn{1}{c}{P[\xi (t+\delta t)=\beta |\xi \left(t\right)=\alpha ,a\left(t\right)=a,u\left(t\right)=u,w\left(t\right)=w]}\\ =q_{\alpha \beta }(x,a,u,w)\delta t+o(x,a,\delta t)\hfill \end{array}$$

(9)

$$\begin{array}{c}P[\xi (t+\delta t)=\alpha |\xi \left(t\right)=\alpha ,a\left(t\right)=a,u\left(t\right)=u,w\left(t\right)=w]\hfill \\ =1+q_{\alpha \alpha }(x,a,u,w)\delta t+o(x,a,\delta t)\hfill \end{array}$$

(10)

where $o(x,a,\delta t)$ is a quantity such that

$$\underset{\delta t\to 0}{\lim }\frac{o\left(x,a,\delta t\right)}{\delta t}=0\mathrm{for\; all}\alpha ,\beta \in \mathrm{B}$$

The corresponding $(3\times 3)$ transition matrix of the semi-Markov chain $\xi (t)$ for the considered system is given by

$$Q\left(w\right)=\left[\begin{array}{ccc}{q}_{11}\hfill & q_{12}\left(a\left(t\right)\right)\hfill & q_{13}(.)\hfill \\ \multicolumn{1}{c}{q_{21}}& q_{22}\hfill & 0\hfill \\ \multicolumn{1}{c}{q_{31}}& 0\hfill & q_{33}\hfill \end{array}\right]$$

(11)

To address the feasibility of the manufacturing system, we introduce as usual, the limiting probabilities ${\pi }_i$ of mode i, $i\in \{1,2,3\}$ that are known to be steady-state solutions of the forward Kolmogorov equations $\stackrel{·}{\pi }(t)=\pi (t)Q(t)$ . When $t\to \infty$ , $\stackrel{·}{\pi }(t)=0$ , as in Ross. ²¹ Therefore, we have

$$\pi (.)Q(.)=0$$

(12)

with the normalizing condition

$$\sum _{i=1}^3{\pi }_i=1$$

(13)

where $\pi (.)=({\pi }_1,{\pi }_2,{\pi }_3)$ and $Q(.)$ is a $(3\times 3)$ transition rates matrix given by equation (11). The manufacturing system is considered feasible if

$${\pi }_1{u}_{\max }⩾d$$

(14)

Let $G(.)$ be the cost rate defined as follows

$$G(\alpha ,x,a,u,w)=c^{+}{x}^{+}+c^{-}{x}^{-}+c^{\alpha }+c\left(u\right)$$

(15)

The constants $c^{+}$ and $c^{-}$ are costs incurred per unit of produced parts and per unit time and are used to penalize parts inventory and backlog, respectively, with $x^{+}=max(0,x),x^{-}=max(0,-x)$ and $c^{\alpha }$ being a constant defined as follows

$$c^{\alpha }=c_r·\mathrm{Ind}\{\xi \left(t\right)=2\}+c_0·q_{31}·\mathrm{Ind}\{\xi \left(t\right)=3\}$$

where $c_r$ is the cost rate (positive constant) for repair activity on the machine and $c_0.q_{31}$ is the replacement cost (positive constant) that depends on the duration of the replacement activity of the machine. Let $c(u)$ be a linear production cost function given by $c(u)=c_{u}u$ . The constant $c_u$ is the cost incurred per unit of produced parts and used here to penalize the production rate of the machine.

The objective of this research is to find the two decision variables, namely, the production rate $u(.)$ and the replacement variable $w(.)$ , that would minimize the expected discounted cost $J(.)$ given by

$$\begin{array}{c}J(\alpha ,x,a,u,w)\hfill \\ =E\{\underset{0}{\overset{\infty }{\int }}{e}^{-\rho t}G(.)\mathrm{dt}|\xi \left(0\right)=\alpha ,x\left(0\right)=x,a\left(0\right)=a\}\forall (u(.),w(.))\in \mathrm{\Gamma }\left(\alpha \right)\hfill \end{array}$$

(16)

where $\rho$ is the discount rate and $\alpha ,x\mathrm{and}a$ are the initial values of the state variables. The set of admissible decisions $\mathrm{\Gamma }(\alpha )$ that define the feasible plan $(u(a,.),w(a,.))$ depends on the stochastic process $\xi (t)$ and is given by

$$\mathrm{\Gamma }\left(\alpha \right)=\left\{\begin{array}{c}(u(a,.),w(a,.))\in {\mathbb{R}}^2,0⩽u(a,.)⩽\hfill \\ \multicolumn{1}{c}{u_{\max }\mathrm{Ind}\{\xi \left(t\right)=1)\},0<w(a,.)⩽1}\end{array}\right\}$$

(17)

The value function of such a problem is given by

$$\begin{array}{c}\nu (\alpha ,x,a)={\inf }_{(u(a,.),w(a,.))\in \mathrm{\Gamma }(\alpha )}J(\alpha ,x,a,u,w)\forall \alpha \in \mathrm{B},x\in \mathbb{R},a\in {\mathbb{R}}_{+}\hfill \end{array}$$

(18)

The properties of the value function leading to HJB equations of the stochastic optimal control problem are presented in the next section.

Properties of the value function and optimality conditions

In this section, we develop the optimality conditions using the optimal control theory based on stochastic dynamic programming. The optimal control policy $(u^{*}(.),w^{*}(.))$ denotes a minimizer over $\mathrm{\Gamma }(\alpha )$ on the right-hand side of equation (16). This policy shows that the value function $\nu (.)$ given by equation (18) satisfies the set of partial differential equations known as the HJB equations. In Appendix 2, we develop the derivation of the HJB equations. However, the differentiability and viscosity properties of the value function can be found in Kenné et al. ²² From equation (15), $G(\alpha ,a,x,u,w)$ can be written as follows: $G(\alpha ,z,U)=h(x)+c^{\alpha }+c(u)$ , where $z=(x,a)$ and $U=(u,w)$ . We extend the domain of the decision variable w serving to describe the controlled replacement to a continuous domain $]0,1]$ , in order to use the notions of continuity and convexity. However, this will not affect the optimization result since for the type of problem we are solving, the optimum (over w) always occurs on the boundary of $]0,1]$ . We should recall that in our case, the replacement is allowed for all possible values of the variable w. Below, we will use the following assumption, definition and lemmas.

Assumption A1

$Q(w)$ is a continuous function in w.

Lemma 1

• 1. $G(\alpha ,z,U)$ is jointly convex if, for each $\alpha \in B$ and for every $z_1,z_2,U_1$ and $U_2$ and $\delta \in [0,1]$ , for example

$$G\left(\alpha ,\delta z_1+\left(1-\delta \right)z_2,\delta U_1+\left(1-\delta \right)U_2\right)⩽\delta G\left(\alpha ,z_1,U_1\right)+\left(1-\delta \right)G\left(\alpha ,z_2,U_2\right)$$

Then, $\nu (\alpha ,z)$ is also convex if, for each $\alpha \in B$ and for every $z_1$ and $z_2$ and $\delta \in [0,1]$ , for example

$$v(\alpha ,\delta z_1+(1-\delta )z_2)⩽\delta v(\alpha ,z_1)+(1-\delta )v(\alpha ,z_2)$$

Moreover, $G(\alpha ,z,U)$ is strictly jointly convex in z if the inequality above of G holding as an equality for some $0<\delta <1$ implies $z_1=z_2$ and $U_1\ne U_2$ . Then, $\nu (\alpha ,z)$ is also strictly convex in z if the inequality above of $\nu$ is strict whenever $z_1\ne z_2$ and $0<\delta <1$ .

• 2. For some constants $C_g$ and $K_g>0$ , if $G(\alpha ,z,U)$ is locally Lipschitz in z, for example

$$\left|G\left(\alpha ,z_1,U\right)-G\left(\alpha ,z_2,U\right)\right|⩽C_g\left(1+{\left|z_1\right|}^{K_g}+{\left|z_2\right|}^{K_g}\right)\left|z_1-z_2\right|$$

Then, $\nu (\alpha ,z)$ is also locally Lipschitz in z, for example

$$|\nu (\alpha ,z_1)-\nu (\alpha ,z_2)|⩽C_g(1+{\left|z_1\right|}^{K_g}+{\left|z_2\right|}^{K_g})|z_1-z_2|$$

Proof

Any function which is convex and locally Lipschitz is continuously differentiable. To prove this property for the value function $\nu (\alpha ,z)$ with respect to z and for each $\alpha \in B$ , we can refer the reader to the book by Sethi et al., ²³ and the proof is similar to their proof of Lemma E.1.

The value function $\nu (\alpha ,z)$ is continuously differentiable in z if and only if $D^{+}v(\alpha ,z)$ and $D^{-}v(\alpha ,z)$ are both singletons. In this case

$$D^{+}v(\alpha ,z)=D^{-}v(\alpha ,z)=\left\{v_z(\alpha ,z)\right\}$$

where $z=(x,a)$ . The HJB equations corresponding to the optimal control problem are written as follows

$$\rho \nu (\alpha ,x,a)=\underset{(u,w)\in \mathrm{\Gamma }\left(\alpha \right)}{min}[G(\alpha ,x,a,u,w)+(u-d)\frac{\partial }{\partial x}\nu (\alpha ,x,a)+\mathrm{ku}\frac{\partial }{\partial a}\nu (\alpha ,x,a)+\sum _{\beta }{q}_{\alpha \beta }(.)\nu (\beta ,x,{\phi }_a(\xi ,a))]$$

(19)

where ${\phi }_a(\xi ,a)$ is the function that restores the age of the machine to as-good-as-new after the machine is replaced and to ABAO after imperfect repair activities. Therefore, at a jump time $\tau$ for the process $\xi$ , one can write this function as follows

$${\phi }_a(\xi ,a)=\{\begin{array}{c}a\left({\tau }^{-}\right),\hfill \\ \multicolumn{1}{c}{0,}\\ \multicolumn{1}{c}{a\left({\tau }^{-}\right),}\end{array}\begin{array}{c}\mathrm{if}\xi \left({\tau }^{+}\right)=1\mathrm{and}\xi \left({\tau }^{-}\right)=2\hfill \\ \multicolumn{1}{c}{\mathrm{if}\xi \left({\tau }^{+}\right)=1\mathrm{and}\xi \left({\tau }^{-}\right)=3}\\ \multicolumn{1}{c}{\mathrm{otherwise}}\end{array}$$

(20)

where $\xi (t)=\alpha \in B$ , and $\partial \nu (\alpha ,x,a)/\partial x$ and $\partial \nu (\alpha ,x,a)/\partial a$ are the first-order partial derivatives of the value function $\nu (\alpha ,x,a)$ with respect to x and a, in that order. The following definition will be used to prove the lemma given below.

Definition 1

The superdifferentiability $D^{+}f(x)$ and subdifferentiability $D^{-}f(x)$ of any function $f(x)$ with respect to x are defined as follows

$$D^{+}f\left(x\right)=\{s\in {\mathbb{R}}^n:\underset{h\to 0}{lim}sup\frac{f(x+h)-f\left(x\right)-h·s}{\left|h\right|}⩽0\}$$

$$D^{-}f\left(x\right)=\{s\in {\mathbb{R}}^n:\underset{h\to 0}{lim}inf\frac{f(x+h)-f\left(x\right)-h·s}{\left|h\right|}⩾0\}$$

Lemma 2

The value function $\nu (.)$ defined in equation (18) is the unique viscosity solution to the HJB equation (19).

Proof

To prove Lemma 2, we use the concept given by Definition 1 to extend Theorem 2 presented by Yan and Zhang. ²⁴

When the value function is available, an optimal control policy can be obtained as a solution of the HJB equation (19). Since an analytical solution is impossible to obtain in general, it is a common practice to develop numerical methods for solving the HJB equations. Boukas and Haurie ¹² implemented a numerical method initially introduced by Kushner and Dupuis ¹³ to solve such a problem in the context of production planning. Kushner’s method used to solve the proposed optimality conditions is presented in Appendix 2. In the subsequent section, we provide a numerical example to illustrate the structure of the control policies.

Numerical example

The computational domain of the state variables is defined as $G_{\mathrm{xa}}=G_x^h\times G_a^h$ such that

$$\begin{array}{c}{G}_x^h=\{x:-10⩽x⩽30\}\hfill \\ G_a^h=\{a:0⩽a⩽100\}\hfill \end{array}$$

(21)

with $h_x=0.5$ and $h_a=2$ . The imperfect repair is characterized by a minimal repair which brings the machine to ABAO conditions. The machine failure rate is assumed to be age dependent. Using the model of the failure rate given by equation (5), we can obtain its trajectory according to the machine age, as illustrated in Figure 4 with values of $A_0={10}^{-4}$ , $A_1^{\infty }=0.01$ and $A_2=5\times {10}^{-6}$ .

Figure 4.

Age-dependent failure rate of the machine.

Other parameters needed in the numerical example are presented in Table 1. These parameters have been chosen such as to ensure that the system is feasible. Using the formula of equation (14), the feasibility of the system was verified and respected on the all computational domains of the age, where $q_{21}^{-1}$ , $q_{31}^{-1}$ and $q_{13}^{-1}$ are the MTTR, the MTTRP and the mean time between the machine successive replacements, respectively.

Table 1.

Parameters of the numerical example.

Parameter	$c^{+}$	$c^{-}$	$c_r$	$c_0$	$c_u$
Unit	$/product/time unit	$/missing product/time unit	$/time unit	$/replacement	$/product
Value	10	150	25	3000	100
Parameter	$u_{\max }$	d	$\rho$	k	w
Unit	Products/time unit	Products/time unit
Value	0.55	0.4	0.01	0.8	$]0,1]$
Parameter	$q_{21}^{-1}$	$q_{13}^{-1}$	$q_{31}^{-1}$
Value	20	45	14

The production rate and replacement policy are presented in Figures 5 and 6, respectively. We can see that the obtained optimal production control policies can be viewed as an extension of the hedging point policy presented in Akella and Kumar. ²⁵ In our case, however, we take into account the effect of the deterioration of the machine along the production phase, as well as the imperfect repairs. The optimal production control policy is presented in Figure 5 and defined by $Z(a)$ , which shows the numbers of parts to hold in inventory in order to hedge against failures of the machine with the age a. This control policy consists of the following rule:

• If the current stock level $x(t)$ is under the threshold level value $Z(a)$ , the production rate is to be set to its maximum value;

• If it is exactly at the threshold level, the production rate is to be set to the demand rate;

• If it is above the threshold level, the production rate is to be set to zero (produce nothing).

Figure 5.

Production policy of the manufacturing system.

Figure 6.

Replacement policy.

Thus, the production control policy satisfies

$$u^{\ast }\left(1,x,a\right)=\{\begin{array}{c}{u}_{max},\mathrm{if}x\left(t\right)<Z\left(a\right)\\ d,\mathrm{if}x\left(t\right)=Z\left(a\right)\\ 0,\mathrm{if}x\left(t\right)\mathrm{>}Z\left(a\right)\end{array}$$

(22)

where $Z(a)$ is the machine age–dependent function that gives the optimal production threshold in operational mode for each value of age of the machine, as illustrated in Figure 5.

We could not prevent the machine from getting older because only minimal repair (ABAO) is possible. Therefore, the machine deteriorated further, and after a certain age, could no longer satisfy the demand. Thus, a replacement policy defines when to replace the machine, taking into account the required stock level allowing the demand rate to be met when the machine is sent for replacement. In other words, the problem to be solved is determining the levels $(x^{*},a^{*})$ at which the machine should be replaced with a new identical one in order to optimize the manufacturing system.

Figure 6 shows that the optimal control for the replacement policy is a bang-bang solution. The reason for this is that the optimal control switches from the upper bound to the lower bound and is restricted somewhere between the two. Let $w(1,x,a)$ denote a switching function based on the stock level $x(t)$ and the age of the machine $a(t)$ , with the maximum value $w_{max}=1$ if the machine must be sent for a replacement activity, and the minimum value (lower value) $w_{\min }={10}^{-5}$ if the option to replace the machine is not recommended. We should recall that we only consider the decision process until the first replacement. Thus, the replacement policy can be written as follows

$$w^{*}(1,x,a)=\{\begin{array}{c}{w}_{\max },\hfill \\ \multicolumn{1}{c}{w_{\min },}\end{array}\begin{array}{c}\mathrm{if}(x\left(t\right),a\left(t\right))\in \mathrm{zone\; B}\hfill \\ \multicolumn{1}{c}{\mathrm{otherwise}}\end{array}$$

(23)

where state variables $(x(t),a(t))$ are the parameters of zone B. We can also see that Figure 6 shows the behavior of the decision variable responsible for replacement, based on the machine age and inventory level. To better understand the obtained policy, we divide the plane $(x,a)$ into two zones, A and B.

Zone A

The replacement policy does not recommend sending the machine for replacement, and the inventory level is low. The machine is still new in this zone and able to satisfy the demand with a rare fear of failure. Thus, it is not necessary to build a significant inventory level. This result is in a good agreement with the “zero-inventory condition” of Bielecki and Kumar, ²⁶ which asserts that a zero-inventory policy can be exactly optimal even in the presence of uncertainty. Hence, the decision variable $w(.)$ is set to its minimum value, $(w(.)={10}^{-5}\approx 0)$ .

Zone B

The machine is aging, and the failure rate increases. Here, the replacement cost is justified, and performing a replacement becomes necessary. However, before that is done, the manufacturing system must ensure a certain inventory level to satisfy the demand and to hedge against possible failures and a nonproductive replacement time. In this case, the decision variable $w(.)$ is set to its maximum value, $(w(.)=1)$ .

To better illustrate the optimal production and replacement policies, we can use their boundaries defined by $Z(a)$ in Figure 5 and $D(a)$ in Figure 6, respectively. The control policy defined by Figure 7 is known as the switching curve policy and is characterize by a vector z of two threshold parameters $z=(Z(a),D(a))$ , the optimal production and replacement switching levels. For every fixed a, when $Z(a)\equiv Z$ , and $D(a)\equiv D$ , the switching curve policy obtained becomes an extension of the so-called hedging point policy, and Z and D are the hedging points. Exploring the impact of aging, we are interested in the functions $Z(a)$ and $D(a)$ , which are the optimal threshold levels of production and replacement for each age a. In the context of deterioration, we note that the production threshold $Z(a)$ presented in Figure 7 increases progressively. Indeed, when the machine is in its early life period, it is still relatively new, and failures are rare. Thus, the inventory level should be maintained almost to a value closer to zero. When the machine is aging and the failure rate increases, building a certain inventory level becomes important, and this level increases with the age of the machine. We note that all values of $w(.)$ chosen between ( ${10}^{-5}\approx 0$ ) and 1 are permitted in order to extend the domain of $w(.)$ to a continuous domain and can apply the theory of convexity, but we know that the optimal solution is reached at the boundary value ${10}^{-5}$ or $1$ , confirming what was mentioned in the section “Properties of the value function and optimality conditions.”

Figure 7.

Production threshold and replacement trace.

In Figure 7, the intersection between the production threshold $Z(a)$ and the replacement trace $D(a)$ defines the feasible zone C, where the manufacturing system resides. This zone recommends optimally the age of the machine and the necessary stock level when the machine should be replaced. Thus, the replacement policy can be rewritten as follows

$$\begin{array}{c}{w}^{*}(1,x,a)=\hfill \\ \{\begin{array}{c}{w}_{\max },\hfill \\ \multicolumn{1}{c}{w_{\min },}\end{array}\begin{array}{c}\mathrm{if}(x\left(t\right),a\left(t\right))\in \mathrm{zone\; C}\hfill \\ \multicolumn{1}{c}{\mathrm{otherwise}}\end{array}\mathrm{with}\mathrm{C}=\mathrm{A}\cap \mathrm{B}\hfill \end{array}$$

(24)

where state variables $(x(t),a(t))$ are the parameters of zone $\mathrm{C}$ . We note that the point $\mathrm{s}$ highlighted on Figure 7 is a point where the production threshold $Z(a)$ intersects with the replacement trace $D(a)$ , and this is a recommended point for machine replacement. In the next section, we will confirm the obtained structure of the control policies through a sensitivity analysis.

Sensitivity analysis

The obtained control policies are validated through a sensitivity analysis by varying some parameters of the model. We analyze the behavior of the production threshold $Z(.)$ and the replacement trace $D(.)$ by varying the following parameters: the costs of backlog, surplus, production and replacement, the MTTR and the MTTRP. In the following cases of the variation of parameters, we can note that the symbols ${\mathrm{s}}_1$ , ${\mathrm{s}}_2$ and ${\mathrm{s}}_3$ are highlighted on the following figures, in order to illustrate the intersection points of the trace $D(.)$ and the production threshold $Z(.)$ when the replacement of the machine is recommended for the first time. The three intersection points ${\mathrm{s}}_1$ , ${\mathrm{s}}_2$ and ${\mathrm{s}}_3$ correspond, respectively, to the small, middle and high values of the parameter which will be varied in the sensitivity analysis.

Backlog cost variation

The results presented in Figure 8 for three different backlog cost values $c^{-}=100,150\mathrm{and}200$ show that when the backlog cost increases, the replacement of the machine is more highly recommended. That is because in this situation, the production threshold $Z(.)$ must be increased as $c^{-}$ increases in order to avoid shortages during periods of future failures of the machine, and to reach this level, the machine must produce more parts at its maximum production rate $u_{\max }$ . In other words, the machine takes more time to produce at $u_{\max }$ . Thus, the machine will deteriorate more rapidly, and for this reason, we replace it earlier: $a({\mathrm{s}}_3)<a({\mathrm{s}}_2)<a({\mathrm{s}}_1)$ . Consequently, this increases the feasible zone C, and we can observe that the variation of the backlog cost is directly linked to the size of the replacement zone.

Figure 8.

Variation of the backlog cost and its effect on the production and replacement policies.

Inventory cost variation

As we can see in Figure 9, the next step in the sensitivity analysis is to examine the inventory cost parameter. The values used are $c^{+}=5,10\mathrm{and}15$ . When the inventory cost $c^{+}$ increases, the replacement of the machine is less recommended. The reason for this is that the production threshold $Z(.)$ must be decreased as $c^{+}$ increases, and in this case, the machine must produce for a shorter period at its maximum production rate $u_{\max }$ . Thus, the machine will deteriorate more slowly, and for this reason, we replace it later: $a({\mathrm{s}}_1)<a({\mathrm{s}}_2)<a({\mathrm{s}}_3)$ ; this means the feasible zone C is decreased as $c^{+}$ increases.

Figure 9.

Variation of the inventory cost and its effect on the production and replacement policies.

We can conclude from Figures 8 and 9 that the effects of the backlog cost on the production and replacement policies are the inverse of what is seen with the inventory cost. We know that the machine deteriorates with age when it produces the parts. When the threshold increases (in the case of $c^{-}$ increases or $c^{+}$ decreases), the machine produces more parts at its maximum production rate $u_{\max }$ , and therefore, it is aging rapidly, and it is recommended that it be replaced, and for this reason, the feasible zone C increases. Conversely, when the threshold decreases (in the case of $c^{-}$ decreases or $c^{+}$ increases), the machine produces fewer parts and ages slowly, and for this reason, the feasible zone C decreases.

Production cost rate variation

We will now analyze the variation of the production cost rate $c_u$ for three values $c_u=100,500\mathrm{and}1000$ . The results presented in Figure 10 show that by increasing the production cost rate, the replacement is less recommended, since when the production is more penalized, the production threshold $Z(.)$ must be decreased, and to reach this level, the machine must take less time to produce parts at its maximum production rate $u_{\max }$ . Thus, the machine will deteriorate more slowly, and we have to replace it later: $a({\mathrm{s}}_1)<a({\mathrm{s}}_2)<a({\mathrm{s}}_3)$ , and the feasible zone C will decrease.

Figure 10.

Variation of the production cost rate and its effect on the production and replacement policies.

Replacement cost variation

Now, we illustrate the effect of the variation of the replacement cost $c_0$ on the production and replacement policies. As we can see from Figure 11, with three different cases, as $c_0=1500,3000\mathrm{and}4500$ , when the replacement cost increases, replacement is less recommended. The reason is that the cost to replace the machine is higher, and it is preferable to keep it longer. That is why we replace the machine later: $a({\mathrm{s}}_1)<a({\mathrm{s}}_2)<a({\mathrm{s}}_3)$ . However, with respect to the production policy, the replacement costs have not reported any influence on the production threshold $Z(.)$ when the machine is at its early stage of life. The change will occur only before entering in the replacement zone, and the system will build a certain stock level to hedge against shortages and to continue to meet customer demand. Once the decision to send the machine for replacement is made, it seems logical that the behavior of the production threshold $Z(.)$ over age remains the same (as observed in all three cases from Figure 11) because the MTTRP is the same in all three cases. Finally, it can be seen that the feasible zone C decreases as the replacement cost increases.

Figure 11.

Variation of the replacement cost and its effect on the production and replacement policies.

MTTR variation

We now discuss the variation of the MTTR for three values, $\mathrm{MTTR}=15,20\mathrm{and}25$ . From Figure 12, we can observe that when the $\mathrm{MTTR}$ increases, replacement is more highly recommended. The reason is that the machine becomes less available, and to continue to meet customer demand, the production threshold $Z(.)$ must be increased; however, to be able to do that, the machine must produce more parts at its maximum production rate $u_{\max }$ . Thus, we will have a more rapid deterioration of the machine, which is why we replace the machine earlier. We conclude that the age at the intersection point is $a({\mathrm{s}}_3)<a({\mathrm{s}}_2)<a({\mathrm{s}}_1)$ , and the feasible zone C will be increased.

Figure 12.

Variation of the mean time to repair and its effect on the production and replacement policies.

MTTRP variation

To complete the sensitivity analysis, we study the variation of the $\mathrm{MTTRP}$ , for the values $10,14\mathrm{and}18$ . We know that the availability of the machine decreases when more time is needed to replace it. As we observe in Figure 13, when the $\mathrm{MTTRP}$ increases, replacement is less recommended. The reason is that when it takes time to replace the machine, for the optimal control policy, it is preferable to keep it longer. Thus, we replace the machine later with the age at the intersection point $a({\mathrm{s}}_1)<a({\mathrm{s}}_2)<a({\mathrm{s}}_3)$ . Once the decision to send the machine for replacement is made, the machine must first produce more parts at its maximum production rate $u_{\max }$ to hedge against periods of nonproduction, when the $\mathrm{MTTRP}$ increases. Thus, the replacement will be performed with a higher age and with a higher stock level. In this case, the feasible zone C will be decreased. We note that the $\mathrm{MTTRP}$ has the same effect on the replacement policy as the replacement cost $c_0$ .

Figure 13.

Variation of the mean time to replace the machine and its effect on the production and replacement policies.

Discussions

The numerical results presented in this article show that the control policy for the considered manufacturing system is not a traditional hedging point policy but rather is a modified one when the machine deteriorates with age, and the repair activities are ABAO. This policy is of hedging point type in the sense that it is fully characterized by a curve within a domain of two parameters: the stock level x and the age a. We can conclude from the sensitivity analysis presented that the structure of the control policy obtained by our proposed approach is maintained when the parameters of the system vary. Incorporating the new production cost factor in the cost function leads to a reduction of the inventory level while continuing to meet the customer demand and keeps the machine longer within a low-cost level because it is aging more slowly. The results presented were obtained with the semi-Markov model because the simpler (Markov) model is not appropriate for systems subject to deterioration caused by machine aging. A careful analysis of applicability of optimization techniques based on a numerical solution of HJB equations was performed. The production and replacement policies are defined simultaneously by the function $Z(a)$ and $D(a)$ , the production threshold trace and the replacement trace, respectively. These two traces, and their intersection point in particular, determine the feasible zone C and the recommended stock and age levels, at which the decision regarding the machine replacement has to be made.

Conclusion

In this article, we determine simultaneously the optimal production and replacement strategies for the case of one machine and one product subject to random failures and repairs. A stochastic optimization model in continuous time has been developed. The considered manufacturing system is under machine’s age deterioration with the combined effect of imperfect repairs and AGAN replacement. By penalizing the production rate, we can explore and observe the impact of the machine aging on the optimal control strategies (production (rate) and replacement policies). From our results, it seems reasonable to incorporate the production parameter into the process of finding optimal control policies in order to get a better solution. This work is strictly related to the age phenomenon and provides a mathematically accurate analysis of the problem at hand in order to ensure the convergence of numerical methods based on the convexity of two-dimensional $(u,w)$ controls. Since the machine is not new after repair activities, a semi-Markov decision process has to be used in describing its dynamics, which seems more realistic in practice. The optimality conditions were developed in the form of the HJB equations using the stochastic optimal control and dynamic programming approach. The solution of the stochastic control problem is obtained using numerical methods. An extensive sensitivity analysis is performed to validate the structure of the obtained control policies. Finally, an extension of this model to the case of several machines remains to be done and is a part of our ongoing research.