Abstract
Multi-weapon production planning contains multi-objective combinatorial optimization and decision-making problems with the NP-hard and large-dimensional natures, which are difficult to be attacked by one single technique successfully. A four-stage hybrid approach is proposed to solve this problem. In the first stage, the multi-weapon production planning problem is formulated with 2N (N > 5) objectives based on operational capability requirements and expected downside risk measure. In the second stage, the formulation addressed is converted into a bi-objective optimization model using goal programming. In the third stage, an algorithm DENS based on differential evolution and nondominated sorting genetic algorithm–II is developed to obtain the Pareto set. Finally, the multiple attribute decision-making method technique for order preference by similarity to ideal solution is employed to acquire the compromise solution from the Pareto set. A case study is given to demonstrate the effectiveness of the proposed approach. The concrete advantages of goal programming, DENS, and technique for order preference by similarity to ideal solution are also validated in this case. This approach can support the weapon production planning in defense manufacturing and is also applicable to solve the multi-level and multi-objective problem in other manufacturing fields.
Keywords
Introduction
With the military requirements of joint operations,1,2 various weapons work together to form an integrated combat force at the right time and the right location in the modern battlefield. This requires an overall plan for development and production of multiple weapon categories in defense manufacturing. Therefore, multi-weapon production planning (MWPP) is put forward to meet the challenge. MWPP, as an operation and management activity, is directly correlated with national military force development and is useful for delivering integrated operational capabilities in future battle spaces. Moreover, it helps to avoid increasing risks in defense manufacturing. 3
Since MWPP plays such an important role in defense manufacturing, top-level managers, such as the senior executives for weapon acquisition in the defense sector as well as chief executive officers of the large military enterprises, pay close attention to approaches for optimization and decision-making problems in MWPP. This is also the research motivation of this article.
MWPP is a concept in which top-level managers select the optimum model from each weapon category simultaneously and subsequently plan the development time and production quantity of each selected weapon model to form a mix of production planning (MPP). An MPP contains all selected weapon models as well as the corresponding development time and production quantity. The development time is a time span that is available to produce, retrofit, and upgrade the selected weapon. Meanwhile, the planned production quantity of each selected weapon must be achieved within the allowed development time.
Many researchers have considered various objectives and proposed different production planning models in recent studies.4–7 Nevertheless, MWPP has its distinctive objectives and formulations that can be summarized as follows:
The production planning problems usually need to formulate the demands of the market and/or customers, while MWPP models the multiple capability requirements of joint operations. The demands of markets8–10 and the capability requirements correspond to different mathematical formulations and real problems. The specific formulations of capability requirements are presented in section “Formulations of the capability requirements.”
The objectives of production planning are usually maximizing the profit and/or productivity while minimizing the various kinds of costs, such as funds, times, and workforce, which are appropriately formulated by an economic measure.11,12 Nevertheless, one category of objectives for MWPP is maximization of meeting multiple capability requirements of joint operations. That is, suppose that there are N (N>5) capability requirements, then the objectives are to minimize the gaps between the N capabilities of the planned weapon mix and the corresponding N capability requirements. The specific formulations are presented in section “Formulations of the capability objectives.”
Since there are many uncertainties in the manufacturing and production processes of these selected weapons, the N risks arise from the MPP failing to achieve the N planned capability value. 13 Therefore, the other category of objectives of MWPP is to minimize the N risks of realizing the N planned capabilities. The risk objectives are formulated with expected downside risk (EDR) measure,14,15 which are presented in section “Formulations of the risk objectives.” To the best of our knowledge, this category of risk objectives has never been considered and discussed in other literature on production planning.
At the same time, the MWPP problem is also a combinatorial optimization problem with NP-hard nature, which will be quantitatively analyzed in section “DENS algorithm.” Therefore, MWPP contains multi-objective combinatorial optimization and decision-making problems with the NP-hard and large-dimensional natures.
There are two primary solutions to solve the multi-objective optimization and decision-making problem in the recent literature:
The first solution converts the multi-objective problem into one objective through a multiple objective decision-making (MODM) method. 16 Then adopts a heuristic and/or meta-heuristic algorithm to solve the problem. However, MWPP involves two kinds of objectives, namely, capability and risk. On the one hand, the capability objectives contradict with each other under the constraint of the overall budget. On the other hand, each capability objective and the corresponding risk objective are also incommensurable and in conflict with each other, and the larger the planned capability value, the higher the corresponding risk. 17 It is clearly nonapplicable and inefficient to integrate all objectives of capability and risk into a single objective for solving the MWPP problem.
The second solution uses a multi-objective optimizing algorithm 18 to obtain the Pareto set, which allows for selection of the best compromise solution from the Pareto set using a multiple attribute decision-making (MADM) method. 19 Nevertheless, this solution also cannot be directly applied to solve the MWPP problem. Because MWPP contains a large-dimensional multi-objective optimization problem that is also named as a many-objective problem 18 in the optimization field, the problem of how to effectively solve the many-objective problem is still one of the main challenges in the multi-objective optimization field. 20 It implies that this solution is also hard to attack the MWPP problem successfully.
Since N capability requirements and N aspired risk levels could be obtained from top-level managers, it is feasible that the N capability objectives and the N risk objectives are respectively converted into one objective by the MODM method goal programming. 21 Because the obtained capability requirements can be recognized as a group of target values for the capability objectives, and the aspired risk levels can also be recognized as a group of target values for the risk objectives, thus, they can be used as the input of the goal programming method. Once converted into a two-dimensional multi-objective problem, MWPP can be appropriately solved by the second solution. Thus, a practical solution based on the above idea is obtained for the MWPP problem.
According to above analysis, we propose a hybrid approach to solve the MWPP problem completely. This approach first converts the MWPP problem into a bi-objective optimization model by using goal programming, followed by the development of an algorithm DENS to solve this model based on nondominated sorting genetic algorithm–II (NSGA-II) 22 and differential evolution (DE). 23 Finally, the technique for order preference by similarity to ideal solution (TOPSIS) 19 is employed to obtain the compromise solution in which two types of weighting operations are discussed. To the best of our knowledge, this is the first description of this hybrid approach in the literature.
Literature review
Production planning is an open issue that contains many research fields, 24 such as aggregate planning, hierarchical planning, capacity planning, manufacture resource planning, and others. Many researchers have proposed various approaches for solving different multi-objective production planning problems in recent literature. The existing approaches can be primarily classified into two types of solutions:
1. The first one is that the multiple objectives are transformed into a single objective by using one MODM method. Many researchers have employed the fuzzy programming technique to deal with the multi-objective problem by considering the preferences of decision makers in several production planning fields.25–28 A number of studies have reported on the progress based on fuzzy programming for the specific applications of production planning. Selim et al. 29 and Jamalnia and Soukhakian 30 studied the fuzzy goal programming approach to solve the multi-objective problem in the distribution planning of supply chain and aggregate production planning, respectively. Lee et al. 7 presented a fuzzy multi-level multi-objective production planning model and developed an efficient two-phase interactive solution procedure to obtain a good compromise solution in a network product manufacturing supply chain. Due to the imprecise nature of the objectives’ aspiration levels, Torabi and Hassini 5 further developed an interactive fuzzy goal programming model to handle four important conflicting objectives simultaneously for multi-site production planning in automobile manufacturing. Sadeghi et al. 31 recently proposed a fuzzy gray goal programming approach for aggregate production planning in which each objective priority is defined as fuzzy number and the related parameters are specified by gray numbers. After the multi-objective is integrated into a single objective, a few researchers considered the meta-heuristic algorithm to solve the single-objective optimization problem in several areas.32,33
This kind of solutions emphasizes preference information (PI) and interaction with the decision makers in order to transform all objectives into a single objective. It does not generate a widespread set of nondominated solutions that would be useful for supporting decision making by managers. In addition, there have not been a sufficient number of studies using meta-heuristic approaches to solve the NP-hard optimization problems that exist in real production planning fields.
2. The second one is to develop a multi-objective optimization algorithm to obtain a widespread set of nondominated solutions. Several studies have explored the multi-objective optimization algorithms based on genetic algorithm and NSGA-II in order to solve their specific production planning fields, such as flexible manufacturing systems,
34
preventive maintenance planning,
35
aggregate production planning for shipbuilding,
6
and multi-objective order scheduling.
36
There are also other multi-objective optimization approaches which are found in recent literature. Lu et al.
37
developed a multi-objective disassembly planning approach with an ant colony optimization algorithm to obtain the Pareto set of objectives of disassembly orientation changes, tool changes, and changes in disassembly operation types. Pan et al.
38
proposed a novel multi-objective particle swarm optimization algorithm for the no-wait flow shop scheduling problems, and Marjani et al.
39
presented a bi-objective heuristics algorithm for a multi-item freight distribution planning problem. Both these studies considered two objectives: make-span and tardiness. To extract a set of Pareto-optimal solutions from a NP-complete problem of multi-objective production planning, Karimi-Nasab and Konstantaras
40
proposed a random search heuristic algorithm considering the three objectives simultaneously. Al-e-Hashem et al.
41
introduced an efficient algorithm that combines an augmented
This kind of solutions focuses on obtaining the Pareto set with good distribution and convergence but does not help decision makers to obtain the best fit compromise solution from the Pareto set. Moreover, if the Pareto set consists of hundreds of solutions, then the decision makers have a difficult time directly selecting the best fit compromise solution by themselves. In addition, this category of the solution is suitable for solving the small-dimensional multi-objective problem but has a difficult time handling multi-level and multiple objectives with a large-dimensional nature.
After review of the aforementioned solutions and various planning approaches in the recent literature, we conclude the special abilities of the proposed hybrid approach in this article as follows:
Compared with the first solution which converts the multi-objective problem into a single objective, the hybrid approach converts multiple objectives into small-dimensional objectives based on their levels of the objectives. Furthermore, the converted small-dimensional objectives are optimized by the proposed DENS algorithm to provide a group of Pareto solutions with good distribution for decision makers.
Compared with the second solution which develops an multi-objective optimization algorithm to obtain the Pareto set, this hybrid approach not only can decrease the dimensions of the large-dimensional multi-objective problem but also support obtaining the best fit compromise solution from the Pareto set. In addition, when managers have a difficult time determining their preference for each objective, our approach also provides the objective weighting operation based on information entropy (IE). 42
Above all, the primary distinction of the proposed planning approach is to integrate multiple techniques that are separately studied in different fields and literature into a comprehensive and practical planning approach, which can simultaneously handle the large-dimensional multi-objective decision-making problem and the NP-hard optimization problem that currently exist in production and manufacturing fields.
Procedure for solving the MWPP problem
In this section, a four-stage procedure is introduced to model and solve the MWPP problem, as shown in Figure 1. In addition, we will elaborate why we choose goal programming, DE, NSGA-II, and TOPSIS to solve the MWPP problem.

The procedure for solving the MWPP problem.
Stage 1. We formulate the capability requirements, capability objectives, and risk objectives in turn, and the risk objectives in particular are based on EDR; the related constraints are then formulated; and finally, the related data are converted as the parameters of the formulations of MWPP. The detailed process will be elaborated in section “Formulations of the MWPP problem.”
Stage 2. The planned capabilities of the MPP must meet the capability requirements and the corresponding risks also must meet the aspired levels. In other words, the capability requirements and aspired risk levels can be recognized as two groups of goals (or target levels), and the aim of the MPP is to reach them as closely as possible. Meanwhile, goal programming as a MODM method is used for minimizing the deviation between the achievements and targets, which has been proved effectively in many fields.43–45
Therefore, goal programming is applied to convert the MWPP problem from a large-dimensional multi-objective problem to a two-dimensional multi-objective one. That is, all the capability/risk objectives can be converted into the overall capability/risk objective. Thus, the MWPP formulations can be built into a bi-objective optimization model by using goal programming. The detailed modeling process will be elaborated upon in section “Bi-objective MWPP model based on goal programming.”
Stage 3. NSGA-II is one of the most popular Pareto dominance–based algorithms in recent decades. It encompasses advanced characteristics like the elitism strategy and fast nondominated sorting operations, but it suffers from low mutation rates and fixed step sizes, which are rooted in the traditional genetic algorithm. 46 DE is proven to be a promising candidate algorithm, with its effective mutation operation in the field of Multi-Objective Evolutionary Algorithm.47,48
Therefore, we develop an algorithm DENS which combines the advantages of DE and NSGA-II in order to solve the bi-objective optimization model of MWPP. The details of this algorithm will be elaborated upon in section “DENS algorithm.”
Stage 4. TOPSIS is a simple and reliable method, and it is attractive because of the limited subjective input acquiring from the managers. TOPSIS is combined with other optimization and decision-making methods to solve these multi-objective optimization and decision problems, which can be found in recent studies.49–51
Accordingly, we will employ TOPSIS to obtain the best compromise solution from the capability-risk Pareto set of the MWPP problem. Meanwhile, we will respectively investigate the objective weighting based on the IE and the subjective weighting based on the PI of the managers. The detailed process will be elaborated upon in section “Compromise solution based on TOPSIS.”
Formulations of the MWPP problem
Formulations of the capability requirements
Suppose that there are N capability requirements and M weapon categories, in which each capability requirement is supported by more than one weapon category, and each weapon category has more than one planned capability. Each weapon category contains a number of weapon models. In each weapon category, the capability values of each weapon model depend on the corresponding development time and production quantity, and the weapon models have different capability values with each other, when they have equivalent values of development time and production quantity.
To formulate the capability objectives, the capability requirements must be formulated first. In existing studies, some of the descriptions on the capability requirements are qualitative;52,53 the others are concentrated in some specific fields.54–56 Therefore, we propose a general formulation of the capability requirements as follows
There are totally N capability requirements, in which CRn represents the nth capability that must achieve the value CRqn in time CRtn. For the sake of modeling the capability objectives, CRqn is normalized in a dimensionless treatment, namely, 0 < CRqn < 1.
Formulations of the capability objectives
In order to formulate the capability objectives, we must first calculate the capability values of each selected weapon and subsequently calculate the overall capability values of the MPP. If weapon
where the L-order polynomial represents the relation between the capability n and the development time of weapon
Suppose that the production quantity of weapon model increases linearly with the growth of time,
After acquiring the capability value n of each weapon model, the nth capability value of the MPP can be calculated through weighted summation, which is presented as follows
where wmn = 0 if weapon category m does not have the capability n. To evaluate the capability gap between the MPP and the nth capability requirement, we calculate it as follows
Formulations of the risk objectives
The risk objectives are that the MPP has the minimum risks to achieve its N planned capabilities and a reasonable measure is requested to evaluate it first. We employ EDR to formulate the risk objectives. Because the practical capability of the MPP is possible lower or higher than the planned capability, while the risk is only associated with those possible capability values that are smaller than the planned capability values.
Suppose that the nth capability of weapon model achieves possible capability value
Therefore, the value of risk n of the MPP based on EDR is formulated as follows
where Cn is the planned capability value n of the MPP, and
Calculation of the constraints
The available ranges of development time and production quantity are evidently not infinite, and the planned capabilities of the MPP are also constrained by the defense budget. Thus, the related constraints are
where equation (9) is the constraint of development time; equation (10) is the constraint of production quantity; equation (11) is the overall budget constraint of planning the MPP, which means that the larger planned capabilities demand more costs.
Proposed approach development
Bi-objective MWPP model based on goal programming
The weighted goal programming
21
model considers all the objective functions
where
For the N capability objectives in the MWPP problem, we hope that the capability values of the MPP exceed the capability requirements values (goals) as much as possible, namely, we only focus on the negative deviations but the positive deviations between each capability value of the MPP and the corresponding capability requirement. Meanwhile, we consider the aspiration levels of the risk objectives as 0, obviously, only positive deviations exist. Therefore, the goal programming based bi-objective optimization model is formulated as follows
where
DENS algorithm
The purpose of DENS algorithm development is to solve the bi-objective optimization model of MWPP and obtain a set of nondominated solutions with good convergence and diversity.
Before development of the addressed algorithm, we analyze the NP-hard nature of the MWPP problem. Since all decision variables take an integer, and the scale of the alternative space S of the MPP and its lower bound are
S increases exponentially with the increase of M, and the computational complexity of MWPP is larger than the knapsack problem 57 which is NP-hard; therefore, it can be inferred that the MWPP problem is also NP-hard.
The mutation operation of DE is excellent at the search efficiency and avoiding premature convergence. We focus on applying the mutation operation of DE into NSGA-II to solve the MWPP problem. The classical mutation operation DE/rand/1 23 has been applied to improve the searching ability of NSGA-II for some specific problems,22,24 but another representative mutation operation DE/rand-to-best/1 23 combined with NSGA-II is not reported because it is hard to determine which individual is the best in the Pareto set. We propose a technique to construct the best individuals set from the Pareto set, which can then be used in the DE/rand-to-best/1 to improve the evolution operation of NSGA-II. The whole process of the algorithm development is presented as follows:
Step 1. Initialize the population.
Define population size P, numbers of generations G, and let count = 1. The initial population Pop_1 is generated and the encoding of each individual is designed as follows
where xm, ym, and zm are, respectively, the sequence number, time, and quantity of the selected model in the weapon category m. Each bit adopts the decimal integer encoding.
Step 2. Evaluate and rank the individuals.
Evaluate each individual by objectives (15) and (16) and constraint (11) and subsequently use the fast nondominated sorting approach 18 to determine the nondomination level and crowding distance of each individual.
Step 3. Execute the binary tournament selection. 18
Generate Pop_2 from Pop_1 with tournament selection. The selection principle is that if the two candidate individuals have a distinct nondomination level, then individual with the minimum nondomination level is selected; if both of them have the same nondomination level, then the individual with the maximum crowding distance is selected as the new individual.
Step 4. Execute the mutation operation of DE.
DE/rand/1 and DE/rand-to-best/1 are respectively applied to execute the mutation operation of DE.
1. DE/rand/1 operates the individuals in the Pop_2 to generate the temporary population Pop_3, which is formulated as follows
where r1, r2, and r3 are the randomly selected individuals in the Pop_2; rand represents a random number between 0 and 1; r′ is the new individual that is put into the Pop_3.
The developed algorithm with the DE/rand/1 mutation operation is called DENS-I.
2. DE/rand-to-best/1 operates the individuals in the Pop_2 as follows
where r is the individual in the Pop_1, and rgbest is the global optimal individual in the single-objective optimization. Thus, we construct the global optimal individuals set for multi-objective optimization as follows:
Step 4.1. Sort the individuals of the Pop_2 in ascending order according to the arbitrary one objective value.
Step 4.2. Set the numbers of the group p (p < P), and initialize count1 = 0.
Step 4.3. Take out a group with a random position from the Pop_2, which is illustrated in Figure 2.
Step 4.4. For the individuals in the random group, normalize the objective values rj_objo to be rj_objo′ as follows
where min(rj_objo) (resp. max(rj_objo)) is the smallest (resp. largest) value of the oth objective from the p individuals.
Step 4.5. Select the individual with the minimum value from the group to put into the global optimal population Pop_gbest; the minimum individual s_obj is formulated as follows
Step 4.6. Set count1 = count1+ 1. If count1 < P, go to step 4.3; otherwise, stop and output the Pop_gbest.
The algorithm with the DE/rand-to-best/1 mutation operation is called DENS-II.
Step 5. Execute the crossover operation of DE.
Each bit of the individual in the Pop_2 is selected with a certain crossover probability to replace the corresponding bit of the individual in the Pop_3.
Step 6. Reload and evaluate Pop_3.
For the individuals of the Pop_3, the bits which infringe upon the constraints are reloaded as the randomly generated numbers within the constraints (9) and (10). Then, evaluate each individual with objectives (15) and (16) and budget constraint (11).
Step 7. Calculate nondominance level and crowding distance.
Calculate the nondominance level and the crowding distance for each individual in the intermediate population which is generated by combining Pop_1 with Pop_3.
Step 8. Generate new population.
Select P individuals from the intermediate population to be the new Pop_1. The selection principle is that the new population is filled by the individuals with superior nondominated level. If the population exceeds P by adding all the individuals in level Lj, the individuals in the level Lj are selected with the large crowding distance.
Step 9. Stop and generate the Pareto set.
Let count = count+ 1, If count < G, go to step 3; otherwise, stop and output the Pareto set.

The random group from the Pop_2.
Compromise solution based on TOPSIS
The purpose of using TOPSIS is to help managers obtain the best fit compromise solution from the capability-risk Pareto set that was obtained with the DENS algorithm.
The concept of TOPSIS is to determine the positive ideal solution
Step 1. Construct the decision-making matrix.
Suppose that there are τ solutions in the Pareto set. The decision-making matrix A is
where
Step 2. Normalize
Step 3. Determine the weight (preference) of each objective.
One way is the subjective weighting according to the preference of the managers, the other way is the objective weighting according to the IE emitted from A. The entropy value of objective j can be calculated as follows:
Step 3.1. The decision matrix A needs to be normalized as follows
Step 3.2. The IE of each objective can be measured by the entropy value ej as
Step 3.3. The weight for objective j is thus given by
Step 4. Construct the weighted (referenced) normalized decision matrix
Step 5. Determine
Step 6. Calculate the separation measures
The separation measure
The separation measure
Step 7. Calculate relative closeness Hi of solution i
Step 8. Choose the best compromise solution which has the largest value of Hi.
Case study
Problem description
The intelligence, surveillance, and reconnaissance (ISR) weapon production planning, which is one of the MWPP problems, is chosen to demonstrate the effectiveness of the proposed hybrid approach.
The scenario is that the ISR weapons must be planned and developed to meet the ISR capability requirements of joint operations in the future. The primary data are shown in Figure 3, such as the categories of the ISR weapons, optional range of development time and production quantity, and the capability requirements.

The candidate ISR weapons and the capability requirements.
The arrows indicate the ISR weapon category that can support the corresponding capability requirements. Each capability requirement consists of two measures, for instance, the capability requirement 1, namely, target identification needs to achieve 0.9 within 8 years. Each ISR weapon category has a distinct optional range for the model, time, and quantity. For instance, the reconnaissance satellite (R-Sate) has five optional models, the optional development time range is from 5 to 10 years, and the optional production quantity range is from 5 to 20. For each category of ISR weapon, if one model is selected and its development time and production quantity are determined, we can obtain the expected capability values of this weapon model according to equation (2).
We assume that each selected weapon model in the MPP will achieve a number of possible capability values. The possible capability values with their probabilities are from the simulated data based on the empirical data and the expert opinions.
After inputting all the parameters, the ISR bi-objective optimization model can be obtained as follows: First, we calculate each planned capability value of the selected weapon model according to equations (2) and (3). Second, each capability value of the MPP is calculated by equation (4). Third, each risk objective value in response to the capability of the MPP is calculated by equations (6)–(8). Fourth, we calculate the constraints of MPP through equations (9)–(11). Finally, we build the bi-objective optimization model for the ISR MWPP problem according to equations (15)–(19).
Validation of the DENS algorithm
After building the ISR bi-objective optimization model, we use NSGA-II, DENS-I, and DENS-II to solve it with MATLAB Version 7.11. The scale of the solutions for this problem is far more than
Convergence (γ). This metric calculates an average distance between the obtained nondominated solutions and the actual optimal Pareto set or reference set. According to this metric, if an algorithm obtains a smaller γ value, it is able to find better convergence.
Spread (Δ). This metric is useful for evaluating diversity among the nondominated solutions. According to this metric if an algorithm obtains a smaller Δ value, it is able to find a better diverse set of nondominated solutions.
MS. This measures the standard deviation of the different distances of the solution values in the solution space. When the solutions are nearly uniformly spaced, the corresponding distance measure will be small. Thus, an algorithm that obtains a set of nondominated solutions with a smaller spacing is better.
IGD. This metric is designed for both convergence and diversity. If the Pareto set has a lower value of the IGD, it will be close to the true Pareto set, which means that the lower the value of IDG, the better the solutions.
When the optimal Pareto set is not known for the actual NP-hard problem, a reference set is needed to calculate these metrics. We first execute these three algorithms 30 times to obtain a
The Pareto sets of the three algorithms corresponding to the best results of 30 independent runs, respectively, are shown in Figure 4. In this figure, the two ends of the curves have been enlarged in order to show the distinctions of the spread between the three algorithms. It can be observed that the Pareto set of DENS-II has the longest distribution curve, followed by DENS-I and NSGA-II. Furthermore, in Figure 4(a), the top-left points of DENS-II, DENS-I, and NSGA-II are (0.0781, 1.1326), (0.1817, 1.0114), and (0.2327, 0.9568), respectively. In Figure 4(c), the bottom-right points of DENS-II, DENS-I, and NSGA-II are (3.3859, 0.1711), (3.3595, 0.1723), and (2.6564, 0.2390), respectively. This implies that the smallest overall capability gap 0.0781 and the smallest overall risk level 0.1711 are obtained by DENS-II. Therefore, it is concluded that the Pareto set of DENS-II has the best spread and diversity, followed by DENS-I. The worst spread and diversity occurs with NSGA-II.

The best Pareto set of NSGA-II, DENS-I, and DENS-II: (a) enlargement of the top-left end, (b) original Pareto set, and (c) enlargement of the bottom-right end.
To more clearly exhibit the convergence of the Pareto sets of the three algorithms, we linearly transform the solutions to be within [0, 1] as follows
The transformed Pareto set for each algorithm is shown in Figure 5. It is observed that the Pareto set of DENS-II has the best convergence, followed by DENS-I and NSGA-II. Furthermore, the convergence of DENS-I advanced significantly based on NSGA-II, while the improvement of the convergence between DENS-II and DENS-I is smaller than former. The quantitative data and statistical comparison of the performance metrics with their best, mean, worst, and standard deviation (σ) values are shown in Table 1.

The linearly transformed best Pareto set with NSGA-II, DENS-I, and DENS-II.
Statistical comparison among NSGA-II, DENS-I, and DENS-II.
NSGA-II: nondominated sorting genetic algorithm–II; MS: minimum spacing; IGD: inverted generational distance.
As shown in Table 1, all of the performance measures of DENS-II have the minimum values on the mean and the best. They quantitatively confirm that the Pareto set of DENS-II outperforms the ones of NSGA-II and DENS-I on convergence, spread, diversity, and uniform distribution. However, the σ and the worst for these four performance measures have the smallest values in DENS-I. Thus, we could conclude that DENS-I has the best robustness among these three algorithms.
Analysis of the ISR MPPs based on TOPSIS
After acquiring the best Pareto set with DENS-II, the compromise solution can be obtained based on the steps of TOPSIS in section “Compromise solution based on TOPSIS.” We obtain two solutions: one is based on the PI, and the other one is based on the IE. The weight of the PI-solution is assumed as W = (0.5, 0.5), which shows a nonpreference for both objectives. The weight of the IE-solution is calculated as W = (0.7876, 0.2124) according to step 3 in section “Formulations of the risk objectives.” Therefore, the two solutions in the Pareto set are shown in Figure 5.
PIS and NIS are, respectively, the positive ideal solution and negative ideal solution. The objective values of the PI-solution and the IE-solution are, respectively, (0.8831, 0.6179) and (0.1938, 0.9894), where the first item is the overall capability gap, and the second item is the overall risk level. The IE-solution puts an emphasis on the overall capability gap compared with the PI-solution because the IE assigns a larger weight to an objective whose values have a broader distribution, namely, if the objective has similar values across alternatives, it will be judged as unimportant to help in decision making. The corresponding MPPs of ISR for these two compromise solutions are shown in Table 2.
The MPPs of ISR based on different weights.
MPP: mix of production planning; ISR: intelligence, surveillance, and reconnaissance; PI: preference information; IE: information entropy; R-Sate: reconnaissance satellite; AWACS: airborne warning and control system; HA-UAS: high-altitude unmanned aerial system; MLA-UAS: medium- to low-altitude unmanned aerial system; R-ship: reconnaissance ship; PAWS: phased array warning system; LOS-DL: line of sight data link; BLOS-DL: beyond line of sight data link; C2S: command and control system.
As shown in Table 2, these two MPPs of ISR weapons select the same model for each category of ISR weapons except the R-Sate and the airborne warning and control system (AWACS). The main difference between these two MPPs of ISR weapons are the development time and production quantity, for instance, both of these MPPs select the third model in the high-altitude unmanned aerial system (HA-UAS) category of ISR weapons, but the development times are planned as 7 and 10 years, respectively, and the production quantities are also different from each other.
For each MPP of ISR weapons shown in Table 2, we investigate the individual capability value and corresponding risk level. The results are shown in Figure 6.

Two compromise solutions in the best Pareto set.
As shown in Figure 7, for the information transition and distribution (ITD; CR4), interoperability (CR5), and C2 (CR7), the IE-based MPP can better meet each individual capability requirement compared to the PI-based MPP, but the individual risk aspiration of PI-based MPP is lower than the IE-based MPP. Accordingly, we should strengthen the development of the ISR weapons that can support the ITD, interoperability, and C2. In addition, both of the MPP of ISR weapons can basically meet the needs of the other capability requirements and their aspired risk levels which are assumed to be 0.

Each planned capability value and risk value for each MPP of ISR.
Conclusion
In this study, we introduced the MWPP problem and the NP-hard, and the large-dimensional multi-objective natures are quantitatively analyzed. A general formulation is first proposed for the MWPP problem. In addition, the risk objectives are formulated with the EDR measure.
A hybrid approach, which combines goal programming, DENS, and TOPSIS, is proposed to solve this problem completely. A case of ISR MWPP has been studied to validate the effectiveness of the proposed approach. Given the related inputs, this approach can generate the Pareto set and the best fit compromise solution of the MPP, which consists of selected weapon model, development time, and production quantity.
The goal programming, as a module of the proposed approach, proved that it can convert the capability objectives and risk objectives into a bi-objective optimization model. This module not only contributes to optimizing the overall capability gap and overall risk level but is also useful for satisfying each capability and risk in good balance. However, the manner in which to weight each objective in the process of building a goal programming model still needs to be investigated.
DENS has been validated as an effective algorithm for the MWPP problem. Furthermore, the statistical comparison of the performance metrics implies that DENS-II outperforms the DENS-I and NSGA-II on diversity, spread, and convergence, nevertheless, DENS-I has the better robustness than DENS-II based on the standard deviation. More examples of the MWPP problem must be studied to validate the performance of DENS for further research.
It has been shown that TOPSIS can effectively obtain the final compromise solution for the MWPP problem. When considering the participation of the managers, TOPSIS can acquire the compromise solution through PI-based weights. Meanwhile, it also generates the objective compromise solution with IE-based weights. It must be noted that the MPP provided by the TOPSIS can be further adjusted and updated by managers.
Although this hybrid approach aims at the production planning problem in the defense manufacturing, it can also be considered as a generic methodology to solve the multi-level and multi-objective planning problem in other engineering manufacture fields. One of the applicable directions of this methodology is the supply chain network management problem7,29,33 which widely exists in the multi-company, competitive manufacturing, and sourcing environments. For example, considering the management of a manufacturing supply chain network with S suppliers, P plants, and D distributors, on the one hand, the profit objective of each partner should be considered, and, on the other hand, the entire supply chain network must maintain an ecological balance and sustainable development. A multi-objective problem with three-level and S + P + D objectives clearly emerges from this supply chain network. According to the proposed methodology, this problem could be converted to a three-objective optimization model using goal programming based on the aspired profit level of each partner, followed by application of the DENS algorithm to solve this model and obtain the Pareto set. Finally, the best fit compromise solution is obtained by employing TOPSIS. This final compromise solution corresponds to a mutually beneficial, win–win partnership that creates sustainable and competitive advantages for the entire supply chain network. The generic extension and application of this hybrid approach are also one of the research directions that will be explored in the future.
Footnotes
Appendix 1
Acknowledgements
We are grateful to the anonymous reviewers for their valuable comments and constructive criticism.
Declaration of conflicting interests
The authors declare that there is no conflict of interest.
Funding
This research is supported in part by the National Natural Science Foundation of China under contract Nos 71031007, 71001104, and 71201168, as well as the Research Fund for the Doctoral Program of Higher Education of China under contract No. 2012430712001.
