Evaluation of the suggestions system performance using robust DEA model: The case of National Iranian Gas Company

Abstract

The suggestions system is a part of total quality management to create individual and group spirit of partnership between staff and increase efficiency in the organization. Also, diagnosis and improvement process is one of the steps of the chain in the processes of suggestions system. In this study, an approach has been proposed to evaluate efficiency of organizations in performing suggestions system with these aims: (1) Reviewing all the elements in the successful implementation of the suggestions system and (2) providing an effective scientific approach to evaluate the organizations on implementing this system considering the uncertainty in the data. Methodology used in this study included the following techniques: (1) Factor analyzing to clarify the internal correlation between significant criteria and detect the major criteria and (2) using robust data envelopment analysis (RDEA) model to evaluate efficiency of organizations in performing suggestions system. The method is based on 3 inputs and 17 outputs in which some outputs are uncertain scores in form of intervals with uncertain bounds. This model has been solved for different Γs, and a value of weights and rankings for each Decision Making Unit (DMU) has been saved by using the obtained values. In the following a simulation has been used to compute the conformity of the rankings from the RDEA model with reality. Doing so shows that the maximum conformity occurs Γ = 6. Therefore, we can conclude that specific values of Γ can maximize conformity and thus more authentic final rankings for the DMUs in this interval of Γ may be expected.

Introduction

The suggestions review and acceptance system is a part of total quality management to create individual and group spirit of partnership between staff, improve processes, and increase efficiency in the organization. Fairbank and Williams introduced suggestions system as a common solution for exploitation from creativity of staffs. ¹ Pluskowski introduced creativity as one of the major capabilities of the man, yet it cannot oblige people to propose ideas in a civilized society. ² They have to be volunteer to propose their ideas. According to the study of Van Dijkand Van Den Ende, ³ suggestions system includes the administrative arrangements and essential infrastructures to collect, judge, and pay reward for the ideas that are proposed by the staffs. ⁴ Indeed, performing suggestions system is an educational and promotional movement to improve the activities of organization through aligning the individual aims with organizational aims and increase organizational belonging, commitment, and motivation to all the affairs of the organization. ⁵ With regard to the point of view of Marksberry et al., the suggestions system is assumed as a way to improve participation by members of organization to help resolving the problems that are not resolved through traditional organizational methods especially at the government sector and constant changes in managements. ⁶ In this system, material rewards are considered for staffs’ thoughts and notions and their participation in decision-making of organization. Observing the principle of continuous monitoring of processes and results, monitoring problems and defects and taking action to solve them, and modifying or redesigning process are considered as requirements for continuous organizational improvements in total quality management. Inspired by European Foundation Quality Management, assessment pattern of suggestions system and assessment checklist of this system have been designed and performed; this model refers to a nonprescriptive framework formed based on eight-dimensional criteria. Some of these criteria develop enabler group and indicate the type of activities of organization and how to accomplish those activities and the residual part of the results, specifying the demands that the organization must achieve by the performing enablers. It seems that success in improvement of suggestions system and achievement of their aims require reviewing these criteria and resolving their defects well suited to the coordinates of aforementioned pattern as well as their generalization in all levels involved in suggestions system. The reasons for lack of success in performing suggestions system in some organizations include lack of belief in participation among staffs, lack of belief by directory of the organization in the significance of participation by staffs in affairs administration of the organization, authoritarian spirit in managers, inadequacy of mechanisms and incentive policies, and superficial look into suggestions systems. The proper assessment of performance of suggestions system based on a scientific principle and method and the suggestion for suitable mechanisms in this context can be a step to resolve some of the deficiencies mentioned above. Performance evaluation of the suggestions system in the current approach has three fundamental problems:

Selecting criteria and their weights is based on unscientific views.

Assessments are based on multi-criteria decision-making which cannot be ensured.

Integrating of assessments is based on unscientific views.

The present research intends to propose an approach to assess the performance of suggestions system, using robust data envelopment analysis (RDEA), such that some of the reasons for lack of success of this system which are related to improper assessment are clarified. This research has been conducted in five sections; the first section examines the literature review and related works on models for assessment of suggestions system, the use of analytic hierarchy process and data envelopment analysis (DEA), and a variety of approaches proposed in exposure with uncertain data. In the second section discusses the research methodology. In this section, the criteria have been screened by considering experts’ and specialists’ views and then the criteria known for enabler criteria are summarized in 17 cases in three groups using exploratory factor analysis. In this section, RDEA has been introduced as an approach for exposure to the problem of data uncertainty, and how to use this method to assess the performance of suggestions system of 60 organizations has been examined. In the third section, the proposed model is introduced and its performance is examined; in the fourth section, the results from the use of RDEA method have been proposed to assess 60 organizations under discussion in this research; ultimately in the last section, we sum up our conclusions, and the findings of research are proposed.

Literature review

The term “assessment” has been defined in various forms in literature review and theoretical background. Nelly et al. has defined performance measurement as the process of quantification of efficiency and effectiveness of activities. ⁷ According to the definition by Simons, performance assessment system has been defined with four major aims including transfer of information, focus on official affairs and procedures, design for use of managers, and supervision of maintenance or modification of organizational activity patterns. ⁸ According to another definition, performance assessment is defined as a systematic process that plans and organizes the tasks and expectations, monitors the performance constantly, creates the implementation capacity, ranks performance periodically, and grants reward to suitable performance. ⁹ To date, numerous studies have been conducted in the context of design of performance assessment systems. Some have been in the context of basic research, resulting in proposal of various performance assessment systems. These performance assessment models include Sink and Tuttle model (1989), performance matrix (1989), the model of results and determinants (1991), the performance pyramid (1991), balanced scorecard (1992), stakeholder analysis (2001), and the business excellence model. ¹⁰ Performance assessment of suggestions system has been regarded as an issue with wide literature review. These studies have been conducted via various techniques and approaches, attempted to introduce a pattern for performance assessment of this system. In one study, Ehsan Jaffarpour et al. proposed a framework using Data Envelopment Analysis (DEA) to assess the performance of this system. ¹¹ In another study, Mohammad Hadi et al. proposed a Context, Input, Process, and Product Evaluation Model (CIPP) model for this purpose. This study has been designed and performed to assess the performance of this system in Medical Science University of Isfahan. ¹² One of the methods to allocate weight to input and output criteria was used in an article by Thompson et al. (1997) entitled “Assurance Region” to select the best place for high-energy physics laboratory. Since there were numerous zero values in optimal weight (u_r *, v_i *) in DEA models, it was indicated that some inputs or outputs are poorer than the rest of the efficient DMUs; thus, this defect in DEA method is resolved through invention of assurance region method. ¹³ Another method introduced for this purpose is referred to as the “cone ratio envelopment” as suggested by Charles and his colleagues. In another study by Brackets et al. cone ratio envelopment has been used to assess performance of banks under the conditions that uncertain aid grant was considered for risk and similar factors. ¹⁴ In 1998, assurance region method was proposed for transfer of capital in Japan in form of a plan. In this plan, the criteria such as distance from Tokyo, access to an international airport, and several other criteria were considered, and that the scores were specified for each place. The assessors considered a weight for each of criteria based on analytic hierarchy process and ultimately obtained an assurance region for each weight through calculating the ratio of weights specified by different assessors. ¹⁵ Due to extensive applications of DEA model in the real-world problems based on the studies by Charnes, Cooper and Rhodes, huge efforts have been made to expand DEA models. Encounter with uncertain data has been regarded as an issue drawn into attention by Cooper et al. ¹⁶ and Despotis and Smirlis. ¹⁷ In ordinary DEA, all the data are assumed as certain numerical values. Yet, the observed values of inputs and outputs in the real-world problems are often uncertain. Uncertain data in assessment and especially in DEA models have been examined in the literature in different forms. The article entitled “Bridge Condition Assessment Using D-Numbers” by Xinyang Deng et al. ¹⁸ and the article entitled “A Hybrid Fuzzy Evaluation Method for Curtain Grouting Efficiency Assessment Based on an AHP Method Extended by D Numbers” by Guichao Fan et al. ¹⁹ are some of these studies. In the former, a hierarchical model has been established for the assessment and then aggregates the assessment results on different factors to obtain an overall assessment for a bridge. During this assessment process, due to lack of precise instrument and the limitation of cost, the assessment result of each factor has been given by bridge monitor through the visual observation according to Deng’s previous experience. Xinyang Deng has extended the Dempster–Shafer theory using D numbers. Deng’s method was simpler and more effective than the Dempster–Shafer’s method. ¹⁸ The second article proposed a hybrid fuzzy evaluation method for curtain grouting efficiency assessment based on a fuzzy comprehensive evaluation method and a D-Number Analytical Hierarchical Analysis (D-AHP) method. The weights of the indicators have been determined by the D-AHP method. The proposed method by Zhonga was successfully applied in the curtain grouting efficiency assessment of a hydropower project in China. ¹⁹ Some researchers have suggested fuzzy DEA and interval DEA (IDEA) in the encounter with uncertain data. In more recent period, uncertain data have been expressed by means of two approaches. IDEA was proposed for the first time by Cooper et al., ¹⁶ and fuzzy DEA was proposed for the first time by Sengupta. Cooper et al. ¹⁶ have extended an interval approach that allows using a mix of uncertain and certain data by means of transformation of DEA model to an ordinary linear planning form. Assessment of lower and upper limits of Decision Making Unit (DMU) efficiencies has been regarded as one of the problems in interval approach. ¹⁶ Despite this problem, some researchers have proposed a variety of interval approaches. ¹⁷ In another article, Guo and Tanaka proposed an approach that changes the fuzzy DEA model to a bilevel linear planning model. ²⁰ Soleimani Damaneh et al. ²¹ indicated that their model cannot be common as a proposed model, having an optimal response under limited conditions. Despite the studies by Guo and Tanaka, used the proposed fuzzy DEA model by Guo and Tanaka and introduced a fuzzy integrative framework to integrate fuzzy values with multiple states. In addition, Guo used the model proposed by Guo and Tanaka in a case study of localization problem of a restaurant in China. ²² Kao and Liu ²³ proposed a technique which transforms fuzzy DEA model to a family of certain DEA models by use of this approach. In another research, Azadeh et al. used fuzzy DEA model to specify relative efficiency of existing units in the electricity generation sectors in Iran. In this research, using data in fuzzy form and creating Charnes, Cooper & Rhodes (CCR) fuzzy model, it has been transformed to a model with interval data via a model based on a cut, and the minimum and maximum efficiency is acquired for the aforementioned units. ²⁴ In another research by Shokouhi et al., an approach based on RDEA was proposed in which the input and output parameters are restricted to a point that they must be considered as an indefinite set with additional restrictions based on the worst corresponding response with uncertain set. The proposed RDEA searches maximum efficiency under this assumption that the worst corresponding efficiency has been defined via indefinite set. ²⁵ In another research, Najafi et al. proposed the use of RDEA to assess organizational excellence projects. In this article, creation of integrated model of DEA and organizational excellence model considering uncertainty in data is considered, and the efficiency of decision-making units is assessed in which the organizational excellence model has been implemented. ²⁶

Research methodology

The methodology used in this assessment has been taken from the applied realities of the subject under discussion. Two categories of input and output data are detected in assessment of suggestions system based on the proposed criteria grounded on excellence approach. Then the efficiency of suggestions system is assessed using DEA. Since the assessment criteria have been qualitative and the data pertaining to inputs and outputs related to these criteria have been given interval value, the approaches are examined which engage in DEA with indefinite data. In this research, we deal with a type of interval data that their limits are indefinite considering this point that assessments are made by different assessors, in which a numerical interval is defined for each input or output. This results in use of the approaches that consider data uncertainty in the problem. The current model of performance assessment in suggestions system is based on a simple form of multi-criteria decision-making, in which a score has been determined based on experts’ view for each of the criteria and subcriteria. With regard to the investigations on executive coordinates of suggestions system, it seems that such assessment model has huge problems despite its simplicity. The problems include incomplete and ambiguous evaluation indices, lack of the criteria that ensure alignment between these assessments and ultimate aims of suggestions system, duality of criteria from quantitative and qualitative perspectives, how to score them, and ignorance of difference between organizations in terms of their size, organizational structure, missions, and so forth. The criteria of model and the scores allocated in the current system are as follows (see Table 1).

Table 1.

Criteria and scores related to each of them in the current assessment system.

Criteria of assessment model for suggestions system
Leadership (180 scores)	Training and synergy (60 scores)	Key results of performance (400 scores)	Processes (160 scores)
	Advertisements and notification (50 scores)
	Organizing (45 scores)
	Knowledge management and information system (45 scores)
	Motivation (60 scores)

Since assessment of organizations in the current model is made based on the scores allocated by assessors, conclusion of these views is not without mistake. Allocation of an interval of the values as the score considered for the related criteria might be a logical solution to encounter with this issue. In this state, some input and output data are specified in the form of numerical intervals; since the assessments are made by different assessors, several intervals are acquired for each input or output that must be entered into the assessment model (DEA model). In the first section of this research, all the criteria and subcriteria are listed and categorized, and they are reduced in number regardless of their significance. For this, the criteria related to the enablers are qualitative; the criteria related to the results that are quantitative were examined and summarized via two separate approaches. In both approaches, point of view of the experts, specialists, and practitioners of suggestions system was considered as the basis for decision-making. In this research, the statistical population consists of the practitioners in suggestions system in over 20 organizational complexes in one of the organizations affiliated to National Iranian Gas Company. To examine the criteria in the first group, two stages are pursued. In the first stage, the significance of criteria was put into question via Likert scale in a pooling, and the insignificant factors were omitted. In the second stage, the remaining criteria from the previous stage were categorized in a way that they get close to each other in terms of subject, and the number of criteria in each category ranges from 20 to 50 to pave the way for use of factor analysis.

Factor analysis

Factor analysis refers to the extension of analysis of major components. In these methods, an attempt is made to clarify the internal correlation between criteria via covariance matrix approximation and detect the major criteria. Since the final number of criteria has not been predetermined in this research, exploratory factor analysis has been firstly used and then the obtained results have been examined via confirmatory factor analysis method. To use factor analysis, significance of each of criteria obtained from the first stage was put into question in form of questionnaires. In each questionnaire, the number of participants was considered five times the number of criteria. Each person has specified his view on the significance of criteria as a value ranging from 0 to 100. Values 0, 10, 15, and 20–35 have less significance, values 40, 45, 50, and 55 have average significance, and values 60, 65, and 70–100 have high significance. In this research, software SPSS [version IBM SPSS Statistics 22] has been used for factor analysis, and Cronbach’s α has been used to determine the reliability of responses. The important point lies on the fact that the criteria that have caused reduction in reliability of pooling have been omitted by overview of the responses obtained during different stages. As mentioned above, the section screened from these criteria has been obtained and named in three categories with 23 indices (X ₁–X ₂₃), 32 indices (X ₁–X ₃₂), and 6 indices (X ₁–X ₃₂) after the first stage of research. With regard to the responses from the criteria in the first section and calculation of their reliability via software SPSS after omission of criteria X ₅ and X ₆, reliability was equal to 0.788, which indicates the average reliability. Therefore, 21 aforementioned criteria were used in the factor analysis. At this stage, 70 persons’ views were asked, that is, three times the number of variables. As a result of this action, univariate statistics including mean, standard deviation and early estimation of the commonalities, eigenvalue of correlation matrix of variables, total percentage of variance described by means of common factors as well as cumulative percentage of variance of factors were obtained. As a result of factor analysis, the aforementioned criteria were categorized in seven categories. Value of Kaiser-Meyer-Olkin (KMO) is greater than 0.7, indicating that the correlation between pair of variables can be determined by means of other variables; thus, the use of factor analysis can be justifiable. Results from Bartlett’s test of sphericity test indicate that there is a significant correlation between variables. At the next stage, the extracted factors were confirmed by some modifications using confirmatory factor analysis. The obtained results based on exploratory and confirmatory analyses for three-dimensional criteria above have been represented in Table 2.

Table 2.

Extraction of enabler factors based on factor analysis method.

CFA				EFA
Bartlett’s test of sphericity	KMO	Percentage of adjusted variance	Variables	Bartlett’s test of sphericity	KMO	Percentage of adjusted variance	Variables	Construct	Enabler criteria
0.000	0.623	64.427	X 13, X 14, X 17	0.000	0.718	74.254	X 13, X 14, X 17	F 11	First section
0.000	0.655	61.476	X 2, X 15, X 16				X 2, X 7, X 15, X 16	F 12
0.000	0.677	69.768	X 11, X 12, X 18				X 10, X 11, X 12, X 18	F 13
0.000	0.600	59.684	X 8, X 9				X 8, X 9	F 14
0.000	0.600	63.888	X 3, X 4				X 3, X 4	F 15
0.012	0.600	65.020	X 21, X 22				X 21, X 22	F 16
0.048	0.600	51.956	X 1, X 19, X 20				X 1, X 19, X 20	F 17
0.000	0.626	47.595	X 18, X 19, X 20, X 23, X 24	0.000	0.750	77.061	X 18, X 19, X 20 X 21, X 23, X 24, X 26	F 21	Second section
0.000	0.643	47.524	X 6, X 9, X 15				X 6, X 9, X 15 X 16, X 17	F 22
0.007	0.677	44.779	X 11, X 12, X 14				X 11, X 12, X 14, X 27	F 23
0.002	0.600	62.595	X 2, X 29				X 2, X 29, X 30	F 24
0.05	0.600	57.952	X 31, X 32				X 13, X 31, X 32	F 25
0.004	0.600	61.832	X 5, X 7				X 5, X 7	F 26
0.04	0.656	42.433	X 3, X 10, X 22				X 3, X 10, X 22, X 28	F 27
0.05	0.594	46.939	X 1, X 2, X 3	0.000	0.619	68.393	X 1, X 2, X 3	F 31	Third section
0.008	0.719	53.319	X 4, X 5, X 6				X 4, X 5, X 6	F 32

EFA: Exploratory Factor Analysis.

Analytic hierarchy process

The criteria related to the results are another important section of the assessment criteria of the units under study drawn into attention with a different approach in this research. This work has been pursued in steps:

Prepare a checklist of existing indices to assess suggestions system and analyze it based on empirical findings and experts’ and managers’ views in the organizations and propose views to complete and create analytic hierarchy process.

Weight the indices based on their significance in achievement of aims and expected results in recommendation system by asking the selected managers’ and experts’ views in form of analytical hierarchy process and assessment of organization in performing this system.

In the following, the criteria and indices that reflect success in achievement of favorable results are examined and specified in the form of a hierarchical diagram. Using the obtained weighted criteria for each of the 60 decision-making units under study in this research, a score was obtained and used as an output in the DEA.

Data envelopment analysis

DEA refers to a methodology based on a linear planning model to assess relative efficiency of decision-making units with the same inputs and outputs. Two basic DEA models include CCR and Banker, Charnes & Cooper (BCC) with return to fixed and varied scale. To any DMU_o, the highest score is allocated to possible efficiency by selecting optimal weights for each of inputs and outputs and considering restrictions of problem. If DMU _o obtained maximum value $E_o=1$ , the unit will be efficient; yet if $E_o<1$ , the unit will be inefficient, because another DMU obtains the highest score of efficiency by allocation of optimal weights for that unit. Accordingly, this model divides DMUs into two groups: efficient and inefficient units. CCR linear planning model specifies efficiency of n units with m inputs and s outputs as follows:

$E_o={\max }_{u_{r\prime }{v}_i}\sum _{r=1}^s{u}_r{y}_{ro}$

1

$\sum _{i=1}^m{v}_i{x}_{io}=1$

$\sum _{r=1}^s{u}_r{y}_{rj}-\sum _{i=1}^m{v}_r{x}_{ij}\le 0\left(j=1,2,\dots ,n\right)$

${\mathit{u}}_{\mathit{r}},{\mathit{v}}_{\mathit{i}}\ge ∊(r = 1,2,\dots s,i=1,2,\dots m)$

where v_i and u_r are the input and output weights assigned to the ith input and rth output, respectively, and ε > 0 is a non-Archimedean element smaller than any positive real number. In order to obtain the value of ε, we apply Mehrabian et al.’s ²⁷ model to this article, which is expressed as follows:

$Max{∊}_o$

$\sum _{i=1}^m{v}_i{x}_{io}=1$

2

$\sum _{i=1}^m{v}_i{x}_{ij} -\sum _{r=1}^s{u}_r{y}_{rj}\ge 0 (j=1,2,\dots ,n)$

${\mathit{u}}_{\mathit{r}},{\mathit{v}}_{\mathit{i}}\ge {∊}_o \left(r=1,2,\dots ,s ,i=1,2,\dots ,m \right)$

where ∊_o is a variable and ${∊}_o^{•}$ is an optimal solution for DMU _o , and ∊ = min{∊ ₁, ∊ ₂,…, ∊_n } is intersection of all ${∊}_o^{•}$ for the assessment of all DMUs.

Robust formulation for linear planning problems

Consider linear optimization problem as follows:

$\begin{array}{l}\text{Maximize }c\text{'}x\\ \text{Subject\hspace{0.17em}to }Ax\le b\\ 1\le x\le u\end{array}$

3

Consider ith row in matrix A and consider J_i as the set of coefficients in ith row which have uncertainty. Any input a_ij ; j ε J_i is modeled as a random symmetric and bounded variable (ã_ij ) proposed with [a_ij − â_ij, a_ij + â_ij ]. Corresponding to each uncertain ã_ij data, a random variable η_ij = (ã_ij − a_ij )/â_ij is defined which follows an unknown distribution and given with the values in interval [−1, 1]. In this regard, the robust formulation will be as follows:

$\begin{array}{l}\text{Maximize\hspace{1em}}c\text{'}x\\ \text{subject\hspace{0.17em}to}\\ \sum _j{a}_{ij}{x}_j+\sum _{j\epsilon J_i}{\widehat{a}}_{ij}{y}_j\le b_i\\ -y_j\le x_j\le y_i\\ 1\le x\le u\\ y\ge 0\end{array}$

4

Assuming x* as optimal response in above formulation, there will be y_j = |x*_j| under optimal conditions, thus

$\sum _j{a}_{ij}{x}^{*}{}_j+\sum _{j\epsilon J_i}{\widehat{a}}_{ij}| x^{*}{}_j|\le b_i\forall i$

Now, we will show that this response will be possible per each real value ã_ij of indefinite data

$\sum _j{\tilde{a}}_{\mathit{\text{ij}}}{x}^{*}{}_j =\sum _j{a}_{ij}{x}^{*}{}_j +\sum _{j\epsilon J_i}{\widehat{a}}_{ij}{\eta }_{ij}{x}^{*}{}_j \le \sum _j{a}_{ij}{x}_j + \sum _{j\epsilon J_i}{\widehat{a}}_{ij}\left|x^{*}{}_j\right| \le b_i\forall i$

It should be noted that η_ij = (ã_ij − a_ij )/â_ij , and thus a_ij +η_ij â_ij = ã_ij and ultimately

$\sum _j{\tilde{a}}_{\mathit{\text{ij}}}{x}^{*}{}_j = \sum _j{\eta }_{ij}{\widehat{a}}_{\mathit{\text{ij}}}{x}^{*}{}_j+\sum _j{a}_{ij}{x}^{*}{}_j$

Therefore

$\sum _j{\tilde{a}}_{\mathit{\text{ij}}}{x}^{*}{}_j = \sum _{j\epsilon J_i}{\eta }_{\mathit{\text{ij}}}{\widehat{a}}_{\mathit{\text{ij}}}{x}^{*}{}_j{}_{}+\sum _j{a}_{\mathit{\text{ij}}}{x}^{*}{}_j$

In row above, η_ij equals 0 per js that do not belong to J_i ; thus j ε J_i has been substituted. ${\sum }_{j\epsilon J_i}{\widehat{a}}_{\mathit{\text{ij}}}$ |x_j| defines the essential protection level for that restriction at interval ∑_j a_ij x*_j and b_i .

New approach in Bertsimas and Sim model ²⁸

In this model, a robust formulation is proposed which is linear, enabled to resist against uncertain parameters in the model under discussion regardless of a significant effect on objective function. Formulation of Bertsimas and Sim model is as follows:

Assume ith restriction of nominal problem as a_i ' X ≤ b_i . Assume J_i as the set of uncertain coefficients and assume these coefficients (ã_ij ; j ε J_i ) given with values in [a_ij − â_ij , a_ij + â_ij ], which have symmetric distribution with mean equal to nominal value (a_ij ). A parameter (Γ _i ) which is not integer is introduced per i which is given a value in [0,|J_i |]. The role of this parameter is to regulate robustness of Bertsimas’s method against conservatism level of response. In other words, it is unlikely intuition that changes all a_ij as much as possible. We can achieve our aim in the encounter with all the states in which the coefficients change and a coefficient a_it changes to the size (Γ _i – [Γ _i ]), that is, the only subset of the coefficients changes in a way with negative effect on response. In addition, robust response will be likely with high probability for the changes in more than [Γ _i ] variable. The result will be formulation of problem below, which is a nonlinear problem

$\text{Max\hspace{0.17em}\hspace{0.17em}}C\text{'}X$

5

$\begin{array}{l}\text{s.t.}\\ \sum _j{a}_{\mathit{\text{ij}}}{x}_j+{\text{max}}_{C_i}\left\{\sum _{j\epsilon S_i}\alpha {^}_{\mathit{\text{ij}}}{y}_j+\left({\Gamma }_i-\lceil {\Gamma }_i\rceil \right)\alpha ^{}_{i{t}_i}{y}_{i{t}_i}\right\}\le b_i,\forall i\end{array}$

$-y_j\le x_j\le y_j,\forall j$

$L\le X\le U$

$Y\ge 0$

where $C_i=\left\{S_i\cup \left\{t_i\right\}|S_i\subseteq J_i,\left|S_i\right|={\Gamma }_i,t_i\epsilon J_i-S_i\right\}$ .

If Γ _i is selected as integer, ith restriction will be as follows:

${\beta }_i\left(x,{\Gamma }_i\right)={\max }_{C_i}\left\{\sum _{j\epsilon S_i}\alpha {^}_{ij}\left|x_j\right|\right\},C_i=\left\{S_i|S_i\subseteq J_i,\left|S_i\right|={\Gamma }_i\right\}$

When Γ _i = 0, there will be β_i (x, Γ _i ) = 0, and the restriction will be corresponding to nominal problem. Further, if Γ _i = |J_i |, there will be swister’s method. Therefore, there will be the possibility to regulate robustness degree against different levels of conservatism in the response by changing this parameter in interval with the above two values. The below theorem is required to rewrite this model in the form of a linear model.

Theorem 1. Vector x* has been given.

${\beta }_i\left(x^{*},{\Gamma }_i\right)={\max }_{C_i}\left\{\sum _{j\epsilon S_i}\alpha {^}_{ij}\left|x^{*}{}_j\right|+\left({\Gamma }_i-\lceil {\Gamma }_i\rceil \right)\alpha {^}_{i{t}_i}\left|x^{*}{}_j\right|\right\}$

where

$C_i=\left\{S_i\cup \left\{t_i\right\}|S_i\subseteq J_i,\left|S_i\right|={\Gamma }_i,t_i\epsilon J_i-S_i\right\}$

The conservative function related to the ith restriction above is equivalent to the objective function of the linear planning problem below

${\beta }_i\left(x^{*},{\Gamma }_i\right)=\max \sum _{j\epsilon J_i}\alpha {^}_{ij}\left|x^{*}{}_j\right|z_{ij}$

6

$\begin{array}{l}\text{s.t.}\\ \sum _{j\epsilon J_i}{z}_{ij}\le {\Gamma }_i\end{array}$

$0\le z_{ij}\le 1\left(j\epsilon J_i\right)$

Now nonlinear model related to Bertsimas’s method can be represented as follows in the form of linear formulation:

$\text{Max\hspace{0.17em}}C\text{'}X$

7

$\begin{array}{l}\text{s.t.}\\ \sum _j{a}_{ij}{x}_j+z_i{\Gamma }_i+\sum _{j\epsilon S_i}{p}_{ij}\le b_i,\forall i\end{array}$

$z_i+p_{ij}\ge \alpha {^}_{ij}{y}_j,\forall i.j\epsilon J_i$

$-y_j\le x_j\le y_j,\forall j$

$l_j\le x_j\le u_j,\forall j$

$p_{ij}\ge 0,\forall i.j\epsilon J_i$

$y_j\ge 0,\forall j$

$z_i\ge 0,\forall i$

By considering the structure of robust formulation, it is clear that [Γ _i ] with coefficients a_ij changes per its bounds, and a coefficient with size (Γ _i – [Γ _i ]) â_it changes; thus, the problem remains unresolved. By proving the theorems below, it can indicate that the robust model is possible with high probability under uncertain data. Parameter Γ _i regulates the balance between the probability for negation and the result of objective function of nominal problem called with robustness.

Theorem 2. Assume x* as optimal response in formulation above and S*_i and t*_I as the indices corresponding to the maximum obtained value for β_i (x*, Γ _i ), hence it will be:

(a) The probability for negation of ith restriction is as follows:

$P\left(\sum _j{\tilde{a}}_{ij}{x}_j>b_i\right)\le P\left(\sum _{j\epsilon J_i}{\gamma }_{ij}{\eta }_{ij}\ge {\Gamma }_i\right)$

${\gamma }_{ij}=\{\begin{array}{c} \\ \begin{array}{l}1,\text{\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em} \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}if} j\epsilon S_i^{\text{*}}\\ \alpha {^}_{ij} \left|x{*}_j\right|/\alpha {^}_{i{r}^{\text{*}}} \left|x{*}_{r*}\right|,\text{\hspace{1em}\hspace{1em} if} j\epsilon J_i-S_i^{\text{*}} \end{array}\end{array}$

$r^{*}=\arg {\min }_r\alpha {^}_{ir}\left|x{*}_r\right|,r\epsilon S_i^{\text{*}}\cup \left\{t_i^{\text{*}}\right\}$

(b) Values γ _ij come true for all the values j ε J_i – S_i in γ_ij ≤ 1.

Theorem 3. If j ε J_i and η_ij are assumed as random symmetric and independent variables distributed in [−1,1].

$P\left(\sum _{j\epsilon J_i}{\gamma }_{ij}{\eta }_{ij}\ge {\text{Γ}}_i\right)\le \text{exp}\left(-\frac{{\text{Γ}}_i^2}{2\left|J_i\right|}\right)$

The proposed model and how to perform it

With regard to what is mentioned, there are 60 DMUs under study with a number of inputs and outputs. Inputs of problem for ith DMU have been displayed with X _1j , X _2j , and X _3j , and 16 outputs related to the enablers which are uncertain scores have been displayed with [Ỹ_ij ^L , Ỹ_ij ^U]. The 17th output represents score of results of performance which is calculated via hierarchical method in the form of a certain value (see Figure 1).

Figure 1.

The diagram representing inputs and outputs to assess suggestions system.

This model can be explained using European Foundation for Quality Management (EFQM) model. In assessments, the elements that contributed to better performance of this system in the organizations are taken into consideration in addition to the criteria considered with EFQM model. In the proposed model, in this research, the weights corresponding to the advantages of each of organizations mentioned as the features of DEA method are used instead of the use of fixed scores for each group of defined factors and criteria, which is mentioned as the most important fault of the current assessment system. The criteria related to three above enablers are assessed by different experts. Each of the above criteria includes several subcriteria to which a value ranging from 0 to 100 is given for trust on assessments. Then the mentioned values are integrated and combined with view of other experts, for example, assume five persons engage in assessment of jth unit (DMU _j ), these five persons represent their views considering the defined scale in the diagram for factor F ₁₁ which defines the practical actions of leadership to progress the aims of suggestions system (see Table 3).

Table 3.

How to get output data based on views of experts, in the proposed model.

With regard to the assessments by five persons, it can consider the obtained result in form of an interval with uncertain bounds [ỹ_rj ^L , ỹ_rj ^U ], in which

${\tilde{\text{y}}}_{rj}{}^L = y_{rj}{{}^L}^{ }\pm {ỷ}_{rj}{}^L,{\tilde{\text{y}}}_{rj}{}^U = y_{rj}{{}^U}^{ }\pm {ỷ}_{rj}{}^U$

In terms above, y_ij ^L and y_ij ^U are called with nominal values in uncertain bounds in above interval, and ỷ _ij ^L and ỷ _ij ^U are considered as their range of variations:

y_ij ^L denotes mean of low bounds of assessors’ views/number of assessors.

y_ij ^U denotes mean of up bounds of assessors’ views/number of assessors.

The probability for negation of restrictions in output-based CCR model by considering different values for Γ is shown with instance, for example, it equals 0.42, 0.32, 0.22, and 0.13 for values 5, 6, 7, and 8, respectively. Therefore, it seems that Γ _j = 6 can be a suitable value for this problem

${\Gamma }_j = 6\to \text{exp}\left(-{\Gamma }_j^2/\left|J_j\right|\right)=\text{exp }\left(-1.5\right) = 0.32$

Now, we present the mathematical details of the RDEA model proposed in this article. Let us consider the DMU _j and assume that J_j is the index set of the imprecise output values. Let us further consider parameter Γ _j , not necessarily integer, that assume value in the bounded interval [0, $\left|J_j\right|]$ , where $\left|J_j\right|$ is the cardinal of J_j . The role of this parameter is to adjust the robustness of the proposed model against the conservatism level of the solution. Indeed, it is unlikely that all of the imprecise outputs will change simultaneously. In other words, we stipulate that only a subset of the output data should change to affect the solution that was defined by Bertsimas and Sim ²⁹ . Then, considering the robust optimization approach, we change it as follows:

$\text{Max\hspace{0.17em}}{E}_o = \sum _{r=1}^s{u}_r{y}_{ro}-{\beta }_o\left(y,{\text{Γ}}_o\right)$

$\begin{array}{l}\text{s.t.}\\ \sum _{i=1}^m{v}_i{x}_{ij}+{\beta }_o\left(y,{\text{Γ}}_o\right)=1\end{array}$

8

$\sum _{r=1}^s{u}_r{y}_{rj}-\sum _{i=1}^m{v}_i{x}_{ij}+{\beta }_j\left(y,{\text{Γ}}_j\right)\le 0\left(j\ne o\right)$

$E_o\le 1$

$u_r,v_i\ge ∊$

We use the robust optimization approach to introduce β_j (y, Γ _j ) to move from the optimistic to the pessimistic viewpoint. In other words, these variables protect the constraints against data uncertainty and keep them feasible

${\beta }_j\left(y,{\Gamma }_j\right)={\max }_{C_j}\left\{\sum _{r\epsilon S_j}{u}_r{ỷ}_{rj}+\left({\text{Γ}}_j-\lceil {\text{Γ}}_j\rceil \right){ỷ}_{r{t}_j}\right\},C_j=\left\{S_j\cup \left\{t_j\right\}|S_j\subseteq J_j,\left|S_j\right|=\Gamma j,t_j\epsilon J_j-S_j\right\}$

Now, with regard to our approach, the defined robust model obtains up limit of efficiency for each of organizations . The difference is that, according to the approach mentioned, here all units are considered in the best condition. If we consider y_ro = y_ro ^U , y_rj = y_rj ^U , and ỷ_rj = ỷ_rj ^U , then model above is solved by considering the following form, which is used to calculate the efficiency and rank of DMUs:

${\text{Max}}_{u_r,v_i} E_o=\sum _{r=1}^s{u}_r{y}_{ro}^U-p_o{\text{Γ}}_j-\sum _{r=1}^s{q}_{ro}$

$\sum _{i=1}^m{v}_i{x}_{io}=1$

9

$\sum _{i=1}^m{v}_i{x}_{ij}-\sum _{r=1}^s{u}_r{y}_{rj}^U-p_j{\text{Γ}}_j-\sum _{r=1}^s{q}_{rj}\ge 0$

$p_j+q_{rj}\ge u_r{ỷ}_{rj}^U$

$E_o\le 1$

$p_j,q_{rj}\ge 0$

$u_r,v_i\ge ∊\left(r=1,2,\dots ,s,i=1,2,\dots ,m\right)$

We should note that the level of conservatism in the output data has to be discretely determined by the decision maker (or expert). In this research, we assumed that all Γ _j are equal to Γ. As a result, we formulate the following nonlinear model:

$\underset{u_r,v_i}{\text{Max\hspace{1em} }}{E}_o=\sum _{r=1}^s{u}_r{y}_{ro}^U-p_o\text{Γ}-\sum _{r=1}^s{q}_{ro}$

$\sum _{i=1}^m{v}_i{x}_{io}=1$

10

$\sum _{i=1}^m{v}_i{x}_{ij}-\sum _{r=1}^s{u}_r{y}_{rj}^U-p_j\text{Γ}-\sum _{r=1}^s{q}_{rj}\ge 0$

$p_j+q_{rj}\ge u_r{ỷ}_{rj}^U$

$E_o\le 1$

$p_j,q_{rj}\ge 0$

$u_r,v_i\ge ∊\left(r=1,2,\dots ,s,i=1,2,\dots ,m\right)$

Research findings and results

Now we follow our approach to solving model (10). This model is solved for different combinations of Γs, and values of weights and rankings for each DMU are saved using the obtained values for E. The DMU efficiencies obtained for each gamma through the RDEA model may result in different rankings of the DMUs (see Table 4).

Table 4.

The efficiency of some DMUs for different gamma levels.

Gamma	DMU6	DMU7	DMU13	DMU18	DMU31	DMU39	DMU45	DMU52	DMU55	DMU60
0	1	0.9924	1	1	0.9639	1	1	1	1	1
1	0.9796	0.9424	0.9948	1	0.9175	0.995	1	0.9423	0.9848	1
2	0.9787	0.9224	0.9771	0.9833	0.886	0.9659	1	0.9229	0.9529	0.9928
4	0.9774	0.9145	0.9505	0.9822	0.8789	0.9646	0.9968	0.9037	0.9285	0.9913
6	0.9762	0.9086	0.9372	0.9813	0.8775	0.9635	0.992	0.9026	0.9271	0.9901
8	0.9751	0.9076	0.9361	0.9804	0.8763	0.9625	0.9908	0.9016	0.9259	0.989
10	0.974	0.9068	0.935	0.9796	0.8752	0.9616	0.9897	0.9009	0.9249	0.988
12	0.9732	0.9061	0.9342	0.9789	0.8742	0.9608	0.9887	0.9002	0.9239	0.9871
14	0.9724	0.9054	0.9333	0.9783	0.8732	0.9601	0.9878	0.8995	0.9231	0.9863
15	0.972	0.9051	0.9329	0.978	0.8727	0.9598	0.9874	0.8992	0.9227	0.9859
16	0.9717	0.9049	0.9327	0.9778	0.8724	0.9595	0.9871	0.899	0.9223	0.9856

In such cases, many analysts allow a decision maker to use his or her preferences in selecting a suitable gamma (see Figure 2). We utilize a graphical presentation of the results enhanced with a Monte Carlo simulation to provide additional insight for making a final decision about gamma and the overall rankings of the DMUs. For each input and output of DMUs, a number is randomly generated using Monte Carlo simulation and by the resulting weights from ${\sum }_{r=1}^s{u}_{r}y{U}_{ro}/{\sum }_{i=1}^m{v}_i{x}_{io}$ . Efficiencies of DMUs are recalculated and ranked again. The conformity value between the rankings resulting from the mathematical model and the simulations are computed and analyzed. This results in two overall rankings for the DMUs: one from the mathematical model and another from the simulation. For example, if DMU 1 is ranked first in both rankings from model 1 and simulation, we assign a 1 to this simulation run; otherwise, we assign a 0. We repeat this procedure for each gamma 1000 times and calculate the percentage of conformity for each DMU and for each gamma. The conformities values are presented in Figure 3. As is shown in the figure, the maximum conformity occurs in Γ = 6.Therefore, we can conclude that the results based on the upper and lower bounds of efficiency do not necessarily have maximum conformity with reality, but specific values of Γ can maximize conformity, and thus, more authentic final rankings for the DMUs in this interval of Γ may be expected (see Table 5).

Figure 2.

The efficiencies of some DMUs for different Γs.

Figure 3.

The average conformity between the ranking of the DMUs using model (10) and the simulation runs for different gamma values.

Table 5.

The average conformity for different gamma values.

Gamma	0	1	2	4	6	8	10	12	14	16
Conformity	0.397	0.68	0.742	0.739	0.752	0.732	0.735	0.742	0.743	0.737

Taking into account the data related to the research and implementation of model IDEA, for each DMU, two upper and lower limit values are achieved. The average of these two values is calculated, and then, the units are ranked based on these obtained values. The results obtained for the first 16 units, and also, the ranking of units obtained with implementation of RDEA model, are shown in Tables 6 and 7.

Table 6.

Average of upper and lower efficiency using IDEA and RDEA.

	Using IDEA (average of upper and lower efficiency)	Using RDEA for Γ = 6	Using RDEA for Γ = 7	Using RDEA for Γ = 8
DMU16	1	1	1	1
DMU20	1	1	1	1
DMU21	1	0.977462501	0.977205337	0.976959419
DMU33	1	1	1	1
DMU40	1	1	1	1
DMU41	1	1	1	1
DMU44	1	1	1	1
DMU53	1	0.997497729	0.997221368	0.996979477
DMU9	0.999922774	0.944704379	0.944222539	0.943971936
DMU39	0.994562246	0.960168342	0.95993351	0.959713256
DMU45	0.992763874	0.962687177	0.962434901	0.96218967
DMU18	0.980679025	0.953338794	0.953136727	0.952940402
DMU60	0.96133913	0.92654416	0.92627313	0.926032007
DMU38	0.946055566	0.944765786	0.944588196	0.944425017
DMU27	0.942304906	0.907485505	0.907034524	0.906597827
DMU13	0.940804848	0.938155977	0.937916004	0.937681215

IDEA: interval data envelopment analysis; RDEA: robust data envelopment analysis; DEA: data envelopment analysis.

Table 7.

Ranking of the first 16 units, using IDEA and RDEA.

DMUs	Using IDEA (average of upper and lower efficiency)	Using RDEA for Γ = 6	Using RDEA for Γ = 7	Using RDEA for Γ = 8
DMU20	1	1	1	1
DMU21	1	3	3	3
DMU33	1	1	1	1
DMU40	1	1	1	1
DMU41	1	1	1	1
DMU44	1	1	1	1
DMU53	1	2	2	2
DMU9	2	8	8	8
DMU39	3	5	5	5
DMU45	4	4	4	4
DMU18	5	6	6	6
DMU60	6	10	10	10
DMU38	7	7	7	7
DMU27	8	11	11	11
DMU13	9	9	9	9

IDEA: interval data envelopment analysis; RDEA: robust data envelopment analysis; DEA: data envelopment analysis.

As can be seen from the above table, by implementing RDEA and increasing the gamma value, power detection and differentiation of the model increase. In the above table, the best results have occurred on Γ = 6. With regard to the amount of gamma, units 16, 21, 33, 40, 41, and 44 are efficient, and other units have been ranked. As can be seen in Figure 4, the ranking of the units for Γ = 6 and greater than 6 was similar, and thus can be concluded that in this research, the results for Γ = 6 and greater than 6 are stable.

Figure 4.

Ranking of the first 16 units, using IDEA and RDEA. IDEA: interval data envelopment analysis; RDEA: robust data envelopment analysis.

Conclusions

In this study, a pattern has been proposed to evaluate the efficiency of organizations in performing suggestions system using RDEA model with the following features:

Two categories of input and output data are detected in the assessment of suggestions system based on the proposed criteria grounded on excellence approach.

There are 60 DMUs under study, with 3 inputs and 17 outputs. Inputs and one of outputs are in form of certain values, and 16 outputs are related to the enablers which are in form of intervals with uncertain bounds.

We deal with a type of interval data that their limits are indefinite.

In the conventional DEA, all the data assume the form of specific numerical values. However, the values of some output data in this study are imprecise. The imprecise or vague data in the DEA models have been examined in the literature in different ways. The imputation methods to estimate the approximations of the imprecise values, the exclusion of the units with imprecise values from the analysis, and the stochastic approach are among the methods most commonly used to model uncertainty in the DEA literature. Recently, the IDEA and the fuzzy DEA are used to deal with the imprecise data in DEA. The contribution of this study is threefold: (1) We consider uncertain and imprecise output data and implement the proposed framework in the real-world (to evaluate efficiency of organizations in performing suggestions system); (2) we propose a robust optimization DEA model in which the output parameters are in form of intervals with uncertain bounds; and (3) we use a method of simulation to specify a range of gamma in which the rankings of the DMUs occur with high probability and then compute the conformity of the rankings resulting from the mathematical model with reality.

Moreover, the use of the proposed model for assessment of suggestions system has useful elements and components such as the possibility to consider views of experts, comprehensiveness of indices in the enablers section and results section, adjustment with EFQM system and its criteria and concepts, and lack of existing conflicts in the current assessment system. The important aim that is pursued in the assessment of suggestions system in organizations is the survey for omitting deficiencies of organization by having successful models than the rest of organizations. An issue under discussion is emphasized in the assessment logic with excellence model, for which no certain executive mechanism has been predicted. The present research indicates that the used model paves the way for such possibility for inefficient units. Detection of reference units in DEA specifies the most accessible models for each of the organizations. An interesting point lies on the fact that most of the organizations especially public organizations fail to select values for input variables such as budget, number of staffs, and/or education level of staffs, and enjoy restricted authorities in this context; thus, they introduce the existing difference in these elements compared to the rest of organizations in selection of the most superior suggestions system. Assessment logic in the current assessment system does not have the possibility for transparency in this context. Yet, the use of input-based models of DEA raises such doubts and clarifies difference in each inefficient organization with reference organizations by comparing outputs. Another point lies on existing flexibility in the proposed model. Parameter Γ in the proposed model provides this flexibility. On the one hand, increasing value of Γ can raise sufficient trust to consider all the assessments by the assessors. It seems that we can use this parameter as a regulating parameter to build balance in this context.