Sage Journals: Discover world-class research

Abstract

This study aims to analyze the research performance of 38 OECD countries in the last 10 years. For this purpose, the research performance of these countries was analyzed using multi-criteria decision-making (MCDM) methods with the productivity and publication impact data of 38 countries. The analyses were conducted both normal data and also with adjusted data based on the population and income level of the countries. Firstly, the importance level of research performance criteria was determined by three different MCDM methods. Using the weight values obtained from these methods, countries were ranked by three different MCDM methods used for ranking purposes. According to the results of the study, the publications in the highest cited 1% were determined as the most important and distinctive criterion. Secondly, the criterion expressing the relative impact of publications on the world was determined as the second most important criterion. In the analyses conducted without normalizing the data, USA showed the highest research performance. In the analyses conducted by considering the population of the countries, Switzerland showed the highest research performance. In the analyses conducted by taking into account the income level of the countries, Estonia has the highest research performance. As R&D expenditures, development level of countries and international collaboration increased, research performance of countries increased. However, with the increase in domestic collaboration, there has been a decrease in research performance. In addition, this research has shown how to use MCDM in the measurement of research performance by using objective methods with different backgrounds.

Plain language summary

Analysis of research performance of OECD countries

This study analyses the research performance of 38 OECD countries over the last 10 years. For this purpose, productivity and publication impact data of 38 countries obtained from the Web of Science database for the last 10 years and the research performance of these countries were analysed using multi-criteria decision making (MCDM) methods. MCDM is a set of methods used in solving problems with multiple criteria and alternatives. These methods can determine the level of importance of criteria and allow decision makers to select, rank, and sort alternatives. Analyses were conducted using both normal and adjusted data for the population and income levels of the countries. First, the importance level of the research performance criteria was determined using three different MCDM methods. Using the weight values obtained from these methods, the countries were ranked using three different MCDM methods. According to the results of the study, the most cited publications in the top 1% were determined to be the most important and distinguishing criterion. Second, the criterion expressing the relative impact of publications on the world was determined to be the second most important criterion. In the analyses conducted without normalizing the data, the USA showed the highest research performance. In the analyses conducted by considering the population of the countries, Switzerland showed the highest research performance. Estonia has the highest research performance in the analyses based on the income level of the countries. As R&D expenditures, development level of countries, and international co-operation increased, research performance of countries also increased. However, as domestic collaboration increased, research performance declined. In addition, this research has shown how MCDM can be used to measure research performance using objective methods with different backgrounds.

Keywords

research performance multi-criteria decision making MCDM performance OECD country

Introduction

Although there is no agreed-upon definition of research performance conceptually, there have been some studies conducted on this concept (Bazeley, 2010; Edgar & Geare, 2013). Research performance consists of two basic components. The first one is research, and the second one is performance. Research includes activities that can be considered as the foundation of performance. Performance, on the other hand, involves sharing what is learned through research with the scientific community in order to become visible. Conceptually, research performance has traditionally been based on publication outputs in various forms, citations as a measure of impact, expert assessments as a measure of the quality of work, and indicators of the reputation of researchers (Bazeley, 2010). In line with the literature, research performance refers to the evaluation of the productivity and impact of research outputs at various levels, including individual researchers, research-related institutions like universities and research centers, and even countries.

Research performance has been frequently examined in recent years. In the literature, it is evident that numerous studies have been conducted on the research performance of higher education institutions, universities, research centers, faculties, researchers, and countries. There are several reasons to examine research performance.

First, some countries rely heavily on the research performance of higher education institutions (Bazeley, 2010). Therefore, government grants to higher education institutions with high research performance increase (Geuna & Martin, 2003). This results in increased efforts by higher education institutions to improve research performance. Second, as higher education institutions face financial constraints, there is an increased demand for productivity and accountability from researchers and higher education institutions (Law & Chon, 2007). Research performance is analyzed based on higher education institutions and countries. In terms of countries, one reason for measuring research performance is that international ranking organizations focus on research performance. Ranking organizations such as THE, ARWU, and QS rank universities and countries according to certain performance criteria. Countries that are highly ranked, especially in the category of top universities, are in a more advantageous position to attract qualified students (Bowman & Bastedo, 2009), positively affecting the reputation of the institution and thus that of the country (Bastedo & Bowman, 2010). A strong academic reputation brings about social and economic advantages for both the university and the region (Fritsch & Slavtchev, 2007; Pressman et al., 1995). In addition, research performance has been shown to impact critical activities such as organizational missions, decision-making, employment, and strategy (Hazelkorn, 2007, 2008). All these factors have led universities and researchers to strive for greater productivity (Law & Chon, 2007).

In recent years, there has been a growing interest in improving research performance (Abdul-Majeed et al., 2021; Barney et al., 2022; Preut et al., 2022). This growing interest in research performance is evident in the literature. Studies have been conducted to examine research productivity and performance for various purposes, including academics (Horta et al., 2020), universities (Abdul-Majeed et al., 2021), and faculties (Khalid & Tadesse, 2022). In addition, studies have been conducted on the performance of countries focused on research in specific fields such as health (Mijumbi-Deve et al., 2021), computer science (Guan & Ma, 2004), cloud computing (Awan & Abbas, 2023), tourism (Cardoso et al., 2020), and physics (Zhang et al., 2017). However, there are few studies in the literature that address research performance holistically, based on cross-country comparisons, rather than focusing on specific fields or topics. Sedláček (2017) examined the impact of governance on the research performance of European countries. Adeyemo (2019) analyzed the research performance of ASEAN (Association of Southeast Asian Nations) countries. However, there is a lack of comprehensive and comparative studies on the research performance of countries in the literature.

One of the important issues related to research performance is the methods by which it will be measured. In a number of studies, research performance is analyzed using descriptive statistical methods (Brocato & Mavis, 2005; Holliday et al., 2014; Khan et al., 2014; Saxena et al., 2022). Moreover, research performance is measured using various methods such as data envelopment analysis (Hung & Chou, 2013; Mutz et al., 2017) and bibliometric methods (Cardoso et al., 2020).

In this study, research performance using with multi-criteria decision making (MCDM) methods, which are differ from the methods used in existing research. MCDM is a part of operations research that is used to classify, categorize, and rank alternatives. MCDM is a concept that encompasses all methods that help decision makers in dealing with multiple conflicting criteria (Ho, 2008). MCDM methods have been developed to assist decision makers in the decision process. These methods provide some techniques to find solutions to existing problems. MCDM places decision makers at the center of the decision making process (Ishizaka & Nemery, 2013). These methods, techniques and approaches improve the quality of decisions by making them more clear, efficient and rational. MCDM provides a method to tackle complex problems by breaking problems into smaller parts (Mardani et al., 2015). Many techniques have been developed in recent years for solving multi-criteria decision problems and this development is still continuing (Wallenius et al. 2008). This development is due to the need to solve different types of problems encountered (Ishizaka & Nemery, 2013). In other words, a large number of MCDM methods have been developed over the years, each with a different theoretical background, able to answer different types of questions and produce different results. MCDM is one of the most widely used decision methods in different disciplines. These methods are used in many fields such as energy, environment, economy (Mardani et al., 2015).

MCDM methods are used to solve real-world problems with multiple criteria and alternatives. MCDM methods have the capacity to evaluate multiple and even conflicting criteria. MCDM can statistically weight the criteria based on a theoretical framework instead of assuming them to be of equal importance or arbitrarily setting an importance level. It can evaluate more than one alternative by considering various criteria. Measurement of research performance is a real problem with multiple criteria and alternatives in accordance with the nature of MCDM. When evaluating the research performance of researchers, institutions, universities, and countries, many criteria such as the number of publications and the impact should be considered. Moreover, if multiple researchers, institutions, universities, or countries are considered, MCDM is capable of ranking, selecting, and sorting these different alternatives. Therefore, the problem addressed by this research is suitable for the nature of MCDM since there are both different performance criteria and the measurement of research performance across multiple countries.

This study aims to examine the research performance of 38 OECD countries over the past 10 years using multiple MCDM methods with different theoretical backgrounds to measure the research performance of countries, rank them accordingly and test the relationship between research performance and various variables. There are two main reasons for analyzing OECD countries in this study. The first reason is that the OECD includes both developed and developing countries from various continents. This has provided new information on research performance for all OECD member countries. The second reason is the availability of comparable data for OECD countries. Therefore, the following three research questions are posed and answers are sought:

(1) What are the criteria weights for research performance?

(2) What are countries in terms of research performance?

(3) Is there a relationship between research performance ranking and a country’s level of development, R&D expenditures, international collaboration rate, and domestic collaboration rate?

This study analyses the research performance of OECD countries and also examines the relationship between research performance and certain variables. Previous studies have shown that research performance increases with an increase in R&D expenditures (Gulbrandsen & Smeby, 2005; Lee, 2021; Meo et al., 2013; Solarin & Yen, 2016; Teitel, 1994; Uyar et al., 2022). Another variable related to research performance is the level of development of countries. The level of development generally refers to the economic development of countries. Previous studies have revealed that developed countries invest more in research and development activities, which has a positive impact on publication outputs (Gonzalez-Brambila et al., 2016; Man et al., 2004). This relationship between economic development and scientific productivity growth has also been proven by other studies (Cimini et al., 2014; Docampo & Bessoule, 2019; Mueller, 2016; Rahman & Fukui, 2003). Another variable associated with research performance is research collaboration. Previous studies have shown that increased international collaboration positively affects research performance (Lee & Bozeman, 2005). In addition, collaboration with researchers in different countries is positively related to the number of citations, which is an indicator of the impact of publications (Aksnes, 2003; Lancho Barrantes et al., 2012). In contrast, an increase in domestic collaboration decreases research productivity and publication impact (Goldfinch et al., 2003). This study also strengthened the existing evidence by revealing how these variables are related to research performance.

This research provides a new perspective on research performance. First, considering that there are many concepts affecting research performance, assessing research performance without considering characteristics such as the level of development and population of countries may not be accurate for countries with different sizes in many respects. Second, evaluations made with different criteria and especially without considering the importance of these criteria may lead to biased results. Finally, evaluating research performance is a real-life problem in line with the main principle of MCDM. Addressing this problem using different MCDM methods, which have their own characteristics, may allow a multidimensional and more reliable evaluation of research performance. This study provides an opportunity to consider research performance from a broad and different perspective.

This research has potential contributions, especially for researchers and policy makers. This research analyzed research performance using different MCDM methods. This may enable researchers to examine research performance from a different perspective. Second, using the methods applied in this study, researchers can analyse research institutions, academic units, universities, and countries. Finally, the methodology presented in this study is not a single and unique methodology. Apart from these methods, research performance can be analyzed using different MCDM methods that achieve the same purpose. As a result, this study will provide researchers with a new methodological perspective. This study analyses the research performance of a comprehensive group of countries using a rich methodology. The results of this study provide important information to help policymakers in policy making on research performance. First, this study analyses the research performance position of a country compared with other countries from different perspectives. Policy makers can rethink how to allocate expenditure on performance more effectively. It can also help countries produce evidence-based policies to improve competitiveness. New strategies can be developed to improve countries’ research performances. In addition, each country can analyse the current situation by taking a closer look at the research performance of higher education institutions and develop new strategies. The performance results of international ranking organizations ignore the size of the countries. This research addresses these important criteria and presents new results specific to the situation of countries.

Method

In this study, MCDM methods were used to evaluate the research performance of countries. MCDM originated as part of operations research, which uses mathematical tools to assist decision makers (Zavadskas et al., 2014).

The implementation model for how MCDM methods will be used in this research is shown in Figure 1. First, the importance level, that is, the weight values, of the research performance criteria were obtained. For this, three different objective MCDM methods were used: (1) Criteria Importance Through Intercriteria Correlation (CRITIC), (2) Method Based on the Removal Effects of Criteria (MEREC), and (3) Entropy. Objective MCDM methods are not based on expert opinions and are used when data on criteria and alternatives are available. After the weight values of the performance criteria were determined, these weight values were included in the analysis with three different MCDM methods used for ranking purposes, and the countries were ranked. The ranking methods used are Additive Ratio Assessment (ARAS), Multi-Attributive Border Approximation Area Comparison (MABAC), and The Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS). The values obtained from the three different weighting methods were analyzed using these ranking methods to obtain nine ranking values for a country. To obtain a single value from the different ranking values, the final ranking of the countries was determined using the Borda function. Considering that there are numerous MCDM methods with different theoretical and statistical backgrounds and that evaluating research performance through a single method may lead to biased results, six MCDM methods were used for weighting and ranking. Another benefit of using more than one method is the diversity of methods, which allows for comprehensive and verifiable analysis.

Figure 1.

MCDM implementation model.

After the research performance rankings of the countries were obtained, Spearman’s rank correlation analysis was conducted to determine whether there is a relationship between the research performance rankings and the development level, R&D expenditures, international collaboration rate, and domestic collaboration rate of the countries. The methods used are detailed in the following section.

CRITIC Method

CRITIC is an MCDM method developed by Diakoulaki et al. (1995) used for objective weighting of criteria. The CRITIC method examines the relationship between criteria and uses correlations between criteria. The implementation steps of this method are as follows (Alinezhad & Khalili, 2019):

Step 1: Normalization of the Decision Matrix

The decision matrix in the CRITIC method is as follows:

{[\begin{matrix} \begin{matrix} r_{11} & \dots & r_{1 j} \\ ⋮ & ⋱ & ⋮ \end{matrix} \begin{matrix} \dots & r_{1 n} \\ ⋱ & ⋮ \end{matrix} \\ \begin{matrix} r_{i 1} & \dots & r_{ij} \\ ⋮ & ⋱ & ⋮ \\ r_{n 1} & \dots & r_{ni} \end{matrix} \begin{matrix} \dots & r_{in} \\ ⋱ & ⋮ \\ \dots & r_{nn} \end{matrix} \end{matrix}]}_{mxn}; i = 1, \dots, m, j = 1, \dots n

The following equations are used to normalize the positive and negative values of the decision matrix, respectively.

x_{ij} = \frac{r_{ij} - r_{i}^{-}}{r_{i}^{+} - r_{i}^{-}}; i = 1, \dots, m, j = 1, \dots n

Where x_ij represents the normalized value of the decision matrix for the jth attribute of each ith alternative, r_i⁺ = max (r₁, r₂, …, r_m), and r_i⁻= min (r₁, r₂, …, r_m).

Step 2: Determination of Correlation Coefficient

The correlation coefficient between attributes is found with the help of the following equation:

p_{jk} = \sum_{i = 1}^{m} (x_{ij} - {\bar{x}}_{j}) - (x_{ik} - {\bar{x}}_{k}) / \sqrt{\sum_{i = 1}^{m} {(x_{ij} - {\bar{x}}_{j})}^{2} \sum_{i = 1}^{m} {(x_{ik} - {\bar{x}}_{k})}^{2}}

where $\bar{x}$ j and $\bar{x}$ k indicate the average of the jth and kth attributes. $\bar{x}$ j is calculated by the following equation. Similarly, the same calculation is made for $\bar{x}$ k. Furthermore, ρ_jk denotes the correlation coefficient between the jth and kth attributes.

{\bar{x}}_{j} = \frac{1}{n} \sum_{j = 1}^{n} (x_{ij}); i = 1, \dots, m

Step 3: Determination of C Index

The standard deviation of each attribute is estimated by the first equation below. The C index is then calculated with the second equation.

σ = \sqrt{\frac{1}{n - 1} \sum_{j = 1}^{n} {(x_{ij} - {\bar{x}}_{j})}^{2}}; i = 1, \dots, m

C_{j} = σ_{j} \sum_{j = 1}^{n} (1 - p_{jk}); j = 1, \dots, n

Step 4: Determination of the Weight of Attributes

The weights of the attributes are calculated with the following equation:

w_{j} = \frac{C_{j}}{\sum_{j = 1}^{n} C_{j}}; j = 1, \dots, n

Step 5: Determination of Final Weighting of Attributes

The weights of the attributes are obtained by sorting them in descending order.

MEREC Method

MEREC is an MCDM method developed by Keshavarz-Ghorabaee et al. (2021). This method is an MCDM method used for objective weighting of criteria. The CRITIC method is based on the removal effect of each criterion on the overall performance of the alternatives to determine the importance of the criteria. The implementation steps of this method are as follows (Keshavarz-Ghorabaee et al., 2021):

Step 1: Creating the Decision Matrix

A decision matrix is created that shows the rating of each alternative or the values for each criterion.

X = [\begin{matrix} \begin{matrix} x_{11} & \dots & x_{1 j} \\ ⋮ & ⋱ & ⋮ \end{matrix} \begin{matrix} \dots & x_{1 m} \\ ⋱ & ⋮ \end{matrix} \\ \begin{matrix} x_{21} & \dots & r_{2 j} \\ ⋮ & ⋱ & ⋮ \\ x_{n 1} & \dots & x_{nj} \end{matrix} \begin{matrix} \dots & x_{2 m} \\ ⋱ & ⋮ \\ \dots & x_{nm} \end{matrix} \end{matrix}]

Step 2: Normalization of the Decision Matrix

A simple linear normalization is used to scale the elements of the decision matrix. The following equation is used for normalization.

n_{ij}^{x} = {\begin{matrix} \frac{\begin{matrix} \min x_{kj} \\ k \end{matrix}}{x_{ij}} ifj \in B \\ \frac{\begin{matrix} \min x_{kj} \\ k \end{matrix}}{x_{ij}} ifj \in H \end{matrix}

Step 3: Calculation of Overall Performance of Alternatives (S_i)

A logarithmic measure with equal criteria weights is applied to calculate the overall performance of the alternatives. The following equation is used for this.

S_{i} = \ln (1 + (\frac{1}{m} \sum_{j} | \ln (n_{ij}^{x} |))

Step 4: Calculating the Performance of Alternatives

The performance of the alternatives is calculated by removing each criterion. The performance of the alternatives is calculated as follows:

S_{i} = \ln (1 + (\frac{1}{m} \sum_{k, k / j} | \ln (n_{ij}^{x} |))

Step 5: Calculating the Sum of Absolute Deviations

In this step, the removal effects of the jth criterion are calculated based on the values obtained in the third and fourth steps.

E_{j} = \sum_{i} | S_{ij}^{’} - S_{i} |

Step 6: Determination of the Final Weight of the Criteria

The objective weight of each criterion is calculated using the lifting effects from step five. The value w_j denotes the weight of the jth criterion.

w_{j = \frac{E_{j}}{\sum_{k} E_{k}}}

Entropy Method

Entropy is an MCDM method used for objective weighting of criteria. This method uses the magnitude of the entropy value and calculates the degree to which the criteria convey decision information, revealing the relative weight of the criteria. The implementation steps of the entropy method are as follows (Chen, 2020):

Step 1: Normalization of the Decision Matrix

M alternatives and n criteria are organized for the initial matrix of the entropy weight evaluation.

X = [\begin{matrix} x_{11} & x_{12} & \dots & x_{1 n} \\ x_{21} & x_{22} & \dots & x_{2 n} \\ ⋮ & ⋮ & ⋮ \\ x_{m 1} & x_{m 2} & \dots & x_{mn} \end{matrix} = (X_{1} X_{2} \dots X_{n})]

Due to the differences between units, each criterion needs to be standardized to eliminate the influence of different units on the evaluation results. At this stage, the conversion method is applied with the following formulas.

x_{ij}^{’} = \frac{x_{ij} - \min {x_{ij}}}{ma x_{i} {x_{ij}} - mi n_{i} {x_{ij}}}

x_{j}^{’} = \frac{x_{ij} - \min {x_{ij}}}{ma x_{i} {X_{j}} - mi n_{i} {X_{j}}} (applicable benefit indicators)

= \frac{ma x_{i} {x_{ij}} - x_{ij}}{ma x_{i} {x_{ij}} - mi n_{i} {x_{ij}}}

x_{j}^{'} = \frac{m a x_{i} {x_{i j}} - x_{i j}}{m a x_{i} {X_{j}} - m i n_{i} {X_{j}}} (a p p l i c a b l e c o s t i n d i c a t o r s)

Step 2: Calculation of Entropy Value

The entropy value of e_j is calculated with the following equation:

e_{j} = - π \sum_{i = 1}^{m} z_{ij} \ln z_{ij} (j = 1, 2, \dots, n)

Step 3: Calculation of Information Utility

The information utility of the jth criterion is calculated with the following equation:

b_{j} = 1 - e_{j} (j = 1, 2, \dots, n)

Step 4: Calculation of Weights of Criteria

The weights of the criteria are calculated with the following equation.

w_{ej} = \frac{b_{j}}{\sum_{j = 1}^{n} b_{j}} = \frac{1 - e_{j}}{n - \sum_{j = 1}^{n} e_{j}} (i = 1, 2, \dots, n)

Step 5: Estimating the Measurement Value

The measurement values of the criteria are obtained with the following equation:

U_{i} = \sum_{j = 1}^{n} z_{ij} w_{ej} (i = 1, 2, \dots, m)

ARAS Method

The ARAS method is an MCDM method developed by Zavadskas and Turskis (2010) and is used to objectively identify alternatives with different attributes. In this method, the degree of contribution of each alternative is determined while ranking the alternatives (Alinezhad & Khalili, 2019). The implementation steps of the ARAS method are as follows (Zavadskas & Turskis, 2010):

Step 1: Creating the Decision Matrix

A decision matrix is created. The decision matrix is as follows:

[\begin{matrix} \begin{matrix} x_{11} & \dots & x_{1 j} \\ ⋮ & ⋱ & ⋮ \end{matrix} \begin{matrix} \dots & x_{1 n} \\ ⋱ & ⋮ \end{matrix} \\ \begin{matrix} x_{i 1} & \dots & x_{ij} \\ ⋮ & ⋱ & ⋮ \\ x_{n 1} & \dots & x_{ni} \end{matrix} \begin{matrix} \dots & x_{in} \\ ⋱ & ⋮ \\ \dots & x_{nn} \end{matrix} \end{matrix}]; i = \bar{0, m}; j = \bar{1, n}

Step 2: Normalization of the Decision Matrix

The initial values of all criteria are normalized and a normalized decision matrix is obtained.

[\begin{matrix} \begin{matrix} {\bar{x}}_{01} & \dots & {\bar{x}}_{0 j} \\ ⋮ & ⋱ & ⋮ \end{matrix} \begin{matrix} \dots & {\bar{x}}_{0 n} \\ ⋱ & ⋮ \end{matrix} \\ \begin{matrix} {\bar{x}}_{i 1} & \dots & {\bar{x}}_{ij} \\ ⋮ & ⋱ & ⋮ \\ {\bar{x}}_{m 1} & \dots & {\bar{x}}_{mi} \end{matrix} \begin{matrix} \dots & {\bar{x}}_{in} \\ ⋱ & ⋮ \\ \dots & {\bar{x}}_{mn} \end{matrix} \end{matrix}]; i = \bar{0, m}; j = \bar{1, n}

The criteria to be maximized are normalized as follows:

{\bar{x}}_{ij} = \frac{x_{ij}}{\sum_{i = 0}^{m} x_{ij}}

The criteria to be minimized are normalized as follows:

x_{ij} = \frac{1}{x_{ij}^{*}}; {\bar{x}}_{ij} = \frac{x_{ij}}{\sum_{i = 0}^{m} x_{ij}}

Step 3: Defining the Normalized Weighted Matrix

In this step, the normalized weighted matrix is defined as follows.

\sum_{j = 1}^{n} w_{j} = 1

[\begin{matrix} \begin{matrix} {\hat{x}}_{01} & \dots & {\hat{x}}_{0 j} \\ ⋮ & ⋱ & ⋮ \end{matrix} \begin{matrix} \dots & {\hat{x}}_{0 n} \\ ⋱ & ⋮ \end{matrix} \\ \begin{matrix} {\hat{x}}_{i 1} & \dots & {\hat{x}}_{ij} \\ ⋮ & ⋱ & ⋮ \\ {\hat{x}}_{m 1} & \dots & {\hat{x}}_{mj} \end{matrix} \begin{matrix} \dots & {\hat{x}}_{in} \\ ⋱ & ⋮ \\ \dots & {\hat{x}}_{mn} \end{matrix} \end{matrix}]; i = \bar{0, m}; j = \bar{1, n}

The normalized weighted values of all criteria are calculated with the following equation:

{\hat{x}}_{ij} = {\bar{x}}_{ij} w_{j}; i = \bar{0, m}

The equation below shows the values of the optimality function.

S_{i} = \sum_{j = 1}^{n} {\hat{x}}_{ij}; i = \bar{0, m}

S_i denotes the value of the optimality function of alternative i. A large S_i value indicates that the alternative is effective. The utility value of the alternative is obtained by comparing it with the best S₀. The following equation is used to calculate the degree of benefit K_i:

K_{i} = \frac{S_{i}}{S_{0}}; i = \bar{0, m}

Step 4: Determining the Final Ranking of Alternatives

K_i values take values between 0 and 1. These values are arranged in descending order and the alternative with the largest K_i value is selected as the best alternative.

MABAC Method

MABAC is an MCDM method used to objectively rank alternatives in MCDM problems. This method, developed by Pamucar and Cirovic (2015), determines the distance of the alternatives to the boundary approximation area of the criterion function and ranks the alternatives accordingly. The implementation steps of the MABAC method are as follows:

Step 1: Creating the Decision Matrix

The decision matrix is as follows. Where x_ij is the value of the ith alternative according to the jth criterion, m is the number of alternatives and n is the number of criteria.

\begin{matrix} C_{1} & C_{2} & \dots & C_{n} \end{matrix}

X = \begin{matrix} A_{1} \\ A_{2} \\ \dots \\ A_{m} \end{matrix} [\begin{matrix} x_{11} & x_{12} & \dots & x_{1 n} \\ x_{21} & x_{22} & \dots & x_{2 n} \\ \dots & \dots & \dots & \dots \\ x_{m 1} & x_{m 2} & \dots & x_{mn} \end{matrix}]

Step 2: Normalization of the Decision Matrix

The normalized version of the initial decision matrix is as follows:

\begin{matrix} C_{1} & C_{2} & \dots & C_{n} \end{matrix}

X = \begin{matrix} A_{1} \\ A_{2} \\ \dots \\ A_{m} \end{matrix} [\begin{matrix} n_{11} & n_{12} & \dots & n_{1 n} \\ n_{21} & n_{22} & \dots & n_{2 n} \\ \dots & \dots & \dots & \dots \\ n_{m 1} & n_{m 2} & \dots & n_{mn} \end{matrix}]

The elements of the normalized decision matrix are determined by the following equations:

(a) For the utility criterion;

n_{ij} = \frac{x_{ij} - x_{i}^{-}}{x_{i}^{+} - x_{i}^{-}}

(b) For the cost criterion;

n_{ij} = \frac{x_{ij} - x_{i}^{+}}{x_{i}^{-} - x_{i}^{+}}

Step 3: Calculation of Elements in the Matrix

The elements in the matrix are calculated with the help of the following equation:

v_{ij} = w_{i} . (n_{ij} + 1)

n_ij is the element of the normalized matrix. w_i is the weighted coefficient of the criterion. The weighted matrix is obtained with the help of the following equation:

\begin{matrix} V = [\begin{matrix} v_{11} & v_{12} & \dots & v_{1 n} \\ v_{21} & v_{22} & v_{2 n} \\ \dots & \dots & \dots & \dots \\ v_{m 1} & v_{m 2} & \dots & v_{mn} \end{matrix}] \\ = [\begin{matrix} w_{1} . (n_{11} + 1) & w_{2} . (n_{12} + 1) & \dots & w_{n} . (n_{1 n} + 1) \\ w_{1} . (n_{21} + 1) & w_{2} . (n_{22} + 1) & \dots & w_{n} . (n_{2 n} + 1) \\ \dots & \dots & \dots & \dots \\ w_{1} . (n_{m} + 1) & w_{2} . (n_{m 2} + 1) & \dots & w_{n} . (n_{mn} + 1) \end{matrix}] \end{matrix}

In this equation, n is the number of criteria and m is the number of alternatives.

Step 4: Determination of Border Approximation Area Matrix

The border approximation area matrix is determined by the following equation:

g_{i} = {(Π_{j - 1}^{m} v_{ij})}^{1 / m}

After calculating the g_i value for each criterion, the boundary approximation area matrix is created.

\begin{matrix} C_{1} & C_{2} & \dots & C_{n} \end{matrix}

G = [\begin{matrix} g_{1} & g_{2} & \dots & g_{n} \end{matrix}]

Step 5: Calculating the Distance of the Alternative from the Border Approximation Area

The distance of the alternatives to the border approximation area (q_ij) is determined as the difference between the elements in the weight matrix and the value of the border approximation area.

Q = [\begin{matrix} q_{11} & q_{12} & \dots & q_{1 n} \\ q_{21} & q_{22} & q_{2 n} \\ \dots & \dots & \dots & \dots \\ q_{m 1} & q_{m 2} & \dots & q_{mn} \end{matrix}]

Q = V = G = [\begin{matrix} v_{11} & v_{12} & \dots & v_{1 n} \\ v_{21} & v_{22} & v_{2 n} \\ \dots & \dots & \dots & \dots \\ v_{m 1} & v_{m 2} & \dots & v_{mn} \end{matrix}] - [\begin{matrix} g_{1} & g_{2} & \dots & g_{n} \\ g_{2} & g_{2} & g_{n} \\ \dots & \dots & \dots & \dots \\ g_{1} & g_{2} & \dots & g_{n} \end{matrix}]

\begin{matrix} Q = [\begin{matrix} v_{11} - g_{1} & v_{12} - g_{2} & \dots & v_{1 n} - g_{n} \\ v_{21} - g_{1} & v_{22} - g_{2} & v_{2 n} - g_{n} \\ \dots & \dots & \dots & \dots \\ v_{m 1} - g_{1} & v_{m 2} - g_{2} & \dots & v_{mn} - g_{n} \end{matrix}] \\ = [\begin{matrix} q_{11} & q_{12} & \dots & q_{1 n} \\ q_{21} & q_{22} & q_{2 n} \\ \dots & \dots & \dots & \dots \\ q_{m 1} & q_{m 2} & \dots & q_{mn} \end{matrix}] \end{matrix}

The belonging of alternative A_i to the approach areas (G, G-, G⁺) is determined by the following equation:

A_{i} \in {\begin{matrix} G^{+} & if & q_{ij} > 0 \\ G & if & q_{ij} = 0 \\ G^{-} & if & q_{ij} < 0 \end{matrix}

Step 6: Ranking of Alternatives

The values of the criterion functions for the alternatives can be calculated as the sum of the distance of the alternatives from the boundary zoom areas (q_i). By taking the sum of the elements of the Q matrix in rows, the final values of the criteria functions of the alternatives are obtained. n is the number of criteria and m is the number of alternatives.

S_{i} = \sum_{j = 1}^{n} q_{ij}, j = 1, 2, \dots, n,, = 1, 2, \dots, m

TOPSIS Method

TOPSIS is an MCDM method developed by Yoon (1980) and Hwang and Yoon (1981) for ranking alternatives. This method uses Euclidean distance to rank alternatives. The proximity of alternatives to the positive ideal solution and their distance from the negative ideal solution are the basis of this method (Zhang et al., 2011). The implementation steps are as follows (Tzeng & Huang, 2011):

Step 1: Creating the Decision Matrix

The decision matrix is created with n alternatives and m criteria as follows, where x is the set of performance ratings and w is the weight of the criteria.

\begin{matrix} Alternatives & C_{1} & C_{2} & \dots & C_{m} \\ A_{1} & x_{11} & x_{12} & \dots & x_{1 m} \\ A_{2} & x_{11} & x_{22} & \dots & x_{2 m} \\ ⋮ & ⋮ & ⋮ & ⋮ & ⋮ \\ A_{n} & x_{n 1} & x_{n 2} & \dots & x_{nm} \\ W & w_{1} & w_{1} & \dots & w_{m} \end{matrix}

Step 2: Normalization of the Decision Matrix

At this stage, the decision-making matrix is normalized. For this, the following equations apply:

r_{kj} (x) = \frac{x_{kj}}{\sqrt{\sum_{k = 1}^{n} x_{kj}^{2}}}, k = 1, \dots, n; j = 1, \dots, m

For the utility criterion;

r_{kj} (x) = (x_{kj} - x_{j}^{-}) / x_{j}^{*} - x_{j}^{-})

For the cost criterion;

r_{kj} (x) = (x_{j}^{-} - x_{kj}) / x_{j}^{-} - x_{j}^{*})

Step 3: Obtaining the Normalized Decision Matrix with Weights

The weighted normalized decision matrix is obtained by the following equation:

v_{kj} (x) = w_{j} r_{kj} (x), k = 1, \dots, n; j = 1, \dots, m

Step 4: Determination of Positive (PIS) and Negative Ideal Point (NIS)

The following equation is used to determine the positive ideal point.

PIS = A^{+} = {v_{1}^{+} (x), v_{2}^{+} (x), \dots, v_{j}^{+} (x), \dots, v_{m}^{+} (x)}

= {(ma x_{k} v_{kj} (x) | j \in J_{1}), (mi n_{k} v_{kj} (x) | j \in J_{2}) | k = 1, \dots, n}

The following equation is used to determine the negative ideal point.

NIS = A^{-} = {v_{1}^{-} (x), v_{2}^{-} (x), \dots, v_{j}^{-} (x), \dots, v_{m}^{-} (x)}

= {(mi n_{k} v_{kj} (x) | j \in J_{1}), (ma x_{k} v_{kj} (x) | j \in J_{2}) | k = 1, \dots, n}

J₁ and J₂ show the benefit and cost criteria.

Step 5: Calculating PIS and NIS Discrimination Between Alternatives

The PIS and NIS discrimination values between alternatives are measured using Euclidean distance. The following equation is used for this purpose:

D_{k}^{*} = \sqrt{\sum_{j = 1}^{m} {[v_{kj} (x) - v_{j}^{+} (x)]}^{2}, k = 1, \dots, n}

D_{k}^{-} = \sqrt{\sum_{j = 1}^{m} {[v_{kj} (x) - v_{j}^{-} (x)]}^{2}, k = 1, \dots, n}

Similarities to PIS are obtained with the following equation:

C_{k}^{*} = \frac{D_{k}^{-}}{D_{k}^{+} + D_{k}^{-}}, k = 1, \dots, n

Step 6: Ranking of Alternatives

The value of $C_{k}^{*}$ is arranged in descending order to identify the best alternatives.

Integration of Rankings: Borda Function

In this study, the Borda function was used to obtain the final ranking value from nine different rankings obtained for the research performance of countries. The Borda function is the fairest method when a single value needs to be obtained from different rankings (Dummett, 1998). By applying this method, a Borda score is obtained for each ranking. This score indicates the superiority of each alternative over the other alternatives. The Borda score is determined by the following formula (Chen et al., 2014):

f_{B} (c_{i}) = \sum_{c_{j} \in C \ {c_{i}}} N (c_{i} > c_{j})

Here $f_{B} (c_{i})$ , denotes the total superiority of candidate c_i over the other candidates, C\[_ci] denotes the set of candidates excluding candidate c_i.

Research Performance Criteria

One of the most important decisions in measuring research performance is to decide which criteria to use. Research performance has characteristics such as productivity, quality, and impact. Different measurement criteria emphasize different aspects of research performance (Auranen & Nieminen, 2010). However, the measurement of research performance relies on publication outputs as a measure of productivity, citations as a measure of impact, and in some cases, quality of work assessments and the reputation indicators of researchers (Bazeley, 2010).

There is no clear and agreed set of indicators by which research performance is measured. However, research performance is generally analyzed in two dimensions. The first is productivity, which depends on publication outputs. The second is impact, which is an indicator of the quality of publications. When studies on research performance are examined, the number of publications as a measure of productivity (Bansal et al., 2022; Chang et al., 2020; Holliday et al., 2014), the ratio of publications in top publications (Craig et al, 2021), the number of publications per academic staff (Johnes & Yu, 2008), the number of publications in Scientific Journal Ranking journals (Saxena et al., 2022), h index (Lai et al., 2022; Praus, 2018; Sadeghi-Bazargani et al, 2019), g index (Ke et al., 2016), i10 index (Cvetanovich et al., 2016; Susarla et al., 2015), and the ratio of publications to total publications (Abramo et al., 2008) are all used. The number of citations (Aldieri et al., 2020; Ermatita et al., 2022), i10 index (Susarla et al., 2015), number of citations per publication, h index, and g index (Ding et al., 2020; Tahira et al., 2018) are used as indicators of impact. In addition to these indicators, the opinions of researchers and academic administrators (Izuagbe, 2021; Law & Chon, 2007; Martin-Sardesai et al., 2017; Quoc et al., 2021; Ryan, 2014) and research grants (Goldstein, 2011; Marisha et al., 2017; Valadkhani & Ville, 2010) were used.

Quantity is important measuring research performance. However, quality is as important as quantity. Therefore, evaluating quantitative and qualitative criteria together is important in measuring performance (Kao & Pao, 2009). For example, the number of publications should not be used to measure performance without considering the impact of the publications. The number of citations is not a reliable criterion without their impact. The number of citations per article is a reasonable criterion, but it is not recommended as a performance criterion because it ignores the size of the alternatives examined and thus creates a bias toward large units (Abdul-Majeed et al., 2021).

Considering all the studies in the literature, in this study, the research performance of countries was analyzed with a total of six criteria, three in the productivity dimension and three in the impact dimension, which comprise important components of research performance. The main productivity measures used are the following: (1) number of documents, (2) number of documents in the top 1%, and (3) number of documents in Q1 journals. However, these criteria may be affected by the size indicators of the units, such as the population of the country and the number of researchers, because larger units can produce more output. Comparing the performance between units by ignoring size may lead to biassed results. The contribution of a performance assessment that ignores size is that it only reveals the most influential countries that contribute the most to science. In light of these considerations, in this study, the productivity outputs of countries are divided into three sets of criteria. First, to analyse the performance of countries with raw data, the size of the countries is ignored and the following set of criteria is used:

number of documents (ND)

number of documents in top 1% (ND%1)

number of documents in Q1 journals (NDQ1)

Secondly, population was taken as a measure of the size of countries and the following criteria were used by normalizing the raw data to population:

number of documents per population (ND_P)

number of documents in top 1% per population (ND%1_P)

number of documents in Q1 journals per population (NDQ1_P)

Third, gross domestic product (GDP) was taken as a measure of the size of countries and the raw data were normalized to GDP and the following criteria were used:

number of documents per GDP (ND_GDP)

number of documents in top 1% per GDP (ND%1_GDP)

number of documents in Q1 journals per GDP (NDQ1_GDP)

The first set of criteria was used to determine the weights of the criteria. This is because these criteria are the main indicators of productivity. These criteria are used in all fields of science and form the basis for the weighting of criteria. The data in the second and third sets were included in the analysis when ranking with the criteria in the first set.

In this study, three different criteria were used as impact indicators. Each of these indicators is independent of unit size and is normalized values. For this reason, the normalization to population and GDP applied to productivity criteria is not applied here. The impact indicators are as follows:

Impact Relative to World (IRW): Demonstrates the impact of publication based on global research impact. The world average is always equal to 1. If this value is greater than 1, it means that the country under study performs above the world average (Incites, 2022).

Category Normalized Citation Impact (CNCI): It is an impact indicator calculated by dividing the citation value of publications with the same publication year, publication type, and subject area by the expected citation value (Incites, 2022).

Average Percentile (AP): The AP value is a normalized impact indicator that shows how a publication performs compared to peer publications in its field, publication type, and year. This value is determined by creating a citation frequency distribution of all publications in the field, publication type, and year, and calculating the percentage of each publication (Incites, 2022).

Table 1 presents the descriptive statistics of the productivity and impact criteria of the countries.

Table 1.

Descriptive Statistics on Research Performance Criteria of Countries.

Descriptive statistics
	N	Minimum	Maximum	Mean	Std. Deviation
(ND)	38	12,402	5,089,006	497,084.68	861,839.332
(NDQ1)	38	2,818	2,035,078	190,618.68	345,942.381
(ND%1)	38	163	99,734	9,415.55	17,035.943
IRW	38	.68	2.04	1.3353	.32811
CNCI	38	.80	1.84	1.2679	.26984
AP	38	33	57	48.24	6.024
ND_P	38	.002	.045	.01979	.011612
NDQ1_P	38	.000	.021	.00771	.005335
ND%1_P	38	.000	.001	.00043	.000342
ND_GDP	38	.0000000191	.0000001155	.000000058762	.0000000222507
NDQ1_GDP	38	.0000000046	.0000000382	.000000021137	.0000000091050
ND_%1_GDP	38	.0000000002	.0000000031	.000000001157	.0000000006109

Data Collection

This study focuses on the research performance of OECD countries over the last ten years. One of the reasons for this is to reveal the recent trends and developments in the research performance of the countries during this period. In this way, a current view of the research performance of the countries has been obtained. Second, in today’s rapidly increasing scientific knowledge, it is not possible for research performance to remain stable. Analyzing the rapidly evolving research environment with old data may not accurately reflect the current situation. Third, this research provides important information for policy makers to develop evidence-based policies. Research results from a recent period provide more accurate evidence for this purpose.

Data on the criteria used to measure research performance were obtained from the InCites Benchmarking and Analytics database (January 16, 2023). Performance data covers reviews and review articles for 38 OECD countries for 2013 to 2022. Documents other than these types of publications were excluded from the dataset.

Data on the population of countries were nine-year data from the World Bank (2022b), covering 2013 to 2021. Because the World Bank population data does not include 2022, population data for 2022 is excluded from the analysis.

The data on the GDP of countries are nine-year data from the World Bank (2022a) covering 2013 to 2021. As 2022 GDP data are not available, data for this year are not considered.

Data on the development levels of countries are based on the Human Development Report prepared by the United Nations Development Programme (UNDP, 2023). While this report was published every ten years from 1990 to 2010, it was published in 2015 and 2018 after 2010 and started to be published every year after 2018. Data on the level of development of countries were selected for the period 2010–2021.

R&D data of countries are taken from the OECD database (OECD, 2022). The most recent data for 2020. For this reason, the dataset was selected to cover 2013 to 2020. As these data will be used for the correlation analysis, 33 countries with complete data were analyzed and data from 5 countries were excluded from the correlation analysis.

In the correlation analysis, two types of data on countries’ collaboration were used. First, the percentage of international collaboration and second, the percentage of domestic collaboration were used. These data are from the InCites Benchmarking and Analytics database (as of January 16, 2023).

Among the data used in this study, data on research performance, population data, GDP data, and data on the development levels of countries are complete for all countries and all years. The R&D data of the countries end in 2020, which is the most recent data year. However, some countries have been missing R&D data for some years. To overcome this problem, only 33 countries with complete data sets for all years were included in the correlation analysis.

Results

Findings Regarding the Weights of the Criteria

To answer the first research question, three MCDM methods were used to objectively weight the productivity criteria. The weighting results of the criteria are given in Figure 2. Figure 2 shows that the CRITIC method gave the highest weight to ND 1%. The lowest weight belongs to NDQ1. In the MEREC method, NDQ1 has the highest weight, while ND has the lowest weight. In the Entropy method, all three criteria have close weight values. When the weighting results of the three methods are analyzed together, it is seen that ND%1 is the most important criterion, while ND is the least important criterion.

Figure 2.

Weight values of productivity criteria.

The weight values of the impact criteria weighted by the three different MCDM methods are given in Figure 3. According to the findings, there are differences between the weight values of the three methods. The AP criterion has the highest weight in the CRITIC method, whereas the IRW criterion has the highest weight in the MEREC and Entropy methods. When the results of the three methods are evaluated together, the IRW criterion is the most important.

Figure 3.

Weight values of impact criteria.

Findings on Research Performance of Countries

After the criteria weights to be used in the measurement of research performance were determined by three different MCDM methods, the values obtained from these methods were analyzed separately by ARAS, MABAC, and TOPSIS methods, and the research performance rankings of the countries were obtained. The analyses were conducted first with raw data, then with data normalized to population, and finally with data normalized to GDP.

Table 2 shows the ranking values of countries in terms of research performance because of the analyses conducted using three different weighting and ranking methods. In the analysis made with raw data, the country with the highest research performance was the USA, followed by the UK, Germany, Australia, and the Netherlands. These countries have the highest research performance when various size indicators are neglected.

Table 2.

Research Performance Rankings of Countries.

Country	ac₁	am₁	ae₁	mc₁	mm₁	me₁	tc₁	tm₁	te₁	tbs₁	fr ₁	ac₂	am₂	ae₂	mc₂	mm₂	me₂	tc₂	tm₂	te₂	tbs₂	fr ₂	ac₃	am₃	ae₃	mc₃	mm₃	me₃	tc₃	tm₃	te₃	tbs₃	fr ₃
Australia	4	4	4	7	7	8	4	4	4	296	4	7	7	7	7	8	8	6	7	7	278	7	11	11	11	10	11	11	11	11	11	244	11
Austria	16	16	16	19	19	19	21	20	20	176	21	15	15	15	14	14	14	15	14	15	211	15	15	15	15	16	15	15	17	15	15	204	15
Belgium	14	14	14	10	10	10	15	15	15	225	13	10	10	10	10	10	10	10	10	10	252	10	10	10	10	8	8	8	13	10	10	255	10
Canada	5	5	5	11	13	13	5	5	5	275	6	16	16	16	16	16	16	16	16	16	198	16	16	16	16	17	16	16	18	17	16	194	16
Chile	31	31	31	30	31	31	32	33	32	60	31	31	31	31	31	30	30	33	31	31	63	31	30	29	29	29	29	30	31	29	30	76	29
Colombia	38	38	38	38	38	38	38	38	38	0	38	38	38	38	38	38	38	38	38	38	0	38	37	37	37	38	38	37	36	37	37	8	37
Costa Rica	37	37	36	36	34	33	37	34	34	24	36	35	35	35	36	35	35	36	35	35	25	35	36	36	35	37	36	35	38	36	35	18	36
Czech Republic	29	28	28	29	29	30	31	30	29	79	28	26	26	26	26	26	26	26	26	26	108	26	17	17	17	24	24	23	15	16	17	172	18
Denmark	15	15	15	5	6	6	14	13	13	240	10	3	3	3	3	3	3	3	3	3	315	3	3	3	3	3	3	3	4	3	3	314	3
Estonia	24	22	21	12	9	7	17	16	16	198	15	14	12	12	11	11	11	14	15	14	228	12	1	1	1	1	1	1	1	1	1	333	1
Finland	18	18	19	16	14	15	20	19	19	184	17	8	8	9	9	9	9	8	8	8	266	9	9	9	9	7	7	7	9	9	9	267	9
France	6	6	6	17	15	14	7	6	6	259	8	24	24	24	23	23	23	25	24	24	128	24	28	27	27	26	26	26	30	28	28	96	27
Germany	3	3	3	8	8	9	3	3	3	299	3	20	19	19	19	19	19	20	20	20	167	19	26	26	26	25	25	25	27	27	27	108	26
Greece	25	26	26	23	23	23	25	25	25	121	25	25	25	25	24	24	24	24	25	25	121	25	12	12	12	14	14	14	10	12	12	230	12
Hungary	30	30	29	26	25	25	29	27	27	94	27	28	28	28	27	27	27	28	28	28	93	28	19	21	21	23	23	22	16	18	18	161	21
Iceland	21	19	17	6	4	2	16	11	10	236	11	2	2	2	2	1	1	2	2	2	326	2	2	2	2	2	2	2	2	2	2	324	2
Ireland	19	20	20	18	18	18	22	22	21	164	22	11	11	11	12	12	12	12	11	11	239	11	23	22	22	19	20	20	25	22	22	147	22
Israel	20	21	22	21	21	21	23	23	23	147	23	17	17	17	17	17	17	18	17	17	188	17	21	20	19	18	18	18	21	20	20	167	19
Italy	7	7	7	14	16	17	6	7	7	254	9	22	22	22	20	20	21	23	23	23	146	22	22	23	23	22	22	24	24	23	23	136	23
Japan	11	11	11	28	27	27	10	12	12	193	16	34	33	33	32	32	32	34	32	32	48	33	35	35	36	34	34	34	35	35	36	28	35
Latvia	34	34	34	31	30	29	33	31	31	55	32	30	30	30	29	29	29	30	30	30	75	30	29	30	30	30	30	29	28	31	31	74	31
Lithuania	35	35	35	32	32	32	35	35	35	36	34	29	29	29	30	31	31	29	29	29	76	29	24	24	25	28	28	28	22	24	25	114	25
Luxembourg	26	23	23	13	12	11	18	18	17	181	18	9	9	8	8	7	6	9	9	9	268	8	25	25	24	20	19	19	26	25	24	135	24
Mexico	33	33	33	34	36	36	34	36	36	31	35	37	37	37	35	36	36	35	36	36	17	36	38	38	38	36	37	38	37	38	38	4	38
Netherlands	8	8	8	3	3	4	8	8	8	284	5	6	5	5	4	4	4	7	6	5	296	5	5	5	5	4	4	4	6	5	5	299	4
New Zealand	23	25	24	20	20	20	24	24	24	138	24	12	14	14	15	15	15	11	12	12	222	14	13	13	13	13	13	13	12	13	13	226	13
Norway	17	17	18	15	17	16	19	21	22	180	19	5	6	6	6	6	7	4	5	6	291	6	18	18	18	15	17	17	20	21	21	177	17
Poland	27	27	27	33	33	34	28	28	30	75	30	33	34	34	34	34	34	32	34	34	39	34	27	28	28	31	31	31	23	26	26	91	28
Portugal	22	24	25	24	24	24	26	26	26	121	26	19	20	20	22	22	22	19	19	19	160	21	6	6	7	11	10	10	5	6	6	275	8
Slovakia	36	36	37	35	35	35	36	37	37	18	37	32	32	32	33	33	33	31	33	33	50	32	32	33	33	33	33	33	29	33	33	50	33
Slovenia	32	32	32	27	28	28	30	29	28	76	29	18	18	18	21	21	20	17	18	18	173	18	4	4	4	9	9	9	3	4	4	292	5
South Korea	13	13	13	25	26	26	13	17	18	178	20	27	27	27	28	28	28	27	27	27	96	27	33	32	32	32	32	32	32	32	32	53	32
Spain	9	9	9	22	22	22	9	10	11	219	14	23	23	23	25	25	25	22	22	22	132	23	20	19	20	21	21	21	19	19	19	163	20
Sweden	12	12	12	9	11	12	12	14	14	234	12	4	4	4	5	5	5	5	4	4	302	4	7	8	8	6	6	6	7	7	7	280	7
Switzerland	10	10	10	4	5	5	11	9	9	269	7	1	1	1	1	2	2	1	1	1	331	1	8	7	6	5	5	5	8	8	8	282	6
Turkey	28	29	30	37	37	37	27	32	33	52	33	36	36	36	37	37	37	37	37	37	12	37	34	34	34	35	35	36	34	34	34	32	34
United Kingdom	2	2	2	2	2	3	2	2	2	323	2	13	13	13	13	13	13	13	13	13	225	13	14	14	14	12	12	12	14	14	14	222	14
USA	1	1	1	1	1	1	1	1	1	333	1	21	21	21	18	18	18	21	21	21	162	20	31	31	31	27	27	27	33	30	29	76	30

Note. ac = ARAS-CRITIC; am = ARAS-MEREC; ae = ARAS-Entropy; mc = MABAC-CRITIC; mm = MABAC-MEREC; me = MABAC-Entropy; tc = TOPSIS-CRITIC; tm = TOPSIS-MEREC; te = TOPSIS-Entropy; tbs = Total Borda Score; fr = Final Rank; 1 = Results with raw data; 2 = Results with values normalized to population; 3 = Results with values normalized GDP. Bold values indicate the combined ranking values from multiple methods.

The second analysis for measuring research performance is based on the population of countries. Because of the analysis made with data normalized according to the population size of the countries, the research performance results of the countries change. According to this analysis, Switzerland has the highest research performance, followed by Iceland, Denmark, Sweden, and the Netherlands.

The final analysis for the measurement of research performance was conducted with data normalized by the GDP of the countries. According to this analysis, the country with the highest research performance is Estonia, followed by Iceland, Denmark, Netherlands, and Slovenia.

When the three findings on research performance are evaluated together, the Netherlands draws attention. The Netherlands is the only country ranked in the top five in all three assessments. While USA had the highest performance in the first analysis, its ranking dropped considerably in the other analyses. Meanwhile, Mexico and Colombia are in the bottom five in all three rankings. Countries that ranked high in the first analysis showed further declines in the second and third analyses. However, the ranking of countries that were at the bottom in the first analysis did not change much in the other analyses. This suggests that the advantage of normalizing population and GDP does not apply to poorly ranked countries.

Factors Related to Research Performance

Table 3 shows the results of Spearman’s rank correlation analysis conducted to determine whether there is a relationship between different rankings of countries’ research performance and the level of development, R&D expenditures, international collaboration rate, and domestic collaboration rate of the country.

Table 3.

Correlation Analysis Results of Factors Related to Research Performance.

	R _R	R _P	R _GDP	R&D	DI	IC	DC
R _R	1
R _P	.693**	1
	(.000)
R _GDP	.483**	.829**	1
	(.002)	(.000)
R&D	−.566**	.004	.208	1
	(.001)	(.982)	(.245)
DI	−.764**	−.873**	−.590**	.289	1
	(.000)	(.000)	(.000)	(.102)
IC	−.268	−.717**	−.605**	−.464**	−.474**	1
	(.103)	(.000)	(.000)	(.007)	(.003)
DC	−.070	.427**	.345*	.718**	.218	−.797**	1
	(.676)	(.008)	(.034)	(.000)	(.188)	(.000)

Note. R_R = ranking with raw data; R_P = ranking with normalized to population; R_GDP = ranking with normalized to GDP; R&D = R&D expenditures; DI =Development Index, IC = percentage of international collaboration; DC = percentage of domestic collaboration. Values in parenthesis represent the significance value.

**Correlation is significant at the.01 level (two-tailed), *Correlation is significant at the.05 level (two-tailed).

When the results of the correlation analysis are analyzed, the R_R variable has a moderate positive and significant relationship with R_P and a low positive and significant relationship with R_GDP. The R&D variable has a moderate negative relationship with R_R, whereas it has no significant relationship with R_P and R_GDP. The DI variable has a high negative correlation with R_R and R_P, while it has a moderate negative correlation with R_GDP. The IC variable does not have a significant relationship with R_R, whereas it has a moderate negative relationship with R_P and a low negative relationship with R_GDP. The DC variable does not have a significant relationship with R_R, whereas it has a low positive relationship with R_P and R_GDP.

Discussion

This study aimed to analyze the research performance of 38 OECD countries using the MCDM methods. For this purpose, three research questions were proposed: (1) What are the weights of the criteria for research performance? (2) What are the research performance rankings of countries? (3) Is there a relationship between research performance ranking and the level of development, R&D expenditures, international collaboration rate, and domestic collaboration rate of the country?

In this study, the research performance of countries was analyzed using multiple MCDM methods. The results of the weighting of the criteria revealed that publications in Q1 journals and publications in the top 1% are an important and distinguishing criterion in the research performance of countries. The number of publications has a lower weight than the other two criteria. An important implication of this finding is that countries should focus on high quality productivity rather than on the number of publications to increase their research performance.

In the measurement of research performance, the data were first analyzed without normalization. The combined results of different MCDM methods revealed that USA, UK, Germany, Australia, and Netherlands are the countries with the highest performance. This result is independent of the various sizes of countries. The reason behind the high performance of these countries is the high number of publications and the higher quality of publications compared with other countries. In addition, the number of scientific journals affects productivity (Basu, 2010). The high performance of the USA, in particular, may be related to the number of USA-based journals indexed in the WOS. Moreover, as shown in this study, countries that engage in international collaborations are more productive (Aksnes, 2003; Goldfinch et al., 2003; Lee & Bozeman, 2005; Wagner et al., 2019). The reason behind the high research performance of these countries is their high level of international research collaboration.

In the performance analysis with data normalized according to the population and GDP of the countries, the performance ranking of the countries changed significantly. Switzerland, which ranked seventh in the first analysis, became the highest performing country when normalized by population. Estonia, which ranked 15th in the first analysis, ranked first when normalized to GDP. The main reason for these changes is that these analyses have different perspectives. Analyses based on raw data ignore the size of the countries. Therefore, the first analysis may reveal a result in favor of countries with more researchers or universities. In the analysis with data normalized to population, the reason for the high performance of high-performing countries may be the high number of researchers compared to the total population, thus contributing to productivity. World Bank data on the ratio of researchers in R&D to the total population confirms this finding. All the top 10 countries in the analysis normalized to population in this study are among the top 18 countries with the highest ratio of researchers in R&D to total population (World Bank, 2022c).

The results of the correlation analysis of the factors associated with research performance revealed interesting findings. There is a positive correlation between the ranking results based on raw data and those normalized to population and normalized to GDP. However, the correlation between the raw data and the ranking normalized to population is stronger than the correlation based on GDP. This suggests that country population is a better criterion to for normalizing unit sizes. The second important result is that as R&D expenditures increase, the performance of countries also increases. This result supports the research of Gulbrandsen and Smeby (2005), Solarin and Yen (2016), Lee (2021), Teitel (1994), Meo et al. (2013), and Uyar et al. (2022). Another important result obtained from the correlation analysis is that research performance increases with the increase in the level of development of countries. Developed countries have higher R&D expenditures. Research has shown a positive relationship between R&D expenditures and publication output (Gonzalez-Brambila et al., 2016; Man et al., 2004). In addition, studies have confirmed the relationship between economic wealth and scientific productivity (Cimini et al., 2014; Docampo & Bessoule, 2019; Mueller, 2016; Rahman & Fukui, 2003). Another important result obtained from the correlation analysis is that research performance decreases with increasing domestic collaboration, whereas research performance increases with increasing international collaboration. This result supports the results of other studies in the literature. The positive correlation between international collaboration and scientific productivity (Lee & Bozeman, 2005), the positive effect of collaboration between researchers from different countries on the number of citations (Aksnes, 2003; Lancho Barrantes et al., 2012), and the negative correlation between the increase in the number of domestic authors and the number of citations (Goldfinch et al., 2003) have been revealed.

In this study, more than one MCDM method was used to weight the criteria. The main reason for this is that MCDM methods have different theoretical and statistical backgrounds. The different criteria weights obtained from the methods in this study justify this decision. In addition, different weighting methods add methodological diversity to the research. Methodological diversity gives countries the opportunity to take advantage of the diversity of methods without being tied to a single method for evaluating research performance. The same applies to the ranking methods. Although the aim of the MCDM methods used for ranking in this study is to rank countries in terms of research performance, each method has a different theoretical background. The fact that the rankings of countries’ research performance vary across methods confirms this idea.

The criteria sets used in this research focus on two important components: research performance-productivity and publication impact. However, research performance can be measured using different criteria. Different measurement criteria emphasize different aspects of performance (Auranen & Nieminen, 2010).

In this research, different MCDM methods were used to weight the criteria; these methods are objective and are not based on expert opinion. Measuring performance based on subjective judgments in determining research performance may cause the results to be questioned (Kao & Pao, 2009). Despite this criticism, subjective MCDM methods based on expert opinion can be used to weight and rank criteria. In addition, a new performance measurement can be made by examining research performance using hybrid MCDM methods that use both objective and subjective methods. In recent years, MCDM methods and approaches have received considerable attention from researchers (Mardani et al., 2015). This interest has led to the emergence of different MCDM methods. Therefore, the methods used in this research are not unique. Apart from these methods, the performance results can be compared using different methods.

This study has some limitations. Research performance data are based on the WOS database alone. Research performance can be measured using databases other than the WOS database or by combining data from multiple databases. Although there are comprehensive other than the WOS database, this database is considered to be a sufficiently comprehensive database for international performance measurement (Auranen & Nieminen, 2010). The second limitation of the study is that the data used in this research are from the last 10 years. However, this period was chosen by the researcher to examine the current research performance of the countries. When using data based on a longer time period, changes in the research performance of countries can be observed. Third, criteria weights may change when the MCDM methods used for weighting criteria change or when different data sets for different universities, countries, or researchers are used. In addition, the inclusion or exclusion of countries other than the 38 countries examined in this study may change the rankings. However, despite these limitations, it can be said that the comparative research performance of countries using different methods will give an idea to researchers and the countries under study. In future research, the performance of countries can be evaluated with different research performance criteria and compared with the results of this study.

Conclusion

Research performance is important because of issues such as the distribution of funds, the productivity and accountability of higher education institutions, the determination of government grants, and institutional and national reputation. Due to this importance, studies on research performance based on researchers, academic units, universities, and countries have gained momentum in recent years. This study aimed to analyse the research performance of 38 OECD countries over the last 10 years using MCDM methods. To ensure methodological diversity and take advantage of methods with different theoretical backgrounds, several MCDM methods were used to assess the research performance of countries. The findings revealed that the number of publications in the top 1% and the IRW impact indicator is important and distinguishing criteria in the research performance of countries. In the analysis with raw data, USA, UK, Germany, Australia, and the Netherlands are the countries with the highest performance. When the data are normalized to population, Switzerland, Iceland, Denmark, Sweden, and the Netherlands show the highest performance. When the data are normalized to GDP, Estonia, Iceland, Denmark, the Netherlands, and Slovenia are the countries with the highest performance. The findings of the study show that as R&D expenditures, countries’ levels of development, and international collaboration increase, research performance also increases. However, with the increase in domestic collaboration, there has been a decrease in research performance. In addition, this study shows how to use MCDM, a part of operations research, to measure research performance using various objective methods.

Footnotes

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Muammer MARAL

Data Availability Statement

The data used to support the findings of this study are available from the corresponding author upon request.

References

Abdul-Majeed

Mahmood

Namer

N. S. M.

(2021). Measuring research performance of Iraqi universities using Scopus data. Scientometrics, 126(3), 2349–2363. https://doi.org/10.1007/s11192-021-03866-6

Abramo

D’Angelo

C. A.

Pugini

(2008). The measurement of Italian universities’ research productivity by a non parametric-bibliometric methodology. Scientometrics, 76(2), 225–244. https://doi.org/10.1007/s11192-007-1942-2

Adeyemo

K. S.

(2019). Measuring research performance of ASEAN higher education. In Adeyemo

K. S.

(Ed.), Higher education policy in the Philippines and ASEAN integration: demands and challenges (pp. 93–107). https://doi.org/10.1163/9789004411326_006

Aksnes

D. W.

(2003). Characteristics of highly cited papers. Research Evaluation, 12(3), 159–170. https://doi.org/10.3152/147154403781776645

Aldieri

Kotsemir

M. N.

Vinci

C. P.

(2020). The effects of collaboration on research performance of universities: an analysis by federal district and scientific fields in Russia. Journal of the Knowledge Economy, 11(2), 766–787. https://doi.org/10.1007/s13132-018-0570-9

Alinezhad

Khalili

(2019). New methods and applications in multiple attribute decision making (MADM). Springer. https://doi.org/10.1007/978-3-030-15009-9

Auranen

Nieminen

(2010). University research funding and publication performance-An international comparison. Research Policy, 39(6), 822–834. https://doi.org/10.1016/j.respol.2010.03.003

Awan

W. A.

Abbas

(2023). Mapping the quantity, quality and structural indicators of Asian (48 countries and 3 territories) research productivity on cloud computing. Library Hi Tech, 41(2), 309–332. https://doi.org/10.1108/lht-07-2021-0233

Bansal

Viswanathan

Meenakshi

J. V.

(2022). Does research performance explain the "leaky pipeline" in Indian academia? A study of agricultural and applied economics. Agricultural Economics, 54(2), 274–288. https://doi.org/10.1111/agec.12744

10.

Barney

C. E.

Clark

B. B.

Veiga

S. P. D.

(2022). Research productivity of management faculty: job demands-resources approach. Career Development International, 27(2), 161–184. https://doi.org/10.1108/cdi-02-2021-0051

11.

Bastedo

M. N.

Bowman

N. A.

(2010). US News & World Report college rankings: Modeling institutional effects on organizational reputation. American Journal of Education, 116(2), 163–183.

12.

Basu

(2010). Does a country’s scientific “productivity” depend critically on the number of country journals indexed? Scientometrics, 82(3), 507–516.

13.

Bazeley

(2010). Conceptualising research performance. Studies in Higher Education, 35(8), 889–903. https://doi.org/10.1080/03075070903348404

14.

Bowman

N. A.

Bastedo

M. N.

(2009). Getting on the front page: Organizational reputation, status signals, and the impact of US News and World Report on student decisions. Research in Higher Education, 50(5), 415–436. https://doi.org/10.1007/s11162-009-9129-8

15.

Brocato

J. J.

Mavis

(2005). The research productivity of faculty in family medicine departments at US medical schools: A national study. Academic Medicine, 80(3), 244–252. https://doi.org/10.1097/00001888-200503000-00008

16.

Cardoso

Silva

de Almeida

G. G. F.

Santos

L. L.

(2020). A bibliometric model to analyze country research performance: Scival topic prominence approach in tourism, leisure and hospitality. Sustainability, 12(23), 1–26. https://doi.org/10.3390/su12239897

17.

Chang

Y. W.

Chen

D. Z.

Huang

M. H.

(2020). Discovering types of research performance of scientists with significant contributions. Scientometrics, 124(2), 1529–1552. https://doi.org/10.1007/s11192-020-03533-2

18.

Chen

C. H.

(2020). A novel multi-criteria decision-making model for building material supplier selection based on Entropy-AHP weighted TOPSIS. Entropy, 22(2), 259. https://doi.org/10.3390/e22020259

19.

Chen

Y. K.

Zou

Chen

(2014). Preference integration and optimization of multistage weighted voting system based on ordinal preference. Abstract and Applied Analysis, 2014, 1–7. https://doi.org/10.1155/2014/186545

20.

Cimini

Gabrielli

Sylos Labini

(2014). The scientific competitiveness of nations. PLoS one, 9(12), 1–11. https://doi.org/10.1371/journal.pone.0113470

21.

Craig

B. M.

Cosh

S. M.

Luck

C. C.

(2021). Research productivity, quality, and impact metrics of Australian psychology academics. Australian Journal of Psychology, 73(2), 144–156. https://doi.org/10.1080/00049530.2021.1883407

22.

Cvetanovich

G. L.

Saltzman

B. M.

Chalmers

P. N.

Frank

R. M.

Cole

B. J.

Bach

B. R.

(2016). Research productivity of sports medicine fellowship faculty. Orthopaedic Journal of Sports Medicine, 4(12), 1–6. https://doi.org/10.1177/2325967116679393

23.

Diakoulaki

Mavrotas

Papayannakis

(1995). Determining objective weights in multiple criteria problems: The CRITIC method. Computers & Operations Research, 22(7), 763–770.

24.

Ding

J. D.

Liu

Kandonga

G. A.

(2020). Exploring the limitations of the h-index and h-type indexes in measuring the research performance of authors. Scientometrics, 122(3), 1303–1322. https://doi.org/10.1007/s11192-020-03364-1

25.

Docampo

Bessoule

J. J.

(2019). A new approach to the analysis and evaluation of the research output of countries and institutions. Scientometrics, 119(2), 1207–1225. https://doi.org/10.1007/s11192-019-03089-w

26.

Dummett

(1998). The Borda count and agenda manipulation. Social Choice and Welfare, 15(2), 289–296.

27.

Edgar

Geare

(2013). Factors influencing university research performance. Studies in Higher Education, 38(5), 774–792. https://doi.org/10.1080/03075079.2011.601811

28.

Ermatita

Sanmorino

Samsuryadi

Rini

D. P.

(2022). Analyzing factors contributing to research performance using backpropagation neural network and support vector machine. Ksii Transactions on Internet and Information Systems, 16(1), 153–172. https://doi.org/10.3837/tiis.2022.01.009

29.

Fritsch

Slavtchev

(2007). Universities and innovation in space. Industry and Innovation, 14(2), 201–218.

30.

Geuna

Martin

B. R.

(2003). University research evaluation and funding: An international comparison. Minerva, 41(4), 277–304. https://doi.org/10.1023/B:MINE.0000005155.70870.bd

31.

Goldfinch

Dale

DeRouen

(2003). Science from the periphery: Collaboration, networks and “periphery effects” in the citation of New Zealand crown research institutes articles, 1995-2000. Scientometrics, 57(3), 321–337. https://doi.org/10.1023/a:1025048516769

32.

Goldstein

(2011). Estimating research performance by using research grant award gradings. Journal of the Royal Statistical Society Series a-Statistics in Society, 174, 83–93. https://doi.org/10.1111/j.1467-985X.2010.00657.x

33.

Gonzalez-Brambila

C. N.

Reyes-Gonzalez

Veloso

Perez-Angón

M. A.

(2016). The scientific impact of developing nations. PLoS One, 11(3), 1–14. https://doi.org/10.1371/journal.pone.0151328

34.

Guan

J. C.

(2004). A comparative study of research performance in computer science. Scientometrics, 61(3), 339–359. https://doi.org/10.1023/B:SCIE.0000045114.85737.1b

35.

Gulbrandsen

Smeby

J. C.

(2005). Industry funding and university professors’ research performance. Research Policy, 34(6), 932–950. https://doi.org/https://doi.org/10.1016/j.respol.2005.05.004

36.

Hazelkorn

(2007). The impact of league tables and ranking systems on higher education decision making. Higher Education Management and Policy, 19(2), 1–24. https://doi.org/10.1787/hemp-v19-art12-en

37.

Hazelkorn

(2008). Learning to live with league tables and ranking: The experience of institutional leaders. Higher Education Policy, 21(2), 193–215. https://doi.org/10.1057/hep.2008.1

38.

(2008). Integrated analytic hierarchy process and its applications—a literature review. European Journal of Operational Research, 186(1), 211–228. https://doi.org/10.1016/j.rser.2009.06.021

39.

Holliday

E. B.

Jagsi

Wilson

L. D.

Choi

Thomas

C. R.

Fuller

C. D.

(2014). Gender differences in publication productivity, academic position, career duration, and funding among U.S. academic radiation oncology faculty. Academic Medicine, 89(5), 767–773. https://doi.org/10.1097/acm.0000000000000229

40.

Horta

Jung

J. S.

Santos

J. M.

(2020). Mobility and research performance of academics in city-based higher education systems. Higher Education Policy, 33(3), 437–458. https://doi.org/10.1057/s41307-019-00173-x

41.

Hung

C. L.

Chou

J. C. L.

(2013). Resource commitment, organizational diversity, and research performance: A case of the national telecommunication program in Taiwan. Project Management Journal, 44(3), 32–47. https://doi.org/10.1002/pmj.21340

42.

Hwang

Yoon

(1981). Multiple attribute decision making methods and applications. Spring-Verlag.

43.

Incites. (2022). Incites help. http://help.prod-incites.com/inCites2Live/overviewGroup/overviewInCites.html

44.

Ishizaka

Nemery

(2013). Multi-criteria decision analysis: Methods and software. John Wiley & Sons.

45.

Izuagbe

(2021). Faculty research performance expectancy of online databases: system design characteristics as facilitating conditions. Journal of Academic Librarianship, 47(2), 1–11. https://doi.org/10.1016/j.acalib.2021.102318

46.

Johnes

(2008). Measuring the research performance of Chinese higher education institutions using data envelopment analysis. China Economic Review, 19(4), 679–696. https://doi.org/10.1016/j.chieco.2008.08.004

47.

Kao

Pao

H. L.

(2009). An evaluation of research performance in management of 168 Taiwan universities. Scientometrics, 78(2), 261–277. https://doi.org/10.1007/s11192-007-1906-6

48.

S. W.

Lin

W. C.

Tsai

C. F.

(2016). Research performance of AACSB accredited institutions in Taiwan: before versus after accreditation. Springerplus, 5, 1–10. https://doi.org/10.1186/s40064-016-2934-6

49.

Keshavarz-Ghorabaee

Amiri

Zavadskas

E. K.

Turskis

Antucheviciene

(2021). Determination of objective weights using a new method based on the removal effects of criteria (MEREC). Symmetry, 13(4), 525. https://doi.org/10.3390/sym13040525

50.

Khalid

Tadesse

(2022). Faculty research productivity at women’s universities through the lens of preference theory. Higher Education, 83(5), 949–968. https://doi.org/10.1007/s10734-021-00717-4

51.

Khan

N. R.

Thompson

C. J.

Taylor

D. R.

Venable

G. T.

Wham

R. M.

Michael

L. M.

Klimo

(2014). An analysis of publication productivity for 1225 academic neurosurgeons and 99 departments in the United States. Journal of Neurosurgery, 120(3), 746–755. https://doi.org/10.3171/2013.11.Jns131708

52.

Lai

K. A.

Saxena

Allen

P. J.

(2022). Research performance of academic psychologists in the United Kingdom. Scientometrics, 127(7), 4139–4166. https://doi.org/10.1007/s11192-022-04424-4

53.

Lancho Barrantes

B. S.

Guerrero Bote

V. P.

Rodríguez

Z. C.

de Moya Anegón

(2012). Citation flows in the zones of influence of scientific collaborations. Journal of the American Society for Information Science and Technology, 63(3), 481–489. https://doi.org/10.1002/asi.21682

54.

Law

Chon

(2007). Evaluating research performance in tourism and hospitality: The perspective of university program heads. Tourism Management, 28(5), 1203–1211. https://doi.org/10.1016/j.tourman.2006.09.022

55.

Lee

Bozeman

(2005). The impact of research collaboration on scientific productivity. Social Studies of Science, 35(5), 673–702. https://doi.org/10.1177/0306312705052359

56.

Lee

Y.-H.

(2021). Determinants of research productivity in Korean Universities: The role of research funding. The Journal of Technology Transfer, 46(5), 1462–1486. https://doi.org/10.1007/s10961-020-09817-2

57.

Man

J. P.

Weinkauf

J. G.

Tsang

Sin

J. H. D. D.

(2004). Why do some countries publish more than others? An international comparison of research funding, English proficiency and publication output in highly ranked general medical journals. European Journal of Epidemiology, 19, 811–817. https://doi.org/10.1023/B:EJEP.0000036571.00320.b8

58.

Mardani

Jusoh

Nor

Khalifah

Zakwan

Valipour

(2015). Multiple criteria decision-making techniques and their applications—a review of the literature from 2000 to 2014. Economic Research-Ekonomska Istraživanja, 28(1), 516–571. https://doi.org/https://doi.org/10.1080/1331677X.2015.1075139

59.

Marisha

Banshal

Singh

V. K.

(2017). Research performance of central universities in India. Current Science, 112(11), 2198–2207. https://doi.org/10.18520/cs/v112/i11/2198-2207

60.

Martin-Sardesai

Irvine

Tooley

Guthrie

(2017). Accounting for research: Academic responses to research performance demands in an Australian university. Australian Accounting Review, 27(3), 329–343. https://doi.org/10.1111/auar.12151

61.

Meo

S. A.

Al Masri

A. A.

Usmani

A. M.

Memon

A. N.

Zaidi

S. Z.

(2013). Impact of GDP, spending on R&D, number of universities and scientific journals on research publications among Asian countries. PLoS One, 8(6), e66449. https://doi.org/10.1371/journal.pone.0066449

62.

Mijumbi-Deve

Parkhurst

Jones

Juma

P. A.

Sobngwi-Tambekou

J. L.

Wenham

(2021). Beyond the metrics of health research performance in African countries. BMJ Global Health, 6(7), 1–8. https://doi.org/10.1136/bmjgh-2021-006019

63.

Mueller

C. E.

(2016). Accurate forecast of countries’ research output by macro-level indicators. Scientometrics, 109, 1307–1328. https://doi.org/10.1007/s11192-016-2084-1

64.

Mutz

Bornmann

Daniel

H. D.

(2017). Are there any frontiers of research performance? Efficiency measurement of funded research projects with the Bayesian stochastic frontier analysis for count data. Journal of Informetrics, 11(3), 613–628. https://doi.org/10.1016/j.joi.2017.04.009

65.

OECD. (2022). Gross domestic spending on R&D. https://data.oecd.org/rd/gross-domestic-spending-on-r-d.htm

66.

Pamucar

Cirovic

(2015). The selection of transport and handling resources in logistics centers using Multi-Attributive Border Approximation Area Comparison (MABAC). Expert Systems with Applications, 42(6), 3016–3028. https://doi.org/10.1016/j.eswa.2014.11.057

67.

Praus

(2018). Statistical evaluation of research performance of young university scholars: A case study. Transinformacao, 30(2), 167–177. https://doi.org/10.1590/2318-08892018000200003

68.

Pressman

Guterman

S. K.

Abrams

Geist

D. E.

Nelsen

L. L.

(1995). Pre-production investment and jobs induced by MIT exclusive patent licenses: A preliminary model to measure the economic impact of university licensing. Journal of the Association of University Technology Managers, 7, 49–82.

69.

Preut

Frosch

K. H.

Debus

E. S.

Grundmann

R. T.

(2022). Publication performance of university orthopedic trauma surgery in Germany. Chirurgie, 93(7), 702–710. https://doi.org/10.1007/s00104-021-01538-y

70.

Quoc

A. N.

M. T.

Pham

H. H.

(2021). The impact of the third mission on teaching and research performance: Evidence from academic scholars in an emerging country. Sage Open, 11(4), 1–12. https://doi.org/10.1177/21582440211054493

71.

Rahman

Fukui

(2003). Biomedical research productivity: factors across the countries. International Journal of Technology Assessment in Health Care, 19(1), 249–252. https://doi.org/10.1017/S0266462303000229

72.

Ryan

J. C.

(2014). The work motivation of research scientists and its effect on research performance. R & D Management, 44(4), 355–369. https://doi.org/10.1111/radm.12063

73.

Sadeghi-Bazargani

Bakhtiary

Golestani

Sadeghi-Bazargani

Jalilzadeh

Saadati

(2019). The research performance of Iranian medical academics: A national analyses. BMC Medical Education, 19(1), 1–6. https://doi.org/10.1186/s12909-019-1892-4

74.

Saxena

Lai

K. A.

Allen

P. J.

(2022). Clustering of UK universities based on the research productivity of psychology departments. Power and Education, 15(3), 385–393. https://doi.org/10.1177/17577438221130983

75.

Sedláček

(2017). The impact of governance on the research performance of European universities in cross-country comparisons. Review of Economic Perspectives, 17(4), 337–362. https://doi.org/10.1515/revecp-2017-0018

76.

Solarin

S. A.

Yen

Y. Y.

(2016). A global analysis of the impact of research output on economic growth. Scientometrics, 108, 855–874. https://doi.org/10.1007/s11192-016-2002-6

77.

Susarla

S. M.

Dodson

T. B.

Lopez

Swanson

E. W.

Calotta

Peacock

Z. S.

(2015). Do quantitative measures of research productivity correlate with academic rank in oral and maxillofacial surgery? Journal of Dental Education, 79(8), 907–913.

78.

Tahira

Abdullah

Alias

R. A.

Bakri

(2018). Evaluation of h and h-type Indices for research performance at researcher’s level. Information Development, 34(1), 64–76. https://doi.org/10.1177/0266666916674251

79.

Teitel

(1994). Scientific publications, R&D expenditures, country size, and per capita income: A cross section analysis. Technological Forecasting and Social Change, 46(2), 175–187. https://doi.org/10.1016/0040-1625(94)90024-8

80.

Tzeng

G. H.

Huang

J. J.

(2011). Multiple attribute decision making: Methods and applications. CRC Press.

81.

UNDP. (2023). Latest human development composite indices tables. https://hdr.undp.org/data-center/documentation-and-downloads

82.

Uyar

Kuzey

Kilic Karamahmutoglu

(2022). Macroeconomic factors, R&D expenditure and research productivity in economics and finance. Managerial Finance, 48(5), 733–759. https://doi.org/10.1108/MF-12-2021-0602

83.

Valadkhani

Ville

(2010). Ranking and clustering of the faculties of commerce research performance in Australia. Applied Economics, 42(22), 2881–2895. https://doi.org/10.1080/00036840801964674

84.

Wagner

C. S.

Whetsell

T. A.

Mukherjee

(2019). International research collaboration: Novelty, conventionality, and atypicality in knowledge recombination. Research Policy, 48(5), 1260–1270. https://doi.org/10.1016/j.respol.2019.01.002

85.

Wallenius

Dyer

J. S.

Fishburn

P. C.

Steuer

R. E.

Zionts

Deb

(2008). Multiple criteria decision making, multiattribute utility theory: Recent accomplishments and what lies ahead. Management Science, 54(7), 1336–1349. https://doi.org/10.1287/mnsc.1070.0838

86.

World Bank. (2022a). GDP (current US$). https://data.worldbank.org/indicator/NY.GDP.MKTP.CD

87.

World Bank. (2022b). Population, total. https://data.worldbank.org/indicator/SP.POP.TOTL

88.

World Bank. (2022c). Researchers in R&D (per million people). https://data.worldbank.org/indicator/SP.POP.SCIE.RD.P6

89.

Yoon

(1980). Systems selection by multiple attribute decision making [PhD thesis, Kansas State University]. Manhattan, Kansas.

90.

Zavadskas

E. K.

Turskis

(2010). A new additive ratio assessment (ARAS) method in multicriteria decision-making. Technological and Economic Development of Economy, 16(2), 159–172. https://doi.org/10.3846/tede.2010.10

91.

Zavadskas

E. K.

Turskis

Kildienė

(2014). State of art surveys of overviews on MCDM/MADM methods. Technological and Economic Development of Economy, 20(1), 165–179. https://doi.org/10.3846/20294913.2014.892037

92.

Zhang

C. L.

L. W.

Zhang

(2011). The evaluation of tourism destination competitiveness by TOPSIS & information entropy—a case in the Yangtze River Delta of China. Tourism Management, 32(2), 443–451. https://doi.org/10.1016/j.tourman.2010.02.007

93.

Zhang

Chen

Gao

Yang

Zhen

Lin

Zhao

(2017). Worldwide research productivity in the field of psychiatry. International Journal of Mental Health Systems, 11(1), 1–6.

Research performance of higher education in OECD countries: A hybrid multi-criteria decision-making approach

Abstract

Plain language summary

Keywords

Introduction

Method

CRITIC Method

Step 1: Normalization of the Decision Matrix

Step 2: Determination of Correlation Coefficient

Step 3: Determination of C Index

Step 4: Determination of the Weight of Attributes

Step 5: Determination of Final Weighting of Attributes

MEREC Method

Step 1: Creating the Decision Matrix

Step 2: Normalization of the Decision Matrix

Step 3: Calculation of Overall Performance of Alternatives (S i )

Step 4: Calculating the Performance of Alternatives

Step 5: Calculating the Sum of Absolute Deviations

Step 6: Determination of the Final Weight of the Criteria

Entropy Method

Step 1: Normalization of the Decision Matrix

Step 2: Calculation of Entropy Value

Step 3: Calculation of Information Utility

Step 4: Calculation of Weights of Criteria

Step 5: Estimating the Measurement Value

ARAS Method

Step 1: Creating the Decision Matrix

Step 2: Normalization of the Decision Matrix

Step 3: Defining the Normalized Weighted Matrix

Step 4: Determining the Final Ranking of Alternatives

MABAC Method

Step 1: Creating the Decision Matrix

Step 2: Normalization of the Decision Matrix

Step 3: Calculation of Elements in the Matrix

Step 4: Determination of Border Approximation Area Matrix

Step 5: Calculating the Distance of the Alternative from the Border Approximation Area

Step 6: Ranking of Alternatives

TOPSIS Method

Step 1: Creating the Decision Matrix

Step 2: Normalization of the Decision Matrix

Step 3: Obtaining the Normalized Decision Matrix with Weights

Step 4: Determination of Positive (PIS) and Negative Ideal Point (NIS)

Step 5: Calculating PIS and NIS Discrimination Between Alternatives

Step 6: Ranking of Alternatives

Integration of Rankings: Borda Function

Research Performance Criteria

Data Collection

Results

Findings Regarding the Weights of the Criteria

Findings on Research Performance of Countries

Factors Related to Research Performance

Discussion

Conclusion

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iD

Data Availability Statement

References

Step 3: Calculation of Overall Performance of Alternatives (S_i)