An Intuitionistic Fuzzy Consensus WASPAS Method for Assessment of Open-Source Software Learning Management Systems

Abstract

Ineffective evaluation of open-source software learning management system (OSS-LMS) packages can negatively impact organizational effectiveness. Clients may struggle to select the best OSS-LMS package from a wide range of options, leading to a complex multi-criteria group decision-making (MCGDM) problem. This evaluates OSS-LMS packages based on several criteria like usability, functionality, e-learning standards, reliability, activity tracking, course development, assessment, backup and recovery, error reporting, efficiency, operating system compatibility, computer-managed instruction, authentication, authorization, troubleshooting, maintenance, upgrading, and scalability. Handling uncertain data is a vital aspect of OSS-LMS package evaluation. To tackle MCGDM issues, this study presents a consensus weighted sum product (c-WASPAS) method which is applied to an educational OSS-LMS package selection problem to evaluate four OSS-LMS packages, namely ATutor, eFront, Moodle, and Sakai. The findings indicate that the priority order of alternatives is Moodle > Sakai > eFront > ATutor and, therefore, MOODLE is the best OSS-LMS package for the case study. A sensitivity analysis of criteria weights is also conducted, as well as a comparative study, to demonstrate the effectiveness of the proposed method. It is essential to note that proper OSS-LMS package evaluation is crucial to avoid negative impacts on organizational performance. By addressing MCGDM issues and dealing with uncertain information, the c-WASPAS method presented in this study can assist clients in selecting the most appropriate OSS-LMS package from multiple alternatives. The findings of this study can benefit educational institutions and other organizations that rely on OSS-LMS packages to run their operations.

Keywords

intuitionistic fuzzy set cross entropy measure consensus based WASPAS OSS-LMS package selection

1 Introduction

The widespread implementation of the internet in recent times has resulted in the integration of Information and Communication Technology (ICT) into the everyday activities of educational institutions, research centres, organizations, government agencies, and individuals (Natarajan, 2015). The growth in ICT and the internet has enabled the development of new learning environments and techniques (Albarrak et al., 2010), providing educational institutions with an innovative educational framework called e-learning. E-learning utilizes multimedia, such as audio, video, animations, and text illustrations, to deliver education and knowledge to students without any location barriers, thus offering an alternative to traditional classroom instruction. The most common form of e-learning is a software application called a learning management system (LMS). An LMS is a digital platform that provides a variety of tools and resources to facilitate and manage online learning. It allows instructors to create and deliver course content, track and assess student progress, and communicate with learners. Similarly, learners can access course materials, participate in discussions, complete assignments, and receive feedback (Ramesh and Ramanathan, 2013). LMSs are commonly used in educational institutions, corporations, and other organizations that offer training and development programs. However, the implementation of an LMS requires significant financial and institutional commitment (Edrees, 2013). According to Caminero et al. (2013), an LMS is also a software system comprised of various tools that support teaching and learning activities. Because of the rapid development of computer and internet-based technologies, LMS has become a critical component in the advancement of educational systems. As a result, many LMSs are now available online, including both authorized and free versions (Cavus, 2007). Many educational institutions spend a significant amount of time and money to implement LMSs (Edrees, 2013). Several LMSs have been presented as open-source software (OSS) licenses, including Moodle, Sakai, ATutor, Dokeos, and eFront, which are especially useful for e-learning (Awang and Darus, 2012). OSS is software that does not require a license and comes with the source code. It is intended to address the rising costs of campus-wide software while allowing the development of learner-centred structures (Abdullateef et al., 2015, 2016b; van Rooij, 2011, 2012).

Selecting the wrong OSS-LMS package can have a negative impact on an organization’s business processes and roles. If an organization selects the wrong OSS-LMS package, it may not meet its requirements and expectations, resulting in inadequate training, lower employee productivity, and decreased organizational performance. An unsuitable OSS-LMS package can result in increased costs and time associated with system modifications or replacements. It is thus crucial for organizations to carefully evaluate and select an appropriate OSS-LMS package that aligns with their specific needs and objectives to ensure efficient and effective management of their e-learning initiatives. However, due to the wide variety of available options, lack of user experience and knowledge, and constant development of information technology, the process of selecting the most suitable OSS-LMS has become increasingly complex (Jadav and Sonar, 2011; Zaidan et al., 2015). Since multi-criteria decision-making (MCDM) methods have wide range of applications (Ulutas˘ et al., 2021; Semenas et al., 2021; Filip, 2021; Krishankumar et al., 2021; Ivanović et al., 2022; Deveci et al., 2022; Hezam et al., 2023a; Deveci et al., 2023; Gökmener et al., 2023; Gokasar et al., 2023a, 2023b; Petrovas et al., 2023), MCDM methods can be used to assess and select the best OSS-LMS package by taking into account multiple conflicting criteria. MCDM methods consider multiple criteria and weigh their relative importance to arrive at a decision that best aligns with the organization’s goals and objectives. The evaluation of OSS-LMS packages requires a thorough examination and investigation of various factors, such as usability, functionality, reliability, security, and compatibility with other systems. Organizations must carefully evaluate each criterion to ensure that the selected OSS-LMS package meets their specific needs and requirements. The application of MCDM methods enables organizations to streamline their decision-making process and make well-informed decisions regarding the selection of an OSS-LMS package.

Literature research has uncovered that various scholars have utilized diverse strategies for the assessment and determination of LMSs. The investigation carried out by Waynet Inc. (2007) depicted an overview style assessment of open-source LMSs aiming to suggest an LMS that could be applied by the Commonwealth of Learning. Hultin’s (2007) studied LMSs and how to assess them, contingent upon the learning condition and the customers’ needs. Graf and List (2005) introduced an appraisal of open-source e-learning stands by concentrating on the adaptation competences of the framework. Another assessment of LMSs was described by Wyles (2007). It was divided into two parts: 1st part portrayed the outcomes of an underlying assessment of open-source LMS software and the 2nd part described the assessment technique applied to choose the best LMS as a feature of the overall platform architecture. Kljun et al. (2007) aimed to assist the individuals who are engaged in e-learning to assess optimal LMS to suit them. The writers categorized the operators of LMSs into three clusters: learners, tutors, and administrators. Arh and Blazic (2007) devised an MCDM model that employs an expert structure to choose the most appropriate and effective LMS from Blackboard 6, Moodle 1.5.2, and CLIX 5.0. Machado and Tao (2007) examined the client experience of two competing LMSs, Moodle and Blackboard, based on their ease of use and viability. Çetin et al. (2010) used Analytical Hierarchy Process (AHP) method to address the problem of LMS evaluation using nine evaluation criteria. Albarrak et al. (2010) evaluated three OSS-LMSs: Jusur, Moodle, and Sakai. Srdevic et al. (2012) used AHP to select the most reasonable LMSs as well. Caminero et al. (2013) utilized a performance assessment technique for three OSS-LMSs, namely dotLRN, Moodle, and Sakai. Edrees (2013) assessed two LMSs, Blackboard, and Moodle, based on their readiness to support Web 2.0. Ramesh and Ramanathan (2013) developed a tool to assess LMSs based on six categories of criteria. Işik et al. (2015) applied fuzzy AHP for choosing the best LMS based on nine considered criteria. Hock et al. (2015) assessed three OSS-LMSs, namely Atutor, Ilias, and Moodle, based on the dependence on the convenience and utilized acknowledgment of the systems. Abdullateef et al. (2016a) introduced the assessment and determination of three OSS-LMSs based on the three directions, namely collection of available three OSS-LMSs, detail of the assessment criteria, and capability of the selection techniques. Karagöz et al. (2017) built up a mobile app for finding analogy of two open-source LMSs and two commercial LMS dependent on some specific criteria. Adewumi et al. (2019) tried to evaluate the LMS software selection using questionnaires for experts and their suggestions. Al Amoush and Sandhu (2020) has focused on the instructor’s perspective for evaluating LMS uses. Santiago et al. (2020) tried to determine the academic efficiency performance by evaluating different LMS systems. Alturki and Aldraiweesh (2021) proposed a model which shows the efficiency of LMS model during covid-19 period. A summary of these works is presented below (see Table 1).

Table 1
Summary of existing works on LMS selection.

Reference Primary focus

Waynet Inc. (2007) Overview style assessment of open-source LMSs

Hultin’s (2007) Assessment of LMSs depending upon the learning condition and the customers’ needs

Graf and List (2005) Appraisal of open-source e-learning stands by concentrating on the adaptation competences of the framework

Wyles (2007) LMS assessment based on OSS software and overall platform architecture

Kljun et al. (2007) To assist the individuals who are engaged in e-learning

Arh and Blazic (2007) Devised an MCDM model that employs an expert structure to choose the most appropriate and effective LMS from Blackboard 6, Moodle 1.5.2, and CLIX 5.0.

Machado and Tao (2007) Examined the client experience of two competing LMSs, Moodle and Blackboard, based on their ease of use and viability

Cetin et al. (2010) LMS evaluation using Analytical Hierarchy Process (AHP)

Albarrak et al. (2010) Evaluated three OSS-LMSs: Jusur, Moodle, and Sakai.

Srdevic et al. (2012) Selection of the most reasonable LMS using AHP tool

Caminero et al. (2013) Performance assessment for three OSS-LMSs, namely dotLRN, Moodle, and Sakai.

Edrees (2013) Assessment of two LMSs, Blackboard, and Moodle, based on their readiness to support Web 2.0

Işik et al. (2015) LMS selection using fuzzy AHP

Hock et al. (2015) Assessment of three OSS-LMSs, namely- Atutor, Ilias, and Moodle based on dependent on the convenience.

Abdullateef et al. (2016b) Determination of the suitable LMS based on collection of available OSS packages, detail of the assessment criteria, and capability of the selection techniques

Karagöz et al. (2017) Built up a mobile app for finding analogy of two open-source LMSs and two commercial LMS

Adewumi et al. (2019) Evaluation of LMS software using experts opinions

Al Amoush and Sandhu (2020) Focused on the instructor’s perspective for evaluating LMS uses

Santiago et al. (2020) Determination of the academic efficiency performance by evaluating different LMS systems

Alturki and Aldraiweesh (2021) Determination of efficiency of LMS model during covid-19 period

Reference	Primary focus
Waynet Inc. (2007)	Overview style assessment of open-source LMSs
Hultin’s (2007)	Assessment of LMSs depending upon the learning condition and the customers’ needs
Graf and List (2005)	Appraisal of open-source e-learning stands by concentrating on the adaptation competences of the framework
Wyles (2007)	LMS assessment based on OSS software and overall platform architecture
Kljun et al. (2007)	To assist the individuals who are engaged in e-learning
Arh and Blazic (2007)	Devised an MCDM model that employs an expert structure to choose the most appropriate and effective LMS from Blackboard 6, Moodle 1.5.2, and CLIX 5.0.
Machado and Tao (2007)	Examined the client experience of two competing LMSs, Moodle and Blackboard, based on their ease of use and viability
Cetin et al. (2010)	LMS evaluation using Analytical Hierarchy Process (AHP)
Albarrak et al. (2010)	Evaluated three OSS-LMSs: Jusur, Moodle, and Sakai.
Srdevic et al. (2012)	Selection of the most reasonable LMS using AHP tool
Caminero et al. (2013)	Performance assessment for three OSS-LMSs, namely dotLRN, Moodle, and Sakai.
Edrees (2013)	Assessment of two LMSs, Blackboard, and Moodle, based on their readiness to support Web 2.0
Işik et al. (2015)	LMS selection using fuzzy AHP
Hock et al. (2015)	Assessment of three OSS-LMSs, namely- Atutor, Ilias, and Moodle based on dependent on the convenience.
Abdullateef et al. (2016b)	Determination of the suitable LMS based on collection of available OSS packages, detail of the assessment criteria, and capability of the selection techniques
Karagöz et al. (2017)	Built up a mobile app for finding analogy of two open-source LMSs and two commercial LMS
Adewumi et al. (2019)	Evaluation of LMS software using experts opinions
Al Amoush and Sandhu (2020)	Focused on the instructor’s perspective for evaluating LMS uses
Santiago et al. (2020)	Determination of the academic efficiency performance by evaluating different LMS systems
Alturki and Aldraiweesh (2021)	Determination of efficiency of LMS model during covid-19 period

1.1 Research Gaps and Our Contributions

The idea of fuzzy sets (FSs) was developed by Zadeh (1965), primarily as a result of taking confusing human judgments into account while resolving practical issues. FS philosophy plays a crucial role in shaping the understanding and interpretation of reality based on computational observations. It acknowledges and embraces the presence of ambiguity, partial belongingness, and inaccuracy in real-world phenomena. By incorporating these aspects into the analysis and decision-making process, FS philosophy provides a more comprehensive and nuanced perspective on complex systems and their behaviours. This allows for a more realistic representation and modelling of uncertain and imprecise information, leading to improved insights and outcomes in various fields, such as artificial intelligence, data science, and decision science. Atanassov (1986) presented the Intuitionistic FS (IFS) as a generalization of FS to deal with situations with incomplete data by using a non-belongingness grade. Since its introduction, IFS has been used by many researchers for solving group decision-making issues. Unfortunately, none of the existing IF decision support models, namely IF-Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) (Rouyendegh et al., 2020), IF-Multi-Objective Optimization on the basis of Ratio Analysis with multiplicative form (MULTIMOORA) (Garg and Rani, 2022), IF-Evaluation based on Distance from Average Solution (EDAS) (Mishra et al., 2020), IF-COmplex PRoportional ASsessment (COPRAS) (Kumari and Mishra, 2020), and IF-measurement of alternatives and ranking according to compromise solution (MARCOS) (Deb et al., 2022) don’t deal with the “consensus-reaching process” for experts. Because of their knowledge and backgrounds, decision makers in multi-criteria group decision-making problems may have opinions that are very different from one another. As a result, a consensus-building process is required for the decision-makers to raise the level of unanimity (Liu and Huang, 2020). Although many consensus models (Herrera et al., 1996; Dong et al., 2010; Herrera-Viedma et al., 2014; Gong et al., 2015; Liao et al., 2016; Wu and Xu, 2016; Zhang et al., 2018; Wu and Liao, 2019) were developed earlier, no consensus model has been developed so far with IF numbers. Moreover, in the aforementioned studies on consensus process, the information was not adjusted before it was aggregated, which may have resulted in irrational decision results.

The selection of appropriate OSS-LMS packages is a significant and uncertain MCDM challenge that is primarily taken into account by many educational organizations due to faulty information, hazy human observation, and time constraints. Numerous academics have focused on the creation of novel MCDM approaches due to the setting’s growing complexity and widespread variations. MCDM methods can be divided into two categories (Saha et al., 2022): (i) utility-based models like COPRAS, Weighted Aggregated Sum Product Assessment (WASPAS), MARCOS, and MULTIMOORA, and (ii) outranking models like Preference Ranking Organization Method for Enrichment Evaluations (PROMETHEE), Elimination and Choice Translating Reality (ELECTRE) and Organization REgarding SYnthesis of the criteria for Rankings of TEchnical alternatives (ORESTE). Due to complicated computations, the outranking techniques have trouble managing numerous choices and criteria. The utility-based models are helpful in treating MCDM issues when there are many experts and criteria involved. WASPAS was developed by Zavadskas et al. (2012) as a unique utility degree-based MCDM method to deal with a variety of realistic decision-making concerns. The benefits of WASPAS are as follows: (a) it employs a straightforward method of calculation, (b) it can select the most preferred alternative by making use of AOs, (c) being a mixture of Weighted Sum Model (WSM) and Weighted Product Model (WPM), it has more accuracy, and (d) it allows us to estimate with the maximum amount of accuracy conceivable. WASPAS method has not been merged with consensus reaching to get a reasonable outcome in OSS-LMS package selection problem.

Motivated by the above facts, this present work is dedicated to address the ambiguity and vagueness that arise during the assessment of OSS-LMS packages. To achieve this, a robust and logical decision-making model has been proposed in this paper. The contributions can be summarized as follows:

•
A consensus IF-WASPAS approach is developed to tackle group decision-making issues.
•
To demonstrate the applicability of the consensus based IF-WASPAS technique, a real-world case study for the selection of OSS-LMS packages is explored.
•
In order to verify the conclusions reached by the proposed method, a sensitivity investigation of criteria weights is also presented.
•
In order to prove the superiority of the developed approach, a comparative analysis is presented.

1.2 Arrangement of the Paper

A brief review of the literature is provided in Section 2. Few essential concepts related to IF sets are introduced in Section 3. An IF consensus WASPAS (IF-c-WASPAS) strategy is designed in this section, where the criteria values are represented by IFNs. The proposed method is clarified using a case study of OSS-LMS package selection in Section 4. Sensitivity analysis of criteria weights and comparative analysis are covered in Section 5. Conclusions are drawn from the entire study, and a summary of the prospects for the future is provided in Section 6.

2 Literature Review

2.1 IFSs

IFS sets are an extension of FSs which introduce the concept of non-membership degree to represent the degree to which an element does not belong to a set. In an IFS, each element is associated with three values: membership degree, non-membership degree, and hesitation degree. Membership degree, non-membership degree, and hesitation degree are fundamental concepts in FS theory. These degrees are used to quantify the relationship between elements and FSs, taking into account uncertainty and imprecision. The membership degree indicates the degree to which an element belongs to a particular FS. It represents the extent of similarity or relevance of the element to the set. A membership degree of 1 indicates full membership, while a degree of 0 indicates no membership at all. The non-membership degree, on the other hand, represents the degree to which an element does not belong to a FS. It captures the degree of dissimilarity or rejection of the element from the set. A non-membership degree of 1 implies complete non-membership, while a degree of 0 suggests full membership. The hesitation degree reflects the level of uncertainty or hesitation in assigning the membership and non-membership degrees to an element. It accounts for situations where the degree of belongingness or non-belongingness is not clearly defined. A higher hesitation degree indicates greater ambiguity or lack of confidence in assigning the membership and non-membership degrees. These three degrees work together to provide a more nuanced and flexible representation of the relationship between elements and FSs, allowing for a more realistic modelling and analysis of complex and uncertain information. IFSs have found several applications in various fields. Mukherjee (2017) selected the best fuel technology for land transportation subject to multiple criteria resulting in a sustainable transportation system. Büyüközkan et al. (2018) evaluated different public bus technologies as urban transportation alternatives. Wang et al. (2018) assessed consumer satisfaction in urban rail travel. Mishra et al. (2019c) solved a multi-criteria IT personnel selection problem with IF measures and Additive Ratio Assessment (ARAS) method. Mishra et al. (2020) utilized IF-EDAS methodology for evaluation of health-care waste disposal technologies. Kumari and Mishra (2020) used IF-parametric measures and IF-COPRAS for selecting green suppliers. Rouyendegh et al. (2020) utilized IF-TOPSIS approach for green supplier choice. Rani et al. (2021) assessed the performance of telecom service providers using IF-grey relational analysis. Yuan and Yang (2021) developed a dynamic MCDM model based on IF data. Tugrul (2022) developed a IFS based decision-making methodology for evaluation of paper quality. Buran and Erçek (2022) presented a business model canvas framework for public transportation organizations. Yan et al. (2022) provided a framework to determine the priority of a rail transit system from the perspective of green and low carbon. Palanisami et al. (2022) developed a new approach of multi-modal medical image fusion using IF sets. Deb et al. (2022) developed a decision-making model for the selection of enterprise resource planning systems. Gohain et al. (2022) developed a symmetric distance formula and applied it to practical problems of decision-making, pattern recognition, and clustering problems. Rasoulzadeh et al. (2022) introduced a multi-objective approach to the portfolio selection problem. Hezam et al. (2022) developed a hybrid IF-method based on the removal effects of criteria (MEREC), ranking sum (RS), and double normalization-based multi-aggregation (DNMA) (MEREC-RS-DNMA) methodology for assessment of alternative fuel vehicles. Senapati et al. (2023) developed an advanced decision-support model for the prioritization of sustainable transportation-sharing practices. Çakır and Taş (2023) defined circular IFSs for solving supplier selection problem. Mishra et al. (2023) used IF fairly aggregation operators and ARAS based model for assessing sustainable industrial buildings. An IF entropy based methodology was proposed by Hezam et al. (2023b) for assessment of sustainable supplier.

2.2 WASPAS method

WASPAS is a weighted aggregation approach in which each criterion is assigned a weight that represents its relative importance in the decision-making process. The performance of each alternative is assessed by multiplying its evaluation score on each criterion by the corresponding weight, and then summing up the weighted scores across all criteria. The alternative with the highest aggregated score is considered the best choice. WASPAS is widely used in various fields, including decision science, operation research, and engineering, to support decision-making processes that involve multiple criteria and alternatives. Deveci et al. (2018) developed interval type-2 FSs-based model with WASPAS and TOPSIS methods. Stanujkic and Karabasevic (2018) proposed IF-WASPAS method to survey the websites. Mishra and Rani (2018) assessed reservoir flood control management using interval-valued IF-WASPAS technique with information measures. Pamučar et al. (2019) identified safety advisors for hazardous material transportation using linguistic neutrosophic WASPAS. Mishra et al. (2019a) assessed the mobile phone service providers using IF-WASPAS. Mishra et al. (2019b) developed hesitant fuzzy-WASPAS method for green supplier selections. Kahraman et al. (2019) introduced Pythagorean fuzzy WASPAS model for the selection of the most reasonable administrators. Keshavarz-Ghorabaee et al. (2019) worked on the assessment of sustainable developed strategies using Type-2 fuzzy-based WASPAS and Sequential Elimination and Choice Translation (SECA) methods. Bid and Siddique (2019) assessed human hazards resultant combination of WASPAS and TOPSIS. Gundogdu and Kahraman (2019) investigated robot selection problem for the industry using WASPAS method with spherical fuzzy data. Krishankumar et al. (2019) worked on the selection of construction project risk technique statistical variance and WASPAS. Schitea et al. (2019) selected the best hydrogen mobility roll-up site utilizing integrated WASPAS, COPRAS, and EDAS under intuitionistic FSs. Dorfeshan and Mousavi (2020) assessed critical paths of aircraft maintenance planning using a coordinated MABAC and WASPAS under the interval type-2 fuzzy setting. To address the doctor recruitment issue, Sharma and Pradhan (2020) examined the machinability criteria for SUS-304L steel using WASPAS model for FSs. Mohagheghi and Mousavi (2020) resolved a sustainable project portfolio problem using WASPAS method with interval-valued Pythagorean FSs. Davoudabadi et al. (2020) addressed a supplier evaluation issue by utilizing a combined approach using TOmada de Decisao Interativa Multicriterio (TODIM), WASPAS, and TOPSIS methods under interval-valued IF setting. For assessing desirable alternative-fuel technology, Rani and Mishra (2020) used WASPAS with q-rung orthopair fuzzy data. Badalpur and Nurbakhsh (2021) investigated the negative impacts of risks on the project using WASPAS. Rudnik et al. (2021) worked on the selection of improvement projects with ordered fuzzy WASPAS method. Simić et al. (2021) solved the issue of selection of last-mile delivery mode using Picture fuzzy WASPAS method. The selection of eco-friendly vendors was done by Liu et al. (2022) under Bipolar complex fuzzy environment with Criteria Importance Through Intercriteria Correlation (CRITIC)-WASPAS model. Senapati and Chen (2022) utilized picture fuzzy WASPAS technique for solving MCDM issues. Handayani et al. (2023) used WASPAS method for selection of online English course. Assis et al. (2023) applied WASPAS tool to select appropriate helicopters for aerial activities. Arisantoso et al. (2023) used WASPAS method for webcam selection.

3 Preliminaries

Definition 1 (Atanassov, 1986).

An IFS ζ on Γ is described by $φ = {⟨ y_{i}, α (y_{i}), β (y_{i}) ⟩ | y_{i} \in Γ}$ , $α, β : Γ \to [0, 1]$ being the membership and non-membership functions respectively, satisfying $0 ⩽ α (y_{i}) + β (y_{i}) ⩽ 1$ . Also we use $π (y_{i}) = 1 - α (y_{i}) - β (y_{i})$ . An IFS ζ transforms to an IF number (IFN) if Γ contains only one element and we write $φ = ⟨ α, β ⟩$ , for $α, β \in [0, 1]$ and $0 ⩽ α + β ⩽ 1$ .

Definition 2 (Deb et al., 2022).

Consider an IFN $φ = ⟨ α, β ⟩$ . Then: $\begin{array}{r} \begin{aligned} s (φ) = \frac{1}{2} (1 + α - β), \\ a (φ) = α + β \end{aligned} \end{array}$ (1) are known as the accuracy and score values of φ, where $s (φ) \in [0, 1]$ and $a (φ) \in [0, 1]$ . For the IFNs $φ_{1} = ⟨ α_{1}, β_{1} ⟩$ and $φ_{2} = ⟨ α_{2}, β_{2} ⟩$ , a ranking rule is: (i)

$φ_{1} ≻ φ_{2} |\begin{array}{l} s (φ_{1}) > s (φ_{2}), \\ s (φ_{1}) = s (φ_{2}), a (φ_{1}) > a (φ_{2}), \end{array}$

(ii)

$ζ_{1} = ζ_{2} | S_{ζ_{1}} = S_{ζ_{2}}$ , $A_{ζ_{1}} = A_{ζ_{2}}$ .

Definition 3 (Deb et al., 2022).

For the IFNs $φ_{1} = ⟨ α_{1}, β_{1} ⟩$ and $φ_{2} = ⟨ α_{2}, β_{2} ⟩$ , the basic operations are:

(i)
$φ_{1}^{c} = ⟨ β_{1}, α_{1} ⟩$ ,
(ii)
$φ_{1} \oplus φ_{2} = ⟨ α_{1} + α_{2} - α_{1} α_{2}, β_{1} β_{2} ⟩$ ,
(iii)
$φ_{1} \otimes φ_{2} = ⟨ α_{1} α_{2}, β_{1} + β_{2} - β_{1} β_{2} ⟩$ ,
(iv)
$λ φ_{1} = ⟨ 1 - {(1 - α_{1})}^{λ}, β_{1}^{λ} ⟩$ $(λ > 0)$ ,
(v)
$φ_{1}^{λ} = ⟨ α_{1}^{λ}, 1 - {(1 - β_{1})}^{λ} ⟩$ $(λ > 0)$ .

Definition 4 (Deb et al., 2022).

Assume $φ_{t} = ⟨ α_{t}, β_{t} ⟩$ , $t = 1, 2, \dots, l$ be IFNs. Then IFWA and IFWG operators are given respectively by $\begin{aligned} IFWA (φ_{1}, φ_{2}, \dots, φ_{n}) = ⨁_{t = 1}^{l} w_{t} φ_{t} = ⟨ 1 - \prod_{t = 1}^{l} {(1 - α_{t})}^{w_{t}}, \prod_{t = 1}^{l} β_{t}^{w_{t}} ⟩, \end{aligned}$ (2) $\begin{aligned} IFWG (φ_{1}, φ_{2}, \dots, φ_{n}) = ⨂_{t = 1}^{l} ζ_{t}^{w_{t}} = ⟨ \prod_{t = 1}^{l} α_{t}^{w_{t}}, 1 - \prod_{t = 1}^{l} {(1 - β_{t})}^{w_{t}} ⟩, \end{aligned}$ (3) where $w_{t}$ is the weight of $φ_{t}$ , with $\sum_{t = 1}^{l} w_{t} = 1$ , $w_{t} \in [0, 1]$ .

4 Consensus WASPAS Method

The weighted sum model (WSM) and weighted product model (WPM) were combined by Zavadskas et al. (2012) to develop a unique utility degree-based MCDM method referred to as WASPAS. This methodology was designed to deal with a variety of realistic decision-making concerns. WASPAS allows decision-makers to flexibly assign weights to criteria based on their relative importance, reflecting the preferences and priorities of the decision-maker. It considers multiple criteria simultaneously, enabling a comprehensive assessment of alternatives based on different dimensions or factors. The method aggregates the performance scores using the weighted sum product approach, which takes into account the interdependencies among criteria and the performance of alternatives. WASPAS method provides a transparent decision-making process, as it allows decision-makers to clearly understand how the final scores are calculated and how each criterion contributes to the overall evaluation. Unfortunately, WASPAS model fails to deal with the “consensus-reaching process” for experts. To tackle this, we present a consensus WASPAS methodology with IF data. The procedural steps of the proposed consensus-based decision-making model are as follows:

Step 1: Construct the initial IF decision matrices.

Assume that m is the number of alternatives $Q_{k}$ $(k = 1, 2, \dots, p)$ and n is the number of criteria $T_{t}$ $(t = 1, 2, \dots, q)$ connected with a group decision-making issue in which each alternative is evaluated by the decision-makers $E_{r}$ $(r = 1, 2, \dots, l)$ under the IF environment. Consider that the initial findings examined by the decision-makers are depicted as the IF decision matrices $M_{r} = {[φ_{r}^{(k t)}]}_{p \times q} = {[⟨ α_{r}^{(k t)}, β_{r}^{(k t)} ⟩]}_{p \times q}$ .

Step 2: Obtain the aggregated IF decision matrix by employing the IFWA (or IFWG) operator.

The aggregated IF decision matrix is ${[φ^{(k t)}]}_{p \times q} = {[⟨ α^{(k t)}, β^{(k t)} ⟩]}_{p \times q}$ , where: $\begin{array}{r} \begin{aligned} φ^{(k t)} & = IFWA (φ_{1}^{(k t)}, φ_{2}^{(k t)}, \dots, φ_{l}^{(k t)}) = ⨁_{r = 1}^{l} (δ_{r} φ_{r}^{(k t)}) \\ or φ^{(k t)} & = IFWG (φ_{1}^{(k t)}, φ_{2}^{(k t)}, \dots, φ_{l}^{(k t)}) = ⨂_{r = 1}^{l} {(φ_{r}^{(k t)})}^{δ_{r}}, \end{aligned} \end{array}$ (4) where $δ_{r}$ is the weight of $E_{r}$ .

Step 3: Find the consensus degree of each decision-maker.

Utilizing the fact that the correlation measure is capable of describing the similarity degree between various opinions, we define the correlation measure $ψ_{t}^{(r)}$ of the decision-maker $E_{r}$ under the criterion $T_{t}$ in this way: $\begin{aligned} ψ_{t}^{(r)} = \frac{\sum_{k = 1}^{p} [\begin{array}{l} (\frac{{Dist}_{k t}^{(r)}}{{Dist}_{t}^{(r)}} - \frac{1}{m} \sum_{k = 1}^{p} \frac{{Dist}_{k t}^{(r)}}{{Dist}_{t}^{(r)}}) \\ \times (\frac{{Dist}_{k t}}{{Dist}_{t}} - \frac{1}{m} \sum_{k = 1}^{p} \frac{{Dist}_{k t}}{{Dist}_{t}}) \end{array}]}{(\begin{array}{l} \sqrt{\sum_{k = 1}^{p} {(\frac{{Dist}_{k t}^{(r)}}{{Dist}_{t}^{(r)}} - \frac{1}{m} \sum_{k = 1}^{p} \frac{{Dist}_{k t}^{(r)}}{{Dist}_{t}^{(r)}})}^{2}} \\ \times \sqrt{\sum_{k = 1}^{p} {(\frac{{Dist}_{k t}}{{Dist}_{t}} - \frac{1}{m} \sum_{k = 1}^{p} \frac{{Dist}_{k t}}{{Dist}_{t}})}^{2}} \end{array})} \\ (t = 1, 2, \dots, q; r = 1, 2, \dots, l), \end{aligned}$ (5) where $\begin{aligned} φ_{r}^{(k t) (+)} = ⟨ max_{k} α_{r}^{(k t)}, min_{k} β_{r}^{(k t)} ⟩, φ_{r}^{(k t) (-)} = ⟨ min_{k} α_{r}^{(k t)}, max_{k} β_{r}^{(k t)} ⟩, \\ φ^{(k t) (+)} = ⟨ max_{k} α^{(k t)}, min_{k} β^{(k t)} ⟩, φ^{(k t) (-)} = ⟨ min_{k} α^{(k t)}, max_{k} β^{(k t)} ⟩, \\ {Dist}_{k t}^{(r)} = Dist (φ_{r}^{(k t)}, φ_{r}^{(k t) (+)}), {Dist}_{t}^{(r)} = Dist (φ_{r}^{(k t) (+)}, φ_{r}^{(k t) (-)}), \\ {Dist}_{k t} = Dist (φ^{(k t) (+)}, φ^{(k t)}), {Dist}_{t} = Dist (φ^{(k t) (+)}, φ^{(k t) (-)}) . \end{aligned}$

Next, the consensus degree $ρ^{(r)}$ of the decision-maker $E_{r}$ can be defined as: $\begin{matrix} ρ^{(r)} = \frac{1}{q} \sum_{t = 1}^{q} ψ_{t}^{(r)} (r = 1, 2, \dots, l) . \end{matrix}$ (6)

It can be verified that $- 1 ⩽ ρ^{(r)} ⩽ 1$ . The greater value $ρ^{(r)}$ means the stronger consensus degree of the decision-maker $E_{r}$ in the group. If ρ denotes the minimum consensus degree, then $ρ^{(r)} ⩾ ρ$ needs to be attained. When $ρ^{(r)} < ρ$ , the FF decision matrices from Step 1 should be modified until $ρ^{(r)} ⩾ ρ$ is obtained for all decision-makers.

Step 4: Normalize the aggregated IF decision matrix.

Suppose that the normalized aggregated IF decision matrix is ${[{\tilde{φ}}^{(k t)}]}_{p \times q} = {[⟨ {\tilde{α}}^{(k t)}, {\tilde{β}}^{(k t)} ⟩]}_{p \times q}$ , where: $\begin{matrix} {\tilde{φ}}^{(k t)} = \{\begin{array}{ll} ⟨ α^{(k t)}, β^{(k t)} ⟩, & if C_{t} is beneficial criterion, \\ ⟨ β^{(k t)}, α^{(k t)} ⟩, & if C_{t} is non-beneficial criterion . \end{array} \end{matrix}$ (7)

Step 5: Estimate the IF “relative significance degree” (RSD) for every alternative.

Suppose $ϑ_{t}$ is the weight of $T_{t}$ ( $t = 1, 2, \dots, q$ ) with $\sum_{t = 1}^{q} ϑ_{t} = 1$ and $0 ⩽ ϑ_{t} ⩽ 1$ .

Step 5.1: The IF-RSD of $Q_{k}$ using WSM is calculated as: $\begin{matrix} \overline{RSD} (Q_{k}) = ⨁_{t = 1}^{q} (ϑ_{t} {\tilde{φ}}^{(k t)}) . \end{matrix}$ (8)

The IF-RSD of $Q_{k}$ using WPM is calculated as: $\begin{matrix} \overline{\overline{RSD}} (Q_{k}) = ⨂_{t = 1}^{q} {({\tilde{φ}}^{(k t)})}^{ϑ_{t}} . \end{matrix}$ (9)

Step 5.2: The overall IF significance degree of $Q_{k}$ is calculated by: $\begin{matrix} η_{k} = (w \overline{RSD} (Q_{k})) \tilde{\oplus} ((1 - w) \overline{\overline{RSD}} (Q_{k})) (k = 1, 2, \dots, p), \end{matrix}$ (10) or $\begin{matrix} η_{k} = {(\overline{RSD} (Q_{k}))}^{w} \tilde{\otimes} {(\overline{\overline{RSD}} (Q_{k}))}^{(1 - w)} (k = 1, 2, \dots, p) . \end{matrix}$ (11) Here, $w \in [0, 1]$ . For $w = 1$ , and $w = 0$ , WASPAS reduces to WSM and WPM, respectively.

Step 6: Compute the scores of the IFNs $η_{k}$ $(k = 1, 2, \dots, p)$ .

Step 7: Generate the ranking order of alternatives and chose the best option.

5 Case Study & Solution

A. Problem Description

In order to avoid face-to-face interactions, the COVID-19 pandemic has forced many educational institutions to quickly move from traditional attendance-based education to online distance learning. Online distance learning can be either synchronous or asynchronous, depending on the mechanism of delivery. During any pandemic situation like Covid-19, it is very essential for an educational institution to select an appropriate OSS-LMS package to manage administration, monitoring, reporting of online classes and training programs, create a virtual classroom where teachers can interact with their students and conduct learning activities online. Suppose an educational institution wants to select an efficient OSS-LMS package out of four OSS-LMS packages: ATutor (Q1) (Graf and List, 2005; Abdullateef et al., 2016a), eFront (Q3) (Abdullateef et al., 2016b), Moodle (Q2) (Caminero et al., 2013; van Rooij, 2011; Abdullateef et al., 2016a, 2016b; Graf and List, 2005 ), and Sakai (Q4) (Caminero et al., 2013; van Rooij, 2011; Abdullateef et al., 2016a, 2016b; Graf and List, 2005). Their information is given in Table 2.

Table 2
Description of the OSS-LMS alternatives.

OSS-LMS Depiction

ATutor ( $Q_{1}$ ) ATutor is an open-source LMS designed for flexibility and convenience. Administrators can easily install or update the software, customize templates for a unique look and feel, and extend functionality with innovative modules. (http://www.atutor.ca/).

MOODLE ( $Q_{2}$ ) Moodle is the most popular open-source LMS, offering teachers, administrators, and students a robust, secure, and integrated system for learning environments. (Moodle.org).

eFront ( $Q_{3}$ ) eFront LMS provides the best open-source solutions for e-learning, with a flexible, powerful, efficient, and fully functional structure. (http://www.efrontlearning.net/).

Sakai ( $Q_{4}$ ) Sakai is an open-source LMS that provides a flexible and versatile platform for teaching, training, research, and other collaborations. It is constantly evolving based on the needs of faculty, students, and organizations. (https://sakaiproject.org/).

OSS-LMS	Depiction
ATutor ( $Q_{1}$ )	ATutor is an open-source LMS designed for flexibility and convenience. Administrators can easily install or update the software, customize templates for a unique look and feel, and extend functionality with innovative modules. (http://www.atutor.ca/).
MOODLE ( $Q_{2}$ )	Moodle is the most popular open-source LMS, offering teachers, administrators, and students a robust, secure, and integrated system for learning environments. (Moodle.org).
eFront ( $Q_{3}$ )	eFront LMS provides the best open-source solutions for e-learning, with a flexible, powerful, efficient, and fully functional structure. (http://www.efrontlearning.net/).
Sakai ( $Q_{4}$ )	Sakai is an open-source LMS that provides a flexible and versatile platform for teaching, training, research, and other collaborations. It is constantly evolving based on the needs of faculty, students, and organizations. (https://sakaiproject.org/).

In this study, four OSS-LMS are being considered as alternatives and will be evaluated based on fifteen criteria. The criteria and their corresponding citations, arranged by publication year within each criterion, are as follows: Activity Tracking (T1): (Graf and List, 2005; Arh and Blazic, 2007; Srdevic et al., 2012; Abdullateef et al., 2016a); Reliability (T2): (Jadav and Sonar, 2011; Srdevic et al., 2012; Abdullateef et al., 2016b); Course Development (T3): (Graf and List, 2005; Arh and Blazic, 2007; Srdevic et al., 2012; Abdullateef et al., 2016a); Assessment (T4): (Arh and Blazic, 2007; Graf and List, 2005; Srdevic et al., 2012; Abdullateef et al., 2016a); Backup and Recovery (T5): (Arh and Blazic, 2007; Srdevic et al., 2012; Abdullateef et al., 2016a); Error Reporting (T6): (Jadav and Sonar, 2011; Abdullateef et al., 2016a); Efficiency (T7): (Abdullateef et al., 2016b); DBMS standards (T8) (Jadav and Sonar, 2011; Abdullateef et al., 2016b); OS compatibility (T9) (Jadav and Sonar, 2011; Abdullateef et al., 2016b); IMS LIP (T10) (Arh and Blazic, 2007; Srdevic et al., 2012; Abdullateef et al., 2016a); AICC Computer managed Instruction (T11) (Arh and Blazic, 2007; Srdevic et al., 2012; Abdullateef et al., 2016a); Authentication (T12) (Arh and Blazic, 2007; Graf and List, 2005; Srdevic et al., 2012; Abdullateef et al., 2016b); Authorization (T13) (Arh and Blazic, 2007; Srdevic et al., 2012; Abdullateef et al., 2016a); Troubleshooting, maintenance and upgrading (T14) (Caminero et al., 2013; Abdullateef et al., 2016b); and Communication synchronous and asynchronous (T15) (Arh and Blazic, 2007; Graf and List, 2005; Srdevic et al., 2012; Abdullateef et al., 2016a).

Details of the criteria based on which OSS-LMS packages are to be evaluated are:

1) Activity Tracking (T1):

Monitoring students’ learning activities is a part of activity tracking in the classroom. The reports are meant to offer the instructor a sense of what occurs in a pedagogically important course. Progress reports are part of course analysis, which often also includes time stamps for the activities’ occurrences. The handling level of the course and participants may be checked by the tutor. The first and last login dates and times are also visible to the course tutor. A tutor can view the amount of time spent on the course or other activities for specific pupils.

2) Reliability (T2):

The software can function continuously without crashing. Software package reliability refers to its capacity to operate consistently under particular circumstances without crashing. The degree of fault tolerance for the software is evaluated using consistency. The number of crashes during a certain task’s execution can also be monitored to gauge dependability.

3) Course Development (T3):

The organization of the course using a web interface satisfies modifying the course outline, the curriculum, the inclusion of customization of the student tools, the communication tools, etc. The course’s content and structure are both easily editable by the author. Content navigation tools are generated automatically by the system. A single zip package may be used to upload and download HTML pages, pictures, and Flash videos. Links must be built between content pages, between courses, and to student tools. The content and course format may both be readily changed by the instructor. The content navigation can be generated automatically by the system.

4) Assessment (T4:)

The test’s questions and format may be simply created or modified by the tutor. The system enables the learner to evaluate themselves. This feature allows the student to evaluate themselves. The method offers online evaluation. The mechanism makes student transcripts available. A quiz editor is provided by the system. With assessment, the tutor has the option to put the student to the test in a variety of ways. several testable possibilities. The test’s questions and content format may simply be edited by the author. Features automated evaluation of the reliability of the test questions and the capacity to import tests from other programs and systems.

5) Backup and recovery (T5):

The software package is capable of providing backup and recovery features. The DBMS backup and recovery subsystem is in charge of recovery. For example, if the computer system breaks in the middle of a complicated update transaction, the recovery subsystem is in charge of restoring the database to the condition it was in before the transaction began. The recovery time target is the maximum amount of time required to get your learning management system back up and running after a failure. When it comes to judging how secure your computer systems are, the RTO is by far the most disregarded criterion. “Are we backed up?” business owners frequently inquire. Usually, the response is “yes”.

6) Error reporting (T6):

One of the most significant requirements is the software package’s error reporting and messaging capability. Sometimes software suffers from various errors or flaws, and prompt reporting of those errors is critical for further resolution in the run time. Error reporting always makes a system possible for smooth processing, which is extremely important for an LMS system since at the time of online assessment, such reporting and rapid remedy are always useful for students and teachers.

7) Efficiency (T7):

Since everyone learns differently, an effective LMS should provide choices for configuring accessibility, display settings, and demonstrating methods in a reasonable period to meet a wide range of courses, learning styles, and accessibility demands. Keep an eye out for an LMS that can be easily utilized for training, learning, and evaluation all at once. The major factor enabling the software package to deliver results in a suitable time is data size.

8) DBMS standards (T8):

A learning management system (LMS) is software used to offer and administer educational courses. It is a client-server system that is often web-based and used to manage student enrollment, course content distribution, test and assignment administration, and associated record keeping. It keeps a record of all information pertaining to students, including their tuition and financial obligations, academic performance, use of school-provided transportation, and frequent attendance at libraries, labs, computer labs, and other facilities. Database applications that are often used include MS-Access, MS-SQL, MS-Excel, Oracle, DB2, Informix, Sybase, MySQL, and Ingrace.

9) OS compatibility (T9):

When using the LMS system, package compatibility with the operating systems MS Windows, Novell, Unix, Linux, and MAC is crucial. When switching from one OS to another, it is necessary to also switch modules, quizzes, courses, etc. OS compatibility is crucial in this circumstance.

10) IMS LIP (T10):

A specification for a common method of storing data on learners is the IMS Learner Information Package (LIP). LIP is made to make it possible to transfer learner data between different software programs, including their current progress and rewards. The Centre for Recording Achievement and CETIS have since modified the LIP standard for usage within the UK HE Sector, resulting in a mapping of the UK HE Transcript to LIP so that crucial student data may be transmitted electronically.

11) AICC Computer managed Instruction (T11):

“Computer Managed Instruction” is referred to as CMI. CMI is a general abbreviation that may be used to describe any type of computer-based learning in that environment. eLearning developers providing less support: AICC is still supported at a basic level by the majority, of course, authoring tools and learning management systems, although instructional designers and course developers are increasingly adopting more recent e-learning standards. The mobility of a course across different CMI learning environments and the communication between a lesson and the learning environment are all covered in the CMI standard.

12) Authentication (T12):

To prevent replay attacks, common security procedures concentrate on how login credentials and subsequent tokens are handled. Application security includes controls over user behaviour and data privacy. The controlling of verification credentials and successive tokens is the main focus of standard security procedures to stop replay attacks.

13) Authorization (T13):

Following successful authentication, authorization processes determine what the user is permitted to do. The majority of web application logging and monitoring is handled by the application framework. Systems that allow anonymous users must be strengthened to validate every user input.

14) Troubleshooting, maintenance and upgrading (T14):

Average independent code module sizes are usually advantageous. The level of module independence can be determined by specifying whether groups or sub-modules must be installed together, even if only a portion is required. The software package’s maximum number of concurrent users it can support should also be noted. Additionally, the software should have the capability to divide into multiple application tiers that can be distributed across multiple servers, as well as the ability to distribute modules across these servers. The software’s ability to be altered is referred to as maintainability, and modifications may include corrections, enhancements, or adaptations of the program to accommodate changes in the environment, requirements, and functional requirements. Measuring maintainability measures in a constrained experimental environment is challenging; they require extensive real-world testing.

15) Communication synchronous and asynchronous (T15):

The LMS emphasizes asynchronous and synchronous communication, mostly in the form of chat rooms and threaded discussion boards. Discussion forums are the major threaders of asynchronous communication. Email communication is crucial in a learning setting. Creators can converse with and observe who is within. Students have access to several discussion platforms for information sharing. Chat rooms, audio conferences, and/or video conferencing are the principal uses of synchronized communications. Wherever uncertainty has to be cleared up or for any other reason, online dialogue between students and instructors is always a smart alternative. The technology allows for the download of all chatroom statistics. Through this device, audio and video conferencing are also possible.

B. Problem Solution

A team of four decision-making specialists was constituted to select the best option among the considered OSS-LMS packages. The linguistic variables and their accompanying IFNs were defined by experts in Table 3. Table 4 gives the IF linguistic decision matrix.

Table 3

Linguistic ratings.

LVs	FFNs
Very Very High (VVH)	$(0.9, 0.1)$
Very High (VH)	$(0.8, 0.15)$
High (H)	$(0.7, 0.25)$
Medium (M)	$(0.5, 0.45)$
Low (L)	$(0.4, 0.55)$
Very Low (VL)	$(0.2, 0.75)$
Very Very Low (VVL)	$(0.1, 0.9)$

Table 4

IF initial assessment matrix.

	Expert 1				Expert 2				Expert 3				Expert 4
Criteria	Q1	Q2	Q3	Q4	Q1	Q2	Q3	Q4	Q1	Q2	Q3	Q4	Q1	Q2	Q3	Q4
T1	VVH	VH	L	H	VVH	M	L	VVH	VH	H	H	M	M	M	VH	VH
T2	VH	VH	H	VVH	L	M	L	L	VH	H	H	VVH	H	VH	VH	M
T3	H	M	H	VL	M	VH	H	H	M	VVH	L	M	L	VVH	M	H
T4	VH	H	M	VVH	L	M	VVL	M	L	VVL	H	H	M	M	M	H
T5	M	M	H	M	VH	L	H	VH	M	M	VL	L	M	VH	H	H
T6	H	VH	L	H	VVL	M	VH	VVH	VVH	M	H	L	VVL	H	M	H
T7	VVL	VVH	M	M	L	VH	M	VH	L	VVL	M	VH	L	M	M	L
T8	VH	L	VVH	VVH	VL	M	M	L	H	M	L	VL	VVH	VVL	M	M
T9	H	M	M	VVL	H	VH	L	M	M	VH	VH	M	M	VL	VVH	VVH
T10	L	VVH	H	L	M	M	M	L	VL	M	M	L	M	L	VVH	VVH
T11	L	VH	VH	M	0.4	VH	VVH	L	H	L	VVH	M	H	H	VL	VVL
T12	H	L	M	L	H	M	M	L	L	VH	L	M	M	L	VL	VH
T13	H	L	VVH	VH	VH	H	VL	VVH	M	H	H	VH	H	VVH	L	VL
T14	M	M	M	L	M	H	L	H	M	M	M	H	H	M	H	L
T15	VVH	H	H	VH	VVH	VVH	M	H	VL	VVH	H	L	L	VH	M	VH

To obtain a reasonable result, we implement the proposed consensus-based IF-WASPAS model to prioritize the considered options. Assume that DMEs’ weights are respectively 0.20, 0.27, 0.30, and 0.23. Then, the aggregated IF decision matrix (Table 5) is obtained by using the IFWA operator. Assume that the minimum consensus degree is $ρ = 0.25$ . The consensus degree of each expert is calculated based on Eqs. (6) and (7) as: $ρ^{(1)} = 0.253$ , $ρ^{(2)} = 0.496$ , $ρ^{(3)} = 0.451$ , and $ρ^{(4)} = 0.210$ . Since $ρ^{(4)} < 0.25$ , the initial assessments for 4th expert should be modified. In the revised assessment matrix, for the 4th expert, the updated entries are: (Q1, T1): H, (Q4, T1): VVH, (Q3, T12): VVL. Then, the revised aggregated IF decision matrix is constructed with the help of IFWA operator. The consensus degrees are recalculated with the help of Eqs. (5) and (6) as: $ρ^{(1)} = 0.252$ , $ρ^{(2)} = 0.496$ , $ρ^{(3)} = 0.446$ , and $ρ^{(4)} = 0.260$ . Since $ρ^{(r)} ⩾ 0.25$ $(r = 1, 2, 3, 4)$ , desired consensus reaching process has been done.

Table 5

Aggregated matrix.

Criteria	Q1	Q2	Q3	Q4
T1	$⟨ 0.8217, 0.1596 ⟩$	$⟨ 0.6429, 0.3028 ⟩$	$⟨ 0.6215, 0.3220 ⟩$	$⟨ 0.7632, 0.2070 ⟩$
T2	$⟨ 0.7046, 0.2396 ⟩$	$⟨ 0.7107, 0.2352 ⟩$	$⟨ 0.6705, 0.3824 ⟩$	$⟨ 0.7651, 0.2239 ⟩$
T3	$⟨ 0.5292, 0.4190 ⟩$	$⟨ 0.8336, 0.1507 ⟩$	$⟨ 0.5846, 0.3626 ⟩$	$⟨ 0.5745, 0.3715 ⟩$
T4	$⟨ 0.5381, 0.4050 ⟩$	$⟨ 0.4615, 0.4926 ⟩$	$⟨ 0.4973, 0.4549 ⟩$	$⟨ 0.7236, 0.2439 ⟩$
T5	$⟨ 0.6096, 0.3345 ⟩$	$⟨ 0.5746, 0.3690 ⟩$	$⟨ 0.5974, 0.3476 ⟩$	$⟨ 0.6333, 0.3103 ⟩$
T6	$⟨ 0.6263, 0.3603 ⟩$	$⟨ 0.6299, 0.3156 ⟩$	$⟨ 0.6526, 0.2919 ⟩$	$⟨ 0.7255, 0.2473 ⟩$
T7	$⟨ 0.3493, 0.6069 ⟩$	$⟨ 0.6625, 0.3048 ⟩$	$⟨ 0.5000, 0.4500 ⟩$	$⟨ 0.6907, 0.2519 ⟩$
T8	$⟨ 0.7200, 0.2460 ⟩$	$⟨ 0.4064, 0.5494 ⟩$	$⟨ 0.6172, 0.3538 ⟩$	$⟨ 0.5617, 0.4099 ⟩$
T9	$⟨ 0.6067, 0.3414 ⟩$	$⟨ 0.6696, 0.2706 ⟩$	$⟨ 0.7244, 0.2418 ⟩$	$⟨ 0.6116, 0.3657 ⟩$
T10	$⟨ 0.4029, 0.5460 ⟩$	$⟨ 0.6221, 0.3488 ⟩$	$⟨ 0.6882, 0.2831 ⟩$	$⟨ 0.6026, 0.3716 ⟩$
T11	$⟨ 0.5845, 0.3621 ⟩$	$⟨ 0.6947, 0.2491 ⟩$	$⟨ 0.8147, 0.1724 ⟩$	$⟨ 0.3987, 0.5572 ⟩$
T12	$⟨ 0.5846, 0.3626 ⟩$	$⟨ 0.5892, 0.3528 ⟩$	$⟨ 0.4116, 0.5375 ⟩$	$⟨ 0.5588, 0.3841 ⟩$
T13	$⟨ 0.6866, 0.2598 ⟩$	$⟨ 0.7323, 0.2371 ⟩$	$⟨ 0.6319, 0.3357 ⟩$	$⟨ 0.7718, 0.1947 ⟩$
T14	$⟨ 0.5554, 0.3931 ⟩$	$⟨ 0.5644, 0.3840 ⟩$	$⟨ 0.5330, 0.4150 ⟩$	$⟨ 0.5958, 0.3509 ⟩$
T15	$⟨ 0.7182, 0.2709 ⟩$	$⟨ 0.8539, 0.1319 ⟩$	$⟨ 0.6127, 0.3354 ⟩$	$⟨ 0.6898, 0.2543 ⟩$

Since all the considered criteria are of benefit type, normalization is not required. Suppose the weights of criteria are: $ϑ_{1} = 0.06$ , $ϑ_{2} = 0.1$ , $ϑ_{3} = 0.2$ , $ϑ_{4} = 0.1$ , $ϑ_{5} = 0.01$ , $ϑ_{6} = 0.05$ , $ϑ_{7} = 0.1$ , $ϑ_{8} = 0.05$ , $ϑ_{9} = 0.1$ , $ϑ_{10} = 0.05$ , $ϑ_{11} = 0.06$ , $ϑ_{12} = 0.05$ , $ϑ_{13} = 0.01$ , $ϑ_{14} = 0.05$ , and $ϑ_{15} = 0.01$ . The IF-RSD of all alternatives using WSM and WPM are then calculated using Eqs. (8) and (9), respectively. The overall IF significance degrees of alternatives are calculated by Eq. (10) (taking $w = 0.5$ ) and are given as: $\begin{aligned} η_{1} = ⟨ 0.8411, 0.1305 ⟩, η_{2} = ⟨ 0.8900, 0.0887 ⟩, \\ η_{3} = ⟨ 0.8595, 0.1206 ⟩, η_{4} = ⟨ 0.8787, 0.1012 ⟩ . \end{aligned}$ The scores of these FFNs are respectively 0.8553, 0.9006, 0.8695, 0.8887 according to which $Q_{2} ≻ Q_{4} ≻ Q_{3} ≻ Q_{1}$ (“≻” means “superior to”) as preference order with $Q_{2}$ as the most suitable option.

6 Discussions

The discussion section is divided into two parts: (A) sensitivity investigation of criteria weights, and (B) comparison of the suggested approach to currently used methods.

A. Sensitivity analysis of criteria weights

In this section, sensitivity analysis is used to assess the impact of a suitable criterion on the results of the model that has been provided. The term “most significant criterion” is used to denote a criterion with the highest weight value. Saha et al. (2021a, 2023a) used Eq. (12) to calculate the weight ratio. $\begin{matrix} ϑ_{c} = (1 - ϑ_{s}) \times \frac{ϑ_{c}^{0}}{Θ_{c}^{0}} = ϑ_{c}^{0} - α_{c} \times Δ x, \end{matrix}$ (12) where $ϑ_{c}$ – variation in criteria weights,

$ϑ_{s}$ – weight of the most prominent criteria,

$ϑ_{c}^{0}$ – original values of criteria weights,

$Θ_{c}^{0}$ – sum of actual values of modified criteria weights,

$α_{c}$ – weight coefficient of elasticity.

The relative significance of the various values of the criteria weights is demonstrated by $α_{c}$ when we associate the variations with the most pertinent criterion weight. It is possible to calculate $α_{c}$ (Kirkwood, 1997) using the following formula: $\begin{matrix} α_{c} = \frac{ϑ_{c}^{0}}{Θ_{c}^{0}} . \end{matrix}$ (13) Hypotheses of the adopted sensitivity analysis are as follows Kahraman (2002):

(1)

$α_{s}$ (an appropriate criterion weight coefficient of elasticity is provided); and

(2)

The ratio of fluctuating weights remains constant during the SA procedure.

The fluctuation degree applied to a set of weight coefficients is given by a parameter $Δ x$ that is specified in terms of the corresponding weighted elastic coefficient, as indicated in Eq. (14). If the weight of the most important characteristic varies, a constraint should be in place. If this is not done, related weights may turn negative and the restrictions on proportionate weights may be broken. Positive and negative values of the parameter could signify an increase or reduction in the degree of importance, respectively. From the following, we may infer the limit values of $Δ x$ . $\begin{matrix} - ϑ_{s}^{0} ⩽ Δ x ⩽ min {\frac{ϑ_{c}^{0}}{α_{c}}} . \end{matrix}$ (14)

The boundaries and original weights of criteria are established and estimated using the pre-defined parameters. The values of a group of weight coefficients are determined by applying Eqs. (15) and (16): $\begin{aligned} ϑ_{s} = ϑ_{s}^{0} + α_{s} \times Δ x, \end{aligned}$ (15) $\begin{aligned} ϑ_{c} = ϑ_{c}^{0} - α_{c} \times Δ x, \end{aligned}$ (16) where $ϑ_{s}^{0}$ – given weight of the most significant criteria,

$ϑ_{c}^{0}$ – given value of changeable weights.

It should be kept in mind that the revised criteria set mentioned above satisfies the equation $\sum ϑ_{s} + \sum ϑ_{c} = 1$ , which is thought of as the fundamental requirement of the percentage of weight coefficients. The rankings of the alternatives are established taking into consideration the updated criteria weight values. T3 has the most significant weight coefficient, which was determined from the analysis, of $ϑ_{3} = 0.2$ , making it one of the most significant criteria in this study. After that, the weight elasticity values are assessed, and it is found that the weight coefficient’s ( $Δ x$ ) fluctuation bounds fall within the range of $- 0.2 ⩽ Δ x ⩽ 0.8$ . Several criteria weight sets (G1, G2, $\dots$ , G15) are then formed based on restrictions for the change of weight coefficient values of criteria.

The weight sets are split into fifteen groups for the range $- 0.2 ⩽ Δ x ⩽ 0.8$ . The weight coefficients are viewed from various perspectives for each set using Eqs. (15) and (16), and these values are shown in Fig. 1. As a result, several criteria weight sets are applied to determine the alternatives’ final scores, which are shown in Fig. 2. The results of this study showed that alternative Q2 is the best choice. After that, we used the results from various weight sets taken into account by various criteria to determine the SRCC values (Saha et al., 2021a). A “high correlation” between alternative ranks is observed, as indicated by the average SRCC value of 0.9 (Saha et al., 2021b). Therefore, stability of the model is established.

Fig. 1

Various criteria weights sets for sensitivity analysis.

B. Comparative Investigation

Fig. 2

Scores of the alternatives for various sets of criteria weights.

This part aims to provide a comparative analysis of the developed consensus-based IF decision support model with the existing IF decision-making methods, namely IF-TOPSIS (Rouyendegh et al., 2020), IF-MULTIMOORA (Garg and Rani, 2022), IF-EDAS (Mishra et al., 2020), IF-COPRAS (Kumari and Mishra, 2020), and IF-MARCOS (Deb et al., 2022). These methods are applied to solve the addressed selection issue of OSS-LMS package selection. According to the comparison results, the ranking order obtained by these existing methods is $Q_{2} ≻ Q_{1} ≻ Q_{3} ≻ Q_{4}$ . On the other hand, the IF-c-WASPAS method generates the order $Q_{2} ≻ Q_{1} ≻ Q_{3} ≻ Q_{4}$ which is exactly the same.

Some advantages of IF-c-WASPAS are as follows:

The consensus-reaching process for decision-makers is integrated into the introduced model, while the available FF methods (Rouyendegh et al., 2020; Garg and Rani, 2022; Mishra et al., 2020; Kumari and Mishra, 2020; Deb et al., 2022) are unable to rectify the consensus level of experts. As a result, our model lessens decision-making process biases, making the process more significant and logical.

The consensus-reaching process using WASPAS methods offers the following advantages: (i) estimation of values can be achieved with the highest degree of precision, (ii) surpasses WPM and WSM in terms of accuracy, (iii) enables the selection of the optimal choice through the utilization of AOs, unlike other methods that only allow for the selection of the option closest to the ideal answer.

The proposed model is useful for assessing and prioritizing OSS-LMS packages under real-life scenarios when there is a lack of quantitative input information.

7 Conclusions

Adaptive online educational practices are a constantly evolving area of study, notably in academic contexts. It is a unique way for people and organizations to acquire knowledge and to fulfill their demands. These demands can be reached by the use of LMSs that are elements of e-learning. LMSs are software-based applications that incorporate a package of schemes for learning and training online. This LMS has now enhanced a top priority and major projects in enterprises and educational organizations. The number of accessible OSS-LMSs is constantly booming and getting significant projection. OSS is enhancing a choice for every institution, as they are advantageous to users in enabling platforms to be transformed according to user specifications, and because of the minimum costs charged to get a more reliable service, compared to other software like commercial ones that need licensing fees to run, with an added subscription and also a maintenance fee to make the LMS software up-to-date. In answer to mounting demands, software enterprises have been recommending a different types of software packages that can be adapted and fulfill the precise needs of an establishment as well as handmade. The evaluation and adoption of an improper OSS-LMS package unfavourably affect the marketing procedures and duties of the institution. This paper focuses on consensus based IF-WASPAS methodology to help a decision-maker or administrators in the education environment to compare and appraise the OSS-LMS packages and choose the suitable one over certain criteria. The comparison is presented between four OSS-LMSs, namely Moodle, Sakai, ATutor, and eFront. The outcomes show that MOODLE is best for the present study compared to other four OSS-LMSs. As potential future research directions, a combination of subjective and objective weights for expert weight determination can be utilized, taking into account assessments of expert similarity (Saha et al., 2021a, 2021b). To enhance the significance of the model, incorporating measures such as dispersion, uncertainty, and cross-entropy can be explored (Saha et al., 2023a, 2023b). The developed consensus approach can also be integrated with MULTIMOORA, MARCOS, DNMA, COPRAS, and VIKOR methodologies to develop novel techniques. Furthermore, the developed model can be extended to encompass dual hesitant fuzzy sets and probabilistic dual hesitant fuzzy sets.

References

Abdullateef

B.N.

Elias

N.F.

Mohamed

Zaidan

A.A.

Zaidan

B.B.

(2015). Study on open source learning management systems: a survey, profile and taxonomy. Journal of Theoretical and Applied Information Technology, 82, 93–105.

Abdullateef

B.N.

Elias

N.F.

Mohamed

(2016a). Open source learning management system: a comparative study. Journal of Engineering and Applied Sciences, 11(3), 519–522.

Abdullateef

B.N.

Elias

N.F.

Mohamed

Zaidan

A.A.

Zaidan

B.B.

(2016b). An evaluation and selection problems of OSS-LMS packages. SpringerPlus, 5, 248. https://doi.org/10.1186/s40064-016-1828-y.

Adewumi

Misra

Omoregbe

Sanz

L.F.

(2019). FOSSES: framework for open-source software evaluation and selection. Software: Practice and Experiment, 49(5), 780–812.

Al Amoush

A.B.

Sandhu

(2020). Digital learning management systems case study: instructors’ perspective. In: Delello

McWhorter

(Eds.), Disruptive and Emerging Technology Trends Across Education and the Workplace. IGI Global, pp. 143–168. https://doi.org/10.4018/978-1-7998-2914-0.ch006.

Albarrak

A.I.

Aboalsamh

H.A.

Abouzahra

(2010). Evaluating learning management systems for University medical education. In: Proceedings of the International Conference on Education, Management, and Technology, Cairo, Egypt, 2010, pp. 672–677. https://doi.org/10.1109/ICEMT.2010.5657569.

Alturki

Aldraiweesh

(2021). Application of Learning Management System (LMS) during the COVID-19 pandemic: a sustainable acceptance model of the expansion technology approach. Sustainability, 13(19), 10991. https://doi.org/10.3390/su131910991.

Arh

Blazic

B.J.

(2007). A multi-attribute decision support model for learning management systems evaluation. In: First International Conference on the Digital Society, ICDS’07, Guadeloupe, French Caribbean, 2007, p. 11. https://doi.org/10.1109/ICDS.2007.1.

Arisantoso

S.M.H.

Sanwasih

Shalahudin

M.I.

(2023). Multi-criteria decision making using the WASPAS method in Webcam selection decision support systems. International Journal of Informatics and Computer Science, 7(1), 1–10.

10.

Atanassov

K.T.

(1986). Intuitionistic fuzzy sets. Fuzzy Sets and Systems, 20(1), 87–96.

11.

Awang

N.B.

Darus

M.Y.B.

(2012). Evaluation of an open source learning management system: Claroline. Procedia – Social and Behavioral Sciences, 67, 416–426.

12.

Badalpur

Nurbakhsh

(2021). An application of WASPAS method in risk qualitative analysis: a case study of a road construction project in Iran. International Journal of Construction Management, 21(9), 910–918. https://doi.org/10.1080/15623599.2019.1595354.

13.

Bid

Siddique

(2019). Human risk assessment of Panchet Dam in India using TOPSIS and WASPAS Multi-Criteria Decision-Making (MCDM) methods. Heliyon, 5(6), e01956. https://doi.org/10.1016/j.heliyon.2019.e01956.

14.

Buran

Erçek

(2022). Public transportation business model evaluation with Spherical and Intuitionistic Fuzzy AHP and sensitivity analysis. Expert Systems with Applications, 204, 117519.

15.

Büyüközkan

Feyzioğlu

Görçer

(2018). Selection of sustainable urban transportation alternatives using an integrated intuitionistic fuzzy Choquet integral approach. Transportation Research Part D: Transportation and Environment, 58, 186–207.

16.

Çakır

Taş

M.A.

(2023). Circular intuitionistic fuzzy decision making and its application. Expert Systems with Applications, 225, 120076.

17.

Caminero

A.C.

Hernandez

R.D.

Ros

Robles-Gómez

Tobarra

(2013). Choosing the right LMS: a performance evaluation of three open-source LMS. In: 2013 IEEE Global Engineering Education Conference (EDUCON), Berlin, Germany, 2013, pp. 287–294. https://doi.org/10.1109/EduCon.2013.6530119.

18.

Cavus

(2007). The Effects of Using Learning Management Systems on Collaborative Learning for Teaching Programming Languages. Dissertation, Near East University, Graduate School of Applied Sciences, Department of Computer Information Systems, Nicosia, Cyprus. http://docs.neu.edu.tr/library/4956059057.pdf.

19.

Çetin

Işık

A.H.

Güler

İ.

(2010). Learning management system selection with analytic hierarchy process. In: 13th International Conference on Interactive Computer aided Learning (ICL2010), Hasselt, Belgium, pp. 921–926.

20.

Davoudabadi

Mousavi

S.M.

Mohagheghi

(2020). A new last aggregation method of multi-attributes group decision-making based on concepts of TODIM, WASPAS and TOPSIS under interval-valued intuitionistic fuzzy uncertainty. Knowledge and Information Systems, 62, 1371–1391. https://doi.org/10.1007/s10115-019-01390-x.

21.

de Assis

G.S.

dos Santos

Basilio

M.P.

(2023). Use of the WASPAS method to select suitable helicopters for aerial activity carried out by the military police of the state of Rio de Janeiro. Axioms, 12(1), 77. https://doi.org/10.3390/axioms12010077.

22.

Deb

P.P.

Bhattacharya

Chatterjee

Saha

Mishra

A.R.

Ahammad

S.H.

(2022). A decision-making model with intuitionistic fuzzy information for selection of enterprise resource planning systems. In: IEEE Transactions on Engineering Management. https://doi.org/10.1109/TEM.2022.3215608.

23.

Deveci

Canıtez

Gökaşar

(2018). WASPAS and TOPSIS based interval type-2 fuzzy MCDM method for a selection of a car sharing station. Sustainable Cities and Societies, 41, 777–791. https://doi.org/10.1016/j.scs.2018.05.034.

24.

Deveci

Krishankumar

Gokasar

Deveci

R.T.

(2022). Prioritization of healthcare systems during pandemics using Cronbach’s measure based fuzzy WASPAS approach. Annals of Operations Research. https://doi.org/10.1007/s10479-022-04714-3.

25.

Deveci

Erdogan

Pamucar

Kucuksari

Cali

(2023). A rough Dombi Bonferroni based approach for public charging station type selection. Applied Energy, 345, 121258. https://doi.org/10.1016/j.apenergy.2023.121258.

26.

Dong

Feng

(2010). The OWA-based consensus operator under linguistic representation models using position indexes. European Journal of Operations Research, 203(2), 455–463.

27.

Dorfeshan

Mousavi

S.M.

(2020). A novel interval type-2 fuzzy decision model based on two new versions of relative preference relation-based MABAC and WASPAS methods (with an application in aircraft maintenance planning). Neural Computing and Applications, 32, 3367–3385.

28.

Edrees

M.E.

(2013). eLearning 2.0: learning management systems readiness. In: 2013 Fourth International Conference on e-Learning “Best Practices in Management, Design and Development of e-Courses: Standards of Excellence and Creativity”, Manama, Bahrain, 2013, pp. 90–96. https://doi.org/10.1109/ECONF.2013.57.

29.

Filip

F.G.

(2021). Automation and computers and their contribution to human well-being and resilience. Studies in Informatics and Control, 30(4), 5–18.

30.

Garg

Rani

(2022). An efficient intuitionistic fuzzy MULTIMOORA approach based on novel aggregation operators for the assessment of solid waste management techniques. Applied Intelligence, 52, 4330–4363.

31.

Gohain

Chutia

Dutta

(2022). Distance measure on intuitionistic fuzzy sets and its application in decision-making, pattern recognition, and clustering problems. International Journal of Intelligent Systems, 37(3), 2458–2501.

32.

Gokasar

Deveci

Isik

Daim

Zaidan

A.A.

Smarandache

(2023a). Evaluation of the alternatives of introducing electric vehicles in developing countries using Type-2 neutrosophic numbers based RAFSI model. Technological Forecasting and Social Change, 192, 122589. https://doi.org/10.1016/j.techfore.2023.122589.

33.

Gokasar

Pamucar

Deveci

Gupta

B.B.

Martinez

Castillo

(2023b). Metaverse integration alternatives of connected autonomous vehicles with self-powered sensors using fuzzy decision making model. Information Sciences, 642, 119192. https://doi.org/10.1016/j.ins.2023.119192.

34.

Gökmener

Oğuz

Deveci

Göllü

(2023). Site selection for floating photovoltaic system on dam reservoirs using sine trigonometric decision making model. Ocean Engineering, 281, 114820. https://doi.org/10.1016/j.oceaneng.2023.114820.

35.

Gong

Zhang

Forrest

(2015). Two consensus models based on the minimum cost and maximum return regarding either all individuals or one individual. European Journal of Operations Research, 240(1), 183–192.

36.

Graf

List

(2005). An evaluation of open source e-learning platforms stressing adaptation issues. In: Fifth IEEE International Conference on Advanced Learning Technologies (ICALT’05), Kaohsiung, Taiwan, pp. 163–165. https://doi.org/10.1109/ICALT.2005.54. Retrieved December 3, 2007, from http://www.campussource.de/aktuelles/docs/icalt2005.pdf .

37.

Gundogdu

F.K.

Kahraman

(2019). Extension of WASPAS with spherical fuzzy sets. Informatica, 30(2), 269–292.

38.

Handayani

Heriyani

Septian

Alexander

A.D.

(2023). Multi-criteria decision making using the WASPAS method for online english course selection. Jurnal Teknoinfo, 17(1), 260–270.

39.

Herrera

Herrera-Viedma

Verdegay

J.L.

(1996). A model of consensus in group decision making under linguistic assessments. Fuzzy Sets and Systems, 78(1), 73–87.

40.

Herrera-Viedma

Cabrerizo

F.J.

Kacprzyk

Pedrycz

(2014). A review of soft consensus models in a fuzzy environment. Information Fusion, 17, 4–13.

41.

Hezam

I.M.

Mishra

A.R.

Rani

Cavallaro

Saha

Ali

Strielkowski

Štreimikienė

(2022). A hybrid intuitionistic fuzzy-MEREC-RS-DNMA method for assessing the alternative fuel vehicles with sustainability perspectives. Sustainability, 14(9), 5463. https://doi.org/10.3390/su14095463.

42.

Hezam

I.M.

Rani

Mishra

A.R.

Alshamrani

(2023a). An intuitionistic fuzzy entropy-based gained and lost dominance score decision-making method to select and assess sustainable supplier selection. AIMS Mathematics, 8(5), 12009–12039.

43.

Hezam

I.M.

Mishra

A.R.

Rani

Saha

Smarandache

Pamucar

(2023b). An integerated decision support framework using single-valued neutrosophic-MASWIP-COPRAS for sustainability assessment of bioenergy production technologies. Expert System With Applications, 211, 118674. https://doi.org/10.1016/j.eswa.2022.118674.

44.

Hock

S.Y.

Omar

Mahmud

(2015). Comparing the usability and users acceptance of open sources Learning Management System (LMS). International Journal of Scientific Research, 5(4), 1–5.

45.

Hultin

(2007). Learning Management Systems (LMS): a review. Retrieved December 17, 2007, from http://se2.isn.ch/serviceengine/FileContent?serviceID=18&fileid=09FD0FDC-723F-5E63-CCDB6AFCDA98BCC8&lng=en.

46.

Işik

A.H.

İnce

Yiğit

(2015). A fuzzy AHP approach to select learning management system. International Journal of Computer Theory and Engineering, 7(6), 499–502.

47.

Ivanović

Saha

Stević

Ž.

Puška

Zavadskas

E.K.

(2022). Selection of truck mixture concrete pump using novel MEREC-DNMARCOS model. Archives of Civil and Mechanical Engineering, 22, 173. https://doi.org/10.1007/s43452-022-00491-9.

48.

Jadav

Sonar

(2011). Framework for evaluation and selection of the software packages: a hybrid knowledge based system approach. Journal of System Software, 84(8), 1394–1407.

49.

Kahraman

Onar

S.C.

Oztaysi

Ilbahar

(2019). Selection among GSM operators using pythagorean fuzzy WASPAS method. Journal of Multi-Valued Logic & Soft Computing, 33(4–5), 459–469.

50.

Kahraman

Y.R.

(2002). Robust Sensitivity Analysis for Multi-Attribute Deterministic Hierarchical Value Models (No. AFIT/GOR/ENS/02-10). Master’s thesis, Air Force Institute of Technology, Wright-Patterson AFB, OH 45433, United States

51.

Karagöz

Oral Güney

L.Ö

Kaya

O.H.

Tecim

(2017). LMS selection process for effective distance education system in organizations. KnE Social Sciences, 1(2), 343–356.

52.

Keshavarz-Ghorabaee

Govindan

Amiri

Zavadskas

E.K.

Antuchevičienė

(2019). An integrated type-2 fuzzy decision model based on WASPAS and SECA for evaluation of sustainable manufacturing strategies. Journal of Environmental Engineering and Landscape Management, 27(4), 187–200. https://doi.org/10.3846/jeelm.2019.11367.

53.

Kljun

Vicic

Kavsek

Kavcic

(2007). Evaluating comparisons and evaluations of learning management systems. In: 2007 29th International Conference on Information Technology Interfaces, Cavtat, Croatia, 2007, pp. 363–368. https://doi.org/10.1109/ITI.2007.4283797.

54.

Kirkwood

C.W.

(1997). Strategic Decision Making: Multi-Objective Decision Analysis with Spreadsheets. Duxbury Press, Belmon.

55.

Krishankumar

Subrajaa

L.S.

Ravichandran

K.S.

Kar

Saeid

A.B.

(2019). A framework for multi-attribute group decision-making using double hierarchy hesitant fuzzy linguistic term set. International Journal of Fuzzy Systems, 21, 1130–1143. https://doi.org/10.1007/s40815-019-00618-w.

56.

Krishankumar

Garg

Arun

Saha

Ravichandran

K.S.

Kar

(2021). An integrated decision-making COPRAS approach to probabilistic hesitant fuzzy set information. Complex and Intelligent Systems, 7, 2281–2298.

57.

Kumari

Mishra

A.R.

(2020). Multi-criteria COPRAS method based on parametric measures for intuitionistic fuzzy sets: application of green supplier selection. Iranian Journal of Science and Technology, Transactions in Electrical Engineering, 44, 1645–1662. https://doi.org/10.1007/s40998-020-00312-w.

58.

Liao

H.C.

Z.S.

Zeng

X.J.

D.L.

(2016). An enhanced consensus reaching process in group decision making with intuitionistic fuzzy preference relations. Information Sciences, 329, 274–286.

59.

Liu

Huang

(2020). Consensus reaching process for fuzzy behavioral TOPSIS method with probabilistic linguistic q-rung orthopair fuzzy sets based on correlation measure. International Journal of Intelligent Systems, 35(3), 494–528. https://doi.org/10.1002/int.22215.

60.

Liu

Saha

Mishra

A.R.

Rani

Dutta

Baidya

(2022). A BCF-CRITIC-WASPAS method for green supplier selection with cross-entropy and Archimedean aggregation operators. Journal of Ambient Intelligence and Humanized Computing, 14, 11909–11933. https://doi.org/10.1007/s12652-022-03745-9.

61.

Machado

Tao

(2007). Blackboard vs. moodle: comparing user experience of learning management systems. In: 2007 37th Annual Frontiers In Education Conference – Global Engineering: Knowledge Without Borders, Opportunities Without Passports, Milwaukee, WI, USA, 2007, pp. S4J-7–S4J-12. https://doi.org/10.1109/FIE.2007.4417910.

62.

Mishra

A.R.

Rani

(2018). Interval-valued intuitionistic fuzzy WASPAS method: application in reservoir flood control management policy. Group Decision and Negotiation, 27, 1047–1078. https://doi.org/10.1007/s10726-018-9593-7.

63.

Mishra

A.R.

Singh

R.K.

Motwani

(2019a). Multi-criteria assessment of cellular mobile telephone service providers using intuitionistic fuzzy WASPAS method with similarity measures. Granular Computing, 4, 511–529. https://doi.org/10.1007/s41066-018-0114-5.

64.

Mishra

A.R.

Rani

Pardasani

K.R.

Mardani

(2019b). A novel hesitant fuzzy WASPAS method for assessment of green supplier problem based on exponential information measures. Journal of Cleaner Production, 238, 117901. https://doi.org/10.1016/j.jclepro.2019.117901.

65.

Mishra

A.R.

Sisodia

Pardasani

K.R.

Sharma

(2019c). Multi-criteria IT personnel selection on intuitionistic fuzzy information measures and ARAS methodology. Iranian Journal of Fuzzy Systems, 17(4), 55–68.

66.

Mishra

A.R.

Mardani

Rani

Zavadskas

E.K.

(2020). A novel EDAS approach on intuitionistic fuzzy set for assessment of health-care waste disposal technology using new parametric divergence measures. Journal of Cleaner Production, 272, 122807.

67.

Mishra

A.R.

Rani

Cavallaro

Heza

I.M.

(2023). Intuitionistic fuzzy fairly operators and additive ratio assessment-based integrated model for selecting the optimal sustainable industrial building options. Scientific Reports, 13, 5055. https://doi.org/10.1038/s41598-023-31843-x.

68.

Mohagheghi

Mousavi

S.M.

(2020). D-WASPAS: addressing social cognition in uncertain decision-making with an application to a sustainable project portfolio problem. Cognitive Computation, 12, 619–641. https://doi.org/10.1007/s12559-019-09679-3.

69.

Mukherjee

(2017). Selection of alternative fuels for sustainable urban transportation under multi-criteria intuitionistic fuzzy environment. Fuzzy Information and Engineering, 9, 117–135.

70.

Natarajan

(2015). Evaluation methods for e-learning: an analytical study. SSARS International Journal of Library and Information Sciences, 1(1), 1–14.

71.

Palanisami

Mohan

Ganeshkumar

(2022). A new approach of multi-modal medical image fusion using intuitionistic fuzzy set. Biomedical Signal Processing and Control, 77, 103762. https://doi.org/10.1016/j.bspc.2022.103762.

72.

Pamučar

Sremac

Stević

Ž.

Ćirovic

Tomić

(2019). New multi-criteria LNN WASPAS model for evaluating the work of advisors in the transport of hazardous goods. Neural Computing and Applications, 31, 5045–5068. https://doi.org/10.1007/s00521-018-03997-7.

73.

Petrovas

Bausys

Zavadskas

(2023). Gestalt principles governed fitness function for genetic Pythagorean neutrosophic WASPAS game scene generation. International Journal of Computers Communications & Control, 18(4), 5475. https://doi.org/10.15837/ijccc.2023.4.5475.

74.

Ramesh

V.M.

Ramanathan

(2013). A rubric to evaluate learning management systems. In: Proceedings of 2013 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE), Bali, Indonesia, 2013, pp. 73–77. https://doi.org/10.1109/TALE.2013.6654402.

75.

Rani

Mishra

A.R.

(2020). Multi-criteria weighted aggregated sum product assessment framework for fuel technology selection using q-rung orthopair fuzzy sets. Sustainable Production and Consumption, 24, 90–104. https://doi.org/10.1016/j.spc.2020.06.015.

76.

Rani

Mishra

A.R.

Ansari

M.D.

Ali

(2021). Assessment of performance of telecom service providers using intuitionistic fuzzy grey relational analysis framework (IF-GRA). Soft Computing, 25, 1983–1993. https://doi.org/10.1007/s00500-020-05269-w.

77.

Rasoulzadeh

Edalatpanah

S.A.

Fallah

Najafi

S.E.

(2022). A multi-objective approach based on Markowitz and DEA cross-efficiency models for the intuitionistic fuzzy portfolio selection problem. Decision Making: Applications in Management and Engineering, 5(2), 241–259.

78.

Rouyendegh

B.D.

Yildizbasi

Üstünyer

(2020). Intuitionistic fuzzy TOPSIS method for green supplier selection problem. Soft Computing, 24, 2215–2228. https://doi.org/10.1007/s00500-019-04054-8.

79.

Rudnik

Bocewicz

Kucińska-Landwójtowicz

Czabak-Górska

I.D.

(2021). Ordered fuzzy WASPAS method for selection of improvement projects. Expert System with Applications, 169, 114471. https://doi.org/10.1016/j.eswa.2020.114471.

80.

Saha

Dutta

Kar

(2021a). Some new hybrid hesitant fuzzy weighted aggregation operators based on Archimedean and Dombi operations for multi-attribute decision making. Neural Computing and Applications, 33, 8753–8776. https://doi.org/10.1007/s00521-020-05623-x.

81.

Saha

Senapati

Yager

R.R.

(2021b). Hybridizations of Generalized Dombi operators and Bonferroni mean operators under Dual probabilistic linguistic environment for group decision-making. International Journal of Intelligent Systems, 36(11), 6645–6679.

82.

Saha

Mishra

A.R.

Rani

Hezam

I.M.

Cavallaro

(2022). A q-rung orthopair fuzzy FUCOM double normalization based multi-aggregation method for healthcare waste treatment method selection. Sustainability, 14(7), 4171. https://doi.org/10.3390/su14074171.

83.

Saha

Pamucar

Gorcun

O.F.

Mishra

A.R.

(2023a). Warehouse site selection for the automotive industry using a Fermatean fuzzy based decision-making approach. Expert System with Applications, 211), 118497. https://www.sciencedirect.com/science/article/abs/pii/S0957417422015809 .

84.

Saha

Simic

Dabic-Miletic

Senapati

Yager

R.R.

Devechi

(2023b). Evaluation of propulsion technologies for sustainable road freight distribution using a dual probabilistic linguistic group decision-making approach. In: IEEE Transactions on Engineering Management. https://doi.org/10.1109/TEM.2023.3253300. https://ieeexplore.ieee.org/document/10077108 .

85.

Santiago

B.J.

Olivares Ramírez

J.M.

Rodríguez-Reséndiz

Dector

García

González-Durán

J.E.E.

Sánchez

F.F.

(2020). Learning management system-based evaluation to determine academic efficiency performance. Sustainability, 12(10), 4256. https://doi.org/10.3390/su12104256.

86.

Schitea

Deveci

Iordache

Bilgili

Akyurt

I.Z.

Iordache

(2019). Hydrogen mobility roll-up site selection using intuitionistic fuzzy sets based WASPAS, COPRAS and EDAS. International Journal of Hydrogen Energy, 44(16), 8585–8600. https://doi.org/10.1016/j.ijhydene.2019.02.011.

87.

Semenas

Bausys

Zavadskas

E.K.

(2021). A novel environment exploration strategy by m-generalised q-neutrosophic WASPAS. Studies in Informatics and Control, 30(3), 19–28.

88.

Senapati

Chen

(2022). Picture fuzzy WASPAS technique and its application in multi-criteria decision-making. Soft Computing, 26, 4413–4421. https://doi.org/10.1007/s00500-022-06835-0.

89.

Senapati

Simic

Saha

Dobrodolac

Rong

Tirkolaee

E.B.

(2023). Intuitionistic fuzzy power Aczel-Alsina model for prioritization of sustainable transportation sharing practices. Engineering Applications of Artificial Intelligence, 119, 105716.

90.

Sharma

Pradhan

(2020). Investigation of machinability criteria during micro-abrasive finishing of SUS-304L steel using fuzzy combined with WASPAS approach. Journal of the Brazilian Society of Mechanical Science and Engineering, 42, 116. https://doi.org/10.1007/s40430-020-2198-5.

91.

Srdevic

Pipan

Srdevic

Arh

(2012). AHP supported evaluation of LMS quality. In: Proceedings of the International Workshop on the Interplay between User Experience Evaluation and Software Development, Copenhagen, Denmark, 2012, pp. 52–57.

92.

Stanujkic

Karabasevic

(2018). An extension of the WASPAS method for decision-making problems with intuitionistic fuzzy numbers: a case of website evaluation. Operations Research in Engineering Sciences: Theory and Applications, 1(1), 29–39. https://doi.org/10.31181/oresta19012010129s.

93.

Tugrul

(2022). Evaluation of papers according to offset print quality: the intuitionistic fuzzy based multi criteria decision making mechanism. Pigment & Resin Technology. https://doi.org/10.1108/PRT-04-2022-0059.

94.

Ulutaş

Stanujkic

Karabasevic

Popovic

Zavadskas

E.K.

Smarandache

Brauers

W.K.M.

(2021). Developing of a novel integrated MCDM MULTIMOOSRAL approach for supplier selection. Informatica, 32(1), 145–161.

95.

van Rooij

S.W.

(2011). Higher education sub-cultures and open sources adoption. Computer Education, 57(1), 1171–1183.

96.

van Rooij

S.W.

(2012). Open sources learning management systems: a predictive model for higher education. Journal of Computer Assisted Learning, 28(2), 114–125.

97.

Wang

Zhang

Sun

(2018). Assessing customer satisfaction of urban rail transit network in Tianjin based on intuitionistic fuzzy group decision model. Discrete Dynamics in Nature and Society, 2018, 4205136. https://doi.org/10.1155/2018/4205136.

98.

Waynet Inc. (2007). COL LMS open source. Under License of Commonwealth of Learning. Retrieved December 4, 2007, from http://www.col.org/colweb/webdav/site/myjahiasite/shared/docs/03LMSOpenSource.pdf .

99.

Z.B.

J.P.

(2016). Managing consistency and consensus in group decision making with hesitant fuzzy linguistic preference relations. Omega, 65, 28–40.

100.

X.L.

Liao

H.C.

(2019). A consensus-based probabilistic linguistic gained and lost dominance score method. European Journal of Operations Research, 272(3), 1017–1027.

101.

Wyles

(2007). Open source software and the New Zealand education system: a response to roy. Journal of Distance Learning, 10(1), 36–41.

102.

Yan

Rong

Huang

(2022). A hybrid intuitionistic fuzzy group decision framework and its application in urban rail transit system selection. Mathematics, 10(12), 2133.

103.

Yuan

Yang

(2021). Dynamic multiple criteria group decision-making method based on intuitionistic fuzzy information. Journal of Control and Decision, 9(4), 397–406.

104.

Zadeh

L.A.

(1965). Fuzzy sets. Information and Control, 8, 338–353.

105.

Zaidan

A.A.

Zaidan

B.B.

Al-Haiqi

Kiah

M.L.M.

Hussain

Abdulnabi

(2015). Evaluation and selection of open source EMR software packages based on integrated AHP and TOPSIS. Journal of Biomedical Informatics, 53, 390–404.

106.

Zavadskas

E.K.

Turskis

Antucheviciene

Zakarevicius

(2012). Optimization of weighted aggregated sum product assessment. Electronics and Electrical Engineering, 122(6), 3–6. https://doi.org/10.5755/j01.eee.122.6.1810.

107.

Zhang

B.W.

Liang

H.M.

Zhang

G.Q.

(2018). Reaching a consensus with minimum adjustment in MAGDM with hesitant fuzzy linguistic term sets. Information Fusion, 42, 12–23.