Sage Journals: Discover world-class research

Abstract

The need for qualitative data to support the evaluation process using the DIVAYANA (Description-Input-Verification-Action-Yack-Analysis-Nominate-Actualization) evaluation model is very important to obtain in-depth information about the effectiveness of e-learning platform utilization in ICT Vocational Schools. Those qualitative data are used to complete the evaluation process for the eight components of the DIVAYANA evaluation model. The eight components included Description component, Input component, Verification component, Action component, Yack component, Analysis component, Nominate component, and Actualization component. The main purpose of this study was to show the qualitative data needed in each evaluation component of the DIVAYANA evaluation model. The tools used to obtain qualitative data on the eight components of the DIVAYANA evaluation model were interview guidelines, checklists, and literature studies. This study approach was qualitative. The technique used to test the qualitative data validity in this study was theoretical triangulation. The findings of this study were the qualitative data needed in each component of the DIVAYANA evaluation model to show the effectiveness of e-learning platform utilization in ICT Vocational Schools. The findings of this study have a positive contribution to the evaluation process in the field of education, especially in strengthening the correctness of the quantitative data results which are usually obtained in the evaluation process. The novelty of this study is the presentation of qualitative data on the yack component, analysis component, and nominate component which does not have by other evaluation models.

Keywords

qualitative data evaluation components divayana evaluation model

Introduction

The role of information technology in the education field in the era of industrial revolution 4.0 is very important. It is because all teaching materials used in the learning process are expected to be quickly transferred to students online. The role of information technology is also certainly significant and important in supporting the smooth learning process, especially since the appearance of the Covid-19 pandemic. Every learning process can be done online from home (Mishra et al., 2020; Rizaldi & Fatimah, 2020). Many free platforms scattered on the internet can be used to support online learning. Those platforms are Schoology, Quipper School, Kelase, Sevima EdLink, and others (Fatahillah et al., 2020; Febrianti et al., 2018; Mahariyanti & Suyanto, 2019; Rojabi, 2021; Sastranegara et al., 2020; Septinawati et al., 2020; Sitinjak, 2020; Jannah et al., 2020).

The reality shows that not all of those platforms can be used optimally in supporting the smoothness of the online learning process (Abidah et al., 2020; Hussain et al., 2020; Iivari et al., 2020; Dhawan, 2020). Therefore it is necessary to make improvements to the platform’s functionality used to support the online learning process. Efforts that can be made to obtain good and appropriate recommendations as a basis for making improvements are evaluation activities.

Several educational evaluation models can be used to obtain appropriate recommendations in evaluation activities for online learning platforms, including the CIPP model, the Countenance model, the CSE-UCLA model, and the Discrepancy model (Ha et al., 2019; Kim, 2018; Wozny et al., 2018; Zhao & Sun, 2018). However, the limitations of those models are not able to show recommendations in the form of priority aspects that need improvements. One of the innovations in the education evaluation model that can show priority aspects that need improvements is called the DIVAYANA model (Divayana, 2020, 2021; Divayana et al., 2021). According to the model’s name, this model consists of eight evaluation components, including Description component, Input component, Verification component, Action component, Yack component, Analysis component, Nominate component, and Actualization component.

The quality of recommendations generated from the DIVAYANA evaluation model is strongly influenced by the qualitative data provided for each evaluation component. Based on that, it is important to know the qualitative data needed in each component of the DIVAYANA evaluation model. The question of this study is “What qualitative data is needed in each evaluation component of the DIVAYANA evaluation model?”

The answer to that question can be answered by showing evidence of qualitative data needed in each component of the DIVAYANA evaluation model. Specifically for this study, the researchers used data related to the e-learning platform utilization (case study at ICT vocational schools) as evidence to be able to show qualitative data in each component of the DIVAYANA evaluation model.

Method

Approach and Design of Study

This study approach was qualitative by exploring in-depth and thorough the qualitative data needed in each component of the DIVAYANA evaluation model. The research design follows the components of the DIVAYANA evaluation model can be seen in Figure 1.

Figure 1.

Design of study.

Data Collection Tools

Several forms of data collection tools were used in this study, including interview guidelines, checklists, and literature studies. The number of questions in the interview guidelines used to collect the qualitative data needed for the Description component was nine questions. Those nine questions are related to the factors causing the appearance of e-learning platforms in ICT Vocational Schools. In addition, the Description component also needed nine questions related to the problems that occur in the e-learning platforms utilization at ICT vocational schools. The number of questions in the interview guidelines used in the Input component was two questions. Those two questions are related to additional elements that cause the stability of the e-learning platforms’ utilization at ICT vocational schools. In addition, the Input component also needed nine questions about alternative solutions for problem-solving related to the e-learning platform utilization at ICT vocational schools.

The number of questions in the interview guidelines used in the Verification component was nine questions. Those nine questions are related to the successful standards of evaluating e-learning platforms at ICT vocational schools. In addition, the Verification component also needed nine checklists are used to verify the suitability between alternative solutions and the successful standards of evaluating e-learning platforms at ICT Vocational Schools. The number of questions in the interview guidelines used in the Action component was nine questions. Those nine questions are related to the effectiveness of alternative problem-solving solutions in e-learning platform utilization at ICT vocational schools. The number of questions in the interview guidelines used in the Yack component was eight questions. Those eight questions are related to the argument agreement between the evaluators and the experts regarding the effectiveness of alternative problem-solving solutions in e-learning platform utilization at ICT vocational schools.

The number of questions in the interview guidelines used in the Analysis component was eight questions. Those eight questions are related to the analysis results of qualitative data in the Action component and the Yack component. The number of questions in the interview guidelines used in the Nominate component was three questions. Those three questions are related to the effectiveness of the calculation process for determining priority aspects that need improvement. In addition, it takes literature studies from several reputable international journals as references to determine the calculation process for determining priority aspects. The number of questions in the interview guidelines used in the Actualization component was three questions. Those three questions are related to the effectiveness of recommendations implementation for weakness improvements in e-learning platforms utilization at ICT vocational schools.

Data Collection Location

The location for this study was several ICT vocational schools in Bali Province. The reason for choosing ICT vocational schools as the location to collect data in this study is because those schools are already accustomed to using e-learning platforms in their learning process.

Data Sources

The qualitative data obtained in this study were sourced from several informants, including the headmaster, school committees, teachers, students, ICT development teams, industry practitioners, experts, and evaluators. Data sourced from the headmaster is used to support the expected findings in the Description component, Input component, and Actualization component. Data sourced from the school committees are used to support the expected findings in the Description component, and the Actualization component.

Data sourced from the teachers are used to support the expected findings in the Description component, Input component, and Actualization component. Data sourced from students are used to support the expected findings in the Description component, and the Actualization component. Data sourced from the ICT development teams are used to support the expected findings in the Description component, Input component, and Actualization component. Data sourced from industry practitioners are used to support the expected findings in the Input component. Data sourced from experts and evaluators are used to support the expected findings in the Verification component, Action component, Yack component, Analysis component, and Nominate component.

Time Duration of Data Retrieval

The time required to collect qualitative data to support the Description component was 10 days. The time required to collect qualitative data to support the Input component was 12 days. The time required to collect qualitative data to support the Verification component was 6 days. The time required to collect qualitative data to support the Action component was 3 days. The time required to collect qualitative data to support the Yack component was 3 days. The time required to collect qualitative data to support the Analysis component was 3 days. The time required to collect qualitative data to support the Nominate component was 8 days. The time required to collect qualitative data to support the Actualization component was 6 days.

Data Analysis Technique

The data obtained in this study were tested for correctness. The correctness test was done by triangulating the theory (Ashour, 2018; Campbell et al., 2020; Kelle et al., 2019; Vogl et al., 2019). Theoretical triangulation is carried out by referring to the use of various theoretical perspectives. It was done to interpret the truth of the data obtained in this study.

Results and Discussion

Results

Qualitative Data in the Description Component

Generally, the Description component in the DIVAYANA model is used to evaluate the basic factors that cause the appearance of the object being evaluated (for example e-learning platforms at ICT vocational schools). Besides, this component is also used to evaluate problems that often occur related to the e-learning platforms at ICT vocational schools. Some of the qualitative data needed in the Description component is related to the basic factors that cause the appearance of the e-learning platforms at ICT vocational schools can be seen in Figure 2. Qualitative data related to problems that occur in the e-learning platforms at ICT vocational schools can be seen in Figure 3.

Figure 2.

Basic factors that cause the appearance of e-learning platforms at ICT vocational schools.

Figure 3.

Problems that occur in the e-learning platforms utilization at ICT vocational schools.

Qualitative Data in the Input Component

Generally, the Input component in the DIVAYANA model is used to evaluate additional elements that support the stability of the object being evaluated. This component also evaluates several alternatives that are used as solutions to problems related to the object being evaluated. Some of the qualitative data needed for the Input component is related to additional elements that support the stability of the e-learning platforms at ICT vocational schools can be seen in Figure 4. Qualitative data related to problem-solving solutions in the e-learning platforms at ICT vocational schools can be seen in Figure 5.

Figure 4.

Additional elements that cause the stability of the E-learning platforms utilization at ICT vocational schools.

Figure 5.

Alternative solutions to problem-solving related to e-learning platforms at ICT vocational schools.

Qualitative Data in the Verification Component

Generally, the Verification component in the DIVAYANA model is used to evaluate the evaluation success standards. Besides, it is used to verify the suitability between alternative solutions and the evaluation success standards. Some of the qualitative data needed in the Verification component related to the evaluation success standards of the e-learning platforms at ICT vocational schools can be seen in Figure 6. Qualitative data related to verification results of the suitability between alternative solutions and evaluation success standards of the e-learning platforms at ICT vocational schools can be seen in Figure 7.

Figure 6.

Evaluation success standards of the e-learning platforms at ICT vocational schools.

Figure 7.

Conformity verification data between alternative solutions and the evaluation success standards of the E-learning platforms at ICT vocational schools.

Qualitative Data in the Action Component

Generally, the Action component in the DIVAYANA model is used by experts or evaluators to evaluate the field trial results or broader scale tests of alternative solutions for problem-solving that had been previously verified through the Verification component. This evaluation component requires qualitative data about the alternative solutions’ effectiveness for problem-solving regarding rules and regulations, the alternative solutions’ effectiveness for problems regarding social support, the alternative solutions’ effectiveness for problems regarding funding, the alternative solutions’ effectiveness for problems regarding infrastructures, and the alternative solutions’ effectiveness for problems regarding human resources. Some of the interview questions needed to get answers regarding the effectiveness of those alternative solutions can be seen in Table 1.

Table 1.

Interview Questions about the Alternative Solutions’ Effectiveness for Problem-Solving in Using the E-Learning Platforms at ICT Vocational Schools.

Question items	Interview questions
Item-1 and item-2 are questions for alternative solutions to problems related to rules and regulations.	Item-1. Has the existence of government regulations been effective in proving the clarity of the legal basis for using the e-learning platforms at ICT vocational schools?
	Item-2. Has the existence of school rules been effective in proving the existence of a concrete form of government regulation implementation related to the use of e-learning platforms at ICT vocational schools?
Item-3 and item-4 are questions for alternative solutions to problems related to social support.	Item-3. Has the existence of evidence documents related to the support from students’ parents through the school committees been effectively used as a strong foundation in the use of e-learning platforms at ICT vocational schools?
	Item-4. Has the existence of evidence documents related to the support from school members (teachers and students) been effectively used as a strong foundation in the use of e-learning platforms at ICT vocational schools?
Item-5 is the question for alternative solutions to problems related to funding.	Item-5. Has the existence of evidence documents about funds support from the school and external parties been effective in proving the existence of funding support in e-learning implementation?
Item-6 is the question for alternative solutions to problems related to infrastructures.	Item-6. Has the existence of evidence documents about hardware, software, and other devices from the school and external parties been effective in proving the existence of supporting facilities and infrastructures for e-learning implementation?
Items-7, item-8, and item-9 are questions for alternative solutions to problems related to human resources.	Item-7. Has training in the operation of the e-learning platform for teachers and students been effective in proving their readiness in operating the e-learning platforms?
	Item-8. Has training for teachers to create interesting material content been effective in proving their readiness in making quality material content that is suitable for implementation in the e-learning platforms?
	Item-9. Has the special training for the development teams and system managers been effective in proving their readiness in managing and maintaining the e-learning platforms?

Qualitative Data in the Yack Component

Generally, the Yack component in the DIVAYANA model is used to evaluate the arguments agreed upon by the evaluators and experts regarding the alternative solutions’ effectiveness for problem-solving of the object being evaluated. Some of the qualitative data needed in the Yack component related to the arguments of the evaluators and experts regarding the alternative solutions’ effectiveness for problem-solving in using the e-learning platforms at ICT vocational schools can be seen in Table 2.

Table 2.

Some Arguments that had been Agreement by Evaluators and Experts Regarding the Alternative Solutions’ Effectiveness for Problem-Solving in the E-Learning Platforms Utilization at ICT Vocational Schools.

Evaluators/experts	Arguments	Experts/evaluators who agree with the arguments
Evaluator-1	In general, the existence of government regulations and school rules has been effective in proving that there is a clear legal basis for using e-learning platforms at ICT vocational schools.	Expert-1, Expert-2, Expert-3, Expert-4, Evaluator-2, Evaluator-3, and Evaluator-4
Evaluator-2	In general, the existence of evidence documents related to the support from students’ parents through the school committees has been effectively used as a strong foundation in the use of the e-learning platforms at ICT vocational schools.	Expert-1, Expert-2, Expert-3, Expert-4, Evaluator-1, Evaluator-3, and Evaluator-4
Evaluator-3	In general, the existence of evidence documents related to the support from school members (teachers and students) has been effectively used as a strong foundation in the use of the e-learning platforms at ICT vocational schools.	Expert-1, Expert-2, Expert-3, Expert-4, Evaluator-1, Evaluator-2, and Evaluator-3
Evaluator-4	In general, the existence of evidence documents about funds support from the school and external parties has been effective in proving the existence of funding in supporting the e-learning implementation.	Expert-1, Expert-2, Expert-3, Expert-4, Evaluator-1, Evaluator-2, and Evaluator-3
Expert-1	In general, the existence of evidence documents about hardware, software, and other devices from the school and external parties has been effective in proving the existence of supporting facilities and infrastructures for e-learning implementation.	Expert-2, Expert-3, Expert-4, Evaluator-1, Evaluator-2, Evaluator-3, and Evaluator-4
Expert-2	In general, training in the operation of the e-learning platforms for teachers and students has been effective in proving their readiness in operating the e-learning platforms.	Expert-1, Expert-3, Expert-4, Evaluator-1, Evaluator-2, Evaluator-3, dan Evaluator-4
Expert-3	In general, training for teachers to create interesting material content has been effective in proving their readiness in making quality material content that is suitable for implementation in the e-learning platforms.	Expert-1, Expert-2, Expert-4, Evaluator-1, Evaluator-2, Evaluator-3, and Evaluator-4
Expert-4	In general, training for the development teams and system managers has been effective in proving their readiness in managing and maintaining the e-learning platforms.	Expert-1, Expert-2, Expert-3, Evaluator-1, Evaluator-2, Evaluator-3, and Evaluator-4

Qualitative Data in the Analysis Component

Generally, the Analysis component in the DIVAYANA model is used to evaluate the analysis results of qualitative data required previously in the Action component and the Yack component. Some of the interview questions needed to obtain the analysis results of qualitative data previously in the Action component and the Yack component can be seen in Table 3.

Table 3.

Interview Questions are used to obtain the Analysis Results of Qualitative Data in the Action Component and the Yack Component.

Question items	Interview questions
Item-1	Are item-1 and item-2 questions for alternative solutions to problems related to rules (in the Action component) appropriate and suitable in obtaining arguments from evaluator-1 to be mutually agreed on the Yack component?
Item-2	Is item-3’s question for alternative solutions to problems related to social support (in the Action component) appropriate and suitable in obtaining arguments from evaluator-2 to be mutually agreed on the Yack component?
Item-3	Is item-4’s question for alternative solutions to problems related to social support (in the Action component) appropriate and suitable in obtaining arguments from evaluator-3 to be mutually agreed on the Yack component?
Item-4	Is item-5’s question for alternative solutions to problems related to funding (in the Action component) appropriate and suitable in obtaining arguments from evaluator-4 to be mutually agreed on the Yack component?
Item-5	Is item-6’s question for alternative solutions to problems related to infrastructures (in the Action component) appropriate and suitable in obtaining arguments from expert-1 to be mutually agreed on the Yack component?
Item-6	Is item-7’s question for alternative solutions to problems related to human resources (in the Action component) appropriate and suitable in obtaining arguments from expert-2 to be mutually agreed on the Yack component?
Item-7	Is item-8’s question for alternative solutions to problems related to human resources (in the Action component) appropriate and suitable in obtaining arguments from expert-3 to be mutually agreed on the Yack component?
Item-8	Is item-9’s question for alternative solutions to problems related to human resources (in the Action component) appropriate and suitable in obtaining arguments from expert-4 to be mutually agreed on the Yack component?

Qualitative Data in the Nominate Component

Generally, the Nominate component in the DIVAYANA model is used to evaluate the calculation process in determining the right recommendations in the form of priority aspects that need improvement. That calculation process is not explained fully in this study. The calculation process can be seen in literature studies about the DIVAYANA formula in reputable international journals (Divayana, 2020, 2021). Some of the interview questions needed to determine the calculation process effectiveness in searching for priority aspects that need improvement can be seen in Table 4.

Table 4.

Interview Questions are used to determine the Calculation Process Effectiveness in searching for Improvement’s Priority Aspects.

Question items	Interview questions
Item-1	Has the calculation process been carried out to determine the improvement of average weight been effective and obtained accurate calculation results?
Item-2	Has the normalization process been carried out to determine the Vector-D values been effective and obtained accurate calculation results?
Item-3	Has the ranking calculation process to determine the Vector-R values as the determinant of improvement’s priority aspects been effective and obtained accurate calculation results?

Qualitative Data in the Actualization Component

Generally, the Actualization component in the DIVAYANA model is used to evaluate the implementation of recommendations that show improvement’s priority aspects. Some of the interview questions needed to determine the implementation effectiveness of recommendations that show improvement’s priority aspects can be seen in Table 5.

Table 5.

Interview Questions are used to determine the Implementation Effectiveness of Recommendations that Show Improvement’s Priority Aspects toward Weaknesses in the E-Learning Platforms Utilization at ICT Vocational Schools.

Question items	Interview questions
Item-1	Have the improvement’s priority aspects toward weaknesses in the use of the e-learning platforms been implemented effectively at ICT vocational schools?
Item-2	Are the improvement’s priority aspects toward weaknesses in the use of the e-learning platforms at ICT vocational schools become the right recommendations and ready to be implemented on wide scales in the field?
Item-3	Do the improvement’s priority aspects have a positive impact on maintaining the stability of effectiveness level using the e-learning platforms at ICT vocational schools in the future?

Discussion

The DIVAYANA model is one of the new innovative models in the field of educational evaluation. This model was created by Dewa Gede Hendra Divayana. This model consists of eight evaluation components, which in principle have a similar function to several other components of the education evaluation model, such as CIPP (Context-Input-Process-Product) and CSE-UCLA (Center for the Study of Evaluation-University of California in Los Angeles). In general, the characteristics of the DIVAYANA model are not the same as model CIPP and CSE-UCLA, but the DIVAYANA model complements some of the limitations of those models especially related to functionality and requirements in the field. For example, the CIPP model consists of four components to evaluate the context, input, process, and product (Duman & Akbaş, 2017; Finney, 2019; Darma, 2019).

The functionality of the Context component in the CIPP model also appears in the Description component of the DIVAYANA model. The functionality of the Input component in the CIPP model also appears in the Input component of the DIVAYANA model. The functionality of the Process component in the CIPP model also appears in the Action component of the DIVAYANA model.

The functionality of the Product component in the CIPP model also appears in the Actualization component of the DIVAYANA model. The functions that are not in the CIPP model included: the verification function, the function in determining of equalization process of weighting values, the ranking function of several recommended alternative aspects, and the socialization function of the existence of the object being evaluated. Those function limitations of the CIPP model are already equipped and solved in the DIVAYANA model.

Likewise, the CSE-UCLA model has five evaluation components, including system assessment, program planning, program implementation, program improvement, and program certification (Mudiarta et al., 2021). The functionality of the System Assessment component in the CSE-UCLA model also appears in the Description component of the DIVAYANA model. The functionality of the Program Planning component in the CSE-UCLA model also appears in the Input component of the DIVAYANA model. The functionality of the Program Implementation and Program Certification components in the CSE-UCLA model also appears in the Actualization component of the DIVAYANA model. The functionality of the Program Improvement component in the CSE-UCLA model also appears in the Action component of the DIVAYANA model. The functions that are not in the CSE-UCLA model, include the verification function, the Yack function in determining of equalization process of weighting values, and the ranking function for several alternative aspects. Those function limitations of the CSE-UCLA model are already equipped and solved in the DIVAYANA model.

Based on some qualitative data in Figures 2 and 3, it can be used to evaluate the use of the e-learning platforms at ICT vocational schools from the Description component perspective in the DIVAYANA model. This is because those data focus on providing an overview of the causes and problems that are often found in the use of e-learning platforms at ICT vocational schools. Based on some qualitative data in Figures 4 and 5, it can be used to evaluate the use of the e-learning platforms at ICT vocational schools from the Input component perspective in the DIVAYANA model. This is because the data in Figure 4 focuses on providing an overview of new factors which are additional inputs to support the stability of the use of the e-learning platforms at ICT vocational schools. Besides, the data in Figure 5 focuses on providing an overview of several alternatives as solutions to solving problems in using the e-learning platforms at the level of ICT vocational schools. Based on some qualitative data in Figures 6 and 7, it can be used to evaluate the use of the e-learning platforms at ICT vocational schools from the Verification component perspective in the DIVAYANA model. This is because the data in Figure 6 focuses on providing an overview of the success standards for evaluating the use of the e-learning platforms at ICT vocational schools. Besides, the data in Figure 7 focuses on providing an overview of the verification of the suitability between alternative solutions and the success standards of evaluating e-learning platforms at ICT vocational schools.

The interview questions shown in Table 1 can be used to evaluate the use of the e-learning platforms at ICT vocational schools from the Action component perspective in the DIVAYANA model. This is because the questions in Table 1 are focused on being used to find qualitative data about the effectiveness of alternative solutions to problem-solving in using the e-learning platforms at ICT vocational schools on a broader test scale. Based on some qualitative data in Table 2 can be used to evaluate the use of the e-learning platforms at ICT vocational schools from the Yack component perspective in the DIVAYANA model. This is because the data in Table 2 focuses on providing an overview of some of the arguments that had been mutually agreed upon between the evaluator and the experts. The interview questions shown in Table 3 can be used to evaluate the use of the e-learning platforms at ICT vocational schools from the Analysis component perspective in the DIVAYANA model. This is because the questions in Table 3 are used to focus on finding data from the qualitative data analysis on the Action component and the Yack component.

The interview questions shown in Table 4 can be used to evaluate the use of the e-learning platforms at ICT vocational schools from the Nominate component perspective in the DIVAYANA model. This is because the questions in Table 4 are focused on being used to determine the calculation process effectiveness in determining improvement’s priority aspects of weaknesses in the use of e-learning platforms at ICT vocational schools. The interview questions shown in Table 5 can be used to evaluate the use of the e-learning platform at ICT vocational schools from the Actualization component perspective in the DIVAYANA model. This is because the questions in Table 5 are focused on being used to determine the effectiveness of recommendations implementation that show improvement’s priority aspects of weaknesses in the use of e-learning platforms at ICT vocational schools.

Several other theories also reveal that the CIPP evaluation model in principle has four evaluation components, including context, input, process, and product (Aldapit & Suharjana, 2019; Suparman & Sangadji, 2019; Umam & Saripah, 2018). The Context component is used to reveal data related to the factors causing the appearance of the object being evaluated and the problems that occur in the object being evaluated (Agustina & Mukhtaruddin, 2019; Tootian, 2019). If seen from Figures 2 and 3, it appears that the data presented in those two figures are appropriate and correct. This is because the data shown in those figures already reveal the same thing as what was revealed in the Context component of the CIPP evaluation model. The Input component is used to reveal data related to additional elements that support the stability of the existence of the object being evaluated and several alternative problem-solving solutions (Esgaiar & Foster, 2019; Prayogo et al., 2020). If seen from Figures 4 and 5, it appears that the data presented in those two figures are appropriate and correct. This is because the data shown in those figures already reveal the same thing as what was revealed in the Input component of the CIPP evaluation model. The Process component is used to reveal data related to the effectiveness of running alternative problem-solving solutions from the object being evaluated (Maksum et al., 2019; Sopha & Nanni, 2019). If seen from Table 1, it appears that the data presented in Table 1 is appropriate and correct. This is because the data shown in Table 1 already reveal the same thing as what was revealed in the Process component of the CIPP evaluation model. The Product component is used to reveal the effectiveness of recommendations implementation on the object being evaluated (Setyadi et al., 2021; Shindi, 2020). If seen from Table 5, it appears that the data presented in Table 5 is appropriate and correct. This is because the data shown in Table 5 already reveal the same thing as what was revealed in the Product component in the CIPP evaluation model.

Several other theories also reveal that the CSE-UCLA evaluation model has five evaluation components, including system assessment, program planning, program implementation, program improvement, and program certification (Alkin & King, 2017; Naibaho, 2021). The system assessment component is used to reveal data related to factors causing the existence of the object being evaluated (Makaria, 2018). If seen from Figure 2, it appears that the data presented in Figure 2 is appropriate and correct. This is because the data shown in Figure 2 already reveal the same thing as what was revealed in the system assessment component of the CSE-UCLA evaluation model. The program planning component is used to reveal data related to alternative fulfillment of object needs that are evaluated in solving the problems (Rantung & Latupeirissa, 2021). If seen from Figure 5, it appears that the data presented in Figure 5 is appropriate and correct. This is because the data shown in Figure 5 already reveal the same thing as what was revealed in the program planning component of the CSE-UCLA evaluation model. The program implementation component is used to reveal data related to efforts in socializing the existence of the object being evaluated (Mukhlasin & Syafaruddin, 2021).

The program improvement component is used to reveal data related to the running process effectiveness of an object as an alternative problem-solving solution (Suyasa & Divayana, 2021). If seen from Table 1, it appears that the data presented in Table 1 is appropriate and correct. This is because the data shown in Table 1 already reveal the same thing as what was revealed in the program improvement component of the CSE-UCLA evaluation model. The program certification component is used to reveal data related to the effectiveness of the recommendations (Rusmulyani et al., 2022). If seen from Table 5, it appears that the data presented in Table 5 is appropriate and correct. This is because the data shown in Table 5 already reveal the same thing as what was revealed in the program implementation and program certification components of the CSE-UCLA evaluation model.

Several theories reveal that the Countenance model has two evaluation matrixes, including a description matrix and a judgment matrix (Gondikit, 2018; Harjanti et al., 2019; Ismail et al., 2018). The description matrix is used to reveal data related to the existence of the object being evaluated, while the judgment matrix is used to reveal data related to evaluation standards and the verification results of conformity between evaluation standards and data from evaluation results in the field (Dewantara, 2017; Zainuddin & Fuad, 2019). If seen from Figures 6 and 7, it appears that the data presented in those two figures are appropriate and correct. This is because the data shown in those two figures already reveal the same thing as what was revealed in the judgment matrix on the Countenance evaluation model.

If seen from Tables 2, 3, and 4, it appears that the data presented in those three tables are correct and follow the reality in the field so it becomes a novelty in this study. The novelty of this study is the discovery of qualitative data related to several things, including (a) the arguments that have been agreed upon by the evaluators and experts about the effectiveness of alternative problem-solving solutions (see Table 2), (b) the analysis results of the qualitative data needed in the Action component and Yack component (see Table 3), and (c) the calculation process effectiveness for determining improvement’s priority aspects (see Table 4).

This study is compatible with other research contexts which also present data on each component in the evaluation model. The research of Purwaningsih & Dardjito (2021) is compatible with this study related to data about the description of an object being evaluated. However, the limitation of Purwaningsih & Dardjito’s research is that it has not shown some qualitative data related to the results of matching the evaluation standards with the evaluation results. This study has contributed to showing qualitative data that has not been shown in Purwaningsih & Dardjito’s research. The research of Kardiyanto et al. (2017) is compatible with this study regarding qualitative data on evaluation standards. However, the limitation of Kardiyanto et al.’s research is that it has not shown some qualitative data related to the agreement of arguments between each expert and evaluator. This study has contributed to showing qualitative data about the agreement of arguments between experts and evaluators that have not been shown in Kardiyanto et al.’s research. Research by Tomczyk et al. (2020) is compatible with this study regarding qualitative data about the causes of the appearance of e-learning platforms. However, the limitation of Tomczyk et al.’s research is that it has not shown some qualitative data related to recommendations implementation in the field. This study has contributed to showing qualitative data that has not been shown in Tomczyk et al.’s research related to the implementation of recommendations.

The position and contribution of this study are also strengthened by several previous research results. Research by Dwi W et al., (2018), and research by Alexsandra et al., (2019), both showed the need for qualitative data in the CIPP evaluation model to strengthen the quantitative results obtained for each evaluation component. The limitations of those two studies were not showing examples of qualitative data that had been used in the evaluation process. Therefore, it is very appropriate this study is an answer to the limitations of those two studies by showing examples of qualitative data that can be used in each component of the DIVAYANA evaluation model.

The research of Hadullo et al. (2017) showed several evaluation models that can be used to evaluate the quality of e-learning. The limitation of Hadullo et al.’s research was not shown qualitative data that corroborates the recommendations from the evaluation results. Therefore, the contribution of this study is very appropriate in showing the existence of qualitative data on each evaluation component.

Research by Mejía-Madrid et al. (2020) showed the components of the Kirkpatrick evaluation model used to evaluate the quality of open learning courses based on qualitative and quantitative data. The qualitative data were obtained using interview guide instruments, while the quantitative data used questionnaire instruments. The limitation of the Mejía-Madrid et al.’s research was not shown the form of interview questions to obtain qualitative data. Therefore, the position of this study is very appropriate as an answer to the limitation of the Mejía-Madrid et al.’s research. In this study, examples of interview questions had been shown in the evaluation component to obtain qualitative data.

Research by Aziz et al. (2018) showed that there were interview guidelines to obtain qualitative data used in each component of the CIPP evaluation. The limitation of Aziz et al.’s research was not shown completely in the form of interview guidelines used to obtain the qualitative data. Therefore, this study has the right contribution in providing an answer to the limitation of Aziz et al.’s research. This study shows the form of interview questions used to obtain the qualitative data required by the DIVAYANA evaluation component.

Conclusion

In general, this study has shown an overview of the qualitative data needed in each component of the DIVAYANA evaluation model. There are eight components of the DIVAYANA evaluation model that require qualitative data to obtain in-depth information regarding the effectiveness of e-learning platform utilization in ICT vocational schools. The eight components that have received qualitative data are the Description component, Input component, Verification component, Action component, Yack component, Analysis component, Nominate component, and Actualization component. The qualitative data in the Description component is related to the factors that cause the appearance of e-learning platforms and the problems that occur in the e-learning platforms’ utilization in ICT vocational schools. Qualitative data in the Input component is related to additional elements that cause stability in the e-learning platforms’ utilization in ICT vocational schools and alternative solutions to problem-solving related to e-learning platforms in ICT vocational schools. Qualitative data in the Verification component is related to the evaluation success standards of the e-learning platform at ICT vocational schools and the verification results of the suitability between alternative solutions and the evaluation successful standards of the e-learning platform at ICT vocational schools. Qualitative data in the Action component is related to the effectiveness of alternative problem-solving solutions in the e-learning platform’s utilization at ICT vocational schools. Qualitative data in the Yack component is related to the argument agreement between the Evaluators and the Experts about the effectiveness of alternative problem-solving solutions in e-learning platforms’ utilization at ICT vocational schools. Qualitative data in the Analysis component is related to the analysis results of qualitative data of the Action component and the Yack component. Qualitative data in the Nominate component is related to the effectiveness of the calculation process for determining priority aspects that need improvement. Qualitative data in the Actualization component is related to the implementation effectiveness of the recommendations that indicate improvements’ priority aspects toward weaknesses in e-learning platforms’ utilization at ICT vocational schools. Most of those qualitative data were obtained using interview questions, but some also used checklists and literature studies. The findings in this study that become novelty are some qualitative data in the Yack component, Analysis component, and Nominate component in the DIVAYANA model. This study will certainly provide sustainable benefits and positive impacts in the future for educational evaluators and experts in the field of education in evaluating information technology-based learning. The benefit is providing an overview to the evaluators and education experts about the appropriate qualitative data according to the needs of each component of the evaluation model.

Footnotes

Acknowledgments

The authors express their gratitude to the Rector and Chair of the Institute for Research and Community Service, Universitas Pendidikan Ganesha for providing encouragement and opportunities to complete this research.

Author Contributions

DGHD was responsible for conceptualization, methodology, data curation, formal analysis, investigation, writing-original draft, writing-reviewing, and editing. NKW and IGR were responsible for data curation and formal analysis. All authors have read and agreed to the published version of the manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding for this research sources from the Special Award for Achievement Lecturers at Universitas Pendidikan Ganesha in 2021 Based on the Decree of the Rector of Universitas Pendidikan Ganesha, Number: 331/UN48/PT/2022.

Data Availability

The data presented in this study are available in article.

ORCID iD

Dewa Gede Hendra Divayana

References

Abidah

Hidaayatullaah

H. N.

Simamora

R. M.

Fehabutar

Mutakinati

(2020). The impact of covid-19 to Indonesian education and its relation to the philosophy of “Merdeka Belajar”. Studies in Philosophy of Science and Education (SiPoSE), 1(1), 38-49. https://doi.org/10.46627/sipose.v1i1.9.

Agustina

N. Q.

Mukhtaruddin

(2019). The CIPP model-based evaluation on integrated english learning (IEL) program at language center. English Language Teaching Educational Journal, 2(1), 22-31. https://doi.org/10.12928/eltej.v2i1.1043.

Aldapit

Suharjana

(2019). CIPP evaluation model for the coaching program of running athletes. Psychology, Evaluation, and Technology in Educational Research, 1(2), 104–116. https://doi.org/10.33292/petier.v1i2.10

Alexsandra Ganefri Usmeldi (2019). Evaluation of vocational school pratic program cipp model. Journal of Educational Research and Evaluation, 3(3), 180-184. https://doi.org/10.23887/jere.v3i3.21860

Alkin

M. C.

King

J. A.

(2017). Definitions of evaluation use and misuse, evaluation influence, and factors affecting use. American Journal of Evaluation, 38(3), 434–450. https://doi.org/10.1177/1098214017717015

Ashour

M. L.

(2018). Triangulation as a powerful methodological research technique in technology-based services. Business & Management Studies: An International Journal, 6(1), 193–208. https://doi.org/10.15295/bmij.v6i1.209

Aziz

Mahmood

Rehman

(2018). Implementation of CIPP model for quality evaluation at school level: A case study. Journal of Education and Educational Development, 5(1), 189–206. http://dx.doi.org/10.22555/joeed.v5i1.1553

Campbell

Goodman-Williams

Feeney

Fehler-Cabral

(2020). Assessing triangulation across methodologies, methods, and stakeholder groups: The joys, woes, and politics of interpreting convergent and divergent data. American Journal of Evaluation, 41(1), 125–144. https://doi.org/10.1177/1098214018804195

Darma

I. K.

(2019). The effectiveness of teaching program of cipp evaluation model. International Research Journal of Engineering, IT & Scientific Research, 5(3), 1–13. https://doi.org/10.21744/irjeis.v5n3.619

10.

Dewantara

I. P. M.

(2017). Stake evaluation model (countenance model) in learning process bahasa Indonesia at Ganesha university of educational. International Journal of Language and Literature, 1(1), 19-29. https://doi.org/10.23887/ijll.v1i1.9615.

11.

Dhawan

(2020). Online learning: A panacea in the time of covid-19 crisis. Journal of Educational Technology Systems, 49(1), 5–22. https://doi.org/10.1177/0047239520934018

12.

Divayana

D. G. H.

(2020). Utilization of DIVAYANA formula in evaluating of suitable platforms for online learning in the social distancing. International Journal of Interactive Mobile Technologies, 14(20), 50–75. https://doi.org/10.3991/ijim.v14i20.15787

13.

Divayana

D. G. H.

(2021). The simulation of DIVAYANA formula calculation to determine the priority of recommendation in evaluation activities. Journal of Physics: Conference Series, 1810(1), 1-7. https://doi.org/10.1088/1742-6596/1810/1/012044.

14.

Divayana

D. G. H.

Suyasa

P. W. A.

Widiartini

N. K.

(2021). An innovative model as evaluation model for information technology-based learning at ICT vocational schools. Heliyon, 7(2), 1–13. https://doi.org/10.1016/j.heliyon.2021.e06347

15.

Duman

S. N.

Akbaş

(2017). Evaluation of Turkish and mathematics curricula according to value-based evaluation model. Cogent Education, 4(1), 1–10. https://doi.org/10.1080/2331186X.2017.1291174

16.

Dwi W

Y.R.

Haryono

Florentinus

T. S.

(2018). The evaluation of the CIPP model in the implementation of character education at junior high school. Innovative Journal of Curriculum and Educational Technology, 7(2), 65-77. http://dx.doi.org/10.15294/ijcet.v7i2.28391.

17.

Esgaiar

Foster

(2019). Implementation of cipp model for quality evaluation at Zawia university. International Journal of Applied Linguistics & English Literature, 8(5), 106-115. https://doi.org/10.7575/aiac.ijalel.v.8n.5p.106.

18.

Fatahillah

Puspitasari

I. D.

Hussen

(2020). The development of schoology web-based learning media with geogebra to improve the ICT literacy on quadratic functions. Journal of Research and Advances in Mathematics Education, 5(3), 304–316. https://doi.org/10.23917/jramathedu.v5i3.10692

19.

Febrianti

Mulyadi Hati

G. M.

(2018). Students’ attitude toward “quipper school” as an english online learning media at SMAN 1 Argamakmur Bengkulu Utara. International Journal of Learning and Development, 2(1), 36–41. https://doi.org/10.33369/jeet.2.1.36-41

20.

Finney

T. L

(2019). Confirmative evaluation: New CIPP evaluation model. Journal of Modern Applied Statistical Methods, 18(2), 1–24. https://doi.org/10.22237/jmasm/1598889893

21.

Gondikit

T. J.

(2018). The evaluation of post PT3 program using Stake’s Countenance model. Malaysian Journal of Social Sciences and Humanities (MJ-SSH), 3(4), 109-118. https://doi.org/10.47405/mjssh.v3i4.137.

22.

A. S.

O’Reilly

J. Y. Y.

Zhang

J. H.

(2019). Evaluating the flipped classroom approach in asian higher education: Perspectives from students and teachers. Cogent Education, 6(1), 1–12. https://doi.org/10.1080/2331186X.2019.1638147

23.

Hadullo

Oboko

Omwenga

(2017). A model for evaluating e-learning systems quality in higher education in developing countries. International Journal of Education and Development Using Information and Communication Technology, 13(2), 185-204. https://ijedict.dec.uwi.edu/viewissue.php?id=48.

24.

Harjanti

Supriyati

Rahayu

(2019). Evaluation of learning programs at elementary school level of “Sekolah Alam Indonesia (SAI)”: (Evaluative research using countenance stake’s model). American Journal of Educational Research, 7(2), 125-132. https://doi.org/10.12691/education-7-2-2.

25.

Hussain

F. N.

Al-Mannai

Agouni

(2020). An emergency switch to distance learning in response to the covid-19 pandemic: Experience from an internationally accredited undergraduate pharmacy program at Qatar university. Medical Science Educator, 30(1), 1393-1397. https://doi.org/10.1007/s40670-020-01079-9.

26.

Iivari

Sharma

Ventä-Olkkonen

(2020). Digital transformation of everyday life– How covid-19 pandemic transformed the basic education of the young generation and why information management research should care? International Journal of Information Management, 55(1), 1-6. https://doi.org/10.1016/j.ijinfomgt.2020.102183.

27.

Ismail

Akabar

Widodo

S. E.

(2018). Evaluation of structuring and equalization policy implementation of civil servant teachers public senior high schools (SMA) and vocational schools (SMK) in Banda Aceh the province of Aceh. International Journal of Scientific Research and Management, 6(4), 207–219. https://doi.org/10.18535/ijsrm/v6i4.el04

28.

Jannah

S. N.

Sobandi

Suwatno (2020). The measurement of usability using use questionnaire on the google classroom application as e-learning media (a case study: SMK Negeri 1 Bandung). Teknodika, 18(2), 94-105. https://doi.org/10.20961/teknodika.v18i2.42486.

29.

Kardiyanto

D. W.

Setijono

Mintarto

(2017). The evaluation of Indonesia paralympic coaching. European Journal of Special Education Research, 2(4), 102–122. https://doi.org/10.46827/ejse.v0i0.1032

30.

Kelle

Kühberger

Bernhard

(2019). How to use mixed-methods and triangulation designs: An introduction to history education research. History Education Research Journal, 16(1), 5–23. https://doi.org/10.18546/HERJ.16.1.02

31.

Kim

(2018). How to evaluate learning in a flipped classroom. Journal of Educational Evaluation for Health Professions, 15:21. https://doi.org/10.3352/jeehp.2018.15.21

32.

Mahariyanti

Suyanto

(2019). The effectiveness of using quipper school teaching materials towards students’ cognitive outcomes of eleventh grade students of MIPA. Advances in Social Science, Education and Humanities Research (ASSEHR), 330(1), 298-301. https://doi.org/10.2991/iceri-18.2019.62.

33.

Makaria

E. C.

(2018). Evaluation study on implementation of home visit service through cse-ucla model in SMP Negeri 2 Anjir Muara. Advances in Social Science, Education and Humanities Research, 274(1), 68-71. https://doi.org/10.2991/iccite-18.2018.16.

34.

Maksum

Luddin

M. R.

Idris

(2019). Evaluation program for the career development of Indonesian navy civilian personnel using the CIPP evaluation model and analytical hierarchy process (AHP). International Journal of Recent Technology and Engineering (IJRTE), 8(4), 1309-1316. https://doi.org/10.35940/ijrte.D6942.118419.

35.

Mejía-Madrid

Llorens-Largo

Molina-Carmona

(2020). Dashboard for evaluating the quality of open learning courses. Sustainability, 12(9), 1–19. https://doi.org/10.3390/su12093941

36.

Mishra

Gupta

Shree

(2020). Online teaching-learning in higher education during lockdown period of covid-19 pandemic. International Journal of Educational Research Open, 1(1), 1-8. https://doi.org/10.1016/j.ijedro.2020.100012.

37.

Mudiarta

I. M. D. R.

Divayana

D. G. H.

Setemen

(2021). The simulation of Alkin evaluation model based on SAW to evaluate flipped learning in IT vocational schools. Journal of Physics: Conference Series, 1810(1), 1-8. https://doi.org/10.1088/1742-6596/1810/1/012063.

38.

Mukhlasin

Syafaruddin

(2021). Evaluation of block learning innovation program in MAN 2 Lubuk Pakam. EDUTEC: Journal of Education and Technology, 5(1), 180-191. https://ejournal.ijshs.org/index.php/edu/article/view/315.

39.

Naibaho

(2021). Online learning evaluation during covid-19 using CSE-UCLA evaluation model at english education department Universitas Kristen Indonesia. Budapest International Research and Critics Institute-Journal (BIRCI-Journal), 4(2), 1987-1997. https://doi.org/10.33258/birci.v4i2.1887.

40.

Prayogo

Rusdarti

Raharjo

T. J.

Kuswardinah

(2020). Distance learning evaluation with the CIPP model. Advances in Social Science, Education and Humanities Research, 574(1), 330-336. https://doi.org/10.2991/assehr.k.211125.062.

41.

Purwaningsih

Dardjito

(2021). Implementation of CIPP model for online learning evaluation during covid-19 pandemic. Getsempena English Education Journal, 8(2), 294-309. https://doi.org/10.46244/geej.v8i2.1394.

42.

Rantung

D. A.

Latupeirissa

(2021). Evaluation of the christian religious education learning program using the CSE-UCLA evaluation model at the PPD social welfare institution in kota Batu. International Journal of Research -Granthaalayah, 9(3), 278-287. https://doi.org/10.29121/granthaalayah.v9.i3.2021.3812.

43.

Rizaldi

D. R.

Fatimah

(2020). How the distance learning can be a solution during the covid-19 pandemic. International Journal of Asian Education, 1(3), 117–124. https://doi.org/10.46966/ijae.v1i3.42

44.

Rojabi

A. R.

(2021). EFL learners’ perceptions on schoology use in the reading class. Voices of English Language Education Society, 5(1), 10–26. https://doi.org/10.29408/veles.v5i1.3219

45.

Rusmulyani

Yudana

I. M.

Natajaya

I. N.

Divayana

D. G. H.

(2022). E-evaluation based on CSE-UCLA model refers to Glickman pattern for evaluating the leadership training program. IJACSA) International Journal of Advanced Computer Science and Applications, 13(5), 279-294. https://doi.org/10.14569/IJACSA.2022.0130534.

46.

Sastranegara

Suryo

Setiawan

(2020). A study of the use of quipper school in history learning during covid-19 pandemic era. International Journal of Learning and Development, 10(3), 20–31. https://doi.org/10.5296/ijld.v10i3.17212

47.

Septinawati

Febriani

R. B.

Tarwana

Syafryadin

(2020). Students’ perceptions toward the implementation of quipper school as an e-learning platform in teaching english. Jadila: Journal of Development and Innovation in Language and Literature Education, 1(2), 223-238. https://doi.org/10.52690/jadila.v1i2.103.

48.

Setyadi

Hidayatullah

M. F.

Purnama

S. K.

(2021). The evaluation of physical health and sport education program at SMA Negeri 2 Ngawi using the CIPP (context, input, process, product) evaluation model. International Journal of Humanities and Education Development, 3(6), 66-77. https://doi.org/10.22161/jhed.3.6.8.

49.

Shindi

(2020). Using the CIPP model to ensure administration evaluation of academic department. International Journal of Psychosocial Rehabilitation, 24(1), 7121-7125. https://doi.org/10.37200/IJPR/V24I1/PR202237.

50.

Sitinjak

A. A.

(2020). The effect of learning method schoology applications and learning style on student learning outcomes. International Journal for Innovation Education and Research, 8(6), 113–119. https://doi.org/10.31686/ijier.vol8.iss6.2380

51.

Sopha

Nanni

(2019). The CIPP model: Applications in language program evaluation. The Journal of Asia TEFL, 16(4), 1360-1367. https://doi.org/10.18823/asiatefl.2019.16.4.19.1360.

52.

Suparman

Sangadji

(2019). Evaluation of learning programs with the CIPP model in college a theoretical review. International Journal of Education, Information Technology, and Others, 2(1), 120-127. https://doi.org/10.5281/zenodo.3666969.

53.

Suyasa

P. W. A.

Divayana

D. G. H.

(2021). Modification of CSE-UCLA and Discrepancy as a new evaluation model in education. Emerging Science Journal, 5(6), 929-942. https://doi.org/10.28991/esj-2021-01321.

54.

Tomczyk

Potyrała

Włoch

Wnęk-Gozdek

Demeshkant

(2020). Evaluation of the functionality of a new e-learning platform vs. previous experiences in e-learning and the self-assessment of own digital literacy. Sustainability, 12(23), 1-22. https://doi.org/10.3390/su122310219.

55.

Tootian

(2019). Evaluation of training courses applied in succession planning in organizations using the CIPP model. International Journal of Human Capital in Urban Management, 4(2), 111-118. https://doi.org/10.22034/IJHCUM.2019.02.04.

56.

Umam

K. A.

Saripah

(2018). Using the context, input, process and product (CIPP) model in the evaluation of training programs. International Journal of Pedagogy and Teacher Education, 2(Special July 2018), 183-194. https://doi.org/10.20961/ijpte.v2i0.26086.

57.

Vogl

Schmidt

E. M.

Zartler

(2019). Triangulating perspectives: Ontology and epistemology in the analysis of qualitative multiple perspective interviews. International Journal of Social Research Methodology, 22(6), 611–624. https://doi.org/10.1080/13645579.2019.1630901

58.

Wozny

Balser

Ives

(2018). Evaluating the flipped classroom: A randomized controlled trial. The Journal of Economic Education, 49(2), 115–129. https://doi.org/10.1080/00220485.2018.1438860

59.

Zainuddin

Fuad

(2019). Program evaluation of continuing professional development at teacher working group. International Journal for Educational and Vocational Studies, 1(2), 130–137. https://doi.org/10.29103/ijevs.v1i2.1524

60.

Zhao

Sun

(2018). Study on the evaluation system of flipped classroom teaching in colleges and universities. Advances in Social Science, Education and Humanities Research, 221(1), 155-159. https://doi.org/10.2991/ceed-18.2018.32.

Qualitative Data Requirements in the Divayana Evaluation Model

Abstract

Keywords

Introduction

Method

Approach and Design of Study

Data Collection Tools

Data Collection Location

Data Sources

Time Duration of Data Retrieval

Data Analysis Technique

Results and Discussion

Results

Qualitative Data in the Description Component

Qualitative Data in the Input Component

Qualitative Data in the Verification Component

Qualitative Data in the Action Component

Qualitative Data in the Yack Component

Qualitative Data in the Analysis Component

Qualitative Data in the Nominate Component

Qualitative Data in the Actualization Component

Discussion

Conclusion

Footnotes

Acknowledgments

Author Contributions

Declaration of Conflicting Interests

Funding

Data Availability

ORCID iD

References