Abstract
OBJECTIVES
This research focuses on assessing non-technical skills (NTS), comprising behavioral aspects that support an individual's expertise in personal and professional performance. Due to varied terminology, including “soft skills” and other terms, NTS is less understood, leading to a potential gap in the literature and limited attention to their development. This study establishes the need for a comprehensive evaluation of NTS to obtain an objective and complete profile of NTS in students pursuing medicine at the Universidad Central del Ecuador (UCE) during 2023, using a standardized and benchmarked psychometric instrument.
METHODS
A cross-sectional study assessed 1035 students from the first to the 12th semester using the CompeTEA® Test, which evaluates 5 factors encompassing 20 competencies and includes sincerity as a reliability condition. Scores were corrected, analyzed, and standardized using TEAcorrige.
RESULTS
The study group comprised 1035 undergraduate medical students (36.8% of students at UCE in 2023), with a mean age of 22.1 years (2.7 standard deviation), and 687 (66.4%) were female. We obtained an average score of 77.7 in sincerity. Factors such as intrapersonal, interpersonal, task development, and managerial skills showed variations throughout the course, with some competencies developing positively in intermediate clinical stages. At the same time, most exhibited a decrease toward the end of the program. Comparisons with population levels revealed 19 competencies at Level 2 and 1 competency predominantly at Level 1.
CONCLUSION
The moderate to low development, with no significant differences based on demographic variables, suggests the need for specific interventions in NTS within comprehensive medical education.
Introduction
Non-technical skills (NTS) are behavioral aspects of performance that support an individual's or professional's expertise. They refer to personal development competencies, constituting an integrated set of knowledge, skills, abilities, aptitudes, personality traits, and motivations inherent or underlying in a person, predisposing them to meet the requirements and demands of their role successfully.1,2 These capabilities reflect cognitive and interpersonal skills that complement technical expertise, ie, technical skills. 3
There is increasing evidence of the importance of developing NTS in all areas, especially for health professions that demand high efficiency, competitiveness, and performance.4–8
Errors in the healthcare environment can be as high as 10%, with human factors responding to 86%. Communication failures, the basis of NTS, comprehends almost half of those human factors. Since healthcare environments are complex systems prone to higher levels of human mistakes, developing NTS could significantly reduce them, increasing patient and professional safety.8,9
The evaluation of the NTS is complex due to the various definitions employed, such as “soft skills,” flexible skills, emotional skills, core skills, and professional competencies. The NTS definition is more inclusive and accurate and, therefore, recommended for standardization. Lozano Fernández 6 and Hinojo Lozano et al 8 provide comprehensive classifications based on performance factors with oneself and others, teamwork, and leadership with various competencies within each factor. Arribas and Pereña Brand 10 offer a complete classification, organizing them into five factors: intrapersonal, interpersonal, task development, environment, and managerial, totaling 20 non-technical competencies.
Several authors emphasize the need for more emphasis on directed learning of NTS in the undergraduate medical curriculum to promote paradigm shifts, particularly among professional educators. Nevertheless, there has yet to be a consensus on evaluating the basal level of NTS or its progression throughout the undergraduate medical curriculum.11–14
In Ecuador, there are no reported evaluations of NTS at the undergraduate medical course level using large samples and standardized instruments, which allow benchmarks to obtain an objective profile. The present study is the first to present an evaluation of such a kind in Ecuador.
Methodology
This descriptive and cross-sectional study aims to obtain an objective profile of NTS in undergraduate medical students from the first up to the 12th semester of 2023 of the Universidad Central del Ecuador (UCE) in Quito, Ecuador.
Inclusion criteria—all students enrolled in undergraduate semesters of the Medicine Program at UCE in Quito during 2023.
Exclusion criteria were (1) failure to complete 100% of responses in the CompeTEA® test; (2) taking more than 25 min to complete the CompeTEA® test; and (3) having perseverative responses, meaning repeatedly answering 1, 1, 1, or 2, 2, 2, 2 more than three times.
After obtaining consent, we presented a dedicated questionnaire to obtain demographic characteristics: age (years), birth gender, number of siblings and hierarchy in the family, nature of funding of fundamental institutions they attended before university admission and previous experience in university courses.
Then, we use a commercially available, standardized psychometric instrument (CompeTEA®) specifically benchmarked for Ecuador. In summary, as described elsewhere, CompeTEA is a self-report questionnaire designed to assess competencies currently understood in the workplace context. 10 The subject has 170 items to complete in up to 30 min, with statements related to various work-related aspects representing multiple behaviors. The individual must respond based on the frequency of these behaviors or the degree of agreement with their content. The assessment covers the 20 most frequently identified competencies in corporate competency dictionaries, grouped into five factors: intrapersonal (self-control and emotional stability, self-confidence, and resilience), interpersonal (communication, relationship building, negotiation, influence, and teamwork), task development (initiative, results orientation, analytical skills, and decision-making), environmental (company knowledge, vision and anticipation, customer orientation, openness, and company identification), and managerial area (leadership, direction, and planning and organization). It also includes a sincerity scale and two qualitative indices: delegation and creativity. Data related to the development of the study (face, content, or construct validity) can be found in the original study by Arribas-Águila. 15 Additionally, the instrument was validated by the Consejo General de la Psicología—España, as evidenced on the website (https://www.cop.es/test/).
We submitted the responses to the online TEA platform. We obtained a direct score and standardized score (SS) for each factor and competency and the classification of NTS development levels for each competency (there is no development level for factors). All the rights of the analyzing process are copyrighted, and the TEA platform provides only the final results.
Statistical analysis
For the calculation of the sample size (students per semester), a 95% confidence level and an estimation error of <7% are considered. With these results, an adjusted sample size of 1103 students to be contacted is obtained.
Descriptive analyses used median and interquartile ranges to measure the central distribution and population dispersion for continuous variables. We used percentages for categorical variables. We used the Kruskal–Wallis and chi-square tests to compare two variables and the Mann–Kendall test to assess the trend of continuous variables over the semesters. We used the Newey–West interrupted time series (ITS) model to determine the impact of curriculum stages (preclinical, clinical, and internship) on competencies and factors. We presented the results as beta coefficients for each semester with their respective 95% confidence intervals. We carried out the analysis and construction of graphs using the statistical software STATA® version 15.
The reporting of this study conforms to the STROBE statement 16 (STROBE_checklist_Supplement).
We obtained ethical approval from the Committee for Research on Human Beings at UCE, endorsed by the Ministry of Public Health in Ecuador (Number 240-CEISH-UCE-2022/code 003-DOCT-FCD-2023).
Results
We enrolled 1443 undergraduate medical students (51.2%) at the Universidad Central del Ecuador (UCE). We excluded 408 students (28.1% out of 1443 enrolled) for inability to answer all questions (205; 50.5% of the 408 students excluded), exceeding the allotted time (152; 37.4% of the 408 students excluded), or exhibiting persistent responses (49; 12.1% of the 408 students excluded). The study group comprised the remaining 1035 students (71.8% of those included and 36.8% of the total students at UCE in 2023). The mean age was 22.1 years (2.7 standard deviations), and 687 (66.4%) were female. We present demographic characteristics in Table 1. Only age and gender were significantly different among the semesters.
Socio-demographic characteristics according to semesters in the undergraduate medical program at Universidad Central del Ecuador (UCE).
We present the SS for every factor and their competencies among the semesters in Table 2.
Median and interquartile range of the standardized scores (SS) of factors with their competencies.
The beta coefficients and 95% confidence intervals obtained for ITS were different for the intrapersonal factor between the first two periods (11.5 [5.95, 17.04]); for the interpersonal factor between the first two periods (2 [0.96, 3.03]) and between the last two periods (−2.25 [−3.86, −0.63]); for the task development factor between the first two periods (3.5 [1.24, 5.75]); and the managerial factor between the last two periods (−8.5 [−12.78, −4.21]). We observed no difference in the environmental factor. Competencies nested within the factors demonstrated similar behavior, showing low development in the preclinical period (first to fourth semesters), increasing toward the clinical period (fifth to eighth semesters), and decreasing toward the practical application period (ninth to 12th semesters). The sincerity factor had an average score of 77.7, with no statistically significant difference (p = 0.744) among the semesters. We present the progression of each competency and factor in Table 3, with ITS graphs in Figure 1.
Interrupted time series graphs of standardized scores of each factor with a cut point in the preclinical (x5 xt5) and clinical (x9 xt9) phases.
Beta coefficient (95% confidence interval (CI)) for the preclinical (x5 xt5) and clinical (x9 xt9) interrupted time series for each competency and factor.
We show the distribution of the students’ performance in four levels of development (Level 1, the lowest up to the highest in Level 4) according to a reference population produced for Ecuador by CompeTEA® in Table 4. Most students reached only 50% of the expected for an external reference in Level 1 (2 competencies) or Level 2 (17 competencies) development. Less than 21.8% of the students were in Level 3 or 4.
Average standardized scores and global competency levels for each skill and competency factor. percentage of students at each level. and the percentage of the total percentage.
Discussion
We found out that there were no significant and sustained increments in NTS throughout the undergraduate medical course in the UCE using a reliable and validated instrument. Besides that, the level of development for NTS in comparison with an external reference was below expected.
NTS comprehends only 8 (9.5%) of the 84 signatures of the formal curriculum at the UCE. Of the total time allotted to the medical course (13,160 h), these 8 signatures totalize 324 h (2.4%). Since the time allocated to NTS in the formal curriculum is low, developing these skills should originate from the student's background (family, pre-university education, culture, etc) or the hidden curriculum, as shown in other studies.17,18 We observed no influence from possible upbringing confounding factors such as the number of siblings and hierarchy in the family or external factors such as the type of school attended (a potential indicator for economic level), except for age and birth gender. Data indicate that female students performed better in the last semester of social, administrative, and health careers at 11 universities. Men perform better in leadership and decision-making, while women excel in organization and planning. Nevertheless, considering the instrument selected in this investigation is already adjusted for gender, age, and culture, we did not evaluate these confounders further to avoid overadjustment.17–20
Evaluating the hidden curriculum can be as challenging as evaluating NTS. We tried to assess that with ITS considering the 3 phases of the curriculum (preclinical, clinical, and internship). Some competencies may have been positively influenced by the preclinical phase, developing up to the clinical stages (fifth to eighth semester). Still, they decline toward the end of the course. Any impact of the hidden curriculum in developing competencies throughout the course is not sustainable.12,21,22
Other researchers used CompeTEA® to evaluate the hidden curriculum of different university courses and in professional settings besides medicine.21–24 They found results similar to those we presented.
Having obtained progressive and positive development of NTS toward the clinical period, we observed a decrease in their development toward the end of the course for several competencies and factors. In other words, there is a decrease when interns establish more significant contact with patients, requiring a greater expression of their performance and NTS for patient safety and self-care.
Besides that, we found a different behavior for each NTS factor evaluated in CompeTEA®. The intrapersonal, interpersonal, and managerial factors seem to be more influenced by the phases of the course than the others. We could expect all factors and competencies to develop uniformly throughout the medical course. But up to now, there is no data regarding the validity of this assumption. It would be more intuitive to expect the development of intrapersonal and interpersonal NTS before the student can pay attention to the other factors. If one proves this correct, these findings could impact curriculum development.8,25–27
Our findings are concerning when benchmarking the results obtained at the population level. Out of the 20 assessed competencies, 2 were predominantly at Level 1, 17 competencies predominantly at Level 2, and 1 competency was equally distributed between Level 1 and Level 2. Only 21.8% of students fall into Levels 3 and 4 in a few competencies. Even though there is yet to be an established level the students should achieve, higher levels are desirable when considering the skills they will have to apply in their jobs. Evaluating NTS more systematically, such as in the current investigation, could provide data for curriculum adjustment and follow-up. Moreno-Murcia and Quintero-Pulgar 21 and Rueda Galvis et al 12 obtained similar results, considering students from 3 university courses at different points in the curriculum. They found, on average, scores below the mean in the 5 major areas, obtaining deficient and insufficient scores in terms of skill development: intrapersonal (24.14), interpersonal (28.92), task development (28.48), environment (21.75), and managerial (27.80).
Unfortunately, our findings regarding the time allocated to NTS in medical training are common in Ecuador. Even though academic authorities know the importance of including NTS in the curriculum, we will need more efficient measurements to implement any change. Moreno-Murcia and Quintero-Pulgar 21 and Martínez et al 23 also recommend changes in the curriculum to increase the time allocated to NTS training. Similar to our findings, they argue that there are no statistically significant differences in soft skills concerning the professional cycle and disciplinary training. They emphasize the need for a critical analysis within educational institutions to promote curriculum harmonization or the design of programs for strengthening and developing NTS.
While a new curriculum usually includes innovative strategies, there is no systematic evaluation throughout the course with constant measurement of NTS in subjects exposed to these methodologies. NTS requires significant strengthening for internalization, where both students and teachers acquire theoretical knowledge and practical NTS. 27
Our study opens up several possibilities for enhancing teaching techniques as it adds a quantitative aspect to NTS. Quantitative assessments will enable a more effective awareness among those responsible for curriculum adjustments regarding the importance of including systematic teaching of NTS, although much still needs to be discussed about how and at what point in the curriculum these should be included and evaluated.
Our study has some limitations. First, as we pointed out, studying NTS is very troublesome since there are multiple instruments to measure it. Using an instrument that could evaluate all 4 levels of Kirkpatrick's framework would be desirable. 28 Nevertheless, since it is very difficult and not cost-effective to do so, we chose an instrument with a solid methodology, that includes standards for an external comparison and has been used for more than 10 years.15,22,23,29 Even though this could be a problem when comparing our results to other studies, the available studies are few, and the results are very similar to ours.
Second, all the limitations concerning transversal studies apply, but it is very informative for an initial evaluation of the NTS status in our institution. When we can implement a continuous assessment of NTS throughout the course, we will have data for correlating teaching strategies with NTS outcomes.
Third, we could only enroll approximately one-third of the students at UCE in our study, but this is the largest and possibly the most reliable survey regarding NTS investigation in undergraduate medical students. We also want to note that we excluded some students who underwent the test. We believe this exclusion did not affect our results, as it is expected that a test of this nature would result in some losses.
Conclusion
In conclusion, we need innovative education strategies and a paradigm shift for professional educators to develop NTS in the undergraduate medical curriculum. Objective measurements of NTS will help to evaluate and adapt the strategy available to reach this goal. Since cultural issues are very important in dealing with NTS, evaluating South American countries is paramount.
Supplemental Material
sj-docx-1-mde-10.1177_23821205241278656 - Supplemental material for Evaluation of Non-Technical Skills in Students Throughout the Medical Course in Ecuador
Supplemental material, sj-docx-1-mde-10.1177_23821205241278656 for Evaluation of Non-Technical Skills in Students Throughout the Medical Course in Ecuador by Olga Cecilia Ortiz Palacios, Mila Inés Moreno Pramatárova and Antonio Pazin-Filho in Journal of Medical Education and Curricular Development
Footnotes
Acknowledgments
Not applicable.
Author contributions
OCOP contributed to the research idea, study design, data collection and analysis, and manuscript writing. MIMP contributed to the data collection and analysis and manuscript writing. APF contributed to the research idea, study design, data analysis, and manuscript writing.
Informed consent
All participants signed an informed consent form approved by the ethics committee.
Ethical approval
We obtained ethical approval from the Committee for Research on Human Beings at UCE, endorsed by the Ministry of Public Health in Ecuador (Number 240-CEISH-UCE-2022/code 003-DOCT-FCD-2023).
DECLARATION OF CONFLICTING INTERESTS
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
FUNDING
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by Fundação de Apoio ao Ensino, Pesquisa e Assistência do Hospital das Clínicas da Faculdade de Medicina de Ribeirão Preto da Universidade de São Paulo (240-CEISH-UCE-2022).
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
