Subjective and objective assessment in skills evaluation: A cross-sectional study among critical care nurses

Abstract

The evidence concerning patient observation skills among critical care nurses is scarce, and these skills have not been evaluated using both subjective and objective assessment. This study aimed to explore associations between self-assessment and knowledge test assessment of critical care nurses’ skills in patient observation, thus combining subjective and objective assessment. The data of this cross-sectional study among Finnish critical care nurses were collected using an instrument combining self-assessment and knowledge test assessment. The study was reported following the STROBE guidelines. Almost half of the critical care nurses responded (n = 372, 49%). Only a weak positive correlation was found between the scores of assessment methods. This highlights the complex nature of skills evaluation. Self-assessment has limitations in reliability and accuracy, but it has potential in promoting nurses’ professional development. There is a need for developing and supporting self-assessment in clinical nursing practice.

Keywords

competency assessment critical care nursing knowledge test nursing skills patient observation self-assessment

Background

Nurses’ high-level skills form a foundation for accountable and good quality healthcare.^1,2 Nurses apply technical, interpersonal and cognitive skills in their daily clinical nursing care.³ Acquisition of these skills continues throughout a nurse's career⁴ because nurses are constantly adopting new skills and updating existing skills due to technological development, societal changes⁵ and other issues, such as sudden outbreaks of infectious diseases.⁶ The importance of constant skills acquisition is accentuated among critical care nurses (CCNs), who work in a rapidly developing field and care for the most severely ill patients.⁷ Therefore, skills development continues to be a highly prioritised topic within nursing research.⁸ To develop skills, nurses need they. Not nurses as this refers to the skills to be evaluated to detect the possible need for improvement.^2,9 Skills evaluation may use either a subjective assessment or an objective assessment method.¹⁰ Subjective assessment is usually based on self-assessment,^11,12 whereas objective assessment is typically based on structured tests,^10,13,14 observations^15,16 and assessment by others.¹⁷

Self-assessment can be defined as ‘the act of monitoring one's processes and products in order to make adjustments that deepen learning and enhance performance’.¹⁸ This definition introduces the formative and summative nature of self-assessment; thus, it may focus on the learning process as well as the product of that process.¹⁹ Self-assessment develops lifelong learning abilities,^20–22 especially self-regulated learning, which is essential in continuous professional development.^19,23 Self-assessment has an important role in skills evaluation in nursing, and in working life in general. It enables an individual to assess a wide range of skills that may be unobservable by other methods and by other people,¹² and it facilitates deep insight into non-technical skills in clinical practice.²⁴ Furthermore, to help ensure patient safety it is essential that healthcare professionals constantly assess their decisions and actions.²⁵

Objective assessment entails collecting evidence representing the truth about an individual's skills.²⁶ Thus, objective assessment is summative in nature and produces information about learning outcomes based on pre-defined criteria.¹¹ Objective assessment includes structured tests, such as knowledge tests.^27,28 Many nursing skills embody application of advanced and situation-specific knowledge; therefore, they can be considered to be cognitive skills, and knowledge tests can be used to assess them.^13,27 Structured observation using objective criteria may be used for assessing skills in a real clinical environment or in simulated scenarios.^15,16 Assessment by others may also yield a relatively high level of objectivity when based on carefully structured objective assessment criteria.¹⁷ However, both observation and assessment by others may be affected by subjectivity bias.²⁹

Ideally, skills evaluation should incorporate multifaceted assessments and techniques to comprehensively cover different skill dimensions.^2,30 The use of both subjective and objective assessments could yield a more comprehensive skills evaluation. For example, nurses’ self-assessment has been combined with assessment by nurse managers^31,32 and peers.¹⁷ Some studies have used self-assessment with structured observations^33,34 as well as with a knowledge test.^35,36

Overall, the results from self-assessments and objective assessments can be incongruous. Nurses have been more critical of their skills in their self-assessment in comparison to assessments made by others.^17,31,32 Whereas observations and an objective knowledge test assessment have suggested that nurses have lower skill levels than when nurses use self-assessment to rate their skills.^33–35 Therefore, self-assessment has been criticised as unreliable. However, recent studies among nurses in post-graduation programmes suggest that self-assessment is a suitable and reliable way to identify the level of nurses’ competence, as the nurses were able to realistically identify the skills they needed to develop.^37,38

The present study focused on skills evaluation among CCNs. CCNs are working in intensive care units (ICUs), and as they care for severely and acutely ill or injured patients they need to frequently verify their skills to respond to the continual development in the field³⁹ and rapidly evolving challenges as seen during the COVID-19 pandemic.⁷ Every day, CCNs apply many skills in their clinical practice; and patient observation is one of these skills.⁴⁰

CCNs are at the front line detecting rapid changes in a patient's clinical condition,⁴¹ thus, the skills to observe patients are a focal element of their work. In order to fulfil their observation responsibilities, CCNs utilise diverse monitoring techniques, such as invasive blood pressure and oxygen saturation measurements,⁴² and non-technical assessment methods such as palpation of peripheral pulses and respiratory auscultation.⁴³ CCNs’ skills in patient observation include information-gaining and information-processing skills. Hence, CCNs gain information regarding the clinical condition of the patient and they process that information for example by combining their observations on various physiological functions.⁴⁰ Patient observation skills entail application of advanced and situation-specific knowledge; therefore, they can be considered to be cognitive skills.^13,27,40

Previous studies have evaluated CCNs’ skills in some areas of patient observation by employing either subjective self-assessment^43–45 or objective assessment in the form of structured tests^46,47 and observations.^42,48 Only a few studies have used both self-assessment and knowledge test assessment,^35,36 even though combining approaches could provide more a comprehensive evaluation. Moreover, those studies have examined some specific physiologic areas and technical methods of patient observation or have included patient observation in general competence evaluation, and therefore do not draw a clear and comprehensive picture of CCNs’ observation skills.

The aim of this study was to explore the associations between subjective self-assessment and objective knowledge test assessment of CCNs’ patient observation skills. The study provides information that may be used in developing skills assessment practices in ICUs. The research question of the study was:

Do subjective and objective assessments associate in evaluation of the patient observation skills among CCNs?

Methods

This was a national multicentre, cross-sectional study, which utilised an instrument specifically developed for the study as a data collection tool among CCNs.⁴⁹ The data were analysed with descriptive and inferential statistics. The study is reported following the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement.⁵⁰

Setting

The study was carried out in all five university hospitals in Finland. Altogether seven adult ICUs participated in the study. There are ICUs outside university hospitals as well, but the level of critical care services provided in these non-university hospitals varies more, and therefore this study was limited to university hospital ICUs providing care at the most demanding level.

Sample

All the CCNs in the ICUs of the university hospital ICUs (N = 767) were invited to participate; thus, total sampling was used. This study is an independent part of a research project with two earlier publications^14,51 using the same sample.

Data collection

The data collection was carried out between September 2017 and January 2018. An instrument for data collection was developed specifically for this study, because there were no tools evaluating the patient observation skills available. The instrument's skill domains, sum-variables (Figure 1) and items were based on the theoretical construction provided by the previous study describing patient observation skills.^14,51 Moreover, previous literature was used to complement item construction. The instrument combined self-assessment and knowledge test assessment, hence subjective and objective assessment methods. Self-assessment focused on the skill domain of information-gaining and the knowledge test assessment focused on the skill domain of information-processing.⁴⁰ Information-gaining skills were divided into three sum-variables: bio-physiological foundation of observation, skills in using observation methods and skills in recognising the changing clinical condition. Respectively, information-processing skills were divided into two sum-variables: skills in evaluating the reliability of observation and analytic thinking skills. All sum-variables contained items covering seven different physiological observation areas (Figure 1).

Figure 1.

The structure of the instrument evaluating critical care nurses’ patient observation skills.

The self-assessment consisted of 56 items demonstrating the CCNs’ skills in gaining information (Figure 1). A visual analogue scale (VAS, 0–100) was used in the assessment (zero representing non-existing skill level and one hundred representing the highest skill level). The objective knowledge test focused on information-processing skills (Figure 1). It comprised 20 items, including multiple-choice questions, true/false statements, and one constructed response question. The knowledge test statements were either brief clinical cases or statements to enhance authenticity,²² thus, requiring the CCNs to use their cognitive skills to analyse and evaluate the content.¹² The maximum score in the knowledge test was 20 points (correct = 1, false = 0). The self-assessment (VAS) scores were categorised as follows: 0–20 = poor, 21–40 = fair, 41–60 = average, 61–80 = good, 81–100 = excellent. Due to the critical importance of patient observation skills in CCNs’ responsibilities, the score of excellent was considered to be a required skill level for a practicing CCN. For the knowledge test, a score of 100% was required for a practicing CCN.

The content validity of the developed instrument was evaluated by a panel of 11 critical care nursing experts (CCNs, educators and researchers). The content validity index (S-CVI) was 0.96 for the self-assessment and 0.93 for the knowledge test, demonstrating excellent content validity.⁵² The pre-test was used to evaluate the instrument's clarity and applicability, and to test the self-assessment items’ distribution and internal consistency. Convenience sampling⁵² among CCNs (n = 46, response rate 46%) at two ICUs (not included in the sample) was used in the pre-test. Distributions for the self-assessment items’ sum-variables were sufficient, as was the internal consistency (Cronbach’s alpha 0.86–0.92).⁵²

Paper-and-pencil questionnaires that contained demographic and educational background questions, self-assessment items (56 items) and knowledge test items (20 items) were distributed to the CCNs (N = 767) by the contact persons in the ICUs. The researcher organised brief information sessions about the study in the ICUs, and an information letter was provided to CCNs before the data collection. The CCNs were instructed to answer independently, based on their perceptions and best understanding. The CCNs were not controlled in any way when answering the questions.

Data analysis

The data were analysed using IBM SPSS Statistics for Windows version 24 (IBM Corp., Armonk, NY, USA). The sample characteristics were analysed and the sums were calculated using descriptive statistics. Cases in which at least 85% of the items were answered were included in sum-variable calculation. Correlations were calculated between continuous variables (Pearson and Spearman). Comparison of self-assessed patient observation skills between the groups was carried out using independent samples t-test. Data from the CCNs who finished answering the background section without continuing to the instrument itself (n = 4) were excluded. The level of statistical significance was set at 0.05.

Ethical considerations

The study protocol was reviewed and accepted by the ethics committee of the University of Turku (approval no. 37/2016). Moreover, the hospitals granted research permission to conduct the study. In the cover letter accompanying the questionnaire, the CCNs were informed that their answers would remain confidential and that their participation was voluntary. The questionnaires were returned in a sealed envelope in a covered box. Data collection did not include identifiable personal data. Completing the questionnaire and returning the envelope were considered an indication of informed consent. The study was conducted in accordance with the Declaration of Helsinki.⁵³

Results

Participants’ characteristics

Nearly half of the CCNs (49%, n = 372) returned the questionnaire. The majority of them were women (n = 316, 85%) and their mean age was 40 years (SD 10.3). The average working experience in nursing was 15 years (SD 9.8), of which 12 years (SD 9.3) were in critical care nursing. Of the CCNs, 15% (n = 54) had accomplished some continuing education courses in critical care nursing (with a certificate) and around two-thirds of the CCNs (n = 252, 70%) had been trained for special tasks in the ICU. These tasks may include, for example, performing renal replacement therapies and extracorporeal membrane oxygenation and working in a rapid response team.

Association between subjective self-assessment and the objective knowledge test scores

Only a weak positive correlation was found between the level of self-assessed skills and the objective knowledge test scores. When examining the correlations at the sum-variables level (Figure 1), only a weak positive correlation was found between the knowledge test score and two self-assessment sum-variables: skills in using observation methods and skills in recognising changing clinical condition. A weak positive correlation was also found between the sum-variables analytic thinking skills (knowledge test) and skills in using observation methods (self-assessment) (Table 1).

Table 1.

Correlations between critical care nurses’ self-assessment and knowledge test scores (n = 368).

Self-assessment	Knowledge test
Self-assessment	b. Information processing skills †	b1. Evaluating the reliability f observation †	b2. Analytic thinking ‡
a. Information-gaining skills	.11*	.06	.09
a1. Bio-physiological foundation of observation	.06	.05	.02
a2. Skills in using observation methods	.13*	.06	.13*
a3. Skills in recognising changing clinical condition	.11*	.06	.10

† = Pearson's correlation. ‡ = Spearman's correlation.

* = p < 0.05.

The mean score of the CCNs’ self-assessed information-gaining skills was 81.9 (SD 11.5), which is considered to be excellent. On average, 77% of the questions in the knowledge test were answered correctly (mean 15.3/20, SD 2.4), which can be considered to be suboptimal. The CCNs who obtained an average score (15.3) or higher in the knowledge test also assessed their information-gaining skills to be a bit higher (mean 82.9, SD 10.7) than the CNNs who obtained a less than average score (mean 80.1, SD 12.8, p = 0.040). The CCNs who obtained an average score or higher in the knowledge test also assessed their skills as being higher in the following sum-variables: skills in using observation methods (mean 84.2, SD 10.1 vs. mean 81.7, SD 12.1, p = 0.042) and skills in recognising the changing clinical condition (mean 83.6, SD 11.3 vs. mean 80.1, SD 13.4, p = 0.014).

Discussion

This study revealed discordance between the subjective and the objective assessment methods in the evaluation of CCNs’ patient observation skills. According to the self-assessment, CCNs’ patient observation skills are at an excellent level, whereas their knowledge test scores suggest that their skills are suboptimal. There was only a weak correlation between the scores of the two assessment methods. Thus, this study's findings align with the results reported in previous studies about the discordance between subjective self-assessment and objective assessment of CCNs’ skills. In those previous studies, the CCNs assessed their skills as being higher than the objective assessment scores indicated.^33–35 However, some agreement between these two assessments can also be found in this study. The CCNs with higher knowledge test scores assessed their patient observation skills to be a bit higher. Presumably, the nurses performing well in the objective test should also be confident in their skill level. However, many of the nurses whose skills were objectively not that good felt confident about their skills, which is a rather surprising and thought-provoking finding given the importance of self-assessment in continuous professional development.²³

Several factors could possibly explain the discordance between the self-assessed skills and the knowledge test scores. First, the accuracy of the CCNs’ self-assessment may be debatable, as previous literature suggests.^33–35 In the current study, the CCNs had been working an average of 12 years in critical care nursing, thus they were experienced. Therefore, it can be assumed that their self-assessment should not be limited by a lack of understanding of the required skill level, which sometimes limits the accuracy of a self-assessment especially among nursing students and newly graduated nurses.^11,20,54 However, the possibility of the required skill level being obscure to some of the CCNs cannot be ruled out; it has to be considered as a possible factor affecting the self-assessment results.⁵⁵ The lack of self-assessment skills is another possible factor limiting the accuracy of the self-assessment.^30,54 There is no evidence about the level of nurses’ self-assessment skills, and it is known that the acquisition of self-assessment skills requires time and training,¹⁹ which may not have been systematically introduced in the ICUs included in the present study. However, it is worth acknowledging that self-assessment does not necessarily yield higher scores than other types of assessments. When assessing one's skills, each individual sets the level one strives to achieve;¹⁹ thus, nurses may also underestimate their skills.^17,33 Some evidence within and beyond the field of nursing has suggested that individuals performing well in an objective assessment tend to underestimate their skills, whereas those performing less well tend to overestimate their skills.^11,21,33,56 Similarly, in the current study, some of the CCNs who scored well in the knowledge test were relatively critical in their self-assessments.

Second, there is a possibility that the level of difficulty in the knowledge test assessment was too demanding. It was developed for the study, and it was used for the first time. Thus, there is no baseline available for comparison. However, the panel of experts evaluated the relevance of the items, indicating excellent content validity. Furthermore, some studies have reported similar findings for suboptimal knowledge test scores in other areas among Finnish CCNs,^33,57 hence supporting the current study's findings.

Third, the highly collaborative nature of critical care may also affect the results. CCNs usually discuss their observations and concerns with each other and consult their colleagues and physicians when needed.³⁵ This obviously improves the quality of the observation, and demonstrates a high level of shared observation skills, which on some occasions may lead an individual CCN to overestimate their skill level. While answering the questions in the knowledge test, the CCNs presumably did not have a chance to consult with their colleagues or with physicians, which may have resulted in lower skill levels.

Fourth, the self-assessment and knowledge test assessment focused on different domains of patient observation skills; the self-assessment focused on information-gaining and the knowledge test focused on information-processing. However, these skill domains are tightly intertwined, and having strong patient observation skills requires mastering both. CCNs process and gain information simultaneously, for example, by evaluating the reliability of the data while performing physiological measurements.^40,58 Therefore, information-processing skills are key to providing a relevant, reliable and holistic interpretation of a critically ill patient's condition.⁴⁰ Thus, this study's findings indicate the need to develop the patient observation skills of CCNs, with a special focus on processing information.

The lack of correlation between the assessment results may challenge the reliability of self-assessments. However, the CCNs who scored better than average in the knowledge test also assessed their skills as being higher. Hence, the CCNs whose skills are objectively at the higher level are able to assess their skills consistently. Whereas the CCNs who performed less well on the knowledge test might have overestimated their skills, as reported in previous studies.^11,21,33,56

Strengths and limitations

The current study was the first to evaluate patient observation skills of CCNs using both subjective assessment and objective assessment, which made the evaluation more comprehensive. Hence, this approach can be seen as the main strength of the study. The second strength is the relatively large sample including all the university hospitals in the country. Furthermore, the sample was diverse in terms of the CCNs’ work-related experience, even though the average working experience in years was rather long.

This study also has some limitations. The first has to do with the evaluation instrument, which was developed for the study and used for the first time. The theoretical structure of the instrument was not supported by the exploratory factor analysis that did not provide a meaningful alternative structure.⁵¹ However, the instrument was structured based on previous evidence reported in the literature. Regarding internal consistency, all the sum-variables in the self-assessment domain had high Cronbach's alphas (0.91–0.97). The second limitation is the response rate (49%). Only about half of the CCNs completed and returned the survey; therefore, the representativeness of the sample is limited. Third, a relatively large proportion of the respondents, around two-thirds of the CCNs in this study, had been educated for special tasks in ICUs. The proportion of CCNs having this kind of training in Finnish ICUs is not known, hence, the representativeness of the sample among Finnish CCNs remains unclear in this sense. Fourth, only CCNs working in university hospitals were included in the study. Therefore, the results may not be applicable in the non-university hospital settings in Finnish. The fifth limitation concerns the uncontrolled test situation. Thus, it is possible that CCNs searched information and discussed it while answering the knowledge test questions. However, the knowledge test items required the application of knowledge; thus, the correct answers were not directly searchable from any source.

Implications for practice

Taking notice of self-assessment skills and practices is of great importance in nursing. Developing and supporting self-assessment in clinical nursing practice is recommended because self-assessment is an essential factor in self-regulated learning and in continuous professional development.^21,24 Nurse managers and nurses themselves should work together to create and maintain open and supportive workplace culture, valuing self-assessment as a method of continuous professional development. Self-assessment should be a systematic and regular practice in clinical nursing, and could be used simultaneously with objective assessment methods. Feedback from the objective assessment could be provided to encourage nurses to develop the accuracy of their self-assessment,^21,56 and to facilitate their professional development. The instrument developed for this study could be used as a framework in an ICU context. It is also important to explicitly and clearly define the skill level criteria in healthcare organisations and units to support self-assessment.⁵⁵ Combining objective methods and self-assessment also deserves attention during nurses’ basic professional training in order to provide graduating nursing students with adequate self-assessment skills.

Conclusion

Discordance between the subjective and objective assessment of patient observation skills highlights the complexity of skills evaluation in clinical nursing practice. Skills evaluation is an important element of assuring good quality care and patient safety; thus, reliable and effective assessment methods are needed. Self-assessment has limitations in terms of its reliability and accuracy, but it has undisputable potential in promoting nurses’ professional development. Self-assessment could be combined with knowledge tests, peer- and mentor-assessment as well as performance assessment in real-world clinical situations. Developing criteria for the required skill levels in critical care nursing is recommended, as is providing CCNs with feedback from their self-assessment. Evidence of the level of self-assessment skills among nurses is needed.

Footnotes

Acknowledgements

The authors wish to express gratitude to all the critical care nurses participating in the study, as well as the contact persons in the ICUs. The authors wish to thank MSc Helena Ollila for her help in recording the research data, MD Waltteri Siirala for checking the medical accuracy of the knowledge test items.

Declaration of conflicting interests

The authors declare that there is no conflict of interest.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Finnish Foundation of Nursing Education, and the Finnish Nurses Association.

ORCID iD

Mika Alastalo

References

Allen-Duck

Robinson

Stewart

. Healthcare quality: a concept analysis. Nurs Forum 2017; 52: 377–386.

Fotheringham

. Triangulation for the assessment of clinical nursing skills: a review of theory, use and methodology. Int J Nurs Stud 2010; 47: 386–391.

International Council of Nurses. Scope of nursing practice and decision-making framework TOOLKIT, https://www.icn.ch/sites/default/files/inlinefiles/2010_ICN%20Scope%20of20Nursing%20and%20Decision%20making%20Toolkit_eng.pdf (2010, accessed 1 March 2022).

Welch

Carter

. Deliberate practice and skill acquisition in nursing practice. J Contin Educ Nurs 2018; 49: 269–273.

Liu

Aungsuroch

. Current literature review of registered nurses’ competency in the global community. J Nurs Scholarsh 2018; 50: 191–199.

Schwerdtle

Connell

Lee

, et al. Nurse expertise: a critical resource in the COVID-19 pandemic response. Ann Glob Health 2020; 86: 1–5.

Brickman

Greenway

Sobocinski

, et al. Rapid critical care training of nurses in the surge response to the coronavirus pandemic. Am J Crit Care 2020; 29: 104–107.

Hopia

Heikkilä

. Nursing research priorities based on CINAHL database: a scoping review. Nurs Open 2020; 7: 483–494.

Windsor

Douglas

Harvey

. Nursing and competencies – a natural fit: the politics of skill/competence formation in nursing. Nurs Inq 2012; 19: 213–222.

10.

Moore

Jr . Assessment of learning and program evaluation in health professions education programs. New Dir Adult Cont Educ 2018; 157: 51–64.

11.

Forsman

Jansson

Leksell

, et al. Clusters of competence: relationship between self-reported professional competence and achievement on a national examination among graduating nursing students. J Adv Nurs 2020; 76: 199–208.

12.

Humburg

van der Velden

. Self-assessments or tests? Comparing cross-national differences in patterns and outcomes of graduates’ skills based on international large-scale surveys. Stud High Educ 2015; 40: 482–504.

13.

Brannan

White

Bezanson

. Simulator effects on cognitive skills and confidence levels. J Nurs Educ 2008; 47: 495–500.

14.

Alastalo M, Salminen L, Vahlberg T, et al. Knowledge of patient observation skills. Nurs Crit Care 2021; 26: 341–351.

15.

Jeon

Lakanmaa

Meretoja

, et al. Competence assessment instruments in perianesthesia nursing care: a scoping review of the literature. J Perianesth Nurs 2016; 32: 542–556.

16.

Lyk-Jensen

Dieckman

Konge

, et al. Using a structured assessment tool to evaluate nontechnical skills of nurse anesthetists. AANA J 2016; 84: 122–127.

17.

Takase

Yamamoto

Sato

. The factors related to self-other agreement/disagreement in nursing competence assessment: comparative and correlational study. Int J Nurs Stud 2018; 80: 147–154.

18.

Andrade

. A critical review of research on student self-assessment. Front Educ 2019; 4. DOI: 10.3389/feduc.2019.00087.

19.

Panadero

Alonso-Tapia

. Self-assessment: theoretical and practical connotations. When it happens, how is it acquired and what to do to develop it in our students? Electron J Res Educ Psychol 2013; 11: 551–576.

20.

Kajander-Unkuri

Leino-Kilpi

Katajisto

, et al. Congruence between graduating nursing students’ self-assessments and mentors’ assessments of students’ nurse competence. Collegian 2016; 23: 303–312.

21.

Lundquist

Shogbon

Momary

, et al. A comparison of students’ self-assessments with faculty evaluations of their communication skills. Am J Pharm Educ 2013; 77: Article 72.

22.

Sambell

McDowell

Montgomery

. Assessment for learning in higher education. New York: Routledge, 2012.

23.

Yan

Brubacher

Boud

, et al. Psychometric properties of the Self-assessment Practice Scale for professional training contexts: evidence from confirmatory factor analysis and Rasch analysis. Int J Train Dev 2020; 24: 357–373.

24.

Osborne

Hawkins

Pournaras

, et al. An evaluation of operative self-assessment by UK postgraduate trainees. Med Teach 2014; 36: 32–37.

25.

Epstein

Siegel

Silberman

. Self-monitoring in clinical practice: a challenge for medical educators. J Contin Educ Health Prof 2008; 28: 5–13.

26.

McDonald

. The nurse educator’s guide to assessing learning outcomes. 4th ed. Burlington: Jones & Bartlett Learning, 2018.

27.

Dankbaar

MEW

Alsma

Jansen

EEH

, et al. An experimental study on the effects of a simulation game on students’ clinical cognitive skills and motivation. Adv Health Sci Educ 2016; 21: 505–521.

28.

Tanner

. The critical state of measurement in nursing education research. J Nurs Educ 2011; 50: 491–492.

29.

Immonen

Oikarainen

Tomietto

, et al. Assessment of nursing students’ competence in clinical practice: a systematic review of reviews. International Int J Nurs Stud 2019; 100: 103414.

30.

Sears

Godfrey

Luctkar-Flude

, et al. Measuring competence in healthcare learners and healthcare professionals by comparing self-assessment with objective structured clinical examinations: a systematic review. JBI Database Syst Rev Implement Rep 2014; 12: 221–272.

31.

Meretoja

Leino-Kilpi

. Comparison of competence assessments made by nurse managers and practising nurses. J Nurs Manag 2003; 11: 404–409.

32.

Numminen

Leino-Kilpi

Isoaho

, et al. Congruence between nurse managers’ and nurses’ competence assessments: a correlation study. J Nurs Educ Pract 2015; 5: 142–150.

33.

Ballangrud

Persenius

Hedelin

, et al. Exploring intensive care nurses’ team performance in a simulation-based emergency situation – expert raters’ assessments versus self-assessments: an explorative study. BMC Nurs 2014; 13: 47.

34.

Williams

Parry

. Knowledge and skills of critical care nurses in assisting with intubation. Br J Nurs 2018; 27: 204–209.

35.

Lakanmaa

Suominen

Ritmala-Castrén

, et al. Basic competence of intensive care unit nurses: cross-sectional survey study. BioMed Res 2015; Article 536724.

36.

McGhee

Woods

. Critical care nurses’ knowledge of arterial pressure monitoring. Am J Crit Care 2001; 10: 43–51.

37.

Taylor

Bing-Jonsson

Wangensteen

, et al. The self-assessment of clinical competence and the need for further training: a cross-sectional survey of advanced practice nursing students. J Clin Nurs 2019; 29: 545–555.

38.

Wangensteen

Finnbakk

Adolfsson

, et al. Postgraduate nurses’ self-assessment of clinical competence and need for further training. A European cross-sectional survey. Nurse Educ Today 2018; 62: 101–106.

39.

World Federation of Critical Care Nurses. Position statement on the provision of a critical care nursing workforce, http://wfccn.org/wp-content/uploads/2019/09/WFCCN-Workforce-PS_17.9.19.pdf (2019, accessed 8 January 2022).

40.

Alastalo M, Salminen L, Lakanmaa RL, et al. Seeing beyond monitors - Critical care nurses' multiple skills in patient observation: Descriptive qualitative study. Intensive Crit Care Nurs 2017; 42: 80–87.

41.

Jones

Johnstone

. Inattentional blindness and failures to rescue the deteriorating patient in critical care, emergency and perioperative settings: four case scenarios. Aust Crit Care 2017; 30: 219–223.

42.

Freysdóttir

Björnsdóttir

Svavarsdóttir

. Nurses’ use of monitors in patient surveillance: an ethnographic study on a coronary care unit. Eur J Cardiovasc Nurs 2019; 18: 272–279.

43.

Cicolini

Tomietto

Simonetti

, et al. Physical assessment techniques performed by Italian registered nurses: a quantitative survey. J Clin Nurs 2015; 24: 3700–3706.

44.

Okumura

Ishigaki

Mori

, et al. Development of an easy-to-use questionnaire assessing critical care nursing competence in Japan: a cross-sectional study. PLOS ONE 2019; 14: e0225668. DOI: 10.1371/journal.pone.0225668.

45.

Sowan

Vera

Fonseca

, et al. Nurse competence on physiologic monitor use: toward eliminating alarm fatigue in intensive care units. Open Med Inform J 2017; 11: 1–11.

46.

Preston

Currey

Considine

. Atrial electrogram interpretation improves after an innovative education program. Can J Crit Care Nurs 2015; 26: 13–18.

47.

Kerr

Bacon

Baker

, et al. Underestimation of pupil size by critical care and neurosurgical nurses. Am J Crit Care 2016; 25: 213–219.

48.

Kebapçı

Dikeç

Topçu

. Interobserver reliability of Glasgow Coma Scale Scores for intensive care unit patients. Crit Care Nurse 2020; 40: 18–26.

49.

Hasson

McKenna

Keeney

. Surveys. In: Gerrish

Lathlean

(eds) The research process in nursing. 7th ed. Chichester: John Wiley & Sons, 2015, pp.255–266.

50.

von Elm

Altman

Egger

, et al. STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol 2008; 61: 344–349.

51.

Alastalo M, Salminen L, Jeon Y, et al. Critical care nurses' self-assessed patient observation skills. Nurs Crit Care 2019; 24: 268–275.

52.

Polit

Beck

. Nursing research: generating and assessing evidence for nursing practice. 10th ed. Philadelphia: Wolter Kluwer, 2017.

53.

World Medical Association. Declaration of Helsinki – ethical principles for medical research involving human subjects, https://www.wma.net/policies-post/wma-declaration-of-helsinki-ethical-principles-for-medical-research-involving-human-subjects/ (2013, accessed 8 January 2022).

54.

Baxter

Norman

. Self-assessment or self-deception? A lack of association between nursing students’ self-assessment and performance. J Adv Nurs 2011; 67: 2406–2413.

55.

Boud

Lawson

Thompson

. The calibration of student judgement through self-assessment: disruptive effects of assessment patterns. High Educ Res 2015; 34: 45–59.

56.

Carroll

. Observations of student accuracy in criteria-based self-assessment. Assess Eval High Educ 2020; 45: 1088–1105.

57.

Jansson

Ala-Kokko

Ylipalosaari

, et al. Critical care nurses’ knowledge of, adherence to and barriers towards evidence-based guidelines for the prevention of ventilator-associated pneumonia: a survey study. Intensive Crit Care Nurs 2013; 29: 216–227.

58.

Hoffman

Aitken

Duffield

. A comparison of novice and expert nurses’ cue collection during clinical decision-making: verbal protocol analysis. Int J of Nurs Stud 2009; 46: 1335–1344.