Abstract
Evaluation of the clinical learning environment in teaching hospitals is an important quality improvement metric. The measure for the anaesthesia theatre educational environment is a validated tool measuring the educational environment in the operating theatre, specific to anaesthesia. Monitoring the learning environment provides information about the impact of interventions, the influence of major events such as a global pandemic and is increasingly linked to accreditation requirements. This study was conducted to establish a baseline for monitoring the impact of interventions designed to improve the learning environment. We surveyed trainees at the Royal Melbourne Hospital, a trauma hospital in Melbourne, Australia, aiming to identify areas for improvement in the clinical learning environment and provide a needs analysis for interventions to help anaesthesia trainees set and achieve learning goals. This single centre survey study occurred over a 6-week period from May 2023. Medical staff who had undertaken a training rotation at Royal Melbourne Hospital in the past 6 months were eligible. Twenty-six responses were received (response rate 39%). Areas for improvement included setting learning goals (mean 3.9, median 5, interquartile range 4–5) and assessment of trainee performance (mean 3.8, median 4, interquartile range 3–5). Over 80% of participants had received no training in how to set a learning goal, despite over 90% indicating that it would be of value to their experience. Trainee perception of the learning environment was positive but there is opportunity for improvement. We are planning interventions to assist trainees and consultants to set learning goals while working in a dynamic clinical learning environment.
Keywords
Introduction
The clinical learning environment (CLE), also referred to as the educational environment, describes a dynamic interaction between the learner and the context in which they function. It is shaped by learner-specific and environmental factors including institutional culture, available resources, and faculty characteristics.1,2 Each clinical setting has a distinct CLE, reflecting the strengths and limitations of that department or practice. 3 A supportive educational environment is critical for knowledge acquisition, skill development, and learner wellbeing. However, direct observation of clinical teaching encounters is often impractical. Educational environment measures (EEMs) are survey instruments administered to learners that have been developed for various clinical settings to serve as a surrogate for direct observation.4 –6
The measure for the anaesthesia theatre educational environment (MATE) is a validated, anaesthesia-specific tool designed to assess teaching in the operating theatre setting. 7 A survey instrument, the MATE questionnaire consists of 33 statements across four domains: teaching preparation and practice, assessment and feedback, procedures and responsibility and overall atmosphere. Participants are asked to rank each statement on a seven-point rating scale (0 = strongly disagree, 6 = strongly agree). Potential applications of MATE in anaesthesia training include its use as an educational key performance index to address areas of concern as they are identified. 7 It was used in this study to assess the CLE within the Department of Anaesthesia and Pain Management at the Royal Melbourne Hospital (RMH), a quaternary trauma centre. There is limited research on how learners interact with curricula in evolving clinical contexts and, to our knowledge, no published research evaluating the CLE for anaesthesia vocational training in a quaternary trauma hospital. By using the MATE tool, we aimed to identify domains for improvement and support targeted educational interventions.
The COVID pandemic significantly disrupted medical education globally, accelerating a shift to online and asynchronous learning formats. 8 Although in-theatre learning persisted, changes to elective surgery, unplanned supervisor and learner absences, and reduced opportunities for formal teaching likely impacted the CLE. 9 Other studies have reported decreased volumes of practice, fewer assessments, and reduced informal learning opportunities across healthcare settings during the pandemic.10 –12 However, the impact of these changes on Australian vocational anaesthesia training, particularly within the operating theatre, have not been explored.
No theatre-specific evaluation tools had been administered in our educational environment. We aimed to:
Establish a baseline evaluation of the clinical learning environment at RMH to meet accreditation requirements.13,14
Identify areas for improvement and inform interventions to support anaesthesia trainees in setting and achieving learning goals.
Explore the perceived impact of the COVID-19 pandemic on the anaesthesia clinical learning environment.
Methods
This was a single-centre, cross-sectional survey study conducted over 6 weeks from May to June 2023 in a quaternary trauma anaesthetic department. Eligible participants included medical staff (prevocational and vocational) who had completed a training rotation (3 months or more) at RMH in the preceding 6 months and were working at least one clinical session per week. Trainees not working clinically at RMH during the study period (due to leave or secondment) were excluded. The Office for Research Ethics and Governance team and a member of the Royal Melbourne Hospital Human Research Ethics Committee (HREC) assessed the project as an evaluation activity not requiring HREC review (QA2023016).
Participants were emailed a link to a one-time REDCap 15 online survey by the Department of Anaesthesia and Pain Management. Supplementary recruitment occurred through departmental social channels and QR codes displayed in staff areas. Reminder emails were sent at 2 and 4 weeks, with the survey closing at 6 weeks.
The survey collected basic demographic data, responses to the 33-item MATE questionnaire, and qualitative responses regarding the impact of the COVID-19 pandemic on anaesthesia training and setting learning goals. Participants were asked to respond to statements relating to the MATE domains: teaching preparation and practice, assessment and feedback, procedures and responsibility, overall atmosphere; using a 0–6 rating scale (0 = strongly disagree, 6 = strongly agree). Free text responses were optional. Completion of the survey implied consent. The survey instrument is provided in Appendix 1.
It was estimated that there would be approximately 60 trainees eligible to undertake the survey, a sample size of convenience based on the number of doctors in training who have undertaken a rotation in the Department of Anaesthesia and Pain Management at RMH during the previous 6 months. Study responses were stored in the study databases (REDCap and Excel spreadsheet) as non-identifiable data.
Quantitative data were reported using descriptive statistics including percentages, mean, median, and interquartile range (IQR). Thematic analysis of qualitative data was performed by two members of the research team, both fellows of the Australian and New Zealand College of Anaesthetists (ANZCA), using a framework analysis based on MATE domains. 7 One was an experienced qualitative researcher (KJA) and both have higher qualifications in education theory and practice. Discrepancies in thematic analysis were resolved through discussion.
Consistent with the original MATE study, 7 individual item scores were summed to determine the overall MATE score out of 198 (33 items each with a maximum score of 6), expressed as a percentage. Domain-specific scores were calculated similarly. Respondents’ scores for the overall MATE were only included if they provided responses for all items, and for each domain only if they provided responses for all items within that domain. Interpretation of scores followed predefined cut-offs 7 : 0–50% = poor, 50.1–60% = below average, 60.1–70% = average, 70.1–80% = good, 80.1–90% = very good, and 90.1–100% = excellent.
Results
Demographics
Demographic data are summarised in Table 1. Of 70 eligible trainees, 27 completed the survey (38% response rate). All responses met the inclusion criteria. Most participants identified as men (63%), were aged 31–35 years (56%), and were at advanced stages of training (63% advanced trainee or provisional fellow).
Demographic data.
The distribution of participant training level is illustrated in Figure 1. This reflects the trainee make-up of the department, where over 50% of training positions are allocated to fellows or advanced trainees. It is also representative of the trainee group at the time the survey was conducted (71% of the trainees working in the department were men). Thirteen participants (50%) had completed the ANZCA Fellowship Exam while three (12%) indicated that they were preparing to sit either the primary or fellowship exam in the upcoming sitting.

Survey responses by training level.
MATE scores
The mean overall MATE score was 76%, with domain scores as follows: ‘teaching preparation and practice’ 71%, ‘assessment and feedback’ 74%, ‘procedures and responsibility’ 80.1 %, and ‘overall atmosphere’ 82.4% (Table 2).
Overall MATE and domain scores recorded as number of responses (percentage of responses).
When assessing individual statements, areas for improvement were identified as statements scoring less than 4 (see Table 1 in Supplemental data). These were setting learning goals for a theatre session (mean 3.9, median 5, IQR 4–5) from the teaching preparation and practice domain and assessment of trainee performance from the assessment and feedback domain (mean 3.8, median 4, IQR 3–5). Statements scoring highest (>5) included ‘I am aware of my duties and responsibilities in theatre’ (mean 5.2, median 5, IQR 5–6), ‘My clinical teachers provided appropriate support when I perform a procedure for the first time’ (mean 5.1, median 5, IQR 5–6), both from the procedures and responsibility domain, and ‘I feel able to ask the questions I want to’ (mean 5.3, median 5, IQR 5–6) from the overall atmosphere domain.
Learning goals
Most respondents (85%) reported receiving no training in setting learning goals, despite 96% indicating that such training would be of value. This was supported by qualitative statements, for example: I need a more structured way of approaching and communicating my learning goals for each theatre list. . . having a formal session on how to formulate goals, and then reflect on how you evaluate goals you set for yourself would help me get the most out of the excellent learning environment that is the one-to-one, consultant-to-junior nature of anaesthetics in theatre.
Responses highlighted common barriers to setting learning goals while undertaking clinical work: time pressures of the list, changes in theatre allocations, high volume of emergency cases, and varying levels of consultant engagement. Several responders also cited unfamiliarity with supervisors as having a negative impact on effective learning: ‘A lack of understanding of my level of skills. Setting goals are usually easier if I work with the same teacher on multiple occasion’, and ‘Also having a new consultant each AM/PM list is good for a lot of reasons, but it can make communicating your learning needs a bit harder’.
Impact of the COVID-19 pandemic
Twenty-one respondents (81%) reported that the COVID-19 pandemic negatively impacted their learning, primarily through reduced access to theatre cases. Encouragingly, 96% indicated that their progression through training was unaffected by the pandemic.
There was also the suggestion of an overall diminished learning experience due to fatigue. Responses included comments such as: ‘Fatigue of senior staff has led to decreased teaching’, ‘Higher stress load and burn out in clinical educators’ and ‘Reduced list availability is probably only a small component but almost all aspects of training are significantly affected in one way or another’.
Perceptions were mixed on clinical topics delivered in an online learning format, with some participants missing the social interactions and others enjoying the increased accessibility. Comments ranged from: ‘I think there were many positive changes like increased flexibility in the way we deliver content (like normalising zoom tutorials/journal clubs etc.) has allowed me to attend more sessions’, to ‘I miss the social interactions with other trainees and its generally less engaging’ and ‘I do not enjoy learning via zoom’ (Table 3).
Respondents’ comments.
Discussion
This pilot study used a validated, anaesthesia-specific tool to evaluate the CLE for anaesthesia trainees in a quaternary trauma hospital. The overall MATE score of 76% falls within the ‘good’ category, suggesting a generally positive clinical learning environment.
However, domain scores varied, and several individual items revealed opportunities for improvement, particularly around setting learning goals and obtaining feedback within the clinical setting. These results provide a useful baseline to support and measure targeted educational initiatives, and meet accreditation requirements.
There is no published research on the CLE for vocational training in a quaternary trauma hospital, and we are unable to provide a direct comparison for our results. In comparison with the MATE development study, 7 three respondents (11.5%) submitted a MATE score of less than 50% (vs. 8.8% in MATE development), while 19 MATE respondents (73%) rated their educational environment as greater than 70% (vs. 68% in the development study). The increase in both the high and low ratings may be due to differences in the characteristics of the respondents, the timing of the survey related to the COVID pandemic, the small number of respondents, or the nature of the quaternary hospital environment. The scores for the teaching preparation and practice (71%) and assessment and feedback (75%) domains were lower than for procedures and responsibility (80%) and overall atmosphere (82.4%). The MATE development study showed similar lower results for these areas. It is not clear what the benchmark for exceptional performance is in these two domains. 7 Initiatives to improve the CLE at our institution will focus on these domains to assist in determining whether a lower score means an actual improvement gap or whether higher scores are unable to be achieved in these categories.
The distribution of responses was consistent with the demographics of the trainees in the department. The demographic data also reflected the composition of the training group at the time (71% men). However, with over half of the participants (61.9%) having completed the ANZCA final exam, this may have resulted in a larger number of positive responses. The survey was completed mid-year, with only three respondents disclosing active exam preparation. Conducting the survey earlier in the clinical year when, typically, more trainees are preparing for primary or fellowship examinations, may capture a different trainee perspective of the learning environment. Finally, the CLE at a quaternary trauma and referral centre may not allow direct comparison with most other teaching hospitals. Even so, piloting the MATE tool at our institution demonstrates that this is a feasible and valuable method to assess and compare trainee experiences.
Setting learning goals is believed to be critical to the successful transfer of knowledge and skills.16,17 Learning goals require the learner to evaluate knowledge, identify gaps, and work with educators to address these gaps. 18 These are often based around predetermined competencies set by regulatory bodies, yet limited explicit training in setting learning goals exists in the ANZCA curriculum. 19 The ANZCA educators program, while including content on setting learning goals, is not a mandatory requirement for anaesthesia training. Our survey found that anaesthesia trainees had not had specific training in developing or implementing a learning goal during a theatre teaching session, providing a target for intervention.
While the COVID-19 pandemic impacted the learning environment, it was encouraging that most reported no effect on training progression. Several responses indicated reduced consultant engagement in teaching as a key factor that affected learning, and a mixed response to hybrid education formats. Interventions incorporating social learning are likely to be more valuable if delivered in person rather than online, while enduring resources address issues related to equitable access.
Deriving a percentage score from a rating scale is accepted practice in the interpretation of EEM results to allow for ease of interpretation across environments and instruments. 7 While there is some evidence that a four-point Likert scale may be sufficiently reliable, 20 a seven-point scale such as we used is less likely to produce inflated scores. Our survey included a 0 marker, allowing us to use a scale from 0% to 100%. The scales were not labelled internally (beyond 0 = strongly disagree and 6 = strongly agree) to reduce acquiescence response bias.
Our study has several limitations. Firstly, the small sample size allowed only descriptive statistics to be used for this pilot. Although the response rate (39%) is consistent for an electronic survey 21 and the completion rate was high (100%), the risk of non-responder bias persists. The response rate may reflect a lack of trainee engagement or polarising experiences of the CLE at our institution that some may not have wanted to report. It can be difficult to obtain honest responses from trainees about their workplaces, as there may be a fear of judgement or perceived risk to career progression. 20
The results of this pilot have provided targets for intervention to improve the trainee experience and enhance clinical learning at our institution. As part of hospital orientation or during allocated teaching time, trainees will be provided with tools and information on developing and implementing a learning goal, as well as asking for and providing feedback. Consultant interviews and focus groups will be conducted to assess what further educational resources can be provided to assist with providing effective clinical teaching. Following intervention, the MATE questionnaire will be repeated on an ongoing basis, not only to evaluate experiences post-intervention, but to continue to identify new potential areas for improvement. We hope to demonstrate an improvement over time with interventions supporting trainees to set learning goals, and a needs analysis for consultants supplemented with targeted education.
Our study demonstrates how using an EEM can identify specific areas for improvement, even in a department with overall positive scores. This study contributes to the literature supporting the use of MATE, which may be used by regulatory bodies in the accreditation of training sites.13,14 Using a validated EEM, results may be compared between regions or states, and larger response numbers will allow for the valuable analysis of demographic subgroups to identify supports and barriers to learning.
Supplemental Material
sj-docx-1-aic-10.1177_0310057X251387719 – Supplemental material for Evaluation of the clinical learning environment in a quaternary trauma anaesthesia department: a pilot project
Supplemental material, sj-docx-1-aic-10.1177_0310057X251387719 for Evaluation of the clinical learning environment in a quaternary trauma anaesthesia department: a pilot project by Amelia R Marshallsea, Kara J Allen, Daryl L Williams and Sidhu N in Anaesthesia and Intensive Care
Footnotes
Appendix 1: Survey instrument
Please rate the following statements as they apply to your perception of teaching in the operating theatres of this department (applies to any site where anaesthesia is delivered, including endoscopy or interventional suites). Provide a rating from 0 to 6, with 0 = strongly disagree and 6 = strongly agree.
Teaching preparation and practice
Assessment and feedback
Procedures and responsibility
Overall atmosphere
Have you had any training on setting learning goals for a clinical session in theatre? Yes/No/Comment
Do you think there is value in setting a learning goal for a theatre session? Yes/No
Why or why not? Comment
How do you decide what goal to set for each clinical session?
What do you think are the barriers to setting an achievable learning goal?
Author contribution(s)
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Data sharing statement
Following direct correspondence with the research team in the interests of further collaboration study protocol, statistical analysis plan and informed consent statements as well as deidentified participant data may be shared in accordance with HREC requirements.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
