Abstract
Different modelling software are developed for building performance simulation. However, modelling uncertainty and variability can impact the reliability of modelling results and contribute to the performance gap. To bridge the gap and improve prediction accuracy at the design stage, the CIBSE TM54 protocol was introduced to promote accurate performance modelling. This study investigated its application in practice by UK practitioners through a modelling task and questionnaire based on a simplified school building. Participants were asked to simulate energy use in IES VE or DesignBuilder using either the template HVAC or detailed component-level HVAC modelling approach outlined by CIBSE TM54. The study revealed significant variations in annual total energy consumption predicted by 20 participants, ranging from 24.2 kWh/m2 to 186.3 kWh/m2. Participants using the detailed HVAC modelling self-reported that the average time taken was 1.5 times longer than participants using template HVAC modelling, however this increased modelling effort did not improve the consistency of the results. It was found that even explicit building information was interpreted differently by different modellers. This points to a gap between modellers and design or engineering teams, and highlights the need for better communication, improved modelling-relevant information from suppliers, and more effective software and data frameworks.
Practical Application
This study demonstrates the performance of different modelling approaches and software in practice, and identifies modelling challenges faced by practitioners under the current modelling framework. The findings provide evidence-based insights to support improvements in the provision of modelling information, modelling processes, and validation steps, with the aim of improving the reliability of simulation results.
Introduction
The Energy Performance of Buildings Directive 1 came into force in the UK in 2003 and introduced Energy Performance Certificates (EPCs) in England and Wales in 2007, which promoted the development of a range of third-party tools for assessing building design compliance and generating EPCs. However, research has shown that compliance models based on standard operating conditions which do not take into account unregulated energy use and real operation settings are not suitable for predicting actual total energy consumption. 2 This actual consumption reflects the energy used during the building’s operational stage, also known as operational energy performance. Unregulated energy refers to energy use from end uses beyond the fixed building services, such as small power equipment, lifts and external lighting. 3 The difference between actual consumption and predicted energy use at the design stage is known as the performance gap. 4 To avoid the misuse of compliance modelling as a design tool or as a benchmark to quantify the performance gap, the CIBSE TM54 5 protocol was developed to facilitate the application of performance modelling to predict operational energy use at the design stage. Two levels of dynamic modelling approaches are now available for this modelling framework, namely (1) template HVAC modelling: users select predefined template HVAC systems in accordance with the UK National Calculation Methodology (NCM) 6 and customise key parameters in these systems; and (2) detailed component-level HVAC modelling: users create project-specific HVAC system schematics to reflect the design as accurately as possible. Each component can individually be modelled using manufacturer data, including performance under part-load conditions, and the simulation incorporates the intended control strategies.
Another performance modelling development in the UK is the emulation of the Australia’s NABERS commitment agreement protocol, for which the industry undertook the Design-for-Performance (DfP) initiative project. 7 One of the most notable feature of NABERS is the high priority given to the HVAC systems in the modelling and the design review process. 8 Meanwhile, the adoption of rating metrics can increase the visibility of expected operational performance and transparency of responsibilities. As the NABERS Commitment Agreement extends beyond modelling alone, models in this initiative project that complied with it may be associated with improved reliability compared to the TM54 template-based models. 9 A potential reason might be that older version of CIBSE TM54 did not cover detailed component-level HVAC modelling. However, given that most UK construction contracts do not specify operational performance targets and that complex modelling would be more time-consuming and costly, the use of component-level modelling remains a niche part of the building simulation industry due to lack of market demand. Currently, there is a lack of research comparing the differences in results between template HVAC modelling and detailed component-level HVAC modelling.
Furthermore, despite the recent improvements in practical frameworks, approaches and software for performance modelling, previous case studies still indicated that the operational energy use generally did not match design expectations.10–13 As building energy modelling increasingly informs regulatory compliance, decarbonisation strategies, and performance verification frameworks in the UK, discrepancies between predicted and actual performance have direct implications for design decision-making and policy implementation. In this context, understanding the sources of variability in simulation outcomes becomes critical for interpreting the reliability of modelling results in practice. Therefore, the aim of this study was to investigate the effect of modeller variability, inter-model variability and modelling approach variability on the results. There were three key objectives to achieve this, firstly, to compare the differences in predicting energy use in a single case study building with different modelling software and approaches. Next, the modelling inputs for key parameters and the reliance on software defaults were examined to investigate potential causes of uncertainty and variability. Finally, a questionnaire was used to collect information on the modelling challenges encountered by users and their decision-making rationale during the modelling process.
This study investigates how modeller variability manifests when applying two distinct HVAC modelling approaches: template HVAC modelling and detailed component-level HVAC modelling. In contrast to previous research, which has primarily focused on modeller and inter-model variability, this work constitutes an initial exploration of modelling approach variability within this context. A key methodological innovation lies in the review of actual models submitted by participants, rather than relying solely on self-reported simulation results. This allows for direct comparison of input data and helps mitigate errors in result interpretation and extraction. Additionally, participants’ records of modelling time are analysed to explore the trade-off between simulation accuracy and modelling effort. Feedback collected through user questionnaires further provides practical insights into opportunities for improving and optimising modelling workflows. After this introduction, section 2 reviews the existing literature exploring modelling variability. The research methodology is outlined in Section 3. Section 4 presents the main findings from three aspects: simulation results, modelling inputs and questionnaire analysis. Section 5 provides the relevant discussion of the results, limitations and future research. Section 6 presents the conclusions.
Literature review
Uncertainty in building performance simulation comes from multiple sources, including specification, modelling, numerical, scenario, and heuristic uncertainty, as classified by De Wit 14 and CIBSE TM 613. This paper focuses on uncertainty caused by variability in the modelling process, specifically considering modeller variability, inter-model variability, and modelling approach variability. The following sections review existing studies from these three perspectives.
Modeller variability
The development of building performance modelling relies on the modeller’s comprehension and decision-making of building information. This design space of input parameters is affected by the personal preferences and biases of a modeller. Generally, the reasons for heuristic uncertainty include the following types: lack of information, differences in parameters between program input and actual design, and differences in geometry creation or import between software that lead to geometric clashes and require manual adjustments, as well as accidental modeller errors.3,15 Previous studies compared the results from different users simulating the same building using the same modelling software and approach. For example, the predicted total energy consumption of an office building by 25 modellers ranged from −46% to +106% compared to the average. 16 Bradley, Kummert and McDowell 17 each modelled two test cases from ASHRAE Standard 140 to compare the results of heating and cooling loads. Despite using a simple shoebox model, the differences in cooling loads can reach 20%. Berkeley, Haves and Kolderup 15 recruited 12 modelers to simulate the energy use of a school administration building within 3 h. The predicted electricity and natural gas consumption were then compared to the best practice model created by the authors, revealing variability ranges of −11% to +104% and −61% to +1535%, respectively. Other studies did not set time limits or require participants to record the time taken. All studies identified that human-introduced uncertainty can have a non-negligible impact on the predicted outputs. For more complex buildings, modellers may introduce greater uncertainty, which could be a potential reason why simulation results for a residential building are typically less variable than for a non-domestic building. 18
Inter-model variability
Each type of software adopts different algorithmic programs and assumptions, which leads to differences in the results they produce. Most of the current studies examined and compared different tools from two aspects: software capabilities or simulation results. In terms of software capabilities, a portion of the research summarised software performance through literature reviews or constructing simple models, which were general evaluations of each software capability but did not collect the individual user’s experience of using the software.19–23 Another body of work used questionnaires or semi-structured interviews to enable users to rank the evaluation criteria and tools themselves, which assessed the suitability of the software for different user groups based on their selection preferences and provided some insights into software choices.24–29 For example, in a survey of 445 architects and 453 engineers, Attia et al.25,26 found that architects preferred software that included intelligent design knowledge bases, while engineers prioritised the accuracy of the tools and the ability to simulate complex components. In addition, IES VE was chosen as the most architect-friendly software, while engineers ranked DesignBuilder and EnergyPlus first. For investigating the impact of internal procedure variations on simulation results, previous studies compared differences in results by having individual users model the same building using different modelling software. Schwartz and Raslan 30 simulated the total energy consumption of a student residential hall using the TAS, EnergyPlus and IES VE software. The IES VE software output was 35% higher than the Tas software and 5% higher than EnergyPlus software. Reeves, Olbina and Issa 22 used IES VE, Ecotect, and Green Building Studio software to predict the energy use of two educational buildings, with the lowest simulation results being 56% (from Ecotect) and 58% (from Green Building Studio) of the maximum values (both generated by IES VE). Three other studies tested more than five software tools, and the results all showed poor consistency.31–33
Modelling approach variability
Bottom-up building energy modelling approaches based on first engineering and building physics principles are divided into two categories: quasi-steady state and dynamic simulation. 34 Jones, Fuertes and De Wilde 35 carried out compliance modelling for a residential building using both steady-state method and dynamic simulation, and the discrepancy between the simulated and measured data from both approaches was significant. Hence, dynamic simulations may not be able to mitigate the prediction limitations of compliance modelling with standardised parameter inputs and without accounting for unregulated loads. Uncalibrated dynamic performance modelling is also insufficient to improve the reliability of simulation results. Ahmad and Culp 36 reported that investigating real operational settings and improving the level of modelling detail, paradoxically, increased the performance gap due to the introduction of additional uncertainty. Wang et al. 37 found that updating models based on design conditions with real operating conditions would make the simulation results more reflective of the actual building energy performance, but there were still performance gaps ranging from 3.0% to 53.5%. This study also categorised the performance gap into HVAC-related and non-HVAC-related. The comparison found that the maximum gap caused by the HVAC system reached 114.7%, while the maximum difference between predicted and measured data for the non-HVAC system was only −4.5%. Furthermore, two studies compared the use of different performance modelling approaches in retrofit analysis. Murray, Rocher and O’Sullivan 38 showed that energy savings predicted using steady-state modelling were closer to measured data in comparison with dynamic modelling. But Jradi 13 stated that using calibrated dynamic simulation models to analyse building retrofitting strategies can effectively reduce the performance gap. One notable difference between the dynamic simulation approaches in the two studies was that the former used template HVAC modelling, whereas the latter used detailed component-level HVAC modelling. Two other studies that used calibrated detailed HVAC modelling both demonstrated more reliable results than either compliance or steady-state modelling.39,40 However, it should be noted that measured energy performance results are not always available to enable building performance model calibration. To conclude, previous research mainly focused on discussing the differences between compliance and performance modelling, and the feasibility of using operational data to calibrate performance models. The potential trade-offs between complexity and accuracy of different dynamic performance modelling were not covered in these studies.
Methodology
Building description
In this study, a 2970 m2 educational building in England was simplified to develop the modelling task. The original building is a sixth form college, which in the UK is a standalone institution providing academic and vocational qualifications for students aged 16-18. As is common practice in modelling and benchmarking, sixth form colleges are often treated as secondary schools due to their similar educational function and occupancy characteristics. Two simplifications were applied in designing the modelling task, merging similar adjacent rooms with the same operating conditions into a single zone, and taking only the main HVAC system into account. These simplifications maintain the representativeness of the model and better reflect the layout and usage patterns of educational buildings. Figure 1 illustrates the building model developed in IES VE software. The model has 31 zones, and the main activity types are teaching, workshops and offices. Table 1 presents an overview of the building information, providing an initial insight into the building specification and services strategy. For the building services systems, gas-fired condensing boilers are used for the main space heating. The ICT-enhanced classrooms and IT rooms are supplied with the variable refrigerant flow (VRF) system for heating and cooling. Two server rooms are equipped with the direct expansion (DX) system for cooling. The automatic vents in three atrium spaces provide natural ventilation by responding to the temperature and CO2 concentrations. The other spaces are mechanically ventilated through an air handling unit (AHU) with heat recovery and there is no dedicated cooling system. Domestic hot water (DHW) is heated by a separate gas water heater. Building model developed in IES VE. The overview of the building information.
Participant recruitment
Participants were recruited through several channels, such as the CIBSE Building Simulation Group, the UCL alumni network and DesignBuilder monthly newsletter for UK subscribers. They were expected to be familiar with the UK building regulations and to have work experience relevant to modelling. Therefore, 20 practitioners with at least 1 year of experience in building performance modelling were recruited and all had a relevant educational background or work experience in the UK. Each modelling approach was tasked by five participants under each software. This study was approved by the University College London, Bartlett School of Environment Energy and Resources (BSEER) Research Ethics Committee (No. 20230419_IEDE_PGR_ETH). It was also registered under reference No Z6364106/2022/12/19 social research in line with UCL’s Data Protection Policy. Informed consent was obtained from all participants (20 in total) involved. All practitioners completed the modelling task and the associated questionnaire. Figure 2 shows the professional composition of the participants and their years of work experience. There were 7 building physicists, 7 sustainability consultants, 2 researchers and 4 individuals who specifically identified their roles as building performance modellers. The questionnaire included options for architects and building services engineers, but no participants identified themselves in these professional roles, which could be indicative of the ‘distance’ between the design practice and building performance modelling in the industry as the complexity of the tasks and projects increasingly grow. Regarding work experience, only 1 participant had more than 10 years 12 participants had 4-10 years of work experience, while the other 7 were in the early stages of their careers (1-3 years). Professional roles involved in the study and years of work experience.
All participants had been involved in projects that predicted operational energy use in non-domestic buildings at the design stage. Nine participants indicated that they use the template HVAC modelling approach for their daily work. Five participants need to use a detailed component-level HVAC modelling approach on a regular basis. Another five require a dynamic modelling approach but not necessarily based on CIBSE TM54. Only one participant uses the quasi-steady state modelling approach in work. Few participants explained in the questionnaire that they do not routinely use the CIBSE TM54 modelling approach because their daily work focuses mainly on compliance modelling (three participants) or overheating assessments (one participant). Additionally, four participants mentioned that they would only use detailed component-level HVAC modelling for projects that require NABERS UK Design for Performance certification.
Design of the modelling task
Participants were asked to choose one of the two modelling approaches (template HVAC modelling or detailed component-level HVAC modelling) proposed by CIBSE TM54, 5 using either IES VE or DesignBuilder software, which are among the most widely used building performance modelling platforms in the UK, to predict the building’s operational energy consumption. Although assigning both approaches to the same modeller would, in principle, control modeller variability, this method was not adopted due to two key concerns: the practical difficulties of recruiting participants with sufficient expertise and time, and the risk of bias. It is likely that modellers would transfer knowledge from the first attempt to the second, or attempt to reconcile discrepancies, thus undermining the independence of the two modelling results.
Template HVAC modelling corresponds to the ‘Apache system’ in the IES VE software and the ‘Simple HVAC method’ in the DesignBuilder software. Detailed component-level HVAC modelling corresponds to the ‘Apache HVAC system’ in the IES VE software and the ‘Detailed HVAC method’ in the DesignBuilder software. In this paper, the terms ‘Simplified’ and ‘Detailed’ are used to refer to the two modelling approaches in the figures and tables presenting the results, respectively. To avoid the introduction of modelling pressure exacerbating modellers’ selection preferences, this study did not impose a time limit, but modellers were required to record the time spent on modelling.
All participants were provided with the same modelling package, which comprised an overview of the modelling task (including detailed thermal performance of the building envelope, building services system characteristics, and occupant information), a gbXML file for the geometry of the building, a weather file, and technical data sheets for key HVAC system equipment, such as boilers, hot water heater, AHU, VRF and DX systems (manufacturer technical data sheets). Additionally, a questionnaire was developed based on this modelling task to obtain feedback from the practitioners’ perspectives for a qualitative analysis of the existing modelling challenges and drivers of variation in the results. It was also used to collect basic information about the participants and the time spent on their modelling.
The authors did not define a reference building model for this study. Establishing a single reference building would require selecting a specific modelling software and modelling approach, which could introduce methodological bias given that this research compares two modelling software and two HVAC modelling approaches. Instead, benchmark values for UK secondary schools were used as a reference point to contextualise the simulation results. 41 For natural gas consumption, the benchmarks defined on the CIBSE Energy Benchmarking platform are 66 kWh/m2 for ‘best practice’ (10th percentile of secondary schools), 86 kWh/m2 for ‘good practice’ (25th percentile), and 113 kWh/m2 for ‘typical practice’ (50th percentile), while for electricity consumption the benchmarks are 34 kWh/m2 for ‘best practice’, 42 kWh/m2 for ‘good practice’, and 51 kWh/m2 for ‘typical practice’. 41 Based on the specification defined for the case study building, which represents a typical post-2010 secondary school, the good practice benchmarks are deemed to provide a plausible reference point for this study. Measured annual energy consumption data were available for the original case study building that was used to define the modelling task, but these were not used as the primary reference point because the modelling task was intentionally simplified and was not designed as an as-built model calibration exercise. Instead, the measured consumption values are discussed as an additional contextual check when interpreting the range of simulation results.
Results
Fuel type breakdown of projections
The distribution of simulated energy consumption results for the 20 participants and the average values are presented in Figure 3. In terms of natural gas consumption, the average simulated energy usage was 105.6 MWh (35.6 kWh/m2), with a standard deviation of 69.6 MWh. This value is lower than the ‘best practice’ benchmark. In terms of electricity, the average simulation result was 168.2 MWh (56.6 kWh/m2), with a standard deviation of 80.4 MWh. This indicates that the building’s average projected electricity use was slightly higher than the ‘typical practice’ benchmark. Across all 20 simulation results, the mean total energy consumption was 273.8 MWh (92.3 kWh/m2), with a standard deviation of 121.3 MWh. Distribution of simulated energy consumption results for 20 participants.
Simulated energy consumption and deviations from the average for 20 participants (mean natural gas consumption = 105.6 MWh; mean electricity consumption = 168.2 MWh; mean total energy consumption = 273.8 MWh).
Figure 4 shows the boxplot comparison of energy consumption under different modelling software and approaches. In terms of the different software, the average natural gas consumption projected by participants using IES VE was approximately 2.3 times higher than that projected by those using DesignBuilder. The average electricity use simulated by both software was almost the same. In terms of different approaches, the simulated averages for natural gas consumption were shown to be nearly identical. Electricity consumption projections were, on average, notably higher (by around 58.5%) with the detailed component-level HVAC modelling approach compared to the simplified one. Boxplot comparison of energy consumption under different modelling software and approaches.
End-use breakdown of projections
Natural gas was consumed for space heating and domestic hot water (DHW), and individual results and comparisons with averages for space heating and DHW energy use projections are shown in Figure 5. The results are grouped by modelling software and approach, with each group containing five participants. The order of participants follows the numbering in Table 2 to maintain consistency across figures. The black dotted lines and numbers in the figures represent the average of all simulated results. To enable effective comparison, the y-axis range in the following two figures is set to the same scale, allowing for a clear visualization of the proportion of these two end-uses in the total natural gas consumption. The simulated mean value of natural gas consumption for space heating was 88.0 MWh, with a standard deviation of 64.8 MWh. The results for the 10 participants using the DesignBuilder software were all below this mean, with an average of 46.2 MWh. In contrast, the simulated mean for the IES VE users was 129.7 MWh, with only three participants simulating results below the overall average. For DHW, the average of all modelling results was 18.3 MWh with a standard deviation of 16.8 MWh, in which one participant did not have a DHW system set up. The investigation of the groups, based on different modelling software and approaches, revealed that the mean values of each group were within ±1.2 MWh of the overall mean. Predicted natural gas end-uses: space heating and domestic hot water.
Electricity consumption included space heating and cooling provided by the VRF system and the DX system, auxiliary energy use, lighting and equipment operation. Figure 6 illustrates individual variations in simulation results for each category. Similarly, the y-axis range in these four figures is also standardised. Regarding electricity use for space heating and cooling, the mean value of the simulation outputs for the 20 participants was 8.6 MWh, with a standard deviation of 11.1 MWh. Generally, the mean value of the simulation results from the DesignBuilder software (11.2 MWh) was higher than that from the IES VE software (6.0 MWh). By modelling approach, the mean value of simulation results using the detailed component-level HVAC modelling approach (12.1 MWh) was higher than that using the template HVAC modelling approach (5.1 MWh). The predicted consumption of auxiliary energy had an average value of 39.4 MWh and a standard deviation of 36.3 MWh. The mean results from different modelling approaches showed a significant difference, with the predicted mean under the template HVAC modelling being 22.4 MWh and the detailed component-level HVAC modelling being 56.3 MWh. The simulated electricity usage for lighting resulted in a mean value of 39.1 MWh with a standard deviation of 22.9 MWh. For equipment, the simulated results had a mean value of 80.6 MWh, with a standard deviation of 31.7 MWh. Both types of end-use energy predictions showed similar average values across the different modelling software. However, when classified according to the different modelling approaches, the simulation results averages for the template HVAC modelling (33.8 MWh for lighting and 67.7 MWh for equipment) were both lower than the detailed component-level HVAC modelling (44.4 MWh for lighting and 93.6 MWh for equipment). Predicted electricity end-uses: space heating & cooling, auxiliary, lighting and equipment.
Discrepancies in modelling inputs
Review of input data: summary of unexpected system configurations in modelling.
Summary of numerical input ranges and usage of software default values.
Average seasonal efficiency of HVAC systems using template modelling approaches.
Participants’ general experience in modelling and feedback
Based on the respondents’ general work experience, regarding the accuracy of the modelling approaches applied to non-domestic buildings, the survey results showed that 14 participants considered the detailed component-level HVAC modelling approach to be the most accurate approach. They mentioned that this approach allows for more detailed modelling information to be entered to simulate the actual operation and control of the HVAC system. However, two participants emphasised that it should not be overly detailed (Response 1) or ignore non-HVAC related parameter inputs (Response 2). The accuracy of the template HVAC modelling approach was chosen to be higher by four participants. One respondent stated that the NCM-based HVAC templates yielded accurate results despite the limited HVAC system setup (Response 3). Another respondent commented that detailed modelling could not offset the inherent uncertainty that exists at the design stage (Response 4). Besides, no one considered quasi-steady state modelling to be superior to the other two dynamic modelling approaches. Response 1: “One should be careful not to overdo the detail. A model should be as simple as it can be to answer the question that it’s used to help answer.” Response 2: “This includes accurately modelling controls, making correct assumptions on people's behaviour when manually operating systems or windows, predicting small power operation etc. If these are not correctly estimated or modelled, then the additional accuracy gained from the component level approach becomes irrelevant.” Response 3: “Some cases show that not all HVAC settings are covered by template HVAC modelling. But, in my opinion, following the steps available when assigning template HVAC leads to more accurate results.” Response 4: “Projecting energy use at the design stage has inherent uncertainty. Therefore, making super-detailed models may not add value to monthly energy use. Probabilistic reporting of results with more straightforward modelling is more efficient.”
In terms of the stage at which detailed component-level HVAC modelling can be introduced, two participants suggested that it should be introduced at the concept design stage, where it would be useful in the design and selection of the HVAC system. 70% of respondents, however, thought that it would be most appropriate to apply it at the detailed design stage. As detailed design information is lacking during the concept design stage, it is worthwhile to move forward with detailed modelling after the overall design strategy is determined (Response 5). One participant recommended applying it at the construction design stage but did not provide reasons. A possible rationale is that all design and construction details would be known at that point. The other three participants argued that modelling should be undertaken at the building commissioning stage. Since real data on system performance is available at this stage, it can be used to calibrate the model and conduct scenario testing of system control. No one opted to introduce this modelling approach while the building was in use. Response 5: “When used at concept design then many component-level variables are set to default values, which contradicts the purpose of using this more advanced approach.”
The respondents’ feedback on the ease of accessing the performance curves for each component in HVAC systems, required for detailed modelling, is shown in Figure 7. Only one respondent reported that accessing this information was relatively easy, but using performance curves in-built in the software and updating system efficiencies based on equipment specifications was the general modelling workflow (Response 6). Participants who chose ‘neutral’ or ‘difficult’ provided three main reasons: (1) specific equipment was sometimes not yet specified at the time of modelling; (2) the design team or building services engineers sometimes did not provide this information; and (3) the manufacturers sometimes did not provide this information. In addition, there were three participants who may not have been involved in the detailed component-level modelling and therefore indicated that they did not need these performance curves. Response 6: “Normally getting the curves is not that difficult. But using it can be quite tedious. I would normally end up simplifying by using a typical curve from DesignBuilder and updating to EER and COPs.” Participants’ perceptions on the availability of HVAC performance curves for detailed component-level HVAC modelling.
Participants’ task-specific experience in modelling
15 participants stated that their organisation commonly used the template HVAC modelling approach for simulation of the building type used in this modelling task. The spreadsheet-based quasi-steady state modelling approach was the least frequently used, selected by only one participant. The other four participants selected the detailed component-level HVAC modelling approach. The participants’ ratings of the perceived accuracy of the three modelling approaches used for this modelling task are shown in Figure 8. Thirteen participants perceived a progressive improvement in accuracy across the three modelling approaches, from quasi-steady state modelling to template HVAC modelling and then to detailed component-level HVAC modelling. One participant perceived quasi-steady state modelling to be superior to template HVAC modelling, and another perceived template HVAC modelling to be superior to detailed component-level HVAC modelling. Two participants considered the accuracy of quasi-steady state modelling and template HVAC modelling to be comparable, while another two participants viewed template HVAC modelling and detailed component-level HVAC modelling to be equally accurate. One other participant commented that there was no difference in accuracy between the three modelling approaches. In summary, 80% of participants perceived the detailed component-level HVAC modelling approach to be the most accurate. Participants’ ratings on perceived accuracy of the three modelling approaches.
Participants self-reported the time taken to complete the modelling task. The question was set with five options including less than 2 hours, 2-5 h, 5-10 h, 10-15 h and more than 15 h. Six participants used 2-5 h to complete the task. Almost half of the participants (nine) spent 5-10 h modelling. Two used 10-15 h to model. The other three modellers reported more than 15 h. No one was able to complete the energy modelling of the building in less than 2 hours. Figure 9 displays the distribution of time taken by each participant to complete the modelling task, categorised by modelling software and approach. The black dotted line and number in the figure represent the average time required. The median value was used as a proxy for the time range, thus the five options are replaced with 2 h, 3.5 h, 7.5 h, 12.5 h and 15 h in the figure. It was observed that three modellers who reported spending more than 15 h on modelling used the same modelling approach and software, which was the Apache HVAC system in the IES VE software. The average modelling time for this group of participants amounted to 11.2 h. In comparison, the average time required to complete the task using the template HVAC modelling approach in the IES VE software was 6.9 h. For the DesignBuilder software, the five participants using the template HVAC modelling approach took an average of 5.9 h and 7.7 h for detailed component-level HVAC modelling. Self-reported time required for the task.
Figure 10 illustrates participants’ assessments of the helpfulness of various information sources in the modelling task. Participants rated modelling assumptions driven by work experience as the most helpful source of information, with an average score of 4.35 out of 5. Technical data sheets from HVAC system suppliers also received a high rating of 4.15, while relevant regulations and standards were the least helpful with a score of 3.60. Participants’ ratings on the helpfulness of different information sources for the modelling task.
Figure 11 further demonstrates the participants’ assessment of the usability of the technical data sheet information provided in the modelling task. Seven participants expressed that the information was somewhat difficult to extract. Others chose neutral and somewhat easy. Specifically, eight respondents thought that the provided technical data sheet lacked key information, with missing specific fan power (SFP) and seasonal efficiencies of the system each mentioned four times (Response 7). Two respondents stated that additional information is usually obtained through communication with building services engineers (Response 8). Furthermore, ten respondents reported that the information in the data sheets did not match the input parameters required by the software (Response 9). Another respondent stated that the difficulty was in sifting through the large amount of information to find the parameters required for modelling. Response 7: “A lot of manufacturers don't tend to list out seasonal efficiency figures on their data sheets. Getting the information from manufacturers which 'translates' into a format used in the energy modelling tools can be difficult.” Response 8: “The data sheet is a generic set of equipment in different capacity. But in reality, we need the MEP engineer to decide which model they are using.” Response 9: “I found it a struggle to know exactly the data required or where they should be placed due to the fluidity of the resources wording compared to software.” Participants’ ratings of the ease of extracting information from technical data sheets.
Discussion
Comparison of simulation results
The findings highlighted marked inconsistencies in the annual natural gas and electricity consumption projected by the 20 participants, as shown in Figure 12. When comparing operational benchmarks for UK secondary schools, the average simulated natural gas consumption was below the ‘best practice’ benchmark, whilst electricity use was slightly above the ‘typical practice’ level. This pattern may partly reflect modelling assumptions commonly adopted in energy simulations, such as standardised occupancy and heating operation schedules, which may result in predicted heating demands being lower than the more variable operating conditions experienced in actual school buildings. Electricity consumption, on the other hand, is strongly influenced by equipment and internal loads, which may remain relatively high in simulations due to the assumed power densities. End-use breakdown of simulated energy consumption.
An additional contextual check was provided by the post-occupancy annual energy data for the original case study building used to define the modelling task. The actual natural gas and electricity consumption were 206.1 MWh and 164.2 MWh, respectively. These measured values fell within the ranges predicted by the 20 participants for each fuel type, and the measured total energy consumption of 370.3 MWh also fell within the simulated total energy consumption. This comparison does not validate any individual model, but it suggests that the range of dynamic simulation outputs captured the broad scale of the building’s operational energy performance. Therefore, despite the large differences between individual predictions, dynamic simulation and performance modelling can still offer useful information for projecting building performance, provided that uncertainty and variation across individual model outcomes are recognised.
Modelling results varied to some extent depending on the simulation software and approaches used. For natural gas consumption, the simulations using the DesignBuilder software resulted in lower predictions. For electricity consumption, the simulations performed under the simplified HVAC modelling approach had lower energy use. Previous studies comparing simulation results from different software concluded that the IES consistently predicts the highest energy use, with two of these studies involving both DesignBuilder and IES software,22,30,32,43 which is in line with the trend found in this research. However, these four studies were based on testing of different software by a single modeller and the present study is still not statistically generalisable despite the expanded sample size.
Variations in simulated energy consumption were also observed across different energy end uses. By comparing the standard deviation of each group, results for natural gas required for space heating were predicted to have the most dispersion. The simulation results were found to be highly dependent on the software used. However, for domestic hot water, the mean value for each group did not fluctuate by more than 6% from the mean value of the total simulation results, whether categorised according to different software or modelling approaches. This demonstrated that space heating accounted for the largest share of natural gas consumption, indicating that it was the main driver of natural gas demand in the modelling. For electricity use predictions, the average simulation results for HVAC-related end-uses under the detailed component-level HVAC modelling approach were higher than for the template HVAC modelling, potentially due to its more comprehensive and granular requirements for system and auxiliary energy input data. However, this was not the reason for the same trend in the simulation results for lighting and equipment operation, as the data inputs for both types of end-use simulation were essentially identical in both modelling approaches, determined by the activity type templates assigned to each space. It was evident that investigating the simulation input parameters to further understand the underlying causes of the differences in the results was necessary.
Comparison of input parameters
To understand the underlying causes of the differences in simulation results, this paper further examined the input choices across the 20 models. The input variations identified included both non-HVAC and HVAC-related issues, ranging from the implementation of material properties and activity templates to auxiliary energy settings and HVAC system configurations. Although the overall analysis places particular emphasis on HVAC system representation, as this is the main distinction between the two dynamic modelling approaches described in CIBSE TM54, the comparison also highlighted the role of building-side input assumptions in modeller variability.
In this modelling task, the scope for variation in the building-side representation was intentionally constrained by providing all participants with a gbXML file containing the building geometry. Participants were therefore not required to recreate the building form or spatial layout independently from drawings. This reduced the potential for variability arising from different interpretations of geometry-related aspects, such as the overall façade arrangement and glazing layout. The remaining building-side differences were therefore mainly related to the implementation of the provided fabric performance data and the allocation of activity type templates. The relatively limited number of fabric-related issues suggests that explicit material and construction information helped constrain this source of variability, although differences could still arise when technical data were translated into software-specific inputs.
A more prominent source of non-HVAC input variation was the selection of activity type templates, which was closely linked to modellers’ familiarity with this type of building. Activity type templates assign several modelling parameters simultaneously, including occupancy density, lighting illuminance, equipment load, and operational schedules based on the intended function of each space. Hence, differences in template selection can affect not only individual input values, but also the internal gains and operational assumptions assigned to the corresponding thermal zones. For example, for the server room with 24-h low-medium internal gains from equipment and transient occupancy, the corresponding NCM template specifies an equipment power density of 50 W/m2. However, the office or computer lab templates, which were misused for this space, have equipment power densities of 12 W/m2 and 30 W/m2, respectively, and associated with a daytime-only operation schedule. The computer lab template with an equipment power density of 30 W/m2 should be used for the ICT-enhanced classroom, but there was a misallocation in the modelling to the office template with an equipment power density of 12 W/m2 or the general teaching area template of 4.7 W/m2. In this context, even under structured modelling instructions, assigning space functions to software templates involved modeller judgement. These differences reflect modeller-level implementation diversity and are not merely numerical deviations from prescribed values.
For HVAC systems, despite providing technical data sheets for each equipment, the range of variability in modelling inputs remained significant. In terms of efficiency input values within the models, the variation range under the detailed component-level HVAC modelling approach was smaller than that of the template HVAC modelling. However, the detailed component-level HVAC modelling approach showed a higher dependency on software defaults compared to the template HVAC modelling approach. This could be attributed to its extensive parameter requirements, which make it more complicated to properly map the available information to the corresponding modelling inputs, thus tending to maintain the default settings. Besides, in template HVAC modelling, the DesignBuilder software requires the input of seasonal efficiencies that represent the entire system, while the IES software can input the seasonal efficiencies of the equipment and system separately. Typically, IES users either (1) input the equipment’s seasonal efficiency and apply the default delivery efficiency, or (2) input both equipment and system seasonal efficiency, with the software automatically calculating and displaying delivery efficiency based on the input values. Data from equipment technical data sheets are used as the overall system efficiency in DesignBuilder, whereas in IES, considering delivery losses results in a value lower than the equipment’s seasonal efficiency as the entire system efficiency. In terms of system configurations, definitions of non-dominant HVAC systems (e.g., DX cooling systems for server rooms) were more likely to be missed in detailed component-level HVAC modelling. In contrast, the application of the template HVAC modelling approach increased the problem of inconsistent, incompatible or duplicate system setups, but there were relatively few missed settings, mainly in auxiliary energy (pumps and fans). This is because, in template HVAC modelling, the settings for pumps and fans are defined by default values associated with the selected HVAC templates. Depending on the software settings, these values may need to be manually entered or adjusted, and can therefore be more easily overlooked. Detailed component-level models, however, account for the required pumps and fans based on the configured HVAC system. In addition, the modelling package provided to participants contained technical specifications for key HVAC equipment, which allowed parameters such as fan characteristics and certain control strategies to be defined when using the detailed modelling approach. For example, system control features such as demand-controlled ventilation can be explicitly represented in detailed HVAC modelling but are not captured within the template HVAC approach. Overall, the dispersion captured above also reflects diversity in interpretation, parameter mapping, and software-specific implementation processes under controlled guidance, which is consistent with typical real-world modelling practice.
Reflections on modelling based on participants’ feedback
This study did not set a time limit for completing the task. The average time taken by the 20 practitioners to complete the modelling was 7.9 h. Under the same modelling approach, the average time required by participants using DesignBuilder was lower overall than that of IES VE users. With the same modelling software, detailed component-level HVAC modelling required 62% more time than template HVAC modelling in IES VE and 31% more time in DesignBuilder. However, the time required for modelling did not correspond to the magnitude of result variations. This demonstrated that increasing the modelling effort and level of detail did not improve the consistency of simulation results between modellers. Adding additional inputs may not only enhance the alignment between the model and the actual system operation, it may also introduce additional uncertainties, making it challenging to assess its impact on result reliability. Another factor may be the cognitive burden associated with complex modelling interfaces, which can affect the decision-making process. As the level of model detail increases, the corresponding rise in interface complexity and modelling duration may lead to cognitive fatigue, which can increase reliance on default values and reduce model fine-tuning efforts.
This interpretation also ties in with previous research on modelling literacy. Imam et al. 44 compared practitioners’ judgements regarding the relative importance of common input parameters with the rankings derived from sensitivity analyses using a validated thermal model. Their study highlighted substantial variation in how modellers prioritise input assumptions, and demonstrated that professional experience or qualifications do not necessarily lead to consistent judgements. Although this study did not assess individual competence or define modelling literacy, the existing literature provides useful context for explaining modeller variability. It suggests that differences between models may arise not only from software choices or HVAC modelling approaches, but also from the professional judgement required to prioritise modelling inputs under practical modelling conditions.
This broader issue of modeller judgement is also reflected in participants’ perceptions of detailed modelling. The questionnaire responses highlight a potential gap between the perceived benefits of detailed modelling and the practical conditions required to implement it reliably. Most participants perceived the accuracy of the detailed component-level modelling approach to be superior to others. Participants’ work experience was largely focused on template HVAC modelling, which may have influenced their evaluation of the detailed modelling approach, such as having higher expectations for the results accuracy of the approach given the complexity of the inputs. Detailed modelling is primarily used in practice associated with the NABERS UK Design for Performance accreditation. Notably, none of the 20 participants identified themselves as architects or building services engineers, and several mentioned that modelling required requesting information from the design team or other engineers, which may indicate that building performance modelling is often outsourced rather than done by the original designers. Gaps in information transfer due to lack of communication between modellers and designers can be a major factor affecting the reliability of modelling. Additionally, participants provided feedback on the difficulty of obtaining usable equipment performance curves in their regular work (as shown in Figure 7). The technical data sheets from manufacturers provided for this task also did not contain performance curves suitable for detailed modelling. This discrepancy between the data granularity of the information available and the level of detail required for performance modelling may further affect the accuracy of the simulation results.
Recommendations and opportunities to improve modelling reliability
The findings of this study point to a series of recommendations directed towards six key stakeholder groups: modelling practitioners, clients and project teams, system suppliers, software developers, policymakers, and researchers.
Modelling practitioners
At the early stages of the project, establish close collaboration between the design and modelling teams and effective interfaces with manufacturers to ensure access to the information required for modelling. Also, provide training and well-defined performance modelling frameworks for modellers to avoid omissions of critical modelling configurations. During modelling, a structured independent peer-review process, similar to the Independent Design Review in the NABERS Design for Performance (DfP) framework (NABERS UK, 2021), should be embedded early in the modelling process to assess key inputs, assumptions and design documentation. This need to be complemented by benchmarking the simulation outputs against industry-recognized references (such as the CIBSE online benchmarking platform), or by comparing them with building operational performance of similar completed projects in shared databases (such as CarbonBuzz), empirical operational evidence from school energy studies, or internal company databases.45–49
Clients and project teams
Dynamic simulation results should be understood as decision-support evidence rather than fixed predictions of building performance. Given the sensitivity of model outputs to modelling approach, software choice and modeller-level assumptions, a single predicted value may overstate the certainty of the prediction. Thus, the modelling scope, key assumptions and relevant uncertainties need to be clearly communicated. This is particularly crucial in evaluating design alternatives, as subtle differences in predicted energy or carbon performance may still fall within the range of variation associated with modelling decisions. In this context, modelling is more suitable for identifying broad performance trends, testing the robustness of design options and highlighting influential assumptions than for making fine distinctions between different options.
The communication of predicted performance is also important for managing expectations after building handover. Excessively low predictions of operational energy consumption may lead clients and building owners to form overly optimistic expectations of actual operating costs and carbon performance. The emergence of significant discrepancies between post-occupancy energy bills or operational results and the figures communicated during the design phase may give rise to conflicts between the client, the design team and the modelling practitioners. Therefore, simulation results should be presented as scenario-based estimates rather than performance commitments, especially when used to guide financial projections or carbon-related decisions.
The same consideration applies to projects working towards client-defined energy or carbon targets. Performance compliance is more appropriately assessed as a risk-informed design process rather than through a single deterministic output. Predicted performance close to the target may be sensitive to modeller variability and input uncertainty, influencing whether the design appears to comply. Transparent assumption reporting, independent review and sensitivity testing of influential inputs can help determine whether the proposed design is robustly below the target, or whether apparent compliance depends on uncertain modelling choices.
System suppliers
Improve the comprehensiveness and standardisation of performance data in equipment technical data sheets, which can bridge the lack of information granularity that results in modelling inputs that are overly reliant on software defaults or modellers’ empirical assumptions. For instance, some equipment data sheets omit part-load performance curves or control logic details, which limits the ability to accurately characterise the performance of the system under realistic operating conditions. In addition, commonly used modelling parameters such as specific fan power (SFP) are often not explicitly reported, forcing practitioners to perform further calculations based on incomplete manufacturer data. Furthermore, provide responsive technical support to help practitioners interpret and apply product data appropriately. And collaborate with software developers to integrate validated product libraries into modelling platforms or independently build open-access collections of product technical parameters to reduce errors introduced by modellers.
Software developers
Enhance the standardisation of modelling software in defining building information and systems to improve their ability to reflect the operational performance of a building, and increase interoperability between different simulation software. For example, some tools require users to input a seasonal efficiency value representing the entire system, while others allow separate inputs for equipment-level and system-level efficiencies. These differences in input requirements can lead to misinterpretation or misuse of information, ultimately resulting in inconsistent modelling results. Meanwhile, allow software to integrate the latest equipment specification data to assist modellers in making more realistic assumptions. Additionally, developers are expected to monitor industry trends and regulatory changes to ensure that the software is updated promptly to meet emerging needs. Training courses should also be offered to users to bridge the gap between the software’s practical application potential and the users’ skill levels.
Policymakers
The degree of consistency between the inputs required by the software and the information provided by the manufacturer largely determines the modelling accuracy. Neither party is able to tailor the data structure to accommodate the diversity of the other’s system. This impediment is particularly reflected in the development of detailed component-level HVAC modelling in the UK building modelling industry. Whereas template HVAC modelling, the modelling approach adopted under regulatory requirements, has extensive software built-in templates and succinct manufacturers’ data, the detailed component-level HVAC modelling approach remains a niche practice due to its complexity and high cost. Thus, concerted market efforts and policy interventions are needed to establish common standards for data formats and to encourage manufacturers to disclose key information required for modelling. In parallel, completed projects should be incentivised to disclose building design and operational performance to strengthen open-access benchmarking databases, such as the existing statutory schemes like Display Energy Certificates and voluntary industry platforms including CarbonBuzz and K2n.
Moreover, policy makers for specific sectors, such as the educational buildings covered by this modelling task, also play a key role in improving the industry standard. Relevant government departments and professional bodies can set expectations for contractors and designers based on general policy goals, while requiring benchmarking results of modelling predictions against existing energy data sets or performance targets. Pilot projects should also be actively carried out to validate the policies and methodologies in practice and to provide practical basis for policy refinement and improvement.
Researchers
Collaborate with industry to compare and evaluate the performance of physical model-based simulation software in practice and propose feasible improvements based on theoretical foundations. On the other hand, explore the use of data-driven approaches to predict energy consumption. These approaches that extract system performance characteristics from past data can effectively reduce human input errors and decrease the dependence of complex physics-based models on detailed data for simulated buildings. Furthermore, it has considerable potential to improve prediction accuracy by employing artificial intelligence and machine learning models that integrate multiple modelling algorithms and optimise hyperparameter configurations.50,51 However, due to insufficient data and the complexity of occupant behaviour, long-term (annual) energy consumption forecasts are currently inadequately researched.50,52
Limitations and future work
As with other similar studies, a limitation of this study is the lack of comparison with actual energy consumption data.15,18 To make the time required for the modelling task manageable for practitioners, and considering the difficulty of recruitment and budget constraints, this study simplified the modelling of a real school building and participants were asked to apply a typical occupancy pattern for simulation that was in accordance with the NCM in England. Not only did this led to an inability to use actual energy consumption data from the original building for comparison, it also neutralised the impact of occupancy uncertainty on the modelling results. In practice, occupancy variability makes the model less reliable. Therefore, it is worthwhile to explore the impact of extracurricular activities and other non-standard occupancy patterns on energy predictions.
Another limitation lies in the sample size of the modelling exercise. Although 20 practitioners participated in this study, the sample has limited statistical power and cannot be considered statistically representative of the broader population of energy modellers. Recruiting practitioners to participate in controlled modelling exercises presents inherent challenges, and the sample size in this study substantially exceeds that of several comparable investigations. Consequently, the findings should not be interpreted as statistically generalisable to the entire industry. Instead, these results contribute to analytic generalisations by identifying patterns and potential mechanisms through which modeller behaviour, modelling approaches, and software implementation may influence simulation outcomes. Future research could further strengthen these insights through larger collaborative modelling exercises or by expanding participation to encompass a wider range of practitioners.
Furthermore, this study focused primarily on variability in the implementation of HVAC-related modelling information, reflecting the intended comparison between the two CIBSE TM54 dynamic modelling approaches. Although the use of a gbXML file helped standardise the building geometry across participants, broader building-side modelling assumptions were not systematically recorded or isolated. These include the treatment of external shading from surrounding buildings, recessed glazing, thermal bridges and other façade-related details, which may influence simulation results in modelling tasks where practitioners are required to interpret drawings and define these elements independently.
Future work could explore the uncertainties that different users or software might introduce when applying BIM-based modelling, in line with the six components (1. Geometry; 2. Constructions and materials; 3. Spaces type; 4. Thermal zones; 5. Space loads; 6. HVAC system and components) proposed by Maile, Fischer and Bazjanac 53 for the ideal transfer of information between building information modelling (BIM) and building energy modelling (BEM). Collecting structured information on both HVAC and non-HVAC modelling assumptions would allow the contribution of building-side modelling decisions to modeller variability to be assessed more explicitly.
Conclusion
This paper analysed the simulation results and key input parameters for a building energy modelling task conducted by 20 participants and gathered their feedback on performance modelling through a questionnaire. The study indicated that modeller decisions, modelling software and modelling approach can have a significant impact on energy simulation results. For detailed component-level HVAC modelling, the increase in modelling detail and effort was not necessarily effective in improving the consistency of results. This was on the one hand because complex modelling led to the introduction of additional parameters and reliance on default values, and on the other hand, the added value that detailed modelling could bring was offset by the insufficiency of the required modelling input information and inconsistency in the data granularity of the available information. This uncertainty was reflected in the wide range of variations in the relevant modelling inputs despite the availability of technical data on equipment. Additionally, analysis of the information gathered from the questionnaire revealed that the distance and lack of communication between building designers, modellers and software developers can be the major reason for the discrepancies. Therefore, this paper concluded by proposing several strategies to improve the performance modelling industry considering five stakeholders: practitioners, system suppliers, software developers, policymakers and researchers. In summary, this study provided evidence-based insights and recommendations through real modelling results and modeller feedback. These insights not only demonstrated the performance of different modelling approaches and software in practice, but also identified the modelling challenges faced by modellers under the current modelling framework. Future research and practice can build on these findings to further optimise and refine the provision of modelling information, the modelling process and the modelling validation process.
Footnotes
Acknowledgement
The authors would like to thank all participants for their valuable time and contributions to this study.
Ethical considerations
The study was approved by the University College London, Bartlett School of Environment Energy and Resources (BSEER) Research Ethics Committee (No. 20230419_IEDE_PGR_ETH). It was also registered under reference No Z6364106/2022/12/19 social research in line with UCL’s Data Protection Policy.
Consent to participate
Informed consent was obtained from all participants involved in this study.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.