Effect of Train-Driving Simulator Practice in the European Rail Traffic Management System: An Experimental Study

Abstract

Many European train drivers face major changes in their work with the introduction of the new train-protection system, the European Rail Traffic Management System (ERTMS), as information retrieval shifts from outside to in-cab, and a new rulebook is introduced. Therefore, many train drivers have to be educated in a short time, to make the transition safe and efficient. The purpose was to find out how a successful ERTMS practice can be designed in a physically low-fidelity but highly functional train-driving simulator. An experimental design was used, with 16 drivers divided into two groups: one group practiced in a simulator, and the other in reality. Standard training methodology was used, and the learning outcome was assessed by both measuring driving errors and via instructor evaluation of a simulator test. The drivers also filled in a questionnaire to capture how different factors, such as repeated practice, experience, and self-estimated confidence, correlate with performance. Results show that the simulator group committed significantly fewer driving errors and received significantly higher scores from the instructor. In addition, the simulator group's better performance is mostly caused by the possibility of repeated training of different special cases. The findings also imply that several of the more common special cases on the ERTMS can hardly be provoked in real train driving. Furthermore, this work strengthens the theory that novices can hardly estimate their own ability. Therefore, we argue that this type of low-fidelity simulator is well suited for research purposes, for practicing special cases, and for train operation companies to assess drivers’ skills.

Keywords

human factors passenger rail transportation rail rail safety railroad simulators train

Introduction

Sweden, as in many other countries in Europe, is in an expansive phase of replacing the national signal- and train-protection system with the European Rail Traffic Management System (ERTMS), aiming for increased railway efficiency and to simplify border crossings in Europe through a more harmonized train-driving environment ( 1 ). This development entails a major change for the drivers’ work tasks, including new regulations, information retrieval shifting from lineside to in-cab ( 2 ), new speed profiles ( 3 ), and a changed way of operational communication ( 4 ). Therefore, this globally changing train-driving environment is becoming a very important issue that needs more attention. For Sweden's train operation companies (TOCs) to make the transition safe and efficient, the rollout of the ERTMS requires educating many drivers in a short time. Five-to-eight days (depending on the TOC) of supplementary education whereof about three days is theoretical is needed for active drivers. However, after this education, train drivers still feel insecure when driving in the new ERTMS environment and their concerns are mainly with regard to handling of special cases (i.e., events that do not occur during a routine drive, but the handling of the situation is regulated). This sense of insecurity can be traced to inadequate practice since reality is difficult to control ( 5 ). In Sweden a (physically) low-fidelity simulator (LFS) is widely used within the education of train drivers ( 6 ), and for this study, such a simulator was utilized to examine how effective ERTMS training can be designed.

Simulator Fidelity

A driving simulator's physical fidelity is categorized by its field of view, the degree of the moving base, and the physical resemblance to a real cab ( 7 ). A high degree of motion and field of view is of importance to reproduce speed control and lane position, similar to real driving ( 7 , 8 ), while physical fidelity is less important for novices and when training special cases ( 9 , 10 ). Since a train does not need steering and the speed handling of a passenger train (with an electro-pneumatic braking system) is fairly simple, a LFS is suitable for training passenger-train drivers in the ERTMS and for practicing different special cases that come with the new system. Important, however, is the functional fidelity, to avoid mislearning ( 3 , 11 ).

Simulator Research and Training

Train-driving simulators have been used for education and research purpose for a long time ( 12 ) and previous studies describe several experiments. Verstappen ( 13 ) studied the effect of distraction and showed that a non-considerate person in the cabin led to poorer task completion than driving alone. Other experiments have shown that fatigue and monotonous driving combined with low task-demands impede performance and lead to increased risk for accident involvement ( 14 , 15 ). The impact of further automation has been investigated and the results are in line with the above-mentioned studies by stating that automatic speed control has a negative effect on drivers’ reaction time when facing a critical event ( 16 ). The introduction of the ERTMS is another technological development that, in one study, seems to decrease driver workload, at least on routes without many speed changes ( 17 ). Impaired situation awareness leading to a higher risk of incidents has been demonstrated when drivers facing two competing tasks ( 18 ), and impaired situation awareness can be the result of time pressure ( 19 ).

However, few studies present guidelines of simulator education for train drivers. Tichon ( 20 ) emphasized the importance of designing stress-training scenarios in a simulator environment where the incident cues should be as reality-like as possible to achieve the most effective training. Olsson et al. ( 21 ) studied the Swedish train-driving reality and found that training of special cases is rare and that simulator training therefore should focus on special cases that hardly can be controlled in reality. Tichon and Wallis ( 22 ) demonstrated in a train-driver simulator experiment that repetition of stress-training scenarios can be effective, especially in a short-time perspective, and that a high-fidelity simulator is not critical for an effective training. The literature in other, more explored, simulator domains (e.g., the aviation, marine, and military industries) also points out the importance of realistic and, for the target group, relevant training scenarios as well as an active instructor to maximize the learning outcome ( 23 – 26 ).

In the Swedish train-driver education, it is fair to say that the traditional training model, where the learner drives a real train together with an experienced instructor, is still the dominant method. Although simulator training has occurred on a small scale for a long time, it is only in the last six to eight years that it has become more widespread. Since 2016, without significant exceptions, only one type of simulator has been used within train driver education. This simulator is used in all basic train-driver education and by most TOCs both in annual training and when learning a new train-protection system ( 6 ). Like Australian operators, TOCs in Sweden state quality ( 10 , 12 , 20 ) and efficiency aspects (e.g., reduce training time via instant repetitions, see Myers et al. [ 10 ] and Naweed [ 12 ]), but also a lack of real trains as motivation for the use of simulator training.

Because of the lack of guidelines in the literature for designing effective train-driving simulator training, this study intends to contribute with important knowledge about how train-driver training in a simulator can be conducted in general, and how to use a LFS for ERTMS education in particular.

Self-Assessment

Several researchers have reported interesting results where the learning outcome of simulator training for novice drivers is measured via the learners' self-assessment ( 27 – 29 ). Self-assessment per se can have a positive effect on the learning experience ( 30 ). However, for novices it is often difficult to correctly calibrate their own ability ( 31 , 33 ). That novices often tend to be overconfident is described as the Dunning–Kruger effect and has been empirically substantiated in various fields ( 32 , 33 ). We argue that the drivers in this study are novices when it comes to the ERTMS with its rules and handling of the train-protection system, even though they are otherwise experienced train drivers in general. In part this study aims to find out if newly ERTMS-trained train drivers can estimate their own ability.

Aims and Research Questions

This paper aims to compare practice in a high-functional but low-physical fidelity simulator to practice in real trains according to a standard training methodology with respect to how many times learner drivers experience different special cases, to examine which variables are important when educating safe and efficient ERTMS train drivers and, finally, to investigate how drivers’ self-estimated confidence correlates with performance. To reach this aim, the following research questions were formulated.

How effective is ERTMS training in a LFS as compared to real train-driving practice? How many training opportunities for different special cases are offered when training in reality?

How does age, experience, number of repetitions of special cases, and internship time, respectively, affect performance?

How do drivers' own estimated confidence ratings correlate with performance scores in a ERTMS simulator test?

Method

Design and Participants

In line with previous driving-simulator studies ( 34 , 35 ) an experimental method was used, with train drivers from the same TOC in Sweden, stationed at two different locations (see Figure 1 for the experimental design). The participants were 11 men and five women aging between 24 and 52 years (M = 35.9 years, SD = 9.2 years) with train driver experience ranging from newly examined to 8 years (M = 1.9 years, SD = 2.5 years). After receiving the theoretical part of the ERTMS education for three days, the drivers were divided into two groups. One group (n = 8) carried out the practical part of training exclusively in a simulator environment, while a control group (n = 8) conducted the practice in reality according to the, by this specific TOC, standard training methods. The practice in reality was preferably done on non-revenue trips; however, there were some exceptions, because of vehicle shortages, during revenue service. Thus, the types and frequencies of special cases practiced could only partially be controlled by the instructor. During simulator practice, the instructor had a set of scenarios that included the special cases to choose from. The instructor could restart the scenarios and practice the different special cases again if needed. However, for some of the special cases, a simulator sequence needed to be completed before the specific case could be practiced, which meant that another special case sometimes occurred before it and thus was practiced once more. At both locations, one and the same instructor educated all drivers theoretically and practically and was responsible for seven and nine drivers, respectively. Thus, all drivers in the same location were educated by the same instructor, regardless of practice group. The number of practicing drivers in the simulator group varied between one and four per education round, and the control group practiced alone or with one other driver (the group size depended on how many drivers the TOC could spare for education during that specific period of time). The time of practice at the TOC did not change because of the number of participants; however, this probably affected the number of repetitions for each participant, which also was intended, since one purpose was to evaluate the effect of practice both as a driver and when watching a colleague practice. In training sessions with more than one driver practicing, watching drivers were expected to do this focused, as that time was also included in the practice time. To examine the training effects, the drivers filled in a questionnaire and conducted a simulator test shortly after the education. Both theory and practice were included in the regular TOC ERTMS education. The study lasted for 12 weeks.

Figure 1.

Experimental design.

Simulator Test

Although all participants at some point earlier in their career had been in contact with the simulator type, the control group made a 15-min test-drive before the actual test, to get familiar with the environment, the buttons, the levers, and other simulator functions. The simulator test consisted of two approximately 15–20 min scenarios, including regular driving, speed changes, and three special cases each. Of the total time, special cases were handled for approximately 70% of the time, and the rest of the time, about 30%, consisted of normal train driving, which included some speed changes as well as observing the lineside and in-cab signaling. During the test, which was recorded via video to enable the assessments, an instructor acted as a signaler (train dispatcher), interacting with the driver according to the regulations. The selection of special cases was as follows.

Scenario 1

Start-up within an operational zone far from a marker board in operational mode staff responsibility (SR).

System failure (SF) occurring at an open line.

Incorrect level-crossing when in the operational mode full supervision (FS).

Scenario 2

Start-up within an operational zone close to a marker board in operational mode staff responsibility (SR).

Trip (TR) occurring when at an open line.

Entering and driving in the operational mode on sight (OS) when facing end of movement authority within an operational zone.

To ensure realism, the scenarios were developed in close collaboration with two ERTMS experts with the aim of creating some of the most “commonly” occurring special cases that need to be handled according to regulations. To enable a decisive, valid, and reliable objective measurement, the test was designed with several points of measurement. See the Appendix for a detailed list of events.

Simulator Test Measurement Methods

The simulator test was assessed both by using an objective protocol measuring driving errors and by instructor evaluation, which are widely used methods for measuring performance in driving simulators ( 36 – 38 ). The purpose of this double assessment was to estimate the reliability of the measurement methods and to make the assessment more nuanced. An instructor's evaluation might capture aspects that the objective protocol cannot, such as the distinction between acting safely and securely or in an uncertain and haphazard way.

Driving Errors

The maximum number of driving errors in the objective protocol, developed by two ERTMS instructors (one of whom is the corresponding author of this paper), was 63 and consisted of 23 rule-based errors (according to the regulations), 19 skill-based errors (e.g., handling the vehicle or, in most cases, the train-protection system), 12 ineffective handling errors (time-costly train driving), and nine combinations of these. Three of the possible errors in each scenario were not linked to any specific event but were of a more general nature (driving too slowly, standing still without cause, and an intervention from the train-protection system). To prevent these being too decisive, a maximum of two errors were counted per scenario. The Appendix contains a complete list of events, expected actions, and possible driving errors.

Instructor Evaluation

Two instructors, one from each train-driver location, who had not been part of the ERTMS education, assessed the drivers’ performance during the two scenarios. A total of nine statements were evaluated according to the instructor's view of the driver's performance. These included the six special cases, a general assessment of special case management, a general assessment of normal situations, and an assessment of how time-efficiently the driver worked. The statements were assessed on a 10-point Likert-type scale ( 39 ), where the instructor selected an alternative between 1 and 10, where 1 = totally disagree and 10 = totally agree. An example of a statement was:

The driver is perceived to be sure of how the initial start-up without a known position – Full supervision in Scenario 1 should be handled (quickly makes the correct decision in relation to the regulations).

Test of Internal Reliability

Individually and independently of each other, the two instructors observed video recordings of two drivers and made individual assessments of the performance.

Transfer to Reality

To assess the transferability of the simulator test to reality, the instructors evaluated the following statement on a 10-point Likert-type scale, where 1 = totally disagree and 10 = totally agree:

If the same scenarios that occurred in the simulator at the time of the study had instead taken place in reality, the result (i.e., the drivers' performance in the study and thus your assessment) would have been the same.

Questionnaire

Before the simulator test the drivers filled in a questionnaire including age, experience, duration of practice, and number of repetitions of different special cases and shunting situations (both as a driver and as a driver plus watching a colleague practice). The questionnaire contained nine situations, of which seven were special cases and two were different forms of shunting. However, when reporting the results of situations practiced, the shunting scenarios are not included and in the rest of the results only the six situations that occurred in the simulator test are presented. The seven special cases were as follows.

Start-up in unknown position in mode staff responsibility (SR).

Trip (TR) and post-trip (PT).

Driving in operational mode on sight (OS).

Passing a marker board in mode staff responsibility (SR) after receiving permission.

System failure (SF).

Incorrect level-crossing. This means that the level-crossing equipment does not for certain protect road users.

Lost radio connection (GSM-Railway, used for communication between train and railway regulation control). This is not included in the simulator test because it is so time-consuming.

The drivers were also asked to rate their confidence when handling special cases and normal situations, and to what extent they felt that they had been given sufficient training in various special cases. Also here, a Likert-type scale was used and the ratings were made on the following type of statement on a 10-point scale, where 10 = totally agree and 1 = totally disagree: “I feel confident in how to handle different special cases at ERTMS”; “I feel confident in how to handle normal situations at ERTMS”; and “I have received enough practice regarding various special cases.”

Ten-Point Likert-Type Scale

A 10-point Likert-type scale was used for measuring driver performance, perceived confidence, and for how transferable this study is to real train driving. The 10-point scale is frequently used ( 39 ), for example in driver performance measuring ( 26 ). It contains no clarifications in words except for the end points, which could be a disadvantage compared to scales with fewer points. However, the advantages, more options for the respondent (and a higher resolution for the researcher, meaning slightly better reliability and statistical power) and people’s familiarity with this type of rating ( 31 , 40 ), outweighed the disadvantages. The scale was intended to be analogous to the percentage of agreement or disagreement. That is, 1–5 correspond to 100%, 78%, 56%, 33%, and 11% disagreement, respectively, whereas 6–10 would indicate 11%, 33%, 56%, 78%, and 100% agreement, respectively. Thus, an 8 is a moderately positive answer and corresponds in this way to a 6 on a 7-point scale (which contains only three positive answer options).

Analysis of Data

To find out what effect the simulator practice had on performance and other independent variables, t-tests for independent samples were used. Also, t-tests for independent samples were used to analyze the effect of simulator practice on the number of repetitions and which types of errors the groups made. To assess the reliability of the two measurement methods, and which variables affected performance, product-moment correlations were used. Cohen's d and partial eta squared ( $η_{p}^{2}$ ) were used as measures of effect size. Internal reliability for the ratings of the two instructors was tested by Cronbach's alpha, and finally, an analysis of variance (ANOVA) was used to test for main and interaction effects pertaining to location. The significance level was 5%.

Apparatus and Environment

The simulator, used for both training and testing, had a fixed base and consisted of a touchscreen PC and a lever for driving and braking, together with a 50-inch flat-screen displaying the outside view (Figure 2). Although, in physical terms, the fidelity is low ( 7 ), the functional fidelity is high with the retardation, acceleration, buttons, and train-protection system corresponding with those of the real train. The simulator software uses the same ERTMS version as reality (baseline 3) and the 16 possible training scenarios, which can be customized via a trigger function in real-time, are developed in collaboration with ERTMS experts from three different TOCs and based on real Swedish ERTMS equipped lines. Therefore, the instructors and drivers could use the same documents (i.e., line descriptions, safety forms, timetables, etc.) as they would in reality. In addition, the instructor acted as a signaler, including performing safety calls and writing forms during the simulator drive. When referring to this simulator as a LFS, it is the physical characteristics that are meant.

Figure 2.

Simulator environment.

Results

Simulator Test Performance

Table 1 demonstrates a significantly better performance by the simulator group than the control group, with 38% fewer driving errors, and a 34% higher score from the instructor. The number of driving errors ranged between 14 and 27 for the control group, while the lowest and highest number of errors in the simulator group were 10 and 17, respectively. The instructor evaluation for the control group ranged between 3.11 and 5.00 and for the simulator group between 4.78 and 7.44. A strong correlation was found between the measured driving errors and the instructors’ evaluation (r = .85, p < .05). The two instructors also submitted a written motivation for their assessment for each situation, even though they were not asked to do so. An example of a motivation for a high score relating to scenario 1, special case 3 was: Does not use the positioning to identify the level-crossing, otherwise well-acted. An example for a low score (3/10) relating to the same scenario was: Careful driving against level-crossing, stops before the marker board and calls the signaler, forgets to signal the horn when passing.

Table 1.

Driver Simulator Test Performance

Driver performance
Measurement method	Simulator (n = 8)		Control (n = 8)		t	p	d
Measurement method	M	SD	M	SD	t	p	d
Driving errors (max. 63)	12.63	2.50	20.50	4.69	4.19	<.01	2.09
Instructors' evaluation (scale 1–10)	5.97	0.82	3.93	0.71	5.34	<.01	2.67

Note: max. = maximum; M = mean; SD = standard deviation; t = Student t-test value; p = p-value, probability that the result is a Type 1 error; d = effect size measured as Cohen's d.

Bold = significant at the 95% confidence level.

Type of Errors

The control group made more errors of all error types, but because of a large standard deviation, only the difference in skill-based errors was significant (Table 2). The percentual difference between the groups was 91% for skill-based errors, 69% for ineffectivity errors, and 34% for rule-based errors. Note that when an error was classified as a combination of two error types, 0.5 error per type was counted. Of the errors not linked to any particular event, the drivers made between zero and two errors, and thus the maximum of two per error was not exceeded.

Table 2.

Driver Error Types

Error type	Group	N	M	SD	t	p	d
Rule-based errors (max. 27)	Simulator	8	5.69	2.33	1.65	.12	2.35
Rule-based errors (max. 27)	Control	8	7.63	2.37	1.65	.12	2.35
Skill-based errors (max. 23)	Simulator	8	4.06	1.43	2.91	.02	2.53
Skill-based errors (max. 23)	Control	8	7.75	3.28	2.91	.02	2.53
Ineffectivity errors (max. 13)	Simulator	8	2.88	1.84	1.68	.12	2.38
Ineffectivity errors (max. 13)	Control	8	4.88	2.81	1.68	.12	2.38

Note: max. = maximum; N = number of participants; M = mean; SD = standard deviation; t = Students t-test value; p = p-value, probability that the result is a Type 1 error; d = effect size measured as Cohen's d.

Bold = significant at the 95% confidence level.

Test of Internal Reliability

Individual evaluations from the two instructors, based on 18 measurement values each for the same two drivers, gave α = .88, reflecting high internal consistency.

Effects of Group and Location

A 2 × 2 between-subjects ANOVA tested for a potential main effect of Location, and a potential interaction between Location and Group, on driving errors and instructor evaluations. As previously found with t-tests, there were main effects of Group on driving errors, F(1, 12) = 25.12, MSE = 11.33, p < .001, $η_{p}^{2}$ = .68; and on instructor evaluations, F(1, 12) = 30.80, MSE = 0.53, p < .001, $η_{p}^{2}$ = .72. However, there was no significant effect for Location on driving errors, F(1, 12) = 1.62, MSE = 11.33, ns, nor on instructor evaluations, F(1, 12) = 1.21, MSE = 0.53, ns. Finally, there was no interaction effect for Location × Group on either driving errors, F(1, 12) = 4.01, MSE = 11.33, ns, or on instructor evaluations, F(1, 12) = 2.09, MSE = 0.53, ns.

Transfer to Reality

Both instructors answered the statement about the transferability of the simulator test to reality with an 8 on a scale of 1–10.

Number of Repetitions During Practice

The simulator group reported a higher number of repetitions for each of the seven special cases, both for own practice (OP) and for watching a colleague practice (WC). For half of the cases the difference was significant (see Table 3). The control group reported more than one repetition of own practice and watching a colleague practice (OP + WC) for only two special cases, while the simulator group declared more than one repetition in all seven special cases (OP + WC). Out of a maximum of 56 (i.e., seven special cases multiplied with eight participants per group), the control group stated no practice of a specific special case 28 times, and more than four repetitions only three times (OP + WC). The participants in the simulator group declared zero repetitions four times and more than four repetitions 18 times.

Table 3.

Number of Repetitions of Special Cases During Practice

Number of repetitions of special cases
Special case	Practice	Group	N	M	SD	t	p	d
Unknown position in staff responsibility (SR)	OP	Simulator	8	2.38	1.51	2.24	.04	1.45
	OP	Control	8	0.75	1.39	2.24	.04	1.45
	OP + WC	Simulator	8	5.88	4.26	3.05	.01	3.20
	OP + WC	Control	8	1.00	1.51	3.05	.01	3.20
Trip (TR)	OP	Simulator	8	1.50	0.76	3.47	.01	0.65
	OP	Control	8	0.38	0.52	3.47	.01	0.65
	OP + WC	Simulator	8	3.38	1.77	4.23	.01	1.36
	OP + WC	Control	8	0.50	0.76	4.23	.01	1.36
On sight (OS)	OP	Simulator	8	4.13	2.90	1.82	.10	2.20
	OP	Control	8	2.13	1.13	1.82	.10	2.20
	OP + WC	Simulator	8	9.50	4.21	4.18	.01	3.17
	OP + WC	Control	8	2.88	1.55	4.18	.01	3.17
Passing a marker board in SR	OP	Simulator	8	1.00	3.88	0.58	.66	1.12
	OP	Control	8	0.75	4.12	0.58	.66	1.12
	OP + WC	Simulator	8	2.88	1.96	0.80	.25	2.08
	OP + WC	Control	8	1.63	2.20	0.80	.25	2.08
System failure (SF)	OP	Simulator	8	0.75	0.46	1.53	.15	0.49
	OP	Control	8	0.38	0.52	1.53	.15	0.49
	OP + WC	Simulator	8	2.5	0.76	6.11	.01	0.66
	OP + WC	Control	8	0.5	0.54	6.11	.01	0.66
Incorrect level-crossing	OP	Simulator	8	0.63	0.74	1.72	.12	0.58
	OP	Control	8	0.13	0.35	1.72	.12	0.58
	OP + WC	Simulator	8	2.00	1.93	2.70	.03	1.39
	OP + WC	Control	8	0.13	0.35	2.70	.03	1.39
Lost radio connection	OP	Simulator	8	0.75	1.04	0.92	.38	0.82
	OP	Control	8	0.38	0.52	0.92	.38	0.82
	OP + WC	Simulator	8	2.13	2.23	1.45	.17	1.69
	OP + WC	Control	8	0.88	0.84	1.45	.17	1.69

Note: N = number of participants; M = mean; SD = standard deviation; t = Students t-test value; p = p-value, probability that the result is a Type 1 error; d = effect size measured as Cohen's d; OP = own practice; WC = watching a colleague practice.

Bold = significant at the 95% confidence level.

Mean Differences Between Groups

An average difference of 11 h more practice time for the control group than the simulator group was found, but this difference was only close to significant (see Table 4). The control group declared significantly fewer (5.7) repetitions of the special cases on average when driving themselves, and when including also watching a colleague practice the special cases, the difference between the groups was even larger (18.2 repetitions).

Table 4.

Drivers’ Questionnaire Results

Questionnaire results
Variable	Simulator (n = 8)		Control (n = 8)		t	p	d
Variable	M	SD	M	SD	t	p	d
Practice time (hours)	9.0	1.1	20.0	16.2	1.92	.10	0.96
Number of own repetitions	9.4	3.3	3.7	3.3	3.4	.01	1.70
Number of own repetitions and seeing others practice	23.2	5.9	5.0	3.6	7.48	.01	3.74
Perceived confidence when handling special cases (1–10)^a	5.4	1.9	5.1	1.2	0.39	.70	0.20
Perceived confidence when handling normal situations (1–10)^a	7.4	2.1	8.25	1.0	1.1	.30	0.54
Enough practice on different special cases (1–10)^a	6.0	3.1	4.1	2.1	1.4	.18	0.71
Train-driver experience (years)	2.4	3.1	1.4	1.7	0.78	.44	0.40
Age	35.1	8.4	36.7	10.4	0.34	.74	0.17

Note: M = mean; SD = standard deviation; t = Students t-test value; p = p-value, probability that the result is a Type 1 error; d = effect size measured as Cohen's d.

Drivers’ own estimates.

Bold = Significant at the 95% confidence level.

The control group included two drivers with at least five years’ experience, and the simulator group one, while all other drivers had three years’ experience or less. Both groups included three drivers over 40 years of age and three below 30. Neither age nor experience differed significantly between groups. There were two female drivers in the control group and three in the simulator group. Three questions covered the drivers’ own estimation of how well a statement agreed with their own opinion. No significant difference with regard to confidence in handling either special cases or normal situations after completing the internship could be seen. Finally, no significant difference between groups with regard to if they thought that they had received enough special case training was found.

Correlations

Correlations between simulator test performance and questionnaire responses from the train drivers are displayed in Tables 5 –7.

Table 5.

Correlations Between Performance and the Questionnaire Responses (n = 16)

	1	2	3	4	5	6	7	8	9	10
1. Driving errors	na	−.85*	.71*	−.45	−.68*	.12	.28	.03	.04	.05
2. Instructors’ evaluation	na	na	−.49	.54*	.77*	−.05	−.22	−.11	−13	−.16
3. Practice time	na	na	na	.10	−.23	.29	.17	.04	.15	−.08
4. Number of own repetitions	na	na	na	na	.82*	.35	−.18	.46	.43	.02
5. Number of own repetitions and seeing others practice	na	na	na	na	na	.13	.25	−.33	.15	.15
6. Perceived confidence when handling special cases	na	na	na	na	na	na	.56*	.66*	−.01	−.11
7. Perceived confidence when handling normal situations	na	na	na	na	na	na	na	.18	−.32	−.20
8. Estimation of enough practice on different special cases	na	na	na	na	na	na	na	na	.42	.01
9. Train-driver experience	na	na	na	na	na	na	na	na	na	.44
10. Age	na	na	na	na	na	na	na	na	na	na

p < .05.

na = not applicable.

For the groups combined (Table 5), the number of repetitions as a driver and by watching a colleague practice correlated with performance as measured by driving errors, r = −.68, p < .05, and by instructors’ evaluation, r = .77 p < .05. Including only the number of repetitions as a driver correlated significantly with performance as measured by instructors’ evaluation, r = .54, p < .05, but failed to reach significance when measured by driving errors, r = −.45, ns.

Practice time correlated with driving errors, r = .71, p < .05, such that more practice time led to more errors. Experience, age, the drivers’ perceived confidence, and estimation of whether they received enough practice were not significantly correlated with performance.

Tables 6 and 7 show correlations for the simulator and control group separately. Since there are few drivers in each group (n = 8), a strong correlation is required to reach the critical value for significance with six degrees of freedom, r = .71.

Table 6.

Correlations Between Performance and the Questionnaire Responses: Control Group (n = 8)

	1	2	3	4	5	6	7	8	9	10
1. Driving errors	na	−.69	.71*	.51	.58	.28	−.03	.53	.37	−.10
2. Instructors’ evaluation	na	na	−.41	−.17	−.15	.01	.31	−.76*	−.29	.32
3. Practice time	na	na	na	.92*	.84*	.72*	.12	.49	.70	−.13
4. Number of own repetitions	na	na	na	na	.90*	.85*	.02	.48	.72*	−.17
5. Number of own repetitions and seeing others practice	na	na	na	na	na	na	.90*	.12	.53	.36
6. Perceived confidence when handling special cases	na	na	na	na	na	na	.31	.43	.32	−.27
7. Perceived confidence when handling normal situations	na	na	na	na	na	na	na	−.48	−.23	−.59
8. Estimation of enough practice on different special cases	na	na	na	na	na	na	na	na	.27	−.46
9. Train-driver experience	na	na	na	na	na	na	na	na	na	.01
10. Age	na	na	na	na	na	na	na	na	na	na

p < .05.

na = not applicable.

Table 7.

Correlations Between Performance and the Questionnaire Responses: Simulator Group (n = 8)

	1	2	3	4	5	6	7	8	9	10
1. Driving errors	na	−.67	−.27	−.61	−.75*	.42	.31	.55	.35	.15
2. Instructors’ evaluation	na	na	.62	.08	.28	−.37	−.10	−.75*	−.65	−.64
3. Practice time	na	na	na	−.52	.28	−.32	.07	−.69	−.82*	−.91*
4. Number of own repetitions	na	na	na	na	.55	.08	−.00	.23	.25	.44
5. Number of own repetitions and seeing others practice	na	na	na	na	na	−.26	−.03	−.16	−.22	−.11
6. Perceived confidence when handling special cases	na	na	na	na	na	na	.72*	.78*	−.15	.04
7. Perceived confidence when handling normal situations	na	na	na	na	na	na	na	.58	−.29	−.38
8. Estimation of enough practice on different special cases	na	na	na	na	na	na	na	na	.43	.48
9. Train-driver experience	na	na	na	na	na	na	na	na	na	.86*
10. Age	na	na	na	na	na	na	na	na	na	na

p < .05.

na = not applicable.

The simulator group's performance measured by driving errors correlates with the number of repetitions as a driver and seeing a colleague practice, r = −.75, p < .05, such that more repetitions are associated with fewer errors.

For both groups, a significant negative correlation is stated between the drivers’ own assessment of sufficient special case training and the instructors’ assessment of performance, r = −.76, p < .05 and r = −.75, p < .05, respectively. Thus, drivers who to a greater extent believed having received sufficient training actually performed worse in the simulator test.

None of the groups demonstrated a significant correlation between performance and the drivers’ perceived confidence, age, or experience.

The control group revealed a strong and significant correlation between practice time and number of repetitions of special cases, r = .92, p < .05, for being a driver, and r = .84, p < .05 when adding watching a colleague practice; however, for the simulator group there was no such significant correlation.

Discussion

LFS Versus Standard Reality Training

Clear evidence from the present results supports the previous literature that practice in a simulated environment is both time- and cost-effective as compared to standard training methods in reality, because it provides better prepared train drivers in a shorter period of time and includes fewer external resources (e.g., vehicle availability, track capacity, help from signalers, and driving to a suitable place) ( 10 , 12 ). It is possible that the simulator groups' familiarity with the simulator had some effect on performance. However, since the control group started with a test-drive, and considering that all drivers, at some point earlier in their career, had had contact with the simulator, it does not explain the large difference between the groups.

Fewer errors were made by the simulator group considering all types of errors, although the largest and single significant difference considered skill-based errors (91%). As skill-based errors mainly consisted of managing the train-protection system, the superiority of the simulator group is probably a result of the mass training of the train-protection system when handling different special cases. An explanation of the difference for ineffectivity errors (69%) is likely an increased uncertainty in the control group because of fewer repetitions, which leads to more careful driving and more consideration during the simulator test. The relatively small difference between the groups for the number of rule-based errors (34%) may be because the regulations to a greater extent (than handling the train-protection system) can be learned via theoretical training.

The strong correlation between the number of driving errors and the instructors’ evaluation provides support not only for the effectiveness of LFS training but also ensures the reliability of the objective protocol measuring driving errors and the instructors’ assessment ability. It is not unlikely that the instructors' written motivation for each assessment may have helped them to be more structured in their evaluation.

Learner Experience of Special Cases During Practice

As previous research has shown, real-life train driving offers few opportunities to practice special cases ( 21 ). The most important information in Table 3 is that, for only two out of seven special cases, the control group stated more than one learning experience on average and a high number of non-practice of special cases in reality. On 28 occasions (i.e., half of the maximum) the control group declared zero repetitions, including watching a colleague practice. Thus, there is a large risk that a driver has not received any practical practice at all on some important special cases after completing standard practice (i.e., in reality). Even if the practice is not carried out during revenue, this study’s data clearly demonstrates the difficulty of provoking several of the more common ERTMS special cases the driver is expected to handle correctly, for example a trip, a system failure, a loss of radio connection, or an incorrect level-crossing.

Repeated Practice Affects Performance

Not surprisingly, and in line with the previous literature, an important reason for simulator training proving to be more effective than reality (i.e., better performance despite shorter practice time) is because it provides the opportunity to practice different special cases repeatedly ( 20 ). Also, results imply that practicing together in small groups is effective, as watching a colleague practice also seem to improve driver performance. Likely, this is a result of drivers being focused when watching others practice, which is in line with the previous literature clearly stating that being active leads to an effective learning experience ( 41 ). Since other research points out that just watching is not enough to acquire skills to perform ( 42 ), it remains to be determined which distribution between watching and practice is most effective.

Practice time and performance correlated positively, quite unexpectedly. However, this is a spurious correlation, since the control group noted significantly more internship time, while the simulator group performed significantly better. One explanation could be that the instructor thought that some of the drivers in the control group needed more training, but that reality is not good enough as a practice arena because it is so difficult to provoke special cases.

The reason that only the control group showed a correlation between practice time and number of repetitions is because of the large variation of internship time in the control group, M = 20 h (SD = 16.2 h), in contrast to the simulator group, which was more homogeneous, M = 9.0 h (SD = 1.1 h). Reality's dependence on external factors probably contributed to the great practice-time variation in the control group.

Self-Estimation

Train-driving experience does not seem to correlate with performance in this ERTMS simulator test. Therefore, it is reasonable to label all newly educated ERTMS drivers as novices (at least when it comes to the specific rules and handling of situations assessed in this simulator test), even if their experience as train drivers in general differs.

The non-existent correlation between self-estimated confidence and performance is in line with the findings of previous studies ( 31 , 32 ) that a novice hardly can estimate her own ability.

A surprising result was that both groups showed a (negative) significant correlation between a high score when self-estimating whether one had received sufficient special training (number 8 in Tables 6 and 7) and a poor performance score from instructor evaluation (number 2). Either the drivers did not understand the statement, or it is a result of coincidence, which then shows the difficulty of estimating their own ability as a novice.

Transfer to Reality

Both instructors rated the test transferability to reality as 8 on a 10-point scale, which indicates a good transferability. This, and that the scenarios were designed by experienced ERTMS experts to be realistic, plus that the handling of special cases does not differ from reality, suggests that the results are also to some extent valid as a performance indicator in real-world ERTMS train driving.

Conclusions and Practical Implications

This study aimed to contributing guidelines about train-driving simulator training in general and ERTMS training in a LFS in particular. Below some guidelines are argued for based on the results of this study.

Firstly, repeated practice in a train-driver simulator environment is, as Tichon and Wallis ( 22 ) also found, effective, at least from a short perspective. As the practical part of the ERTMS training is not aimed at learning to drive a train (which the drivers already can), but rather at putting the theoretical knowledge into a practical context, and above all learning to handle new special cases, a (highly functional) low-fidelity train-driving simulator is excellent for the purpose.

Secondly, a LFS may, based on the results of this study, be useful in other industries where new technology or regulations have been introduced, such as new special cases in a somewhat new environment within the profession task.

Thirdly, when a simulator is used as a supplement to reality practice, it is particularly important to focus on practice of the special cases that are difficult to provoke in reality. The large difference between the groups for skill-based errors, which primarily includes handling the train-protection system, suggests that the simulator training on special cases has a special focus on handling the train-protection system.

Fourthly, the strong correlation between the measurement methods suggests that both the protocol measuring driving errors and the instructors' structured evaluation can be used for reliable assessments of a driver's performance, not only to assess ERTMS knowledge, but also for other examinations in train driving and other driving industries.

Fifthly, and finally, since self-assessments made by novices are unreliable, and that reality is difficult to control, examination of the ERTMS should be done practically, preferably with the help of standardized tests in a simulator where instructors can assess the train drivers’ knowledge in different situations. Relying on the drivers’ self-estimated confidence and not testing the drivers’ practical knowledge in the handling of special cases before driving by themselves could potentially lead to costly mistakes.

Limitations and Suggestions for Future Work

Even though the simulator function is very similar to reality, the same emotions (e.g., stress or fear) can hardly be evoked by this type of LFS. Consequently, it is possible that a driver's performance in reality could be both better (more focused) and worse (more stressed or less engaged) than in a simulated environment without real consequences. Also, the test-time is relatively short and therefore the effects of fatigue or monotonous driving are not included in this study. Although simulator sickness was not detected in this study, it has been shown to be a problem in simulator training, particularly among older drivers. It could be a problem to consider if ERTMS training is to take place exclusively in a simulator environment.

This study is limited to passenger-train driving with an electro-pneumatic braking system (the train react almost directly to the driver’s actions with the driving and brake levers). The speed handling of a freight train with an ordinary pneumatic braking system is more difficult and places other demands on the driver (much longer reaction time between drivers’ brake-handling and actual brake-reaction), which also would be of interest to investigate further.

Because the simulator test is carried out immediately after completion of training, the interpretation of the training effects is limited to a short-term perspective. A more long-term perspective would be useful to gain an understanding of how well the training effects age over time.

A larger set of data, including more drivers, should improve the power of the variables studied in this article and provide opportunities for a more sophisticated and rigorous data analysis.

The present study shows evidence that repeated practice (as a driver or watching a colleague practice) are important for achieving a good learning outcome but says nothing about how the training sessions should be designed. To examine the effect of different educational methods in a train-driver simulator would benefit both the research community and practical train-driver education.

Supplemental Material

sj-docx-1-trr-10.1177_03611981221135802 – Supplemental material for Effect of Train-Driving Simulator Practice in the European Rail Traffic Management System: An Experimental Study

Supplemental material, sj-docx-1-trr-10.1177_03611981221135802 for Effect of Train-Driving Simulator Practice in the European Rail Traffic Management System: An Experimental Study by Niklas Olsson, Björn Lidestam and Birgitta Thorslund in Transportation Research Record

Footnotes

Acknowledgements

This research was funded by the Swedish Transport Administration. The authors also wish to thank the TOC and the participants for their collaboration and contribution.

Author Contributions

The authors confirm contribution to the paper as follows: study conception and design: N. Olsson, B. Lidestam; data collection: N. Olsson; analysis and interpretation of results: N. Olsson, B. Lidestam; draft manuscript preparation: N. Olsson, B. Thorslund, B. Lidestam. All authors reviewed the results and approved the final version of the manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by the Swedish Transport Administration (TRV 2019/70073).

ORCID iDs

Niklas Olsson

Björn Lidestam

Data Accessibility Statement

The datasets generated and/or analyzed during the current study are not publicly available because of confidentiality reasons but are available from the corresponding author on reasonable request.

Supplemental Material

Supplemental material for this article is available online.

References

Młyńczak

Toruń

Bester

European Rail Traffic Management System (ERTMS). In Intelligent Transportation Systems – Problems and Perspectives. Studies in Systems, Decision and Control ( Sładkowski

Pamuła

, eds.), Vol. 32, Springer International Publishing, Cham, Switzerland, 2016, pp. 217–242.

Young

Stanton

Walker

In Loco Intellegentia: Human Factors for the Future European Train Driver. International Journal of Industrial and Systems Engineering, Vol. 1, No. 4, 2006, pp. 485–501.

Rosberg

Cavalcanti

Thorslund

Prytz

Moertl

Driveability Analysis of the European Rail Transport Management System (ERTMS) — A Systematic Literature Review. Journal of Rail Transport Planning & Management, Vol. 18, 2021, p. 100240.

Smith

Kyriakidis

Majumdar

Ochieng

Impact of European Railway Traffic Management System on Human Performance in Railway Operations: European Findings. Transportation Research Record: Journal of the Transportation Research Board, 2018. 2374: 83–92.

Kecklund

Lindgren

Nordlöf

Investigating the Effects of the ERTMS Driver Machine Interface on Train Driver Behaviour and Railway Safety. In Rail Human Factors: Supporting Reliability, Safety and Cost Reduction ( Dadashi

Scott

Wilson

Mills

, eds.), Taylor & Francis Group, London, 2013, pp. 201–205.

Thorslund

Rosberg

Lindström

Peters

User-Centered Development of Train Driver Simulator for Education and Training. Presented at 8th International Conference on Railway Operations Modelling and Analysis (ICROMA 2019), Norrköping, Sweden, 2019.

Wynne

Beanland

Salmon

Systematic Review of Driving Simulator Validation Studies. Safety Science, Vol. 117, 2019, pp. 138–151.

Mecheri

Lobjois

Steering Control in a Low-Cost Driving Simulator: A Case for the Role of Virtual Vehicle Cab. Human Factors, Vol. 60, No. 5, 2018, pp. 719–734.

Noble

The Relationship Between Fidelity and Learning in Aviation Training and Assessment. Journal of Air Transportation, Vol. 7, No. 3, 2002, pp. 33–54.

10.

Myers

Starr

Mullins

Flight Simulator Fidelity, Training Transfer, and the Role of Instructors in Optimizing Learning. International Journal of Aviation, Aeronautics, and Aerospace, Vol. 5, No. 1, 2018, p. 6.

11.

Naweed

Balakrishnan

That Train has Already Left the Station! Improving the Fidelity of a Railway Safety Research Simulator at Post-Deployment. Proceedings of the Institution of Mechanical Engineers, Part F: Journal of Rail and Rapid Transit, Vol. 227, No. 5, 2013, pp. 419–426.

12.

Naweed

Simulator Integration in the Rail Industry: The Robocop Problem. Proceedings of the Institution of Mechanical Engineers, Part F: Journal of Rail and Rapid Transit, Vol. 227, No. 5, 2013, pp. 407–418.

13.

Verstappen

The Performance of Dutch Train Drivers Based on the Impact of the Presence of a Second Person in the Cab. Proceedings of the Institution of Mechanical Engineers, Part F: Journal of Rail and Rapid Transit, Vol. 231, No. 10, 2017, pp. 1130–1140.

14.

Dorrian

Roach

Fletcher

Dawson

Simulated Train Driving: Fatigue, Self-Awareness and Cognitive Disengagement. Applied Ergonomics, Vol. 38, No. 2, 2007, pp. 155–166.

15.

Dunn

Williamson

Driving Monotonous Routes in a Train Simulator: The Effect of Task Demand on Drving Performance and Subjective Experience. Ergonomics, Vol. 55, No. 9, 2012, pp. 997–1008.

16.

Brandenburger

Jipp

Effects of Expertise for Automatic Train Operations. Cognition, Technology & Work, Vol. 19, No. 4, 2017, pp. 699–709.

17.

van der Weide

. ERTMS Roll out From a Train Driver’s Perspective. Presented at Sixth International Human Factors Rail Conference, London, 2017.

18.

Hickey

Collins

Disinhibition and Train Driver Performance. Safety Science, Vol. 95, 2019, pp. 104–115.

19.

Suter

Stoller

Use of Simulators to Investigate Complex Issues Relating to Human Factors. Presented at FORMS/FORMAT 2014 10th Symposium on Formal Methods for Automation and Safety in Railway and Automotive Systems, Inst. Fuär Verkehrssichertheit und Automatisierungstechnik, Braunschweig, Germany, 2014, pp. 86–96.

20.

Tichon

The Use of Expert Knowledge in the Development of Simulations for Train Driver Training. Cognition, Technology and Work, Vol. 9, No. 4, 2007, pp. 177–187.

21.

Olsson

Lidestam

Thorslund

The Practical Part of Train Driver Education: Experience, Expectations, and Possibilities. European Transport Research Review, Vol. 13, No. 1, 2021, p. 52.

22.

Tichon

Wallis

Stress Training and Simulator Complexity: Why Sometimes More is Less. Behaviour & Information Technology, Vol. 29, No. 5, 2010, pp. 459–466.

23.

Aronsson

Artman

Brynielsson

Lindquist

Ramberg

Design of Simulator Training: A Comparative Study of Swedish Dynamic Decision-Making Training Facilities. Cognition, Technology & Work, Vol. 23, No. 1, 2021, pp. 117–130.

24.

Sellberg

Lindmark

Rystedt

Learning to Navigate: The Centrality of Instructions and Assessments for Developing Students’ Professional Competencies in Simulator-Based Training. WMU Journal of Maritime Affairs, Vol. 17, No. 2, 2018, pp. 249–265.

25.

Hjelmervik

Nazir

Myhrvold

Simulator Training for Maritime Complex Tasks: An Experimental Study. WMU Journal of Maritime Affairs, Vol. 17, No. 1, 2018, pp. 17–30.

26.

McClernon

McCauley

O’Connor

Warm

Stress Training Improves Performance During a Stressful Flight. Human Factors, Vol. 53, No. 3, 2011, pp. 207–218.

27.

Saus

Johnsen

Saus

Johnsen

Eid

Perceived Learning Outcome: The Relationship Between Experience, Realism and Situation Awareness During Simulator Training. International Maritime Health, Vol. 62, No. 4, 2010, pp. 258–264.

28.

Renganayagalu

Mallam

Nazir

Ernstsen

Haavardtun

Impact of Simulation Fidelity on Student Self-Efficacy and Perceived Skill Development in Maritime Training. TransNav, the International Journal on Marine Navigation and Safety of Sea Transportation, Vol. 13, No. 3, 2019, pp. 663–669.

29.

Oskarsson

P.-A.

Nählinder

Training Effects in a Low Fidelity Combat Vehicle Simulator. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 56, No. 1, 2012, pp. 1566–1570.

30.

McMillan

Hearn

Student Self-Assessment: The Key to Stronger Student Motivation and Higher Achievement. Educational Horizons, Vol. 87, No. 1, 2008, pp. 40–49.

31.

Boud

Falchikov

Quantitative Studies of Student Self-Assessment in Higher Education: A Critical Analysis of Findings. Higher Education, Vol. 18, No. 5, 1989, pp. 529–549.

32.

Mahmood

Relationship of Students’ Perceived Information Literacy Skills With Personal and Academic Variables. Libri, Vol. 63, No. 3, 2013, pp. 232–239.

33.

Kruger

Dunning

Unskilled and Unaware of it: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments. Journal of Personality and Social Psychology, Vol. 77, No. 6, 1999, pp. 1121–1134.

34.

Power

S. M.

Power

MacKinnon

Simões Ré

Effect of Simulator Training on Novice Operators’ Abilities to Navigate in Ice. Presented at International Conference and Exhibition on Performance of Ships and Structures in Ice (ICETECH), Anchorage, AK, 2010.

35.

Hirsch

Bellavance

Transfer of Skills Learned on a Driving Simulator to On-Road Driving Behavior. Transportation Research Record: Journal of the Transportation Research Board, 2017. 2660: 1–6.

36.

Lee

Cameron

Lee

Assessing the Driving Performance of Older Adult Drivers: On-Road Versus Simulated Driving. Accident Analysis & Prevention, Vol. 35, No. 5, 2003, pp. 797–803.

37.

Shechtman

Classen

Awadzi

Mann

Comparison of Driving Errors Between On-the-Road and Simulated Driving Assessment: A Validation Study. Traffic Injury Prevention, Vol. 10, No. 4, 2009, pp. 379–385.

38.

de Winter

de Groot

Mulder

Wieringa

Dankelman

Mulder

Relationships Between Driving Simulator Performance and Driving Test Results. Ergonomics, Vol. 52, No. 2, 2009, pp. 137–153.

39.

Likert

A Technique for the Measurement of Attitudes. Archives of Psychology, Vol. 22, No. 140, 1932, pp. 1–55.

40.

Dawes

Do Data Characteristics Change According to the Number of Scale Points Used? An Experiment Using 5-Point, 7-Point and 10-Point Scales. International Journal of Market Research, Vol. 50, No. 1, 2008, pp. 61–77.

41.

Prince

Does Active Learning Work? A Review of the Research. Journal of Engineering Education, Vol. 93, No. 3, 2004, pp. 223–231.

42.

Kardas

O’Brien

Easier Seen Than Done: Merely Watching Others Perform can Foster an Illusion of Skill Acquisition. Psychological Science, Vol. 29, No. 4, 2018, pp. 521–536.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB