Interrater Reliability of the Simulation Fidelity (SiFi) Scale

Abstract

A limitation of the extant research on the training effectiveness of high versus low fidelity simulation is the lack of a standardized methodology to define high and low fidelity in way that is generalizable across studies. This paper presents a novel approach to standardizing the definition of fidelity using The Simulation Fidelity (SiFi) scale. The SiFi scale is a 6-point scale used to rate the fidelity of a simulation. The present study examines the inter-rater reliability of the SiFi scale to define the fidelity of simulated task elements with respect to their real-world referent. Two subject matter experts (SMEs) completed a series of aviation maneuvers in a virtual reality (VR) simulator. Following the maneuvers, each SME provided 117 ratings of simulated elements using the SiFi scale. Cohen’s k was calculated to determine interrater reliability. Results of this analysis indicate strong reliability of the SiFi scale.

Get full access to this article

View all access options for this article.

References

American Psychological Association. (n.d.). Reliability. In APA dictionary of psychology. Retrieved January 06, 2021, from https://dictionary.apa.org/reliability

Beaubien

J. M.

Baker

D. P.

(2004). The use of simulation for training teamwork skills in health care: how low can you go?. Quality & safety in health care, 13(Suppl 1), i51–i56. https://doi.org/10.1136/qhc.13.suppl_1.i51

Champney

R. K.

Stanney

K. M.

Milham

Carroll

M. B.

Cohn

J. V.

(2017). An examination of virtual environment training fidelity on training effectiveness. International Journal of Learning Technology, 12(1), 42-65.

Cohen

. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104

Harris

D. J.

Bird

J. M.

Smart

P. A.

Wilson

M. R.

Vine

S. J.

(2020). A Framework for the Testing and Validation of Simulated Environments in Experimentation and Training. Frontiers in psychology, 11, 605. https://doi.org/10.3389/fpsyg.2020.00605

Hays

R. T.

Singer

M. J.

(1989). Simulation fidelity as an organizing concept. In Simulation fidelity in training system design (pp. 47-75). Springer, New York, NY.

Jeon

(2015). The virtual flier: The link trainer, flight simulation, and pilot identity. Technology and culture, 28-53.

Lapkin

Levett-Jones

. (2011), A cost–utility analysis of medium vs. high-fidelity human patient simulation manikins in nursing education. Journal of Clinical Nursing, 20, 3543-3552. https://doi.org/10.1111/j.13652702.2011.03843.x

Liu

Macchiarella

N. D.

Vincenzi

D. A.

(2008). Simulation Fidelity. In Human factors in simulation and training (pp. 61-73). CRC Press, Boca Raton, FL.

10.

McCoy-Fisher

Mishler

Bush

Severe-Valsaint

Natali

Riner

(2019). Student Naval Aviation Extended Reality Device Capability Evaluation. Naval Air Warfare Center Training Systems Division. Orlando, FL.

11.

McCracken

J. H.

Aldrich

T. B.

(1984). Analyses of selected LHX mission functions: Implications for operator workload and system automation goals. US Army Research Institute Aviation Research and Development Activity. Fort Rucker, AL.

12.

McHugh

M. L

. (2012). Interrater reliability: the kappa statistic. Biochemia medica, 22(3), 276–282.

13.

Miller

R. B.

(1954). Psychological Considerations in the Designs of Training Equipment. Wright Air Development Center, Wright Patterson AFB, OH.

14.

Pedregosa

Varoquaux

Gramfort

Michel

Thirion

Grisel

Blondel

Prettenhofer

Weiss

Dubourg

Vanderplas

Passos

Cournapeau

Brucher

Perrot

Duchesnay

. (2011). Scikit-learn: Machine learning in Python. The Journal of Machine Learning Research, 12, 2825-2830.

15.

Salas

Bowers

C. A.

Rhodenizer

(1998). It is not how much you have but how you use it: Toward a rational use of simulation to support aviation training. The international journal of aviation psychology, 8(3), 197-208. https://doi.org/10.1207/s15327108ijap0803_2

16.

Schricker

Franceschini

Johnson

(2001). Fidelity evaluation framework. In Proceedings of the IEEE 34th Annual Simulation Symposium, Seattle, WA.

17.

Shou

Sellbom

Chen

H. F.

(2021). Fundamentals of Measurement in Clinical Psychology. In Reference Module in Neuroscience and Biobehavioral Psychology. Elsevier. https://doi.org/10.1016/B978-0-12-8186978.00110-2

18.

Stevens

S. S.

(1946). On the Theory of Scales of Measurement. Science, 103, 677-680.

19.

Talbot

Walker

(1996). Behavioral fidelity requirements analysis, In Proceedings of the Interservice/Industry Training, Simulation and Education Conference, Orlando, FL.

20.

Backlund

Ding

Ning

(2019). Fidelity in simulation-based serious games. IEEE Transactions on Learning Technologies, 13(2), 340-353.