Understanding Human Readiness Levels

Abstract

The Human Readiness Levels (HRL) scale is a simple nine-level scale developed as an adjunct to complement and supplement the existing Technology Readiness Levels (TRL) scale widely used across government agencies and industry. A multi-agency working group consisting of nearly 30 members representing the broader human systems integration (HSI) community throughout the Department of Defense (DOD), Department of Energy (DOE), other federal agencies, industry, and academia was established in August 2019. The working group’s charter was to mature the HRL scale and evaluate its utility, reliability, and validity for implementation in the systems acquisition lifecycle. Toward that end, the working group examined applicability of the HRL scale for a range of scenarios. This panel will discuss outcomes from the working group’s activities regarding HRL scale structure and usage.

Get full access to this article

View all access options for this article.

References

Acosta

(2010, May). Human readiness levels: Implementing HSI – Connecting some dots. Panel discussion presented at the 81st Annual Scientific Meeting of the Aerospace Medical Association, Phoenix, Arizona.

Brooke

(1986). System usability scale (SUS): A quick-and-dirty method of system evaluation user information. Reading, UK: Digital Equipment Co Ltd.

Del Giudice

Hale

Johnston

. (2015). Development of a scale for quantifying a system’s human factors readiness. Procedia Manufacturing, 3, 5192-5198.

Demaree

R. G.

James

L. R.

Wolf

(1993). Rwg: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306-309.

Endsley

(2014). Human system integration: Challenges and opportunities. PowerPoint slides retrieved from http://www.acq.osd.mil

Garcia

Ganey

Wilbert

(2017). Human readiness assessment: A multivariate approach. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 61, 106-109.

Hallgren

(2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8, 23-34.

Kosnik

Acosta

(2010). HSI as a method for embedding human-centered design in Air Force rapid acquisition programs. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 54, 2165-2168.

Kosnik

O’Neill

Zimmerlin

(2018). Human systems integration risk management tool. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 62, 325-329.

10.

Lacson

F. C.

Risser

M. R.

Gwynne

J. W.

Kosnik

W. D.

(2017). The human systems integration framework. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 61, 1720-1724.

11.

Mankins

J. C.

(2009). Technology readiness assessments: A retrospective. Acta Astronautica, 65, 1216-1223.

12.

Miller

Thomas

Rusnock

(2016). Extending system readiness levels to assess and communicate human readiness. Systems Engineering, 19, 146-157.

13.

Olsen

Shorrock

(2010). Evaluation off the HFACS-ADF safety classification system: Inter-coder consensus and intra-coder consistency. Accident Analysis and Prevention, 42, 437-444.

14.

O’Neil

M. P.

(2014). Development of a human systems integration framework for Coast Guard acquisition (Unpublished master’s thesis). Naval Postgraduate School: Monterey, CA.

15.

Phillips

E. L.

(2010). The development and initial evaluation of the human readiness level framework (Unpublished master’s thesis). Naval Postgraduate School: Monterey, CA.

16.

Phillips

(2015). DOD HFE TAG: Human readiness level (HRL) working group. PowerPoint slides from william.kosnik.1@us.af.mil

17.

Phillips

Stohr

Pharmer

Seely

(2015). HSI progress & risk specification tool (HPRST). Paper presented at DOD HFE TAG 69, Orlando, Florida.

18.

Sadin

S. R.

Povinelli

F. P.

Rosen

(1989). The NASA technology push towards future space mission systems. Acta Astronautica, 20, 73-77.

19.

See

J. E.

Craft

Morris

J. D.

(2019). Human readiness levels in the systems engineering process at Sandia National Laboratories (Report SAND2019-312). Albuquerque, NM: Sandia National Laboratories.