Hierarchy of Evidence: An Appraisal Tool for Weighting the Evidence in Healthcare Design Research Based on Internal Validity

Abstract

Purpose:

This inquiry aims to develop an appraisal tool to offer greater granularity in weighing evidence in the field of healthcare design research.

Background:

In evidence-based design (EBD), the goal is to implement interventions that result in a meaningful and optimal effect based on current best evidence. Although multiple appraisal tools (many adopted from medical disciplines) have been instrumental in evaluating studies in the field of healthcare design research to identify the best evidence, they do not necessarily consider the unique contexts of healthcare design research, and methodologies appropriate to the field.

Methods:

Five basic types of studies are ranked based on the level of confidence that they offer regarding the estimate of an effect: (1) meta-analysis studies, (2) causal studies, (3) correlational studies, (4) descriptive studies, and (5) anecdotal evidence. Causal studies are further divided into four levels based on the interaction of two factors: (1) type of intervention and (2) groups’ equivalency and extraneous variable control.

Results:

An eight-level hierarchy of evidence for healthcare design research is proposed that is expected to improve upon previous hierarchies in three major ways: (a) including research methods that are more relevant to healthcare design research, (b) enhancing evaluation accuracy and reliability by providing a clearer definition of studies based on their key components rather than using study labels alone, and (c) distinguishing different levels of evidence, particularly in causal studies.

Conclusions:

The proposed appraisal tool is developed specifically for EBD by reflecting on the unique context of healthcare design research and practice.

Keywords

evidence-based design appraisal tool hierarchy of evidence quality of evidence weight of evidence credibility

Get full access to this article

View all access options for this article.

References

Alpi

K. M.

Evans

J. J.

(2019). Distinguishing case study as a research method from case reports as a publication type. Journal of the Medical Library Association, 107(1), 1–5. https://doi.org/10.5195/jmla.2019.615

Andrade

(2018). Internal, external, and ecological validity in research design, conduct, and evaluation. Indian Journal of Psychological Medicine, 40(5), 498–499. https://doi.org/10.4103/IJPSYM.IJPSYM_334_18

Armola

R. R.

Bourgault

A. M.

Halm

M. A.

Board

R. M.

Bucher

Harrington

Heafey

C. A.

Lee

Shellner

P. K.

Medina

. (2009). AACN levels of evidence: What’s new? Critical Care Nurse, 29(4), 70–73. https://doi.org/10.4037/ccn2009969

Aronson

J. K.

(2003). Anecdotes as evidence. British Medical Journal, 326(7403), 1346.

Blumberg

R. S.

Dittel

Hafler

von Herrath

Nestle

F. O.

(2012). Unraveling the autoimmune translational research process layer by layer. Nature Medicine, 18(1), 35–41. https://doi.org/10.1038/nm.2632

Burns

P. B.

Rohrich

R. J.

Chung

K. C.

(2011). The levels of evidence and their role in evidence-based medicine. Plastic and Reconstructive Surgery, 128(1), 305–310. https://doi.org/10.1097/PRS.0b013e318219c171

Craig

Cooper

Gunnell

Haw

Lawson

Macintyre

Ogilvie

Petticrew

Reeves

Sutton

Thompson

. (2012). Using natural experiments to evaluate population health interventions: New Medical Research Council guidance. Journal of Epidemiology and Community Health, 66(12), 1182–1186. https://doi.org/10.1136/jech-2011-200375

Creswell

J. W.

Creswell

J. D.

(2017). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). Sage.

Crowe

Sheppard

(2011). A review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed. Journal of Clinical Epidemiology, 64(1), 79–89. https://doi.org/10.1016/j.jclinepi.2010.02.008

10.

Crowe

Cresswell

Robertson

Huby

Avery

Sheikh

(2011). The case study approach. BMC Medical Research Methodology, 11(1), 100. https://doi.org/10.1186/1471-2288-11-100

11.

Deeks

Higgins

Altman

(2019). Chapter 10: Analysing data and undertaking meta-analyses. In Higgins

Thomas

Chandler

Cumpston

Page

Welch

(Eds.), Cochrane handbook for systematic reviews of interventions (2nd ed.). John Wiley & Sons.

12.

Dinardo

(2010). Natural experiments and quasi-natural experiments. In Durlauf

S. N.

Blume

L. E.

(Eds.), Microeconometrics (pp. 139–153). Palgrave Macmillan UK. https://doi.org/10.1057/9780230280816_18

13.

Evans

(2003). Hierarchy of evidence: A framework for ranking evidence evaluating healthcare interventions. Journal of Clinical Nursing, 12(1), 77–84. https://doi.org/10.1046/j.1365-2702.2003.00662.x

14.

Gerring

(2004). What is a case study and what is it good for? American Political Science Review, 98(2), 341–354.

15.

Glynn

(2006). A critical appraisal tool for library and information research. Library Hi Tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154

16.

Guyatt

G. H.

Oxman

A. D.

Vist

G. E.

Kunz

Falck-Ytter

Alonso-Coello

Schünemann

H. J.

(2008). GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. British Medical Journal, 336(7650), 924–926. https://doi.org/10.1136/bmj.39489.470347.AD

17.

Hamilton

D. K.

(2011). What constitutes best practice in healthcare design? Health Environments Research & Design Journal, 4(2), 121–126. https://doi.org/10.1177/193758671100400210

18.

Jansen

C. W. S.

(1999). Case reports and case studies: A discussion of theory and methodology in pocket format. Journal of Hand Therapy, 12(3), 230–232. https://doi.org/10.1016/S0894-1130(99)80051-9

19.

Johnson

Christensen

L. B.

(2017). Educational research: Quantitative, qualitative, and mixed approaches (6th ed.). Sage.

20.

Linden

(2017). Challenges to validity in single-group interrupted time series analysis. Journal of Evaluation in Clinical Practice, 23(2), 413–418. https://doi.org/10.1111/jep.12638

21.

Lorusso

L. N.

Bosch

S. J.

(2018). Impact of multisensory environments on behavior for people with dementia: A systematic literature review. The Gerontologist, 58(3), e168–e179. https://doi.org/10.1093/geront/gnw168

22.

Marquardt

Bueter

Motzek

(2014). Impact of the design of the built environment on people with dementia: An evidence-based review. Health Environments Research & Design Journal, 8(1), 127–157. https://doi.org/10.1177/193758671400800111

23.

Martin

D. W.

(2008). Doing psychology experiments (7th ed.). Thomson/Wadsworth.

24.

Maxwell

J. A.

(2004). Causal explanation, qualitative research, and scientific inquiry in education. Educational Researcher, 33(2), 3–11. https://doi.org/10.3102/0013189X033002003

25.

Murad

M. H.

Katabi

Benkhadra

Montori

V. M.

(2018). External validity, generalisability, applicability and directness: A brief primer. BMJ Evidence-Based Medicine, 23(1), 17.

26.

Naci

(2017). What to do (or not to do) when randomization is not possible. The Journal of Heart and Lung Transplantation, 36(11), 1174–1177. https://doi.org/10.1016/j.healun.2017.07.008

27.

Nielsen

M. E.

Reilly

P. L.

(1985). A guide to understanding and evaluating research articles. Gifted Child Quarterly, 29(2), 90–92. https://doi.org/10.1177/001698628502900210

28.

Pati

(2011). A framework for evaluating evidence in evidence-based design. Health Environments Research & Design Journal, 4(3), 50–71. https://doi.org/10.1177/193758671100400305

29.

Pati

Freier

O’Boyle

Amor

Valipoor

(2016). The impact of simulated nature on patient outcomes: A study of photographic sky compositions. Health Environments Research & Design Journal, 9(2), 36–51. https://doi.org/10.1177/1937586715595505

30.

Pati

Valipoor

Lorusso

Mihandoust

Jamshidi

Rane

Kazem-Zadeh

(2019). The impact of the built environment on patient falls in hospital rooms: An integrative review. Journal of Patient Safety. Publish Ahead of Print. https://doi.org/10.1097/PTS.0000000000000613

31.

Peer-reviewed literature [training material and manuals]. (n.d.). Retrieved June 21, 2022, from https://www.nlm.nih.gov/nichsr/stats_tutorial/section3/mod6_peer.html

32.

Phillips

Ball

Sackett

Badenoch

Straus

Haynes

… Howick

. (2009, March). Oxford Centre for evidence-based medicine: Levels of evidence (March 2009) [Web page]. Center for Evidence-Based Medicine. Retrieved April 25, 2022, from https://www.cebm.ox.ac.uk/resources/levels-of-evidence/oxford-centre-for-evidence-based-medicine-levels-of-evidence-march-2009

33.

Pluye

Gagnon

M.-P.

Griffiths

Johnson-Lafleur

(2009). A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. International Journal of Nursing Studies, 46(4), 529–546. https://doi.org/10.1016/j.ijnurstu.2009.01.009

34.

Ponce

O. J.

Alvarez-Villalobos

Shah

Mohammed

Morgan

R. L.

Sultan

Falck-Ytter

Prokop

L. J.

Dahm

Mustafa

R. A.

Murad

M. H

. (2017). What does expert opinion in guidelines mean? A meta-epidemiological study. BMJ Evidence-Based Medicine, 22(5), 164–169. https://doi.org/10.1136/ebmed-2017-110798

35.

Reeves

Deeks

Higgins

Shea

Tugwell

Wells

(2019). Chapter 24: Including non-randomized studies on intervention effects. In Higgins

Thomas

Chandler

Cumpston

Page

Welch

(Eds.), Cochrane handbook for systematic reviews of interventions (2nd ed.). John Wiley & Sons.

36.

Salkind

N. J.

(Ed.). (2010). Encyclopedia of research design (Vols. 1–3). Sage.

37.

Singal

A. G.

Higgins

P. D. R.

Waljee

A. K.

(2014). A primer on effectiveness and efficacy trials. Clinical and Translational Gastroenterology, 5(1), e45. https://doi.org/10.1038/ctg.2013.13

38.

Stetler

C. B

. (2001). Evidence-based practice and the use of research: A synopsis of basic concepts & strategies to improve care. Nova Foundation.

39.

Stichler

J. F.

(2010). Weighing the evidence. Health Environments Research & Design Journal, 3(4), 3–7. https://doi.org/10.1177/193758671000300401

40.

Stichler

J. F.

Hamilton

D. K.

(2008). Evidence-based design: What is it? Health Environments Research & Design Journal, 1(2), 3–4. https://doi.org/10.1177/193758670800100201

41.

Tacq

(2011). Causality in qualitative and quantitative research. Quality & Quantity, 45(2), 263–291. https://doi.org/10.1007/s11135-009-9293-0

42.

Taylor

Hignett

(2014). Evaluating evidence: Defining levels and quality using critical appraisal mixed methods tools. Health Environments Research & Design Journal, 7(3), 144–151. https://doi.org/10.1177/193758671400700310

43.

Translational Science Spectrum. (2021). National Center for Advancing Translational Sciences. Retrieved October 30, 2022, from https://ncats.nih.gov/translation/spectrum

44.

Trochim

W. M. K.

Donnelly

J. P.

(2008). Research methods knowledge base (3. ed). Cengage Learning.

45.

Ulrich

R. S.

(1984). View through a window may influence recovery from surgery. Science, 224(4647), 420–421.

46.

Weerasuriya

Henderson-Wilson

Townsend

(2019). Accessing green spaces within a healthcare setting: A mixed studies review of barriers and facilitators. Health Environments Research & Design Journal, 12(3), 119–140. https://doi.org/10.1177/1937586718810859