Lessons learned in evaluating system interdependencies using qualitative methods

Abstract

This article shares lessons learned while evaluating the system interdependencies for a clinical and translational research centre (CTR). It explores the methodological challenge of discussing system concepts (e.g., cascading failures, feedback loops, and reflex arcs) with layperson participants during evaluation interviews. The article discusses the iterative process of moving from structured interview approaches in favor of an open narrative approach for data collection and lessons learned.

Keywords

evaluation interview methods qualitative methods system interdependencies systems concepts systems evaluation

Get full access to this article

View all access options for this article.

References

AEA . (2018). Principles for effective use of systems thinking in evaluation. Systems in Evaluation TIG, American Evaluation Association.

Arnold

R. D.

Wade

J. P.

(2015). A definition of systems thinking: A systems approach. Procedia Computer Science, 44, 669-678. https://doi.org/10.1016/j.procs.2015.03.050.

Austin

Sutton

(2014). Qualitative research: Getting started. The Canadian Journal of Hospital Pharmacy, 67(6), 436-40. https://doi.org/10.4212/cjhp.v67i6.1406.

Bagodi

Mahanty

(2013). Double-loop learning in the Indian two‐wheeler service sector. The Learning Organization, 20(4/5), 264-278. https://doi.org/10.1108/TLO-04-2012-0029.

Bolton

M. K.

(1999). The role of coaching in student teams: A “just-in-time” approach to learning. Journal of Management Education, 23(3), 233-250. https://doi.org/10.1177/105256299902300302.

Braun

Clarke

(2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa.

Burman

C. J.

Moerschell

Mamabolo

Aphane

Delobelle

(2015). Re-imaging decision making: Addressing a discrete social drive of HIV/AIDS through the lens of complexity science. African Journal of AIDS Research, 14(1), 75-84. https://doi.org/10.1191/1478088706qp063oa.

Byrne

Callaghan

(2013). Complexity theory and the social sciences: The state of the art. Routledge.

Cabrera

Trochim

W. M.

(2006). A theory of systems evaluation. Retrieved from: https://ecommons.cornell.edu/bitstream/handle/1813/3094/Theory06.pdf?sequence=1&isAllowed=y.

10.

Capra

Luisi

P. L.

(2014). The systems view of life: A unifying vision. Cambridge University Press. https://doi.org/10.1017/CBO9780511895555.

11.

Checkland

(1999). Systems thinking. Rethinking management information systems. In Currie

Gallers

(Eds), Rethinking: Management information systems: An interdisciplinary perspective (pp. 44-56). Oxford Printing.

12.

Checkland

(2000). Soft systems methodology: A thirty-year retrospective. Systems Research and Behavioral Science, 17(S1), S11-S58. https://doi.org/10.1002/1099-1743(200011)17:1+<::AID-SRES374>3.0.CO;2-O.

13.

Chen

Hailey

Wang

(2014). A review of data quality assessment methods for public health information systems. International Journal of Environmental Research and Public Health, 11(5), 5170-207. https://doi.org/10.3390/ijerph110505170.

14.

Coar

Sim

(2009). Interviewing one’s peers: Methodological issues in a study of health professionals. Scandinavian Journal of Primary Health Care, 24(4), 251-256. https://doi.org/10.1080/02813430601008479.

15.

Coşkun

Akande

Renger

(2012). Using root cause analysis for evaluating program improvement. Evaluation Journal of Australasia, 12(2), 4-14. https://doi.org/10.1177/1035719X1201200202.

16.

Creswell

J. W.

Poth

C. N.

(2016). Qualitative inquiry and research design: Choosing among five approaches (4th ed.). SAGE.

17.

Davidson

E. J.

(2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. SAGE.

18.

Davis

A. C.

Stroink

M. L.

(2016). The relationship between systems thinking and the new ecological paradigm. Systems Research and Behavioral Science, 33(4), 575-586. https://doi.org/10.1002/sres.2371.

19.

Doyle

(1997). Heard any really good stories lately? A critique of the critics of narrative in educational research. Teaching and Teacher Education, 13(1), 93-99. https://doi.org/10.1016/S0742-051X(96)00039-X.

20.

DiCicco‐Bloom

Crabtree

B. F.

(2006). The qualitative research interview. Medical Education, 40(4), 314-321. https://doi.org/10.1111/j.1365-2929.2006.02418.x.

21.

Eoyang

G. H.

Holladay

R. J.

(2020). Adaptive action: Leveraging uncertainty in your organization. Stanford University Press. https://doi.org/10.1515/9780804785402.

22.

Fairclough

(2010). Critical discourse analysis: The critical study of language (2nd ed.). Routledge.

23.

Forss

Schwartz

(2011). Introduction. In Forss

Marra

Schwartz

(Eds), Evaluating the complex: Attribution, contribution, and beyond (Vol. 1, pp. 1–32). Transaction Publishers.

24.

Ganga

Scott

(2006). Cultural and the issue of positionality in qualitative migration research: Moving “across” and moving “along”: Researcher-participant divides. Cultural and the issue of positionality in qualitative migration research: Moving. Forum Qualitative Social Research, 7(3). https://doi.org/10.17169/fqs-7.3.134.

25.

Gee

J. P.

(2004). An introduction to discourse analysis: Theory and method (2nd ed.). Routledge. https://doi.org/10.4324/9780203005675.

26.

Gesch-Karamanlidis

(2015). Reflecting on novice qualitative interviewer mistakes. Qualitative Report, 20(5), 712-726. https://doi.org/10.46743/2160-3715/2015.2145.

27.

Goff

P. A.

Mentovich

Martin

K. D.

(2013). (The need for) A model of translational mind science justice research. Journal of Social and Political Psychology, 1, 385-399. https://doi.org/10.5964/jspp.v1i1.75.

28.

Gery

(1991). Electronic performance support systems: How and why to remake the through the strategic application of technology. Weingarten Publications

29.

Gray

D. E.

(2019). Doing research in the real world (3rd ed.). SAGE.

30.

Hamstra

(2017). Complexity storytelling: The science of complexity within the art of communication. Emergence: Complexity and Organization, 19(1), 1-6. https://doi.org/10.17357.076ac0318796a9ab519c0a5463154e88.

31.

Hill

C. E.

(2014). Psychotherapy research: Foundations, process, and outcomes. In Gelo

Pritz

Rieken

(Eds.), Consensual qualitative research (CQR): Methods for conducting psychotherapy research (pp. 485-499). Springer. https://doi.org/10.1007/978-3-7091-1382-0_23.

32.

Hill

C. E.

Thompson

B. J.

Williams

E. N.

(1997). A guide to conducting consensual qualitative research. The Counseling Psychologist, 25(4), 517-572. https://doi.org/10.1177/0011000097254001.

33.

Hummelbrunner

(2011). Systems thinking and evaluation. Evaluation, 17(4), 395-403. https://doi.org/10.1177/1356389011421935.

34.

Holley

Colyar

(2012). Under construction: How narrative elements shape qualitative research. Theory Into Practice, 51(2), 114-121. https://doi.org/10.1080/00405841.2012.662866.

35.

Hollway

Jefferson

(2008). The free association narrative interview method. In Given

L.M.

(Ed), The SAGE encyclopedia of qualitative research methods. 296–315). SAGE.

36.

Ison

R. L.

(2008). Systems thinking and practice for action research. In Reason

R.W.

Bradbury

(Eds), The SAGE handbook of action research, participative inquiry and practice. 139–158). SAGE.

37.

Jamshed

(2014). Qualitative research method-interviewing and observation. Journal of Basic and Clinical Pharmacy, 5(4), 87-8. https://doi.org/10.4103/0976-0105.141942.

38.

Jackson

B. A.

Faith

K. S.

Willis

H. H.

(2012). Evaluating the reliability of emergency response systems for large-scale incident operations. RAND Corporation. https://doi.org/PMC4945240.

39.

Jones

M. J.

(2001). Just-in-time training. Advances in Developing Human Resources, 3(4), 480-487. https://doi.org/10.1177/15234220122238409.

40.

Kastens

K. A.

Manduca

C. A.

Cervato

Frodeman

Goodwin

Liben

L. S.

Mogk

D. W.

Spangler

T. C.

Stillings

N. A.

Titus

(2011). How geoscientists think and learn. Eos, Transactions American Geophysical Union, 90(31), 265-266. https://doi.org/10.1029/2009EO310001.

41.

Koskinen

K. U.

(2011). Project-based companies as learning organizations: Systems theory perspective. International Journal of Project Organization and Management, 3(1), 91-106. https://doi.org/10.1504/IJPOM.2011.038866.

42.

King

McKegg

Oakden

Wehipeihana

(2013) (In this issue). Evaluative rubrics: A method for surfacing values and improving the credibility of evaluation. Journal of Multidisciplinary Evaluation, 9(21), 11-20. https://www.alnap.org/system/files/content/resource/files/main/374-1266-1-pb.pdf.

43.

Kurtz

C. F.

Snowden

(2007). Bramble bushes in a thicket: Narrative and the intangibles of learning networks. In Gibbert

Durand

(Eds), Strategic networks: Learning to compete. 121–137. Blackwell Publishing.

44.

Macias

J. A.

Castells

(2001). An authoring tool for building adaptive learning guidance systems on the web. Unpublished manuscript.

45.

Madill

Gough

(2008). Qualitative research and its place in psychological science. Psychological Methods, 13(3), 254-271. https://doi.org/10.1037/a0013220.

46.

McCormack

Ambler

Martin

Waite

Wilson

(2016). Narrative-based evaluation demonstrates the value of a higher education professional learning network. Studies in Educational Evaluation, 50, 79-87. https://doi.org/10.1016/j.stueduc.2016.07.003.

47.

McKenney

Reeves

(2014) (In press). Methods of evaluation and reflection in design research. Zeitschrift für Berufs- und Wirtschaftspädagogiek, 27, 141-153. https://ris.utwente.nl/ws/files/7007413/McKenneyReevesZBW2014-PrePrint.pdf.

48.

Meadows

D. H.

(2008). Thinking in systems: A primer. Chelsea Green Publishing.

49.

Midgley

Boyd

Geerling

Gregory

Murray

Walsh

Kagan

(2002). Systemic evaluation: Three flexible and inclusive approaches evaluation. Revista Gerencia y Políticas de Salud, 1(2), 6-16. https://www.redalyc.org/pdf/545/54510202.pdf.

50.

Moen

(2006). Reflections on the narrative research approach. International Journal of Qualitative Methods, 5(4), 56-69. https://doi.org/10.1177/160940690600500405.

51.

Monat

J. P.

Gannon

T. F.

(2015). What is systems thinking? A review of selected literature plus recommendations. American Journal of Systems Science, 4(1), 11-26. https://doi.org/10.5923/j.ajss.20150401.02.

52.

Morrow

S. L.

(2005). Quality and trustworthiness in qualitative research in counseling psychology. Journal of Counseling Psychology, 52(2), 250-260. https://doi.org/10.1037/0022-0167.52.2.250.

53.

Murray

(1998). Authoring knowledge-based tutors: Tools for content, instructional strategy, student model, and interface design. Journal of the Learning Sciences, 7(1), 5-64. https://doi.org/10.1207/s15327809jls0701_2.

54.

Paliszkiewicz

Koohang

Nord

J. H.

(2014). Management trust, organizational trust, and organizational performance: Empirical validation of an instrument. Online Journal of Applied Knowledge Management, 2(1), 28–39.

55.

Patton

M. Q.

(2008). Utilization-focused evaluation. SAGE.

56.

Patton

M. Q.

(2010). Developmental evaluation. Evaluation Practice, 15(3), 311–319.

57.

Patton

M. Q.

(2013). The future of evaluation in society: Top ten trends plus one. In Donaldson

S. I.

(Ed), The future of evaluation in society: A tribute to Michael Scriven. 45–62. Information Age Publishing.

58.

Petit

Verner

Brannegan

Buehring

Dickinson

Guziel

HaffendenPhillips

R.J.

Peerenboom

(2015). Analysis of critical infrastructure dependencies and interdependencies. Argonne National Laboratory. https://publications.anl.gov/anlpubs/2015/06/111906.pdf.

59.

Pickering

Garrod

(2007). Do people use language production to make predictions during comprehension? Trends in cognitive sciences, 11(3), 105-10. https://doi.org/10.1016/j.tics.2006.12.002.

60.

Puranam

Raveendran

Knudsen

(2012). Organization design: The epistemic interdependence perspective. Academy of Management Review, 37(3), 419-440. https://doi.org/10.5465/amr.2010.0535.

61.

Rapley

T. J.

(2001). The art(fulness) of open-ended interviewing: Some considerations on analysing interviews. Qualitative Research, 1(3), 303-323. https://doi.org/10.1177/146879410100100303.

62.

Raybould

(1990). Solving human performance problems with computers. Performance & Instruction, 29(11), 4-14. https://doi.org/10.1002/pfi.4160291004.

63.

Reynolds

Gates

Hummelbrunner

Marra

Williams

(2016). Towards systemic evaluation. Systems Research and Behavioral Science, 33(5), 662-673. https://doi.org/10.1002/sres.2423.

64.

Sainsbury

E. J.

Walker

(2007). Same words, different meanings: Learning to talk the scientific language of pharmacy. Sydney University Press.

65.

Renger

(2015). System evaluation theory (SET): A practical framework for evaluators to meet the challenges of system evaluation. Evaluation Journal of Australasia, 15(4), 16-28. https://doi.org/10.1177/1035719x1501500403.

66.

Renger

Keogh

Hawkins

Foltysova

Souvannasacd

(2018). Reworks: A robust system efficiency measure. Evaluation Journal of Australasia, 18(3), 183-191. https://doi.org/10.1177/1035719x18796611.

67.

Renger

Basson

M. D.

Hart

Van Eck

Souvannasacd

Renger

Foltysova

(2020). Lessons learned in evaluating the infrastructure of a Centre for Translational Research. Evaluation Journal of Australasia, 20(1), 6-22. https://doi.org/10.1177/1035719x20909910.

68.

Senge

P. M.

(2006). The fifth discipline: The art and practice of the learning organization. Currency.

69.

Snowden

(2011). Naturalizing sensemaking. In Mosier

K. L.

Fischer

U. M.

(Eds), Informed by knowledge: Expert performance in complex situations. 237-248. Psychology Press.

70.

Susarla

Adcock

Van Eck

Moreno

Graesser

A. C.

(2003). Development and evaluation of a lesson authoring tool for tutors. In Aleven

Hoppe

Kay

Mizoguchi

Pain

Verdejo

Yacef

(Eds.), Supplemental proceedings. 378–387. University of Sydney School of Information Technologies

71.

Sweller

(1994). Cognitive load theory, learning difficulty, and instructional design. Learning andinstruction, 4(4), 295-312. https://doi.org/10.1016/0959-4752(94)90003-5.

72.

Taut

Brauns

(2003). Resistance to evaluation: A psychological perspective. Evaluation, 9(3), 247-264. https://doi.org/10.1177/13563890030093002.

73.

Taylor‐Powell

Boyd

H. H.

(2008). Evaluation capacity building in complex organizations. New Directions for Evaluation, 120, 55-69. https://doi.org/10.1002/ev.276.

74.

Toole

Heift

(2002). The tutor assistant: An authoring system for a web-based intelligent language tutor. Computer Assisted Language Learning, 15(4), 373-386. https://doi.org/10.1076/call.15.4.373.8266.

75.

van Merriënboer

J. J. G.

Sweller

(2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17(2), 147-177. https://doi.org/10.1007/s10648-005-3951-0.

76.

Vindrola-Padros

Pape

Utley

Fulop

N. J.

(2017). The role of embedded research in quality improvement: A narrative review. BMJ Quality & Safety, 26(1), 70-80. https://doi.org/10.1136/bmjqs-2015-004877.

77.

Williams

(2010, November 12). Fitting the key to the lock: Matching systems methods to evaluation questions. Paper presented at the 24th Annual Conference of the American Evaluation Association, San Antonio, TX.

78.

Williams

Hummelbrunner

(2020). Systems concepts in action. Stanford University Press.

79.

Williams

Imam

(2007). Systems concepts in evaluation: An expert anthology. EdgePress.

80.

Weiss

R. S.

(1995). Learning from strangers: The art and method of qualitative interview studies, Simon & Schuster

81.

Yarbrough

D.B.

Shula

L.M.

Hopson

R.K.

Caruthers

F.A.

(2010). The program evaluation standards: A guide for evaluators and evaluation users (3rd. ed.). Corwin Press.

82.

Zerhouni

E. A.

Alving

(2006). Clinical and translational science awards: A framework for a national research agenda. Translational Research, 148(1), 4-5.