Towards systematic reviews that inform health care management and policy-making

Abstract

Objectives

To identify ways to improve the usefulness of systematic reviews for health care managers and policy-makers that could then be evaluated prospectively.

Methods

We systematically reviewed studies of decision-making by health care managers and policy-makers, conducted interviews with a purposive sample of them in Canada and the United Kingdom (n=29), and reviewed the websites of research funders, producers/purveyors of research, and journals that include them among their target audiences (n=45).

Results

Our systematic review identified that factors such as interactions between researchers and health care policy-makers and timing/timeliness appear to increase the prospects for research use among policy-makers. Our interviews with health care managers and policy-makers suggest that they would benefit from having information that is relevant for decisions highlighted for them (e.g. contextual factors that affect a review's local applicability and information about the benefits, harms/risks and costs of interventions) and having reviews presented in a way that allows for rapid scanning for relevance and then graded entry (such as one page of take-home messages, a three-page executive summary and a 25-page report). Managers and policy-makers have mixed views about the helpfulness of recommendations. Our analysis of websites found that contextual factors were rarely highlighted, recommendations were often provided and graded entry formats were rarely used.

Conclusions

Researchers could help to ensure that the future flow of systematic reviews will better inform health care management and policy-making by involving health care managers and policy-makers in their production and better highlighting information that is relevant for decisions. Research funders could help to ensure that the global stock of systematic reviews will better inform health care management and policy-making by supporting and evaluating local adaptation processes such as developing and making available online more user-friendly ‘front ends’ for potentially relevant systematic reviews.

Get full access to this article

View all access options for this article.

References

Egger

, Davey Smith

, Altman

eds. Systematic Reviews in Health Care: Meta-Analysis in Context 2nd edn. London: BMJ Books, 2001

Dixon-Woods

, Agarwal

, Jones

, Young

, Sutton

. Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Pol 2005; 10: 45–53

Mays

, Popay

, Pope

. Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Pol 2005; 10(Suppl 1):6–20

Greenhalgh

, Robert

, Macfarlane

, Bate

, Kyriakidou

. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 2004; 82: 581–629

Pawson

, Greenhalgh

, Harvey

, Walshe

. Using realist methods to produce syntheses of evidence for use by managers and policy makers. J Health Serv Res Pol 2005; 10(Suppl 1):21–34

Davies

HTO

, Nutley

, Smith

. What Works? Evidence-based Policy and Practice in Public Services. Bristol: The Policy Press, 2000

Walshe

, Rundall

. Evidence-based management: from theory to practice in health care. Milbank Q 2001; 79: 429–457

Tranfield

, Denyer

, Smart

. Towards a methodology for developing evidence-informed management knowledge by means of systematic review. Br J Manage 2003; 14: 207–22

Davies

. Is evidence-based government possible? Paper Presented at the Fourth Annual Campbell Collaboration Colloquium. Washington, DC, 19 February 2004

10.

Lavis

, Becerra Posada

, Haines

, Osei

. Use of research to inform public policymaking. Lancet 2004; 364: 1615–21

11.

Lavis

, Davies

HTO

, Oxman

, Denis

J-L

, Golden-Biddle

, Ferlie

. Towards systematic reviews that inform healthcare management and policymaking. Report Submitted to the Canadian Health Services Research Foundation and the NHS Service Delivery and Organization R&D Programme, November 2004

12.

Innvaer

, Vist

, Trommald

, Oxman

. Health policymakers, perceptions of their use of evidence: a systematic review. J Health Serv Res Pol 2002; 7: 239–44

13.

Florio

, DeMartini

. The use of information by policymakers at the local community level. Knowledge 1993; 15: 106–23

14.

Rudat

. Evaluation of the Effective Health Care Bulletin. London: Office for Public Management, 1998; 1–46

15.

Harries

, Elliott

, Higgins

. Evidence-based policy-making in the NHS: exploring the interface between research and the commissioning process. J Public Health Med 1999; 2: 29–36

16.

Elliott

, Popay

. How are policymakers using evidence? Models of research utilization and local NHS policy making. J Epidemiol Community Health 2000; 54: 461–8

17.

McNeece

, DiNitto

, Johnson

. The utility of evaluation research for administrative decision-making. Admin Social Work 1983; 7: 77–87

18.

Ibbotson

, Long

, Sheldon

, Mason

. An initial evaluation of Effective Health Care bulletins as instruments of effective dissemination. J Manage Med 1993; 7: 48–57

19.

Glaser

, Taylor

. Factors influencing the success of applied research. Amer Psychol 1973; 28: 140–6

20.

Burns

, Charlwood

, Darling

. Better Information, Better Outcomes? The Use of Health Technology Assessment and Clinical Effectiveness Data in Health Care Purchasing Decisions in the United Kingdom and the United States. New York: Milbank Memorial Fund, 2000; 1–26

21.

Patton

, Grimes

, Guthrie

, Brennan

, French

, Blyth

. In search of impact: an analysis of the utilization of federal health evaluation research In: Weiss

, ed. Using Social Research in Public Policymaking. Toronto: Lexington Books, 1977; 141–64

22.

Sunesson

, Nilsson

. Explaining research utilization: beyond ‘functions’. Knowledge: Creation, Diffusion, Utilization 1989; 10: 140–55

23.

Trostle

, Bronfman

, Langer

. How do researchers influence decision-makers? Case studies of Mexican policies. Health Pol Plan 1999; 14: 103–14

24.

Lavis

, Ross

, Hurley

. Examining the role of health services research in public policymaking. Milbank Q 2002; 80: 125–54

25.

Caplan

, Morrison

, Stambaugh

. The Use of Social Science Knowledge in Policy Decisions at The National Level: A Report to Respondents. Ann Arbor, MI: The University of Michigan, 1975; 1–63

26.

Caplan

. A minimal set of conditions necessary for the utilization of social science knowledge in policy formulation at the national level In: Weiss

, ed Using Social Research in Public Policy Making. Toronto: Lexington Books, 1977; 183–98

27.

Van de Vall

, Bolas

. Using social policy research for reducing social problems: an empirical analysis of structure and functions. J Appl Behav Sci 1982; 18: 49–67

28.

Boyer

, Langbein

. Factors influencing the use of health evaluation research in congress. Eval Rev 1991; 15: 507–32

29.

Ross

. The use of economic evaluation in health care: Australian decision-makers, perceptions. Health Pol 1995; 31: 103–10

30.

, Rich

. Explaining the use of information in public policymaking. Knowledge and policy. Int J Knowledge Transfer Util 1996; 9: 3–35

31.

. Explaining the impact of policy information on policy-making. Knowledge and policy. Int J Knowledge Transfer Util 1997; 10: 22–55

32.

Soumerai

, Ross-Degnan

, Fortress

, Walser

. Determinants of change in Medicaid pharmaceutical cost sharing: does evidence affect policy? Milbank Q 1997; 75: 11–34

33.

Canadian Health Services Research Foundation. Policy Synthesis: Program Overview. Ottawa: Canadian Health Services Research Foundation, 2004; www.chsrf.ca/funding_opportunities/commissioned_research/polisyn/descrip_e.php (accessed 31 October 2004)

34.

Canadian Health Services Research Foundation. Reader-Friendly Writing - 1:3:25. Ottawa: Canadian Health Services Research Foundation, 2001; http://www.chsrf.ca/knowledge_transfer/pdf/cn-1325_e.pdf. (accessed 5 November 2004)

35.

Ad Hoc Working Group for Critical Appraisal of the Medical Literature (R B Haynes, Chairman). A proposal for more informative abstracts of clinical articles. Ann Intern Med 1987; 106: 598–604

36.

Haynes

, Mulrow

, Huth

, Altman

, Gardner

. More informative abstracts revisited: a progress report. Ann Intern Med 1990; 113: 69–76

37.

Oxman

, Guyatt

. A consumers, guide to subgroup analyses. Ann Inter Med 1992; 116: 78–84

38.

GRADE Working Group. Grading quality of evidence and strength of recommendations. BMJ 2004; 328: 1490

39.

Black

, Carter

. Public accountability: one rule for practitioners, one for scientists? J Health Serv Res Pol 2001; 6: 131–2

40.

Oxman

, Cook

, Guyatt

. Users, guides: VI: How to use an overview. JAMA 1994; 272: 1367–71