In the realm of risk management, and policy-making more generally, “public engagement” is often advocated as an antidote to pathologies associated with traditional methods of policy-making, and associated deficit-model-driven communication strategies. The actual benefits of public engagement are, however, difficult to establish without thorough evaluation of specific engagement processes. Unfortunately, rigorous evaluation is difficult, and, perhaps for this reason, it has rarely been undertaken. In this paper we highlight a number of these difficulties in the light of our experiences in evaluating a major engagement initiative, namely the GM Nation? publice debate on the possible commercialization of transgenic crops, which took place in Britain in 2003. The difficulties we identify seem likely to be relevant to many, if not most, engagement evaluations. They are concerned with both theoretical/normative (how one should evaluate) and practical (how one does evaluate) issues. We suggest a number of possible solutions to these evaluation difficulties.
Get full access to this article
View all access options for this article.
References
1.
AEBC (2001) Crops on Trial. London: Agriculture and Environment Biotechnology Commission .
2.
Bloor, M. (1978) “On the Analysis of Observational Data; a Discussion of the Worth and Uses of Inductive Techniques and Respondent Validation,”Sociology12(3): 545–557 .
3.
Bryman, A. and Cramer, D. (1997) Quantitative Data Analysis. London: Routledge .
4.
Cabinet Office (2002) Risk: Improving Government's Ability to Handle Risk and Uncertainty. London: Cabinet Office Strategy Unit .
5.
Chess, C. (2000) “Evaluating Environmental Public Participation: Methodological Questions,”Journal of Environmental Planning and Management43(6): 769–784 .
6.
Chess, C. and Purcell, K. (1999) “Public Participation and the Environment: Do We Know What Works?,”Environmental Science and Technology33(16): 2685–2692 .
7.
Clarke, A. (1999) Evaluation Research: an Introduction to Principles, Methods and Practice. London: SAGE .
8.
Fiorino, D. J. (1990) “Citizen Participation and Environmental Risk: a Survey of Institutional Mechanisms,”Science, Technology, & Human Values15(2): 226–243 .
9.
Frewer, L. J. (1999) “Risk Perception, Social Trust, and Public Participation into Strategic Decision-making: Implications for Emerging Technologies,”Ambio28: 569–574 .
10.
Funtowicz, S. and Ravetz, J. (1992) “Risk Management as a Post-normal Science,”Risk Analysis12(1): 95–97 .
11.
Gaskell, G. and Bauer, M.W. (2001) Biotechnology 1996–2000: The Years of Controversy. London: The Science Museum .
12.
Glaser, B. and Strauss, A. (1967) The Discovery of Grounded Theory: Strategies for Qualitative Research. London: Weidenfeld & Nicholson .
13.
Halford, N. G. (2004) “Prospects for Genetically Modified Crops,”Annals of Applied Biology145(1): 17–24 .
14.
Horlick-Jones, T. (1998) “Meaning and Contextualisation in Risk Assessment,”Reliability Engineering and System Safety59: 79–89 .
15.
Horlick-Jones, T. (2004) “Experts in Risk?... Do They Exist?,”Health, Risk & Society6(2): 107–114 .
16.
Horlick-Jones, T. and Rosenhead, J. (2002) “Investigating Risk, Organisations and Decision Support through Action Research,”Risk Management: an International Journal4(4): 45–63 .
17.
Horlick Jones, T. , Pidgeon, N. , Walls, J. and Rowe, G. (2002) Proposal for the Evaluation of the UK Public Debate on the Possible Commercialisation of Genetically Modified Crops . Paper submitted to the GM Public Debate Steering Board, October 2002.
18.
Horlick-Jones, T., Walls, J., Rowe, G., Pidgeon, N., Poortinga, W. and O'Riordan, T. (2004) A DeliberativeFuture? An Independent Evaluation of the GM Nation? Public Debate about the Possible Commercialisation of Transgenic Crops in Britain, 2003. University of East Anglia, Programme on Understanding Risk, Working Paper 04-02.
19.
Horlick-Jones, T., Walls, J., Rowe, G., Pidgeon, N., Poortinga, W. and O'Riordan, T. (submitted) “On Evaluating the GM Nation? Public Debate about the Commercialisation of Transgenic Crops in Britain.”
20.
Horlick-Jones, T., Walls, J., Rowe, G., Pidgeon, N., Poortinga, W., O'Riordan, T., Murdock, G., Tait, J. and Bruce, A. (2003) “Memorandum Submitted by the Understanding Risk Team and Collaborators,” in House of Commons Environment, Food and Rural Affairs Committee Conduct of the GM Public Debate. Eighteenth Report of the Session 2002–03 HC 1220, Ev50–Ev56. London: The Stationery Office.
21.
House of Commons Environment, Food and Rural Affairs Committee (2003) Conduct of the GM Public Debate. Eighteenth Report of the Session 2002–03 HC 1220. London: The Stationery Office .
22.
House of Lords Select Committee on Science and Technology (2000) Science and Society Third Report. HMSO , HL Paper 38.
23.
Irwin, A. and Wynne, B. (eds) (1996) Misunderstanding Science? The Public Reconstruction of Science and Technology. Cambridge: Cambridge University Press .
24.
Jasanoff, S. (1990) The Fifth Branch: Scientific Advisors as Policy Makers. Cambridge, MA: Harvard University Press .
25.
Joss, S. (1995) “Evaluating Consensus Conferences: Necessity or Luxury?,” in S. Joss and J. Durant (eds) Public Participation in Science: The Role of Consensus Conferences in Europe, pp. 89–108. London: The Science Museum .
26.
Lord Phillips of Worth Matravers , Bridgeman, J. and Ferguson-Smith, M. (2000) The BSE Inquiry [The Phillips Report]. London: The Stationery Office .
27.
Myhr, A.I. and Traavik, T. (2003) “Genetically Modified (GM) Crops: Precautionary Science and Conflicts of Interests,”Journal of Agricultural and Environmental Ethics16(3): 227–247 .
28.
National Research Council (1996) Understanding Risk: Informing Decisions in a Democratic Society. Washington DC: National Academy Press .
29.
Nielsen, C.P. , Thierfelder K. and Robinson, S. (2003) “Consumer Preferences and Trade in Genetically Modified Foods,”Journal of Policy Modeling25(8): 777–794 .
Patton, M. (1990) Qualitative Evaluation and Research Methods, 2nd edn.London: Sage .
32.
Pidgeon, N.F. , Poortinga, W. , Rowe, G. , Horlick-Jones, T. , Walls, J. and O'Riordan, T. (2005) “Using Surveys in Public Participation Processes for Risk Decision-making: the Case of the 2003 British GM Nation? Public Debate,”Risk Analysis25(2): 467–479 .
33.
POST (2001) Open Channels: Public Dialogue in Science and Technology. Parliamentary Office of Science and Technology, Report 152.
34.
Public Debate Steering Board (PDSB) (2003) GM Nation? The Findings of the Public Debate. Department of Trade and Industry . URL: www.gmnation.org.uk.
35.
Robson, C. (2002) Real World Research, 2nd edn.Oxford: Blackwell .
36.
Rossi, P.H. , Freeman, H.E. and Lipsey, M.W. (1999) Evaluation: a Systematic Approach, 6th edn.London: Sage .
37.
Rowe, G. and Frewer, L.J. (2000) “Public Participation Methods: a Framework for Evaluation,”Science, Technology, & Human Values25(1): 3–29 .
38.
Rowe, G. and Frewer, L.J. (2004) “Evaluating Public Participation Exercises: a Research Agenda,”Science, Technology, & Human Values29(4): 512–556 .
39.
Rowe, G. and Frewer, L.J. (2005) “A Typology of Public Engagement Mechanisms,”Science, Technology, & Human Values30(2): 251–290 .
40.
Rowe, G., Horlick-Jones, T., Walls, J., Poortinga, W. and Pidgeon, N. (submitted) “Analysis of a Normative Framework for Evaluating Public Engagement Exercises: Reliability, Validity and Limitations.”
41.
Rowe, G. , Marsh, R. and Frewer, L.J. (2004) “Evaluation of a Deliberative Conference,”Science, Technology, & Human Values29(1): 88–121 .
42.
Shaw, I. (1999) Qualitative Evaluation. London: Sage .
43.
Silverman, D. (1998) Interpreting Qualitative Data: Methods for Analysing Talk, Text and Interaction. London: SAGE .
44.
Thorpe, A. and Robinson, C. (2004) “When Goliaths Clash: US and EU Differences over the Labelling of Food Products Derived from Genetically Modified Organisms,”Agriculture and Human Values21(4): 287–298 .
45.
UK Government (2002) UK Government Response to AEBC Advice Submitted in April 2002. Press release, URL: http://www.aebc.gov.uk/aebc/reports/public_debate_advice.shtml.
46.
Walls, J. , Pidgeon, N. , Weyman, A. and Horlick-Jones, T. (2004) “Critical Trust: Understanding Lay Perceptions of Health and Safety Risk Regulation,”Health, Risk & Society6(2): 133–150 .
47.
Wynne, B. (1991) “Knowledges in Context,”Science, Technology, & Human Values16(1): 111–121 .
