Abstract
Delphi Groups are an increasingly popular method, not least because electronic communications have made it easier to assemble a ‘virtual’ expert panel, but there have been a number of review articles which have pointed to a lack of rigour. Using an extended case study of a Delphi Group designed to establish agreed policy recommendations and deriving from a project using observational and semi-structured interview methods, this article examines the value of Delphi Groups as part of a mixed method research design. The article includes a narrative of the sequence of events in the Delphi Group’s deliberations, a detailed examination of how the group process led to the modification of one draft policy recommendation, a further examination of a contested additional policy recommendation, and a list of pragmatic recommendations on the conduct of Delphi Groups in respect of size, composition, recruitment, contestation, timing, closure and scope.
Introduction
Delphi Groups have been defined as ‘A method for achieving consensual agreement among expert panellists, through repeated iterations (usually by email) of anonymized opinions and of proposed compromise statements from the group moderator’ (Bloor and Wood, 2006). The technique has its origins in attempts to inform military strategy during the Cold War (see Rescher, 1998) and has evolved to become a quite popular method in policy studies, particularly in the health field (e.g. Adler and Ziglio, 1996; Keeney et al., 2011). Previous logistical difficulties have diminished with the advent of internet access, which has allowed the assemblage of ‘virtual’ panels of experts. However, review articles have been critical of the standard of much of Delphi Group research and it has developed an unenviable ‘quick and dirty’ reputation. Typical problems include: panel recruitment bias, poor specification of questions/tasks by the moderator, high rates of panel attrition, problematic consensus where it is unclear what actually constitutes consensus, and ‘specious consensus’ where panellists conform out of sheer boredom (see reviews by Rowe and Wright, 1999; Garrod and Fyall, 2005).
Potentially, Delphi Group methods can assist in the communication of research findings to policy-makers. The relationship between sociological research and policy-making is contested and problematical (see Bloor, 2004). Becker’s famous rhetorical question ‘Whose side are we on?’ (1967) was a call for a partisan sociology which championed the underdog against the elites, including the policy-maker elites, and Burawoy (2005) called for a ‘public sociology’ that would appeal directly to the public over the heads of policy-makers. Harvey Moloch (1994) has ruefully pointed out that sociologists who wish to ‘speak truth to power’ actually have great difficulty finding anyone to speak to at all. In contrast, Bulmer (1982) has described a different model of sociological influence, the ‘enlightenment model’, where social scientists influence policy-makers osmotically through extended informal contact, but it has to be objected that, in Anglo-Saxon countries at least (France and Scandinavia might possibly form contrasting cases), sociologists are rare visitors to the corridors of power. It follows that, while policy-makers may be the anticipated audience for much research, many sociologists are inexpert in drawing out the policy implications of research findings and could use some expert help.
This paper does not seek to rehabilitate Delphi Groups as a stand-alone research method, but rather considers the possible limited role of Delphi Groups within multi-method research projects, where expert input from a Delphi Group in the end-phase of the project can potentially serve a number of useful functions. There is a large literature on the virtues and drawbacks of mixed method designs that has been widely reviewed elsewhere (Barbour, 1999; Brannen, 1992; Bryman, 2006; Cresswell and Plano-Clark, 2007; Johnson and Onwuegbuzie, 2004; Morse, 2005; O’Cathain, 2010; Parry-Langdon et al., 2003), a handbook on mixed methods has been published (Tashakkori and Teddie, 2003), and there is a journal, the Journal of Mixed Methods Research 1 devoted to mixed methods studies, so only a brief recapitulation is necessary here. The most recent review (O’Cathain, 2010) lists four virtues of mixed methods. One of these virtues, ‘development or facilitation’, is a mere commonplace – the use of one method, say a postal survey, to facilitate the recruitment of a sub-sample for another method, say semi-structured face-to-face interviews. A second virtue, that of ‘emancipation’ or giving voice to the marginalised or oppressed, is controversial and open to dismissal as tokenism.
The third claimed virtue of mixed methods is also controversial, but less readily dismissible. Where a Delphi Group is part of a mixed method design and the expert panellists of the Delphi Group have also been among the research participants in the wider study, as interviewees or subjects of ethnographic observation, the Delphi Group method then becomes, in effect, a variant of that form of ‘methodological triangulation’ (Denzin, 1970) known (misleadingly) as ‘member validation’ (Emerson and Pollner, 1988). While triangulation – the comparison of findings on the same research topic generated by different methods – is often seen as a mark of good research practice, member validation is usually dismissed as a well-meaning but wrong-headed attempt to demonstrate the accuracy of a social science account of a social activity by appeal to the judgement of those persons taking part in that social activity. The problems with member validation have been discussed elsewhere (Bloor, 1997; Emerson and Pollner, 1988; Seale, 1999: 61–72). There is a lurking positivist assumption of just a single paramount reality available for sociological description. There is also the concern that collectivity members, shown the sociologist’s account and asked to validate it, may find it too politically sensitive and may fear that it will be used as a weapon against them by others. Alternatively, members shown an account for validation purposes may be quite unconcerned and may only read the sociologist’s report in a superficial and uncritical way. Further, there is the possibility that members’ views may change over time – what they previously would have validated, they now reject, or vice versa. And, finally and most importantly, there is a failure on the part of the investigator to appreciate that the member validation exercise is not a neutral information-gathering medium: like all methodological processes, it shapes the product it is designed to produce. However, there is no need for a Delphi group exercise to be oriented towards the validation of research findings: it may still be a very useful exercise if it produces a critical panellist’s response to preliminary findings and leads to the reconsideration of earlier data, to the amendment or qualification of preliminary findings, and/or to the sharpening of conclusions and recommendations. This indeed is the fourth (and uncontroversial) virtue of mixed method designs, that of comprehensiveness, whereby a topic can be addressed more widely and completely by a range of datasets. Thus, a Delphi Group constituted from research participants (as in the case below) may be simply a useful additional dataset in a multi-method research design, without there being any ambition to conduct a member validation exercise.
For illustration, we report on the progress of a Delphi Group exercise where a virtual expert panel was tasked to reach consensus on the policy recommendations to be drawn from a project on the effectiveness of enforcement of new international regulations on ship emissions. A methods section briefly describes the wider study within which the Delphi Group was conducted and describes in more detail the composition and conduct of the group. Two further sections illustrate, first, the process whereby a draft recommendation of the study was modified through the Delphi process, and second, the process whereby a possible additional recommendation (proposed by two separate Delphi Group members) was rejected. This is followed by an extended conclusion, embracing some pragmatic suggestions on the conduct of Delphi Groups and an examination of the strengths and weaknesses of Delphi Groups within mixed method research designs.
However, first of all, some background material needs to be presented on the regulation of ship emissions for readers to make sense of the reported Delphi deliberations. Until very recently, most vessels burnt a cheap residual fuel oil, a by-product of oil refineries (that which is not sold for marine fuel is sold for road tar). However, the exhaust plume from burning that residual fuel contains pollutants – sulphur compounds (SOx), nitrogen compounds (NOx) and fine particulate matter – that have contributed to acidification and to pulmonary and coronary diseases (Winebrake et al., 2009). The UN agency, the International Maritime Organization (IMO) brought into force regulations to control these emissions in 2005 and 2010. Emission Control Areas (ECAs) were set up in the Baltic (2006), the North Sea/English Channel (2007) and the coasts of North America (2012). Initially, the sulphur limit on fuel in the ECAs was set at 1.5%, reducing to 1.00% on 1 July 2010 and set to further reduce to 0.1% on 1 January 2015. Bunker fuel suppliers are able to reduce the sulphur content of residual fuel oil to 1.0% (known as low sulphur fuel oil) by judicious blending of residual fuel oils and distillate. However, a limit of 0.1% sulphur will eliminate the possibility of vessels burning residual fuel oil unless they fit expensive exhaust abatement technology – a dispensation which need not concern us here. Effectively, vessels in the ECAs will be required to switch to burning much more expensive distillate fuel and ship operators (at a time of over-capacity and falling freight rates) will find their fuel costs exceeding their crewing costs. In addition, the EU has introduced, from January 2010, an additional requirement in EU ports for all ships at berth (where only auxiliary engines are in use) to burn fuel with 0.1% sulphur.
The main means of enforcing these and other IMO regulations is by ‘port-State control’, that is, the equal enforcement of international shipping regulations by the port State, regardless of a ship’s flag. Since 1982, the Paris Memorandum of Understanding on Port State Control (2010) has established a common methodology of ship inspections for foreign-flagged vessels berthing in European ports. The inspections are not exhaustive: in a frequently used phrase, port-State control is ‘a sample, not a survey’. Port State Control Officers are required to use their discretion, and officers who encounter initial problems are likely to then inspect progressively more areas of the ship’s operation and equipment, while those who do not encounter initial problems may proceed to a ‘light-touch’ inspection. Officers are empowered to detain ships with serious deficiencies until those deficiencies are rectified, a sanction which can be expensive for ship operators. But arguably a more powerful deterrent is the ‘naming and shaming’ of ships with deficiencies on industry websites, which impacts on those vessels’ abilities to attract custom from charterers (note however that those industry websites currently only report vessels’ IMO deficiencies, not their EU deficiencies – see the section below, ‘Delphic Modification of a Policy Recommendation’). The most widely used method of enforcing the new sulphur regulations is for officers to check the vessel’s previous Bunker Delivery Notes (which the vessel is required to retain), which record the sulphur content of the bunkered fuel, and for officers to check the Oil Record Book, where a switch-over to a different fuel must be recorded by the Chief Engineer. The European Commission has requested Member States to take samples of fuel, on occasion, for testing of the sulphur content. However, only Denmark, Germany, Holland and Sweden have so far equipped their officers with fuel sampling kits. Testing of fuel is expensive and port-State control is not a revenue-generating activity.
Finally, it should be noted that, over time, the operation of port-State control appears to have served to increase the quality of shipping, at least in respect of those vessels trading in and out of European ports. A ‘culture of compliance’ is said to have recently grown up among ship operators (Bloor, Sampson et al., 2013). However, the potential rewards for non-compliance with the sulphur regulations are very substantial: currently (July 2013), the price difference between high sulphur fuel oil outside the ECAs and the compliant 1.0% low sulphur fuel oil is around $18 per tonne, and the price difference between high sulphur fuel oil and 0.1% distillate is a massive $325 per tonne. There are therefore concerns that, if the fuel regulations are ineffectively policed, then this new culture of compliance may disintegrate.
Methods
The wider research project (funded by the UK’s Economic & Social Research Council) was comparative between Sweden and the UK and involved observation of 16 ship inspections (involving in turn extended visits to seven different Port State Control offices in the UK and Sweden) and 50 qualitative interviews with inspectors, regulators, shipping industry stakeholders (ship operator managers, shipping industry representatives, fuel experts, trade unionists and others) and environmental NGOs. A final report was drafted of around 9,000 words and circulated to seven Delphi Group members; the report concluded with seven policy recommendations. It will be appreciated that, in a design that had already incorporated semi-structured interviews, it would only be possible to gauge reactions to the draft policy recommendations by additional use of either an electronic ‘real-time’ focus group or an electronic Delphi Group, and with research participants having crowded diaries and other urgent commitments, a Delphi Group was much easier to convene. The Delphi Group members were drawn from the 50 interviewees, but they were not selected randomly. Instead, they were selected purposively as likely to have a range of viewpoints, and (on the basis of their views at interview) likely to be sufficiently committed to the study (or the topic) to be willing to take part in the rather arduous Delphi Group process. The seven members comprised two regulators, two ship operator managers, one shipping industry representative, one inspector (aka Port State Control Officer), and one member of an environmental NGO. All seven agreed to take part.
The purposive selection of Delphi Group participants with a range of backgrounds increased the likelihood of receiving contributions from Delphi Group members which exhibit partial disagreement with the seven policy recommendations. In this instance, the Delphi Group process and the researcher’s analytic response are therefore analogous to the search for ‘deviant cases’, which is at the heart of the longstanding procedure known as ‘analytic induction’ (Cressey, 1953; Bloor, 1978), whereby the analyst searches for deviant cases that do not fit with a preliminary hypothesis and seeks to modify and extend the hypothesis to accommodate the deviant cases, thereby extending and deepening the analysis. In Seale’s judgement: ‘as a reminder to be systematic in the search for negative instances and to modify one’s theories in the light of new evidence, the spirit of analytic induction seems worth preserving in qualitative work’ (Seale, 1999: 85).
The Delphi Group process was scheduled to take place through the month of September 2012. On Monday 3 September, the first author (acting as group moderator) re-emailed the draft final report with the recommendations (it had been circulated in the first place as part of the Delphi Group recruitment process), with a request that individuals send their suggested changes to the recommendations back to the moderator by the end of the day on 5 September. This was ‘Round 1’. The moderator would then circulate the anonymised suggestions (and the moderator’s response) to the rest of the group, who would then have an opportunity to respond by the end of 10 September. Those responses would then lead to a further iteration of the process (‘Round 2’), with the prospect of further iterations occurring to follow, in the hope of eventually arriving at consensual agreement with the re-drafted recommendations by the end of the month. The initial 3 September email concluded: Your participation in the group is based on your special expertise […], but you are not being asked to participate as a representative of your organisation or company. I would therefore ask you not to discuss the draft recommendations and your suggestions with colleagues. At the end of the Delphi Group, the final report together with the (hopefully) consensual recommendations will be made publicly available. To reiterate, no reports of the project will name any of the Delphi Group members.
In the event, only four of the seven responded to the first email. Fortunately, however, these four still represented a range of expertise – one regulator, one ship operator manager, one inspector and one member of an environmental NGO. All four participants had suggestions on modifications to at least one of the recommendations (one group member offered suggestions on four of the seven recommendations), and two participants independently suggested an additional recommendation (see the section ‘Delphic Suggestion of an Additional Policy Recommendation’ below). The moderator responded privately to each of the participants, in part in order to seek more clarity on their suggestions, and then (in Round 2) circulated all the comments (with some abridgement and clarification), plus his responses, plus revised recommendations.
In response to Round 2, two of the initial participants offered further comments. None of the Round 1 non-respondents commented, although they received invitations to do so. The two responses to Round 2 resulted in some further small changes, circulated along with the Round 2 comments and moderator’s responses as Round 3. No further suggestions were received after Round 3 (sent out on 16 September). However, further information later received, in response to enquiries the moderator made in following up one of the suggestions made in Round 2, led to a further small modification and a final version of the recommendations (Bloor, Baker et al., 2013) being sent out on 25 September.
Delphic modification of a policy recommendation
In all, three of the seven initial recommendations dispatched to the panellists in Round 1 underwent some important modifications. We illustrate the Delphi process by reference to the second of those recommendations, which read as follows: 2. That all EU countries inform the European Commission of the IMO number of all vessels founded to be non-compliant in respect of the 0.1% sulphur port fuel regulations, with a view to the Commission seeking to ensure that these non-compliant cases appear on Equasis.
Whereas the final version read as: 2.
This recommendation relates to the previously mentioned importance of the deterrent effect of ‘naming and shaming’ non-compliant vessels on shipping industry websites (see, for example, the discussion in Bloor et al., 2006). Such websites (the most influential of which is Equasis) are widely consulted by charterers, insurers and others, and vessels with poor compliance records will find the freight rates that they can command will be severely adversely affected. Equasis obtains its information on the inspection records of vessels from the THETIS website of the Paris Memorandum of Understanding on Port State Control. Since THETIS is the Paris MoU website (and since the Paris MoU includes a number of non-EU members), it only records the results of inspections in respect of IMO (and International Labour Organization) regulations, not in respect of EU regulations. However, since the European Commission bankrolls THETIS, it seems reasonable for the Commission to request the inclusion in THETIS of inspection records showing non-compliance with EU regulations on sulphur levels. Once those inspection records have appeared in THETIS, they will automatically appear on other websites like Equasis.
The final version of Recommendation 2 incorporated suggested changes from three panellists. Two of the changes were minor: first, the final version incorporates a suggestion from one panellist that each recommendation should have its own sub-heading (in this case, ‘Publication of Non-compliant Vessels on Equasis’); and second, the final version incorporates a full summary of all the recommended information on the vessel to be published on Equasis (i.e. the vessel’s name, and the place and date of inspection), as well as the vessel’s unique IMO number and the regulatory violation. The third change was more important and concerned a suggestion from a panellist that the violation of the EU regulation on the sulphur level of fuel to be burnt in port could still be recorded on the THETIS website as if it were a violation of an IMO regulation, if the inspector in question recorded the violation as a deficiency under the IMO’s International Ship Management (ISM) Code, Paragraph 1.2.3, of which states that the vessel’s safe management system should ensure ‘compliance with mandatory rules and regulations’ (IMO, 2002). The ISM code is a requirement for all vessels to have formally laid-down management systems for all the standard procedures involved in vessel operation (berthing, cargo-handling, etc., etc.).
The third suggestion was a valuable contribution to the study. Treating violation of the EU regulation as if it were a deficiency under the IMO’s ISM code would be a powerful incentive to regulatory compliance: not only would it result in the naming and shaming of the vessel in Equasis, but (if the violation was identified as grounds for detention of the vessel in port by the inspector in question) the ISM code deficiency would automatically trigger the requirement for an additional audit by the vessel’s flag-State, with the possibility of delay and expense for the ship operator. None of the Port State Control Officers interviewed in the study had suggested that the ISM code could be used in this manner and, on the one occasion when an observed inspection found a vessel to be burning non-EU-compliant fuel in port, the inspector in question did not treat the violation as a deficiency under the ISM code. However, the wording of the ISM code clearly requires compliance with all mandatory rules and regulations (i.e. regional EU rules, as well as global IMO rules), and the ISM code is expressly designed to be a tool for the punishment of poor ship management practices (such as the failure to provide the vessel with compliant fuel). Drawing on his particular expertise, this Delphi Group member had been able to suggest a possible policy option which had not, thus far, occurred to the interviewed inspectors, all of them members of a generic inspectorate grappling with the enforcement of a new regulatory regime. Such a development is, of course, a strength of a multi-method research design: the additional data provided by the Delphi Group led to a re-analysis and an extension of the findings.
However, it remained unclear how this possible policy option was to be drafted. Port-State inspections, in common with much inspection practice in other jurisdictions, follow a discretionary methodology, allowing surveyors latitude in the depth and foci of inspections and in the actions required from non-compliant vessels. The relevant Paris MoU instruction to surveyors states that the surveyor ‘should use professional judgement to determine whether to detain the ship or to allow it to sail with deficiencies which do not pose an unreasonable threat of harm to the environment’ (Paris MoU, 2010: 1). Therefore it would not be appropriate to have a study recommendation that Port State Control Officers be instructed to treat a violation of the EU port regulations as a deficiency under the ISM code. One possibility would have been to incorporate the panellist’s suggestion within Recommendation 7 (‘Port State Control Training’), but as the panellist had raised the topic in response to Recommendation 2, it was felt more appropriate to report the possibility of treating the EU fuel violation as an ISM deficiency as part of Recommendation 2.
Delphic suggestion of an additional policy recommendation
Two of the four panellists independently suggested that the research report should recommend that substantial fines be levied on non-compliant ships. This issue was topical at the time of the operation of the Delphi Group because a draft European Commission Directive published at that time had stated the need for Member States to fine the operators of non-compliant ships. In drafting the research report, the possibility of recommending fines had been previously been considered, but rejected. Rejection was based on the evidence collected in the course of the study on the Swedish experience of fining operators using non-compliant fuel. Sweden had been taking and testing fuel samples from around 200 quasi-randomly selected vessels per year, since 2006, but not a single vessel had been prosecuted between 2006 and 2010. This information had not been included in the draft report and was thus unavailable to the panellists.
The lead author, who conducted the Delphi Group, accordingly decided not to adopt the fines suggestion in Round 2, but to state at length the reasons for not doing so. At the same time, the offer was made to include a recommendation on fines in Round 3 if the panellists still felt this was necessary. Here is an abridged version of the moderator’s Round 2 response: I should say that my colleagues on the project and I have considered the fines issue. And you are right of course that substantial fines do have a deterrent effect. However, I decided against including this in the recommendations because of the Swedish experience with prosecutions. As you know, Sweden has been taking fuel samples as part of the enforcement of the fairway dues system […] but from 2006 (the start of the Baltic ECA) until the end of 2010 there wasn’t a single prosecution […] In each case, the [State] prosecutors decided that the higher standard of proof required for a successful court case had not been achieved (e.g. no proof that the sample tested in the laboratory was the same as that taken on the ship, no proof that the fuel sample had not been tampered with en route to the laboratory, etc.). I believe there could be a similar problem in the UK courts. […] It’s currently unclear how effective the US fines regime will be, but we felt that in the context of the European courts the high standard of proof of non-compliance that might be required could mean that relatively few cases would come to court. Our previous work in this research centre on Port-State Control indicated that the threat of detention was the major deterrent for ship operators, because the publication of a detention on Equasis and other industry websites has a major impact on the vessel’s future freight rates […]. So the report focuses on making port-State enforcement more effective. That’s why we’re recommending that breaches of EU port 0.1% fuel regulations be publicised on Equasis (they aren’t at the moment) and that departed vessels found on testing to have been burning non-compliant fuel be inspected at the next port (they aren’t at the moment). Nevertheless, if the general feeling of the group is that we should include a recommendation on fines, then of course we’ll do that.
In their responses to Round 2 of the Delphi Group, only one of the panellists returned to the fines issue: he did not explicitly state that he wanted his initial suggestion on fines to stand. Instead, he thanked the lead author for taking his previous suggestion so seriously and for explaining why the recommendation had not been included in the earlier draft of the report. He then copied a press report about a fine levied by an Italian court on the operator of a cruise ship for burning non-compliant fuel in port, while stating that ‘I have to admit, though, that this is one of the few examples I’ve seen’. This was taken to be concurrent with the view taken in the report that few cases of non-compliance would come to court and receive fines. The same panellist’s response to the final Round 3 included the statement: ‘Thanks for the new draft recommendations – they are all fine’. So it would appear that this panellist at least had come round to the view that his initial suggestion for a further recommendation on fines could now be disregarded.
Conclusion: Delphi Group strengths and limitations
We begin with two caveats. First, and self-evidently, a Delphi Group is not the only additional qualitative method that can be introduced with advantage into a mixed method design: focus groups and semi-structured interviews are obvious alternatives. But an electronic Delphi Group, allowing the remote polling of experts, does seem a particularly appropriate choice for a policy research project.
A second caveat: this discussion is based on the experience of conducting one particular Delphi Group exercise and it should be acknowledged that, with the benefit of hindsight, that exercise could have been better conducted. In particular, it was a mistake not to explicitly request that the panellists communicate their agreement with the Round 3 recommendations. Although, as seen above, one panellist chose to reply endorsing the final recommendations, none of the others did so. The email accompanying Round 3 was poorly phrased and stated: I have only received one further set of comments on the suggested modifications in Round 2. These comments, my response and the further suggested modifications are set out in the attachment. Please could I have your further suggestions by the end of Wednesday Sept 19th? If I receive no further suggestions, then I will conclude the Delphi Group.
It is likely that the other panellists construed this email as taking the position that non-response would be taken as agreement and therefore, being in agreement, they felt no need to respond. But of course non-response can also indicate drop-out – what has been termed elsewhere ‘specious consensus’ (Sackman, 1975). Good practice should require that all participating panellists explicitly state their agreement before consensus is claimed and the Delphi Group is concluded.
Having stated this caveat, it seems fitting to draw on the experience of this particular Delphi Group exercise to include some tentative, pragmatic suggestions on ‘dos and don’ts’ in Delphi Group conduct:
Size: this Delphi Group exercise operated quite successfully with just four active participants, but that is surely a minimum number. However, more important than size is ‘balance’ – it was sheer luck that our four active participants represented a range of viewpoints, expertise and interests.
Recruitment: given the onerous nature of the exercise, recruitment is most likely to be successful among a population of research participants, who had previously shown commitment to the study. However, in this instance seven research participants had formally agreed to serve as panellists, but in the event only four sent responses. It would therefore seem prudent to recruit more panellists than are needed in order to allow for ‘no-shows’. Again, balance is critical.
Composition: recruitment needs to be tailored to the specific task at hand, kindred to purposive sampling. A Delphi Group tasked to consider policy recommendations will need to include some panellists with policy expertise. Balance is critical, but representativeness is not.
Moderation: the moderator’s role extends beyond the facilitation of the timely completion of the Delphi Group and requires the exercise of discernment in responding to inputs from each round. Not all Delphi suggestions need to be incorporated in the final version of the recommendations. Where other data justify it, seemingly inappropriate recommendations can be courteously contested.
Timing: a virtual Delphi Group, operating by email, is the easiest commitment in the world to extinguish, by the simple expedient of remaining silent. It is therefore important that panellists realise that their commitment is not open-ended. This Delphi Group was told that it would end after a month, regardless of whether or not the repeated iterations had achieved consensus. The specific function of the Delphi Group, and the group’s time-limited task, differentiate a Delphi Group from the expert Advisory Groups that often accompany large-scale research projects.
Closure: profit by our mistake and ensure that, when sending out the final iteration you ask all respondents to confirm their agreement, rather than give seeming assent through their silence.
Modesty: treat the Delphi Group simply as a useful additional data collection exercise (focusing on policy implications in this particular case). Panellists’ agreement does not equate to either public participation or member validation. Nevertheless, the very fact that the Delphi Group is a purposive sample, chosen for the range and depth of their expertise, ensures that the additional data they provide will have particular added value, especially where the researcher employs an analytic induction approach to the study of ‘deviant cases’.
Beyond these ‘dos and don’ts’, some general points can be made on the Delphi Group method. First, it is clear that Delphi Groups can bring certain advantages within a multi-method research design. As was illustrated by the panellist’s suggestion on the use of IMO’s International Ship Management Code to detain vessels with non-compliant fuel, a Delphi Group, by being an additional vehicle for the presentation and collection of new evidence/data/arguments, can extend and deepen an analysis. The nature of the Delphi Group method is such that the new evidence/data/argument that is generated is likely to be particularly valuable, since it is likely to be in qualified opposition to early findings: the analysis has to be inductively modified and extended (or corrected) to embrace these opposing additional data. The advantage being claimed here is analogous to that claimed by Irwin (1995) and others for the use of lay experts in an ‘extended peer review’ of scientific findings. But this is an analogy only: we are not claiming here that Delphi Groups enhance public participation in sociological research. Delphi Groups are composed, after all, of experts not the laity, and other methods (most notably citizens’ juries – Coote and Lenaghan, 1997; Dunkerley and Glasner, 1998) have been specially developed to involve the public in policy-making processes. Nor are we granting a privileged epistemological status to our expert panellists: the Delphi Group is simply acting as a spur to further inductive analysis.
Relatedly, as was illustrated by the response to the panellists’ suggestions about fines, the extension of the analysis that can occur as a result of exposure to opposing viewpoints need not necessarily involve the researcher in accepting antithetical arguments and attempting to incorporate them into the research findings against his/her better judgement. The role of the moderator in the Delphi Group process allows for the reasoned exclusion of antithetical arguments, with the consent of the panellists. As the late Gordon Horobin remarked, ‘Let a thousand flowers bloom, but don’t ask me to water them all’.
Further, the very act of compiling a set of policy recommendations for submission to a Delphi Group can itself act to sharpen up the presentation and accessibility of research findings: the consciousness of the need to be understood by a non-sociological audience can both act as a spur to clarity and as a censor of political quietism and excessive obscurantist self-reference. And, of course, the panellists in turn can make suggestions to further improve presentation. A minor suggestion on presentation (the use of sub-headings for each separate recommendation) has already been illustrated. But additional and more substantial suggestions were also made in respect of another recommendation (not reported here) on sampling and testing fuel, which gave that recommendation more clarity.
Relatedly again, Delphi Groups may be particularly useful when deliberating policy recommendations simply because some of the panellists (at least) may be members of the policy community and more expert in framing policy recommendations than researchers. While a Delphi Group may be composed with an eye to a number of different specific functions within a research design, a policy-orientated Delphi Group may be of particular value.
Nevertheless, Delphi Groups can be, and have been, composed of members other than policy experts and can deliberate on topics other than policy recommendations. For example, service-users and service-providers have both been enrolled in past Delphi Groups. Technological advances have made light of the problem of remoteness, but the some of the problems mentioned by past reviewers of Delphi Group studies remain, most notably the problems of high panel attrition, and of poor specification of tasks by the moderator where the focus of the Delphi Group is more diffuse than a stated set of policy recommendations. The Delphi Group is not a tool suitable for all research occasions.
Footnotes
Acknowledgements
We thank the ESRC for their support, we thank the UK’s Maritime & Coastguard Agency, and Sweden’s Sjofartsverket and Transportstyrelsen for their assistance, and we thank our anonymous research participants for their time and their kindness.
Funding
This project was supported by a grant from the UK’s Economic and Social Research Council (grant number: RES-062-23-2644).