Using a Modified Delphi Approach to Explore California’s Possible Transportation and Land Use Futures

Abstract

Many methods exist for engaging experts in interactive groups to explore, clarify, and/or decide on various issues. In an investigation of four possible future scenarios concerning transportation and land use in California, we developed a novel “hybrid policy Delphi” method for use with a panel of 18 experts. We applied it to explore the policies and practices that would likely lead to each of the four scenarios and the consequences that would result from them. Through our process, panel members discussed and reflected on the scenarios in multiple ways. The scenario they considered most desirable they also deemed least likely to occur, and they foresaw the likely trajectory of California transportation and land use leading to less desirable scenarios. Our mix of discussion and questionnaires traded the benefit of anonymity for the benefit of exploratory, interactive discussion. In addition, our use of surveys before and after meetings allowed us to track changes in panel opinion on a central question and discuss the survey results at meetings, at the cost of greater administrative effort. We discuss the results of this hybrid policy Delphi approach, reflect on how it worked, and conclude with a discussion of limitations and future directions.

Researchers employ numerous strategies to involve expert panels in their research, each with a particular emphasis and purpose. Some are multi-stage and iterative; others involve one-off engagements. Some rely on surveys of large groups of people who remain anonymous to each other; others involve smaller groups in face-to-face discussion.

This paper explores the methods and processes used in a panel study on the future of transportation, land use, and planning in California. The purpose of this study, commissioned by the California 100 Initiative, was to develop and describe four future scenarios that experts thought likely for the state. Results from the study were included in a report intended for policymakers, leaders, journalists, and the public interested in California’s future. We explored the likelihood, desirability, and implications of four transportation/land use scenarios for 2050 with a panel of 18 experts with professional experience representing a wide range of disciplines and sectors.

We initially modeled our approach on the well-established Delphi method, used by our colleagues to recently explore pandemic-related uncertainties in transportation planning ( 1 ), to assess urban development impacts of high-speed rail ( 2 ), and to evaluate strategies in transit-oriented development ( 3 ). However, this project’s duration was too short for us to use the time-intensive Delphi, and we sought more concept exploration and discussion than the Delphi entails. Instead, we ultimately developed a research strategy whose mix of quantitatively measured survey responses and qualitative discussion design elements align with those of other methodologies, such as the nominal group technique (NGT), the policy Delphi, the hybrid Delphi, and the focus group. Our method most closely resembles the hybrid Delphi but differs in process; therefore, we have named it the “hybrid policy Delphi.”

Types of Group Research Processes

Workshop

The workshop is a single convening of panelists (experts, non-experts, or both) to discuss topics without anonymity. Workshops are useful for exploring issues and their loosely structured format means researchers can collect many perspectives.

Focus Group

Focus groups aim to obtain qualitative data from a targeted group of people. A meta-study of focus groups describes them as relatively small: typically between three and 21 participants, with a median of 10, and typically involve a one-time convening of participants giving open-ended responses to thematic prompts ( 4 ).

Nominal Group Technique

The NGT is similar to a focus group but smaller (7–10 participants) and more structured ( 5 , 6 ). It blends group discussion with panelists’ written ideation and feedback, which moderators use to gauge interest in or agreement with a given topic, direct discussion, and order results by rank accordingly. Compared to workshops, focus groups, and other interacting groups, the NGT avoids the problems of dominant speakers and reticent participants through its use of independent writing ( 5 ).

Delphi Method Processes

“Traditional” Delphi

The Delphi method as first formulated involves “a series of intensive questionnaires interspersed with controlled opinion feedback” ( 7 ). Its four distinct features are (1) the anonymity of panelists to each other, (2) iteration of the process with controlled feedback (i.e., questionnaire results are shared with panelists, sometimes along with limited written responses), (3) statistical group response (i.e., results can be quantified), and (4) and expert input ( 8 ).

The method is designed to narrow the range of panelists’ responses over time. Over a series of surveys, panelists receive the same question(s), along with the panel’s aggregated responses to previous iterations. Importantly, the method avoids “direct confrontation of the experts with one another” ( 7 ). According to its formulators, it is “more conducive to independent thought” than a direct discussion, which can harden initial opinions or lead other panelists to be swayed too rashly ( 7 ).

The Delphi method confers several advantages for researchers seeking expert forecasts. Firstly, the format captures both an initial set and a “corrected” range of responses. Secondly, the results capture panelists’ individual reflections at each stage. Thirdly, the results offer a useful range of answers that reflects both individual input and a collective response. Together, these facets allow researchers to examine the direction, speed, and completeness of any convergence (or divergence).

Although convergence on consensus has come to typify the Delphi, it is sometimes misunderstood: the authors of an early authoritative book on the Delphi method clarify that the goal is not to achieve consensus but rather a stability of responses—that is, a point at which respondents’ answers do not change in successive rounds. A split distribution of responses can be a result and is in fact revealing, although many applications of the Delphi method do not pay attention to such divergences ( 9 ).

Some scholars have criticized the Delphi method for providing a narrower but no more accurate range of responses. The final judgments of the panel may represent a compromise or a bandwagon effect rather than best judgments. The type of feedback that panelists receive affects shifts in opinion: argumentative written feedback prompts less change than statistical feedback ( 10 , 11 ). Nevertheless, some researchers argue that the Delphi method is better than other techniques at avoiding the pressure of conformity, although they concede the presence of the bandwagon effect. However, one study that gave false feedback to panelists yet still resulted in convergence raises questions about whether conformity pressure is actually absent in Delphi panels, despite panelists’ anonymity and isolation ( 8 , 10 , 12 , 13 ).

“Policy” Delphi

One early variation of the Delphi method, the policy Delphi, uses the Delphi method’s iteration, interaction, and anonymity but emphasizes strong opposing views and is typically designed to present all options and supporting evidence to policymakers. Rather than narrowing a range of responses over two or three iterations, the policy Delphi uses four or five rounds to explore and expand on responses. The first round explores ideas on the topic; the second and third narrow areas of interest and establish group views; the fourth and fifth involve discussions and decisions based on opinions established in the third round. This method can inform but not decide on policy matters. The policy Delphi shares many of the advantages and disadvantages of other Delphi approaches: it is economical, provides respondents the time and opportunity to re-think responses, and is flexible, although it is lengthy. Strengths and weaknesses particular to the policy Delphi include a tendency to produce many ideas and evaluations of ideas, although these gathered ideas can be difficult to synthesize. For this reason, some researchers think the policy Delphi is suitable as a precedent to or foundation for workshops or other group discussions ( 14 , 15 ), although others argue that it can be used to facilitate consensus on issues that are not well defined ( 16 ).

“Hybrid” Delphi

Economists in Spain developed another variation, the hybrid Delphi, which precedes the Delphi process with an in-person convening, so as to draw on the strengths of the Delphi method while mitigating its shortcomings (i.e., its limited interactions, respondents’ possible misuse of anonymity, and the possible excess and uncorrected influence of the survey designer in setting the problem definition) ( 17 ). In a hybrid Delphi, researchers lead a panel of 5–10 experts through first a focus group and then a NGT exercise, before conducting a traditional Delphi exercise by correspondence among the same group. By doing so, researchers garnered new ideas and identified new aspects of a particular issue (a strength of the NGT), which informed the questionnaire that panelists answered in the first Delphi round—which in turn informed the second Delphi round. As its developers note, the hybrid Delphi differs from other variations of the Delphi in that the Delphi process itself is not modified per se but rather supplemented ( 17 ).

A New Hybrid Method

Hybrid Policy Delphi

Given the broad and important remit and the tight (three month) timeline of the research, time did not allow for the type of iterative correspondence used in the full Delphi process. We nevertheless thought that the Delphi’s concept of iteration would serve our project well, alongside other exploratory methods.

Given our charge and timeline, we developed our own variation of the Delphi method that has many elements in common with the policy and hybrid Delphi methods but also some in common with the NGT and focus groups.

Our method lost some of the Delphi benefits of individual assessment, conducted in isolation without confrontation. However, our modified approach allowed for learning from thought processes and conversation still informed by iterative questionnaires completed privately. Our approach took steps to mitigate the concern over “hasty formulation of preconceived notions” raised by Delphi’s creators ( 7 ) and to capture both some Delphi-style convergence and the added nuance of respectful and diverse group discussion. In addition, we added “backcasting” to our method (described further below) so as to explore how a preferred scenario might be attained.

Table 1 outlines the similarities and differences among these strategies and denotes the elements our method has in common with others, either fully (boldface type with double asterisks [**]) or partially (italic typeface with single asterisk [*]). Note that this table characterizes each strategy with attributes that are typical, as described in academic literature, but many deviations exist in practice ( 8 , 18 ).

Table 1.

Comparison of Group Research Methods

	Workshop	Focus group	Nominal group technique	Delphi	Policy Delphi	Hybrid Delphi	Hybrid policy Delphi (our method)
Typical method	Meeting	Meeting	Meeting	Survey	Survey	Survey and meetings**	Survey and meetings**
Multi-stage/iterative	No	Sometimes *	No	Yes**	Yes**	Yes**	Yes**
Panelists develop topics	Yes	No**	No**	No**	Yes	No**	No**
Rounds	1	1–4	1	2–3*	4–5	4	3**
Statistical group feedback shared with participants	No	No	Yes**	Yes**	Yes**	Yes**	Yes**
Written questionnaires	No	No	Yes**	Yes**	Yes**	Yes**	Yes**
Typical size	Variable	3–21**	7–10	10–30**	10–50**	5–10	18 **
Typical panelists	Experts or non-experts	Experts or non-experts	Experts or non-experts	Experts**	Experts or non-experts	Experts**	Experts**
Process	Exploratory**	Explanatory, clarifying*	Prioritizing	Narrowing responses	Exploratory**	Exploratory, prioritizing*	Exploratory, narrowing scope**
Result	Collection of perspectives	Themes, understanding, clarification	Decisions, ranked ideas	Range of quantitative answers with rationales	Range of outcomes, considerations	List of ordered proposals, arguments	Themes, perspectives, quantitative answers**
Role	Decision-making or -facilitating*	Decision-facilitating**	Decision-making	Decision-making	Decision-facilitating**	Decision-facilitating**	Decision-facilitating**
Role orientation of group ( 5 )	Socio-emotional group maintenance	Socio-emotional group maintenance	Socio-emotional and task-focused**	Task-focused	Task-focused	Socio-emotional and task-focused**	Socio-emotional and task-focused**
Anonymity	No**	No**	No**	Yes	Yes	No**	No**
Controlled feedback	No	No	Yes**	Yes**	Yes**	Yes**	Yes**
Convergence-focused	No	No	No	Yes**	No	Yes**	Yes**
Precursor to group processes	No**	No**	No**	No**	Yes	No**	No**
Administrative costs	Low	Low	Medium**	High	High	Medium**	Medium**
Panelist time requirements	Low	Low	Low	High**	High**	High**	High**
Process time	Very short	Short	Short	Very long	Very long	Medium**	Medium**

Note: Representation of the elements our method has in common with others: either fully (boldface type with double asterisks [**]) or partially (italic typeface with single asterisk [*]).

Source: ( 4 – 6 , 10 , 12 , 14 – 17 , 19 ).

Scenarios

The four scenarios that we discussed over the period of the panel study flow from combinations of two factors–land use policies and transportation policies–whose intersection results in four distinct outcomes ( 20 ), as represented in a 2 × 2 matrix that we created (see Figure 1). Although each outcome describes a variation across only two axes (multimodality and density), we discussed each as a discrete and mutually exclusive outcome, which is why we referred to them as “scenarios.”

Figure 1.

Scenario Matrix.

Two important, related transportation concepts shaped the scenarios: accessibility and mobility. Accessibility is the ability to avail oneself, household, firm, or institution of goods, services, activities, and opportunities. Mobility, by contrast, refers to the ability to move about. Walking for 10 min. or driving for 10 min. can yield the same access but entail vastly different levels of mobility (while automobiles tend to provide the most mobility benefits in the U.S. metropolitan landscape today, one could imagine an extensive, high-speed transit network acting comparably). Mobility often conveys access, but more mobility does not necessarily mean more access—and sometimes means less, when, for example, long distances mean that time spent traveling to destinations is time away from activities at destinations ( 21 ). Greater accessibility, in contrast, means that people need not travel far or make long trips in any mode to reach their desired destinations. Land uses can support greater accessibility via higher development densities, allowing for destinations to be closer to each other. In this context, accessibility may be enhanced, even as mobility is hindered, by the traffic delays that high densities can engender. This is because the higher density of buildings does not typically provide sufficient road and parking capacity that fast point-to-point mobility via driving requires. Transportation policy, planning, and engineering are in the midst of a significant, albeit gradual, shift from a mobility focus to an accessibility one ( 22 ), and that shift figures into the scenarios we presented to the panelists. We note, however, that differences in accessibility and mobility among the scenarios (given in parentheses below) do not strictly map onto our two scenario axes, as the particular intersection of land use and transportation futures interacts to affect both concepts.

Scenario: “You’ll Need a Car to Get Around”

Car-Centered/Low-Density (More Mobility, Less Accessibility)

This is the postwar norm in California that still describes most suburban areas (Scenario 3 in Figure 1). Building densities are low, land uses are separated, streets are wide, parking is abundant, and almost every trip is made by motor vehicle for those with cars. Single-family neighborhoods, for those who can afford them, are pleasant, but travel distances are often long, and many arterial roads and most freeways are chronically congested. Most new transportation investments support increasingly electric and autonomous vehicles, ever-widened roads frequently re-congest, and new housing continues to be built primarily on the fringes of metropolitan areas.

Scenario: “Lots of Travel Choices, but Most Will Drive”

Multimodal/Low-Density (Less Mobility, Variable Accessibility)

This is the new normal in much of metropolitan California, where transportation investments go increasingly toward walking, biking, scootering, and public transit infrastructure, although most trips are still made by car (Scenario 4 in Figure 1). Looking ahead, public transit service continues to expand and policies to rein in unfettered driving—such as pricing driving to reduce congestion and emissions and encourage much more multimodal travel—are gradually phased in. However, outside of already built-up central cities, most development remains dispersed and poorly served by modes other than driving, while housing, particularly in-town affordable housing, is chronically undersupplied.

Scenario: “More City Living and Lots of Traffic”

Car-Centered/Higher-Density (Variable Mobility, Less Accessibility)

Under this scenario, policymakers prioritize urban infill development and limit suburban expansion into fire-prone and agricultural areas (Scenario 1 in Figure 1). Development densities increase in central cities and inner-ring suburbs, raising the supply of in-town housing and affordable housing. However, rather than investing in multimodal travel, public officials accede to popular calls to widen boulevards and freeways (even double-decking the most heavily trafficked ones) and build parking decks to store the mass of cars in central areas. Walking increases, but chronic traffic slows cars and buses to a crawl, increases emissions, and prompts ever more calls for expanded road and parking capacity.

Scenario: “Easy to Get Around without a Car”

Multimodal/Higher-Density (Less Mobility, More Accessibility)

This scenario entails the largest break from current patterns, wherein the multimodal-focused transportation policies in the “Lots of Travel Choices, but Most Will Drive” scenario are combined with the land use policies of the “More City Living and Lots of Traffic” scenario (Scenario 2 in Figure 1). Road and parking access is managed (such as by pricing driving and parking) to substantially reduce car trips that drivers deem not worth their price (which reduces congestion and emissions and makes the driving experience better for the remaining car trips). A fast, frequent transit service reduces waits and makes riding more attractive. Denser, mixed-use development puts more destinations in walking distance and more affordable housing where it is most demanded.

We wanted panelists to consider each scenario in several ways. The four pre-developed scenarios remained the same throughout, but how we asked panelists to think about them changed over the course of the exercise. Firstly, panelists assessed current conditions and future scenarios—specifically, what aspects and statements were “most applicable” and “least applicable” to any one of the four scenarios—and to forecast what scenario was most likely to materialize.

In the second round, we asked panelists to “backcast”—that is, to consider what kinds of changes and policies would be needed today to realize—the “Easy to Get Around without a Car” scenario in the future, as the consensus among panelists was that it was the preferred (but least likely) scenario. We told panelists to assume the “Easy to Get Around without a Car” scenario exists in the year 2050 and asked what policies and conditions had to have been in place to have arrived at this scenario. By removing the uncertainty about the future outcome—because it was given—backcasting appeared to give panelists more comfort connecting policy choices to outcomes (albeit in a different direction of causality). Finally, in the third round, we used a survey to confirm agreement/disagreement with summary statements from the discussion in Meeting 2 and to collect quantitative data on those agreements (using a seven-point Likert scale).

Panel of Experts

To help us explore the dimensions of uncertainty and possibility across our scenarios, we assembled a panel of 18 experts with professional experience covering a wide range of disciplines and sectors, touching on transportation and land use in diverse ways. To identify potential panelists, we sought recommendations and nominations from members of the UCLA Institute of Transportation Studies Advisory Board, UCLA faculty and staff, and nominees themselves. From an initial list of 155 candidates, we chose 60 to poll for interest and availability. From this group, we enrolled 18 participants, 14 of whom participated in all activities throughout our study. As an incentive for continued participation, we offered an honorarium of US$500 to each participant who attended both meetings and completed all surveys and proportionately smaller honoraria for panelists who missed a meeting or survey.

Our modified Delphi involved three “rounds” that combined surveys (with some questions repeated to track changes over time) and convenings (conducted virtually) (Table 2). Each of the panel discussions (Meetings 1 and 2) was held in two sessions to allow for scheduling flexibility. Between seven and nine participants attended each session; the sessions consisted of different mixes of participants each time.

Table 2.

Summary of Modified Delphi Activities Conducted, Their Purpose, and Results

Activity	Approximate time	Purpose	Result
First Round
Survey 1	45 min.	Record panel’s initial thoughts on the importance of California trends and the desirability and likelihood of each of four future scenarios	Initial opinions and quantitative record on scenario likelihood established
Meeting 1	90 min.	Present Survey 1 results, discuss results, and hear panel’s specific rationales for Survey 1 answers	Rationales for survey choices captured qualitatively
Second Round
Survey 2	15 min.	Ask panel about the importance of specific policies for achieving a multimodal, higher-density scenario and to probe further for reasons for Survey 1 responses (based on Meeting 1 discussion)	Relative importance of policies identified; revised quantitative record of scenario likelihood captured
Meeting 2	120 min.	Present Survey 2 results; focus panel on most desirable scenario and discuss what would need to change for it to happen (“backcasting” exercise)	Key themes and challenges identified
Third Round
Survey 3	15 min.	Quantify panelists’ agreement or disagreement with the themes identified in Meeting 2 and with statements that synthesized comments made in Meeting 2	Quantitative record of agreement/disagreement with Meeting 2 summary statements; revised quantitative record of scenario likelihood captured

Unlike the original Delphi application, we did not seek from our panelists a specific number but rather an assessment of which scenario was most desirable and which most likely. For the former, in particular, our method allowed us to hear panelists’“thoughts out loud.”

Process and Results

Round 1

We used Survey 1 as a baseline to understand what opinions the panel had at the outset and to identify areas of agreement and disagreement. We presented survey findings in Meeting 1 and explored rationales for survey responses.

Survey 1

Our first survey asked panelists to rate the importance and likely persistence of current trends in, or factors affecting, California transportation and land use. The survey also asked respondents to report agreement or disagreement on statements concerning California’s transportation system and the equity and efficacy of its development and use. Finally, respondents were presented with 40 statements describing aspects of a potential transportation future and asked to choose the scenario for which each statement was most applicable and least applicable. One question, for example, asked panel members to answer (on a Likert scale) to what extent they agreed with the statement “California’s transportation system generally has a diversity of mobility options”; a second part of the question listed all four scenarios and prompted panelists to select one as “least applicable” and one as “most applicable.”

The survey results showed some areas of strong consensus. Table 3 shows that panelists generally disagreed or strongly disagreed that community participation in transportation and land use decision-making is equitable, and that California’s transportation system adequately serves the needs of communities of color, disabled individuals, children, rural communities, immigrant communities, and older adults. All panelists strongly disagreed or disagreed with the claim that California’s transportation system has addressed past injustices on disadvantaged communities, and most also disagreed with the statement that California’s transportation system “generally has a diversity of mobility options.”

Table 3.

Panelists’ Degree of Consensus on Select Survey 1 Statements

Statement	Strongly disagree	Disagree	Undecided	Agree	Strongly agree
Community participation in transportation and land use decision-making is generally equitable.	43%	50%	7%	0%	0%
California’s transportation system generally serves the needs of … well.
Communities of color	21%	71%	7%	0%	0%
Disabled individuals	36%	57%	7%	0%	0%
Children	29%	64%	7%	0%	0%
Rural communities	36%	21%	36%	7%	0%
Immigrant communities	43%	36%	21%	0%	0%
Older adults	57%	21%	14%	7%	0%
California’s transportation system…
…has addressed past injustices on disadvantaged  communities.	71%	29%	0%	0%	0%
…generally has a diversity of mobility options.	0%	71%	7%	21%	0%
Technology…
…makes it easier for many Californians to travel  without a privately owned automobile.	14%	7%	7%	64%	7%
…reduces the need for non-work travel for many  Californians.	7%	36%	14%	43%	0%
…has helped to reduce the transportation sector’s  greenhouse gas emissions.	7%	21%	50%	21%	0%
…has helped to improve mobility options for most  Californians.	14%	29%	14%	36%	7%

Table 3 also shows that the panel was split or was unsure, though, on questions concerning the role of technology in transportation. While most agreed or strongly agreed with the statements that technology “makes it easier for many Californians to travel without a privately owned automobile” and that it reduces their need for work-related travel, they diverged over whether it reduces Californians’ need for non-work travel. Meanwhile, the statement that technology has helped to reduce the transportation sector’s greenhouse gas emissions received a neither-agree-nor-disagree response from half of the panelists. In addition, panelists were divided evenly on the key question of whether technology has helped to improve mobility options for most Californians.

Finally, in response to the list of different aspects of possible transportation futures, panelists deemed the most favorable aspects “most applicable” to the “Easy to Get Around without a Car” scenario (multimodal/higher-density) and the most unfavorable aspects “most applicable” to the “You’ll Need a Car to Get Around” scenario (car-centered/low-density).

Meeting 1

At Meeting 1, we presented the survey results to panelists as prompts for further discussion, which added further context and texture to each scenario. Panelists discussed each scenario in depth and considered the relative desirability of each scenario.

Out of specific scenario discussion arose panelists’ immediate reactions and concerns. For example, panelists raised safety and equity concerns about the urban developments described in some scenarios. On one hand, less car-centered travel patterns risk making people more vulnerable to police interaction as they ride transit or otherwise travel in non-private spaces; on the other, more automobile-centered futures put pedestrians and bicyclists at greater risk. Other themes that arose out of the discussion of scenarios included social isolation, displacement, equity in housing access, equity in job access, “not in my backyard” objections to changes in the urban environment, and unintended consequences/disparate impacts of such changes. To a varying degree, these themes were discussed for each scenario.

Round 2

Survey 2

Our second survey was more specific than the first. We asked panelists to opine on what changes they thought would be likely by 2050 and what catalysts for change they thought were most important. Because the panel showed some convergence around one scenario being most desirable but another being most likely, we also added questions to Survey 2 to probe why panelists thought the most desirable scenario was relatively unlikely to materialize.

Although nearly all (94%) panelists identified that a multimodal, higher-density scenario aligned best with California’s social, economic, and environmental goals, panelists thought other less-aligned scenarios are more likely. They identified the multimodal, low-density scenario (“Lots of Travel Choices, but Most Will Drive”) as being the most likely: 61% of panelists ranked it as first-most likely. Next came the car-centered, low-density future (“You’ll Need a Car to Get Around”), with 56% ranking it as second-most likely. In comparison, 50% ranked the “Easy to Get Around without a Car” scenario as fourth-most (i.e., least) likely, and only 6% thought it was most likely (Figure 2).

Figure 2.

Panelists’ Views on Scenario Likelihood over the Course of Three Surveys.

When considering the “Easy to Get Around without a Car” scenario, panelists clearly perceived cities as benefiting the most: 73% said urban centers and 40% said urban areas would fare “much better.” Compact suburban areas would also fare “better” (73%), while dispersed suburban areas would fare “worse” (40%). Panelists were split on rural areas’ prognosis under this scenario, with 40% foreseeing “neutral” effects, 33% saying “better,” 13% saying “worse”, and 13% saying “much worse.”

Meeting 2

With a single scenario now in focus (“Easy to Get Around without a Car”: multimodal/higher-density), panel discussions centered on how it could be realized, and with the influence of what change agents and necessary preconditions.

Housing emerged as a dominant issue, important for a multimodal transportation future. Panelists deemed the state’s housing shortage dire. While higher-density scenarios would (by definition) provide more housing supply, the state’s decades-long failure to provide adequate housing caused panelists to have doubts about future development. Panelists cited a need to change incentives, policies (such as Proposition 13’s stringent caps on property tax collection), and attitudes around higher-density development. Broadly, a lack of a shared vision and “thinking on a community level” seemed to resonate as a reason that panelists thought that opposition arises to new housing construction. Panelists also perceived that many constituents who might press for more housing solutions are left out of the political process, which inflates the influence of opponents who know how to interact with elected officials. The results often frustrate hope for change and erode trust in institutions and their competency, which emerged (to our surprise) as a significant theme in this meeting and in subsequent panel interaction.

The role of technology also emerged as a theme. Echoing earlier responses, panelists expressed some ambivalence about the role of technology in a future multimodal/higher-density scenario. Panelists noted the role that technology could play in improving mobility options (for example, by enabling mobility-as-a-service platforms, better service integration with transit, and better transit service planning and by facilitating shared rides and scooter-share). However, some skepticism arose around whether technology will improve land use and transportation outcomes (based on its failure to do so in the past), how powerful tech companies can be reined in to operate more in the public interest, and how a lack of diversity among the people who fund and build technology affects the equity of its uses. One panelist commented on technology “solving small problems, but not changing the large land use patterns.” Another said that investor interests lead tech companies to offer “frivolous,” disconnected solutions.

Without action taken to communicate a shared vision, inspire greater trust in the government’s efficacy and good faith, and defend the goals of the multimodal/higher-density future, many panelists appeared to share the view that California could attempt to implement the multimodal/higher-density scenario (“Easy to Get Around without a Car”) and “sleepwalk” into the car-centered/higher-density scenario (“More City Living and Lots of Traffic”), as one panelist put it. The consensus among panelists was that this risk was greatest if land use policies and concerns continued to focus only on the needs of and demand from wealthier people and white Californians.

Round 3

Survey 3

In our final survey, we sought to focus on topics raised in Meeting 2, so we asked panelists to identify the degree to which they agreed or disagreed with summary statements about California’s present; California’s future; prerequisite conditions and policy changes for achieving the “Easy to Get Around without a Car” scenario; and necessary policy changes for avoiding poor, undesirable, and/or inequitable outcomes of the scenario.

In the results, panelists again identified the “Easy to Get Around without a Car” scenario as the most desirable but least likely. Notably, the panel’s opinions shifted from the prior survey on the most important change agents for enabling the “Easy to Get Around without a Car” scenario, now naming elected local leaders, rather than California state legislators in Survey 2.

The Survey 3 results clarified quantitatively where the panel stood as a group in their agreement or disagreement with ideas that arose in Meeting 2. Table 4 shows an excerpt from the results with the percentage of panelists agreeing or disagreeing with summary statements. From the Survey 3 results, we were able to positively identify what “essential ingredients” the panel thought were important for improving the prospects of better land use and transportation options for future Californians. Specifically, most panelists identified improving trust in government as essential to implementing scenarios that require a change from the status quo. This finding, which raised sweeping questions about the role and conduct of the public sector vis-à-vis the myriad constituencies it ostensibly serves that reach well beyond planning and transportation, was beyond our initial scope and expectations, but it is among the most significant to emerge from this research.

Table 4.

Excerpt from Survey 3 Results

Statement	Strongly disagree	Disagree	Somewhat disagree	Not sure	Somewhat agree	Agree	Strongly agree	Likert scale weighted average (−3 to +3)
Governments and people in California have a shared vision for the state’s transportation and land use future	12%	59%	18%	6%	0%	6%	0%	−1.59
Californians trust the government to implement changes to transportation and land use	0%	35%	47%	0%	18%	0%	0%	−1.00
Californians have a strong sense of place in their neighborhoods or local communities	0%	0%	12%	0%	35%	41%	12%	+1.41
California is resilient to future climate change impacts	35%	47%	12%	0%	6%	0%	0%	−2.06
Decision-makers in California are responsive to the needs of wealthy constituents	0%	6%	0%	0%	35%	24%	35%	+1.76
Decision-makers in California are responsive to the needs of low-income constituents	0%	35%	47%	0%	18%	0%	0%	−1.00
Government in California does not always realize or recognize its own power	0%	0%	24%	18%	24%	29%	6%	+0.76
Many transportation agencies and public transit operators have relatively little power over factors that influence ridership (such as land use)	0%	0%	18%	0%	12%	41%	29%	+1.65

Discussion

Delphi in Action: Convergence and Divergence

In each of our three surveys, we asked panelists which scenario they thought was the most likely to materialize by 2050. The scenario definitions did not change, but the responses did, likely explained by panelists having time and reflection on their own and others’ thoughts between their surveys.

Panelists’ responses were mostly consistent between Survey 1 (which preceded Meeting 1) and Survey 2 (which followed Meeting 1). However, responses shifted notably between Surveys 2 and 3, suggesting that the content of the medial Meeting 2 changed several participants’ minds. The focus of Meeting 2 on the policies necessary to arrive at the singular scenario (“Easy to Get Around without a Car”: multimodal/higher-density) that the panel had previously established was most desirable and on what policies were necessary for that scenario appear to have motivated panelists to find it somewhat more likely. Talking about the actions and factors necessary to enable the scenario may have made it seem more plausible to panelists. It also seems probable that the “backcasting” method we used in Meeting 2 was particularly effective in enabling panelists to envision feasible paths forward and therefore view the scenario as more likely. Because backcasting assumes a future scenario will occur, it removes uncertainty about the future outcome and frees participants to think about how that scenario came to be.

Figure 2 illustrates how responses to the same question about the likelihood of each scenario changed with each survey. The percentages represent the share of panelists who thought each scenario was the most likely, second-most likely, third-most likely, and least likely, in each of the three surveys. The most dramatic shifts occurred between Surveys 2 and 3, as mentioned above, but there was also a sizable shift between Survey 1 and 2 for the “More City Living and Lots of Traffic” (car-centered/higher-density) scenario. It is possible that the definition of this scenario was more ambiguous than the others at the outset and that clarification in Meeting 1 changed responses.

The results of Survey 3 show less convergence than the prior two. In Survey 2, half of the panelists ranked the “More City Living and Lots of Traffic” scenario (car-centered/higher-density) as the third-most likely, but in Survey 3, more than half of respondents thought it the most likely. Responses for this scenario also became more scattered, with a sizable minority ranking it second-most likely as well. It could be that a greater common understanding of all scenarios led to this shift, but it also seems possible that a panelist’s single, salient comment in Meeting 2—that California could aim for the “Easy to Get Around without a Car” scenario but “sleepwalk” into the “More City Living and Lots of Traffic” scenario—might have influenced participants’ thinking. This result reflects a strength of iteration in our Delphi-like method: participants have the opportunity to reflect on others’ input and change their response thoughtfully without being put on the spot (since the question in Survey 3 followed Meeting 2).

Modifying the Delphi: Lessons Learned

The traditional Delphi method poses specific questions of its panelists to get a range of responses that narrows with iteration. For this project, however, we developed four scenarios in advance and examined their likelihoods, policies that might lead to or result from each, and obstacles to each. Defining the scenarios de novo with a panel would have been time-consuming, and although we did not try it, the traditional Delphi method does not appear to be well-suited for “from-scratch” scenario development. Iteration, not ideation, is the Delphi method’s best use.

Our hybrid policy Delphi method for identifying convergence on panelists’ opinions differs from the classic Delphi and other Delphi-like methods in another way: whereas the traditional Delphi and hybrid Delphi methods use iterative questioning to narrow the range of responses to a specific problem, and the policy Delphi does not narrow responses at all, our hybrid policy Delphi used iterations primarily to narrow the scope of considerations. For example, we first discussed all four scenarios but then decided to focus on the panel’s most desirable (but least likely) scenario for further discussion in Meeting 2.

What Worked Well

Our hybrid policy Delphi combined online surveys completed individually with (virtual) meetings. This retained the benefit of private surveys while also allowing panelists to digest survey results with the group. While we did not see any evidence that seeing survey results changed opinions within the meetings, their presentation possibly contributed to the subsequent shifts in responses between Surveys 2 and 3.

The meetings did sacrifice panelists’ anonymity, which is not trivial. As described above, individuals’ reputation, rank, or force of opinion risks “halo” and “bandwagon” effects ( 23 ). Similarly, panel meetings also carry the risk of “direct confrontation” ( 7 ) discussed earlier. Mixing panelists in each of the two sessions to accommodate panelists’ schedules may have reduced any bandwagon effects, or influence by individuals with higher status or who were more assertive.

Despite these risks and potential shortcomings, the panel meetings did offer at least one significant advantage to our research: they allowed us to hear specific themes that developed in conversation. As noted above, in Meeting 2, a discussion about inequitable outcomes in some scenarios became an exchange about the more fundamental problem that trust in governmental action is lacking but is necessary for achieving any deviation from the status quo. This topic (and political science problem) is not one we anticipated in our urban planning study, yet it emerged as a major finding in our study that most of our panelists believed this to be a significant factor affecting possible scenario outcomes—one we likely would not have heard about without panel discussion. Thus, for our exploratory study, our hybrid policy Delphi approach yielded richer results than a traditional Delphi process would have.

Limitations

This expansive, fast-moving project had multiple objectives, and our hybrid policy Delphi method was well-suited for some but not others. For instance, we wanted to learn more about why panelists answered the way they did in Survey 1, and the discussion in Meeting 1 facilitated that better understanding. However, there was no iterative questioning that could lead to convergence. For projects in which it is important for rationales to be understood and to “match” the survey response, such questions could be better asked on the survey itself, through careful wording or in individual follow-up interviews.

Panel discussions are inherently difficult to direct, especially when enthusiastic experts are engaged on topics as wide-ranging and expansive as ours. In Meeting 2, for example, we intended to narrow the focus to the challenge of implementing the “Easy to Get Around without a Car” scenario. While the panel discussed this, panelists also identified new challenges that broadened discussion in ways that might have complicated convergence. At the same time, it was in this discussion that the major finding about trust in government emerged. Considering these linked limitations and benefits, discussion of the sort we undertook may be more suitable for projects in which it is not critical to achieve convergence or a narrower range of responses. Indeed, our study did not aim to test a pre-established hypothesis, which could have been more difficult using our method.

Future Directions

The Delphi method is a valuable tool for arriving at a stable, often narrower range of responses among experts on a defined question that requires (or benefits from) a range of initial responses. The hybrid Delphi method similarly enables a narrower range of proposals (from a broader range of inputs sourced from initial focus group and NGT activities). The policy Delphi is a useful tool for exploring (and even expanding on) a range of options. Our hybrid policy Delphi used Delphi-type iteration and NGT-style activities to both explore emergent themes and track respondents’ change of opinion.

The results of our hybrid policy Delphi investigation informed a California 100 report on policies and future scenarios ( 20 ), which the California 100 Initiative publicized, and two UCLA Institute of Transportation Studies reports ( 24 , 25 ).

Based on our experience with our own variation of the Delphi method, we would recommend it on future projects that:

• have sufficient time to allow for multiple iterations of interrogation;

• concern a topic whose conditions are unlikely to change over the duration of the project;

• have a high degree of uncertainty involved, a need for forecasting, or both;

• have predefined scenarios to evaluate;

• do not require brainstorming or ideation;

• would benefit from expert opinions.

Backcasting proved to be effective at focusing panelists’ attention on catalysts and causes of future conditions rather than on the uncertainty of a future condition. While backcasting complemented our method, it is largely independent of it, and we believe backcasting could be used effectively with other variations of the Delphi method. Our project used two steps—forecasting and backcasting. On a similar future project, we might incorporate four steps to explore future scenarios.

1. Positive forecasting: Panelists assess current conditions and predefined “business as usual” future scenarios.

2. Positive backcasting: Panelists imagine each “business as usual” scenario existing in the year 2050 and are asked what policies, conditions, or both, needed to remain in place to have arrived at this scenario.

3. Normative forecasting: Panelists imagine what the future would look like if a given (alternative) policy or condition were put in place today.

4. Normative backcasting: Panelists imagine an ideal or desirable scenario in 2050 and are asked what present-day changes in policies, conditions, or both, would be needed to support that “what if.”

Based on our experience with the first two steps, we expect that these four steps would result in greater definition of potential scenarios and a better overall understanding of the connection between policy choices and specific scenario outcomes (i.e., what policies in what conditions are most important for attaining the most desirable scenario).