Abstract
The regional learning environment (RLE) embeds planning student learning in a real-world participatory planning process. This study investigates the effectiveness of the RLE and its “boundary crossing” design characteristics with respect to student learning. The quasi-experimental study combines a quantitative competence test (N = 225) with qualitative student and teacher learning reports. The RLE, in general, stimulates planning students’ learning. “Working in multidisciplinary student groups” and “high coaching intensity” have specific added value for learning. “Intense student–stakeholder collaboration” does not significantly improve learning; however, qualitative data indicate this to be a powerful boundary crossing design principle of the RLE.
Keywords
The Need for Effective Learning Environments in Planning Education
Professional planners need to cooperate and facilitate cocreation with a variety of stakeholders from different disciplinary fields and with different perspectives. Planners must be able to operate across different practices and, therefore, need “boundary crossing” competence (Akkerman and Bakker 2011; Walker and Nocon 2007). Consequently, planning education needs learning environments that effectively stimulate boundary crossing. Up to now, the effectiveness of learning environments in planning education from a boundary crossing perspective has not been demonstrated. This study investigates the effectiveness of the real-life multistakeholder regional learning environment (RLE), designed from a boundary crossing perspective, for planning students’ learning.
Scholars are well aware of the need to incorporate real-world planning practices in planning curricula to prepare future students for their profession (e.g., Angotti, Doble, and Horrigan 2011; Dalton 2007; Edwards and Bates 2011). Indeed, different authentic learning environments have been developed, that is, learning environments that represent the contemporary planning practice and that provide students with the opportunity to develop their professional competence in a real-life context (Newmann and Wehlage 1993). Well-known examples are the planning studio and service learning environments (Angotti, Doble, and Horrigan 2011; Long 2012).
Several studies have portrayed learning experiences in studios, service learning, and in mixed variants of these two, resulting in suggested guidelines for the educational design of these learning environments (e.g., Balassiano 2011; G. Harris 2004; Higgins, Aitken-Rose, and Dixon 2009; Sletto 2010; Winkler 2013). However, these studies provide only limited, anecdotal evidence for the effectiveness of these authentic learning environments in terms of student learning. More specifically, evidence lacks for the effect of typical learning environment design characteristics (e.g., working with multiple stakeholders) that represent the current planning profession, on student learning. Such evidence is needed to develop education programs that meet the demands of the current profession (Young 2009), as well as to support evidence-based design and pedagogy of authentic learning environments in planning education (Németh and Long 2012; Roakes and Norris-Tirrell 2000).
The multistakeholder RLE is a new authentic learning environment in Dutch planning education programs (Foorthuis, Lutz, and Rippen 2012; Meijles and Van Hoven 2010). A unique feature of the RLE is that student learning is embedded in a real multistakeholder participatory planning process, aimed at stimulating both student learning and “regional learning.” As such, the RLE represents the current planning profession (Albrechts 2013; Boelens and De Roo 2014). It requires students to “cross boundaries” between multiple disciplines and multiple perspectives.
This quasi-experimental study explores, in a mixed method pre- and posttest design, the effectiveness of the RLE for planning students’ learning. Moreover, the study analyzes whether “working in multidisciplinary student groups” and “working in close collaboration with multiple external stakeholders” enhance this learning. These two characteristics of the RLE are specifically meant to stimulate students to cross boundaries, and are therefore referred to as the “boundary crossing learning environment characteristics.”
The results of this study highlight the effectiveness of the new RLE for planning students’ learning in view of current professional requirements for planners (Dalton 2007; Frank et al. 2014). These results may guide the future design and pedagogy of RLEs and of various other authentic learning environments in planning education, considering their parallels. On the curriculum level, insights obtained in the study are relevant to develop planning curricula that show an increasing outbound focus, establishing long-term partnerships between universities and the outside world to build on social relevance and civil engagement (e.g., Balassiano and West 2012; Winkler 2013).
We will first outline the theoretical considerations that guided this study and explain the importance of boundary crossing in the professional practice of planners. Next, the educational design of the RLE will be described, including how boundary crossing is explicitly stimulated in the RLE.
Coevolutionary Planning and the Rising Need for Boundary Crossing Planners
The context in which planning operates has changed considerably over the past three decades. Society has become increasingly globalized and networked, and faces complex issues with unpredictable changes in land use systems through multiple scales. Planning processes worldwide have transitioned from an exclusive governmental affair to complex, coevolutionary multistakeholder processes (Albrechts 2013; De Roo and Silva 2010). Planning as a profession responded, and still responds, to these changes by moving beyond more structuralist planning approaches, toward developing and applying approaches that try to address the “wicked” character of current issues and processes (Albrechts 2013; Boelens and De Roo 2014; Booher and Innes 2002; Boonstra and Boelens 2011; Forester 1999; Healey 1997; Rittel and Webber 1973). All these approaches share the idea that tackling planning issues requires taking into consideration the perspectives of multiple actors with diverse backgrounds, values, and interests and who have multiple understandings and interpretations of reality (Albrechts 2013; Domingo and Beunen 2012; Healey 1997, 2003). Traditional planning and decision-making approaches that were commonly focused on achieving explicit goals gradually move toward working on “communicative ideals where planning actions enhance the achievement of consensus on perceived situations” (Boelens and De Roo 2014, 43).
Planners increasingly become participating mediators (De Roo and Silva 2010), also called “social entrepreneurs” (Boelens and De Roo 2014, 19), instead of the more technical facilitators from the times of governmental led blueprint and strategic planning practices (Healey 2003; Nienhuis, Van Dijk, and De Roo 2012). Nowadays, planners are expected to mediate the coevolutionary processes by sharing perspectives of multiple stakeholders and situations and supporting the cocreation of new communicative and situational knowledge. This mediating role requires new competencies of planners, and the development of those in their education. Alongside long-established technical, communicative, and facilitating competencies (Dalton 2007; Healey 1997), more mediating competencies become relevant for planners. Examples of these new competencies, recently identified by planning scholars, are being able (1) to quickly identify and understand multiple disciplines, cultural traditions, interests, values, and perspectives; and (2) to admit to and intertwine these; (3) to cultivate a safe, respectful and stimulating collaborative climate; and (4) to support the reflection on process, products, and performance and mutually learn from each other to achieve shared understanding and reasoning (e.g., Balassiano 2011; Domingo and Beunen 2012; Thomas 2012; Umemoto 2001).
The concept of “boundary crossing competence” encapsulates these new competencies and as such is a key competence of current planners. Boundary crossing competence is defined here as the ability to manage and integrate multiple discourses and practices across different sociocultural boundaries (Akkerman and Bakker 2011; Umemoto 2001; Walker and Nocon 2007).
The RLE is designed to explicitly stimulate boundary crossing in planning education. The next section describes the RLE in more detail and shows how its design reflects the boundary crossing planning practice. To position the RLE in the planning education landscape and explicate its expected added value, the RLE is compared to planning studios and service learning environments.
The Authentic Multistakeholder Regional Learning Environment
Regional Planning and the RLE
In the Netherlands as well as in other European countries, regions have become a subnational, supralocal focus point for spatial development (Haughton and Counsell 2004; Lagendijk 2001). In 2005, a regional partnership between nine municipalities, two provinces, two water boards, and several educational institutes in the Dutch Peat District (a region in the northern part of the Netherlands), developed and experimented with a regional learning arrangement to collaboratively face regional developmental issues. This experiment was the starting point for the further development of the RLE, supported by the Dutch Ministry of Economic Affairs (Foorthuis, Lutz, and Rippen 2012).
The RLE is meant to be a catalyst for “regional learning”—a learning and working community in which students, teachers, researchers, policy makers, members of NGOs, entrepreneurs, and/or citizens cooperatively face complex issues of regional spatial development while mutually learning (Foorthuis, Lutz, and Rippen 2012; Meijles and Van Hoven 2010; Scholz and Steiner 2015). The RLE has been established so far in thirteen Dutch, mostly rural, regions that are characterized by having high landscape and biodiversity values, recreational pressure, and economic and demographic decline. The RLE aims to facilitate the collaborative creation of new knowledge toward sustainable regional development. Educational institutions are always one of the partners in the RLE. Appendix A (Supplementary Material) provides a comprehensive illustration of the state of affairs in one of the studied RLEs.
Educational Design of the RLE
From an educational perspective, the RLE is an authentic, multistakeholder learning environment. Here, authentic refers to an instructional approach in which learning takes place in a real-life context to allow students to meaningfully construct knowledge and develop skills for real life (e.g., Herrington and Oliver 2000; Newmann and Wehlage 1993). Multiple stakeholders in this study are defined as a collection of persons or parties, from (semi)government, the business sector, NGOs, research bodies, and/or society, who either have a stake in the issue(s) at hand, and/or participate in the cocreation of knowledge (Freeman 1984).
The RLE is always typified by the following learning environment design characteristics:
The RLE’s general aim is twofold, namely (1) to support students’ and other parties’ learning in terms of both domain-specific expertise and professional skills development, and (2) to contribute to sustainable regional development.
Students are exposed to real trans-disciplinary problems in a real-world situation (Scholz and Steiner 2015); i.e., regional planning problems identified and commissioned by an external client. Working on the assignment engages students in authentic, complex tasks. Appendixes A and B (Supplementary Material) show examples of assignments that have been carried out in the RLEs included in this study.
Knowledge is collaboratively constructed between students, between students and their teachers, and preferably between students, teachers, and multiple stakeholders.
Students work in student groups.
Working in the RLE results in a realistic product that has value for the external client(s) and contributes to regional development. In practice, deliverables vary depending on agreements between the students, their teachers, and the external client(s). Next to the product for the client, process reflection reports are required as a deliverable. Product and process deliverables are both part of the assessment, but assessment criteria and procedures vary between RLEs.
The teacher’s role is to facilitate and/or coach the learning process rather than to transfer knowledge as an expert. Additionally, the teacher is a “learner” himself, working in an almost equal relationship with the students to collaboratively tackle complex regional problems.
The RLE preferably has two additional design characteristics that are expected to explicitly stimulate students to work and learn across boundaries between different disciplines and different perspectives (Foorthuis, Lutz, and Rippen 2012; Meijles and Van Hoven 2010):
Students work in multidisciplinary student groups, which means that the groups consist of students from different study programs, that is, disciplines.
Students collaborate intensively with multiple stakeholders to enable the inclusion of diverse perspectives and interests when solving complex, transdisciplinary regional problems.
These characteristics are referred to as the boundary crossing learning environment characteristics, upon which this study specifically sheds a light.
The RLE Compared to the Planning Studio and Service Learning
The RLE shares educational characteristics with the commonly used authentic planning studio, service learning environment, and mixed variants of these. However, the RLE characteristics purposely differ from and add to these existing authentic learning environments when it comes to explicitly addressing boundary crossing. Providing insights in similarities and differences between the characteristics of the RLE, studio, and service learning enables readers to use the results of this study for the further educational development of both the RLE and the planning studio (Higgins, Aitken-Rose, and Dixon 2009; Long 2012; Németh and Long 2012) and of service learning (Angotti, Doble, and Horrigan 2011; Roakes and Norris-Tirrell 2000; Sletto 2010).
For the RLE, planning studio, or service learning, no internationally standardized pedagogy has been described. However, all three learning environments contain elements of social constructivist learning theories (Vygotsky 1978), authentic learning (Herrington and Oliver 2000), and experiential learning (Dewey 1938). Students collaboratively construct knowledge together with other students, teachers, and society; they work on real-world assignments and learn through experience (Watson 2001).
The planning studio aims to expose students to a professional experience by introducing them to real-world problems in a quasi-real–world situation. Learning outcomes, although often not clearly described (Long 2012; Németh and Long 2012), focus on student learning in reference to the integration and application of theory into practice, and development of various practical skills (Frank et al. 2014; Németh and Long 2012). The planning studio mostly starts with an open-ended complex problem. The problem takes account of current issues in the real world, but is mostly described by the teacher, and not always offered by real external clients (Balassiano and West 2012). Students work individually or in groups. The planning studio is finalized with a final presentation to faculty and/or the client, sometimes but not necessarily with an authentic product that offers added value for the client. As a remnant from design studios, assessment often takes place through several rounds of formative assessments (jury-crits) involving presentations of students and feedback from diverse assessors (peers, tutor, and experts). Although the intention is to base assessment on absolute, criterion-referenced quality standards, standards are still lacking (Németh and Long 2012). The teacher fulfills the role of expert, providing a flexible though strong instructional frame. Recently, scholars have advocated a broader role of stakeholders and to run studio courses with mixed groups of students from various disciplines (Balassiano 2011; Long 2012).
We believe that the RLE adds three typical design characteristics to the current mainstream characteristics of the planning studio; that is, (1) students always work on real-world transdisciplinary assignments identified by and relevant for one or more external stakeholders; (2) students work in groups, preferably in multidisciplinary groups; and (3) students, their teachers, and multiple external stakeholders collaboratively construct knowledge and mutually learn.
Service learning is defined as a pedagogy that aims to integrate meaningful community service with formal education (Giles and Eyler 1994; Jacoby 2014). Next to the objective of strengthening communities, student learning outcomes focus on civic responsibility, critical problem solving skills, adaption to challenging and unexpected situations, and critical reflexivity (Roakes and Norris-Tirrell 2000; Sletto 2010). In service learning, students learn how to use the knowledge and skills from a specific course to provide service to society. Learning outcomes are in most cases closely connected to those of a related theoretical course. Students work individually or in groups on real-world, societal assignments that ideally are relevant for the community at the local or sublocal level (G. Harris 2004; Sletto 2010). The assignments are often, but not necessarily, multidisciplinary. Service learning allows students to work closely with community organizations on applied field projects. Knowledge construction explicitly takes place through structured reflective thinking, mainly between the students and their teacher. The final product provides added value to the community and is presented to the client. Assessment criteria and procedures are not commonly defined (S. C. Harris and Irazábal 2011). Teachers and students work in a master–apprenticeship model in which the teacher fulfills the role of facilitator of the process and reflection (G. Harris 2004; Roakes and Norris-Tirrell 2000; Schön 1987; Sletto 2010). In planning education practice, many different service learning courses have been developed and evaluated, for example, the service learning studio (Forsyth, Lu, and Mc Girr 1999; Winkler 2013) and the research seminar (G. Harris 2004). In these service learning variants, both faculty members and students maintain a close relationship with external organizations and deliver a formal community service report.
Service learning environments and the RLE share the characteristic of students working on a real-world assignment with actual relevance for one or more external parties. In both service learning and the RLE, contact with external parties is a prerequisite for a sufficient project result, and an oral presentation of the final product to client and/or other external parties is part of the assessment. However, RLE assignments always have a transdisciplinary character and are always demand driven, while in service learning they can also be launched for the purpose of a (service learning) course. In addition, RLE students preferably work in multidisciplinary student groups. So far, working in multidisciplinary student groups is not regarded as a prerequisite for service learning. The process of reflection to transform an experience into learning, and thus stimulate reflective practices, is, to date, less pronounced in the RLE, while it is a core element of service learning.
To sum up, the RLE adds to the planning studio and to service learning, a set of typical boundary crossing characteristics namely that students in the RLE (1) always work on a demand-driven, transdisciplinary assignment identified by one or more external actors, and (2) students always work in groups, preferably multidisciplinary student groups. Alternatively, the element of systematic, critical reflection to explicate and learn from experiences across the boundaries, an explicit characteristic of service learning, could add to the boundary crossing learning potential of the RLE.
Studying the Learning Potential of Regional Learning Environments
The new RLE is increasingly used in planning education in the Netherlands. However, still undiscovered is the effectiveness of the RLE in terms of students’ learning and/or competence development, and the actual added value of learning in multidisciplinary student groups and with multiple external stakeholders, that is, the two typical boundary crossing characteristics. Investigating the educational effectiveness of the RLE is important to confirm the hypothesized added value of this new learning environment, and add to the small record of systematic, large-scale, quantitative studies on the effectiveness of studios and service learning (Angotti, Doble, and Horrigan 2011; Long 2012).
This study addresses four research questions:
To what extent do RLEs stimulate planning students to develop competencies identified as relevant for working in an RLE setting?
Do the two boundary crossing characteristics of RLEs (multidisciplinary student groups and intensive student–stakeholder collaboration) enhance student learning?
What learning outcomes do teachers perceive as a result of working in multidisciplinary student groups and with multiple external stakeholders?
What preconditions do teachers perceive as necessary for exploiting the learning potential of RLEs?
Method
This study has been carried out in a quasi-experimental mixed method pre- and posttest design combining quantitative data on students’ competence development with qualitative student and teacher reports on learning outcomes of the RLE.
Participants
Five RLEs implemented in different planning education programs were monitored: three in academic study programs (n = 81, 64, 52) and two in professional higher education programs (n = 15, 13). Students in all the RLEs worked in student groups of mostly five or six students. Each student group worked on a different project assignment of which the results were meant to contribute to the development of the respective region (see Appendix B, Supplementary Material). Table 1 shows the general characteristics of the five RLEs and mean age and gender of the participating students.
Characteristics of and Developed Competencies in the Studied Regional Learning Environments.
p < 0.05.
d < 0.2 = small, d ≈ 0.5 = medium, d > 0.8 = large.
To answer research questions 3 and 4, twenty-five teachers participated in a semistructured workshop. All teachers were experienced in the development and implementation of RLEs, including the five monitored RLEs.
Classification of Students on Boundary Crossing Learning Environment Characteristics
All studied RLEs met the general educational characteristics. The RLEs differed regarding educational level, study load, size of the student groups, number of students involved (Table 1) and focus in content (Appendix B, Supplementary Material). To answer research question 2, individual students in every RLE were classified for the two studied boundary crossing design characteristics. A student was classified as “working in a mono-disciplinary student group” when he or she worked only with other planning students. A student was classified as “working in a multidisciplinary student group” if he or she worked with students from other study programs (e.g., landscape architecture, environmental sciences, and management and economics). A student was classified as “working on a low level of multistakeholder collaboration” if he or she only read information about the stakeholders and their opinions without contacting them personally or only asked the stakeholders informative questions that were answered without any discussion. A student was classified as “working on a high level of multistakeholder collaboration” if he or she discussed project-related issues with one or more stakeholders or worked together in collaborative working sessions with one or more stakeholders. The attribution of the classification for multistakeholder collaboration was based on three weighed ratings: (1) teachers’ observations of the level of stakeholder collaboration during the projects (rating low or high), (2) researchers’ observations of the level of stakeholder collaboration during the projects (rating low or high), and (3) students’ reported level of stakeholder collaboration in an evaluation questionnaire. In this questionnaire, the students ticked off how many stakeholders they collaborated with and how the collaboration took place (ranging from “finding information on the Internet about this stakeholder” to “collaborating with the stakeholder via personal contact during the whole project”). A scoring scheme translated these answers into six possible relationships for which scores 1–3 were classified as “low level of collaboration” and scores 4–6 as “high level of collaboration.”
Classification of Students on Coaching Intensity
Besides classifying for the above two typical learning environment characteristics, a classification for the independent variable “coaching intensity” was added. This was done as observations of the RLEs illuminated coaching intensity as an important varying variable between the RLEs. Students in RLEs who were classified as obtaining “low coaching intensity” only met their teacher coach once a week for a state-of-the-art group discussion mainly focused on the product and the process toward a final result. Students in RLEs who were classified as obtaining “high coaching intensity” followed an intense parallel coaching trajectory in which their learning experiences with collaboration across boundaries were explicitly addressed and utilized to optimize the RLE product and process. Both group and individual meetings with a teacher coach were organized purposely. Students’ classification for coaching intensity was based on the descriptions of coaching in the study manuals.
Measuring Competence Development
To answer research questions 1 and 2, a validated pre- and posttest questionnaire assessed the perceived level of domain-specific professional expertise and eight generic competencies (Table 2).
Competencies as Assessed in Pre- and Posttest. a
For a description per competency, see Appendix C (supporting information).
This total of nine competencies was identified as relevant for working in RLEs by (1) a group of fifty-six academic planning students working in an RLE project that was used as a pilot project for the setup of this study (June 2011), and (2) ten planning teachers experienced in working in RLEs, including the five monitored RLEs. Their selection was based on a list of twenty-five generic competencies for Dutch vocational education (COLO 2006) as developed on the basis of the SHL Universal Competency Framework (Bartram 2011; www.cebglobal.com). Students and teachers were asked to rate each of the twenty-five competencies on relevance for working in the RLE on a four-point scale (1 = certainly not relevant to 4 = certainly relevant). Nine competencies, including “domain-specific professional expertise” were convincingly rated as certainly relevant by the respondents. For this reason, these nine were selected for the purpose of this current study. The questionnaire, part of the validated competency measurement instrument COM (Khaled et al. 2014), consisted of a description of each of the nine competencies (supporting information Appendix C) and four to six performance indicators per competency derived from this description. In both pre- and posttests, students awarded themselves a score for each performance indicator on a ten-point scale. A competence mean score was based on students’ rating of the four to six performance indicators per competency. At the start of the project, directly after being informed about their project assignments, students filled out the pretest. At the end of the project, right after the final presentation of the project result, they filled out the posttest. The scales were reliable (α > 0.80). RLEs were compared based on their development scores between pre- and posttest (dependent variables).
Measuring Students’ Other Learning Outcomes
To answer research question 2 in terms of “other learning outcomes,” the posttest asked students the open question “What did you learn more from your RLE project? Please write down as many of your ideas as possible regarding your learning in this project.”
Measuring Teacher Perceptions on Student Learning in the RLE
During a semistructured workshop, teachers first individually and then in five groups of four to six participants wrote down experienced learning outcomes typically resulting from (1) working in multidisciplinary student groups and (2) multistakeholder collaboration in RLEs. Every statement was individually written on a Post-It to allow coding and counting. Additionally, teachers wrote down statements regarding what they experienced as preconditions for optimal learning in the RLE.
Analysis
Paired sample t tests were used to calculate development of the students per RLE on the nine competencies (research question 1). Effect size for the paired sample t tests was measured in Cohen’s d with d <0.2 showing a small effect, d around 0.5 showing a medium effect, and d >0.8 showing a large effect (Cohen 1988). Three multivariate general linear models (GLMs) compared competence development across RLEs using mono-/multidisciplinary student groups, low/high multistakeholder collaboration, and low/high coaching as independent variables (research question 2, quantitative part). Effect size for the GLMs was measured in partial eta-squared (partial η²), with partial η² ≈ 0.01 showing a small effect, partial η² ≈ 0.06 showing a medium effect, and partial η² ≈ 0.14 showing a large effect (Cohen 1988).
The reported learning outcomes from the students (research question 2, qualitative part) were deductively coded on “referring to working in multidisciplinary student groups,” “referring to collaborating with multiple stakeholders,” or “referring to other learning environment characteristics” (Miles, Huberman, and Saldana 2014). The interrater reliability kappa (κ) of this coding step was 0.85 (two independent raters), which represents an almost perfect strength of agreement (Landis and Koch 1977). After coding, the percentages per code were calculated for students that worked in either mono- or multidisciplinary student groups or with a low or a high level of stakeholder collaboration (i.e., the two typical boundary crossing learning environment characteristics). Finally, two illustrative examples of learning outcomes per category were chosen.
The statements resulting from the teacher workshop were first coded by the same two independent researchers that coded the student learning outcomes. They used deductive coding (Miles, Huberman, and Saldana 2014) on learning outcomes resulting from either multidisciplinary group work or multistakeholder collaboration (research question 3), and open, inductive coding (Glaser and Strauss 1967; Miles, Huberman, and Saldana 2014) on preconditions for utilizing the learning potential of RLEs (research question 4). They clustered the codes into meaningful learning outcome and precondition categories, after which axial coding was used (Strauss and Corbin 1998) with an interrater reliability к of 0.88. Additionally, learning outcome statements were subcoded (Miles, Huberman, and Saldana 2014) as representing one of the nine competencies as measured in the pre- and posttest or “another learning outcome.” The interrater reliability к of this coding step was 0.84, which also shows an almost perfect strength of agreement (Landis and Koch 1977).
Results
In response to research question 1, results showed differences in competence development between the five RLEs. Competence development ranged from no significant development in RLE 5, to significant development of three (RLE 4), four (RLE 3) and five (RLE 2) competencies, to significant development of all competencies in RLE 1 (p < 0.05, see Table 1 for developed competencies and effect sizes). Analyses did not show statistical differences in competence development per RLE for gender and age. Although not statistically analyzed, the data in Table 1 suggest the absence of systematic differences in competence development for study load and educational level. In the four RLEs that showed competence development, “domain-specific professional expertise” always developed and always developed the most. The competency “collaborating and discussing” also developed in all four RLEs that showed competence development.
In response to research question 2, regarding students’ competence development resulting from three typical learning environment characteristics, GLM analyses showed a large (partial η² ≥ 0.14) positive multivariate effect of the learning environment characteristics:
Working in multidisciplinary student groups (F[9, 113] = 2.432, p < 0.05, partial η² = 0.162): Competence development scores (posttest minus pretest) were significantly higher for students working in multidisciplinary student groups for the two competencies “deciding and initiating activities” and “collaborating and discussing.” The trend showed a higher, but nonsignificant, development score for all competencies of students working in multidisciplinary student groups, except for “domain-specific professional expertise.”
A high coaching intensity (F[9, 113] = 2.373, p < 0.05, partial η² = 0.159): Development scores (posttest minus pretest) were significantly higher for students working with a high coaching intensity for four competencies. These four competencies were “deciding and initiating activities,” “showing attention and understanding,” “planning and organizing,” and “collaborating and discussing.” In addition, the trend showed a higher, but nonsignificant, development score for all competencies of students who worked with a high coaching intensity.
Multistakeholder collaboration showed no significant multivariate effect, but it did show a trend of higher, nonsignificant, competence development scores for all competencies of students working on a high level of stakeholder collaboration.
In response to research question 2 regarding students’ other learning outcomes related to the typical learning environment characteristics, we found that students working in multidisciplinary student groups referred in 14 percent of the reported learning outcomes to learning from multidisciplinary group work. Students working in mono-disciplinary student groups did so in 7 percent of the reported learning outcomes. Students working on a high level of multistakeholder collaboration referred in 26 percent of the reported learning outcomes to learning from multistakeholder collaboration. Students working on a low level of multistakeholder collaboration did so in 27 percent of the reported other learning outcomes (see Table 3 for percentages and examples of reported learning outcomes).
Percentages of Students’ Reported “Other Learning Outcomes” Referring to Multidisciplinary Group Work and Multistakeholder Collaboration per Category of Learning Environment Characteristics.
In response to research question 3 on teachers’ perceived learning outcomes, we found six typical learning outcome categories resulting from working in multidisciplinary groups (Table 4). Eighty-five percent of the teacher statements in these categories related to one of the measured competencies. Fifty-three percent of these 85 percent related to “domain-specific professional expertise,” including statements like “development of domain-specific professional expertise through explaining your own expertise knowledge to others.” Next, we found eight typical learning outcome categories resulting from multistakeholder collaboration (Table 4). Fifty-nine percent of the teacher statements in these categories were related to the measured competencies (e.g., “learning to acknowledge diverse, often conflicting interests” related to “showing attention and understanding”). Other statements referred to additional learning outcomes such as “developing insight into the profession” and “finding ways to effectively deal with diverse interests and perspectives during the whole process.”
Learning Outcome Categories Typically Resulting from Multidisciplinary Group Work and Multistakeholder Collaboration.
In response to research question 4, we deduced six categories of preconditions for exploiting the learning potential of RLEs (Table 5). The largest numbers of statements were attributed to the precondition categories “intensive process coaching” (ten of thirty-two) and “real collaboration with the stakeholder” (seven of thirty-two).
Categories of Preconditions for Optimal Learning in the RLE.
Discussion
Results for research question 1 show the RLE to be an effective learning environment for developing domain-specific professional expertise, as well as various generic competencies, all identified as relevant for working in RLEs by participating planning students and teachers. Four of five studied RLEs show significant competence development for three to nine competencies. Domain-specific professional expertise significantly developed and was the most developed competency in all four RLEs that showed development. This is a relevant finding with regard to concerns about students’ development of professional knowledge and expertise in innovative and authentic learning environments for higher education such as competence-based education (Biemans et al. 2004) and problem-based education (Schmidt, Vermeulen, and Van der Molen 2006). The competency “collaborating and discussing” also developed in all four RLEs that show development. This confirms the potential of the RLE for developing collaborative capacities of planners (Healey 2003; Seltzer and Ozawa 2002).
Answering the second research question, the study found that the typical RLE “boundary crossing” learning environment characteristic of working in multidisciplinary student groups enhanced students’ competence development. Working in multidisciplinary compared with monodisciplinary student groups led to higher competence development scores, and to more student-reported learning outcomes referring to learning from working in a multidisciplinary setting. In response to research question 3, the study found that RLE teachers also identify student learning outcomes resulting from multidisciplinary group work, in most cases related to one of the measured competencies, which underwrites the quantitative findings. This combination of quantitative and qualitative exposure of learning from working in multidisciplinary groups supports the added value of this boundary crossing characteristic of the RLE (Akkerman and Bakker 2011).
The second typical boundary crossing learning environment characteristic, working on a high level of multistakeholder collaboration, did not significantly enhance students’ competence development more than working on a low level of collaboration. Also, the number of reported student learning outcomes referring to learning from multistakeholder collaboration did not differ for a high or a low level of multistakeholder collaboration. The lack of an enhancing effect of intense multistakeholder collaboration for planning student learning is disappointing because this disputes the RLE to be an effective boundary crossing learning environment for upcoming planners. Remarkably, the percentages of the total amount of student learning outcomes that referred to learning from multistakeholder collaboration (26 and 27 percent of the total number of excerpts) were much higher than the percentages of student learning outcomes that referred to learning from working in a multidisciplinary setting (14 and 7 percent of the total number of excerpts). This could be caused by the fact that students are excited by what they learn from “real” external partners (in case of low intense stakeholder collaboration from the external client), more than what they learn from their peers. This suggests at least some kind of impact of working with external stakeholders on learning. Also, the participating teachers identified student learning outcomes resulting from multistakeholder collaboration. Different from multidisciplinary group work, these learning outcomes were in only half of the cases related to one of the measured competencies. The COM, as used in this study, may not have measured the things that students actually learn from working with external stakeholders. This could have contributed to the lack of a significant quantitative effect of multistakeholder collaboration on student learning. Probably, students learned other things than, or next to, the competencies measured in the COM. A second reason for the lack of a significant effect of intense collaboration with multiple stakeholders on competence development could be that learning with and from stakeholders was not explicated as a learning objective and typical learning activity in the studied RLEs. We were supported in this idea by the fact that the students who participated in the selection of competencies being relevant for working in the RLE before the start of this study (see Method section and Table 2) did not select the competency “building relationships and networking” as being relevant for working in the RLE. Apparently, they seemed not to be aware, at least at the start of their project, of the relevance of this competence for working in the RLE. A third reason for the lack of effect could be that stakeholder collaboration was not explicitly addressed in instructions or object of coaching during the RLE process. Just being embedded in a multistakeholder environment seems to not guarantee explicit learning thereof. Not explicitly aiming for and coaching student–stakeholder collaboration in the studied RLEs contrasts with the results from research question 4 where participating teachers identified collaboration with stakeholders to be an important precondition for optimal learning in the RLE. Overlooking these considerations, and regarding the many promising reported student learning outcomes on learning with and from multiple stakeholders, we argue that the RLE contains an unemployed learning potential of multistakeholder collaboration caused by the lack of explicating and supporting learning with and from the stakeholders in the RLE.
Results for the added independent variable coaching intensity showed that working in RLEs with a high coaching intensity compared to a low coaching intensity led to higher competence development scores. This result adds significance to the statements of teachers—from the results of research question 4—that process coaching and reflection are important preconditions for exploiting the learning potential of RLEs (Table 5).
Looking into how the majority of the students per RLE were classified (Table 1) might partly explain the differences in competence development per RLE. The two RLEs that showed most development (RLE 1 and RLE 2) both included all three typical characteristics. The two RLEs that showed the least development did not include the typical characteristics. This observed relation supports our statistical findings for the enhancing effect of the three typical characteristics and may explain differences in competence development between the studied RLEs.
Two limitations of the study may have influenced the results. First, competence development in this study was mainly measured by students’ own perception thereof. This was a consequence of differences in assessment strategies per RLE, and as such the nonavailability of comparable assessment data. It has recently been advocated that self-reports provide results that are as reliable as third-party data (Braun et al. 2012; Chan 2009). However, to be cautious, we decided to combine the quantitative measurements with qualitative student data on learning outcomes of the RLE, and with teacher data on student learning outcomes and preconditions for learning in the RLE. Next, all measurements took place before the students were graded on their RLE work, and both oral and written instructions stressed that the results of the measurements did not influence students’ RLE grading. The second limitation of the study relates to its quasi-experimental design. Since the study was conducted in educational practice, the RLEs differed in more than the studied variables (e.g., characteristics of the assignment [perceived authenticity, urgency, complexity etc.] or willingness of the stakeholders to really collaborate with the students and show an explanatory attitude). Although we did not see systematic variations or influences of these variables in our qualitative data, observations and discussions with students and teachers, it is appropriate to state that we did not study five equal RLEs and that the results for the value of typical design characteristics cannot completely be attributed to the studied variables. However, the study obtained evidence for student learning in a quasi-experimental design, and as such meets the need for studies in planning education that go beyond the description of an educational innovation (Wu and Brooks 2011). Implications of the results for planning education and future research are discussed in the concluding section.
Conclusions and Implications
This study confirms the effectiveness of the new, authentic multistakeholder Regional Learning Environment for planning students’ learning. The study specifically shows the added value of the learning environment characteristic of “working in multidisciplinary student groups” and identifies the learning potential of “intense collaboration between students and multiple stakeholders.” These learning environment characteristics prompt boundary crossing, inherent to current planning practice. Additionally, this study found a high coaching intensity to be a preconditional learning environment characteristic for boundary crossing learning in the RLE, and to enhance students’ competence development in the RLE.
Having identified similarities between RLEs, planning studios, and service learning, this study adds to the scientific debate and its practical implications on the optimization of studio and service learning pedagogy in planning education (Angotti, Doble, and Horrigan 2011; Long 2012). First, the confirmation that working in multidisciplinary instead of monodisciplinary student groups more strongly stimulates various competencies regarded as important for professional planners provides an argument for more explicitly facilitating multidisciplinary group work in planning studio courses (Long 2012) and in service learning (G. Harris 2004; Sletto 2010). Second, the revealed importance of intense process coaching and reflection in order to facilitate and explicate learning experiences in the RLE evidentially supports the important role that reflection has been given in service learning (Roakes and Norris-Tirrell 2000; Sletto 2010). For the future improvement of coaching and reflection in the RLE, especially with regard to students’ collaboration with stakeholders, we suggest RLE teachers make use of insights in systematic reflection strategies as extensively utilized in service learning environments. Third, although this study showed insignificant effects of student–stakeholder collaboration for students’ competence development, teachers supported the hypothesized learning potential of intense collaboration with multiple stakeholders. Planning studios and service learning are recommended, just like RLEs, to further experiment with using real-world, transdisciplinary assignments and collaborative knowledge construction between students, teachers, and multiple stakeholders. This will allow us to examine when and how multistakeholder collaboration fosters student learning. We also recommend addressing student–stakeholder collaboration more explicitly in learning objectives and coaching. Part of the learning outcomes of multistakeholder processes cannot be predicted beforehand because of unexpected learning opportunities during authentic collaborative trajectories. Therefore, we suggest authentic learning environments in planning education to experiment with the allowance of “learning surprises” (Scardamalia et al. 2012) to gain insight into what students really learn from working and learning with other stakeholders. Allowing such surprises and trying to address these in learning objectives and assessment strategies could support students in their learning to plan for the “undefined becoming” (Boelens and De Roo 2014).
Looking ahead to future research and reviewing some limitations of this study, we suggest carrying out follow-up studies to strengthen the effects of this study and deepen understanding of the RLEs’ learning potential, especially of its boundary crossing characteristics. These studies are specifically required for further investigation of the unexpected, insignificant findings of “intense collaboration between students and multiple stakeholders.” Adding the competency “building relationships and networking” to the competence test would be interesting since this competency is likely to be directly influenced by working in the multistakeholder RLE. In addition, adding teacher ratings to the student reported competence levels could increase the reliability of the quantitative data. Next to the more large-scale and quasi-experimental studies as described in this article, more qualitative, in-depth studies could also shed interesting light on the actual learning processes taking place or hampering learning in the RLE. Next, intervention studies, both quantitative and qualitative, could investigate whether active support of stakeholder collaboration effects student learning in the RLE. Both follow-up studies and intervention studies can reveal our identified “hidden” learning potential of stakeholder collaboration.
Beyond an effective innovation in planning education, the RLE can be positioned as a new model of knowledge production in planning practice. This supports the idea of different planning scholars to expand the boundaries of authentic learning environments in planning education outside the educational context (Angotti, Doble, and Horrigan 2011; Balassiano and West 2012; Long 2012; Sletto 2010; Winkler 2013). The potential of this idea could be investigated by broadening the scope of effectivity studies from student learning outcomes to learning outcomes for all parties involved, and to results for spatial development.
In conclusion, RLE insights and experiences provide a range of opportunities for the future development of planning education on our way to educate collaborative, “boundary crossing” planners.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available with the manuscript on the JPER website.
Author Biographies
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
