Abstract
Low-code platforms can provide a learning environment that integrates academic theory with practical experiences, allowing students to experience real-world ISD projects. Such a pedagogy is known as work-integrated learning (WIL). In a low-code WIL course, the focus is on the practical development experiences and the delivery of a business application based on client’s requirements. Yet, the real-world ISD experiences may inhibit metacognition, which is essential for students to succeed in future learning of digital innovations. Without metacognition, students struggle to learn how to learn and be reflective about their learning. This study uses a mixed-method design and draws on Flavell’s (1979) theoretical framework to examine metacognition in the context of low-code WIL. In particular, the study examines the effect of learner-oriented factors and task-oriented factors on metacognition and its impact on learning outcomes, such as knowledge confidence, app delivery, and grade. The quantitative results indicate that metacognition is influenced by motivation, the strategic approach, and autonomy, whereas self-efficacy beliefs have no effect. The learning outcomes are also achieved. Qualitative results (students’ essays and apps) corroborate the quantitative results while also providing additional insights into the learning in low-code WIL showing the impact of the platform and agile ISD practices on metacognition. Overall, this study offers IS educators a better understanding of how low-code platforms can foster learning-to-learn while providing real-world practical experiences.
Keywords
Introduction
Globally, information systems (IS) educators seek to enhance their students’ workplace readiness because of alarming reports that graduates are not ready to work in their respective jobs (Baska, 2019; Tonkin, 2019). Employers’ concerns are that university education favours theory over practice, with students having limited practical experiences while not getting equipped with essential workplace skills (e.g. abilities for problem-solving, self-management, and flexibility) (CMI-Insights, 2021). In response, higher education adopts work-integrated learning (WIL) to strongly and explicitly provide students with practical experiences of real-world workplaces (Jackson, 2017; Patrick et al., 2009). For example, WIL in an information systems development (ISD) course can be designed so that students undertake IT consulting projects with real clients, real CASE tools, requirements, deadlines and customer acceptance testing. Thus, students experience real-world, authentic conditions of software development, including uncertainties, scoop creep, and changing dynamics of technology-driven workplaces (Hennel & Rosenkranz, 2021; Wiesche, 2021).
Scholars present low-code platforms as an innovative ISD approach to creating business applications (apps) faster and with less effort than in traditional development (Bock & Frank, 2021; Elshan et al., 2023 ). Using the low-code approach allows for addressing the need to adapt flexibly, and change IS in organizations (Krell et al., 2011). Thus, a low-code platform would provide students with real-world ISD experiences (Crumbly & Field, 2020) while facilitating technology-mediated learning (Bayerlein et al., 2021; Söllner et al., 2018; Strecker et al., 2018). Low-code platforms are praised for empowering non-IT professionals with minimal coding expertise (e.g. programming syntax, test, automation techniques, and database integration) to develop fully functional business apps (Beranič et al., 2020). Consequently, learning in low-code WIL provides students with opportunities to gain rich practical experiences.
In low-code WIL, student efforts are focused and directed towards creating a practical result – the delivery of the business app. Specifically, students must create an authentic output of value at the end of the course, which is a valuable working software that addresses the requirements and is error-free (Wang & Wang, 2021). The theoretical ISD teaching is tailored to provide students with foundational knowledge, including requirements gathering, ISD methods, and project management (Beranič et al., 2020). Yet, the teaching is focused on the app and the low-code platform, which assigns autonomy to students to master ISD challenges (e.g. client requirements or technology changes). In WIL, as in a real-world ISD project, students focus on the product, and feedback is given to address specific ISD challenges. Students seek to implement client’s demands under time pressure, leading to heightened development stress and potential cutting-corners (Licorish et al., 2022). Because a low-code WIL is designed to provide students with practical and real-world experiences, some scholars question whether WIL may introduce too much practice into the learning, which in turn takes away time and attention from students’ metacognitive thinking (Rowe, 2015; Tran & Soejatminah, 2017).
Metacognition encapsulates learning how to learn so that a person can master future innovations (Flavell, 1979). When people, including students, possess metacognition, they have the skills to learn new skills. Reflections are core to metacognition as they enable a person to think about and critically challenge their thinking (Prins et al., 2006). The extent to which a person is reflective indicates how well a person can learn new knowledge (Garner, 1987). Through metacognition, learners understand their thinking processes which in turn allows for 'deeper, more durable, and more transferable learning' (Rickey & Stacy, 2000). IS educational research highlights the importance of metacognitive skills for IS students to adapt to changing business demands (Fadel et al., 2015). Learners with metacognitive skills can optimize their learning and become successful lifelong learners (Anthonysamy et al., 2020). We know from the literature that students can develop metacognition when being taught to program software (Prather et al., 2018) or when being taught to design applications (Santoso et al., 2017). Yet, a low-code WIL context is different because of the undertaking of a real-world, authentic IT project in which students are expected to create the client app at the end of the course. Unfortunately, few studies investigate how students learn in a low-code WIL context (real client, real requirements, need to deliver), esp. which factors impact on students’ metacognitive skills.
This study addresses the gap and examines metacognitive skills in a low-code WIL context. As the overarching theory, we build on Flavell’s (1979) three-dimensional framework. The framework postulates that metacognition and the skills one can acquire are impacted by the person, the task, and the situation. Drawing on the WIL situational context of low-code ISD, we theorize factors related to the students (motivation, self-efficacy) and related to the ISD task (autonomy, strategic approach) as impactful on metacognitive skills and learning outcomes. Particularly, we use a perceptual measure of learning (students’ knowledge confidence) and two performance measures (grade and app delivery). Hence, our research question asks how students learn (and learn to learn) in low-code WIL? In particular, Study 1 asks 'What impact do learner-oriented and task-oriented factors have on metacognitive skills in a low-code WIL and in turn, what impact do metacognitive skills have on learning outcomes?'. Study 2 asks: 'How do students experience learning in low-code WIL regarding metacognitive skills and technology-enhanced scaffolds as well as the learner-oriented and task-oriented factors?'
The research deploys a two-study mixed-methods design (quantitative and qualitative) to gain deeper insights into low-code WIL and metacognitive skill building. In the first quantitative study, we test our hypotheses with 417 students in an ISD course at a large Australian university. The course is a designated WIL course by the authors’ university. Students cooperatively work with industry/external stakeholders and academics to deliver an app using the low-code platform Mendix. In the second study, we draw on qualitative data from students’ metacognitive reflections (reflective essays) and the low-code apps they developed to understand better how students learn to learn new knowledge and skills. Findings from Study 1 show that the students’ motivation, autonomy, and the strategic approach to app development aid in building metacognitive skills; however, the students’ self-efficacy was not significantly correlated. Findings from Study 2 provide insights into low-code WIL learning with a particular focus on the technology-enhanced scaffolds (Janson et al., 2020; Sharma & Hannafin, 2005). It allowed us to show that students successfully created metacognition, esp. when drawing on technology-enhanced scaffolds.
The research contributes to the IS education literature by demonstrating the opportunities for integrating theory with practice via simulated low-code WIL. We also provide new knowledge on low-code ISD platform use in educational settings. Further, we contribute to an enhanced understanding of metacognition by demonstrating the influence of learner- and task-oriented factors on technology-mediated learning (Dang et al., 2016; Zhang et al., 2016). Finally, we provide a detailed account of low-code WIL and technology-enhanced scaffolds.
Theoretical background
Work-integrated learning with low-code platforms
WIL is an educational approach that 'integrate[s] theory with the meaningful practice of work as an intentional component of the curriculum' (Wood et al., 2020, p. 331). Students draw on theoretical concepts through a purposefully designed curriculum to solve practical problems. WIL focuses on authentic and meaningful work tasks that demand timely completion (Patrick et al., 2009). The integration of theory and practice seeks to emulate the functions of a workplace to provide real-life experiences (Jackson, 2017). Consulting projects are often undertaken in which students, industry stakeholders, and educators cocreate valuable outputs from the learning activities (Cooper et al., 2010). A simulated WIL is ' … an immersive WIL experience in a context created to emulate the functions of a workplace with input by the workplace/community, educational institution, and the student' (Wood et al. 2020, p. 333). The WIL is considered ‘simulated’ because the ISD team members are students in the classroom and not temporarily placed in the company as employees, which might be during a placement or internship. A course is considered a WIL course when it involves three stakeholders: students, academics, and industry/external stakeholders. Without the external element, the WIL becomes a full simulation; albeit technology-supported simulations may be a pedagogic practice within a WIL (Patrick et al., 2009). Active stakeholder involvement accelerates the project work and facilitates the outputs that are of value to the business (Leidner & Jarvenpaa, 1995). Prior research shows that WIL increases the work readiness of students, and thus, universities worldwide are increasing their WIL offerings.
An ISD project aiming to develop a minimal viable product is an authentic and meaningful work task for WIL in an IS course. Organizations and customers demand the delivery of tailored IS at an increased speed to meet heightened pressures from digital innovations and short-time-to-market (Wiesche, 2021). Thus, the WIL needs to mimic the demands for fast delivery, frequent changes, and complex requirements. IS educators attempt to provide such learning environments through agile simulation games (Hefley & Thouin, 2016) and by providing technology-enhanced scaffolds (Janson et al., 2020). The next advances in learning would mean that students work with inputs from external partners (i.e. industry). Such an advancement would ensure that students are provided with meaningful theory-practice integrations (Wood et al., 2020). In particular, with the risks of failing to deliver an output (e.g. software), the relevance and utility of WIL are unlimited (Effeney, 2020).
Low-code ISD and the respective platforms 1 are development environments increasingly used in industry for 'developing apps fast [while also] changing them fast and inexpensively' (Phalake & Joshi, 2021; p. 690). The low-code platforms provide declarative, high-level programming abstraction and model-driven development (Richardson & Rymer, 2016), thereby extending the functionalities of past WYSIWYG editors (e.g. markup (HTML) editors). Both low-code platforms and WYSIWYG editors have easy-to-use graphical interfaces that create automatic code; albeit scholars reject the notion that markup is a programming code (Phalake & Joshi, 2021; p. 690). Furthermore, some research argues that low-code platforms are not a new technology and that the components of these platforms have existed for some time. However, the innovation of low-code platforms comes from the integration 'in one environment, [of] multiple well-known and traditional system design components' (Bock and Frank, 2021, p. 739). The advancements are integrated low-code platform functionalities such as automated database initiation and extensions, and automated testing (setting up the test environment and running the test continuously) (Beranič et al., 2020; Litman & Field, 2018). The advancements of low-code platforms compared to traditional systems are particularly relevant when considering the differences between no-code and low-code platforms. The latter provides expert developers with an integrated ISD environment, whereas no-code platforms are mainly for business users with little IT skills (Hurlburt, 2021).
Low-code platforms are increasingly used as teaching technology in higher education (Adrian et al., 2020; Metrôlho et al., 2020; Vikebø & Sydvold, 2019). The literature presents several reasons why low-code platforms are deemed suitable teaching technologies, including component integration, easy platform access for students and educators (through cloud solutions), and availability of rich training materials. The decision to use low-code platforms as a teaching technology needs to be made considering limitations raised about them (Luo et al., 2021). For example, studies find the lack of design and layout customization are unique challenges for low-code developers (Alamin et al., 2022). Similarly, practitioners find high pricing models and steep learning curves of low-code platforms to be barriers to adoption (Luo et al., 2021). These limitations demand that educators use free-of-charge platforms and engage in deep curriculum preparation to personally master the use of low-code platforms for teaching and learning. The benefits of the platform can be complemented with agile ISD method principles (Harvey et al., 2019), particularly autonomy (Matook et al., 2016), so that learners have discretion in making decisions on low-code platform aspects, for example, to overcome customization challenges in their interface design.
In sum, with low-code platforms, IS students can develop apps for real-world clients quickly (within the timeframe of a semester), especially should their ISD expertise be that of a non-IT expert. The low-code WIL provides an environment that simulates a real-world ISD project, allowing students to develop and deliver working software quickly.
Educational information systems research and metacognition
Research on digital learning and teaching has a long tradition in the IS discipline with seminal papers dating back 20–30 years (see, for example, Gupta and Bostrom (2009); Leidner and Jarvenpaa (1993, 1995); Piccoli et al. (2001)). This body of work shows that technology improves learning processes and can enhance the effectiveness of a teaching method 2 (Leidner & Jarvenpaa, 1993). Ongoing advancements in new technologies provide more integrated and immersive learning in simulated environments (Gupta & Bostrom, 2013). For example, educational research began with examining IBM technologies that projected results to overhead screens (Leidner & Jarvenpaa, 1993) but advanced to web-based training and collaboration methods that had greater positive effects on multiple learning outcomes (Gupta & Bostrom, 2013).
Technology is also used for learning simulations and game-based learning (Leemkuil & De Jong, 2012), including gamified learning (Schöbel et al., 2023) and serious games (López et al., 2021). Especially in management education, gamified online training programs with badges and points create positive learning outcomes (such as better problem-solving) through the increased emotional engagement of learners. Because such learning environments are simulations without needing inputs from an industry/external stakeholder (Wood et al., 2020), teaching and learning are flexible while still realistic. Fewer accounts exist in the literature on technology-rich learning environments that involve all three stakeholders (i.e. students, academia, and industry/external stakeholders) to cocreate outcomes (learning outcomes and product outcomes).
Prior research describes different IS educational models to present the determinants of learning (see, for example, Gupta and Bostrom (2009); Leidner and Jarvenpaa (1993, 1995); Piccoli et al. (2001)). Depending on the learning context, its authenticity (realism), and the learner’s control over the technology, the learning manifests differently (observational repetition, own creation, affective responses) (Leidner & Jarvenpaa, 1995). The models place the learner as a central factor within the learning process (Leidner & Jarvenpaa, 1993). The human dimension – and its specifics – is critical for learning success within a given teaching method (Piccoli et al., 2001). Theoretical diversity exists in other design determinants to account for the differences in learning structures, content, interaction, and technology (Gupta & Bostrom, 2009; Piccoli et al., 2001). Learning outcomes in digital settings are the extent to which learning goals are achieved following the means-end understanding (Matook, 2013). The goals include skill, cognitive, effective, and metacognitive factors (Anderson & Sosniak, 1994). Especially, metacognition is highlighted as an important indicator of students’ ability to learn and to understand their own learning, albeit skills and cognitive goals are most often studied. Educational scholars consequently called for research on metacognition in digital learning and teaching (Gupta & Bostrom, 2009, p. 692).
Metacognition is conceptualized as 'one’s knowledge concerning one’s own cognitive processes and products or anything related to them' (Flavell, 1979, p. 232). At the core, metacognition is about reflections of a person’s cognition and is thus referred to as 'thinking about thinking' (Dinsmore, 2017). In the context of learning, metacognition captures how learners think about how to acquire new information (Flavell, 1979). Contrasting traditional views of learning, which put the educator at the centre of learning, metacognition accredits the learner with an active role in learning (de Fátima Goulão & Menedez, 2015). Metacognitive learners think critically about their learning and adapt when perceived improvements are required (Flavell, 1979). Therefore, learners assume responsibility for the learning outcome and take ownership of the learning process (Flavell, 1979). Through metacognition, learners plan their learning, monitor their progress, and self-assess their performance (Tanner, 2012).
Metacognition draws on reflections as deliberate and focused thinking processes. The reflection aids in understanding one’s own capabilities, learning strategies, and the requirements of a task (Coutinho & Neuman, 2008). Without reflective abilities, learners struggle with planning, and the cognitive monitoring and evaluation of their actions. When learners are given time and opportunity for reflection, they can consciously register when learning occurs and derive conclusions for improvements (Boor & Cornelisse, 2021). Reflexivity makes learning more personal (Clem et al., 2014; Dafei, 2007). Through regular reflective behaviours, learners better internalize the planning, monitoring, and evaluation of learning. Thus, metacognitive skills serve as a ‘surplus value’ to students’ intelligence while contributing to their academic success (Veenman et al., 2014).
Metacognitive learning theorizing dimension of person and task for a situation.
Differences in metacognitive skills can be explained by examining variations in the knowledge of a learner and the learning tasks (Livingston, 2003). For example, differences in metacognitive skills were due to learners’ differences in language learning (Goh & Vandergrift, 2021; Kessler, 2021), mathematic problem-solving (Lee et al., 2019; Tian et al., 2018), and programming (Chetty & van der Westhuizen, 2014; Prather et al., 2020). Metacognition comes to fruition when a learner is strategic in learning new knowledge (knowing how) and has an understanding of when and why certain behaviours are to be selected (Paris et al., 1983). Flavell (ibid) dimensions serve as influencing factors for metacognitive skills, but scholars must contextualize these factors for unique learning situations (such as learning programming or math problem-solving).
Metacognition is an important factor for education because it can explain how students effectively learn new information (Tanner, 2012). Students acquire current knowledge through their university education but they face an ever-changing workplace as graduates (Wiesche et al., 2019). The environment demands ongoing learning and students must have obtained metacognitive skills as part of their graduate learning. Metacognitive students can plan, monitor and evaluate their learning, which is useful in mastering workplace demands. For example, metacognitively skilled students can reflect and reframe their thinking to new learning processes. Drawing on metacognition, the learning becomes deeper and more effective (Schraw & Dennison, 1994). However, the learning outcomes (e.g. grades or confidence in learned knowledge) vary among students depending on their metacognitive skills (Zhao & Ye, 2020).
Metacognition is investigated in the literature of computer science and IS, but focuses predominantly on programming, which is only one ISD activity (see Appendix A, and references such as Chetty & van der Westhuizen, 2014; Lopez, 1997). In this context, metacognition is theorized as affecting programmers’ choices and the ISD activity of programming for creating software artefacts (Ambrose, 2003; Prather et al., 2018; Shaft, 1995). Metacognitive skills of programmers can impact their comprehension of code (Shaft, 1995). When programmers are given metacognitive instructions during programming, they improve their understanding of the program and can better describe the strategies used (Loksa et al., 2016), albeit only through specific metacognitive heuristics (Shaft, 1995). In addition, novice programmers benefit from using metacognitive strategies to enhance their coding expertise (Prather et al., 2018).
Through reflections and engaging in higher-order thinking, novice programmers understand their needs for scaffolding and knowledge gaps and subsequently, ask for assistance to perform better. Even for senior programmers, their competency depends on metacognitive factors, which in turn influence programming behaviours, resulting in different performance outcomes (Ambrose, 2003). Consequently, this body of research provides evidence of metacognition's influential role in software programming (Havenga et al., 2013) due to the highly intellectual nature of creating and orchestrating these complex systems (Maruping & Matook, 2020). However, programming is only one of many activities during the ISD process; others including planning, analysis, and design have received limited attention (Matook et al. 2021). Two noteworthy exceptions are Santoso et al. (2017) and VanDeGrift et al. (2011) which examine metacognition during the analysis and design activities of an ISD project, but with little attention paid to programming and delivery.
More recently, IS studies examine metacognitive influences in the context of technology adoption, including AI algorithmics and social media analytics (Kessler, 2021; Sagara et al., 2020), but also as general learning aids in IS courses (Lang, 2018; Prather et al., 2020). Collectively, these studies illustrate that students with metacognitive skills think more often about how to acquire new knowledge and are more aware of their abilities to learn (Vu et al., 2000). However, past research is exploratory using qualitative methods such as reflective journals of users and learners (Kessler, 2021; Shaft, 1995), and metacognitive mental maps that elicit reflective episodes and thinking processes (Lee & Baylor, 2006). To a lesser extent, we find quantitative studies (e.g. surveys) with large samples in IS that examine antecedents and impacts of metacognition in IS learners, except for one large experiment study (Prather et al., 2018). Notably, as Appendix A shows, past research mainly focuses on programming. Because prior research remains unclear about metacognition in other contexts than programming, especially learner-centric ISD contexts with real-world outputs and the three-stakeholder model (i.e. WIL), more research is needed.
Mixed-methods design
The mixed-method research approach draws on both quantitative and qualitative methods to develop novel theoretical perspectives within one inquiry about a specific phenomenon of interest (Venkatesh et al., 2016). The mixed-method approach can be especially helpful when results are inconclusive or incomplete (Venkatesh et al., 2013). The deeper levels of understanding about a single phenomenon increase the existing knowledge base about it. The current study on low-code WIL uses a mixed-method design to provide a more robust understanding of digital innovations (i.e. low-code platforms in higher education) while relying on 'the strengths and minimizes the weaknesses of both methods' (Venkatesh et al., 2016, p. 437). For both studies, we formulated research questions to gradually gain a more complete understanding and a more complementary view on the phenomenon. Thus, our research purpose fits the ‘completeness and complementary’ categories (Venkatesh et al., 2016). We use a theory-testing deductive design in the quantitative study and an explanatory, interpretative design for the qualitative study.
We selected a ‘mixed-methods multistrand’ design (Venkatesh et al., 2016) in which the deductive quantitative paradigm is the dominant approach (Tashakkori & Teddlie, 1998). Data strategies regarding collection and analysis focused for both studies on the students in the respective university course while the analysis used a sequential QUAN-QUAL approach. This study uses a population sampling approach as the entire population of enrolled students in the course was invited to participate. In sum, the first study is a quantitative survey of 417 students who participated in the low-code WIL learning. The second study is a qualitative study of students’ reflective essays as data to understand how students learn to learn in WIL that uses the digital technology of a low-code platform (Mendix). Appendix B presents our design choices following mixed-methods guidelines.
Study 1: Impact on metacognitive skills and learning outcomes by learner- and task-oriented factors
Hypotheses development
Overarching theoretical model
Our theoretical foundation is based on the metacognitive framework by Favell (1979). The situational context is that of a WIL course of low-code app development which provides real-world experiences of undertaking an ISD client project. Based on Leidner and Jarvenpaa (1995), our theoretical foundation is that of constructivism of learning with the adaptation that knowledge is a highly personal experience within a real context. The low-code platform supports the ‘informate down’ vision in which 'control and content of learning is in the hands of students' (Leidner & Jarvenpaa, 1995, p. 283).
For the learner-oriented factors, we integrate Bandura’s self-efficacy theory (1979) and motivation theory (Noe & Schmitt, 1986) to capture the students’ beliefs and motivation to learn skills to deliver the app in this learning situation. These two are specific factors of the learner and are presented as central self-variables in the literature (see Leidner and Jarvenpaa (1995) Piccoli et al. (2001). We theorize them in hypotheses 1 and 2.
For the task-oriented factors, we draw on theoretical writing in the ISD literature (Conboy, 2009; S. Matook et al., 2021; Matook et al., 2016) to account for the task of developing a business app. The low-code platform empowers students to deliver the app because of the inbuilt citizen development focus. The task-oriented factors are autonomy and a strategic approach to the ISD task (delivery of a valuable working app) which hypotheses 3 and 4 theorize as enhancing students’ metacognitive skills.
As the research model in Figure 1 shows, we also use a perceptual measure of learning (students’ knowledge confidence) and two performance measures (grade and app delivery) to capture the impact of metacognitive skills on outcomes. In sum, the hypotheses postulate relationships between learner- and task-oriented factors in the situational context of low-code WIL to impact students’ metacognitive skills and learning outcomes. Research model Study 1.
Factors influencing metacognitive skills
In the context of low-code WIL, self-efficacy captures perceptions about a student’s ability to achieve a set learning task (i.e. app delivery). A student’s high self-efficacy would translate into the belief of having control over their behaviours to produce the business app (Chen et al., 2001). To successfully deliver the app, students would believe they are capable of using the low-code platform. Any future efforts a student invests in the development work depend on their perceptions of their ability to create the app (Bandura, 1997). The capabilities of low-code platforms (Bock and Frank, 2021) are useful in this process because they provide ongoing feedback about the development progress, number of errors, and feature correctness. Thus, students can use the platform’s feedback to define their ability levels and conclude about their self-efficacy.
We argue that students in low-code WIL can use their self-efficacy perceptions to create metacognitive skills. When students develop their apps, they are expected to engage in planning (design sprints, requirements planning), monitoring (progress via Kanban boards), and evaluation activities (retrospectives). The low-code platform provides students with information to appraise how well they perform these ISD activities. Through client showcases, business feedback on the app (utility for the industry purpose) is also provided, all of which creates awareness of past ISD work and provides opportunities for reflection on how to do it better. Students with high self-efficacy beliefs are expected to work extensively on the app to achieve the delivery goal (Bandura, 1986). Consequently, we argue that students with high self-efficacy beliefs are likely to choose ISD activities that enable the creation of metacognitive skills (Bandura, 1997). With no doubt about their capabilities, highly self-efficient students will be more persistent when facing development challenges and be more reflective on thinking about how to create a valuable working app. This reflective thinking process is an activity that creates metacognitive skills. In contrast, students who fear failure and doubt their abilities to develop the app (i.e. low self-efficacy) are more likely to refrain from acquiring performance feedback, and forgo any progress monitoring and critical evaluations. These students miss out on activities for reflecting on what went wrong in their thinking and thus lack opportunities for improving their thinking in the future (i.e. miss out on metacognitive skills). Therefore, we hypothesize:
When students are highly motivated to acquire skills required for delivering their low-code WIL outcome (i.e. business app), they create metacognitive skills. For example, students who plan the implementation of a user story carefully and test its correctness based on clients’ expectations, would also undertake metacognitive activities (e.g. reflection on interface design and data entry options). Usually, ISD coding involves multiple ways of coding a feature (small and large technical debt) (Topi & Spurrier, 2019), and so does the ‘coding’ in the low-code platform. Thus, students keen to deliver a working app that satisfies the client’s need would seek to optimize their drag-and-drop ‘coding’ (Litman & Field, 2018). When students are highly motivated to learn to deliver the app, they direct their attention to reflective thinking about undertaking the best ISD activities and approaches. Through the low-code WIL development work and reflections, students train their metacognitive skills. Indeed, the low-code platform provides opportunities for reflection such that the higher the students’ motivation, the more they create metacognitive skills. Thus, we hypothesize:
We argue that a strategic approach also creates metacognitive skills in low-code WIL. Opportunities for reflection may emerge when students present the outputs. For example, when students present the app in a showcase to the clients, they obtain feedback on their outputs. The showcase provides opportunities for reflective thinking and relating existing knowledge about the app to newly discovered insights from the feedback. The feedback can be used to improve the app. We argue that students adopting a strategic approach, which aims to satisfy clients’ requirements and, by extension, seeking a high grade for the course project, can create metacognitive skills. When students fully immerse into the low-code WIL, they conduct ISD activities (acceptance testing and retrospective) that stimulate ‘thinking about the thinking’, namely, a metacognitive experience of app delivery. Consequently, we argue that the more strategically the students develop a working app aligned with the client’s requirements, the more they undertake active planning, monitoring, and evaluation of the app delivery. These activities contribute to metacognitive skills (Schoenfeld, 1983). Thus, we hypothesize:
ISD research refers to autonomy as 'the extent to which the software team is empowered with the authority and control in making decisions to carry out the project' (Lee & Xia, 2010, p. 88). Students with autonomy about the ISD task have the authority and control over their development decisions (Highsmith, 2004). In an autonomous ISD environment, decision rights are allocated to the developers so that relevant decisions are made by those who conduct the ISD tasks. Any resource allocation or implementation decision can be performed without the need for executive management approval. When ISD autonomy sits at the operational level, the speed and frequency of decision-making are enhanced, and as such the delivery of outputs is increased (Matook et al., 2016). Particularly, the agile ISD method benefits from autonomy to ensure an ever shorter time-to-markets (Kautz & Zumpe, 2008; Lee & Xia, 2010).
We argue that autonomy given to the students in low-code WIL provides opportunities for creating metacognitive skills. When students are in control of their learning (e.g. app development), they are more engaged in producing outputs. Perceptions of autonomy show when students engage in and thus enjoy learning about the low-code platform, they are open to novel ISD experiences and continue with the low-code WIL despite any challenges (Macaskill & Taylor, 2010). Prior research shows that autonomy-triggered engagement results in deeper learning (Blumenfeld et al., 2006). When students use their autonomy to improve ISD activities, they undertake planning, monitoring, and evaluation of the design and coding for the app. In addition, educators encouraged (but did not force) students to regularly perform reflections (retrospective, planning, and estimation sessions). The more students use their autonomy to engage in metacognitive activities, the better they become at reflective thinking. Consequently, we argue that the more autonomous students work, the more likely they are also engaged in metacognitive skill building. Thus, we hypothesize:
Learning outcomes in low-code WIL
We capture the effects of metacognition in low-code WIL to show the impact of reflective thinking on relevant student outcomes. In particular, we examine the influence of metacognition on outcomes, that is, knowledge confidence, app delivery, and grade (Klein et al., 2006; Zulkiply, 2009).
Metacognitive skills in low-code WIL enable students to reflect, monitor and control and adapt their learning based on their assessment. The acquired feedback from educators and learning materials further allow for reflection and deep thinking (Flavell, 1979). App development requires students to continuously think about how to best design and implement the business requirements. The low-code platform provides means for fine-grained monitoring and testing because the platform automatically parses ISD activities in the background and immediately notifies about any errors in the app (Crumbly & Field, 2020). Some students may find multiple error notifications once they implement a user story. Thus, students with advanced metacognitive skills can plan and adjust their learning activities to reduce errors in the future. As a result, the improvements in the low-code WIL experiences may enhance students’ knowledge, and they feel more competent to create low-code business apps at their future workplaces. Thus, we hypothesize:
Students with metacognitive skills reflect deeply on how they learn and adapt their learning (Flavell, 1979). In a low-code WIL, the metacognitive skills should empower students to reflect on the learning artefact they create. Through the app, the learning becomes more tangible and progress more visible. Enriched through the ongoing theoretical input of low-code platform features, students can reflect on how they implemented specific client requirements and undertake adaptations to their app. Thus, the more metacognitive skills students possess, the better they should master the low-code development tasks. In turn, we would expect that higher metacognitive skills contribute to students’ ability to deliver an app of high quality. In contrast, students with poor metacognitive skills may be challenged to deliver an app as they struggle with their reflective thinking. Thus, we hypothesize:
Extant literature shows that advanced metacognitive skills lead to academic success (Zulkiply, 2009). Students with knowledge of how to plan, monitor, and evaluate their learning (Flavell, 1979) can continuously improve their learning. Subsequently, these students better achieve their set performance targets (per the assessment rubrics) and thus obtain higher grades. Hence, in alignment with prior research (Anthonysamy, 2021; Ibabe & Jauregizar, 2010; Zulkiply, 2009), we posit that students’ metacognitive skills in low-code WIL result in better grades for the respective assessment. Thus, we hypothesize:
Methodology of quantitative Study 1
In Study 1, we mainly use data from a cross-sectional survey of IS student in a low-code WIL course.
Research context
The survey study was conducted in a large IS course Business Information Systems Analysis and Design at an Australian public university. The curriculum is that of a simulated WIL requiring students to complete a professional project with industry partners. It is referred to as simulated because students are not placed with the client, but the client comes to the classroom. It is considered a WIL course because it involves three parties: students, academics, and external stakeholders. The task for students is to develop and deliver an app to a business client, a local non-governmental organization (NGO). The clients and the respective customer representatives provide the requirements and undertake acceptance testing. The project vision outlines a communication and event management app for the NGO to allow its members (vulnerable children and their families) to stay connected in a secure online environment. Through features such as a discussion board and events page, the NGO members can discuss relevant topics, and plan and register for upcoming events.
A second industry partner, Siemens Digital Industries provides technical mentoring for the low-code platform. Following market research reports (Gartner, 2021), Mendix from Siemens is the chosen low-code platform for the course. Based on the report, Mendix is considered one of the leading low-code platforms (Gartner, 2021). Mendix provides the entire suite of common and occasional features of low-code platforms (Bock & Frank, 2021), including a data definition capability, GUI designers, and agile project management. For example, a Unified Modeling Language (UML)–based conceptual modelling tool assists with building data structures (Mew & Field, 2018). The drag-and-drop GUI and the widgets minimize hand-coding efforts and accelerate delivery speed (Rymer & Koplowitz, 2019). Likewise, the workflow component, a Business Process Modeling Notation (BPMN) engine, captures the processing logic of the business activities.
The low-code WIL adopts the agile method of Scrum for the app development (Poe & Mew, 2019). Following the Scrum method, various Scrum practices were offered throughout the course. In a design sprint session, students could interview customer representatives from the NGO to elicit requirements and capture them as user stories in Mendix’s product backlog. Furthermore, planning meetings and sprint retrospectives were held to improve the app and the development process. Using the team collaboration tool Miro, students reviewed what went well during the sprint, identified problems they encountered, and set actionable goals for the next sprint. The Siemens mentors and the NGO’s customer representatives provided feedback on intermediates in showcases and through acceptance testing of specific features. Appendix D (columns 1 and 2) captures the numerous details of different activities the students engaged in during the app development.
In addition, as part of the final course assessment, students undertake a 750-word reflective essay. The task is to write a reflection on the low-code WIL learning, esp. about their intensive learning moments (reflection on challenges and resolutions). The educational goal is to create awareness among students about their metacognitive skills and to make it transparent for each student which WIL experience contributes to building their metacognitive skills. The reflective essay is a personal recording of the learning journey, similar to a diary study (Hosein & Rao, 2017). Consequently, the essay provides insight into the thinking processes and how students learn to learn. In the research study, we also used the students’ essays as qualitative data.
The low-code WIL is a contextualized experience for the students to provide an authentic experience of a real-world IT project. Figure 2 illustrates ISD learning with a low-code platform and details the various opportunities for metacognitive reflections. Appendix D details low-code metacognitive skill building in relation to development stages and WIL activities. WIL process in the low-code course examined (for more details, see Appendix D).
Participants and data collection
Participants' demographics and app details.
Measures
We adapted existing scales from the literature, wherever possible, for measuring our core constructs. However, we adjusted the wording of some items to account for the contextualized situation of a low-code WIL in a university course. All constructs were measured by multiple items using a 5-point Likert scale. A pilot of the instrument demonstrated satisfactory reliability of the measures (Cronbach’s alpha >0.7). The constructs and items are shown in Appendix E.
Independent variables: We measured the learner-oriented factor of self-efficacy with a five-item reflective construct adapted from Chen et al. (2001), and motivation with a three-item reflective construct adapted from Noe and Schmitt (1986). The task-oriented factor of strategic approach was measured by using a reflective four-item scale adapted from Entwistle et al. (1997) and Tait et al. (1998), and autonomy as a four-item reflective construct adapted from Macaskill and Taylor (2010).
Metacognitive skills were assessed with a reflective seven-item scale adapted to our low-code WIL context from Schmidt and Ford (2003), Ford et al. (1998), and Pintrich et al. (1991). The scale assessed to what extent students control their thinking about obtaining knowledge by monitoring, reflecting, and adapting their app development.
Dependent variables: Knowledge confidence assessed the extent to which students feel competent to develop low-code apps was assessed. It was measured with a reflective four-item scale adapted to the low-code WIL context from Jarvenpaa et al. (1998) and Mast et al. (2009).
The second variable is Grade, which captures the marks the students received for the essay about the low-code WIL development. Using students’ performance requires heightened protection of their privacy. Thus, as part of the ethics application (and the subsequent approval), it was explained that 'Data will only be published in an aggregated form and no single respondent's data will be provided'. We also explained the matching processes of grades, essays and low-code apps, and survey data: 'Responses and data from students are collected at different points in time. We therefore collect students’ ID at the beginning of the survey. We explicitly state in the Information Participation Sheet that we only use the ID to match the different points of data collection'. The third dependent variable is that of app delivery. It captures the software artefact each student created as the WIL output. App delivery is conceptualized as the extent to which students deliver an application that meets the client’s needs. Importantly, the factor is not a perceptional measure captured in the survey but created by the researchers, using three objective dimensions of the app: (1) requirements completeness: how many client requirements the students included in the app, (2) user interface design: how many of the lecture’s design principles were considered in the app, and (3) implementation complexity: how many of the advanced low-code platform capabilities were used for developing the app and/or were implemented in the app. Each dimension had 10 criteria and was assessed on 5 levels (see criteria in Table C1). For each dimension, a rating of 5 was given if more than 8 criteria were achieved, whereas a rating of 1 was given if less than 3 criteria were captured.
To create the app delivery construct, two coders coded the apps (N = 417). To ensure a consistent coding process the two coders started by first coding 20 apps independently, and then met and discussed their coding until an agreement was reached. Following, the coders coded another batch of apps (N = 30), discussed and agreed on the coding done, and the remaining apps were coded. During the coding process, the coders met repeatedly to ensure high coding quality. Once the coding process was finalized, the factor served as a construct in the model. To seek support for the coding consistency, we calculated the interrater reliability (IRR; Pearson's correlation coefficient) for the subdimensions of the app delivery factor; n = 417, p < .001; rrequirement completeness = 0.884, rUI design = 0.888, rcomplexity = 0.892). We also determined interrater agreement (IRA; intra-class correlation coefficient (ICC)) for the subdimensions; ICCrequirement completeness = 0.88 (Mean = 4.02; F = 16.10, p < .001; Cronbach’s α = 0.94), ICCUI design = 0.89 (mean = 4.00; F = 16.83, p < .001; Cronbach’s α = 0.94); ICCcomplexity = 0.89 (mean = 3.93; F = 17.30, p < .001; Cronbach’s α = 0.94). The results indicate strong reliability and high agreement between both coders (Hallgren, 2012; LeBreton & Senter, 2008).
Control variables: We include different control variables commonly used in educational studies to rule out alternative explanations for learning outcomes (Becker et al., 2016). First, we include age because prior research shows that younger learners have higher academic performance than older learners, particularly evidenced in IT-related studies (Cassidy, 2012). Second, we include gender because female students were found to excel in their studies and outperform their male counterparts in higher education, irrespective of how learning outcomes were measured (Shibley Hyde & Kling, 2001). Third, we include studying time and prior academic performance because research suggests that students’ studying time (e.g. Cerna & Pavliushchenko, 2015) and prior academic performance (e.g. Cassidy, 2012) influence their academic achievement. Students who spend more time studying and have excellent prior academic achievement often exhibit better learning outcomes (Crawford & Wang, 2016). We measured studying time by the amount of time (hours/week) spent studying, while prior academic achievement was measured by assessment marks students received pre-WIL in the course. Finally, we controlled for software development expertise of students to account for the level of expertise in software development pre-WIL (Wiedenbeck et al., 2004).
Marker variable: To account for common method bias (CMB), we included a four-item construct blue attitude (Schuetz et al., 2021) unrelated to our topic.
Results of quantitative Study 1
The research model was tested using covariance-based structural equation modelling (CB-SEM) via LISREL 10.30. We chose CB-SEM because the literature notes that it avoids 'statistical blind spots' (e.g. the multicollinearity and measurement error) that cause Type I errors for path estimates (Goodhue et al., 2017). CB-SEM sidesteps the blind-spot by accounting for random measurement error and construct correlations (Goodhue et al., 2012). Further, CB-SEM is more robust for a large data sample such as ours (N = 417), and it is recommended as the technique of choice when the sample size exceeds 250 (Hair et al., 2012).
Before presenting the results, we detail the results of a comparative calculation of metacognitive skills pre-WIL and post-WIL. If we can demonstrate an increase in the value of metacognitive skills, then it can be suggested that students’ metacognitive skills have increased. As part of the internal university reporting, we captured students’ metacognitive skills at the beginning of the semester (i.e. t1). The measurement in t1 serves as a baseline for determining a possible average increase in metacognition. Thus, we compared the mean for metacognitive skills in t1 = 3.93 with the mean at t2 = 4.19 (end of the semester, after app completion). An independent t-test shows a significant difference between the two times (p < 0.001). Next, we present first the results for the measurement model and then the structural model (i.e. hypothesis testing) results (Anderson & Gerbing, 1988).
Measurement model assessment
Measurement model fit evaluation.
Reliability and validity of measurements.
Inter-construct correlations.
Note: Square root of the average variance extracted (AVE) is the diagonal.
As per Table 5, we noticed the correlation between autonomy and self-efficacy (r = 0.73) is close to the 0.8 cut-off (Franke, 2010; Shrestha, 2020), which may point to multicollinearity. We checked the severity of potential multicollinearity issues via variance inflation factors (VIFs) and the tolerance via condition indices and variance proportions (Shrestha, 2020; Thompson et al., 2017). Results in Appendix F show that multicollinearity is less likely an issue in our data.
One threat to validity is common method biases (CMBs). To control for CMB, we followed established guidelines for procedural and statistical remedies (Podsakoff et al., 2003). Procedural remedies included proximal separation, increasing the physical distance between the measures of the independent and dependent variables. This was achieved by distributing the questions on different pages of the survey. Statistical remedies included Harman’s single factor test and a partial correlation procedure (e.g. marker variable). First, we conducted Harman’s single-factor test (Harman, 1976). We loaded all variables into an EFA and examined the unrotated factor solution. Six factors that emerged from the dataset accounted for 70.39% of the variance, and no single factor explained more than 50% variance. Then, we performed a partial correlation test using a marker variable to examine the influence of CMB on the observed relationships between constructs (Lindell & Whitney, 2001). The results show changes in the partial correlation were non-significant. Overall, these results indicate that CMB is unlikely to pose a strong threat to our dataset.
Structural model assessment
A structural model evaluates the relationship between the theoretical constructs. To test our hypotheses, we assessed the structural model. The results are shown in Figure 3 and Table 6. The structural model presented adequate fit (χ2/df = 1.54, GFI = 0.90, NFI = 0.91, CFI = 0.97, SRMR = 0.05, and RMSEA = 0.04). Appendix G presents the model fit testing results. Results of research model. Structural model results. Notes: **p < 0.01, ***p < 0.001. ns p > 0.05; effect size ƒ2 = 0.02 = small, ƒ2 = 0.15 = medium, ƒ2 = 0.35 = large. Following Cohen (1988), we calculated effect sizes with the formula: ƒ2 = (R2included − R2excluded)/(1 − R2included).
The coefficient of determination (R2), which is the amount of explained variance of the endogenous latent variable, should be above 0.20 to indicate acceptable explanatory power (Zikmund, 2003). Structural model tests results (see Table 6) indicate that our model explained 56.7% of the variance in knowledge confidence (R2 = 0.567; a large effect size), 56.9% of the variance in app delivery (R2 = 0.569; a large effect size), 47.9% of the variance in grade (R2 = 0.479; a large effect size), and 53.6% the variance in metacognitive skills (R2 = 0.536; a large effect size).
As per Table 6, motivation (γ = 0.36; t = 5.23, p < .001), strategic approach (γ = 0.22; t = 5.05, p < .001), and autonomy (γ = 0.25; t = 2.46, p < .05) significantly influence metacognitive skills. Thus, H2, H3, and H4 are supported. As the effect of self-efficacy on metacognitive skills is not significant (γ = −0.01; t = −0.06, p > .05), H1 is not supported. The results show that the effects of metacognitive skills on knowledge confidence (β = 0.73; t = 15.26, p < .001), app delivery (β = 0.75; t = 17.04, p < .001), and grade (β = 0.17; t = 2.76, p < .01) are significant. Thus, H5, H6, and H7 are supported.
Among the controls, gender significantly influences knowledge confidence (β = −0.14; t = −3.39, p < .001) and grade (β = −0.29; t = −4.58, p < .001). Studying time (β = 0.40; t = −6.26, p < .001), development expertise (β = 0.20; t = 3.16, p < .01), and prior academic performance (β = 0.46; t = 7.28, p < .001) are significant on grade. App delivery is only significantly influenced by prior academic performance (β = 0.10; t = 2.18, p < .05). Age is not significant with any dependent variable.
To check the robustness of the results, we examined the effect sizes (ƒ2) (Table 6). The results show that metacognitive skills have a large effect (Cohen, 1988) on both knowledge confidence (ƒ2 = 1.198) and app delivery (ƒ2 = 1.290), and a medium to large effect on grade (ƒ2 = 0.258). Strategic approach and autonomy have a small effect of 0.043 and 0.028. Motivation has a small to medium effect of 0.116. Self-efficacy does not have a significant effect.
Post-hoc analysis – mediation tests
Mediation test results.
Notes: MC skills = metacognitive skills; ns p> 0.05, *p< 0.05, **p < 0.01, ***p < 0.001.
Study 2: Student voices about metacognitive skill building in low-code WIL
Motivation
The quantitative analysis in Study 1 provides insights into learner-oriented and task-oriented influences in low-code WIL on metacognition and learning outcomes. The results determine that motivation, autonomy, and the strategic approach significantly impact on metacognition, but self-efficacy does not. Thus, we conducted this Study 2 to examine the non-significant finding (Venkatesh et al., 2016) while also enhancing the understanding of how the design choices regarding technology aspects (low-code platform, agile method, and development task) impact on learning. In our qualitative engagement with the data, we sought for deeper insights into low-code WIL with respect to Study 1 factors, and their interplay with the technology. We aimed to provide a better understanding of how (in which ways) learner-oriented and task-oriented factors (as part of Study 1 model) impact students’ metacognition and the role the technology aspects play. We benefited from the literature on scaffolding (Lajoie, 2005; Wood et al., 1976) to delineate students’ individual learning experiences.
Scaffolding is the temporary support by experts (human and objects) for learners to accomplish a task that is beyond the learner’s individual capabilities (Wood et al., 1976). Prior research conceptualizes different scaffolding approaches that guide and facilitate learning (Hannafin et al., 2004), viz. procedural (knowing how), conceptual (knowing what), metacognitive (reflecting on the how and what), strategic (identifying alternative solutions), and technology-enhanced scaffolds (IT for learning and teaching) (Janson et al., 2020; Sharma & Hannafin, 2005). For a low-code WIL, the technology-enhanced scaffolds are most interesting because the platform and the ISD method can serve as the supporting expert for students. Technology-enhanced scaffolds are technological tools and resources to support learning (Kim & Hannafin, 2011), including static web-based materials (e.g. online books, learning portals) and dynamic learning environments (e.g. artificial intelligence (AI)–based tutors (Wambsganss et al., 2021)).
The design of any scaffolds – timing, intensity, and richness – impacts learning processes, giving rise to individual experiences of learners. In Appendix D, we detail the learning activities in the low-code WIL regarding metacognitive skill building. In each low-code development stage, multiple ISD activities are undertaken. For example, in the app creation stage, students participate in model-driven development, a retrospective, and feedback activities. For the students’ learning, the low-code experts are educators, industry partners, and technology aspects. The scaffolding provided by human experts fades over the teaching time when students gain ISD capabilities. In contrast, the technology-enhanced scaffolds remain in place (e.g. Mendix learning paths, platform error messages); albeit these support structures are dynamic, feedback on learning varies depending on learning activities. As the detailed account of the designed low-code learning in Appendix D shows, scaffolds and educational efforts are directed towards building metacognition in low-code WIL. However, one needs to involve the learners and gather their perspectives to better understand the learning experience. Thus, our guiding question for Study 2 is: 'How do students experience learning in low-code WIL regarding metacognitive skills and technology-enhanced scaffolds as well as the learner-oriented and task-oriented factors?'
Methodology of qualitative Study 2
The reflective essays are a written account of students’ learning. We treated the essays as qualitative data that provides insights into student learning (challenges and mastery) in the low-code WIL. The reflections describe a student’s process of learning; how to learn new knowledge and how to learn low-code app development within the WIL pedagogy. Of particular interest are the factors in our research model (Figure 1). In this study, we followed a realist ontology and approached the data with a positivist mindset (Sarker et al., 2018). We sought to triangulate the non-significant finding from Study 1 while also gaining deeper insights into students’ learning experiences in the low-code WIL environment.
Two coders, experienced in qualitative research, coded the 417 student essays. An initial coding list guided the analysis with a focus on the learner-oriented factors, task-oriented factors, and their relationship to metacognitive skills. The coders first coded 35 essays independently and then discussed their coding to understand agreement and address coder differences. Subsequently, each coder coded 191 students’ essays and they regularly met to discuss the coding findings. Through creative collaboration, the coders arrived at a shared understanding.
Results of qualitative Study 2
The students describe the WIL as a self-learning experience that demands their active participantion in the learning. The acknowledgement of the Self shifts the locus of responsibility from the teacher to the learner. This means learning accountability for creating and delivering the low-code app is placed with the students. By accepting the learning responsibilities, students show that they possess educational readiness for WIL. In the essays, students use terms such as ‘self-study’ and ‘self-learning’. One student explained: [Mendix development] … is self-learning and exploration. Before, I only obtain knowledge from lectures instead of participating in mining knowledge and exploring knowledge by myself. For example, I was unaware of how to convert the user requirement into an app. After I
We also found various examples in the data describing how technology-enhanced scaffold of the low-code platform helped with metacognitive skill building. For example, students explain that implementation errors stimulated thinking about better ways to learn low-code development. As Appendix D illustrates, in the stage of 'App creation with Mendix low-code platform', feedback is provided by the platform as part of the app creation process before deployment. As this technology-enhanced scaffold depends on the quality and errors of the app, it is student-dependent and dynamic feedback. Importantly, the error message also explains the type of error and provides a link to the online handbook with tips on how to debug it. Mendix itself also provides a hint of all kinds of error messages, which helped me get into the habit I think the learning of Mendix is a good experience for me as
In analyzing the qualitative data, we particularly sought to explain the non-significant impact of self-efficacy on meta-cognition from Study 1. In their essays, students highlight how the development on the low-code platform and the interactions with the app improved their beliefs in themselves and their abilities to develop a working app in a short time (a few weeks of the semester). Students’ confidence extends beyond the classroom, indicating that they applied the skills gained further. The technology-enhanced scaffolds in the form of materials on low-code development (i.e. the Learning Path as part of the Mendix Academy) are online accessible. Thus, support was readily available, albeit students need to search for the respective section in the materials: My teacher recommended the learning path on the Mendix website, where I was able to enhance my skills and upgrade what I had learned, as well Another important feature of Mendix … is it’s debugging feature where it clearly helps user understand what the error is and what is required to resolve the same. …
The reflections performed by students during the app development also strengthen their self-efficacy beliefs. Students benefited from the teacher demonstrations of the low-code activities, but then continued with the ISD work on their own, in the belief they could deliver an app to the client’s satisfaction. I should first understand what each step is doing, what is the role and purpose, and I found that the fundamental principle of Information System is similar. The creation of the Domain Model and microflow resemble the domain class diagram and activity diagram, respectively. Therefore, with the basic knowledge
We also found evidence in the data that students at times were not self-efficacious in their development work, such that they felt overwhelmed and did not trust their abilities. Students lacking self-efficacy sought out various learning resources (technology-enhanced scaffolds) to allow them to continue with the app development, for example, teaching instructions, Mendix Academy materials, and lecture video recordings. Through these activities, they still engaged in metacognitive thinking because they reflected on how to learn low-code development and adjusted accordingly. Again, the technology-enhanced scaffolds supported the students’ learning and contributed to their high perceptions that they could deliver the app. I thought
The qualitative data also provides details on the convergent results, esp. the learner-oriented factors of motivation and the task-oriented factors of autonomy and strategic approach. Regarding motivation, the student voices explicitly mentioned it as enabling ‘thinking about their own thinking' (i.e. metacognition) because the feedback received in the showcase presentations (see Figure 2) stimulated their motivation and a meta-thinking process: In the WIL process, I do have some
In addition, the technology-enhanced scaffolds, esp. the drag-and-drop coding of the low-code platform, facilitated students’ motivation. For example, Mendix provides an effortless, easy-to-use visual user interface that motivates students to low-code. Similarly, the online Mendix Academy offers teacher and classroom independent learning resources. Using the material enables students to grow their mastery of low-code development. The quotes below suggest that students were surprised about their self-motivation to develop the low-code app, while the quotes also show that the availability of the technology-enhanced scaffolds motivated them to learn and continue to learn, maybe even more so than traditional scaffolds: What I found more surprising was that this The first is self-study. Through my online study of the Mendix Academy -Rapid Application Developer Analyst course, I learned how to schedule the study progress
The experience of with the technology-enhanced scaffolds also benefits the task-oriented factors of autonomy and strategic approach. Using a strategic approach for low-code app development was seen as the best way to deliver the app and to build metacognitive skills. Students explored various platform-related materials (e.g. Mendix Academy materials) on how to use the low-code platform: During the Mendix development, I try to determine which things I do understand well and adjusted Become a Rapid Developer course offered by Mendix has greatly improved my self-study skills. The course
Autonomy cultivated in students while undertaking the development task was repeatably mentioned as a benefit by the students. The teacher-recommended solution for the app was perceived only as an alternative option. At the same time, students embraced the freedom to decide on their own, engaged freely with the platform and its features, and tried various development possibilities. Making almost the same app according to the steps taught by the teacher actually limits our imagination and understanding of Mendix in a certain way. Because the I rebuilt my app from the first step, thinking about the logical relationship between data insert and new item creation. I realized I should distinguish different roles’ requirements and meet their demands. Soon, I created a new 'Attendee List' page by myself, and I can also read the system error report without much effort and correct errors without asking for help. The most important thing was that
Although students appreciated the autonomy, they expressed the need to enhance further their skills and knowledge about low-code ISD and metacognitive knowledge of learning processes. Then followed the given instructions,
Subsequently, metacognitive skills emerged, which the students enjoyed building as it was a personal challenge that enhanced their employability: The whole learning experience is enjoyable to me because it enhances my problem-solving capability and upskills my ability for my future career.
Similarly, a student explained that the authentic nature of the WIL – having an actual client in the classroom – furthered their thinking and enhanced their workplace readiness: The WIL this semester taught me to accurately collect the required information from clients and interact with clients online. How to memorize clients’ needs and use them in my own work task. I hope I
Essay-based relations between low-code platforms, independent variables, and metacognitive skills.
Discussion
Learning and ‘learn to learn’ in low-code WIL environments
The research examined how students in low-code WIL build metacognitive skills, which are essential skills for lifelong learning and for future workplaces. Study 1 investigated the impact of learner-oriented and task-oriented factors (Flavell 1979) on metacognitive skills leading to relevant learning outcomes. The empirical results indicate that students’ motivation, autonomy, and the strategic approach aid in developing metacognition, albeit students’ self-efficacy perceptions are not relevant. Study 2 focused on students’ experiences in the low-code WIL environment aiming to understand how learning takes place and the role of technology-enhanced scaffolds in supporting learning and metacognitive learning. Findings show that students use extensively different technology-enhanced scaffolds that allow building metacognition at different stages of the work (see Appendix D). Through the results of Study 2, we can address the divergent finding regarding self-efficacy.
In Study 1 and Study 2, we show that motivation impacts metacognition. When students are highly motivated to learn the skills to deliver a valuable and working business app to the NGO, they more often encounter problems because they are learners, not professional developers with expert knowledge. Thus, students need to think and reflect on the development work and explore alternatives to find solutions to their ISD problems. These reflections also encourage them to think about their development approaches, the methods used, and their past development activities. For some requirements, additional knowledge was necessary and that was not given to them by the lecturer. Only when the students reflected on how they had planned and executed their development can they realize mistakes, errors, and oversights, and subsequently, fix them. For example, the NGO did not mention the feature of password recovery in the planning meeting, but in a later showcase, it became an important non-functional requirement. When confronted with the missing feature, students re-designed their interface while capturing other non-functional requirements. The reflection in the showcases and a strong motivation stimulated their thinking about how to learn better to undertake ISD activities. Thus, reflection can improve the low-code ISD work through metacognition.
Both studies provide convergent results for the task-oriented factor. We propose that students can acquire metacognitive skills when they strategically learn how to develop apps with low-code platforms. The WIL course used a low-code platform and the agile ISD method, which is known for exploration opportunities (Vidgen & Wang, 2009). Thus, students were able to be reflective when exploring innovative ways. For example, the WIL allowed students to challenge their interface design, reflect on alternative data structures, and think about a better way to implement a user story by adopting different agile practices, esp. showcase presentations, retrospectives, and planning meetings. The technology-enhanced scaffolds, which relate to planning (user stories) and the ISD process (testing and deployment), support the students in achieving their development results, such as Mendix Microflows. Our results demonstrate that students can translate their strategic and result-oriented ISD efforts into metacognitive learning and thinking about better ways to develop the app.
Autonomy – a second task-oriented factor – is an important aspect of the low-code WIL and we obtained convergent findings for it through the two studies. Prior research shows that autonomy is essential within the agile ISD method (Lee & Xia, 2010). Students had control over their ISD decisions, guided only by the client’s requirements. Our results support a positive relationship between autonomy and metacognition. Students used the autonomy to plan, monitor, and assess not only their ISD activities but also their thinking about how to develop the app. Because the WIL took place in a blended learning setting (Zhang et al., 2016), many ISD activities had to be done outside class hours, and without the teaching staff’s input. In utilizing their autonomy, students engaged in thinking about how to best use the Mendix platform, obtain solutions to error fixing, and monitor their development work. Consequently, our findings provide empirical support for educators who promote students’ independent ISD work as it creates metacognitive skills. Without traditional human scaffolding (e.g. procedural and conceptual), students need to think on their own about how to advance their skills in planning, monitoring, and assessing the intermediary outputs (app). However, the technology-enhanced scaffolds were always available to them and did not fade over time.
In this research, we encountered one divergent finding. In Study 1, the relationship between self-efficacy and metacognitive skills was not significant. However, in Study 2, we found evidence in the qualitative data that self-efficacy contributes to metacognitive skill building. In consulting prior research, we find that differences in fading of human and technology-enhanced scaffolds (Collins et al., 1988) explain the divergent result. The low-code WIL curriculum was designed as a rich practical experience with extensive ongoing support, including technology-enhanced scaffolds (platform’s features, learning materials), the agile ISD method practices, and support from teaching staff. However, the literature on scaffolds suggests that they should be only temporarily available and must fade so that learners take responsibility for their learning (Sharma & Hannafin, 2005). The continuing availability of the scaffolds might result in students’ overconfidence in their abilities (Littrell et al., 2020; Zamary et al., 2016) while also preventing them from beneficial moments of confusion (Lodge et al., 2018).
Our meta-inference about self-efficacy and metacognitive skills is that students who used technology-enhanced scaffolds and esp. used them more or in addition to human scaffolds were able to build metacognitive skills. We suggest this meta-inference because (1) both types of scaffolds did not fade contributing to students feeling self-confident about their abilities for the tasks, (2) human scaffolds were individualized and calibrated providing students with immediate and correct help, and (3) technology-enhanced scaffolds were not calibrated and not tailored to each specific student motivating reflections. Considering these conditions, the findings from Study 2 need to be supported which illustrate the building of metacognition related to high levels of self-confidence. This conclusion is supported by the theory of metacognition in the field of programming, which provides empirical evidence on how metacognitive skills form when learners write software code and are supported by scaffolds (Loksa et al., 2016). Thus, students drawing on technology-enhanced scaffolds need to reflect (i.e. engage, plan, and monitor) on learning with the low-code platform, the agile method, and the ISD task. For example, when the testing and deployment fail, they need to reflect on why this happened, what they know about how to rectify it, and how to proceed. Therefore, different technology-enhanced scaffolds are available, such as links to the Mendix handbook, lecture recordings, and Mendix developer discussion group. In this process, students build metacognitive skills. Consequently, we suggest that students, who used the human scaffolds more than the technology-enhanced scaffolds, had fewer opportunities to build metacognitive skills.
Finally, our results show that metacognitive skills impact multiple learning outcomes. Indeed, students’ metacognitive skills enhanced their confidence, the app delivery quality, and the grade, all of which are important aspects for becoming workplace ready. When students are confident in what they learned and deliver an app to the satisfaction of the clients, they can effectively undertake the ISD activities that eventually enhance their employability in IS careers (e.g. Business Analyst). A high grade achieved in the course is important because it signals knowledge and skills to future employers. The three factors in concert contribute to creating deep metacognitive thinking processes and the knowledge to deliver low-code apps. The mediation testing further illustrates that metacognitive skills also partially or fully mediate the relationships between the independent variables (except for self-efficacy) and dependent variables (except for grade). The test provides however no evidence that metacognitive skills mediate the effects of the independent variables on grade, albeit strategic approach, motivation, autonomy, and metacognitive skills alone impact grade. A plausible reason for this is that there may exist other factors that mediate the relationships between the independent variables and grade.
Implications and contributions
This research provides several contributions to the literature. First, it enriches our understanding of the use of digital educational technologies, viz. low-code platforms (Bock & Frank, 2021), on students’ ability to learn how to learn (i.e. metacognition) (Bennett, 2017; Flavell, 1979; Tanner, 2012). This study provides evidence of the antecedents of metacognition in ISD environments where problems are ill-structured and complex (Matook et al., 2021). For students with little experience in authentic ISD workplaces, app development may appear to be overwhelmingly challenging. Important ISD factors aid in building metacognition that in turn, influences learning outcomes. Because the hypothesized relationships were significant (except for self-efficacy), this study is one of the first to demonstrate how low-code WIL also promotes learning and skills to learn-to-learn.
The study provides an example of how digital educational technologies enhance practical experiences for learners in IS education. The use of technology in student learning was examined in blended learning environments (Dang et al., 2016; Zhang et al., 2016) and programming (Prather et al., 2020). In these settings, the technology is a support media, whereas in a low-code WIL, the technology is the focal point of learning. The platform and the app manifest as one artefact such that an enhanced understanding of the platform translates into a feature-rich app, while feedback on app improvements stimulates advanced knowledge of the platform. We believe the study provides new knowledge on the catalyst role of technology for effective learning (Leidner & Jarvenpaa, 1995).
Second, we also contribute to the IS literature on low-code development as an ISD method for non-IT professionals (Bock & Frank, 2021; Frank et al., 2021). Prior research shows how low-code platforms utilize visual interfaces, model-driven development, and automatic code generation to create apps with minimal hand-coding (Litman & Field, 2018). Extensive research is currently conducted to theoretically investigate low-code methods and cloud-based platforms regarding adoption, use, and benefits. Our study builds on these findings to demonstrate how students in business school IS programs experience low-code development. The business students can be contextualized as non-IT developers, and their development tasks can be a trigger to revive the discussion of user-led ISD initiatives (Lawrence & Low, 1993). Similarly, the findings contribute to the long tradition in ISD research on democratizing IS development (Bjerknes & Bratteteig, 1995; Muller & Kuhn, 1993). Our empirical results provide evidence that non-IT professionals can be empowered to develop valuable solutions while limiting the control of the ISD process and outcome by IT departments and experts.
Third, we also contribute to the literature on learning approaches in the field of education. A lively discussion surrounds the concept of learning styles with strong views about their validity (Nancekivell et al., 2021). While we believe this is an ongoing discussion, we contribute to it with new knowledge about the balanced approach. Particularly, as we do not subscribe to the strict position that a learning style is learners’ disposition valid across different learning situations (Dembo & Howard, 2007). Our results demonstrate that within low-code WIL, the strategic approach with a result-oriented focus enables achieving both objectives, viz. gaining workplace experiences and metacognitive skills. With an achievement focus, students learn how to learn and have practical experiences in the future workplace. Our findings provide new knowledge suggesting that integrating theory and practice in contemporary IS teaching affords educational skill building. Through metacognition, students acquire skills that support the mastery of future innovations in the workplace.
Fourth, we contribute to the literature on motivation, particularly on learners’ motivation to acquire skills for delivering results to the ISD clients (NGO business in the case of this WIL environment). Motivation theory contextualized to the field of education (Colquitt et al., 2000) and studies on motivation of learners (Esnaashari et al., 2020; Klein et al., 2006) demonstrate the importance of students taking responsibility for their learning. In the low-code WIL, we focused on the motivation of students to learn the skills for producing a valuable, working product and their willingness to engage in ISD activities (Noe & Schmitt, 1986). Especially in WIL, when learners believe in the authenticity of the project, they seek to become more skilful at their development. This goal can be achieved through the reflection of learning approaches, the planning, monitoring, and improvements of ISD tasks. Our study enriches prior literature by showing that when learners invest time and efforts in creating apps, they build metacognitive skills that enhance their learning outcomes. Thus, motivation is an internal factor of a learner and to a large extent, under the learner’s control. The factor’s mean (Table 4) was among the highest (4.24/5), which provides evidence that real-world practical experiences indeed stimulated students’ motivation.
Finally, we contribute to research on autonomy, a concept highly valued in agile ISD method (Lee and Xia, 2010). Educational scholars show that students with large degrees of decision freedom succeed at a learning task (Chene 1983). Similarly, our study shows that low-code WIL also supports students working autonomously with the creation of metacognition. Consequently, research that proposes to treat students as active learning partners rather than passive knowledge recipients is also supported by our study (Wang et al., 2013). Observations from our own experiences as educators suggest that through the autonomy afforded by low-code WIL, students’ metacognitive skills (e.g. reflective thinking) improved, evidenced by the less demanding weekly consultations compared to previous years. Through building metacognitive skills, students explored and implemented better processes for their app development.
Findings impact on teaching practices in higher education, on ISD work using low-code platforms with citizen developers (i.e. students), and on learners’ understanding of how to become workplace ready. At universities, the WIL teaching benefits from a focus on metacognition as a means for enhancing employability. It is of crucial importance to build skills at the ‘meta’ level to encourage learners’ reflections on their thinking processes. Our findings also impact on ISD practices by providing insights into the use of low-code platforms by novice and citizen developers. The students serve as a proxy for practitioners who lack programming expertise but have received ISD analysis and design training. The WIL shows how non-IT professionals can self-direct the development of apps. Finally, learners can draw on the findings to understand how they can control their success in learning and, ultimately, their future careers. When learners are motivated to acquire skills for delivering results, engage in an output-oriented strategic approach, and develop apps autonomously, they build metacognitive skills, which are useful for mastering the demands of today’s digital workplace.
Limitations and future research
Some limitations of the current research need to be acknowledged. First, we have used the proxy measures (knowledge confidence, app delivery, and grades) for employability. The extent to which WIL helped enhance students’ workplace readiness and improved success in their future careers can only be determined in a longitudinal study. A future research study (qualitative diary study) should accompany learners from ‘being students’ to ‘becoming graduates’ and follow them through the beginnings of workplace life. Such a study would provide details on the effects of university education, esp. WIL, on employability.
A second limitation may be the delivery mode of the course being exclusive to online teaching due to the global health pandemic. All students and educators interacted virtually but in real-time. In fact, all teaching was delivered as live lectures, and recordings were only used as an additional learning aid. Still, the virtuality may have impaired some low-code WIL activities and how metacognition was integrated into the curriculum. However, virtual work was the new norm during the years 2020–2021, and the teaching settings simulated even this real-world aspect of remote working and distributed software development. Thus, a future research study is encouraged to compare low-code WIL in exclusively virtual teaching to that in a hybrid mode.
It may be that self-efficacy perceptions are impacted by the differences in the technology-enhanced scaffolds in virtual and hybrid learning (Thatcher & Perrewe, 2002). Whereas our focus was on WIL self-efficacy, future studies may want to theorize the more specific concept of computer self-efficacy. Regarding the divergent finding of self-efficacy, it needs to be noted that we did not collect an additional dataset, but instead provide empirical and theoretical explanations for the difference. Such an approach is possible following guidelines by Venkatesh et al. (2016).
Further research may theorize WIL elements and examine how specific agile practices and low-code experiences (e.g. agile retrospectives, talk out loud about interface design) contribute to building metacognitive skills. Another study may take an employer’s perspective to evaluate metacognitive teaching practices undertaken in the classroom. In designing a qualitative study, interviews with practitioners may provide insights into the relevance and effectiveness of the learning. A resulting taxonomy of metacognitive ISD teaching practices would be useful for educators.
Supplemental Material
Supplemental Material - Metacognitive skills in low-code app development: Work-integrated learning in information systems development
Supplemental Material for Metacognitive skills in low-code app development: Work-integrated learning in information systems development by Sabine Matook, Yazhu Maggie Wang, Nuria Koeppel, and Simon Guerin in Journal of Information Technology
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Australian Research Council (DP160100431).
Supplemental Material
Supplemental material for this article is available online.
Notes
Author biographies
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
