Abstract
Postgraduate students need to be equipped with self-determined learning skills in order to meet the demands of higher degree and employment in the fourth industrial revolution (IR 4.0) environment. However, the lack of analysis conducted on item level for validating the measurement of postgraduate students’ self-determined learning skills prompted the development of The Postgraduate Self-determined Learning Questionnaire (PSLQ) to fill in the research gaps. This paper aimed to assess the psychometric properties of the newly developed 42-item instrument through a mixed method approach as no study has evaluated the psychometric properties of PSLQ at the item level through Rasch model analysis. Quantitatively, 440 postgraduate students were sampled from a higher education institution in Malaysia to answer the questionnaire. Qualitatively, 10 respondents from science and non-science disciplines were interviewed. The paper reports findings on the psychometric properties of PSLQ through Rasch Rating Analysis and participants’ perspectives on the item validity. The results revealed that the items in PSLQ demonstrated good psychometric properties for the measurement of self-determined learning skills of postgraduate students. Implications of the study were discussed in this paper. Future studies can further investigate the psychometric properties of PSLQ according to students’ discipline of study.
Plain Language Summary
Self-determined learning is a crucial skill for postgraduate students in higher education and future working environment. However, there is a lack of established measures to gauge this skill. The Postgraduate Self-determined Learning Questionnaire (PSLQ) is a newly developed measure to gauge postgraduate students’ self-determined learning skills. The objective of this study was to assess the properties of this newly developed questionnaire. A total of 440 postgraduate students from a university in Malaysia have participated in this research. Data were gathered through online survey. In addition, nine postgraduate students from the same university were interviewed. These students were taken from both science and non-science disciplines. The results of the analysis showed that PSLQ can be used to measure postgraduate students’ self-determined learning. However, some minor improvement is required. The instrument can be used by researchers, lecturers and university administrators as a tool to determine self-determined learning at the postgraduate level. Implications and suggestions for future investigations were discussed in this paper.
Introduction
The fourth industrial revolution (IR 4.0) has a profound impact on higher education and human capital development (Halili et al., 2021; Man & Man, 2019). To thrive in today’s innovation-driven economy, graduates must possess competencies that allow them to meet the demand of the future workforce. Self-determined learning is one of the fundamental skills that needs to be nurtured and strengthened in higher education (Kamaruddin et al., 2020). Self-determined learning is also called heutagogy. The term was first coined by researchers in 2000 (Hase & Kenyon, 2000). Heutagogy was an attempt to advance self-directed learning to become a more inclusive and comprehensive concept (Hase, 2015). It places students at the center of the teaching and learning processes. Students must “learn how to learn.” They need to identify own learning needs, formulate learning goals, utilize learning resources to achieve their goals, solve learning problems proactively and engage in self-reflection throughout the learning processes to optimize learning and develop own capabilities (Blaschke, 2012). The learning process is supported by digital technology and non-liner in nature. Self-determined learning can produce students who are “future ready” and capable to thrive in work and live in the fourth industrial age and beyond. The promotion of self-determined learning skill in higher education is in line with the national policies of Malaysia to produce the necessary human capital for the country’s development (e.g., Malaysian Education Blueprint-Higher Education 2015–2025, the 11th Malaysian Plan, the Future of Curriculum Philosophy and the National Higher Education Strategic Plan).
Self-determined learning is mostly witnessed in postgraduate education (Uday, 2019) as students are expected to play active roles in research activities (Dietz & Eichler, 2013; Eichler & Dietz, 2013; Kerry, 2013). They are given greater autonomy to determine what, when and how to learn (Blaschke & Hase, 2019; Davis & Hase, 2001; Hase, 2011; Hase & Kenyon, 2000). Considering the importance of self-determined learning skill, it is crucial to measure and enhance this skill among postgraduate students. However, literature reviews indicate that there is a lack of validated instrument to measure this skill among postgraduate students. The existing instruments that assess self-determined learning have primarily been designed to gauge elements and practices of heutagogy at the undergraduate level (Jailani et al., 2020; Mohaffyza et al., 2020; Mohamad et al., 2020; Rascón-Hernán et al., 2019). Other instruments do not specifically measure self-determined learning, but instead focus on constructs that are similar, such as self-directed learning and self-regulatory abilities (Rascón-Hernán et al., 2019; Torabi et al., 2013). Currently, there is no validated instrument for measuring self-determined learning among postgraduate students. The PSLQ aims to address this gap by assessing this construct. To date, the Postgraduate Self-determined Learning Questionnaire (PSLQ) has not undergone the evaluation of psychometric properties using the Rasch model analysis on a Malaysian sample, which highlights the importance of this work to ensure it is a robust and valid instrument in measuring self-determined learning. The validity and reliability of the PSLQ in the context of postgraduate education have yet to be established through Item Response Theory, Rasch model analysis.
To fill in the research gaps, the Postgraduate Self-determined Learning Questionnaire (PSLQ) was developed by the researchers (Abdullah et al., 2022). It is crucial to assess the psychometric properties of the items in this newly developed instrument (e.g., Dabaghi et al., 2020; Stolt et al., 2021). This is because results obtained from instruments on self-directed learning (Cheng et al., 2010; Fisher et al., 2001; Hendry & Ginns, 2009) could not be generalized to explain self-determined learning and the psychometric tests done were mostly through classical test theory (CTT) (Dabaghi et al., 2020). CTT could not produce different precision estimates for the measured construct with varied levels. (Zamora-Araya et al., 2018). This is because the test assumes that the value of an attribute is represented by an observed score, which is the sum of a true score and the measuring error. This implies that if the test items are endorsed by the majority of the respondents, the scores in the higher level of the trait will be estimated with less precision (more error) than scores in the lower end. On the contrary, if the items are endorsed by only a few respondents, scores in the lower level of the trait will be estimated with less precision (more error) than scores in the higher end. Founded on the above-mentioned limitation, researchers have called for a more complicated model and new theories in testing the psychometric properties of new instruments (Momani, 2017).
The Rasch model, which is based on item response theory (IRT), is recommended as a more practical and appropriate approaches in assessing the psychometric properties of new instrument. IRT is a modern test theory approach that can be used to assess the psychometric properties of instruments (Bortolotti et al., 2013). Also, it examines the items included in the instrument and the people who are using it. IRT assumes that the probability of responding correctly or incorrectly to a single item is related to both the person’s ability and the difficulty of the item (Bond & Fox, 2015). Trait levels are modeled as a mathematical function of the difference between the person and the item parameter (Bond & Fox, 2001). Hence, the analysis allows generalizability across items and samples (Tennant & Conaghan, 2007). The aim of this study is to examine the item psychometric properties and item validity of the PSLQ using a mixed methods approach. To achieve the first research objective, a quantitative approach was employed to evaluate the psychometric properties of the items in PSLQ using the Rasch model analysis. The second objective was addressed using a qualitative approach that involved exploring the perspectives of science and non-science postgraduate students on the suitability of items in the PSLQ. This approach, which involves collecting participants’ perspectives on item content through interviews, allows for a more exploratory inquiry. By combining these methods, the researchers aim to shed light on the understudied construct of self-determined learning in postgraduate education contexts.
Research Objectives
To evaluate the psychometric properties of the items in PSLQ via the Rasch model analysis.
To explore postgraduate students’ perspectives on the validity of items in the PSLQ via semi-structured interviews.
Literature Review
Self-determined learning or heutagogy is a teaching approach that empowers students to take control of their learning. The concept was first introduced by Steward Hase and Chris Kenyon in 2000 (Hase & Kenyon, 2000). It puts the students at the center of the teaching and learning process and enables them to actively participate in their own education. If the students are guided to take charge of their learning environment through self-control in learning, they will be better equipped to engage in self-determined learning and tackle challenges in the future (Nwagu et al., 2018). This approach prepares students for the rapidly evolving job market and the increasing need for continuous learning and adaptation. Past studies (Handayani et al., 2022; Snowden & Halsall, 2016) found that pedagogical approaches in higher education emphasizes students’ knowledge acquisition and skill development to solve real problems in the working environment, which are closely aligned with self-determined learning.
There are four core principles underlying this approach: student-centered learning, non-linear learning, double-loop reflection, and capacity development. The first principle, student-centered learning, involves students setting their own learning goals, selecting their own materials, and regularly evaluating their own progress. Autonomy and scaffolding provided by lecturers to promote student-centered learning among postgraduate students can enhance learning and research activities. As found by Amani et al. (2022), ongoing support and mentorship help students improve their learning and research skills and enhance their success in postgraduate education. Self-determined learning skills can also be fostered through project-based learning, experiential learning and problem-based learning.
Non-linear learning is a crucial aspect of self-determined learning as it acknowledges that the learning process does not always follow a straightforward path. This approach allows students to explore new information and develop their knowledge at individual pace and in a way that suits their own learning style. Students, therefore, have more opportunities to engage with the learning materials in a meaningful manner, making connections between their prior knowledge and new concepts, which lead to deeper and more personalized understanding. Non-linear learning also promotes learner’s agency and autonomous motivation, which is the characteristics of self-determined learners (Frederico et al., 2020).
Double-loop reflection is the fourth principle of double-loop reflection. It occurs when learners question and test own personal values and assumptions as being central to “learning how to learn” (Newman & Farren, 2018). Double-loop reflection involves a deeper examination of values, assumptions and beliefs in learning. It is an important principle of self-determined learning. Students are empowered to take control of their own growth by engaging in self-reflection and making informed decisions about their learning continuously. Students will be more self-aware and confidence in self-determined learning process. This metacognitive skill fosters deeper understanding and long term retention of knowledge and skills among students, which improves their learning and performance in higher education.
In the context of self-determined learning, capacity development refers to the development of a range of competencies, skills and resources such as critical thinking, goal-setting, time management, and progress evaluation. These transferrable skills are crucial for students to thrive in the 21st century. Through self-determined learning, students become more capable and confident learners as they are equipped with competencies to manage their own learning. By fostering a culture of self-determined learning, postgraduate students can build the capacity that they need to succeed in their chosen field and achieve their individual’s career goals.
Theoretical Background
The first key principle of self-determined learning measures by the PSLQ is student-centered learning, which emphasizes on autonomous learning. As proposed by the agency theory (Blaschke, 2018), humanistic theory (Rogers, 1980), and constructivism (Piaget, 1980; Vygotsky, 1978), students have voices, choice and ownership in learning because they are the key agent of learning. As learning should be personalized and autonomous in nature, heutagogy postulates that the focus of education should place on what and how the learner wants to learn, instead of what is to be taught by the educators (Hase & Kenyon, 2013). Hence, students need to be given autonomy to make decisions concerning what and how they want to learn. Postgraduate students who are self-determined learners will set learning goals according to self-interest and learning needs. They plan and monitor own progress and engage with others (e.g., supervisors, researchers, peers) to gather feedback to improve their achievement. Studies have shown that student-centered learning can be crafted in the teaching and learning within higher education institutions (Mike, 2018; Tsegay, 2015). A study by Mike (2018) investigated undergraduate students’ experience of undertaking Research Methods for the Social Science (SP220) module. Eight students were invited to provide insights and perceptions on the eight semi-structured questions. The findings suggested that student-centered learning approaches were adopted by the lecturers to promote students’ continuous engagement with the materials and tasks.
The second principle of self-determined learning explains about non-linear learning. The exponential development of information and communication technologies at the turn of the 21st century is changing the world of learning. As proposed by the system theory (Emery & Trist, 1965), students drive their own learning actively by interacting with the learning environment and with other learners. With the support of digital technologies and Web 2.0, learning processes are non-linear, more personalized, interactive and dynamic. Self-determined learning highlights the notion of non-linear learning in which learning can take place virtually at anytime and anywhere (Blaschke & Hase, 2015). The application of ICT has been prevalent in the context of higher education (Hamdan et al., 2021; Rana & Rana, 2020; Shen & Ho, 2020; Tømte et al., 2019). For example, an experimental study by Yilmaz and Keser (2016) investigated the impact of technology-enhanced learning combined with reflective thinking activities on students’ academic success, social presence perception, and motivation. The researchers discovered that the use of podcast supported by reflective activities enhanced students’ success and motivation. Additionally, Shen and Ho (2020) conducted a bibliometric analysis on the use of technology-enhanced learning in higher education. Results from the hybrid bibliometric analysis indicated that the use of technology-enhanced learning such as social media, podcasting, and blended learning were the main streams to the development of non-linear learning.
The third principle of self-determined learning measures by the PSLQ is double-loop reflection. With reference to the work by Argyris and Schön (1974), the scholars have differentiated two kinds of learning namely single-loop learning and double-loop learning. In single-loop learning, students focus on correcting their errors without questioning the underlying assumptions, learning goals set and their own beliefs. Double-loop learning extends what students experience in single-loop learning where students engage in deeper thinking and ponder hard about what, why and how they learn. Learners are prompted to make self-reflection on the individual learning process as in understanding how certain knowledge has been learned (Blaschke, 2013). Double-loop reflection stresses the importance of developing students’ metacognitive abilities, which include self-reflection, self-evaluation, self-planning, and re-evaluation skills (Blaschke, 2012; Hase & Kenyon, 2013). Double-loop reflection occurs when learners question and test own values with the aim to improve learning (Hase & Kenyon, 2013). In this process, learners attempt to locate the most competent people when making their decision at the same time try to build viable decision-making networks which functions to maximize each individual’s contributions. When the development of a synthesis occurs, the exploration of views could happen in the widest possibility (Argyris, 1976). As supported by the agency theory (Blaschke, 2018), humanistic theory (Rogers, 1980) and constructivism (Piaget, 1980; Vygotsky, 1978), double loop learning is regarded as one of the key principles of self-determined learning. It promotes active, deep, and meaningful learning. The fourth principle of self-determined learning measures by the PSLQ is capacity development. Capacity refers to students’ acquisition of knowledge and transferrable skills that are crucial for them to thrive in the 21st century (Kereluik et al., 2013). Students need to be more prepared for the challenges and uncertainty in the future, they need to be self-determined learners who are able to think, learn, and solve problems independently (Blaschke & Hase, 2015). Self-determined learning skills will enable students to develop capacity in foundational literacies (e.g., digital skills), competencies (e.g., interpersonal skills) and character qualities (e.g., taking initiative) (Jose, 2021; World Economic Forum, 2015), which are crucial in fourth industrial revolution environment. Together, the four principles clearly formulate the construct of self-determined learning. Studies have emphasized on the importance of developing competencies among undergraduate students (Barrie, 2004; Chan et al., 2017; Lozano et al., 2017). A systematic review by Chan et al. (2017) proved that competencies were crucial to students’ development. Self-motivation of both teachers and students were required in order to promote competencies effectively during the teaching and learning process. Several studies also highlighted the need for competence-based education to ensure students’ development of competence in higher education. A study by Ceulemans and De Prins (2010) shed light on a variety of student-centered methods that could enhance sustainability and competence in the classroom including audio-visual aids, higher-order thinking activities, case studies, collaborative work, assignments, problem-oriented education, oral presentations, inquiry-based learning, and so on. However, more efforts are still needed for the implementation of relevant pedagogical approaches to enable the acquisition of competencies among students (Lozano et al., 2017; Martins et al., 2006; Sibbel, 2009).
With the knowledge guided by the past literature, the researchers thus proposed a theoretical framework which detailed the constructs of self-determined learning. Figure 1 shows the theories that underpin the instrument framework of the PSLQ. The framework is supported by Agency Theory (Milgram, 1974), Humanistic Theory (Rogers, 1980), Constructivism (Piaget, 1980; Vygotsky, 1978) and System theories (Emery & Trist, 1965). There are four principles underlying the theory of self-determined learning namely (a) student-centered learning, (b) non-linear learning, (c) double-loop reflection, and (d) capacity development (Blaschke, 2018; Blaschke & Hase, 2015; Hase & Kenyon, 2000; Tiew & Abdullah, 2021). Therefore, the PSLQ is also grounded by four principles of self-determined learning.
Theoretical framework.
As of today, there is no instrument measuring the four principles of self-determined learning. Researchers have thus far examined the validity and reliability of the PSLQ through exploratory factor analysis (EFA), which only showed the use of factor analysis for the evaluation of the instrument (Abdullah et al., 2022). To date, no study has evaluated the PLSQ through Rasch model analysis and interviews among postgraduate students in Malaysia higher education. Examining the validity of the PSLQ through advanced measurement method namely IRT proves its importance to conduct research across cultures (Hambleton et al., 2004). Therefore, Rash model analysis was needed because it deals with measurement quality at the item level which can overcome the limitations of Cronbach’s alpha and EFA (Park et al., 2021). This type of analysis can provide more in-depth information on item analysis (Che Lah et al., 2022).
Methodology
This study used an explanatory sequential mixed-method design to evaluate the psychometric properties of the items in the adapted Postgraduate Self-determined Learning Questionnaire (PSLQ), a questionnaire which was developed based on the information and concepts gathered and guided from past literature (Abdullah et al., 2022; Hase & Kenyon, 2013) to measure students’ self-determined learning in postgraduate education. The rationale of using mixed methods design is that mixed methods techniques emphasize the strengths of one method to address the weaknesses of another (Creswell & Clark, 2011; Johnson & Christensen, 2014). In this study, the rationale of including qualitative data is to supplement the quantitative findings. Therefore, quantitative and qualitative data were triangulated to provide more insights on the psychometric properties of the items in the PSLQ. The purpose of the interviews aimed to seek in-depth insights on the suitability of the items in the PSLQ in addition to the results from Rasch analysis. The interview findings could provide insights on why some PSLQ items were easy or difficult to be agreed by the respondents. Additionally, the qualitative data was also used as validity evidence to support the interpretations on the basis of test scores (Johnson & Christensen, 2014). It aimed to determine the face validity to examine the degree of the PSLQ in measuring what it claims to measure (Mills & Gay, 2019). With the understanding of the difficulty of each item, such information can be used to guide the development of the PSLQ. The Rasch model can also provide a context as in what principles of self-determined learning a student has acquired when explaining the result (Boone, 2016). Therefore, the understanding can help researcher to check what items in the instrument requires further attention and decision. In this study, the researchers aimed to triangulate the quantitative and qualitative data for the theoretical validity of the instrument. Theoretical validity is known as the degree to which the theoretical explanation fits the data (Johnson & Christensen, 2014, p. 415) and to what extent the findings from a qualitative study explain the phenomenon in relation to a theory (Mills & Gay, 2019). The theoretical goal of this study was to test how student-centered learning, non-linear learning, double-loop learning, and capacity development contributed to self-determined learning.
Participants
A total of 440 postgraduate students from a higher education institution in Malaysia volunteered to participate in this study. As the study aimed to understand if there was any difference in self-determined learning among postgraduate students from different disciplines, the researchers decided to gather data from science and non-science disciplines. Simple random sampling technique was applied to select 220 postgraduate students from science and non-science disciplines respectively. In this study, science discipline included postgraduate students from physics, medical, and chemistry program whereas non-science discipline consisted of postgraduate students from business, education, and language program. Participants for the qualitative phase were 10 postgraduate students made up of five science and five non-science students, who voluntarily participated in the individual semi-structured interview sessions. Purposive sampling was applied to select the interview participants who met the criteria for selection and their identities were kept anonymous using pseudonyms (SP for science participant and NSP for non-science participant). The participants’ content was obtained and their sessions were audiotaped. The interviewees from the science discipline were labeled as SP1 to SP5, while those from the non-science discipline were labeled as NSP1 to NSP5.
Instrument
The PSLQ was made up of 42 items that measure four principles of self-determined learning namely student-centered learning, non-linear learning, double-loop reflection, and capability development (Abdullah et al., 2022). The items were developed based on the four principles of heutagogy and guided by theoretical and relevant literature reviews. The responses were gathered through 4-point Likert scale from “strongly disagree” to “strongly agree.” The PSLQ was sent to participants via an online link. The data collection was completed in the academic year 2021 to 2022. In addition to the questionnaire, the researchers conducted semi-structured interviews with the chosen participants to explore their perceptions of survey items. During the interviews, the participants were asked to provide feedback on the misfitting items.
The Rasch Measurement Model
According to Rasch (1960), Rasch analysis demonstrates the examination in terms of how well items in a test or rating scale contribute to the useful measurement of an assumed one-dimensional construct variable. The psychometric properties of the Postgraduate Self-determined Learning Questionnaire (PSLQ) were analyzed through Rasch Measurement Model. According to Kean et al. (2018), Rasch analysis is based on Item Response Theory (IRT) models, which are commonly used for development, validation, and refinery of latent construct that is made up of multiple items; each item evaluates different positions individually in the latent construct. IRT enables researchers to obtain invariant items and latent trait estimates (Kliem et al., 2015). The application of Rasch measurement has been evident in a vast variety of disciplines, including education, school psychology, nursing, leadership, and many other fields (Boone & Noltemeyer, 2017; Fox & Jones, 1998). The Rasch model can be used to analyze dichotomous and polytomous data (Alias et al., 2019; Andrich, 1978). Furthermore, Bichi et al. (2019) explained that the interpretation via Rasch analysis provides thorough information about the item structure, thus, can be used to validate students’ ability and appropriateness of the items. Scholars explain that Rasch model analysis may offer a robust assessment for reliability and validity with a small sample as well as provide an elaboration of the respondents and item analysis (Che Lah et al., 2022, 2023). In the present study, the researchers used IRT to examine the psychometric properties of the PSLQ as well as understand the difficulty respondents encountered in agreeing with items assessing self-determined learning.
The unit of measurement for Rasch analysis is known as logit, where persons and items can be measured on the same metric scale (Norhayati et al., 2020). Besides that, Rasch model can also be applied for the development of survey instrument (Boone, 2016). As the researcher aimed to gather data about postgraduate students’ self-determined learning ability in managing their own learning in postgraduate education, a prediction was needed to know which survey items tap into different ranges of self-determined learning ability. Since the PSLQ is a relatively new instrument, the Rasch Measurement Model is suitable to evaluate the psychometric properties of items to ensure the scale validity and reliability in measuring postgraduate students’ self-determined learning. Generally, it is understood that all items are usually not of the equal difficulty which means that survey items should not be assumed to be equally agreeable (Boone, 2016). In current study, Rasch Rating Model (RSM) was applied to diagnose the rating scale structure of PSLQ as Rasch measurement provides a practical method for rating scale analysis (Rasch, 1960). Therefore, it is appropriate to use the Rasch rating-scale model to deal with polytomous data from the Likert response in the present study (Andrich, 1978). Therefore, in this study, Rasch model was used to determine the item difficulty of PSLQ and identify students’ ability in answering the items and further understand the extent they agree or disagree with each item in the instrument.
This formula contains three main parameters as follows:
β n is the measure (ability) of person n,
δ i is the difficulty of item i,
τi is the threshold parameters
Where P (Xni = x) is the probability that a person n is observed in the rating scale category x on item i, which has m + 1 rating scale categories, and
Procedure of Analysis
Before conducting the Rasch model analysis, the content of the PSLQ was validated by three experts from a Malaysian higher education institution, who had extensive experience in online teaching and learning. Researchers validated the instrument through the use of EFA (Abdullah et al., 2022). Subsequently, the Rasch model analysis was performed to measure the psychometric properties of the PSLQ. WINSTEPS software version 3.74.0 software was employed to analyze the item functionality of the PSLQ, including (1) unidimensionality based on principle component analysis of residuals (PCA); (2) item fit based on infit and outfit values; (3) item polarity based on the value of point measure correlation (PTMEA CORR); (4) category fit; (5) reliability and separation index for items and persons; and (6) Wright distribution map. To supplement the quantitative data, the research was then continued with the qualitative phase where 10 selected postgraduate students participated in semi-structured interviews to gain deeper insights into the suitability of the items in the PSLQ. After collecting the data, the interview data were analyzed and interpreted through thematic analysis.
Results
Unidimensionality Analysis of the PSLQ
The requirement for unidimensionality is a prerequisite for the concept of fit in Rasch’s Model (Bond & Fox, 2015, p. 314). Unidimensionality explains only one construct or dimension for response over a pool of items at a time (David et al., 2018). This analysis is vital for the content and construct validity of an instrument (Al Ali & Shehab, 2020). Based on the guideline by scholar, raw variance explained by measures should be above 40%, and unexplained first contrast of should be below 15%.)
In this study, unidimensionality analysis seeks to confirm whether items in PSLQ focus on the same latent trait. A construct is considered unidimensional when it investigates the same latent trait (Aryadoust, 2012). It is also regarded as the preliminary step to conduct differential item functioning (DIF) analysis (Linacre, 2009). Table 1 portrays the unidimensionality results of the PSLQ. The value of principal component analysis of residual (PCA) showed that the raw variance explained by measures recorded at 39%, which was close to the expected variance of 39.8%. Based on the result, it shows that the raw variance explained fulfilled the minimum required value of unidimensionality which was 20%. In addition, the unexplained variance in first contrast recorded a value of 4.5 which is considered good in terms of the dimensionality of the construct (Fisher, 2007), explaining that no second dimension was present. Besides, the unexplained variance in the first to the fifth contrasts of PCA of residuals fell between 3% and 5% indicated that the instrument was of very good quality criteria (Fisher, 2007). Overall, the analysis proved that the items in PSLQ measure a unidimensional dimension, meaning that the items in the questionnaire function in unison, measuring self-determined learning as one latent variable.
Item Fit.
Item Fit & Item Polarity of PSLQ
Item fit is necessary for the confirmation of the construct measures and it is used to identify misfitting items and determines the patterns of responses from each respondent (Cordier et al., 2018). Polarity analysis for items is tested to examine if the item in the construct function in the same direction for the measurement of the developed construct. The construct is considered well functional when it displays positive Point Measure Correlation (PTMEA Corr) value (Samsudin et al., 2019). The item polarity can be measured by point measure correlation coefficient (PTMEA Corr). The results of Rasch Analysis (Table 2) demonstrated that the mean-squared (MNSQ) infit ranged from 0.61 to 1.81 logits and the mean-squared (MNSQ) outfit was between 0.63 and 2.24 logits. The MNSQ outfit value between 0.5 and 1.5 is considered acceptable for items in Likert scale which also implies that the item is productive for the measurement scale (Norhayati et al., 2020). According to the guideline by Linacre (1999), it is suggested that the MNSQ outfit should not exceed 2.0. In the present study, only Item Q2 exceed MNSQ outfit of 2.0.
Category Fit.
For the principle of student-centered learning, Items Q2 and Q3 did not fit the Rasch model assumption. Item Q2, “I choose the research topic after considering many factors (e.g., self-interest, supervisor’s research focus, availability of grants, feasibility of the study)” recorded MNSQ infit of 1.81 and MNSQ outfit of 2.24 whereas Item Q3, “I negotiate with my supervisor(s) on my research work” recorded MNSQ infit of 1.09 and MNSQ outfit of 1.57. Next, the construct in the principle of double-loop learning recorded MNSQ infit value of 0.61 to 0.89 and MNSQ outfit value of 0.62 to 0.98, indicating that all the items in the specific dimension fit the Rasch model assumption. However, the items in the principle of non-linear learning displayed MNSQ infit value of 0.80 to 1.57 and MNSQ outfit value of 0.77 to 1.76, indicating that some items did not fit the Rasch model. Five items in the principle of non-linear learning construct exceeded MNSQ fit range. These items were Q12, “I share my academic ideas online (e.g., Facebook, Research Gate, university student portal) to let others know my opinion” (MNSQ outfit = 1.52); Q13, “I use social apps (e.g., Facebook, Twitter, Whatsapp) to engage with others (e.g., supervisor(s), researchers, peers) in academic discourse” (MNSQ outfit = 1.59); Q15, “I use online platform (e.g., Facebook, Research Gate, Twitter) to exchange ideas with others to learn more about my research work” (MNSQ outfit = 1.66); Q17, “I engage with related stakeholders (e.g., ministry, agency, organisation) when carrying out my research” (MNSQ outfit = 1.63); and Q18, “I disseminate my research work online (e.g., Slide Share, Research Gate, Facebook, and Twitter) to let others know my research work” (MNSQ outfit = 1.66). The fourth principle of the construct, capability development, recorded MNSQ infit value of 0.70 to 1.39 and MNSQ outfit value of 0.69 to 1.5, indicating that all the items in the dimension fit Rasch model assumption.
Based on the overall results, seven items (Q2, Q3, Q12, Q13, Q15, Q17, Q18) in the PSLQ have exceeded the MNSQ outfit value of 1.5. The result showed that Item Q3 and Item Q12 slightly exceeded the recommended MNSQ outfit value of 1.5. In other words, further speculation was needed on other properties of these items before making the decision to discard them. Hence, the researchers referred to the point measure correlation coefficient of the item. A high PTMEA Corr explains that an item can distinguish between the respondents’ abilities; on the other hand, if the PTMEA Corr is negative or zero, it indicates that the items are not consistent or conflict with the construct (Bond & Fox, 2015). In this study, the results showed that the point measure correlation of the seven items was .39 and above, indicating no negative or zero value. Therefore, the positive values explained that the seven items did not conflict with the construct and distort the measurement.
Category Fit of the PSLQ
The category fit assumes that the measurement fits and functions at the expected way (Samsudin et al., 2019). It means the measurement category value is more positive when the scale value is higher. The analysis (Table 3) indicates that the category mean-squared infit of PSLQ ranged between 0.93 and 1.46 whereas the category mean-squared outfit was between 0.89 and 1.74. Additionally, the category label as 1 recorded a category measure of −3.12 logits, category label 2 had −1.21 logits, followed by category label 3, which showed 0.99 logits, and category 4 contained a measure of 3.50 logits. These category measures fulfilled the assumption of Rasch model by following a pattern sequenced from easy to difficult. This explains that the four response scales for each item in the instrument functioned well.
Unidimensionality of PSLQ (in Eigenvalue Units).
Item and Person Reliability of the PSLQ
The item reliability is known as the expected consistency of the item placement on the logit scale if the item is attempted by a group of different respondents with the same ability (Bond & Fox, 2001). In this case, high item reliability in the present instrument indicates that the items are consistent and measure the same construct, which is self-determined learning. The person reliability refers to the estimated internal consistency of the measure that is equivalent to Cronbach’s alpha (Cordier et al., 2018). According to Bond and Fox (2015), person and item reliability values of 0.8 and above are acceptable, values between 0.6 and 0.8 are less acceptable, and values below 0.6 are not acceptable. In this study, Table 4 reveals that the item reliability of PSLQ was recorded at 0.98, indicating that the instrument achieved excellent value of item reliability. This indicates that the instrument is expected to measure the level of self-determined learning consistently. Next, the item separation index recorded a value of 7.20 which is considered excellent according to the quality criteria set by Fischer. The item separation index explains that the person’s ability to discriminate the 42 items into seven levels of agreement, revealing that the 42 items provided a good spread in measuring the level of self-determined learning and the item difficulty of the items can be confirmed. Additionally, the findings in Table 5 show that the person reliability index of an individual recorded a value 0.95, which is categorized as excellent quality. The person separation index was 4.27, which is very good, explaining that there were four levels of ability among the respondents in PSLQ. It gives a positive indication that the instrument can differentiate the respondents’ self-determined learning ability into at least one level.
Item Reliability.
Person Reliability.
Wright Distribution Map of the PSLQ
Wright map can be used to further understand Rasch analysis for instrument development. It provides both person measures and item measures on the same linear scale, it can guide researchers in determining how well the items are distributed across the test takers’ ability (Boone, 2014). The analysis showed that the respondents had more difficulties to endorse items located on the top of the map whereas self-determined learning items located at the bottom of the map were more easily endorsed by the respondents. As shown in Figure 2, four items in the PSLQ were considered as the most difficult items for respondents to agree with. They are: “I share my academic ideas online (e.g., Facebook, Research Gate, university student portal) to let others know my opinion” (Q12), “I engage with related stakeholders (e.g., ministry, agency, organisation) when carrying out my research” (Q17), and “I disseminate my research work online (e.g., Slide Share, Research Gate, Facebook, and Twitter) to let others know my research work” (Q18) and “I know how to commercialise my research product” (Q39). In short, the respondents have most difficulties to agree with Items Q12, Q17, Q18, and Q39. When the respondents were less likely to agree with the more difficult items, it gave an indication that the respondents found it more difficult to demonstrate their self-determined learning ability through the items. On the other hand, Items Q1 (I am aware of my learning goals before enrolling in postgraduate program), Q21 (I reflect on the way I learn to develop more effective learning approaches), Q22 (I reflect on the problems to find solution), Q25 (I like to think over what I have been doing), Q26 (Based on the outcome of my evaluation, I plan what needs to be done), Q33 (I am aware of my own weaknesses in learning), and Q34 (I know what I still need to learn throughout the program) were considered as the easiest items for the respondents to agree upon. It means the items were able to reflect the respondents’ self-determined learning ability more easily among the other items in the questionnaire. According to Bond and Fox (2015), an easy test would yield mean-person estimate with a large positive value. Based on the analysis, the researcher reported that the PSLQ was considered as a relatively easy questionnaire for the sample. The value of standard deviation of 1.37 logits for person estimates indicates greater spread of person measures in the measures than evident in the item measures. The result also indicated that the sample found the items in the questionnaire were comparatively agreeable and useful to measure their level of self-determined learning. In other words, the items were able to measure the continuum of self-determined learning from easy to more difficult items to agree with.
Wright distribution map.
Participants’ Perspectives From Interviews
As this study aimed to examine the psychometric properties of the items in PSLQ through mixed methods approach, the researchers conducted semi-structured interviews with 10 postgraduate students to gather qualitative data. This section presents the qualitative data specifically on science and non-science students’ perspectives on the unfit and more difficult items. The findings showed that postgraduate students from both science and non-science disciplines perceived the survey items to be relevant in measuring self-determined learning. They agreed that the measure of the PSLQ was an accurate assessment to determine postgraduate students’ level of self-determined learning.
Autonomous Learning
Based on the qualitative findings, all interviewees agreed that they could become more independent and autonomous during their learning in postgraduate program. For example, “…this PhD I feel it’s entirely on my own…so I got to tell myself all the time to be discipline to spend at least two hours to three hours every day looking at my research.” (NSP1) and “…we need to be independent and we submit our proposal we submit our chapters on our at our own pace…” (NSP3); “It allows me to organise my schedule and my supervisors…they don’t micromanage.” (SP1) and “When you are doing your research you know everything you do on your own.” (SP4).
When the interviewees were asked to give opinion on Q2 in the PSLQ, “I choose the research topic after considering many factors (e.g., self-interest, supervisor’s research focus, availability of grants, feasibility of the study),” most of them agreed that choosing a research topic was a challenging task as it required some considerations. For example, SP1 said, “I would say yes because you know to choose a research topic it will take some time for me to think about that…there were many factors to consider before I actually chose what I am doing now. I need to understand and identify the lecturer’s expertise and whether I can really complete what I proposed. Am I being too ambitious like maybe it is too much work and so on. So, I think it is quite a process for us to decide a research topic ya..” SP2 added, “I can’t just simply choose a research topic right. So, I need to think about it carefully by considering many factors especially lecturer’s area of interest. And of course I need to ask myself also what I am interested in. If I don’t even have interest in the topic, I won’t choose it…sometimes we think it is doable but maybe there are some factors like time and cost. So, ya there are many things to think of before choosing the topic.”. Participants from non-science discipline also expressed similar view. For instance, NSP1 explained, “…choosing a research topic is an important decision. I have to think it thoroughly and look at the factors lecturer’s expertise and my interest which may affect my choice on the research topic…there are some factors to consider when choosing a research topic. I think this is very relevant to us postgraduates..” Interviewee NSP3 added, “…it is quite a big thing when choosing a research topic. You are gonna stick with it until the end of your study, so to me it is a very important decision. I have to consider many things like supervisor’s expertise, my own interest…We cannot expect our supervisor to give us a research to do…it is our own effort to decide the topic and think of what we can do about it.”
Besides, when interviewees were asked to evaluate the suitability of Q3 in the PSLQ, “I negotiate with my supervisor(s) on my research work,” some interviewees mentioned they were given freedom to choose their research topic and negotiate their research work. For example, “…It is my research and certain things I need to make decision instead of depending on my supervisor. I should be the one deciding the direction of my research. I am given the freedom to conduct my research so that’s where negotiation takes place. There are opportunities for me to negotiate my research with my SV, instead of following her entirely.” (NSP1). Interviewee NSP1 highlighted that she decided the direction of her research. Three interviewees, NSP2, NSP3, NSP4, expressed that it was important for them to negotiate with supervisors and decide the direction of their research.
In addition to the viewpoints given by non-science students, two science students, SP1 and SP2 expressed similar perspectives. For example, “…I tend to evaluate what my supervisor has told me. Then I would clarify if I didn’t understand or maybe I would discuss more with my supervisor to explain my points… why I did this way why not that way.” (SP1) and “… we students have to make adjustment on our RQ or RO, instead of just taking what our supervisors say. I have the responsibility to do that because it is my own research work and supervisors are there to guide us, not dictating us… there are some negotiations for the improvement of my work.” (SP2). Interviewees SP1 and SP2 could seek clarification or discuss further with supervisor to reach a mutual understanding or find a solution to the problem. Their perspectives clearly illustrated the importance of student-centered learning where they took full responsibility and autonomy of their own learning.
Multidirectional Learning
Some excerpts were extracted to illustrate interviewees’ viewpoints on the occurrence of non-linear learning during their learning process. For example, “It doesn’t matter it’s from YouTube… printed material…online news article, Google Scholar, databases…special report…I just take the information in and use it in my research.” (NSP3). Google Scholar was used by Interviewee NSP3 to connect with the researchers and made request to get their published articles.
Regarding interviewees’ opinions on Q13, “I use social apps (e.g., Facebook, Twitter, Whatsapp) to engage with others (e.g., supervisor(s), researchers, peers) in academic discourse,” interviewees SP2, NSP2, NSP3, NSP4, and NSP5 gave a positive viewpoint. First, Interviewee SP2 said, “I actually depend on social media to actually blast my study so that I can get there like participants.” She highlighted the importance of social media to connect with others and explain her study, aiming to get potential participants for her study. Four non-science interviewees, NSP2, NSP3, NSP4, and NSP5, explained that it was important to use WhatsApp to engage in discussion of research with people such as supervisors and peers. For instance, “…a necessity for us to find information using online, YouTube, or Facebook.” (NSP2). “…we do communicate with each other on WhatsApp I mean with supervisor… when I have questions I can ask my supervisor on WhatsApp instead of email which is much slower… using WhatsApp to stay connected with others is a very convenient way” (NSP3), “Well I really only use WhatsApp to discuss with my supervisors occasionally” (NSP4) and “Yes I do use WhatsApp and Facebook to connect with my supervisor and peers for discussion on my research work. I think using social media is another convenient way to stay connected with them and make things more efficient” (NSP5).
Several positive viewpoints were given by interviewees on Q15, “I use online platform (e.g., Facebook, Research Gate, Twitter) to exchange ideas with others to learn more about my research work.” SP1 said, “I do use Facebook and Research Gate sometimes. I join some research community on Facebook and also followed some experts on Research Gate… I can connect with them and share my questions with them… it definitely improves my understanding and make my research work better because I can obtain different inputs from people around the world.” Similar perspective was shared by SP2 who said, “I have a Research Gate account… it allows me to post questions and discuss with researchers not just local but international. There are many experts willing to answer my questions and I can obtain many different insights from them.” The view was also supported by non-science students. NSP1 explained, “Sometimes I do ask questions on Research Gate…they are many experts around the world so if I can get some insights or ideas from them that would be great for my work. It is not only supervisor and me but I can reach out to other experts when I have doubts”. NSP3 added, “I think to a certain extent yes because I can seek advice or insights from experts worldwide. When I use Research Gate to post questions, I can obtain many answers and some researchers even included references to solve my problems…”. Some interviewees agreed that they learned through communicating and exchanging knowledge with researchers from other universities. For example, “…I got to talk to people from other countries who conducted similar research…” (NSP1), “I communicate with researchers from overseas.” (NSP3). The interviewees shared some mixed viewpoints on Q17, “I engage with related stakeholders (e.g., ministry, agency, organization) when carrying out my research,” when the researchers asked about their engagement with related stakeholders for research purposes. Two non-science interviewees mentioned that they dealt with stakeholders. For example, “…. we must apply from the government agency and the school also… to collect data” (NSP1) and “I do engage with expatriates… in order to carry out my research… and permission from organization… very important part of my work” (NSP4) . In contrast, three science interviewees explained that they were usually involved in laboratory work. For instance, “… not much… I am carrying out animal study” (SP1), “Basically no. Because I conducted the experiment by myself. I did not involve any stakeholders or deal with them during the process. So, I did my work individually most of the time.” (SP2), and “… doing the chemistry and dealing with machines” (SP4), and “… go to the lab to test some other established procedure.” (SP5). When the interviewees were asked to give opinion on Q18, “I disseminate my research work online (e.g., Slide Share, Research Gate, Facebook, and Twitter) to let others know my research work,” some of them explained that they shared their research work with others. For instance, “…I also uploaded my journal online like Research Gate and Google Scholar you know to share my research work with people around the world” (NSP1) and “I believe it is important to do so…knowledge should be available worldwide because I can let more people understand my work. Perhaps it is not benefitting to this group but it might be useful for other groups of people” (SP1). Interviewee SP2 added, “…currently I am still working on the manuscript, so maybe in the future I can publish my work and let others know things that I do. I mean it is beneficial to us students because we may gain some insights from the experts… By sharing my research work, I can connect with more people around the world and we can learn from each other. I think that’s good”. They agreed that they exchanged knowledge with others by uploading research papers on research platforms. The findings revealed that the participants engaged in non-linear form of learning such as finding information from various sources, attending online conference, exchanging knowledge with other researchers, connecting with others through various medium.
Reflective Learning
All the interviewees expressed positive perspectives that their reflective ability was enhanced in postgraduate education. By engaging in double-loop learning, the participants were given opportunities to reflect on their mistakes and make amendments on them. “… you tend to find where your mistakes… so in that sense I think self-reflect is very important.” (SP4), “… supervisor give us feedback… it will make us… think over on like what’s going on or what’s wrong.” (NSP2), and “… you have to constantly reflect do a lot of reflection and from there you will understand what’s wrong.” (NSP3). Additionally, interviewee SP1 explained that when she discovered a new finding, she would relook at her previous knowledge and try to make sense of it. Interviewee SP2 stated that feedback from her supervisor prompted her thinking and reflection to improve from her mistakes. In short, the results from Rasch model analysis and qualitative data confirmed the robustness and appropriateness of the Items Q21 to Q34 in PSLQ that measure double-loop reflection.
21st Century Competencies Development
Generally, the interviewees agreed that the learning process in postgraduate education facilitated the development of their 21st century competencies including subject knowledge, critical thinking, communication, technology literacy, and so on. Some excerpts were shown to illustrate the students’ development of 21st century competencies throughout their learning. “… when we talk to lecturers who are more experienced, they definitely make us think outside the box.” (SP1); “I compare my work to their work… whether I can do similar thing… use similar angle to analyze my data how to form the discussion.” (NSP3); “I’ve learned how to actually convince the people that actually involve work in my study.” (SP2); “Technology literacy… we are using all kinds of technology that is available.” (NSP2). When the interviewees were asked to express opinion on Q39, “I know how to commercialize my research product,” whether they could develop their ability to commercialize their research product during their learning process, some mixed perspectives were discovered. Three science interviewees, SP2, SP3, and SP5, disagreed with the item. For example, interviewee SP3 disagreed by saying “Not so much on the entrepreneurial aspect.” Interviewee SP2 mentioned, “I don’t think so. For my research, I don’t do this. Even though I deal with scientific experiments, there is no commercialisation of product involved. But I believe it will be relevant to studies from other fields.” Additionally, interviewee SP5 expressed, “I tried to put up the prototype but I wasn’t able to proceed to the next stage of commercialising it. It’s not easy to be honest.” He explained that he tried to put up his prototype but did not proceed to the stage of commercializing it. In contrast, three interviewees, SP1, NSP2, NSP4, expressed that they could develop entrepreneurial abilities throughout their studies. SP1 said, “So, what stood out in my class would be another one is entrepreneurial skills… in research we do not just do it just to discover new findings at the end of the day we have to serve the public in return like our findings should benefit the public. So what we’re trying to work on is to probably use our findings to benefit the public by for example coming up with new devices… We’re trying to develop something like that for cancer for other diseases as well. So the cost for diagnosing a patient would be lower when they can get all these kits from their nearby pharmacies.” Interviewee SP1 highlighted the use of entrepreneurial skills as she believed that the findings from her study should benefit the public such as lowering the cost for diagnosing and treating patients. Interviewee NSP2 expressed, “… commercial research product I think yes as well because I will interview a bunch of woman entrepreneur so yeah the key is the entrepreneur skill, innovations on their product as well.” Interviewee NSP4 said that she learned some basic entrepreneurial skills by seeing how the company executives managed their business.
In short, the interview data provided deeper insights about the unfit and more difficult items assumed by Rasch model. The positive perspectives indicated that further evaluation should be made before finalizing the decision on the items.
Discussion
The psychometric properties of the PSLQ were analyzed from the aspects of Item Fit, Item Polarity, Category Fit, Unidimensionality, Item and Person Reliability as well as Wright Distribution Map. The results of Rasch model analysis revealed that even though seven items (Q2, Q3, Q12, Q13, Q15, Q17, Q18) recorded slightly higher MNSQ outfit value, these items showed PTMEA Corr values above zero, suggesting that each item did not distort the construct and could contribute to the measurement of postgraduate students’ self-determined learning (Bond & Fox, 2001). The remaining items of the PSLQ displayed substantive positive PTMEA Corr values. When items are removed, the construct validity and conceptual meaning are reduced and the quality of results is also degraded, so the researchers only considered item removal as a last resort (Finaulahi et al., 2021; Hopkins et al., 2021; Medvedev et al., 2020). Additionally, the instrument strength and item validity of the PSLQ were reinforced via semi-structured interviews with five postgraduate students from science and non-science disciplines respectively. This study portrayed the importance of considering postgraduate students’ perspectives on the items to be included in the PSLQ. This study also proved the strength of Rasch analysis in overcoming the limitations of the classical test theory approach (de Vent et al., 2023; Hergesell, 2022).
The interview data served as the supplementary data to provide further details and explanation on the results of Rasch model analysis. The interview results supported the connection of the four principles of self-determined learning in heutagogic framework designed by Blaschke and Hase (2015). All the interviewees unanimously agreed that they could become more independent and autonomous to manage their own learning in postgraduate education. The results revealed that Items Q2 and Q3 should be retained as these items supported student-centered learning, one of the key principles of self-determined learning. The interviewees highlighted that it is important for postgraduate students to have interest in a research topic as well as to express their opinions about the research work freely. They also claimed to have some forms of negotiation with their supervisor such as deciding their own path of research, selecting appropriate information, and seeking clarification. This finding showed that Items Q2 and Q3 were in line with student-centered learning, the principle of self-determined learning as interest in a research topic could act as driving force in research. The participants also mentioned that they had opportunities to discuss and voice out their perspectives when doing their research work. They could critically review and challenge their supervisor’s opinions on the subject matter. In fact, many things needed to be considered especially their interest in the research before finalizing the research topic. Hence, Items Q2 and Q3 were retained. These items reflect students as the key agent in deciding what and how to learn (Hase & Kenyon, 2013), making the students more independent and autonomous in learning.
The researchers found that all the interviewees agreed that it is a must for them to learn in a multidimensional way. This learning method was aligned with the concept of non-linear learning as an important principle of self-determined learning where students learn from different sources and choose their own path to facilitate learning (Blaschke & Hase, 2015). Items Q12 and Q18, for instance, are related to sharing of academic ideas via online platform. Even though postgraduate students found it challenging, they acknowledged that it is important for them to be self-determined learners and actively share their work online as this will open up opportunities for them to connect and share information with the relevant audience (Blaschke & Hase, 2019). In addition, the interviewees also acknowledged that their engagement with stakeholders was limited, particularly among postgraduate students from the science disciplines. Nevertheless, they agreed that engagement with stakeholders such as the relevant ministry, agency, industry, or community is needed to obtain approval to conduct the study and to ensure that their research meet the industry or community’s needs and have significant contributions to the field. Founded on these reasons, Item Q12, Q17, and Q18 were retained as the items to measure non-linear learning in the PSLQ.
Based on the interview data, most interviewees agreed that they could become more reflective during their venture of postgraduate education. The findings were consistent with the concept of self-reflection and double loop learning, which is known as a key principle of self-determined learning postulated by Blaschke and Hase (2015). The interviewees mentioned that they often engaged in reflecting on the new knowledge and what they had learned. They also challenged their existing beliefs and perceptions when encountering contradictory information during their learning process. It clearly supported that double-loop reflection was one of the principles emphasized in self-determined learning. As a result, all the items (Q21-Q34) measuring the principle of double-loop reflection were retained in the PSLQ. Lastly, the findings indicated that the students’ capacity development could be fostered in their postgraduate programs. Although the Rasch analysis highlighted that postgraduate students found it difficult to agree with Item Q39, which emphasized on commercializing their research product, the interviewees agreed that as postgraduate students it is important for them to learn how to commercialize their research products. The results indicated that Item Q39 was in line with entrepreneurial skills, which is regarded as one of the important 21st century capabilities that students should develop (Ghafar, 2020). As the interviewees agreed that capacity development was crucial for them to become self-determined in learning, the findings supported the theory of self-determined learning which emphasizes on capacity development as one of the key principles (Blaschke & Hase, 2015). Hence, Item Q39 was retained so that postgraduate students’ capacity development in commercialization of research products can be measured.
Based on the results of Wright Distribution Map, four items (Q12, Q17, Q18, Q39) were identified as more difficult among the respondents from both science and non-science disciplines. Students faced difficulties mainly in non-linear learning, particularly in sharing their academic ideas online, engaging with stakeholders when carrying out research, disseminating research work online and commercializing their research products. Although Wright Distribution Map analysis identified the four items as more difficult items, arguably they still received some positive perceptions from the interviewees who supported the importance of the four items contributing to the construct of self-determined learning. Additionally, the findings contributed to the new discovery Wright Distribution Map clearly illustrated the ability of self-determined learning of each student to manage their own learning in postgraduate education. Such an application of Wright Map can also be seen in the past literature to investigate students’ actual abilities in tests (Boone, 2016; Hilaliyah et al., 2019) and attitudes (Jimam et al., 2021; Sabah et al., 2013). The item-person map allows researchers to examine the utility of items for the sample and the extent of how well the items are targeted to the sample (Bailes & Nandakumar, 2020). The items displayed in Wright Map also provided detailed information on the continuum of the items from the highest to lowest complexity for measuring self-determined learning. Hence, it gives a positive indication that the items remain relevant to be included in the instrument to measure postgraduate students’ ability of self-determined learning. The high item reliability index of 0.98 suggests that 98% replication of this order of item estimates can be readily relied on when the PSLQ is given to other suitable samples (Bond & Fox, 2015). Besides, the high person reliability indicated that the current instrument could expect the estimated consistency of the inferences on the low to high scoring among the respondents to measure the same construct (Bond & Fox, 2015), which refers to self-determined learning in this case. The high person separation index explained that the construct has the ability to separate person along their level of self-determined learning.
As a result of the feedback given by the interviewees, all the seven misfitting items such as Items Q2, Q3, Q12, Q13, Q15, Q17, and Q18 were still relevant and aligned with the four principles of self-determined learning. In earlier studies, these four principles have been significantly correlated with self-determined learning (Blaschke & Hase, 2015; Hase & Kenyon, 2013). Hence, the seven items were retained as the participants confirmed the usability and acceptability of those items. The findings also implied that all the 42 items fit the theoretical explanation and fundamental building principles of self-determined learning by Hase and Kenyon (2013). As a result, all 42 items in the PSLQ were retained after considering both quantitative and qualitative findings. The findings suggested that the items in the PSLQ demonstrated good psychometric properties to measure self-determined learning of postgraduate students as a unidimensional construct. The findings confirmed the psychometric properties, appropriateness, and robustness of the PSLQ. The instrument can be employed to determine self-determined learning among postgraduate students from both “science and non-science disciplines.” The findings also suggested that the qualitative data provided deeper insights on top of the Rasch analysis to achieve the content validity of the instrument.
Conclusions
This study aimed to assess the psychometric properties of the PSLQ, a relatively developed instrument that was designed to measure postgraduate students’ self-determined learning. Rasch measurement provides scale and item-level insights that facilitate the adaptation of scales to a specific research population and context (Hergesell, 2022). Additionally, the use of the estimation method for the Rasch model provides clear and understandable output figures as well as increases the understandability of fit statistics based on individual unexpected answers (Törmäkangas, 2011). Overall, the findings revealed that the 42 items in PSLQ are found to have good psychometric properties that function well in assessing self-determined learning of postgraduate students in the context of Malaysian higher education institution.
The current research concluded some significant implications to the academic and practical aspects. As the literature on heutagogy or self-determined learning generally focuses on the implementation or heutagogical design elements, consequently there is a lack of studies on measuring scale of self-determined learning. Therefore, the findings of this study fill in the research gaps by evaluating and establishing a newly developed instrument. Although extensive use of Rasch Modeling can be seen in educational measurement and various areas of social science (Drehmer et al., 2000), it has not been applied in the research area of self-determined learning. This study contributed to the field of self-determined learning and context of the research method as the researchers employed Rasch model analysis as the approach to evaluate questionnaire in the educational field. When Rasch model analysis was applied, this study contributed to evaluating the accuracy and usefulness of the questionnaire. As such, this paper contributed to a better understanding of the item and scale properties to encourage its use in self-determined learning research. The analysis confirmed the robustness of the PSLQ items for its appropriate assessment in measuring postgraduate students’ level of self-determined learning. In terms of practical implications, the current study shed light on an instrument that can be used by relevant stakeholders (e.g., lecturers, administrators, Institute of Postgraduate Studies, Student Support Unit) to measure postgraduate students’ self-determined learning skills, which is fundamental for success in higher education and in IR 4.0 environment. Moreover, the findings served to validate the framework proposed in developing the PSLQ including the principles of self-determined learning stated by Hase and Kenyon (2013). The principles were concluded as student-centered learning, non-linear learning, double-loop learning, and capacity development that transformed consistently into the items that reflected the construct in the PSLQ. Hence, the findings of the study supported the underlying framework and attested the important roles of student-centered learning, non-linear learning, double-loop learning, and capacity development in fostering students’ self-determined learning within postgraduate education. To sum up, the study contributed to the theoretical validity of the study as the scale connected consistently with the theoretical stance maintained by the earlier studies (Blaschke & Hase, 2015; Hase & Kenyon, 2013). The constructs in the PSLQ corresponded to the principles of self-determined learning postulated by Hase and Kenyon (2013) when they explained about the fundamentals of heutagogy in their early work. The results implied that the constructs functioned well and formulated the principles of self-determined learning in postgraduate education. Therefore, these results successfully demonstrated the implications in terms of reinforcing the theoretical stance of this study as well as confirming the item psychometric properties and development of the PSLQ.
The PSLQ can be used to gauge and monitor postgraduate students’ self-determined learning skills (Kamaruddin et al., 2020). The findings found that postgraduate students experienced difficulties in certain aspects of self-determined learning. They face difficulties in sharing their academic ideas online, engaging with stakeholders when carrying out research, disseminating research work online, and commercializing their research products. Postgraduate from the non-science disciplines in particular had more difficulties in non-linear learning, one of the key aspects of self-determined learning. These are important information that can guide lectures and university administrators to customize effective intervention and support to enhance postgraduate students’ self-determined learning skills. There are needs to provide postgraduate students with more opportunities to optimize the use of ICT in their learning and capacity development. This study is not without limitations. There are few limitations to be considered in this study. Firstly, the PSLQ was tested on postgraduate students from one higher education institution. Even though the sample size is adequate for statistical analysis, it is recommended that future studies include larger sample size and cover people from different geographical backgrounds and institutions. In addition, this study has investigated the psychometric properties of PSLQ among postgraduate students from science and non-science disciplines in general. Detailed analysis was not run according to individual science or non-science programs, thereby suggesting that future studies can further investigate the psychometric properties of PSLQ according to different postgraduate programs. Finally, this study demonstrates the use of modern test theory for the evaluation of psychometric properties of items and instrument development. Furthermore, it is a unique contribution to the field of heutagogy, as it provides the means to measure self-determined learning among postgraduate students.
This research was conducted while Tiew Chia Chun was at Universiti Sains Malaysia. He is now at Universiti Tunku Abdul Rahman and may be contacted at tiewcc@utar.edu.my.
Footnotes
Acknowledgements
The authors would like to thank the Ministry of Higher Education Malaysia and the Institute of Postgraduate Studies, Universiti Sains Malaysia for the support to undertake this study and to all the postgraduate students and lectures who have facilitated and supported the data collection processes.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Ministry of Higher Education Malaysia under the Fundamental Research Grant Scheme (FRGS) with Project Code: FRGS/1/2019/SS109/USM/02/3
Ethical Approval
This study has obtained ethical approval to carry out the research from the Human Research Ethics Committee with the approval number: JEPeM USM Code: USM/JEPeM/19090509
Data Availability Statement
Data is available on reasonable request from the corresponding author.