Sage Journals: Discover world-class research

Abstract

One of the questionnaires that will be used to evaluate social learning environments such as Facebook is the Online Social Learning Environment Instrument (OSLEI). The aim of this study was to evaluate the OSLEI using alternative method of analysis via Rasch model toward 49 undergraduate students in Malaysia. To date, no study has evaluated the OSLEI through the lens of Rasch model analysis among Malaysian samples. Firstly, the OSLEI underwent content validation by three experts. The results of the overall percent agreement on eight constructs (perceived usefulness, usage of learning groups, learning outcomes, affective outcomes, self-efficacy, trust, privacy, and teaching member presence) of the OSLEI was 86% for 48 items. Next, 10 measurement properties of Rasch model analysis were employed with the OSLEI. The results revealed that 23 respondents needed to be dropped. Subsequently, the data from the remaining 26 respondents proceeded to Rasch Model analysis of the 43 items in the OSLEI. The results of the item analysis revealed that 13 items and the construct of privacy needed to be dropped. The findings demonstrated that 35 items were robust, valid, and reliable, and could be used to measure online social learning environments.

Keywords

Rasch evaluation validity reliability social learning questionnaire

Introduction

In recent years, the Coronavirus disease 2019 (COVID-19) pandemic has necessitated that most teaching and learning activities be conducted via online learning platforms (Ali, 2020; Daniel, 2020; Lapitan et al., 2021). The learning environment may potentially impact the effectiveness of such activities, especially during the current difficulties. One of the online learning environments identified thus far is the online social learning environment, which has been described in Chawinga’s (2017) work. This term refers to an environment in which students can share knowledge and information, co-create content, and disseminate resources among their peers and lecturers, whereby they are equipped with the ability to control and manage their own learning progress. Furthermore, it allows them to undertake sharing sessions and discussions on pertinent information with their peers and lecturers, which can occur in either a formal or an informal setting.

Accordingly, Raspopovic et al. (2017) have revealed that the social learning environment can offer an interactive learning space for student-teacher communication. Similarly, students are motivated by its presence to engage in active learning through collaborative problem-solving tasks, which is possible via the implementation of the appropriate pedagogical methods. In particular Noyens et al. (2019) emphasized the importance of such learning environments due to their capability to foster students’ social integration, thereby supporting the learning process of less-motivated students throughout the educational field.

Theoretically, the incorporation of online social learning environments is possible by utilizing various social learning tools. Tangentially, social networking tools such as Twitter, Facebook, Myspace, and Instagram allow researchers the space to conduct research and assess the implementation of such tools as a platform for learning, especially in view of the multitude of different emerging online social learning tools. Facebook, in particular, is one of the most popular social networking tools; this is reflected in the growing body of research and interest dedicated to online learning in the educational field in the context of this platform as an online social learning environment. According to Giannikas (2020), Facebook consists of interactive elements that are conducive to a learning environment for tertiary education. Similarly, scholars such as Jumaat et al. (2019) have underlined its status as a platform that advocates social interactions and meaningful learning. Thus, Facebook is undeniably proven to offer extensive potential for the purpose of online learning.

During COVID-19, Facebook has emerged as one of the most widely utilized alternative tools for conducting online teaching and learning activities (Docimo et al., 2020; Mbatha, 2020). Its usage in such activities throughout this period is deemed highly attractive according to many educators and students, especially those hailing from educational institutions (Azlan et al., 2020; Chung et al., 2020). The platform is denoted as being capable of facilitating student-instructor interactions; Facebook offers a new, all-encompassing, and exciting learning climate for students and instructors, as well as serving as an alternative platform to enable students to collaborate on any subject matter (Lee et al., 2021). Concurrently, student participation (Camus et al., 2016) and peer-to-peer learning (Dalsgaard, 2016) are encouraged via this tool.

Moreover, previous scholars such as Wu and Chen (2015) and Ahern et al. (2016) are among those who have underlined positive findings pertaining to the use of Facebook for teaching and learning purposes. In light of its potential use in the online social learning environment, a robust instrument is necessary to measure its function as an online social learning environment. One such instrument is a questionnaire proposed by Hong and Gardner (2014) to measure online social learning environment, which has been tested using statistical analysis, such as Confirmatory Factor Analysis (CFA). Its reliability and validity have been ascertained using Cronbach’s alpha values, Exploratory Factor Analysis (EFA), and CFA, which are widely employed methods for such assessment.

Cronbach’s alpha values, EFA, and CFA were used to validate the online social learning environment instrument in a previous study, and the results showed that it was appropriate for the selected samples. Cronbach’s alpha values, EFA, and CFA, on the other hand, do not deal with measurement quality at the item level because neither analysis used a rigorous and comprehensive analysis like the Rasch model analysis (Embretson & Hershberger, 1999; Park et al., 2021).To overcome the limitations of Cronbach’s alpha values, EFA, and CFA, Rasch model analysis was utilized to assess the items in the current study, which builds on Hong and Gardner’s (2014) questionnaire, dubbed the Online Social Learning Environment Instrument (OSLEI).

To date, no studies have successfully evaluated the above mentioned questionnaire by utilizing Rasch model analysis. This type of analysis may offer a robust assessment that can be used with a small sample, thus providing a detailed evaluation of the study respondents and item analysis. For example, Vogel and Engelhard (2011) opted to incorporate a sample size of just 44 participants for their examination of two instructional approaches for teaching and learning French grammar by employing Rasch model analysis. When dealing with small sample sizes in Rasch analysis, one of the potential concerns is that it is less precise, and item calibration instability may arise. However, O’Neill et al. (2020) discovered that in Rasch analysis, lowering sample size has little influence on person-ability estimation.

Similarly, Rasch model analysis has been undertaken in several studies to establish the validity and reliability of the questionnaires incorporated. The scholars and their respective works include the following: Che Lah et al. (2021) evaluated an online problem-solving skills inventory (OPSI) among 49 samples, while Clinton et al. (2014) examined Ethical Issues Scale (EIS) among 59 samples. According to their findings, Rasch model analysis is a powerful tool for establishing validity and reliability with small sample size. Thus, the current study opts to evaluate the OSLEI by using evidence from Rasch model analysis that focus on applied measurement in education.

Practically, this work aims to evaluate the OSLEI via the use of comprehensive techniques, whereby robust evidence is obtained by implementing Rasch model analysis. Interestingly, it will contribute to the evaluation of the questionnaire and it has had a significant impact on the literature across different contexts; in particular, the study identifies which respondents are fit to participate in the OSLEI assessment, identifies redundant items in the instrument, confirms its constructs, reports on its item and respondent reliability, and ascertains which items can be deemed robust and valid.

Literature Review

Online Social Learning Environment Instrument (OSLEI)

Hong and Gardner (2014), as the pioneers of this instrument, proposed a questionnaire to measure Facebook as a social learning environment for teaching and learning purposes. Eight dimensions were thus measured using a total of 45 items, namely perceived usefulness (5 items), usage of learning groups (13 items), learning outcomes (7 items), affective outcome (4 items), self-efficacy (6 items), trust (3 items), privacy (3 items), and teaching member presence (4 items). Then, the validity and reliability of the original instrument were reported by using the values generated from CFA, Composite Reliability (CR), Average Variance Extracted (AVE), Cronbach’s alpha, and discriminant validity index. The instrument was evaluated using a sample of 258 students enrolled in the University of Auckland with a commerce background.

The scholars then revealed that all reliability, average variance extracted, and Cronbach’s alpha values achieved acceptable construct validity and reliability (Hong & Gardner, 2014). According to Awang (2015), the minimum value for composite reliability is .6 and the average variance extracted is .5. Accordingly, the values generated by the constructs for both elements are as follows: perceived usefulness (CR = .909, AVE = .715), usage of learning groups (CR = .89, AVE = .576), learning outcomes (CR = .924, AVE = .634), affective outcome (CR = .879, AVE = .644), self-efficacy (CR = .909, AVE = .633), trust (CR = .908, AVE = .832), privacy (CR = .878, AVE = .707), and teaching member presence (CR = .838, AVE = .721).

Hong and Gardner (2014) also reported the Cronbach’s alpha values obtained, which were as follows: perceived usefulness (.867), usage of learning groups (.852), learning outcomes (.904), affective outcome (.816), self-efficacy (.871), trust (.798), privacy (.792), and teaching member presence (.613). Besides, the instrument yielded discriminant validity values indicating that no constructs were redundant, namely perceived usefulness (.845), usage of learning groups (.759), learning outcomes (.796), affective outcome (.803), self-efficacy (.814), trust (.912), privacy (.841), and teaching member presence (.849).

However, several limitations have been highlighted regarding the validity and reliability of the original instrument. First, the previous analysis required a large sample. This is a requirement of CFA, for which the minimum sample size is between 100 and 150 depending on the number of constructs. Second, CFA assesses each item without eliminating unfit respondents from the analysis, thus rendering the analysis less comprehensive and robust than Rasch model analysis. Moreover, Hong and Gardner’s (2014) instrument has been revealed to contain redundant double-barreled items.

Therefore, this study modifies the instrument proposed by Hong and Gardner (2014), which is then named the Online Social Learning Environment Instrument (OSLEI). The OSLEI consists of two parts (Parts A and B) and includes several adjustments, such as the addition of new infographic items. Furthermore, the inclusion of several items with a specific element allows the measurement of the online social learning environment in Part B, while some items are modified due to their double-barreled property. To date, an evaluation of the OSLEI has yet to be undertaken using a Malaysian sample via Rasch model analysis implementation, rendering this work critical to ensure that it is a more valid and robust instrument following its analysis using such a powerful method.

Rasch Model Analysis

Rasch (1966) is a Danish mathematician who is credited for developing the Rasch measurement model as a special case of the general linear model (GLM). The Rasch model is based on the Item Response Theory (IRT), which allows the calculation of the probability of correct responses to the test items and the strength of endorsement for the rating scale items (Shea et al., 2012). The model analysis thus serves as a powerful and comprehensive method, establishing the instrument’s validity and reliability and providing a detailed analysis of the respondents who complete the instrument. Similarly, respondents who have yet to answer the questionnaire can also be easily detected. Rasch model analysis can also easily classify fit respondents based on their level into several categories with Rasch probability.

Furthermore, the Rasch measurement model is capable of transforming ordinal data to ratio data, allowing data to be obtained for many types of appropriate analyses. This is attributable to the data being of the highest-ranking type for analysis. Moreover, Rasch model analysis could increase the questionnaire validity for conducting research. Besides that, Rasch model analysis also could categorization of each item in the questionnaire used into several levels, such as easy, moderate, and difficult items. Concurrently, the Rasch model analysis permits the detection of redundant items and the rating scale in the questionnaire, while also revealing the Cronbach’s alpha value, item reliability, and person reliability. Thus, Rasch model analysis is justifiably named as a powerful tool geared for establishing the reliability and validity of questionnaires.

In theoretical terms, subjecting instruments to Rasch model analysis is an increasingly popular method for education-based works, particularly in educational filed (Bode & Wright, 1999). To date, a multitude of studies has implemented this method in education and online learning-focused contexts, such as Lah and Tasir’s (2018) validity and reliability measurement of an online social presence questionnaire. Meanwhile, Yang et al. (2018) assessed an instrument which measures Grade 4 to 8 students’ understanding of interdisciplinary science throughout multiple semesters. Yan’s (2018a, 2018b) work validated the self-assessment practice scale (SaPS). Meanwhile, the psychometric properties of the Motivated Strategies for Learning Questionnaire (MSLQ) were evaluated by Bonanomi et al. (2018) among high school students, whereas Toland et al. (2021) evaluated the Assessment, Evaluation, and Programing System Test—Third Edition (AEPS-3). Other comparable studies also include Rahayu et al.’s (2021) validation of an Indonesian version of the “What is happening in this Class?” (WIHIC) questionnaire. In brief, these prior works have revealed that Rasch model analysis is a powerful analytical tool capable of providing valid, reliable, and robust items for their respective instruments.

Furthermore, evaluation of instrument utilization has gradually been deemed acceptable with Rasch model analysis in the field of applied measurement in education, as evidenced by the works of authors such as Abbakumov et al. (2020) who explored formative assessments in MOOCs, Randall and Engelhard (2010) who measured high stakes reading assessment and DeMars (2021) who conducted Violation of Conditional Independence in the Many-Facets Rasch Model.

According to Bond and Fox (2007), Rasch model analysis is a comprehensive assessment tasked with probing the validity of an instrument. Coe (2008) and Wilmot et al. (2011) are of the opinion that its incorporation offers evidence of the relationship between questionnaire items and the respondents. Its analysis encompasses elements such as rating scale, items, persons, and other facets, such as raters (Pomeranz et al., 2008). Alternatively, Tennant and Conaghan (2007) have listed 11 measurement properties of Rasch model analysis as follows: (i) model chosen, (ii) appropriate ordering categories, (iii) fit of items, (iv) fit of person, (v) summary statistics, (vi) local independence of items, (vii) response dependency, (viii) unidimensionality, (ix) presence of Differential Item Functioning (DIF), (x) targeting of the scale, and (xi) person separation reliability.

In light of the empirical evidence identified by conducting validity and reliability testing via Rasch model analysis, scholars have been driven to employ this method to evaluate their instruments. Examples of such scholars and their works include Rahayu et al. (2021) and Eddy et al. (2021). Performing the analysis renders it easier for researchers to trace the respective responses of respondents to specific items and the manner in which each item can fit or is suitable for the questionnaire. Thus, this study employs Rasch model analysis to evaluate the OSLEI as alternative method, thereby addressing the gap in the literature with regard to the application of this method to establish robust items in the instrument. This will generate a powerful method to establish validity and reliability of questionnaire.

Method

Instrument

This study adapted a questionnaire from Hong and Gardner (2014), with some adjustments made to several items, and thus renamed it as the OSLEI. It includes two parts—Part A and Part B—which denote the sections on Demographics and Online Social Learning Environment, respectively. Part B consisted of eight constructs, namely: (i) perceived usefulness (4 items); ii) usage of learning groups (8 items); (iii) learning outcomes (7 items); (iv) affective outcome (4 items); (v) self-efficacy (10 items); (vii) trust (3 items); (vii) privacy (3 items); and (viii) teaching member presence (4 items). A 5-point scale was thus employed for the respondents to answer each item as follows: 1 = Strongly Disagree, 2 = Disagree, 3 = Somewhat Disagree, 4 = Agree, and 5 = Strongly Agree. A summary of the constructs and items included in the OSLEI’s Part B is provided in Table 1.

Table 1.

Constructs and Items in Part B.

Constructs	Number of items	Total items
Perceived usefulness	4, 5, 6, 7	4
Usage of learning groups	8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20	13
Learning outcomes	21, 22, 23, 24, 25, 26, 27	7
Affective outcome	28, 29, 30, 31	4
Self-efficacy	32, 33, 34, 35, 36, 37, 38, 39, 40, 41	10
Trust	42, 43, 44	3
Privacy	45, 46, 47	3
Teaching member presence	48, 49, 50, 51	4
Overall items		48

Software

Two software programs were utilized in this study, namely IBM’s Statistical Package for Social Sciences (SPSS) and WINSTEPS software version 3.72. The first program was employed to analyze Part A of the OSLEI (i.e., frequencies and percentages), while the second program undertook the assessment of the 10 measurement properties in the Rasch model analysis in Part B.

Procedure of Analysis

Content analysis of the OSLEI was first performed by three experts, following which items in Part A were subjected to descriptive analysis while Rasch model analysis was performed on items in Part B which employed rating scales items which are the respondents need to answer five-point scale such as 1 = Strongly Disagree, 2 = Disagree, 3 = Somewhat Disagree, 4 = Agree, and 5 = Strongly Agree. The 10 measurement properties of Rasch model analysis evaluated included: (i) person infit mean-square (MNSQ), (ii) person infit z-standardized (ZSTD), (iii) person point measure correlation, (iv) item infit MNSQ, (v) item infit ZSTD, (vi) item point measure correlation, (vii) item measure value, (viii) standardized residual correlation, (ix) dimensionality, and (x) summary statistics.

In this study, the misfit respondents were analyzed by using three measurement properties, namely person infit MNSQ, person infit ZSTD, and person point measure correlation. Once they were identified, they were dropped, while the remaining fit respondents would proceed with item analysis. Here, item infit MNSQ and item infit ZSTD were performed on the OSLEI, whereby the item point measure correlation was then carried out after the items were deemed fit. Next, singling out redundant items was ensured by assessing the measurement properties of item measure value and standardized residual correlation on the instrument, followed by unidimensionality evaluation to ensure its validity in measuring the element requiring measurement. Subsequently, the reliability of the items and respondents in the OSLEI was reviewed by using summary statistics.

Measurement Properties of Rasch Model Analysis

In this work, the OSLEI was subjected to the 10 measurement properties of Rasch model analysis and the resulting outcomes are detailed in Table 2.

Table 2.

Measurement properties of Rasch model analysis in OSLEI.

Rasch analysis	Purpose	Measurement properties	Acceptable values	References
Person fit	To assess whether each respondent in the completing the OSLEI were fit and they answered OSLEI in the manner.	Person infit MNSQ	0.4 < x < 1.5	Linacre (2002); Wright et al., (1994)
		Person infit ZSTD	−2 < x < +2	Aziz et al. (2013)
		Person point measure correlation	Positive value	Aziz et al. (2013)
Item fit	To assess whether each item fit is really fit to measure OSLEI.	Item infit MNSQ	0.4 < x < 1.5	Linacre (2002); Wright et al., (1994)
		Item infit ZSTD	−2 < x < +2	Aziz et al. (2013)
	To assess the all the items are aligned to measure OSLEI.	Item point measure correlation	Positive value	Aziz et al. (2013)
Item measure	Item measure value is the analysis of detecting item in OSLEI which are measuring the same things in a same construct.	Item measure	• There is no same value of item measured in the same construct • The selected items based on the criteria infit MNSQ near to 1 and infit ZSTD near to 0	Aziz et al. (2013)
Standardized residual correlations	To examine the dependent item which are measuring the same things in OSLEI.	Standardized residual correlations	• Correlation is <.7The retention item, the MNSQ should be selected as close to 1 and ZSTD as close to 0	Linacre (2002)
Dimensionality	To validate the OSLEI could be to measure online social learning environment.	Principal component analysis (PCA)	Raw variance explained by measures should be >40%, and unexplained first contrast of should be <15%	Fisher (2007)
Summary statistic	To reveal the result of item separation, item reliability, person reliability, person separation, and Cronbach Alpha in OSLEI.	Summary statistic	• Item separation • Item reliability • Person • Reliability • Person separation • Cronbach’s alpha.	Fisher (2007); Ghandi (2012)

Participants

The OSLEI was administered to the study participants and the first part of the questionnaire gathered their demographic information, such as gender, experience in online learning, and experience of using Facebook for teaching and learning purposes. However, Rasch model analysis was not employed for these demographic items. According to Linacre (1994), performing Rasch model analysis necessitates a minimum requirement of five to six respondents per instrument. Therefore, a total of 49 undergraduate students (18 males and 31 females) enrolled in one course, namely a multimedia applications and web design course at a university in Malaysia, were selected as the study sample by using a purposive sampling technique. According to Etikan et al. (2016) the purposive sampling techniques also known as judgement sampling, it is intentional choice based on the included qualities of the participants. The rationale of using purposive sampling in this study is to focus on particular characteristics of a population that are of interest.

The inclusion criteria for respondents for this study selection included: (i) enrolment in a computer-based subject, (ii) being equipped with any experience in an online learning environment, and (iii) experience of using Facebook in teaching and learning activities. These criteria have been chosen since this study focused on particular characteristics of samples. Demographic results subsequently revealed that all respondents (100%) had experience with online learning and used Facebook for teaching and learning purposes.

Results

Percent Agreement Validity of the OSLEI

The content validity of OSLEI was determined by consulting three educational technology specialists, all of whom were senior academics at one of Malaysia’s institutions. Their selection was based on their extensive teaching experience of more than 15 years, as well as their standing as specialists in online learning. As a guideline, percent agreement was utilized to achieve the assessor scores as per the suggestion of Nordin and Bakar (2008). The scores given by the assessors to evaluate the constructs are: 1—items unsuitable for measurement, 2—measurable item, and 3—item requiring improvement. Here, the percent agreement outcome for each construct is as follows: perceived usefulness (100%), usage of learning groups (83%), learning outcomes (62%), affective outcome (100%), self-efficacy (60%), trust (100%), privacy (100%), and teaching member presence (83%). Furthermore, the overall percentage agreement validity of the OSLEI is 86%. Table 3 displays the details of the instrument’s percentage agreement

Table 3.

Percent-Agreement OSLEI.

Construct	No	Item	Agreement Expert 1			Agreement Expert 2			Agreement Expert 3			Agreement between Expert 1 with Expert 2	Agreement between Expert 1 with Expert 3	Agreement between Expert 2 with Expert 3	Total agreement	Percent-Agreement (%)
Construct	No	Item	1	2	3	1	2	3	1	2	3	Agreement between Expert 1 with Expert 2	Agreement between Expert 1 with Expert 3	Agreement between Expert 2 with Expert 3	Total agreement	Percent-Agreement (%)
Perceived usefulness	Q4	I think Facebook learning groups can be useful in my courses.		√			√			√		1	1	1	3/3	100
	Q5	I think that by joining Facebook learning groups, I will get information I need quickly.		√			√			√		1	1	1	3/3
	Q6	I feel that by joining Facebook learning groups, will have support for my learning.		√			√			√		1	1	1	3/3
	Q7	I feel that by joining Facebook learning groups, my understanding/learning of that course will improve.		√			√			√		1	1	1	3/3
Usage of learning groups	I use Facebook groups to
	Q8	Clarify study materials.		√			√			√		1	1	1	3/3	83
	Q9	Get guidance for (e.g., assignments, missed lecture content)		√			√			√		1	1	1	3/3
	Q10	Get clearer explanations on course topics.		√			√			√		1	1	1	3/3
	Q11	Retrieve relevant materials that are useful to course.		√			√			√		1	1	1	3/3
	Q12	Retrieve admin information (e.g., due dates, notifications, general FAQ).		√			√			√		1	1	1	3/3
	Q13	Retrieve procedural information (e.g., marks now available on. . ..).		√			√			√		1	1	1	3/3
	I usually ask questions
	Q14	To clarify (i.e., my answers, due dates,) what I should do to complete course-related tasks/activities in Facebook groups.		√			√			√		1	1	1	3/3
	Q15	When I am confused with the course content (did not fully understand with what was being covered in lectures or tutorials) in Facebook groups.		√			√			√		1	1	1	3/3
	Q16	In Facebook groups, when I am not able to link the various concepts taught in the course in a holistic manner.		√			√				√	1	0	0	1/3
	I tend to post
	Q17	Course-related content queries (i.e., topics covered, assignments, tests, or exams) in Facebook groups.		√			√			√		1	1	1	3/3
	Q18	Admin-related queries (e.g., notifications, procedural information, due dates, general admin queries) in Facebook groups.		√			√			√		1	1	1	3/3
	Q19	How I feel toward (i.e., assignments, tests, topics) in Facebook groups.		√				√		√		0	1	0	1/3
	Q20	Or comment more frequently during period nearing to due dates (of i.e., assignments/tests/exams) in Facebook groups.		√				√		√		0	1	0	1/3
Learning outcome	Q21	Using Facebook learning group has improved my grades.			√		√			√		0	1	0	1/3	62
	Q22	Using Facebook learning group increases my productivity.		√			√			√		1	1	1	3/3
	Q23	I feel that less time is needed to comprehend study materials when using Facebook learning groups.		√			√				√	1	0	0	1/3
	Q24	Using Facebook learning group enables me to accomplish tasks (i.e., assignments) more quickly.		√			√			√		1	1	1	3/3
	Q25	I perceive that I have an increase knowledge of that course when using Facebook learning group		√			√				√	1	0	0	1/3
	Q26	I am more able to critically analyze course materials when using Facebook learning groups.		√			√				√	1	0	0	1/3
	Q27	The use of Facebook learning groups improves my understanding of course materials.		√			√			√		1	1	1	3/3
Affective outcome		By participating in Facebook groups. . .. . .. . .														100
	Q28	I feel more satisfied with my study progress.	√				√			√		1	1	1	3/3
	Q29	I feel more confident in interacting with strangers as a result of participating in Facebook learning groups.	√				√			√		1	1	1	3/3
	Q30	I feel more motivated to do better in my studies.	√				√			√		1	1	1	3/3
	Q31	I feel less frustrated with my studies because questions asked in Facebook learning groups are being answered in a quick manner.	√				√			√		1	1	1	3/3
Self-efficacy	I feel confident to															60
	Q32	Decide where and how to find the information I need in Facebook groups.			√		√			√		0	0	1	1/3
	Q33	Post questions in Facebook groups.			√		√			√		0	0	1	1/3
	Q34	Comment on posts in Facebook groups.			√		√			√		0	0	1	1/3
	Q35	Interact with others even though I might not know them in the first place in Facebook groups.			√		√			√		0	0	1	1/3
	Q36	Follow and learn along with an online discussion created by others in Facebook groups.			√		√			√		0	0	1	1/3
	I feel competent to
	Q37	Decide where and how to find the information I need in Facebook groups.		√			√			√		1	0	1	1/3
	Q38	Post questions in Facebook groups.		√			√			√		1	1	1	3/3
	Q39	Comment on posts in Facebook groups.		√			√			√		1	1	1	3/3
	Q40	Interact with others even though I might not know them in the first place in Facebook groups.		√			√			√		1	1	1	3/3
	Q41	Follow and learn along with an online discussion created by others in Facebook groups.		√			√			√		1	1	1	3/3
Trust	In Facebook groups, I trust that															100
	Q42	The information provided is correct.		√			√			√		1	1	1	3/3
	Q43	My course mates will help in answering my queries.		√			√			√		1	1	1	3/3
	Q44	My course mates will be understanding if constructive criticisms are made in regards to their posts.		√			√			√		1	1	1	3/3
Privacy	Q45	I am concerned that my personal information might be exposed as a result from participating in Facebook learning groups.		√			√			√		1	1	1	3/3	100
	Q46	I would create a separate account in Facebook learning groups for use in my courses.		√			√			√		1	1	1	3/3
	Q47	I feel the need to change the privacy settings of my Facebook account to keep my account private.		√			√			√		1	1	1	3/3
Teaching member presence	Q48	I find lecturers/teaching members on Facebook learning groups encourages me from participating.		√			√				√	1	0	0	1/3	83
	Q49	I feel more confident that posts are correct when lecturers/teaching members are participating in/part of Facebook learning groups.		√			√			√		1	1	1	3/3
	Q50	I like having lecturers/teaching members participating/commenting in the Facebook learning groups.		√			√			√		1	1	1	3/3
	Q51	Regular support from teaching members on Facebook Learning Groups is important to my success in learning.		√			√			√		1	1	1	3/3
Overall																86

Note. 0 = different level of agreement between experts; √ = agreement from expert based on the three scales of agreement, which are: Scale 1—items that are unsuitable for measurement, Scale 2—items that can be measured, Scale 3—items that should be improved.

1 = Same level of agreement between experts.

Person infit MNSQ and ZSTD of the OSLEI

Table 4 displays the results of person infit MNSQ and ZSTD. The values from the first analysis revealed that eight respondents should be dropped due to their non-adherence to the person infit MNSQ and ZSTD criteria. The respondents were: respondent number 6 (MNSQ: 7.54, ZSTD: 9.9), respondent number 37 (MNSQ: 2.96, ZSTD: 6.3), respondent number 21 (MNSQ: 2.24, ZSTD: 4.9), respondent number 12 (MNSQ: 1.86, ZSTD: 4.1), respondent number 3 (MNSQ: 1.79, ZSTD: 2.6), respondent number 18 (MNSQ: 1.64, ZSTD: 2.2), respondent number 10 (MNSQ: 2.96, ZSTD: 6.3), and respondent number 28 (MNSQ: 1.61, ZSTD: 2.1).

Table 4.

Summary for Person Infit MNSQ and ZSTD of OSLEI.

Analysis	Respondents number	Infit MNSQ	Infit ZSTD	Total respondents dropped
First	6	7.54	9.9	8
	37	2.96	6.3
	21	2.24	4.9
	12	1.86	4.1
	3	1.79	2.6
	18	1.64	2.2
	10	1.56	2.1
	28	1.61	2.1
	39	0.56	−2.2	14
	38	0.55	−2.4
	35	0.55	−2.2
	16	0.48	−3.3
	29	0.49	−3.0
	30	0.48	−2.5
	1	0.39	−3.7
	32	0.39	−3.7
	31	0.40	−4.0
	19	0.32	−4.5
	41	0.25	−4.4
	42	0.16	−5.4
	25	0.06	−4.0
	44	0.06	−4.5
Second	4	1.53	3.0	1
Overall total				23

Note. MNSQ = Mean square, 0.4 < x < 1.5; ZSTD = z-standard, − < x < +2.

Furthermore, another 14 respondents were to be dropped due to their non-adherence to the person infit ZSTD criteria. They were: respondent number 39 (ZSTD: −2.2), respondent number 38 (ZSTD: −2.4), respondent number 35 (ZSTD: −2.2), respondent number 16 (ZSTD: −3.3), respondent number 29 (ZSTD: −3.0), respondent number 30 (ZSTD: −2.5), respondent number 1 (ZSTD: −3.7), respondent number 32 (ZSTD: −3.7), respondent number 31 (ZSTD: −4.0), respondent number 19 (ZSTD: −4.5), respondent number 41 (ZSTD: −4.4), respondent number 42 (ZSTD: −5.4), respondent number 25 (ZSTD: −4.0), and respondent number 44 (ZSTD: −4.5).

In the second analysis, only one respondent needed to be dropped due to non-adherence to the person infit MNSQ and ZSTD criteria (respondent number 4; MNSQ: 1.53, ZSTD: 3.0). According to Linacre (2002) and Wright et al. (1994), the recommended value for the person infit MNSQ should be within the range of 0.4 < x < 1.5, whereas Aziz et al. (2013) have suggested that the value of person infit ZSTD should be within the range of −2 to +2. In total, 23 out of 49 respondents were removed from the analysis because they yielded values outside the range of the criteria above.

Person Point Measure Correlation of the OSLEI

The remaining 26 respondents retained in the study were then subjected to person point measure correlation in the OSLEI, which yielded interesting outcomes. In particular, all 26 values generated by these respondents were positive, meaning that they were retained for the subsequent process. Aziz et al. (2013) have suggested that the value of the person point measure correlation should be positive for person polarity. Table 5 depicts the results of the person point measure correlation in the OSLEI, whereby all 26 respondents proceeded with item analysis accordingly.

Table 5.

Person Point Measure Correlation of OSLEI.

Analysis	Respondents number	Point measure correlation	Total respondents dropped
First	27	.36	0
	47	.33
	23	.08
	40	.15
	34	.17
	33	.15
	36	.13
	2	.43
	45	.48
	48	.47
	8	.27
	14	.44
	24	.24
	5	.43
	46	.01
	17	.21
	43	.27
	49	.26
	11	.41
	9	.17
	26	.09
	20	.31
	7	.58
	22	.15
	15	.37
	13	.42
Overall total			0

Note. Point measure correlation = positive value.

Item Infit MNSQ and ZSTD in the OSLEI

An assessment for the item infit MNSQ and ZSTD in the OSLEI (i.e., 48 items in Part B) was performed based on the 26 respondents retained thus far. The first analysis revealed that six items should be dropped due to non-adherence to the item infit MNSQ and ZSTD criteria, specifically three, two, and one items non-adhering to the item infit MNSQ and ZSTD, item infit MNSQ, and item infit ZSTD, respectively. The three items that were non-adherent to the item infit MNSQ and ZSTD criteria were: item number Q47 (MSNQ: 2.15, ZSTD: 2.9), item number Q46 (MSNQ: 2.24, ZSTD: 3.7), and item number Q45 (MSNQ: 1.77, ZSTD: 2.1). Meanwhile, the two items that were non-adherent to the item infit MNSQ criteria were: item number Q10 (MSNQ: 1.58) and item number Q15 (MSNQ: 1.52), whereas the one item non-adherent to the item infit ZSTD criteria was item number Q37 (ZSTD: −2.6).

According to Linacre (2002) and Wright et al. (1994), the range of item infit MNSQ values should be within 0.4 < x < 1.5, whereas the value of item infit ZSTD should be within −2 < x < +2, as suggested by Aziz et al. (2013). Thus, the privacy construct was dropped, because none of its items (i.e., Q45, Q46, and Q47) adhered to the criteria of item infit MNSQ and ZSTD. Table 6 shows the results of item infit MNSQ and ZSTD in the OSLEI that needed to be dropped.

Table 6.

Item infit MSNQ and ZTSD of OSLEI.

Analysis	Item number	Infit MNSQ	Infit ZSTD	Total item dropped
First	Q47	2.15	2.9	3
	Q46	2.24	3.7
	Q45	1.77	2.1
	Q10	1.58	1.6	2
	Q15	1.52	1.4	2
Second	Q37	0.37	−2.6	1
Overall total				6

Note. MNSQ = Mean square, 0.4 < x < 1.5; ZSTD = z-standard, −2 < x < +2.

Item Point Measure Correlation of the OSLEI

The remaining 42 OSLEI items retained thus far were then subjected to item point measure correlation, which also indicated a positive value for all items. Thus, no items were dropped in this step. This echoes Aziz et al. (2013) statement that the point measure correlation for items analyzed should yield a positive value. Tables 7 and 8 below provides the results of item point measure correlation for all 42 items retained in the OSLEI.

Table 7.

Item Point Measure Correlation of OSLEI.

Item number	Point measure correlation	Total item dropped
Q19	.65	0
Q42	.75
Q8	.54
Q16	.49
Q26	.65
Q21	.72
Q23	.68
Q24	.68
Q28	.74
Q9	.64
Q11	.60
Q16	.53
Q18	.48
Q44	.40
Q13	.29
Q20	.59
Q22	.85
Q31	.58
Q38	.68
Q48	.53
Q30	.64
Q43	.69
Q33	.70
Q35	.63
Q40	.57
Q12	.40
Q41	.55
Q5	.63
Q32	.69
Q34	.70
Q27	.66
Q29	.24
Q36	.59
Q50	.68
Q6	.70
Q14	.45
Q25	.72
Q49	.64
Q4	.64
Q7	.64
Q51	.58

Note. Point measure correlation = positive value.

Table 8.

Summary of Item Measure Value in OSLEI.

Construct	Item number	Measure value	Infit MNSQ	Infit ZSTD	Result	Total item dropped
Learning outcomes	Q23	0.51	0.85	−0.5	Retained	1
Learning outcomes	Q24	0.51	0.71	−1.1	Dropped	1
Usage of learning group	Q9	0.30	0.96	0.0	Retained	3
	Q11	0.30	0.93	−0.2	Dropped
	Q16	0.30	1.21	0.8	Dropped
	Q18	0.30	1.34	1.2	Dropped
	Q13	0.19	1.33	1.1	Dropped	1
	Q20	0.19	1.33	1.1	Retained	1
Perceived usefulness	Q4	−0.81	0.83	−0.5	Dropped
Perceived usefulness	Q7	−0.81	1.05	0.3	Retained	1
Total						6

Note. MNSQ = Mean square; ZSTD = z-standard.

Item Measure Value in the OSLEI

Next, the item measure values of all 42 retained OSLEI items were ascertained. This analysis identified the redundancy of some items. This was revealed following the measurement of the same items under one construct, yielding the same measure value. Accordingly, the results revealed that two items in the learning outcomes construct measured the same thing, namely Q23 “I feel that less time is needed to comprehend study materials when using Facebook learning groups” and Q24 “Using Facebook learning group enables me to accomplish tasks (i.e., assignments) more quickly.” Meanwhile, four items (i.e., Q9 “I use Facebook groups to get guidance for (e.g., assignments, missed lecture content,” Q11 “I use Facebook groups to retrieve relevant materials that are useful to course,” Q16 “I usually ask questions in Facebook groups, when I am not able to link the various concepts taught in the course in a holistic manner,” and Q18 “I tend to post admin-related queries (e.g., notifications, procedural information, due dates, general admin queries) in Facebook groups” were detected to measure the same thing in the usage of learning group construct, whereas two items (i.e., Q13 “I use Facebook groups to retrieve procedural information (e.g., marks now available on)”and Q20 “I tend to post or comment more frequently during period nearing to due dates (of i.e., assignments/tests/exams) in Facebook groups.” measured the same thing in the same construct. Moreover, items Q4 “I think Facebook learning groups can be useful in my courses” and Q7 “I feel that by joining Facebook learning groups, my understanding/learning of that course will improve” measured the same thing in the perceived usefulness construct.

The above outcomes proved the redundancy of several items, requiring them to be dropped. Identifying the retained items necessitated the implementation of criteria selection based on item infit MNSQ near to 1 and infit ZSTD near to 0 as per the recommendations of Aziz et al. (2013). Thus, items that were retained were items Q23 (infit MNSQ: 0.85, infit ZSTD: −0.5) and Q9 (infit MNSQ: 0.96, infit ZSTD: 0), whereas items Q13 (infit MNSQ: 0.85, infit ZSTD: −0.5) and Q20 (infit MNSQ: 0.85, infit ZSTD: −0.5) generated the same value for their infit MNSQ and infit ZSTD properties.

Furthermore, the selection of retained items is based on those generating a high value of item point measure correlation, namely items Q13 (0.29) and Q20 (0.59). Moreover, items Q20 (infit MNSQ: 0.85, infit ZSTD: −0.5) and Q7 (infit MNSQ: 1.05, infit ZSTD: 0.3) were also retained in line with the recommendations of Aziz et al. (2013), as shown in Table 7. However, six items were dropped due to non-adherence to the item measure value criteria, namely: item Q24 (infit MNSQ: 0.71, infit ZSTD: −1.1), item Q11 (infit MNSQ: 0.93, infit ZSTD: −2), item Q16 (infit MNSQ: 1.21, infit ZSTD: 0.8), item Q18 (infit MNSQ: 1.34, infit ZSTD: 1.2), item Q13 (infit MNSQ: 1.33, infit ZSTD: 1.1), and item Q4 (infit MNSQ: 0.83, infit ZSTD: −0.5). They were found to be misfits in the instrument based on the item measure value analysis.

Standardized Residual Correlations of the OSLEI

The subsequent step of standardized residual correlations was then undertaken for the remaining 36 items retained in the OSLEI. This analysis further proved that no items were redundant. According to Linacre (2002), this value should be <.7. Table 9 depicts the adherence of all items to this suggestion, except for items Q5 and Q6 (0.85). Thus, one of these items should be dropped. The selection of retained items was guided by the following criteria: the MNSQ should be close to 1 while the ZSTD should be close to 0 (Linacre, 2002). Thus, item Q5 (infit MNSQ: 0.96, infit ZSTD: 0) was retained, as it adhered to the criteria above, whereas item Q6 (infit MNSQ: 0.84, infit ZSTD: −4) was dropped due to its non-adherence. This was the only 1 out of 36 items to be dropped in this analysis.

Table 9.

Standardized Residual Correlations.

Correlation	Item number	Item number
.84	Q5	Q6
.51	Q41	Q44
.49	Q23	Q25
.47	Q26	Q42
.47	Q26	Q31
.44	Q6	Q22
−.62	Q9	Q23
−.51	Q25	Q38
−.48	Q6	Q29
−.47	Q31	Q34

Note. Correlation is <.7.

Unidimensionality of the OSLEI

Next, all 35 retained items of the OSLEI proceeded to unidimensionality analysis, which would substantiate its accuracy in measuring the social learning environment. Here, the residual Principal Component Analysis (PCA) provided the percentages of raw variance explained by the measures and unexplained first contrast for the instrument. According to Fisher (2007), the value of the former element should be > 40% to ensure the instrument’s validity in terms of measuring what it is supposed to measure, whereas the suggested value of the latter element is <15%. As such, the results obtained revealed that the raw variance explained by the measures was 39.2%, while the unexplained first contrast was 9.7% prior to Rasch model evaluation of the OSLEI, whereas these values were increased and decreased, respectively, to 43.4% and 7.5% following the assessment. Thus, the OSLEI was proven to be a valid instrument to measure the online social learning environment. Table 10 below portrays the unidimensionality results of the OSLEI in detail.

Table 10.

Unidimensionality of OSLEI.

	Raw variance explained by measures (%)	Unexplained variance in first contrast (%)
OSLEI before using Rasch model analysis (i.e., before items were deleted)	39.2	9.7
OSLEI after using Rasch model analysis (i.e., after items were deleted)	43.4	7.5

Summary Statistics of the OSLEI

The next step denoted an analysis of the summary statistics for the remaining 35 items retained thus far in the OSLEI, revealing that the value of person separation was 4.6, which is very good, according to Fisher’s (2007) suggestion that this value should be in the range of 4 to 5. Person reliability was .95, which is excellent, as recommended by Fisher (2007), who suggests a range of 0.94. Cronbach’s alpha was .95, which is excellent, according to Ghandi (2012), as the recommended range is 0.9 to 1.0. Item separation was 1.21 which is acceptable, based on the criteria suggested by Fisher (2007); and item reliability was .59, which is acceptable, as recommended by Fisher (2007). Table 11 provides the summary statistics generated for the 35 items, for which 26 respondents were thus retained.

Table 11.

Summary Statistic for OSLEI.

Summary statistic	Value
Person separation	4.24
Person reliability	.95
Item separation	1.32
Item reliability	.63
Cronbanch’s alpha	.95

Robust Items in the OSLEI

Table 12 provides a summary of the 35 items deemed robust in the OSLEI after it had been subjected to the Rasch model analysis. In particular, the first construct of perceived usefulness was measured by two robust items, namely Q5 and Q7, while the second construct of usage of learning group was measured by seven robust items, namely Q8, Q9, Q12, Q14, Q17, Q19, and Q20. Next, the third construct of learning outcomes was measured by six robust items, namely Q21, Q22, Q23, Q25, Q26, and Q27, whereas the fourth construct of affective outcome was measured by four robust items, namely Q28, Q29, Q30, and Q31. Following this, the fifth construct of self-efficacy was measured by nine robust items, namely Q32, Q33, Q34, Q35, Q36, Q38, Q39, Q40, and Q41, while the sixth construct of trust was measured by three robust items, namely Q42, Q43, and Q44. Finally, the last construct of teaching member presence was measured by four robust items, namely Q48, Q49, Q50, and Q51.

Table 12.

Summary of OSLEI (Part B) Evaluation With Rasch Model Analysis.

Construct	Item number		Infit MNSQ	Infit ZSTD	Point measure correlation
Perceived usefulness	Q5	I think that by joining Facebook learning groups, I will get information I need quickly.	0.96	0.0	.60
Perceived usefulness	Q7	I feel that by joining Facebook learning groups, my understanding/learning of that course will improve.	1.04	0.2	.66
Usage of learning group	Q8	I use Facebook groups to clarify study materials	1.07	0.4	.52
	Q9	I use Facebook groups to get guidance for (e.g., assignments, missed lecture content)	0.98	0.0	.61
	Q12	I use Facebook groups to retrieve admin information (e.g., due dates, notifications, general FAQ).	1.47	1.4	.37
	Q14	I usually ask questions to clarify (i.e., my answers, due dates) what I should do to complete course-related tasks/activities in Facebook groups.	1.08	0.4	.46
	Q17	I tend to post course-related content queries (i.e., topics covered, assignments, tests, or exams) in Facebook groups.	1.36	1.3	.46
	Q19	I tend to post how I feel toward (i.e., assignments, tests, topics) in Facebook groups	1.35	1.3	.64
	Q20	I tend to post or comment more frequently during period nearing to due dates (of i.e., assignments/tests/exams) in Facebook groups.	1.31	1.1	.59
Learning outcomes	Q21	Using Facebook learning group has improved my grades	0.91	−0.3	.70
	Q22	Using Facebook learning group increases my productivity	0.71	−1.0	.83
	Q23	I feel that less time is needed to comprehend study materials when using Facebook learning groups.	0.89	−0.3	.65
	Q25	I perceive that I have an increase knowledge of that course when using Facebook learning groups	0.77	−0.7	.69
	Q26	I am more able to critically analyze course materials when using Facebook learning groups.	0.72	−1.1	.66
	Q27	The use of Facebook learning groups improves my understanding of course materials.	1.06	0.3	.67
Affective outcomes	Q28	By participating in Facebook groups, I feel more satisfied with my study progress.	0.92	−0.2	.72
	Q29	By participating in Facebook groups, I feel more confident in interacting with strangers as a result of participating in Facebook learning groups.	1.34	1.1	.28
	Q30	By participating in Facebook groups, I feel more motivated to do better in my studies.	0.72	−0.9	.63
	Q31	By participating in Facebook groups, I feel less frustrated with my studies because questions asked in Facebook learning groups are being answered in a quick manner.	0.85	−0.4	.58
Self-efficacy	Q32	I feel confident to decide where and how to find the information I need in Facebook groups	0.65	−1.1	.70
	Q33	I feel confident to post questions in Facebook groups.	0.76	−0.7	.68
	Q34	I feel confident to comment on posts in Facebook groups.	1.28	0.9	.71
	Q35	I feel confident to interact with others even though I might not know them in the first place in Facebook groups.	1.07	0.3	.68
	Q36	I feel confident to follow and learn along with an online discussion created by others in Facebook groups.	0.78	−0.6	.62
	Q38	I feel competent to post questions in Facebook groups.	0.73	−0.9	.67
	Q39	I feel competent to comment on posts in Facebook groups.	1.25	0.9	.63
	Q40	I feel competent to interact with others even though I might not know them in the first place in Facebook groups.	1.21	0.7	.60
	Q41	I feel competent to follow and learn along with an online discussion created by others in Facebook groups	0.83	−0.5	.60
Trust	Q42	In Facebook groups, I trust that the information provided is correct.	0.71	−1.1	.75
	Q43	In Facebook groups, I trust that my course mates will help in answering my queries.	1.01	0.1	.69
	Q44	In Facebook groups, I trust that my course mates will be understanding if constructive criticisms are made in regards to their posts.	1.16	0.6	.45
Teaching member presence	Q48	I find lecturers/teaching members on Facebook learning groups encourages me from participating.	0.79	−0.6	.54
	Q49	I feel more confident that posts are correct when lecturers/teaching members are participating in/part of Facebook learning groups.	0.77	−0.7	.63
	Q50	I like having lecturers/teaching members participating/commenting in the Facebook learning groups.	1.16	0.6	.68
	Q51	Regular support from teaching members on Facebook learning groups is important to my success in learning.	1.19	0.7	.58

Note. MNSQ = mean square, 0.4 < x < 1.5; ZSTD = z-standard, −2 < x < +2; point measure correlation = positive value.

Discussion

The current study aimed to evaluate the quality of OSLEI items by using Rasch model analysis, whereby percentage agreement was conducted pre-analysis and no items were deleted. Rasch analysis is also capable of evaluating a person’s fit analysis. This is crucial since the majority of the studies related to educational technology have been ignored to validate the persons, who are important for the real measurement (Islam et al., 2020). Following the person fit assessment, 26 respondents were determined to be valid for further incorporation in evaluating the item fit of the OSLEI instrument, whereas 23 respondents were dropped for non-adherence to the criteria as per their values generated. Theoretically, a Rasch model analysis is capable of respondent analysis even for a small sample of 5 to 10 respondents (Linacre, 1994). Thus, the 23-person respondent pool identified for this study was deemed valid and reliable, with misfit respondents being detected and analyzed before proceeding to item analysis. Pallant and Tennant (2007) suggested the removal of such misfit respondents and items to prevent any influence on the resulting study findings. Moreover, Rasch model analysis is able to detect respondents that answered the OSLEI without reading the items properly.

In the analysis of item infit MSNQ and ZSTD, six items (i.e., Q47, Q46, Q45, Q10, Q15, and Q37) were dropped, as they were considered infit in the OSLEI. Following this, the Rasch model analysis proved their inability to measure the online social learning environment accurately. Moreover, the privacy construct was dropped altogether because all of its items were determined as misfits. Alternatively, the item point measure correlation revealed positive values for all retained items, indicating their appropriateness for measuring the environment. Meanwhile, the item measure values led to the identification of six redundant items, which measured the same thing in their respective constructs. In particular, the items consisted of one item (Q24) in learning outcome, four items (Q11, Q16, Q18, and Q13) in usage of learning group, and one item (Q4) in perceived usefulness.

Similarly, utilizing the standardized residual correlations also revealed redundant items in the instrument, resulting in one item being dropped (Q6). Regardless, the validity of the OSLEI was confirmed via its unidimensionality, whereby the value was increased after 13 items were dropped. Meanwhile, all 35 fit items in the instrument were proven by the raw variance explained by measures and unexplained variance in first contrast, which was 43.1% (increasing from 31%) and 4.4% (decreasing from 7.6%), respectively. These outcomes successfully met the minimum requirement of unidimensionality for Rasch model analysis as per Fisher’s (2007) suggestion, thereby proving the 7 constructs and their 35 items valid for measuring the online social learning environment.

Furthermore, all 35 retained items in the OSLEI were strongly supported by summary statistics following Rasch model analysis, whereby the values obtained for person separation, person reliability, item separation, and item reliability were 4.6, .95, 1.21, and .59, respectively. The outcomes for the first two values were very good and excellent, respectively, whereas the latter two values were deemed acceptable as per Fisher’s (2007) criteria. Besides, these values successfully met the minimum requirements set out by Fisher (2007) according to the rating scale for ascertaining instrument quality. Similarly, all 35 retained items in the OSLEI yielded strong support following the Rasch model analysis in view of a high Cronbach’s alpha value (.95). Here, Gandhi (2012) has underlined that a value of .9 to 1.0 is excellent for internal consistency.

Accordingly, the study findings imply that the OSLEI is a culturally valid instrument to assess social learning environments such as Facebook for the Malaysian population in the setting of educational field. Linacre (1994) has previously indicated that 5 to 10 responses per item are considered sufficient; however, the strength of this study revolves around the large number of respondents retained for further use in the item analysis for the OSLEI. However, Rasch model analysis does not require a large number of respondents. Besides, the instrument strength examined in this study was reinforced via a secondary content analysis carried out by three field experts following item drop in comparison with other instruments.

Even though only 35 items were retained in the OSLEI, the instrument was deemed valid and reliable for measuring an online social learning environment; in theory, the number of items is not important as long as the instrument can measure what it is supposed to measure. According to Harvey et al. (1985), four items are sufficient to measure a construct, while Cook et al. (1981) have recommended a minimum of three items in a construct to attain internal reliability. Hinkin (1998) suggests a minimum of four to six items to provide evidence of validity. In this study, the perceived usefulness construct only contained two items, but the instrument validity was 43.1% following Rasch model analysis. This value increased after the deletion of misfit items. Concurrently, the construct’s Cronbach’s alpha value was .95, suggesting a very high internal consistency reliability.

Thus, the current assessment of the OSLEI by using Rasch model analysis was highly valid and accurate to measure the online social learning environment in the educational setting, whereby the items assessing the environment were strongly reliable in the context of applied measurement in education. Following the analysis, all 35 items retained revealed more robust constructs for measuring said environment. In general, Rasch model analysis is known for its ability to provide empirical evidence of a powerful tool for the measurement context, such as a questionnaire (Salzberger, 2000). Additionally, the study findings offer a comprehensive reference for educational researchers seeking to employ this method with other instruments, such as for the evaluation of questionnaires in the context of social research.

Conclusion

Practically, this study contributes with revealed that 26 respondents and 35 OSLEI items were deemed valid and reliable following a comprehensive and powerful evaluation using Rasch model analysis. These 35 items undoubtedly contribute to the rigid item in OSLEI that can be used to measure online social learning environment. This instrument was modified from the original instrument, rendering it contributary to the body of literature in providing robust items to measure an online social learning environment, such as Facebook. Furthermore, this study contributed to the context of the research method thanks to its approach of evaluating questionnaires in the educational field by using an alternative method, namely Rasch model analysis. The critical analysis thus confirmed the robustness of items in the OSLEI for use in appropriate evaluation of questionnaire.

In particular, Rasch model analysis suggested dropping the privacy construct and only 35 items were retained, which were valid for measuring Facebook as an online social learning environment. In conclusion, measuring the platform as such an environment would render it meaningful to utilize 35 robust items of the OSLEI. Similarly, Rasch model analysis is particularly well known for its strength in establishing the reliability and validity of instruments for the purpose of practical assessment, research, and evaluation of questionnaire. Last but not least, others researcher are encouraged to employ Rasch model analysis as one of alternative method to evaluate the questionnaire through rigorous methodology. Previous study used CFA analysis to establish the validity and reliability of the questionnaire. When applying Rasch Model Analysis, this study contributes to determining if the questionnaire will be more accurate, rigid, and powerful.

Limitations and Future Research

This study is not without limitations, one of which is the selected sample. The respondents are chosen from only one university, thereby suggesting that a sample comprising people of different geographical backgrounds would generate an interesting assessment. Similarly, this study is limited by the students and their backgrounds, specifically undergraduate students enrolled in a multimedia applications and web design course. Thus, it is suggested that future research should employ a sample consisting of respondents from different subject areas or courses, allowing generalization of the results based on their educational backgrounds. Besides, this study focuses only on undergraduate students, but postgraduates could be selected instead in future works, as these students are also involved in online learning during COVID-19. Lastly, this study does not undertake an analysis of the rating scales used, which makes it possible for future works to include additional evaluation such as calibration structure by using Rasch model analysis. This step may ensure the accuracy of the rating scales used for the OSLEI.

Footnotes

Acknowledgements

The authors would like to thank the Universiti Pendidikan Sultan Idris and Universiti Teknologi Malaysia for their support in making this research possible.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Noor Hidayah Che Lah

References

Abbakumov

Desmet

Van den Noortgate

(2020). Rasch model extensions for enhanced formative assessments in MOOCs. Applied Measurement in Education, 33(2), 113–123.

Ahern

Feller

Nagle

(2016). Social media as a support for learning in universities: An empirical study of Facebook groups. Journal of Decision Systems, 25(Suppl 1), 35–49.

Ali

(2020). Online and remote learning in higher education institutes: A necessity in light of COVID-19 pandemic. Higher Education Studies, 10(3), 16–25.

Aziz

A. A.

Masodi

M. S.

Zaharim

(2013). Asas model pengukuran Rasch: Pembentukan skala dan struktur pengukuran. Penerbit Universiti Kebangsaan Malaysia.

Awang

(2015). SEM made simple: A gentle approach to learning Structural Equation Modeling. MPWS Rich Publication, Bangi.

Azlan

C. A.

Wong

J. H. D.

Tan

L. K.

Huri

M. S. N. A.

Ung

N. M.

Pallath

Lay Tan

C. P.

Yeong Chai

K. H.

(2020). Teaching and learning of postgraduate medical physics using Internet-based e-learning during the COVID-19 pandemic—A case study from Malaysia. Physica Medica, 80, 10–16.

Bode

R. K.

Wright

B. D.

(1999). Rasch measurement in higher education. In Higher education: Handbook of theory and research (pp. 287–316). Springer, Dordrecht.

Bode

R. K.

Wright

B. D.

(1999). Rasch measurement in higher education. In Smart

J. C.

Tierney

W. G.

(Eds.). Higher education: Handbook of theory and research (pp. 287–316). Springer. https://doi-org.ezproxy.utm.my/10.1007/978-94-011-3955-7_7

Bonanomi

Olivari

M. G.

Mascheroni

Gatti

Confalonieri

(2018). Using a multidimensional Rasch analysis to evaluate the psychometric properties of the Motivated Strategies for Learning Questionnaire (MSLQ) among high school students. Testing, Psychometrics, Methodology in Applied Psychology, 25, 83–100. http://doi.org/10.4473/TPM25.1.5

10.

Bond

T. G.

Fox

C. M.

(2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd edition). Lawrence Erlbaum.

11.

Camus

Hurt

N. E.

Larson

L. R.

Prevost

(2016). Facebook as an online teaching tool: Effects on student participation, learning, and overall course performance. College Teaching, 64(2), 84–94.

12.

Chawinga

W. D.

(2017). Taking social media to a university classroom: teaching and learning using Twitter and blogs. International Journal of Educational Technology in Higher Education, 14(1), 3.

13.

Che Lah

N. H.

Tasir

Jumaat

N. F

. (2021). Applying alternative method to evaluate online problem-solving skill inventory (OPSI) using Rasch model analysis. Educational Studies, 1–23. https://doi.org/10.1080/03055698.2021.1874310

14.

Chung

Subramaniam

Dass

L. C.

(2020). Online learning readiness among university students in Malaysia amidst COVID-19. Asian Journal of University Education, 16(2), 46–58.

15.

Clinton

Alayan

El-Alti

(2014). Rasch analysis of Lebanese nurses’ responses to the EIS questionnaire. SAGE Open, 4(3), 2158244014547182.

16.

Coe

(2008). Comparability of GCSE examinations in different subjects: An application of the Rasch model. Oxford Review of Education, 34, 609–636

17.

Cook

J. D.

Hepworth

S. J.

Wall

T. D.

Warr

P. B.

(1981). The experience of work. Academic Press

18.

Dalsgaard

(2016). Students’ educational use of Facebook groups. Educational Media International, 53(4), 261–273.

19.

Daniel

(2020). Education and the COVID-19 pandemic. Prospects, 49(1), 91–96.

20.

DeMars

C. E

. (2021). Violation of Conditional Independence in the Many-Facets Rasch Model. Applied Measurement in Education, 34(2), 122–138.

21.

Docimo

Jacob

Seras

Ghanem

(2020). Closed Facebook groups and COVID-19: An evaluation of utilization prior to and during the pandemic. Surgical Endoscopy, 14, 1–5.

22.

Eddy

L. H.

Preston

Mon-Williams

Bingham

D. D.

Atkinson

J. M.

Ellingham-Khan

Hill

L. J.

(2021). Developing and validating a school-based screening tool of Fundamental Movement Skills (FUNMOVES) using Rasch analysis. PloS One, 16(4), e0250002.

23.

Embretson

S. E.

Hershberger

(1999). The new rules of measurement: What every psychologist and educator should know. Lawrence Erlbaum.

24.

Etikan

Musa

S. A.

Alkassim

R. S.

(2016). Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics, 5(1), 1–4.

25.

Fisher

W. P. J.

(2007). Rating scale instrument quality criteria. Rasch Measurement Transaction 21(1), 1095.

26.

Ghandi

P. S.

(2012). Calculating and interpreting reliability estimates for achievement test in graph theory (a modern branch of mathematics). International Indexed & Referred Research Journal, 5, 36–38.

27.

Giannikas

(2020). Facebook in tertiary education: The impact of social media in e-learning. Journal of University Teaching and Learning Practice, 17(1), 3.

28.

Harvey

R. J.

Billings

R. S.

Nilan

K. J.

(1985). Confirmatory factor analysis of the Job Diagnostic Survey: Good news and bad news. Journal of Applied Psychology, 70(3), 461–468. https://doi.org/10.1037/0021-9010.70.3.461

29.

Hinkin

T. R.

(1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1(1), 104–121. https://doi.org/10.1177/109442819800100106

30.

Hong

Gardner

(2014). Facebook groups: Perception and usage among undergraduates in the context of learning. Completed Research Paper. Thirty Fifth International Conference on Information Systems, Auckland.

31.

Islam

A. Y. M. A.

Crook

Spector

J. M.

(2020). Assessment of ICT in tertiary education applying structural equation modeling and Rasch model. SAGE Open, 10(4). https://doi.org/10.1177/2158244020975409

32.

Jumaat

N. F.

Ahmad

Abu Samah

Ashari

Z. M.

Ali

D. F.

Abdullah

A. H.

(2019). Facebook as a platform of social interactions for meaningful learning. International Journal of Emerging Technologies in Learning, 14(4), 151–159.

33.

Kliem

Beller

Kröger

Stöbel-Richter

Hahlweg

Brähler

(2015). A Rasch re-analysis of the Partnership Questionnaire. Sage Open, 5(2), 2158244015588958.

34.

Lah

N. H. C.

Tasir

(2018). Measuring reliability and validity of questionnaire on online social presence: A Rasch model analysis. Advanced Science Letters, 24(11), 7900–7903.

35.

Lapitan

L. D.

Jr Tiangco

C. E.

Sumalinog

D. A. G.

Sabarillo

N. S.

Diaz

J. M.

(2021). An effective blended online teaching and learning strategy during the COVID-19 pandemic. Education for Chemical Engineers, 35, 116–131.

36.

Lee

Pham

Crosby

Peterson

J. F.

(2021). Digital collaboration in design education: how online collaborative software changes the practices and places of learning. Pedagogy, Culture & Society, 29(2), 231–245.

37.

Linacre

J. M.

(1994). Sample size and item calibration stability. Rasch Measurement Transactions, 7(4), 328.

38.

Linacre

J. M.

(2002). What do infit, outfit, mean-square and standardise mean? Rasch Measurement Transactions, 16, 878.

39.

Mbatha

(2020). Crossing the bridge: transitioning to Facebook as a short-term response amidst the COVID-19 pandemic. In Alternation African Scholarship Book Series (AASBS) (p. 120). CSSALL Publishers.

40.

Nordin

A. B.

Bakar

B. A.

(2008). Penaksiran dalam pendidikan dan sains sosial. Universiti Pendidikan Sultan Idris.

41.

Noyens

Donche

Coertjens

Van Daal

Van Petegem

(2019). The directional links between students’ academic motivation and social integration during the first year of higher education. European Journal of Psychology of Education, 34(1), 67–86.

42.

O’Neill

T. R.

Gregg

J. L.

Peabody

M. R.

(2020). Effect of sample size on common item equating using the dichotomous Rasch Model. Applied Measurement in Education, 33(1), 10–23. https://doi.org/10.1080/08957347.2019.1674309

43.

Pallant

J. F.

Tennant

(2007). An introduction to the Rasch measurement model: an example using the Hospital Anxiety and Depression Scale (HADS). British Journal of Clinical Psychology, 46(1), 1–18.

44.

Park

E. Y.

Seo

Blair

K. S. C.

Kang

M. C.

(2021). Rasch analysis of the Korean-Child Behavior Checklist (K-CBCL) to assess emotional and behavioral problems in children with autism spectrum disorder. SAGE Open, 11(3), 21582440211040790.

45.

Pomeranz

J. L.

Byers

K. L.

Moorhouse

M. D.

Velozo

C. A.

Spitznagel

R. J.

(2008). Rasch analysis as a technique to examine the psychometric properties of a career ability placement survey subtest. Rehabilitation Counseling Bulletin, 51(4), 251–259.

46.

Rasch

(1966). An item analysis which takes individual differences into account. British Journal of Mathematical and Statistical Psychology, 19(1), 49–57.

47.

Rahayu

Putra

M. D. K.

Rahmawati

Hayat

Koul

R. B.

(2021). Validating an Indonesian version of the What Is Happening In this Class? (WIHIC) questionnaire using a multidimensional Rasch model. International Journal of Instruction, 14(2), 919–934.

48.

Rahayu

Putra

M. D. K.

Sulaeman

Koul

R. B.

(2021). Development and validation of Online Classroom Learning Environment Inventory (OCLEI): The case of Indonesia during the COVID-19 pandemic. Learning Environments Research, 25(1), 97–113.

49.

Randall

Engelhard

Jr. (2010). Using confirmatory factor analysis and the Rasch model to assess measurement invariance in a high stakes reading assessment. Applied Measurement in Education, 23(3), 286–306.

50.

Raspopovic

Cvetanovic

Medan

Ljubojevic

(2017). The effects of integrating social learning environment with online learning. International Review of Research in Open and Distributed Learning, 18(1), 142–160.

51.

Salzberger

(2000). An alternative way of establishing measurement in marketing research-its implications for scale development and validity. ANZMAC 2000 visionary marketing for the 21st century: Facing the challenge (pp. 1111–1117). Griffith University, School of Marketing and Management.

52.

Shea

Cooper

B. K.

De Cieri

Sheehan

(2012). Evaluation of a perceived organisational performance scale using Rasch model analysis. Australian Journal of Management, 37(3), 507–522. https://doi.org/10.1177/0312896212443921

53.

Steele

A. R.

Day

D. V.

(2020). Development and preliminary validation of the interest in leadership scale. SAGE Open, 10(1), 2158244019900182.

54.

Tennant

Conaghan

P. G.

(2007). The Rasch measurement model in rheumatology: What is it and why use it? When should it be applied, and what should one look for in a Rasch paper? Arthritis Care & Research, 57(8), 1358–1362.

55.

Toland

M. D.

Grisham

Waddell

Crawford

Dueber

D. M.

(2021). Scale evaluation and eligibility determination of a field-test version of the assessment, evaluation, and programming system third edition. Topics in Early Childhood Special Education. Advance online publication. https://doi.org/10.1177/0271121420981712

56.

Vogel

S. P.

Engelhard

Jr. (2011). Using Rasch measurement theory to examine two instructional approaches for teaching and learning of French grammar. The Journal of Educational Research, 104(4), 267282. https://doi.org/10.1080/00220671003733815

57.

Wilmot

B. D.

Schoenfeld

Wilson

Champney

Zahner

(2011). Validating a learning progression in mathematical functions for college readiness. Mathematical Thinking and Learning, 13, 259–291.

58.

Wright

B. D.

Linacre

Gustafsson

J. E.

Martin-Loff

(1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370.

59.

C.-H.

Chen

S.-C.

(2015). Understanding the relationships of critical factors to Facebook educational usage intention. Internet Research, 25(2), 262–278.

60.

Yan

(2018a). The self-assessment practice scale (SaPS) for students: Development and psychometric studies. The Asia-Pacific Education Researcher, 27(2), 123–135.

61.

Yan

(2018b). How teachers’ beliefs and demographic variables impact on self-regulated learning instruction. Educational Studies, 44(5), 564–577.

62.

Yang

Liu

(2018). Validation of an instrument for measuring students’ understanding of interdisciplinary science in grades 4–8 over multiple semesters: A Rasch measurement study. International Journal of Science and Mathematics Education, 16, 639–654. http://doi.org/10.1007/s10763-017-98057

Item number	Point measure correlation	Total item dropped
Q19	.65	0
Q42	.75
Q8	.54
Q16	.49
Q26	.65
Q21	.72
Q23	.68
Q24	.68
Q28	.74
Q9	.64
Q11	.60
Q16	.53
Q18	.48
Q44	.40
Q13	.29
Q20	.59
Q22	.85
Q31	.58
Q38	.68
Q48	.53
Q30	.64
Q43	.69
Q33	.70
Q35	.63
Q40	.57
Q12	.40
Q41	.55
Q5	.63
Q32	.69
Q34	.70
Q27	.66
Q29	.24
Q36	.59
Q50	.68
Q6	.70
Q14	.45
Q25	.72
Q49	.64
Q4	.64
Q7	.64
Q51	.58

Item number	Point measure correlation	Total item dropped
Q19	.65	0
Q42	.75
Q8	.54
Q16	.49
Q26	.65
Q21	.72
Q23	.68
Q24	.68
Q28	.74
Q9	.64
Q11	.60
Q16	.53
Q18	.48
Q44	.40
Q13	.29
Q20	.59
Q22	.85
Q31	.58
Q38	.68
Q48	.53
Q30	.64
Q43	.69
Q33	.70
Q35	.63
Q40	.57
Q12	.40
Q41	.55
Q5	.63
Q32	.69
Q34	.70
Q27	.66
Q29	.24
Q36	.59
Q50	.68
Q6	.70
Q14	.45
Q25	.72
Q49	.64
Q4	.64
Q7	.64
Q51	.58

An Evaluation of the Online Social Learning Environment Instrument (OSLEI) Using Rasch Model Analysis

Abstract

Keywords

Introduction

Literature Review

Online Social Learning Environment Instrument (OSLEI)

Rasch Model Analysis

Method

Instrument

Software

Procedure of Analysis

Measurement Properties of Rasch Model Analysis

Participants

Results

Percent Agreement Validity of the OSLEI

Person infit MNSQ and ZSTD of the OSLEI

Person Point Measure Correlation of the OSLEI

Item Infit MNSQ and ZSTD in the OSLEI

Item Point Measure Correlation of the OSLEI

Item Measure Value in the OSLEI

Standardized Residual Correlations of the OSLEI

Unidimensionality of the OSLEI

Summary Statistics of the OSLEI

Robust Items in the OSLEI

Discussion

Conclusion

Limitations and Future Research

Footnotes

Acknowledgements

Declaration of Conflicting Interests

Funding

ORCID iD

References

Item number	Point measure correlation	Total item dropped
Q19	.65	0
Q42	.75
Q8	.54
Q16	.49
Q26	.65
Q21	.72
Q23	.68
Q24	.68
Q28	.74
Q9	.64
Q11	.60
Q16	.53
Q18	.48
Q44	.40
Q13	.29
Q20	.59
Q22	.85
Q31	.58
Q38	.68
Q48	.53
Q30	.64
Q43	.69
Q33	.70
Q35	.63
Q40	.57
Q12	.40
Q41	.55
Q5	.63
Q32	.69
Q34	.70
Q27	.66
Q29	.24
Q36	.59
Q50	.68
Q6	.70
Q14	.45
Q25	.72
Q49	.64
Q4	.64
Q7	.64
Q51	.58