Sage Journals: Discover world-class research

Abstract

Inconsistency in the use of Virtual Learning Environments (VLEs) has led to dissatisfaction amongst students and is an issue across the Higher Education sector. This paper outlines research undertaken in one faculty within one university to ascertain staff and student views on minimum standards within the VLE; how the VLE could reduce student dissatisfaction; and to propose a conceptual framework surrounding student satisfaction with the VLE.

A questionnaire was sent to staff and students asking if they agreed with the need to introduce minimum standards in the VLE and what criteria they wanted. The National Student Survey (NSS) results were analysed for six schools within the faculty over a 4-year period. Many of the NSS results were relevant to developing minimum standards with the VLE.

The questionnaire results showed the vast majority of staff and students favour the introduction of minimum standards and identified specific items that should be included, for example handbooks, contact information for staff, access to previous modules, assessment information, further reading, etc. The NSS data showed that students wanted lectures available in the VLE, improved feedback, more computers for students and information about cancelled sessions/timetable changes in the VLE.

The results suggest the presence of many minimum standards may reduce student dissatisfaction with the VLE. However, a distinction is made between those pre-potent factors that cause dissatisfaction and those that lead to satisfaction, using Herzberg’s Two-Factor Theory as a theoretical basis.

When considering minimum standards as ‘hygiene factors’, their presence can prevent student dissatisfaction and provide the foundations for innovation in technology-enhanced learning.

Keywords

Minimum standards VLEs baseline NSS hygiene factors satisfaction

Introduction

The need to remain competitive in the Higher Education (HE) marketplace means universities no longer have a choice regarding implementing e-learning strategies (Kregor et al., 2012). This rings true at a local level, evidenced by the recent ‘Making the Most of IT’ campaign from the Student Union at the University of Liverpool (Bee, 2013). Their campaign, conducted during the autumn term of the 2012/2013 academic year, was developed to ascertain the ‘student voice’ on the issue of technology-enhanced learning (TEL), and involved face-to-face discussion with students followed by a series of focus groups. Their subsequent report suggested students expect the effective use of technology to support and enhance their studies, and raised concern at the lack of consistency between modules in the Blackboard Virtual Learning Environment (VLE) – an online platform used to support, supplement and enhance teaching and learning across the university. As a result, their recommendations call for policies requiring all modules to have a VLE presence including module specifications, lecture notes, reading lists and past exam papers. It further recommends the university should increase the take up of recorded lectures. The student voice, in this instance, is the key driver to research the issue of consistency in the VLE further, and furthermore implement appropriate strategy and policy at the institution under study.

Despite the range of innovation across educational technology, universities’ implementation of a consistent approach to the use of the VLE appears to vary a great deal. Such lack of consistency, where some module spaces are empty whilst others overflow with administrative information, lecture content and support materials, is an ongoing topic of discussion amongst educational technology user groups and online mailing lists. This suggests the topic of consistency in the VLE is a topic of current interest across the educational technology landscape. Universities across the UK are considering how they can introduce templates, ‘baseline requirements’ and minimum standards for the use of the VLE (Ahmed and Morley, 2010; Reed, 2014); however, this topic seldom appears in formal publications such as books and journal articles.

The Blackboard Catalyst – Exemplary Course Programme (http://www.blackboard.com/community/catalyst-awards/exemplary-course-program.aspx) is one international initiative aiming to support universities to implement good practice in the VLE. This is through the use of a course rubric focusing on four major aspects: Course design; Interaction and Collaboration; Assessment; and Learner Support. There are correlations here to quality assurance in TEL; however, this is not something universities appear to be signing up to en masse. Rather, Blackboard are engaging with learning technologists and select academic staff ‘champions’. Thus it could be considered a ‘bottom-up’ approach to raising standards as opposed to ‘top-down’ policy and strategy within institutions, and takes a broader view of good practice beyond the sole focus of minimum standards.

Whilst such bottom-up approaches are inspirational, Ellis and Calvo (2007) emphasise the role of leadership (e.g. a ‘sponsor’ of sufficiently high level within the institution) and policy in successfully implementing ‘minimum indicators’ to assure quality in the use of VLEs. They go on to suggest that in the absence of policy and planning, ‘one cannot realistically expect enduring and influential growth and change’ (Ellis and Calvo, 2007).

Despite the ease of many TEL implementations as well as the driving force of the student voice, academic staff do not always respond with action. Kregor et al. (2012) posit reasons for a potential difference in attitudes between staff and students regarding the use of technology in learning and teaching in one university in the United States, suggesting:

We therefore attribute the difference to respective roles where time saving and flexibility gains for students may inversely require additional workload or skill demands for some staff. It is self-evident that the two groups have very different relationships with the technology as a function of role. (Kregor et al., 2012: 1397)

However, many of the suggestions for inclusion in VLE minimum standards are relatively basic, for example contact details for module leaders and timely information about summative assessment requirements, etc. This in contrast to more sophisticated uses of technology, such as the use of social media in learning and teaching (Reed, 2013). Therefore the scenario whereby universities are struggling with consistent adoption of the VLE whilst simultaneously implementing innovative approaches (Hill et al., 2012) could appear illogical – more so as basic implementations require relatively little time and technical expertise. The differences attributed by Kregor et al., therefore, could relate to more advanced uses of technology, or perhaps identify significance of competing time pressures on academic staff.

In addition, the National Student Survey (NSS) has gained increasing importance and become a key focal point of student satisfaction across the United Kingdom (Anderson, 2012; Cocksedge and Taylor, 2013; NSS, 2013). The NSS is given to students in all publicly funded higher education institutions (HEIs) in England, Scotland, Wales and Northern Ireland (between January and April of the year of graduation) and is part of a global trend for evaluating students’ experiences of their higher education. Similar surveys take place in the United States (The National Survey of Student Engagement [NSSE]) and New Zealand and Australia (Australasian Survey of Student Engagement [AUSSE]), and the Republic of Ireland has recently introduced the Irish Survey of Student Engagement. These surveys can also be seen as successors to the European Union (EU) declarations and communiqués such as Bologna 1999, Prague 2001, and Berlin 2003, which have placed emphasis on evaluating the quality of outcomes in HE.

The emphasis here is that the UK NSS results are used to guide prospective students in their decision-making and are widely published. The questionnaire contains 23 questions under the following six themes:

Teaching on the course;

Assessment and feedback;

Academic support;

Organisation and management;

Learning resources;

Personal development; and,

Overall satisfaction with their course and institution (NSS, 2013).

The quantitative and qualitative results from the NSS are collated by Ipsos MORI (a UK-based market research company) and sent to HEIs later in the year to distribute to relevant departments/schools. Whilst the NSS does not ask directly about the VLE, some of the questions do ask about issues which impinge on the VLE, such as question 17: ‘I have been able to access general IT resources when I needed to’. The theme ‘Assessment and feedback’ has obvious correlations to the increased consistency and standardisation from the introduction of minimum standards. Respondents also have opportunity to leave qualitative free text responses at the end of the questionnaire, and are invited to write generally about positive or negative parts of their institution/course. Although they may not be questioned specifically about the VLE at their institution, many comments are related to this area.

This paper illustrates the views of staff and students from six schools based in the Faculty of Health and Life Sciences at one Russell Group university in the United Kingdom, considering the implications of the NSS for the VLE and the introduction of minimum standards. Finally, the paper also offers a theoretical/conceptual framework within which to consider the baseline use of, or minimum standards for, the VLE.

Minimum standards – why, what, where?

As suggested above, one of the key drivers for encouraging minimum standards is to ensure a certain degree of consistency between module areas in the VLE. There is very little published work in the area of minimum standards or baseline agreements for the use of the VLE, which makes any literature review difficult. Indeed, both authors of the present study conducted independent literature searches using various tools including using the University’s ‘Discover’ service to search databases and journal collections, as well as separate searches using Google Scholar. The key terms related to the topic were used, for example ‘VLE Minimum Standards’, ‘VLE Baseline’, ‘VLE consistency’, etc. The limited relevant literature returned was used to identify further published works via the citations made, and all are incorporated within this paper. The lack of published literature is in itself interesting, given the topic is a popular issue amongst user groups and mailing lists in the UK. However, a report by the University’s Student Union (Bee, 2013) suggests the variation in experience across VLE areas is a ‘significant factor for their [student’s] dissatisfaction’ (Bee, 2013: 6), a notion common across the UK HE sector (Ahmed and Morley, 2010). Furthermore, the success of implementations such as the VLE is often a direct result of managing and realising learner expectations (Jackson and Fearon, 2014; Wu et al., 2010), and there is no doubt that students today expect to use the VLE to support their studies (Bee, 2013). Coverdale-Jones (2012) also suggests that a ‘push’ strategy (i.e. top-down policy) of minimum standards was a key factor in ‘the cultural change and successful development of e-learning as a normal activity for academics’ (Coverdale-Jones, 2012: 123), which strengthens Ellis and Calvo’s (2007) call for strong leadership and policy in this area. Collectively, this provides suitable justification for universities to take action with regards to the introduction of minimum standards; however, as demonstrated in this study, any such action should be evidence based.

Active discussions within communities of practice, for example The Association for Learning Technology Closed Mailing List, reinforce Ahmed and Morley’s (2010) suggestion that such inconsistent student experience is a more common problem across the HE sector. Jackson and Fearon (2014) report on various studies on VLE satisfaction, and suggest:

When a mismatch occurs between initial expectations and the actual (benefits success) outcome of technology change, it can give rise to a situation of expectations dissatisfaction, more commonly referred to as a “disconfirmation of expectation”. (Jackson and Fearon, 2014)

In an era combining increased student fees and widespread usage of technology on a daily basis, students have already begun to voice their dissatisfaction around VLE inconsistency (Ahmed and Morley, 2010; Bee, 2013; Coverdale-Jones, 2012; Hill et al., 2012), with increasing evidence to suggest ‘e-learning is rarely seen as separate or special by learners’ (Hall, 2009: 32).

It is recognised that many of the suggestions for inclusion in minimum standards focus on administrative tasks and supportive information, rather than factors that necessarily enhance learning and teaching. Although some believe(d) using the VLE as a place to put lecture notes was evidence that innovation had ‘slowed considerably’ (Stiles, 2007), students are now increasingly relying on these basic uses and services to fundamentally support their studies.

The introduction of minimum standards is not considered to be innovative in itself, but could, however, provide solid foundations for innovation to flourish. Ahmed and Morley (2010) suggest the basic uses of VLEs prevented low scores in module evaluations (i.e. prevented dissatisfaction) at the University of Huddersfield (UK); however, those modules that incorporated blended learning, for example integrated elements of online learning within the face-to-face courses, raised evaluations to greater levels (i.e. indicating greater satisfaction). This potentially suggests high levels of satisfaction could be related to a greater and more interactive use of the VLE, whilst minimum standards could prevent dissatisfaction.

Sun et al. (2008) report seven variables to have critical relationships to e-learner’s satisfaction (with e-learning in this case referring to completely online learning) across two public universities in Taiwan; however, the authors did not explicitly identify factors that cause dissatisfaction. Two of these variables include learners’ ‘perceived usefulness’ and ‘ease of use’ of the system (Sun et al., 2008). It is anticipated that these factors will prove critical in the present study as factors that cause/prevent dissatisfaction rather than satisfaction.

Without questioning the validity of previous studies exploring the use of technology alongside student satisfaction, further clarification is required to identify factors that truly contribute towards student satisfaction and student dissatisfaction (as separate entities). This might reinforce the idea that many of the criteria within minimum standards often serve as ‘hygiene’ (or dissatisfying) factors, and such standards, whilst ‘raising the floor’ for technology use, also provide foundations for real innovation to enhance learning and teaching.

A model for student (dis)satisfaction with the VLE – Herzberg’s Two-Factor (motivation-hygiene) Theory

In the mid 1960s, management theorist Frederick Herzberg was interested in the correlation between employee attitude and workplace motivation. From his research he proposed his Two-Factor Theory (or motivation-hygiene theory), suggesting:

Job dissatisfaction was influenced by hygiene factors, and

Motivation was influenced by motivating factors (or satisfiers).

Hygiene Factors describe factors that cause dissatisfaction in the workplace. They are usually extrinsic or independent of the work itself, such as politics, quality of leadership, relationships, job security, compensation, etc. They do not motivate employees, but when missing/lacking they can cause employee dissatisfaction. They focus on making employees feel secure and comfortable.

Motivators/satisfiers are related to motivating employees, and consider intrinsic factors of the job, such as responsibility, job satisfaction, recognition, achievement, opportunity for growth and advancement.

Herzberg proposes that the opposite of ‘satisfaction’ is not dissatisfaction – it is simply, ‘no satisfaction’. Similarly, the opposite of ‘dissatisfaction’ is ‘no dissatisfaction’. Herzberg suggests the key satisfiers relate to the job itself, whereas the hygiene factors focus on the ‘characteristics of the context in which the job is done’ and seldom bring about job satisfaction (Herzberg et al., 1967: 63).

Therefore, the things that make us unhappy at work are unlikely to be the same things that make us overjoyed if they were to be reversed.

Applying Two-Factor Theory to education

Youn et al. (2005) used Herzberg’s motivation-hygiene theory as a conceptual framework for their qualitative study into satisfying and dissatisfying factors in online learning (Youn et al., 2005). Two questions led the focus of the research:

What factors of an e-learning system do e-learners express as satisfying factors (i.e. motivation factors) and what factors do they express as dissatisfying factors (i.e. hygiene factors)? (Youn et al., 2005: 103)

In the context of their study (US-based online education), the authors propose four strong satisfiers and two strong dissatisfiers (Table 1).

Table 1.

Satisfiers and dissatisfiers from Youn et al. (2005).

Satisfiers/motivators	Dissatisfiers/hygiene factors
Learning content itself	Instructional directions/expectations
Instructor’s teaching methods/styles	Instructors participation level
Instructor’s subject matter expertise
Types of learning activities

Again, this reinforces Herzberg’s suggestion that hygiene factors/dissatisfiers impact on the characteristics of the context.

The authors go on to suggest the importance of addressing strong dissatisfying factors before implementing motivation factors, since: ‘an e-learning environment with serious dissatisfying factors will likely cause dissatisfaction despite positive effects of the satisfying factors’ (Youn et al., 2005: 110). Chyung’s SIEME model is referred to as a systematic approach to dealing with motivational and hygiene factors in the context of attrition in distance learning, but may be repurposed in the context of the present study:

Separate unhealthy attrition and healthy attrition;

Investigate hygiene and motivation factors;

Eliminate or reduce the hygiene factors;

Maintain and/or add motivation factors; and

Evaluation attrition in DE [distance education] continuously.

(Chyung, 2005)

It might also be possible to relate these basic uses of technology to Maslow’s Hierarchy of Needs (Maslow, 1943) (Figure 1), which identifies five basic needs (physiological, safety, belonging, self-esteem, self-actualisation) to underpin research on motivation, whereby ‘human needs arrange themselves in hierarchies of pre-potency. That is to say, the appearance of one need usually rests on the prior satisfaction of another, more pre-potent need. Man is a perpetually wanting animal’ (Maslow, 1943: 370).

Figure 1.

Maslow’s Hierarchy of Needs (1943).

This ties in with the notion that satisfaction cannot be achieved whilst dissatisfying factors are present. The obvious challenge in applying the work of Youn et al. (2005) and Chyung (2005) is the specific context in which they were introduced, that is, online/distance education in the United States with a particular focus on attrition. In the UK, with the exception of the Open University, few universities place such a great emphasis on online education, with a preference for face-to-face and blended modes of delivery. The questions raised here therefore relate to the identification of hygiene factors and motivators within VLE minimum standards; their relationship to Maslow’s Hierarchy of Needs; and the transferability of the SIEME model to the implementation of VLE minimum standards.

Methodology

To examine interest in this area, a pre-research activity sought to ‘crowdsource’ a broad picture of minimum standards adoption across HEIs by requesting volunteers to populate information about their own institution. This was followed by an online questionnaire to investigate staff and student perceptions of the introduction of minimum standards within one faculty in the university under study. Finally, the data were triangulated with qualitative and quantitative data received from the NSS for this faculty within the institution. It was felt this process would be useful by identifying existing practice across the sector, and relate student feedback from the NSS to specific questionnaire data related to minimum standards.

Pre-research activity

An email was sent to members of the ‘Association for Learning Technology Closed Mailing List’ inviting volunteers to collaboratively edit a shared document, providing details of institutions either looking to introduce, or having already introduced, VLE minimum standards. The initial email emphasised that the data would be publicly available via various dissemination paths, for example blog posts, conference presentations and publications.

Details were received relating to 23 HEIs; 75% (n = 18) already had a series of minimum standards/baseline agreements, with a further 25% (n = 6) looking to introduce such standards in the near future (Reed, 2014). A number of those with standards in place also included scope for further development and progression in an attempt to encourage greater use of learning technology (e.g. Exeter), whilst some others had automated at least some standards through tight system integrations (e.g. Newcastle). The Blackboard Catalyst – Exemplary Course Programme has also provided rubrics for good practice in some other HEIs (e.g. Aberystwyth). Some 24 different criteria were identified for inclusion, with the most common items being:

Dedicated VLE areas (100%, n = 17);

Staff Profiles/Contact Details (88%, n = 15);

Further/Recommended Reading (82%, n = 14);

Module Description/Outline inc. learning outcomes, etc (82%, n = 14);

Assessment Requirements (82%, n = 14);

Lecture handouts/slides (70%, n = 12).

The data obtained are of interest, offering the potential to inform future policy and cross-check against the views of staff and students.

Staff and student questionnaires

With the broad aim of investigating VLE minimum standards in various schools in one Russell Group University in the UK, a number of questions guided the research:

Do staff agree with the need to introduce VLE minimum standards and what do they think should be included?

What specific criteria do students want in VLE minimum standards?

Can the introduction of minimum standards impact on the NSS?

The research was conducted within the Institute of Learning and Teaching (ILT), Faculty of Health and Life Sciences. The ILT comprises the Schools of Dentistry, Health Sciences, Life Sciences, Medicine, Psychology and Veterinary Science. Staff and students across these six schools completed an online questionnaire related to the introduction of minimum standards in the VLE. Links to the online questionnaires were sent to staff (August 2013) and students (October 2013) via email and VLE announcements.

Emails and announcements contained an opening invitation for participation in this research. It was made explicit that participation was both completely voluntary and responses were anonymous, in order to align with the British Educational Research Association’s (BERA) Guidelines for Educational Research (BERA, 2011). Data capture via the questionnaires was done using survey software hosted internally at the University. Ethical approval was awarded by the ILT Ethics Review Group (Ref 201404155). Furthermore, the NSS is a voluntary national survey administrated independently which does not require ethical approval for further analysis within specific institutions. No personal data or identifiers were included in the questionnaires or NSS study.

In both questionnaires, respondents were asked to identify specific elements that should be included in any such series of minimum standards (presented in a list as follows):

A dedicated VLE area for each module

‘Welcome’ to the VLE area

Contact details (Module Leader)

Contact details (Other teaching staff)

Module specification

Module timetable/schedule

Module learning outcomes

Module assessment strategy/requirements

Recommended/further readings

Lecture notes/handouts

Past exam papers (where appropriate)

Ability to submit coursework online

Opportunity for draft assignment (formative) feedback

Online discussion forums

Other (please specify)

As this could be a long list, the use of ‘Other’ enabled respondents to detail items that were missing.

There were some differences in the staff and student questionnaires:

The staff questionnaire contained an initial question: ‘Do you agree with the introduction of minimum standards to help achieve greater consistency between VLE areas?’

The student questionnaire asked respondents to identify their gender (Table 3); however, the staff questionnaire did not. The staff questionnaire was conducted before the student questionnaire, and this was one addition that was felt could provide further insight.

Statistical tests

A Pearson product-moment correlation coefficient test was computed to assess the relationship between data obtained from staff responses and student responses to the items within minimum standards.

Furthermore, a number of two-sample t-tests were used to determine statistical significance between data obtained, for example:

Staff responses and student responses to the items within minimum standards;

The number of selections made for inclusion in minimum standards between staff and students;

The number of selections made for inclusion in minimum standards between female and male students.

These particular statistical tests were chosen as they might prove/disprove a relationship between staff and student responses and male and female responses in relation to their views on what should be included in minimum standards.

NSS analysis

As part of a wider project to look at trends in the NSS, the quantitative (from 2010 to 2013 inclusive) and qualitative data (from 2011 to 2013 inclusive) relating to the six schools were collected and analysed.

Quantitative questions use 5-point Likert scale responses: definitely agree (5); mostly agree (4); neither agree nor disagree (3); mostly disagree (2); definitely disagree (1); and ‘not applicable’. Qualitative free text responses at the end of the questionnaire enable more general feedback about positive/negative parts of a course. From 2007 onwards results for the NSS are published as a percentage response to each individual question by subject. For example:

Subject	Question Number	Agree %
X	1. Staff are good at explaining things	75%

The percentage figure indicates the proportion of students who gave the question a response of either 4 or 5, that is students chose the ‘mostly agree’ or ‘definitely agree’ response to that question.

Detailed thematic analysis (Boyatzis, 1998) has been undertaken on all students’ qualitative comments on the last 3 years (2011–2013) of the NSS results. Comments were categorised by school and cohort, and were given a ‘theme’, enabling quick identification of similar comments. Progression through student responses either reinforced existing themes or generated new themes. Only key points related to the use of the VLE are included in this paper.

Results

Online questionnaire data

In total, 865 responses were received to the online staff (n = 101) and student (n = 764) questionnaires (Table 2). Table 3 demonstrates a breakdown of student responses by level of study and gender. Due to the methods of distribution an accurate response rate is difficult to ascertain; however, there are circa 4500 students across ILT, representing roughly a 17% response rate.

Table 2.

No. of responses to online questionnaires.

Questionnaire	Responses (n)
Staff	101
Student	764
Total	865

Table 3.

Breakdown of student respondents (n).

Level of study	Male	Female	Gender unknown	Total
Undergraduate	169	529	2	700
Postgraduate	15	49	0	64
Total	184	578	2	764

Of staff respondents, 90% (n = 91) agree with the introduction of minimum standards. Of the 10% who did not agree, zero (n = 0) provided further detail in the follow-up question exploring their reasoning.

Chart 1 demonstrates student (blue) and staff (red) responses to the selection of criteria to be included in minimum standards.

Chart 1.

What should be included in minimum standards.

A very small number of respondents to the staff (3%, n = 3) and student (3.4%, n = 24) surveys selected ‘Other’ from the list of criteria for inclusion. Staff responses included: Lectures as audio/video podcast; module handbook where appropriate; and mitigating circumstances. Three other comments included: Support for staff is currently poor; and Module spec contains learning outcomes and assessment strategy. One final comment of interest included:

Depends who is available to create data or if it can be automatically added from university databases and if anyone reviews content to ensure all is there. If it has to be done by academics, the minimum set must be small. If there are FM [sic] staff to do it, the set can be larger. Module spec will give much of the other content listed. (Anonymous respondent)

The most common ‘other’ response from students related to provision of recorded lectures (n = 8) and formative quizzes (n = 7). Other responses included: A search area; access to previous modules; exam info; integration with cloud services like dropbox; webmail; further reading; social media interaction; useful websites; and alumni access to content.

Table 4 shows an extract of the data to demonstrate the top five responses from both staff and students. The data show some overlap between the most common responses.

Table 4.

Data extracted to identify top five responses for staff and students.

Top criteria	Staff	No. of responses	Students	No. of Responses
#1	Lecture Notes	72% (n = 72)	Lecture notes	95% (n = 733)
#2	Module Leader Contact Details	71% (n = 71)	Past exam papers	93% (n = 716)
#3	Further Reading	65% (n = 65)	Further reading	88% (n = 679)
#4	Timetables	59% (n = 59)	Timetables	86% (n = 666)
#5	Contact Details for Other Teaching Staff	59% (n = 59)	Module leader contact details	83% (n = 642)

Table 5 demonstrates the descriptive statistics related to the number of items/criteria selected by staff and student respondents for inclusion in minimum standards, with further breakdown by gender for student responses. Table 6 present further data from statistical tests.

Table 5.

Descriptive statistics for staff and student selections for inclusion in minimum standards.

Respondents		Mean selections	St. Dev.
Staff		9.32	3.07
Students		10.91	2.75
Student	Female	11.1	2.53
Student	Male	10.25	3.28

Table 6.

Statistical data for staff and student selections for inclusion in minimum standards.

Statistical Test	Result	Notes
Pearson product-moment correlation between staff and student responses	r = 0.560	Strong positive correlation
Two-sample t-test between staff and student responses	t(26) = 4.28; p = 0.0002	Extremely significant difference
Two-sample t-test between female and male student responses	t(760) = 3.25; p = 0.0012	Extremely significant difference
Two-sample t-test between number of items selected by staff and student responses	t(843) = 4.887; p = 0.0001	Extremely significant difference

NSS data (summary)

Overall, the quantitative results pertaining to Information Technology (IT) were positive, though there were some differences between the schools. Tables 7 and 8 demonstrate quantitative marks for questions on access to general IT resources (Table 7) and learning resources (Table 8).

Table 7.

NSS responses to question 17: ‘I have been able to access general IT resources when I needed to’ (in percent).

Schools/ year	Dentistry	Health sciences	Life sciences	Medicine	Psychology	Veterinary science	University average
2010	93	92	95	95	88	94	90
2011	96	91	89	96	94	85	89
2012	94	91	88	97	87	91	87
2013	93	84	88	95	79	88	86

Table 8.

NSS responses to the theme of learning resources (in percent).

Schools/ year	Dentistry	Health sciences	Life sciences	Medicine	Psychology	Veterinary science	University average
2010	87	90	90	89	89	90	87
2011	92	89	89	91	93	84	87
2012	89	86	86	94	86	87	86
2013	92	87	87	91	83	89	85

Despite the general positive response demonstrated above, the qualitative comments highlighted some conflicting issues. Students from all six schools complained about the lack of lecture notes that were made available, with many suggestions that the student experience could be improved if lectures were made available in PowerPoint, podcast/lecture recording, or notes formats. There were also complaints that some lecturers were difficult to understand (either lecture content or because English was not their first language), so having access to extra information would be beneficial in these circumstances. Students from all schools also complained about cancelled lectures or classes, that is, they would travel into the University and find a session cancelled. The School of Psychology had the highest number of complaints about lack of computers and for 2013 this received the lowest satisfaction rating across the ILT for question 17 (Table 7).

The lowest scores and the greatest variation in scores were on the theme assessment and feedback (Table 9). It also received the most negative comments from all the students across the ILT. The students wanted more relevant, timely feedback on all aspects of their work. Many of the programmes within the ILT have practical and clinical aspects where students are assessed undertaking skills or tasks, for example Objective Structured Clinical Exams (OSCEs). The School of Dentistry scored well in this area, and had a higher proportion of assessment and feedback taking place using technology, for example the use of iPads in clinics, compared with the School of Medicine, which still largely uses paper-based assessment. Whilst more information would be required to determine statistical significance here, it does provide an interesting trend.

Table 9.

NSS responses to the theme of Assessment & Feedback (in percent).

School/ year	Dentistry	Health sciences	Life sciences	Medicine	Psychology	Veterinary science	University average
2010	61	61	65	31	55	52	59
2011	83	64	60	34	65	48	61
2012	76	69	57	43	60	57	64
2013	72	75	63	41	67	57	68

There were a number of negative comments from across ILT related to poor communication between the schools and students, particularly in response to email queries, for example requiring clarification on topics from lectures. Also, students from the School of Health Sciences and Veterinary Sciences in particular complained about long placement hours and being away from the campus.

Discussion

The following discussion considers the results from online questionnaires, NSS results, and the suitability of Herzberg’s Two-Factor Theory as a conceptual framework for the introduction of minimum standards.

Discussion of questionnaire data

The data obtained from the staff and student surveys provide a unique insight into the introduction of minimum standards for VLE areas; however, it is recognised that the response rate for academic staff (n = 102) was low, possibly due to the time of year (August). Repeating the questionnaire during term time might lead to a greater response rate and therefore more generalisable data. Furthermore, student responses were weighted more towards females than males (3:1 ratio), which is not unusual based on the student demographics in the faculty under study. Follow-up questionnaires and focus groups could investigate these aspects in greater detail.

A perceived limitation of this study could be the response rate. The response rate for the NSS survey was generally high compared with response rate to the questionnaire part of this research. This is to be expected, given the high-profile nature of the NSS and the fact the University and Schools spend a lot of time publicising the NSS and encouraging students to take part. Average response rates for these kinds surveys, however they are distributed, are often low (Armstrong and Lusk, 1987). On average, online survey response rates are 11% below mail and phone surveys (Monroe and Adams, 2012), with rates as low as 2% reported (Petchenik and Watermolen, 2011). However, to receive over 750 responses for the student questionnaire can be seen to represent a good sample, especially as the ILT represents a variety of programmes from clinical courses such as medicine to scientific courses such as zoology and psychology. Therefore, a wide range of views has been gathered. Also, it is possible to triangulate the questionnaire results with the higher NSS results so the results from the minimum standards questionnaire can be extrapolated to a wider base. Finally, the questionnaire asked what respondents felt should be included in minimum standards, rather than reflecting on personal experiences, which should reduce the risk of bias within the respondents.

Some 90% of respondents in the staff survey were in favour of introducing a baseline to help introduce consistency across VLE areas. Generally, staff and student responses follow a similar pattern in relation to criteria for inclusion (Chart 1, r = 0.560); however, items were consistently more important to students than staff (p = 0.0002). These results suggest staff perceptions of what should be included in minimum standards, although correlated, are not an accurate indication of what students feel should be included in minimum standards. The single anomaly here relates to the inclusion of a ‘welcome to the module’, where this was slightly more important for staff than student respondents.

Table 4 highlights staff and students’ top five criteria for inclusion in minimum standards. There is some overlap here, with three criteria from the staff list appearing in students’ top five items, demonstrating a clear awareness, and some correlation to the importance of lecture notes, further reading and timetables.

The areas where the staff opinion differs mostly from the student opinion are in relation to:

Provision of past exam papers (46% difference)

Online submission of coursework (41% difference)

Provision of a module specification (38% difference)

Opportunity for draft/formative feedback on work (34% difference)

In addition to these differences, Table 5 and related t-test data demonstrate significant differences for the number of items selected by staff and students. Students are significantly more likely to select a greater number of items (mean = 10.91) than staff (mean = 9.32); p = 0.0001. These data, together with the anonymised quote provided, might add weight to Kregor et al.’s (2012) suggestion that academic staff view TEL initiatives differently to students based on the required increased workload, but it could also be evidence for the increasing view of the student as a customer paying £9000 per year, and thus wanting more value for money.

If increasing workload is the issue, it presents a clear challenge because certain items, for example module specifications, already exist. It is therefore not immediately apparent as to why academic staff are less open to making them available. This could be related to presuming students do not want (or need) them, or perhaps related to the perceived time requirements for making them available. Also, the difference between responses for the inclusion of past exam papers could be related to students’ desire to ‘train for the test’, or because staff are in the habit of reusing exam questions and want to avoid giving students an advantage. Regardless, it is not possible to draw solid conclusions here, and thus further work is required to clarify exactly why students want criteria included.

Finally, a significant difference was observed related to the number of items selected for inclusion between female students and male students. From the data collected, female students are statistically more likely to select more items for inclusion in minimum standards than males (p = 0.0012); however, the effect size of the sample raises concerns related to the female:male ratio amongst respondents. Again, further work would be required to clarify this area; however, in this study, it could be concluded that females have greater expectations in relation to minimum standards. In practice, this will likely make little difference as the provision of minimum standards would be for all students regardless of gender. It is, however, an interesting perspective to consider in researching the needs and desires of student use of technology in learning and teaching.

Discussion of NSS data

It is important to note that respondents to the NSS are not specifically prompted to reflect on the use of the VLE and associated tools, but in this instance, many of the issues raised could have a solution through the provision of minimum standards in the VLE. The data from the NSS demonstrate clear overlap and triangulation with the data demonstrated in Chart 1 (student identification of criteria for minimum standards). Qualitative comments identified specific criteria such as recommended/further readings, lectures notes/handouts, timetable details and more feedback. As demonstrated in Table 4, these were the most popular items for inclusion in minimum standards, suggesting clear overlap between what students want when asked about the VLE, and when they make more general comments in the NSS.

Particular emphasis from qualitative comments included the following themes:

Provision of learning resources, in particular lecture notes or recorded lectures;

General communication between staff and students;

Cancelled lectures (although qualitative comments suggest it was the lack of communication, rather than the cancellations themselves, that was the problem);

Assessment and feedback, for example appropriate and timely feedback to support students;

Time away from the University on placement.

Assessment and feedback is the single largest concern across the sector, and has long been considered to be not ‘up to scratch’ (Times Higher Education, 2006). In 2007, The Guardian reported that ‘Universities received the lowest score in the area of assessment and feedback, where 62% of students were satisfied’ (Lipsett, 2007). Students wanted feedback on how to improve their performances in exams, written work and, where applicable, on practical skills. Despite a small improvement in NSS figures in this category, there is still much room for improvement. Issues such as useful and timely feedback could be solved by providing opportunity for formative feedback on draft work – something almost 70% of students identified as desirable in the questionnaire. Considering the concerns of staff workload, (formative) peer or self-assessment could still have a positive impact. Similarly, provision of past exam papers was essential for around 90% of respondents and could help prepare for summative assessment.

Reproducing the good practice in Dentistry could provide enhancements to the outdated paper-based mechanisms in Medicine, for example; however, the implementation of iPads is likely beyond the scope of any minimum standards. Generally, these concerns about assessment and feedback have also been seen in the Australasia and Republic of Ireland equivalent of the NSS (Coates, 2010).

Communication is also an obvious theme for improvement. Cancelled lectures are unavoidable over the course of a degree lasting between 3 to 5 years, but communication tools within the VLE could be helpful. Use of announcements or discussion forums to inform students and prevent unnecessary travel could be a solution to impact on dissatisfaction. Medical students also felt that there was poor communication from/with the medical school. Again, forums could be used for students to voice concerns, pose questions, and engage with staff and other students. As detailed in the student questionnaire, over half of student responses suggested the inclusion of discussion forums in VLE areas.

Time away from the university on placement is unavoidable for many of the courses across ILT. However, greater use of the VLE for students on placement could impact on the negative comments related to long placement hours, in particular those from students in the School of Health Sciences. Effective use of the VLE could make students feel more supported and in closer contact with the University, for example more content or recorded lectures, discussion forums or even reflective blogs, the aim of which would be to maintain student–school engagement whilst on placement.

As discussed, the data from the NSS concur with the data received from the student survey for the provision of minimum standards. Such implementations could have a significant impact on NSS results. Of particular interest is that themes from qualitative comments were consistent across the different schools in the ILT, suggesting the results have implications beyond these cohorts. These implications are not just relevant to the UK, as results from other similar student surveys in Ireland and Australasia have also shown concerns, for example, around interaction between staff and students, which could take place online (Coates 2010).

Discussion of suitability of Herzberg’s Two-Factor Theory as a conceptual framework for the introduction of minimum standards

The dissatisfaction reported by the Student Union (Bee, 2013) related to inconsistent use of the VLE, coupled with the data obtained in this study, might suggest these factors are suitable to be considered within a conceptual framework such as Herzberg’s. We propose that many of these ‘hygiene factors’, true to Herzberg’s suggestions, are the cause of student dissatisfaction. Including them in a series of baseline / minimum standards, however, is unlikely to cause satisfaction. Rather, it will simply lead to ‘no dissatisfaction’, and can be thought of as in Maslow’s basic needs, where the satisfaction of one need rests upon the satisfaction of another. To reinforce this notion, qualitative comments in the NSS were only negative in relation to points that align with hygiene factors.

Further work is required to verify these suggestions and delve deeper into factors that lead to student satisfaction (motivators/hygiene); however, from the data obtained, the following mapping of hygiene factors to Maslow’s hierarchy (Figure 2) provides a conceptual framework for understanding student satisfaction with the VLE, and triangulates the ‘Needs’ suggested by Maslow (visible on the left); mapped by the ‘Criteria for VLE minimum standards’ (right); and Herzberg’s hygiene factors and motivators (middle). This suggests hygiene factors map to Maslow’s more basic needs, that is, ‘Physiological’, ‘Safety’, and to a lesser degree, ‘Belonging’, evidenced through criteria such as the availability of learning outcomes, timetables and assessment requirements. True to Maslow’s definition of the physiological need, these are essential criteria for students to ‘survive’ at university. Students’ ‘Safety’ needs are catered for through criteria such as lecture notes, contact details, further reading and past exam papers. Other criteria to cater for needs of belonging, self-esteem and self-actualisation are catered for via online forums, feedback and peer review.

Figure 2.

Mapping of minimum standards (hygiene factors and motivators) to Maslow’s Hierarchy of Needs.

In line with both the suggestions from Maslow and Herzberg, the hygiene factors and basic needs must be satisfied before building further; that is, the physiological needs must be satisfied before focusing on safety needs; and hygiene factors pre-suppose motivators. Furthermore, the dissatisfaction regarding the basic needs/hygiene factors will negatively impact upon the more advanced needs/ motivators. In this regard, and as Herzberg relates to hygiene in the medical sense, hygiene factors are not ‘curative’ in nature, but rather ‘preventative’ – they prevent dissatisfaction. Moreover, the factors that lead to positive attitudes do so because ‘they satisfy individual’s need for self-actualisation’ (Herzberg et al., 1967: 114)

Implementing minimum standards

Data obtained as part of this study suggest that the introduction of minimum standards for VLE areas could impact on student satisfaction. However, significant work must be carried out in order to implement minimum standards consistently. A decision must be made on exactly what criteria should be included based on both the information students want/need, as well as practical issues for their inclusion. For example, the use of discussion forums in all VLE areas was important to more than half of student respondents (54%), and could potentially be included easily in VLE templates. However, they would require some monitoring by academic staff in each module, thus providing additional workload. This might raise further questions around responsibility, as many modules (and elements of teaching in non-modular subjects) utilise staff from across departments or specific research institutes, whose main priority is research.

To help overcome the additional workload for academic staff, it is often possible to harvest many of the criteria from existing systems and display them to students dynamically. This is the approach taken at Newcastle University and Manchester Metropolitan University (identified from the pre-research activity), thus enabling greater focus on enhancing learning and teaching through motivating factors. Removing this workload might provide some counterbalance to the significant differences between the number of items selected for inclusion by staff and students.

Whilst the technical requirements to fulfil other criteria might be minimal, provision of appropriate staff development (workshops/online support materials, etc) should not be underestimated, and topics should cover a range of aspects from introductory uses of the VLE through to more advanced uses, as well as a focus on the pedagogic rationale for the introduction of technology to support and enhance learning, teaching and assessment (Reed, 2014).

Introducing minimum standards based on the suggestions of the student body is likely to reduce the dissatisfaction expressed in the Student Union report (Bee, 2014). It is possible student dissatisfaction is related to an expectation that such information would be available consistently across VLE areas, and thus its provision could reduce dissatisfaction and address Jackson and Fearon’s (2014) notion of ‘disconfirmation of expectation’. It is also recognised that the ‘one size fits all’ approach is not suited to education, and in particular, the use of the VLE. However, the implementation of VLE minimum standards aims to provide a consistent basic use of the system to which individual modules can build upon and further include specific tools and strategies to support teaching and learning. Therefore ultimately, a student might traverse module spaces that are different based on the specific tools/strategies used; however, they will all consistently include the basic fundamental aspects.

Furthermore, in line with Ellis and Calvo (2007), it is felt senior management buy-in and a policy to encourage academic staff to engage is a critical factor in the successful implementation of minimum standards.

Reapplying the SIEME model

Based on this investigation into the introduction of VLE minimum standards, Chyung’s SIEME model might be repurposed as a systematic approach to impact upon motivational and hygiene factors in the context of student satisfaction in HE, not simply in relation to satisfaction with technology, but related to their studies more generally. It is proposed that hygiene factors are not specifically related to student satisfaction with technology, as students are beginning to see technology as part of, not distinct from, their overall student experience. As such, motivating factors might also include non-technology related items, for example charismatic and knowledgeable teaching staff, etc. Therefore, the SIEME model for student satisfaction might be represented by the following five points and diagrammatical model (Figure 3):

Separate concepts of satisfaction and dissatisfaction;

Investigate hygiene and motivation factors;

Eliminate or reduce the hygiene factors;

Maintain and/or add motivation factors; and

Evaluation of student satisfaction continuously.

Figure 3.

Applying the SIEME model to student satisfaction.

A clear challenge exists in designing suitable research methods to explore these issues further. In particular, we should investigate the specific criteria that impact on student satisfaction and dissatisfaction, potentially through focus groups and student interviews, as well as means to measure the impact of the implementation of minimum standards. This paper has only dealt with factors impacting upon student dissatisfaction (hygiene); however, we should also focus on factors that lead to satisfaction. It is also clear that further work is required to understand the differences between staff responses for minimum standards and student responses.

Conclusion

This study provides an insight into staff and student perspectives on the introduction of VLE minimum standards. As a topic under discussion across the sector with limited published literature, this paper is likely to be of interest to academics, academic managers, learning technologists and IT departments across HE; however, it should be noted that the data obtained represent staff and student views at a particular point in time, and within a single faculty at one Russell Group university. The proposed framework for student satisfaction should be taken as a flexible conceptual framework to guide further exploration at other institutions in order to fully understand factors that lead to student satisfaction (motivators) and dissatisfaction (hygiene factors).

If these are truly hygiene factors in Herzberg’s use of the word, they will not necessarily make students’ HE experience completely satisfactory; rather, they will reduce the likelihood students will be dissatisfied in a preventative sense. These factors could be present but other aspects central (or intrinsic) to the teaching and learning experience could be missing, thus preventing students from extreme satisfaction. This notion relates well to Maslow’s hierarchy, whereby the satisfaction of one need rests on the satisfaction of another, and is reinforced by the negative comments raised in the NSS.

As Jackson and Fearon (2014) suggest, ‘organisations can achieve a strong sense of VLE benefits success’ by managing expectations of users (Jackson and Fearon, 2014). Therefore, the introduction of hygiene factors (or what we might also call minimum standards) is not about creating outstanding learning experiences directly, but about ensuring the foundations are in place to enable outstanding experiences to flourish in the future.

Footnotes

Acknowledgements

The authors would like to express gratitude to ELESIG (Evaluation of Learners’ Experiences of e-learning Special Interest Group) for the award of an ELESIG small grant to support the dissemination of this research.

References

Ahmed J and Morley G (2010) VLE a blessing or a curse: VLE use by HE Academic Staff. In: Global Learn Asia Pacific 2010 - Global Conference on Learning and Technology. Available at: http://eprints.hud.ac.uk/8901/.

Anderson

(2012) Dentistry, Medicine, Veterinary and Animal Science. National Student Survey Discipline Report. Higher Education Academy. https://www.heacademy.ac.uk/node/4050.

Armstrong

Lusk

(1987) Return postage in mail surveys a meta-analysis. Public Opinion Quarterly 51(2): 233.

Bee

(2013) Making the Most out of IT: report to TELWG, Liverpool: University of Liverpool.

BERA (2011) Ethical Guidelines for Educational Research, London, Nottingham: British Educational Research Association. Available online at: https://www.bera.ac.uk/researchers-resources/resources-for-researchers. [last accessed 12/01/2015].

Boyatzis

(1998) Transforming Qualitative Information: Thematic analysis and code development. Transforming qualitative information, Thousand Oaks, London & New Delhi: SAGE Publications.

Chyung S (2005) Hoping to reduce attrition? Follow the SIEME model and investigate motivation-hygiene factors. In: Paper presented at the Distance Teaching & Learning conference (pp. 1–6). Madison, WI. Available at: https://opwl.boisestate.edu/faculty-staff/faculty/yonnie-chyung/.

Coates

(2010) Development of the Australasian survey of student engagement (AUSSE). Higher Education 60: 1–7.

Cocksedge

Taylor

DCM

(2013) The National Student Survey: Is it just a bad DREEM? Medical Teacher 35(12): e1638–e1643.

10.

Coverdale-Jones

(2012) Success factors in the implementation of e-learning in a UK higher education institution: The case of one faculty. In: Clothey

Austin-Li

Weidman

(eds) Post-Secondary Education and Technology: A Global Perspective on Opportunities and Obstacles to Development, Palgrave Macmillan, pp. 111–125.

11.

Ellis

Calvo

(2007) Minimum indicators to assure quality of LMS-supported blended learning. Educational Technology & Society 10(2): 6–70.

12.

Hall

(2009) Towards a fusion of formal and informal learning environments: The Impact of the read / write web. Learning 7(1): 29–40.

13.

Herzberg

Mausner

Block Synderman

(1967) The Motivation to Work, 2d edn. New York, New York, USA: John Wiley & Sons.

14.

Hill

Fresen

Geng

(2012) Derivation of electronic course templates for use in higher education. Research in Learning Technology 20(1063519): 1–18.

15.

Jackson

Fearon

(2014) Exploring the role and influence of expectations in achieving VLE benefit success. British Journal of Educational Technology 45(2): 245–259.

16.

Kregor

Breslin

Fountain

(2012) Experience and beliefs of technology users at an Australian university: Keys to maximising e-learning potential. Australasian Journal of Educational Technology 28(8): 1382–1404.

17.

Lipsett A (2007, September). Students’ biggest concern is feedback. The Guardian. Available at: http://education.guardian.co.uk/students/news/ story/0,,2167546,00.html (accessed 14 February 2008).

18.

Maslow

(1943) A theory on human motivation. Psychological Review 50: 370–396.

19.

Monroe

Adams

(2012) Increasing response rates to web-based surveys. Journal of Extension 50(6): Available at: http://www.joe.org/joe/2012december/pdf/JOE_v50_6tt7.pdf.

20.

NSS (2013) The National Student Survey 2013. Available at: http://www.thestudentsurvey.com/the_nss.html (accessed 20 November 2013).

21.

Petchenik

Watermolen

(2011) A cautionary note on using the internet to survey recent Hunter education graduates. Human Dimensions of Wildlife 16(3): 216–218.

22.

Reed

(2013) Hashtags and retweets: Using Twitter to aid community, communication and casual (informal) learning. Research in Learning Technology 21(1063519): 1–21.

23.

Reed P (2014) Sector-wide subscription to VLE minimum standards. The Reed Diaries. Available at: http://thereeddiaries.blogspot.co.uk/2014/03/sector-wide-subscription-to-vle-minimum.html (accessed 21 March 2014).

24.

Reed P (2014) Staff experience and attitudes towards technology-enhanced learning initiatives in one Faculty of Health and Life Sciences. Research in Learning Technology 22. doi:10.3402/rlt.v22.22770. Available online at: http://www.researchinlearningtechnology.net/index.php/rlt/article/view/22770.

25.

Stiles M (2007) Death of the VLE?: A challenge to a new orthodoxy. Serials: The Journal for the Serials Community 20(March). Available at: http://serials.uksg.org/content/bvygddk41gj7/?genre=issue&issn=0953-0460&volume=20&issue=1.

26.

Sun

P.-C

Tsai

Finger

(2008) What drives a successful e-Learning? An empirical investigation of the critical factors influencing learner satisfaction. Computers & Education 50(4): 1183–1202. doi:10.1016/j.compedu.2006.11.007.

27.

Times Higher Education (2006, August). Courses deliver, but feedback falls short. Times Higher Education Supplement. Available at: http://www.timeshighereducation.co.uk/news/courses-deliver-but-feedback-falls-short/204943.article.

28.

J-H

Tennyson

Hsia

T-L

(2010) A study of student satisfaction in a blended e-learning system environment. Computers & Education 55(1): 155–164.

29.

Youn

Chyung

Vachon

(2005) An investigation of the profiles of satisfying and dissatisfying factors in e-learning. Performance Improvement Quarterly 18(2): 97–113.

Hygiene factors: Using VLE minimum standards to avoid student dissatisfaction