Technology-mediated learning (TML) is a major trend in education, since it allows to integrate the strengths of traditional- and IT-based learning activities. However, TML providers still struggle in identifying areas for improvement in their TML offerings. One reason for their struggles is inconsistencies in the literature regarding drivers of TML performance. Prior research suggests that these inconsistencies in TML literature might stem from neglecting the importance of considering the process perspective in addition to the input and outcome perspectives. This gap needs to be addressed to better understand the different drivers of the performance of TML scenarios. Filling this gap would further support TML providers with more precise guidance on how to (re-)design their offerings toward their customers’ needs. To achieve our goal, we combine qualitative and quantitative methods to develop and evaluate a holistic model for assessing TML performance. In particular, we consolidate the body of literature, followed by a focus group workshop and a Q-sorting exercise with TML practitioners, and an empirical pre-study to develop and initially test our research model. Afterward, we collect data from 161 participants of TML vocational software trainings and evaluate our holistic model for assessing TML performance. The results provide empirical evidence for the importance of the TML process quality dimension as suggested in prior literature and highlighted by our TML practitioners. Our main theoretical as well as practical contribution is a holistic model that provides comprehensive insights into which constructs and facets shape the performance of TML in vocational software trainings.
AlaviM. (1994). Computer-Mediated Collaborative Learning: An Empirical Evaluation. MIS Quarterly, 18 (2), 159–174.
2.
AlaviM., and LeidnerD.E. (2001a). Research Commentary: Technology-Mediated Learning-A Call for Greater Depth and Breadth of Research. Information Systems Research, 12 (1), 1–10.
3.
AlaviM., and LeidnerD.E. (2001b). Review: Knowledge management and knowledge management systems: Conceptual foundations and research issues, MIS Quarterly: 107–136.
4.
AlaviM., MarakasG.M., and YooY. (2002). A comparative study of distributed learning environments on learning outcomes. Information Systems Research, 13 (4), 404–415.
5.
AlaviM., WheelerB.C., and ValacichJ.S. (1995). Using IT to Reengineer Business Education: An Exploratory Investigation of Collaborative Telelearning. MIS Quarterly, 19 (3), 293–312.
6.
AlaviM., YooY., and VogelD.R. (1997). Using Information Technology to Add Value to Management Education. The Academy of Management Journal, 40 (6), 1310–1333.
7.
AlonsoF., ManriqueD., MartinezL., and VinesJ.M. (2011). How Blended Learning Reduces Underachievement in Higher Education: An Experience in Teaching Computer Sciences. IEEE Transactions on Education, 54 (3), 471–478.
8.
Aragon-SanchezA., Barba-AragónI., and Sanz-ValleR. (2003). Effects of training on business results1. The International Journal of Human Resource Management, 14 (6), 956–980.
9.
ArbaughJ.B. (2000). How Classroom Environment and Student Engagement Affect Learning in Internet-based MBA Courses. Business Communication Quarterly, 63 (4), 9–26.
10.
ArbaughJ.B. (2001). How Instructor Immediacy Behaviors Affect Student Satisfaction and Learning in Web-Based Courses. Business Communication Quarterly, 64 (4), 42–54.
11.
ArbaughJ.B. (2013). Does Academic Discipline Moderate CoI-Course Outcomes Relationships in Online MBA Courses?The Internet and Higher Education, 17, 16–28.
12.
ArbaughJ.B., and Benbunan-FichR. (2007). The importance of participant interaction in online environments. Decision Support Systems, 43 (3), 853–865.
13.
ArmstrongS.J., AllinsonC.W., and HayesJ. (2004). The effects of cognitive style on research supervision: A study of student-supervisor dyads in management education. Academy of Management Learning and Education, 3 (1), 41–63.
14.
BeckerJ.-M., KleinK., and WetzelsM. (2012). Hierarchical Latent Variable Models in PLS-SEM: Guidelines for Using Reflective-Formative Type Models. Long Range Planning, 45 (5-6), 359–394.
15.
Benbunan-FichR. (2010). Is Self-Reported Learning a Proxy Metric for Learning? Perspectives From the Information Systems Literature. Academy of Management Learning and Education, 9 (2), 321–328.
16.
Benbunan-FichR., and ArbaughJ.B. (2006). Separating the Effects of Knowledge Construction and Group Collaboration in Learning Outcomes of Web-based Courses. Information and Management, 43 (6), 778–793.
17.
BiggsJ. (1993). What do inventories of students’ learning processes really measure?: A theoretical review and clarification. British Journal of Educational Psychology, 63 (1), 3–19.
18.
BiggsJ., KemberD., and LeungD.Y. (2001). The revised two-factor Study Process Questionnaire: R-SPQ-2F. British Journal of Educational Psychology, 71 (1), 133–149.
19.
BitzerP., and JansonA. (2014). Towards a Holistic Understanding of Technology-Mediated Learning Services – a State-of-the-Art Analysis, ECIS 2014 Proceedings.
20.
BitzerP., SöllnerM., and LeimeisterJ.M. (2013). Evaluating the Quality of Technology-Mediated Learning Services, ICIS 2013 Proceedings.
21.
BordensK.S., and AbbottB.B. (2011). Research design and methods: A process approach (8th ed.). New York: McGraw-Hill.
22.
BrowerH.H. (2003). On emulating classroom discussion in a distance-delivered OBHR course: Creating an on-line learning community. Academy of Management Learning and Education, 2 (1), 22–36.
23.
CenfetelliR.T., and BassellierG. (2009). Interpretation of formative measurement in information systems research. MIS Quarterly, 33 (4), 689–707.
24.
ChinW. (1998). The partial least squares approach for structural equation modeling. In MarcoulidesG. A. (ed.), Modern methods for business research (pp. 295–336). Mahwah, N.J: Lawrence Erlbaum.
25.
ChinW.W., ThatcherJ.B., and WrightR.T. (2012). Assessing common method bias: Problems with the ULMC technique. MIS Quarterly, 36 (3), 1003–1019.
26.
ChoiD.H., KimJ., and KimS.H. (2007). ERP Training with a Web-Based Electronic Learning System: The Flow Theory Perspective. International Journal of Human-Computer Studies, 65 (3), 223–243.
27.
ChurchillG.A.Jr. (1979). A paradigm for developing better measures of marketing constructs, Journal of Marketing Research: 64–73.
28.
ChwelosP., BenbasatI., and DexterA.S. (2001). Research Report: Empirical Test of an EDI AdoptionModel. Information Systems Research, 12 (3), 304.
29.
ColeM.S., FeildH.S., and HarrisS.G. (2004). Student learning motivation and psychological hardiness: Interactive effects on students’ reactions to a managementclass. Academy of Management Learning and Education, 3 (1), 64–85.
30.
ColquittJ.A., LePineJ.A., and NoeR.A. (2000). Toward an integrative theory of training motivation: A meta-analytic path analysis of 20 years of research. Journal of Applied Psychology, 85 (5), 678–707.
31.
de MatosC.A., and RossiC.A.V. (2008). Word-of-mouth communications in marketing: A meta-analytic review of the antecedents and moderators. Journal of the Academy of Marketing Science, 36 (4), 578–596.
32.
DeciE.L., and RyanR.M. (2000). The “What” and “Why” of Goal Pursuits: Human Needs and the Self-Determination of Behavior. Psychological Inquiry, 11 (4), 227–268.
33.
DelenE., LiewJ., and WillsonV. (2014). Effects of interactivity and instructional scaffolding on learning: Self-regulation in online video-based environments. Computers and Education, 78, 312–320.
34.
DeLoneW.H., and McLeanE.R. (1992). Information systems success: The quest for the dependent variable. Information Systems Research, 3 (1), 60–95.
35.
DeLoneW.H., and McLeanE.R. (2003). The DeLone and McLean model of information systems success: a ten-year update. Journal of Management Information Systems, 19 (4), 9–30.
36.
DijkstraT., and HenselerJ. (2015). Consistent Partial Least Squares Path Modeling. Management Information Systems Quarterly, 39 (2), 297–316.
37.
EomS.B. (2011). Relationships Among E-Learning Systems and E-Learning Outcomes: A Path Analysis Model. Human Systems Management, 30 (4), 229–241.
38.
EomS. (2015). Effects of Self-Efficacy and Self-regulated Learning on LMS User Satisfaction and LMS Effectiveness, AMCIS 2015 Proceedings.
39.
EomS.B., WenH.J., and AshillN. (2006). The Determinants of Students’ Perceived Learning Outcomes and Satisfaction in University Online Education: An Empirical Investigation*. Decision Sciences Journal of Innovative Education, 4 (2), 215–235.
40.
EvansC., and GibbonsN.J. (2007). The interactivity effect in multimedia learning. Computers and Education, 49 (4), 1147–1160.
41.
FreyJ.H., and FontanaA. (1991). The group interview in social research. The Social Science Journal, 28 (2), 175–187.
42.
GarrisonD.R., and KanukaH. (2004). Blended learning: Uncovering its transformative potential in higher education. The Internet and Higher Education, 7 (2), 95–105.
43.
GregorS. (2006). The nature of theory in information systems. MIS Quarterly, 30 (3), 611–642.
44.
GuptaS., and BostromR. (2009). Technology-Mediated Learning: A Comprehensive Theoretical Model. Journal of the Association for Information Systems, 10 (9), 686–714.
45.
GuptaS., and BostromR. (2013). An Investigation of the Appropriation of Technology-Mediated Training Methods Incorporating Enactive and Collaborative Learning. Information Systems Research, 24 (2), 454–469.
46.
GuptaS., BostromR.P., and HuberM. (2010). End-user Training Methods: What We Know, Need to Know,. SIGMIS Database, 41 (4), 9–39.
47.
HairJ.F., HultG.T.M., RingleC.M., and SarstedtM. (2014). A primer on partial least squares structural equations modeling (PLS-SEM).Los Angeles: Sage.
48.
HannafinM.J., KimM.C., and KimH. (2004). Reconciling research, theory, and practice in web-based teaching and learning: The case for grounded design. Journal of Computing in Higher Education, 15 (2), 3–20.
49.
HansenD.E. (2008). Knowledge Transfer in Online Learning Environments. Journal of Marketing Education, 30 (2), 93–105.
50.
HattieJ., and YatesG.C.R. (2014). Visible learning and the science of how we learn.New York: Routledge.
51.
HenselerJ., RingleC.M., and SarstedtM. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43 (1), 115–135.
52.
HiltzS.R., CoppolaN., RotterN., ToroffM., and Benbunan-FichR. (2000). Measuring the importance of collaborative learning for the effectiveness of ALN: A multi-measure. Online Education: Learning Effectiveness and Faculty Satisfaction, 1, 101–119.
53.
JacobsS.C., HuprichS.K., GrusC.L., CageE.A., ElmanN. S., ForrestL. (2011). Trainees with professional competency problems: Preparing trainers for difficult but necessary conversations. Training and Education in Professional Psychology, 5 (3), 175.
54.
JansonA., SöllnerM., BitzerP., and LeimeisterJ.M. (2014). Examining the Effect of Different Measurements of Learning Success in Technology-mediated Learning Research, ICIS 2014 Proceedings.
55.
JansonA., SöllnerM., and LeimeisterJ.M. (2017). Individual Appropriation of Learning Management Systems: Antecedents and Consequences, AIS Transactions on Human-Computer Interaction (forthcoming).
56.
JansonA., and Thiel de GafencoM. (2015). Engaging the Appropriation of Technology-mediated Learning Services – A Theory-driven Design Approach, ECIS 2015 Proceedings.
57.
JohnsonR.D., GueutalH., and FalbeC.M. (2009). Technology. Trainees, Metacognitive Activity and E-Learning EffectivenessJournal of Managerial Psychology, 24 (6), 545–566.
58.
KimJ., KwonY., and ChoD. (2011). Investigating Factors that Influence Social Presence and Learning Outcomes in Distance Higher Education. Computers and Education, 57 (2), 1512–1520.
59.
KirkpatrickD., and KirkpatrickJ. (2005). Transferring learning to behavior: Using the four levels to improve performance:Berrett-Koehler Publishers.
60.
KleinH.J., NoeR.A., and WangC (2006). Motivation to Learn and Course Outcomes: The Impact of Delivery Mode, Learning Goal Orientation, and Perceived Barriers and Enablers. Personnel Psychology, 59 (3), 665–702.
61.
KolfschotenG.L., BriggsR.O., de VreedeG.-J., JacobsP.H.M., and AppelmanJ.H. (2006). A conceptual foundation of the thinkLet concept for Collaboration Engineering. International Journal of Human-Computer Studies, 64 (7), 611–621.
62.
KuoY.-K., and YeK.-D. (2009). The causal relationship between service quality, corporate image and adults’ learning satisfaction and loyalty: A study of professional training programmes in a Taiwanese vocational institute. Total Quality Management and Business Excellence, 20 (7), 749–762.
63.
LadhariR. (2009). A review of twenty years of SERVQUAL research. International Journal of Quality and Service Sciences, 1 (2), 172–198.
64.
LandisJ.R., and KochG.G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33 (1), 159.
65.
LehmannK., and SöllnerM. (2014). Theory-driven design of a mobile-learning application to support different interaction types in large-scale lectures, ECIS 2014 Proceedings.
66.
LehmannK., SöllnerM., and LeimeisterJ.M. (2016). Design and Evaluation of an IT-based Peer Assessment to Increase Learner Performance in Large-Scale Lectures, ICIS 2016 Proceedings.
67.
LeidnerD., and JarvenpaaS. (1995). The Use of Information Technology to Enhance Management School Education: A Theoretical View, Management Information Systems Quarterly19 (3).
68.
LimH., LeeS.-G., and NamK. (2007). Validating E-Learning Factors Affecting Training Effectiveness. International Journal of Information Management, 27 (1), 22–35.
69.
LinH.-F. (2007). Measuring Online Learning Systems Success: Applying the Updated DeLone and McLean Model. CyberPsychology and Behavior, 10 (6), 817–820.
70.
LinnenbrinkE.A., and PintrichP.R. (2002). Motivation as an enabler for academic success. School Psychology Review, 31 (3), 313.
71.
López-PérezM.V., Pérez-LópezM.C., and Rodríguez-ArizaL. (2011). Blended learning in higher education: Students’ perceptions and their relation to outcomes. Computers and Education, 56 (3), 818–826.
72.
MackenzieS.B., PodsakoffP.M., and JarvisC.B. (2005). The Problem of Measurement Model Misspecification in Behavioral and Organizational Research and Some Recommended Solutions. Journal of Applied Psychology, 90 (4), 710–730.
73.
MilesM.B., and HubermanA.M. (1994). Qualitative data analysis: An expanded sourcebook:Sage.
74.
MooreM.G. (1989). Editorial: Three types of interaction. American Journal of Distance Education, 3 (2), 1–7.
75.
NahmA.Y., RaoS.S., Solis-GalvanL.E., and Ragu-NathanT.S. (2002). The Q-sort method: assessing reliability and construct validity of questionnaire items at a pre-testing stage. Journal of Modern Applied Statistical Methods, 1 (1), 15.
76.
OzkanS., and KoselerR. (2009). Multi-dimensional Students’ Evaluation of E-Learning Systems in the Higher Education Context: An Empirical Investigation. Computers and Education, 53 (4), 1285–1296.
77.
ParasuramanA., ZeithamlV.A., and BerryL.L. (1985). A conceptual model of service quality and its implications for future research, The Journal of Marketing: 41–50.
78.
PetterS., DeLoneW., and McLeanE. (2008). Measuring information systems success: models, dimensions, measures, and interrelationships. European Journal of Information Systems, 17 (3), 236–263.
79.
PiccoliG., AhmadR., and IvesB. (2001). Web-Based Virtual Learning Environments: A Research Framework and a Preliminary Assessment of Effectiveness in Basic IT Skills Training, Management Information Systems Quarterly25 (4).
80.
PintrichP.R., and De GrootE.V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82 (1), 33.
81.
PodsakoffP., MacKenzieS., LeeJ., and PodsakoffN. (2003). Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology, 88 (5), 879–903.
82.
PooleM.S., de Ven, Van AndrewH., DooleyK., and HolmesM.E. (2000). Organizational change and innovation processes: Theory and methods for research.Oxford, New York: Oxford Univ. Press.
83.
RaschT., and SchnotzW. (2009). Interactive and Non-Interactive Pictures in Multimedia Learning Environments: Effects on Learning Outcomes and Learning Efficiency. Learning and Instruction, 19 (5), 411–422.
84.
ReberR.A., and WallinJ.A. (1984). The effects of training, goal setting, and knowledge of results on safe behavior: A component analysis. Academy of Management Journal, 27 (3), 544–560.
85.
RingleC.M., SarstedtM., and StraubD.W. (2012). A Critical Look at the Use of PLS-SEM in MIS Quarterly, MIS Quarterly36 (1): iii–xiv; s3-s8 (supplement).
86.
RingleC.M., WendeS., and BeckerJ.-M. (2015). SmartPLS [www document] www.smartpls.de.
87.
RubinD.B. (1987). Multiple imputation for nonresponse in surveys.New York: Wiley.
88.
SanthanamR., LiuD., and ShenW.-C.M. (2016). Research Note – Gamification of Technology-Mediated Training: Not All Competitions Are the Same. Information Systems Research, 27 (2), 453–465.
89.
SanthanamR., SasidharanS., and WebsterJ. (2008). Using Self-Regulatory Learning to Enhance E-Learning-Based Information Technology Training. Information Systems Research, 19 (1), 26–47.
90.
SanthanamR., YiM., SasidharanS., and ParkS.-H. (2013). Toward an Integrative Understanding of Information Technology Training Research across Information Systems and Human-Computer Interaction: A Comprehensive Review. AIS Transactions on Human-Computer Interaction, 5 (3), 134–156.
91.
SattlerH., VölcknerF., RiedigerC., and RingleC.M. (2010). The Impact of Brand Extension Success Factors on Brand Extension Price Premium. International Journal of Research in Marketing, 27 (4), 319–328.
92.
SchankR. (2005). Lessons in Learning, e-Learning, and Training. Perspectives and Guidance for the Enlightened Trainer., Pfeiffer: John Wiley.
93.
SeydaS., and WernerD. (2012). IW-Weiterbildungserhebung 2011: Gestiegenes Weiterbildungsvolumen bei konstanten Kosten. IW-Trends–Vierteljahresschrift zur empirischen Wirtschaftsforschung, 39 (1), 37–54.
94.
SharmaR., YettonP., and CrawfordJ. (2009). Estimating the Effect of Common Method Variance: The Method-Method Pair Technique with an Illustration from TAM Research. MIS Quarterly, 33 (3), 473.
95.
SiauK., ShengHong, and NahF.F.-H. (2006). Use of a classroom response system to enhance classroom interactivity. Education, IEEE Transactions on, 49 (3), 398–403.
96.
SimsR. (2003). Promises of Interactivity: Aligning Learner Perceptions and Expectations With Strategies for Flexible and Online Learning. Distance Education, 24 (1), 87–103.
97.
SitzmannT., ElyK., BrownK.G., and BauerK.N. (2010). Self-Assessment of Knowledge: A Cognitive Learning or Affective Measure?Academy of Management Learning and Education, 9 (2), 169–191.
98.
SmithS.M., and WoodyP.C. (2000). Interactive Effect of Multimedia Instruction and Learning Styles. Teaching of Psychology, 27 (3), 220–223.
99.
SollnerM., HoffmannA., HoffmannH., WackerA., and LeimeisterJ.M. (2012). Understanding the Formation of Trust in IT Artifacts, ICIS 2016 Proceedings.
100.
SöllnerM., HoffmannA., and LeimeisterJ.M. (2016). Why Different Trust Relationships Matter for Information Systems Users. European Journal of Information Systems, 25 (3), 274–287.
101.
SousaN., AlvesG.R., and GericotaM.G. (2010). An Integrated Reusable Remote Laboratory to Complement Electronics Teaching. IEEE Transactions on Learning Technologies, 3 (3), 265–271.
102.
StevensJ.P. (1992). Applied Multivariate Statistics for the Social Sciences, 2nd ed., Hillsdale: Lawrence Erlbaum Associates.
103.
StraubD.W. (1989). Validating Instruments in MIS Research. MIS Quarterly, 13 (2), 147–169.
104.
SultanP., and WongH.Y. (2010). Service quality in higher education – a review and research agenda. International Journal of Quality and Service Sciences, 2 (2), 259–272.
105.
SunP.-C., TsaiR.J., FingerG., ChenY.-Y., and YehD. (2008). What drives a successful e-Learning? An empirical investigation of the critical factors influencing learner satisfaction. Computers and Education, 50 (4), 1183–1202.
106.
SwellerJ., and ChandlerP. (1991). Evidence for Cognitive Load Theory. Cognition and Instruction, 8 (4), 351–362.
107.
TannenbaumS.I., MathieuJ.E., SalasE., and Cannon-BowersJ.A. (1991). Meeting trainees’ expectations: The influence of training fulfillment on the development of commitment, self-efficacy, and motivation. Journal of Applied Psychology, 76 (6), 759.
108.
ThurmondV., and WambachK. (2004). Understanding interactions in distance education: A review of the literature. International Journal of Instructional Technology and Distance Learning, 1 (1), 9–26.
109.
TseD.K., and WiltonP.C. (1988). Models of Consumer Satisfaction Formation: An Extension. Journal of Marketing Research, 25 (2), 204–212.
110.
van der RheeB., VermaR., PlaschkaG.R., and KickulJ.R. (2007). Technology readiness, learning goals, and eLearning: Searching for synergy. Decision Sciences Journal of Innovative Education, 5 (1), 127–149.
111.
van DykeT.P., KappelmanL.A., and PrybutokV.R. (1997). Measuring information systems service quality: concerns on the use of the SERVQUAL questionnaire, MIS Quarterly: 195–208.
112.
VollmarM. (2013). Berufliche Weiterbildung in Unternehmen 2010, Methodik und erste Ergebnisse. Wirtschaft und Statistik, H, 4, 276–287.
113.
WangY.-S. (2003). Assessment of learner satisfaction with asynchronous electronic learning systems. Information and Management, 41 (1), 75–86.
114.
WangW., and BenbasatI. (2005). Trust in and adoption of online recommendation agents. Journal of the Association for Information Systems, 6 (3), 72–101.
115.
WangY.-S., WangH.-Y., and SheeD.Y. (2007). Measuring E-Learning Systems Success in an Organizational Context: Scale Development and Validation. Computers in Human Behavior, 23 (4), 1792–1808.
116.
WegenerR., MenschnerP., and LeimeisterJ. (2012). Design and Evaluation of a Didactical Service Blueprinting Method For Large Scale Lectures, ICIS 2012 Proceedings.
117.
WoldH.O.A. (1982). Soft Modeling: The Basic Design and Some Extensions. In JöreskogK. G., and WoldH. O. A. (eds.), Systems under indirect observation: Causality, structure, prediction (pp. 1–54). Amsterdam: North-Holland.
118.
YiM.Y., and DavisF.D. (2003). Developing and Validating an Observational Learning Model of Computer Software Training and Skill Acquisition: Information Systems Research. Information Systems Research, 14 (2), 146–169.
119.
YooY., KanawattanachaiP., and CitursA. (2002). Forging into the Wired Wilderness: A Case Study of a Technology-Mediated Distributed Discussion-Based Class. Journal of Management Education, 26 (2), 139–163.
