Mixed Methods for Human–Computer Interactions Research: An Iterative Study Using Reddit and Social Media

Abstract

Researchers have called for additional empirical studies associated with video games. However, every game is novel in terms of mechanics, content, context, and agency; known variables may be operationalized differently based on the game involved. It is incumbent upon researchers to leverage or create the best tools when extracting data from games. This article models instrument development of a scale intended to catalog users’ actions in a novel context (i.e., the game League of Legends) using a mixed methods approach. Specifically, this work outlines data collection and validation strategies using an online social news aggregation, rating, and discussion resource (i.e., Reddit), involving multiple cycles to elicit expert input and queries to generate consensus from expert gamers, followed by analysis of responses and an exploratory factor analysis for scale construction. Reddit provided three unique functions relative to the process: (a) access to experts who informed the development of scale items, (b) a social space and mechanism to validate scale items, and (c) the opportunity to capture data necessary to establish the psychometric properties of the instrument. Findings associated with scale development (i.e., item generation, theoretical and psychometric analyses) are presented. Overall, implications for instrument development in continually evolving contexts are discussed.