The Comparative Impact of Critics and Consumers: Applying the Generalisability Theory to Online Movie Ratings

Abstract

This study employs a new measurement theory (i.e. Generalisability Theory) to investigate the comparative influence of early movie ratings from professional critics versus ordinary consumers on latent movie performance. The empirical results show that both ordinary consumers and critics have great impact on the latent movie performance. In particular, the main effect of rater sources and the two-way interaction between raters and movies are substantial contributors to the variation in movie performance, with the contribution from ordinary consumers even more substantial than that from professional critics. However, professional critics provide more reliable ratings (a higher G coefficient) than ordinary consumers. Moreover, we found that genre familiarity is an important factor that moderates the differential effect of these two sources of ratings. Professional critic ratings contribute more to the total variance of movie performance evaluations in the case of less familiar genres, while ordinary consumer ratings contribute more to that in the case of more familiar genres. The aggregate level validity (correlation) results for each rater source indicate that professional critics consistently provide better concurrent and predictive validity than ordinary consumers. While our analyses focused on the impact of two sources of ratings on movie performance evaluations, the findings have implications not limited to the movie industry. They are also applicable to the broad category of experience goods such as music, restaurants, video games and books, where consumers could seek opinion from both experts and ordinary consumers.