Updating Generalizability Theory in Management Research

Abstract

In the management literature, generalizability theory (GT) has been typically used to investigate the reliability of assessment center and job performance ratings. However, the management field has yet to take full advantage of the information GT can offer regarding the reliability of measurement. It is likely that GT has not been adopted because of the complexities involved with its notation and practical application. Moreover, current methods for obtaining accurate interval estimates around estimated variance components or their reliability coefficients are not easily implementable. Alternatively, Bayesian methods provide a different method for estimating GT variance components. Bayesian methods enable management researchers to estimate the posterior distributions of each GT variance component as well as the GT reliability coefficients. From these posterior distributions, researchers can easily obtain the interval estimates for each variance component and the corresponding reliability estimates. Conducting two studies, the authors examine what priors should be used when conducting a Bayesian GT analysis and what estimates should be used to summarize a variance component’s posterior distribution. Additionally, the authors find that under certain conditions, Bayesian methods perform better than frequentist methods.