Generalizability Theory and Experimental Design: Incongruity Between Analysis and Inference

Abstract

The inferences drawn from many statistical analyses are not congruent with the analyses performed. The analysis commonly employed when cognitive or affective measures serve as the dependent variable yields an inference that is rigorously generalizable only to the specific set of scales or items that were employed, and not to the intended universe of items of which the set used is viewed as a representative sample. The arguments in this paper show that if the items are incorporated into the design as levels of a random facet via generalizability theory, the inferential question in the desired universe of inference can be examined statistically. An estimate of the reliability (generalizability) of the outcome variable also accrues from the proposed analysis; consequently the statistical and measurement fidelity questions are unified in a single analysis. Advantages of nesting items within subjects are proposed.