Mixed group validation (MGV) is a statistical model for estimating the diagnostic accuracy of tests. Unlike the more common approach to estimating criterion-related validity, known group validation (KGV), MGV does not require a perfect external validity criterion. The present article describes MGV by (a) specifying both the standard error associated with MGV validity estimates and the effect of assumption violation, (b) recommending required sample sizes under various study conditions, (c) evaluating whether assumption violation can be identified, and (d) providing a simulated example of an MGV with imperfect base rate estimates. It is concluded that MGV will always have a wider margin of error than KGV, MGV performs best when the research design approximates a KGV design, the effect of assumption violation depends on the severity of the assumption violation and also the value of the base rates, and that assumption violation may only be detected in severe cases.
AchenC. H.ShivelyW. P. (1995). Cross-level inference. Chicago, IL: University of Chicago Press.
2.
AltmanD. G.BlandJ. M. (1994a). Diagnostic tests 1: Sensitivity and specificity. British Medical Journal, 308, 1552.
3.
AltmanD. G.BlandJ. M. (1994b). Diagnostic tests 2: Predictive values. British Medical Journal, 309, 102.
4.
American Psychological Association. (2006). Evidence-based practice in psychology. American Psychologist, 61, 271-285.
5.
BudingerT. F.DerenzoS. E.GullbergG. T.GreenbergW. L.HuesmanR. H. (1977). Emission computer assisted tomography with single-photon and positron annihilation photon emitters. Journal of Computer Assisted Tomography, 1, 131-145.
6.
CobbS.HuntP.HarburgE. (1969). The intrafamilial transmission of rheumatoid arthritis—II. Journal of Chronic Diseases, 22, 203-215.
7.
CrawfordE. F.GreeneR. L.DupartT. M.BongarB.ChildsH. (2006). MMPI-2 assessment of malingered emotional distress related to a workplace injury: A mixed group validation. Journal of Personality Assessment, 86, 217-221.
8.
DawesR. M. (1967). How clinical probability judgmentmaybe used to validate diagnostic signs. Journal of Clinical Psychology, 23, 403-410.
9.
DawesR. M.FaustD.MeehlP. E. (1989). Clinical versus actuarial judgment. Science, 243, 1668-1674.
10.
DawesR. M.MeehlP. E. (1966). Mixed group validation. Psychological Bulletin, 66, 63-67.
11.
DuncanO. D.DavisB. (1953). An alternative to ecological correlation. American Sociological Review, 18, 665-666.
12.
EfronB.TibshiraniR. J. (1993). An introduction to the bootstrap. New York, NY: Chapman & Hall.
13.
FidlerF.ThomasonN.CummingG.FinchS.LeemanJ. (2004). Editors can lead researchers to confidence intervals, but can’t make them think. Statistical reform lessons from medicine. Psychological Science, 15, 119-126.
14.
FleissJ. L. (1973). Statistical methods for rates and proportions. New York, NY: Wiley.
15.
FrederickR. I. (2000). Mixed group validation: A method to address the limitations of criterion group validation in research on malingering detection. Behavioral Sciences & the Law, 18, 693-718.
16.
FrederickR. I.BowdenS. C. (2009a). The test validation summary. Assessment, 16, 215-236.
17.
FrederickR. I.BowdenS. C. (2009b). Evaluating constructs represented by symptom validity tests in forensic neuropsychological assessment of traumatic brain injury. Journal of Head Trauma Rehabilitation, 24, 105-122.
18.
GartJ. J.BuckA. A. (1966). Comparison of a screening test and a reference test in epidemiologic studies II: A probabilistic model for the comparison of diagnostic tests. American Journal of Epidemiology, 83, 593-602.
19.
GelmanA.ParkD. K.AnsolabehereS.PriceP. N.MinniteL. C. (2001). Models, assumptions and model checking in ecological regressions. Journal of the Royal Statistical Society. Series A. Statistics in Society, 164, 101-118.
20.
GoldbergL. R. (1970). Man versus model of man: A rationale, plus some evidence, for a method of improving on clinical inferences. Psychological Bulletin, 73, 422-432.
21.
GoodmanL. (1953). Ecological regression and the behavior of individuals. American Sociological Review, 18, 663-664.
22.
GoodmanL. (1959). Some alternatives to ecological correlation. American Journal of Sociology, 64, 610-625.
23.
HagenaarsJ.McCutcheonA. (Eds.). (2002). Applied latent class analysis models. New York, NY: Cambridge University Press.
24.
HuiS. L.ZhouX. H. (1998). Evaluation of diagnostic tests without gold standards. Statistical Methods in Medical Research, 7, 354-370.
25.
Institute of Medicine. (2001). Crossing the quality chasm: A new health system for the 21st century. Washington, DC: National Academies Press.
26.
KnottnerusJ. A.van WeelC.MurisJ. W. M. (2002). Evaluation of diagnostic procedure. British Medical Journal, 324, 477-480.
27.
KnowlesE. E.SchroederD. A. (1990). Concurrent validity of the MacAndrew Alcoholism Scale: Mixed-group validation. Journal of Studies on Alcohol, 51, 257-262.
28.
LinnR. L. (1967). A note on mixed group validation. Psychological Bulletin, 67, 378.
29.
MeehlP. E. (1986). Diagnostic taxa as open concepts: Metatheoretical and statistical questions about reliability and construct validity in the grand strategy of nosological revision. In MillonT.KlermanG. (Eds.), Contemporary directions in psychopathology: Toward the DSM-IV (pp. 215-231). New York, NY: Guilford Press.
30.
MeehlP. E.RosenA. (1955). Antecedent probability and the efficiency of psychometric signs, patterns, or cutting score. Psychological Bulletin, 52, 194-216.
31.
MontgomeryD. C.PeckE. A. (1982). Introduction to linear regression analysis. New York, NY: Wiley.
32.
RorerL. G.DawesR. M. (1982). A base-rate bootstrap. Journal of Consulting and Clinical Psychology, 50, 419-425.
33.
StokesD. E. (1969). Cross-level inference as a game against nature. In BerndJ. L. (Ed.), Mathematical applications in political science (pp. 62-83). Charlottesville: University Press of Virginia.
34.
StrausS. E.RichardsonW. S.GlasziouP.HaynesR. B. (2005). Evidence-based medicine: How to practice and teach EBM (3rd ed.). Edinburgh, Scotland: Elsevier.
35.
StraussM. E.SmithG. T. (2009). Construct validity: Advances in theory and methodology. Annual Review of Clinical Psychology, 5, 1-25.
36.
TaylorJ. R. (1982). An introduction to error analysis. Mill Valley, CA: University Science Books.
37.
TolinD. F.SteenkampM. M.MarxB. P.LitzB. T. (2010). Detecting symptom exaggeration in combat veterans using the MMPI-2 symptom validity scale: A mixed group validation. Psychological Assessment, 22, 729-736.
38.
ValensteinP. N. (1990). Evaluating diagnostic tests with imperfect standards. American Journal of Clinical Pathology, 93, 252-258.
39.
WypijD. (2005). Binomial distribution. In ArmitageP.ColtonT. (Eds.). Encyclopedia of biostatistics (Vol. 1, pp. 447-450). Chichester, England: Wiley.
40.
YerushalmyJ. (1947). Statistical problems in assessing methods of medical diagnosis, with special reference to X-ray techniques. Public Health Reports, 62, 1432-1449.
