A Meta-Analytic Evaluation of Naglieri Nonverbal Ability Test: Exploring Its Validity Evidence and Effectiveness in Equitably Identifying Gifted Students

Abstract

The Naglieri Nonverbal Ability Test (NNAT) was developed to more equitably identify students of color, as it advertises itself as a culture-fair measure. In this meta-analytic evaluation, we aimed to investigate (a) the generalizability of validity evidence of NNAT by checking its construct and criterion validity with other measures (Part I) and (b) whether NNAT truly meets its goal to identify more culturally diverse students (Part II). After reviewing 1,714 studies, a total of 29 studies met our criteria (59 effect sizes from 22 studies for Part I and 7 effect sizes from 7 studies for Part II). In Part I, we investigated empirical evidence of validity of NNAT in relationship with different types of measures (overall effect size of r was .44); The results revealed that the correlation between NNAT and the achievement test results was 0.68, followed by the intelligence measures similar to NNAT (e.g., Cognitive Abilities Test, Otis–Lennon School Ability Tests; r = .31) and other alternative measures often used to identify gifted students (e.g., teacher-rating scale; r = .20). The moderator analysis results showed high correlations between NNAT and other measures when Naglieri is an author of the study. In Part II, although NNAT identified more students of color compared with other nonverbal tests (overall effect size of risk ratio was 0.42), findings revealed that students of color remain underrepresented in gifted programs and services.

Keywords

Naglieri Nonverbal Ability Test NNAT gifted identification validity check underrepresented students

Get full access to this article

View all access options for this article.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.

*Arrambide

T. O.

(2016). The validity of three instruments in identifying academically gifted Hispanic kindergarten English language learners (Order No. 10182942) [Doctoral dissertation, Sam Houston State University]. ProQuest Dissertations & Theses Global.

*Balboni

Naglieri

J. A.

Cubelli

(2010). Concurrent and predictive validity of the Raven Progressive Matrices and the Naglieri Nonverbal Ability Test. Journal of Psychoeducational Assessment, 28(3), 222-235. https://doi.org/10.1177/0734282909343763

Berkeljon

Baldwin

S. A.

(2014). An introduction to meta-analysis for psychotherapy outcome research. In Lutz

Knox

(Ed.), Quantitative and qualitative methods in psychotherapy research (pp. 221-234). Routledge. https://doi.org/10.4324/9780203386071-14

Borenstein

Hedges

L. V.

Higgins

J. P.

Rothstein

H. R.

(2010). A basic introduction to fixed-effect and random-effects models for meta-analysis. Research Synthesis Methods, 1(2), 97-111. https://doi.org/10.1002/jrsm.12

Borenstein

Hedges

L. V.

Higgins

J. P.

Rothstein

H. R.

(2011). Introduction to meta-analysis. John Wiley.

*Botella

Fürst

Myszkowski

Storme

Pereira

Costa

Luminet

(2015). French validation of the Overexcitability Questionnaire 2: Psychometric properties and factorial structure. Journal of Personality Assessment, 97(2), 209-220. https://doi.org/10.1080/00223891.2014.938750

*Bracken

B. A.

Brown

E. F.

(2008). Early identification of high-ability students: Clinical assessment of behavior. Journal for the Education of the Gifted, 31(4), 403-426. https://doi.org/10.4219/jeg-2008-794

Brown

Sherbenou

R. J.

Johnsen

S. K.

(1997). Test of nonverbal intelligence (3rd ed.). Pro_Ed.

10.

Brown

Sherbenou

R. J.

Johnsen

S. K.

(2010). Test of nonverbal intelligence (4th ed.). Pearson.

11.

Brown

R. P.

Day

E. A.

(2006). The difference isn’t black and white: Stereotype threat and the race gap on Raven’s advanced progressive matrices. Journal of Applied Psychology, 91(4), 979-985. https://doi.org/10.1037/0021-9010.91.4.979

12.

*Brulles

Peters

S. J.

Saunders

(2012). School wide mathematics achievement within the gifted cluster grouping model. Journal of Advanced Academics, 23(3), 200-216. https://doi.org/10.1177/1932202X12451439

13.

Canivez

G. L.

Rains

J. D.

(2002). Construct validity of the Adjustment Scales for Children and Adolescents and the Preschool and Kindergarten Behavior Scales: Convergent and divergent evidence. Psychology in the Schools, 39(6), 621-633. https://doi.org/10.1002/pits.10063

14.

*Carman

C. A.

Taylor

D. K.

(2010). Socioeconomic status effects on using the Naglieri Nonveral Ability Test (NNAT) to identify the gifted/talented. Gifted Child Quarterly, 54(2), 75-84. https://doi.org/10.1177/0016986209355976

15.

Carman

C. A.

Walther

C. A.

Bartsch

R. A.

(2018). Using the cognitive abilities test (CogAT) 7 nonverbal battery to identify the gifted/talented: An investigation of demographic effects and norming plans. Gifted Child Quarterly, 62(2), 193-209. https://doi.org/10.1177/0016986217752097

16.

Cizek

G. J.

(2016). Validating test score meaning and defending test score use: Different aims, different methods. Assessment in Education: Principles, Policy & Practice, 23(2), 212-225. https://doi.org/10.1080/0969594X.2015.1063479

17.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum.

18.

Cooper

(1998). Synthesizing research: A guide for literature reviews (3rd ed.). Sage.

19.

Cooper

(2010). Research synthesis and meta-analysis (4th ed.). Sage.

20.

Cooper

Hedges

L. V.

Valentine

J. C.

(Eds.). (2009). The handbook of research synthesis and meta-analysis. Russell Sage Foundation.

21.

Davis

G. A.

Rimm

S. B.

Siegle

D. B.

(2013). Education of the gifted and talented: Pearson new international edition. Pearson.

22.

DerSimonian

Kacker

(2007). Random-effects model for meta-analysis of clinical trials: An update. Contemporary Clinical Trials, 28(2), 105-114. https://doi.org/10.1016/j.cct.2006.04.004

23.

*Edmonds

M. M.

(2016). Identifying gifted minorities using nonverbal tests: The cognitive abilities test, Form 7, versus the Naglieri nonverbal ability test, second edition (Order No. 3730349) [Doctoral dissertation, Regent University]. ProQuest Dissertations & Theses Global.

24.

Ellis

P. D.

(2010). The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge University Press.

25.

*Esquierdo

J. J.

(2006). Early identification of Hispanic English language learners for gifted and talented programs (Order No. 3219154) [Doctoral dissertation, Texas A&M University]. ProQuest Dissertations & Theses Global.

26.

Fagerland

M. W.

(2015). Evidence-based medicine and systematic reviews. In Laake

Benestad

H. B.

Olsen

B. R.

(Eds.), Research in medical and biological sciences (pp. 431-461). Elsevier Ltd. https://doi.org/10.1016/B978-0-12-799943-2.00012-4

27.

Fernandes

Lynch

Jr. Netemeyer

(2012). A meta-analytic and psychometric investigation of the effect of financial literacy on downstream financial behaviors. Advances in Consumer Research, 40, 1052. https://www.acrwebsite.org/volumes/1012956/volumes/v40/NA-40

28.

Field

A. P.

(2001). Meta-analysis of correlation coefficients: A Monte Carlo comparison of fixed-and random-effects methods. Psychological Methods, 6(2), 161-180. https://doi.org/10.1037/1082-989X.6.2.161

29.

Fletcher

R. B.

Hattie

(2011). Intelligence and intelligence testing. Routledge.

30.

Ford

D. Y.

Wright

B. L.

Washington

Henfield

M. S.

(2016). Access and equity denied: Key theories for school psychologists to consider when assessing Black and Hispanic students for gifted education. School Psychology Forum: Research in Practice, 10(3), 265-277. https://www.researchgate.net/publication/312121099_Access_and_Equity_Denied_Key_Theories_for_School_Psychologists_to_Consider_When_Assessing_Black_and_Hispanic_Students_for_Gifted_Education

31.

Gentry

(2009). Myth 11: A comprehensive continuum of gifted education and talent development services. Gifted Child Quarterly, 53(4), 262-265. https://doi.org/10.1177/0016986209346937

32.

Gentry

Desmet

Chowkase

Lee

(2020, November 12-17). Intelligence tests to identify students with gifts and talents: Still perpetuating inequity [Conference presentation]. 66th Annual Convention of the National Association for Gifted Children, Orlando, FL.

33.

Gentry

Gray

Whiting

Maeda

Pereira

(2019). Gifted education in the United States: Laws, access, equity, and missingness across the country by locale, Title I school status, and race [Executive summary]. https://www.education.purdue.edu/geri/new-publications/gifted-education-in-the-united-states/

34.

Gerlsma

Emmelkamp

P. M.

Arrindell

W. A.

(1990). Anxiety, depression, and perception of early parenting: A meta-analysis. Clinical Psychology Review, 10(3), 251-277. https://doi.org/10.1016/0272-7358(90)90062-F

35.

*Giessman

J. A.

Gambrell

J. L.

Stebbins

M. S.

(2013). Minority performance on the Naglieri Nonverbal Ability Test, versus the Cognitive Abilities Test, Form 6: One gifted program’s experience. Gifted Child Quarterly, 57(2), 101-109. https://doi.org/10.1177/0016986213477190

36.

Higgins

Thompson

S. G.

(2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539-1558. https://doi.org/10.1002/sim.1186

37.

Hodges

Tay

Maeda

Gentry

(2018). A meta-analysis of gifted and talented identification practices. Gifted Child Quarterly, 62(2), 147-174. https://doi.org/10.1177/0016986217752107

38.

*Humble

Dixon

Schagen

(2018). Assessing intellectual potential in Tanzanian children in poor areas of Dar es Salaam. Assessment in Education: Principles, Policy & Practice, 25(4), 399-414. https://doi.org/10.1080/0969594X.2016.1194257

39.

Kaya

Stough

L. M.

Juntune

(2017). Verbal and nonverbal intelligence scores within the context of poverty. Gifted Education International, 33(3), 257-272. https://doi.org/10.1177/0261429416640332

40.

Lassiter

K. S.

Harrison

T. K.

Matthews

T. D.

Bell

N. L.

(2001). The validity of the comprehensive test of nonverbal intelligence as a measure of fluid intelligence. Assessment, 8(1), 95-103. https://doi.org/10.1177/107319110100800109

41.

Lau

Ioannidis

J. P.

Terrin

Schmid

C. H.

Olkin

(2006). The case of the misleading funnel plot. British Medical Journal, 333(7568), 597-600. https://doi.org/10.1136/bmj.333.7568.597

42.

*Lewis

J. D.

DeCamp-Fritson

S. S.

Ramage

J. C.

McFarland

M. A.

Archwamety

(2007). Selecting for ethnically diverse children who may be gifted using Raven’s Standard Progressive Matrices and Naglieri Nonverbal Abilities Test. Multicultural Education, 15(1), 38-42. https://files.eric.ed.gov/fulltext/EJ780591.pdf

43.

*Lindsey

C. D.

(2013). An investigation of the combined assessments used as entrance criteria for a gifted English middle school program (Order No. 3525542) [Doctoral dissertation, University of New Orleans]. ProQuest Dissertations & Theses Global.

44.

Lohman

D. F.

(2005). Review of Naglieri and Ford (2003): Does the Naglieri Nonverbal Ability Test identify equal proportions of high-scoring White, Black, and Hispanic students? Gifted Child Quarterly, 49(1), 19-28. https://doi.org/10.1177/001698620504900103

45.

Lohman

D. F

(2013). Cognitive Ability Test form 7: Planning and implementation guide. Riverside.

46.

Lohman

D. F.

Gambrell

J. L.

(2012). Using nonverbal tests to help identify academically talented children. Journal of Psychoeducational Assessment, 30(1), 25-44. https://doi.org/10.1177/001698620504900103

47.

*Lohman

D. F.

Korb

K. A.

Lakin

J. M.

(2008). Identifying academically gifted English-language learners using nonverbal tests: A comparison of the Raven, NNAT, and CogAT. Gifted Child Quarterly, 52(4), 275-296. https://doi.org/10.1177/0016986208321808

48.

*Mann

R. L.

(2005). The identification of gifted students with spatial strengths: An exploratory study (Order No. 3180228) [Doctoral dissertation, University of Connecticut]. ProQuest Dissertations & Theses Global.

49.

*Mann

E. L.

(2008). Parental perceptions of mathematical talent. Social Psychology of Education, 11(1), 43-57. https://doi.org/10.1007/s11218-007-9034-y

50.

*Martin

(1996). The performance of a multilingual South African sample on two measures of nonverbal assessment (Order No. 9630930) [Doctoral dissertation, The Ohio State University]. ProQuest Dissertations & Theses Global.

51.

Matthews

D. J.

(1988). Raven’s Matrices in the identification of giftedness. Roeper Review, 10(3), 159-162. https://doi.org/10.1080/02783198809553115

52.

McBee

M. T.

Makel

M. C.

(2019). The quantitative implications of definitions of giftedness. AERA Open, 5(1), 1-13. https://doi.org/10.1177/2332858419831007

53.

McBee

M. T.

Peters

S. J.

Miller

E. M.

(2016). The impact of the nomination stage on gifted program identification: A comprehensive psychometric analysis. Gifted Child Quarterly, 60(4), 258-278. https://doi.org/10.1177/0016986216656256

54.

McBee

M. T.

Peters

S. J.

Waterman

(2014). Combining scores in multiple-criteria assessment systems: The impact of combination rule. Gifted Child Quarterly, 58(1), 69-89. https://doi.org/10.1177/0016986213513794

55.

Mills

C. J.

Ablard

K. E.

Brody

L. E.

(1993). The Raven’s Progressive Matrices: Its usefulness for identifying gifted/talented students. Roeper Review, 15(3), 183-186. https://doi.org/10.1080/02783199309553500

56.

Naglieri

J. A.

(1997). Planning, attention, simultaneous, and successive theory and the Cognitive Assessment System: A new theory-based measure of intelligence. In Flanagan

D. P.

Genshaft

J. L.

Harrison

P. L.

(Eds.), Contemporary intellectual assessment: Theories, tests, and issues (pp. 247-267). The Guilford Press.

57.

*Naglieri

J. A.

(2003). Naglieri nonverbal ability test: Individual administration manual. Multi-Health Systems.

58.

*Naglieri

J. A.

(2011). Naglieri nonverbal ability test manual: Technical information and normative data (2nd ed.). NCS Pearson.

59.

*Naglieri

J. A.

(2018). Naglieri nonverbal ability test manual levels A-D (3rd ed.). Pearson.

60.

*Naglieri

J. A.

Booth

A. L.

Winsler

(2004). Comparison of Hispanic children with and without limited English proficiency on the Naglieri Nonverbal Ability Test. Psychological Assessment, 16(1), 81-84. https://doi.org/10.1037/1040-3590.16.1.81

61.

*Naglieri

J. A.

Ford

D. Y.

(2003). Addressing underrepresentation of gifted minority children using the Naglieri Nonverbal Ability Test (NNAT). Gifted Child Quarterly, 47(2), 155-160. https://doi.org/10.1177/001698620304700206

62.

Naglieri

J. A.

Ford

D. Y.

(2005). Increasing minority children’s participation in gifted classes using the NNAT: A response to Lohman. Gifted Child Quarterly, 49(1), 29-36. https://doi.org/10.1177/001698620504900104

63.

Naglieri

J. A.

Ford

D. Y.

(2015). Misconceptions about the Naglieri Nonverbal Ability Test: A commentary of concerns and disagreements. Roeper Review, 37(4), 234-240. https://doi.org/10.1080/02783193.2015.1077497

64.

*Naglieri

J. A.

Ronning

M. E.

(2000). The relationship between general ability using the Naglieri Nonverbal Ability Test (NNAT) and Stanford Achievement Test (SAT) reading achievement. Journal of Psychoeducational Assessment, 18(3), 230-239. https://doi.org/10.1177/073428290001800303

65.

National Association for Gifted Children. (n.d.). What is giftedness? Author. http://www.nagc.org/resources-publications/resources/what-giftedness

66.

National Association for Gifted Children. (2011). Identifying and serving culturally and linguistically diverse gifted students. Author. http://www.nagc.org/sites/default/files/Position%20Statement/Identifying%20and%20Serving%20Culturally%20and%20Linguistically.pdf

67.

National Association for Gifted Children. (2019). Position statement. Author. https://www.nagc.org/sites/default/files/Position%20Statement/Definition%20of%20Giftedness%20%282019%29.pdf

68.

Newton

P. E.

Shaw

S. D.

(2013). Standards for talking and thinking about validity. Psychological Methods, 18(3), 301-319. https://doi.org/10.1037/a0032969

69.

Otis

A. S.

Lennon

R. T.

(2003). Otis-Lennon school ability test (8th ed.). Pearson.

70.

Pearson. (n.d.). Stanford achievement test series (10th ed.). Author.

71.

Peters

S. J.

Gentry

(2012). Group-specific norms and teacher-rating scales: Implications for underrepresentation. Journal of Advanced Academics, 23(2), 125-144. https://doi.org/10.1177/1932202X12438717

72.

Pfeiffer

S. I.

Blei

(2008). Gifted identification beyond the IQ test: Rating scales and other assessment procedures. In Handbook of giftedness in children (pp. 177-198). Springer. https://doi.org/10.1007/978-0-387-74401-8

73.

Raven

J. C.

Court

J. H

. (1998). Manual for Raven’s progressive matrices and vocabulary scales. Oxford Psychologists Press.

74.

Raven

J. C.

Court

J. H

. (2003). Manual for Raven’s progressive matrices and vocabulary scales. Harcourt Assessment.

75.

*Rosado

J. I.

(2009). Validation of the Spanish version of the gifted rating scales (Order No. 3340757) [Doctoral dissertation, Florida State University]. ProQuest Dissertations & Theses Global.

76.

Rossen

E. A.

Shearer

D. K.

Penfield

R. D.

Kranzler

J. H.

(2005). Validity of the comprehensive test of nonverbal intelligence (CTONI). Journal of Psychoeducational Assessment, 23(2), 161-172. https://doi.org/10.1177/073428290502300205

77.

*Runyon

(2010). Identification of highly gifted 5- and 6-year-old children: Measures to predict academic achievement (Order No. 1485564) [Master’s Thesis, University of North Texas]. ProQuest Dissertations & Theses Global.

78.

Schmidt

F. L.

Hunter

J. E.

(1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62(5), 529-540. https://doi.org/10.1037/0021-9010.62.5.529

79.

Sim

Arnell

(1993). Measurement validity in physical therapy research. Physical Therapy, 73(2), 102-110. https://doi.org/10.1093/ptj/73.2.102

80.

Sterne

J. A.

Sutton

A. J.

Ioannidis

Terrin

Jones

D. R.

Lau

Higgins

(2011). Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. British Medical Journal, 343(7818), 302-307. https://doi.org/10.1136/bmj.d4002

81.

Sutton

A. J.

Abrams

K. R.

Jones

D. R.

Sheldon

T. A.

Song

(2000). Methods for meta-analysis in medical research. Wiley.

82.

The Psychological Corporation. (2002). Wechsler Individual Achievement Test (2nd ed.). Author.

83.

Torrance

P. E.

(2006). Torrance Tests of Creative Thinking. Scholastic Testing Service.

84.

VanTassel-Baska

(2007). Alternative assessments with gifted and talented students. Prufrock Press.

85.

Van Rooy

D. L.

Viswesvaran

. (2004). Emotional intelligence: A meta-analytic investigation of predictive validity and nomological net. Journal of vocational Behavior, 65(1), 71-95. https://doi.org/10.1016/S0001-8791(03)00076-9

86.

Wechsler

(2003). Wechsler Intelligence Scale for Children (4th ed.). The Psychological Corporation.

87.

Wechsler

Naglieri

J. A.

(2006). Wechsler nonverbal scale of ability. Pearson.

88.

*Wills

A. J.

(2013). A study of the relationship between kindergarten nonverbal ability and third-grade reading achievement (Order No. AAI3505945) [Doctoral dissertation, University of Missouri-St. Louis]. ProQuest Dissertations & Theses Global.

89.

*Worthington

J. J.

(2002). A comparison of gifted identification methods using measures of achievement, ability, multiple intelligences, and teacher nominations (Order No. 3048673) [Doctoral dissertation, University of San Francisco]. ProQuest Dissertations & Theses Global.

90.

*Yang

(2016). The influential factors of math achievement in mathematically promising English language learners (Order No. 3664191) [Doctoral dissertation, St. John’s University]. ProQuest Dissertations & Theses Global.

91.

Yoon

S. Y.

Gentry

(2009). Racial and ethnic representation in gifted programs: Current status of and implications for gifted Asian American students. Gifted Child Quarterly, 53(2), 121-136. https://doi.org/10.1177/0016986208330564