Ecologically valid indicators of executive functions are designed to capture dysfunction not easily measured in a lab setting. Here, we present two studies on the development and validity analyses of a behavioral screener for executive functions among young adults. In Study 1, we derived a four-factor (problem solving, attentional control, behavioral control, and emotional control) behavioral screener using a sample of 765 individuals. We used invariance analyses to evaluate the screener’s measurement reliability across sex. In Study 2, we replicated the screener derivation analyses using an independent sample of 197 undergraduates. To further examine the screener’s validity, we evaluated it against a well-known executive functions rating scale. The four-factor model was supported in both samples and analyses provided support for this screener as a valid and reliable measure for everyday executive functions among young adults.
AndersonV.AndersonP. J.JacobsR.Spencer SmithM. (2008). Development and assessment of executive function: From preschool to adolescence. In AndersonV.JacobsR.AndersonP. J. (Eds.), Executive functions and the frontal lobes: A lifespan perspective (pp. 123-154). New York, NY: Taylor & Francis.
2.
BaarsM. A. E.BijvankM. N.TonnaerG. H.JollesJ. (2015). Self-report measures of executive functioning are a determinant of academic performance in first-year students at a university of applied sciences. Frontiers in Psychology, 6, 1131. doi:10.3389/fpsyg.2015.01131
3.
BarkleyR. A. (2015). Assessment of executive functioning using the Barkley Deficits in Executive Functioning Scales. In NaglieriJ. A.GoldsteinS. (Eds.), Handbook of executive functioning (pp. 245-264). New York, NY: Springer.
4.
BlakemoreS. J.ChoudhuryS. (2006). Development of the adolescent brain: Implications for executive function and social cognition. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 47, 296-312. doi:10.1111/j.1469-7610.2006.01611.x
5.
BoelemaS. R.HarakehZ.OrmelJ.HartmanC. A.VolleberghW. A. M.van ZandvoortM. J. E. (2014). Executive functioning shows differential maturation from early to late adolescence: Longitudinal findings from a TRAILS study. Neuropsychology, 28, 177-187. doi:10.1037/neu0000049
6.
BryantF. B.KingS. P.SmartC. M. (2007). Multivariate statistical strategies for construct validation in positive psychology. In OngA. D.Van DulmenM. H. M. (Eds.), Oxford handbook of methods in positive psychology (pp. 61-82). New York, NY: Oxford University Press.
7.
BurgessP. W.AldermanN.ForbesC.CostelloA.CoatesL. M-A.DawsonD. R.. . . ChannonS. (2006). The case for the development and use of “ecologically valid” measures of executive function in experimental and clinical neuropsychology. Journal of the International Neuropsychological Society, 12, 194-209. doi:10.1017/S1355617706060310
8.
CampbellD. T.FiskeD. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105. doi:10.1037/h0046016
9.
CaseyB. J.TottenhamN.ListonC.DurstonS. (2005). Imaging the developing brain: What have we learned about cognitive development?Trends in Cognitive Sciences, 9, 104-110. doi:10.1016/j.tics.2005.01.011
10.
CheungG. W.RensvoldR. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255. doi:10.1207/S15328007SEM0902_5
11.
CrewsF.HeJ.HodgeC. (2007). Adolescent cortical development: A critical period of vulnerability for addiction. Pharmacology Biochemistry and Behavior, 86, 189-199. doi:10.1016/j.pbb.2006.12.001
12.
CronbachL. J.MeehlP. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302. doi:10.1037/h0040957
13.
DelisD. (2012). Delis-Rating of Executive Function (D-REF). Bloomington, MN: Pearson.
14.
DevineR. T.HughesC. (2014). Relations between false belief understanding and executive function in early childhood: A meta-analysis. Child Development, 85, 1777-1794. doi:10.1111/cdev.12237
15.
DiamondA. (2013). Executive function. Annual Review of Psychology, 64, 135-168.
16.
DugganE. C.Garcia-BarreraM. A. (2015). Intelligence and executive functioning. In NaglieriJ. A.GoldsteinS. (Eds.), Handbook of intelligence (pp. 435-458). New York, NY: Springer. doi:10.1007/978-1-4939-1562-0_27
17.
EgelandJ.FallmyrØ. (2010). Confirmatory Factor Analysis of the Behavior Rating Inventory of Executive Function (BRIEF): Support for a distinction between Emotional and Behavioral Regulation. Child Neuropsychology: A Journal on Normal and Abnormal Development in Childhood and Adolescence, 16, 326-337. doi:10.1080/09297041003601462
18.
FournetN.RoulinJ.-L.MonnierC.AtzeniT.CosnefroyO.Le GallD.RoyA. (2014). Multigroup confirmatory factor analysis and structural invariance with age of the Behavior Rating Inventory of Executive Function (BRIEF)-French version. Child Neuropsychology: A Journal on Normal and Abnormal Development in Childhood and Adolescence, 21, 379-398. doi:10.1080/09297049.2014.906569
19.
FrazerJ. (2012). The integrative neuropsychological theory of executive-related abilities and component transactions (INTERACT): A novel validation study (Unpublished doctoral dissertation). University of Victoria, Victoria, British Colombia, Canada.
20.
FuglestadA. J.WhitelyM. L.CarisonS. M.BoysC. J.EckerleJ. K.FinkB. A.WozniakR. (2015). Executive functioning deficits in preschool children with Fetal Alcohol Spectrum Disorders. Child Neuropsychology, 21, 716-731. doi:10.1080/09297049.2014.933792
21.
FurrR. M.BacharachV. R. (2014). Psychometrics: An introduction (2nd ed.). Thousand Oaks, CA: Sage.
Garcia-BarreraM. A.KamphausR. W.BandalosD. (2011). Theoretical and statistical derivation of a screener for the behavioral assessment of executive functions in children. Psychological Assessment, 23, 64-79. doi:10.1037/a0021097
24.
Garcia-BarreraM. A.KarrJ. E.DuranV.DirenfeldE.PinedaD. A. (2015). Cross-cultural validation of a behavioral screener for executive functions: Guidelines for clinical use among Colombian children with and without ADHD. Psychological Assessment, 27, 1349-1363. doi:10.1037/pas0000117
25.
Garcia-BarreraM. A.KarrJ. E.KamphausR. W. (2013). Longitudinal applications of a behavioral screener of executive functioning: Assessing factorial invariance and exploring latent growth. Psychological Assessment, 25, 1300-1313. doi:10.1037/a0034046
26.
GioiaG. A.IsquithP. K. (2004). Ecological assessment of executive function in traumatic brain injury. Developmental Neuropsychology, 25, 135-158. doi:10.1080/87565641.2004.9651925
GioiaG. A.IsquithP. K.KenealyL. E. (2008). Assessment of behavioral aspects of executive function. In AndersonV.JacobsR.AndersonP. J. (Eds.), Executive functions and the frontal lobes: A lifespan perspective (pp. 179-202). New York, NY: Taylor & Francis.
29.
GioiaG.KenworthyI.IsquithP. K. (2010). Executive function in the real world: BRIEF lesions from Mark Ylvisaker. Journal of Head Trauma Rehabilitation, 25, 433-439. doi:10.1097/HTR.0b013e3181fbc272
30.
GoldsteinS.NaglieriJ. A.PrinciottaD.OteroT. M. (2015). Introduction: A history of executive functioning as a theoretical and clinical construct. In NaglieriJ. A.GoldsteinS. (Eds.), Handbook of executive functioning (pp. 3-12). New York, NY: Springer. doi:10.1007/978-1-4614-8106-5_1
31.
GraceJ.MalloyP. F. (2001). Frontal Systems Behavior Scale: Professional manual. Lutz, FL: Psychological Assessment Resources.
32.
HairJ. F.BlackB.BabinB.AndersonR. E. (2010). Multivariate data analysis. Upper Saddle River, NJ: Prentice Hall.
33.
HersenM.BeidelD. C. (Eds.). (2012). Adult psychopathology and diagnosis (6th ed.). Hoboken, NJ: Wiley.
34.
HoyleR. H.SmithG. T. (1994). Formulating clinical research hypotheses as structural equation models: A conceptual overview. Journal of Consulting and Clinical Psychology, 6, 429-440.
35.
HuL.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. doi:10.1080/10705519909540118
36.
HuizingaM.DolanC. V.van der MolenM. W. (2006). Age-related change in executive function: Developmental trends and a latent variable analysis. Neuropsychologia, 44, 2017-2036. doi:10.1016/j.neuropsychologia.2006.01.010
37.
IsquithP. K.RothR. M.GioiaG. (2013). Contribution of rating scales to the assessment of executive functions. Applied Neuropsychology: Child, 2, 1-8. doi:10.1080/21622965.2013.748389
38.
JacksonD. L.GillaspyJ. A.Purc-StephensonR. (2009). Reporting practices in confirmatory factor analysis: An overview and some recommendations. Psychological Methods, 14, 6-23. doi:10.1037/a0014694
39.
KarrJ. E.Garcia-BarreraM. A.KernsK. A.MuellerU.BaronI. S.LitmanF. (2013, April). Derivation of a behavioral executive functioning screener in preschoolers using a latent variable approach. Paper presented at the biennial meeting of the Society for Research in Child Development, Seattle, WA.
40.
KerteszA.DavidsonW.FoxH. (1997). Frontal Behavior Inventory: Diagnostic criteria for the frontal lobe dementia. Canadian Journal of Neurological Science, 24, 29-36.
LeeK.BullR.HoR. M. (2013). Developmental changes in executive functioning. Child Development, 84, 1933-1953. doi:10.1111/cdev.12096
43.
LehtoJ. E.JuujärviP.KooistraL.PulkkinenL. (2003). Dimensions of executive functioning: Evidence from children. British Journal of Developmental Psychology, 21, 59-80. doi:10.1348/026151003321164627
44.
LezakM. D.HowiesonD. B.BiglerE. D.TranelD. (2012). Neuropsychological assessment (5th ed.). New York, NY: Oxford University Press.
45.
ManchesterD.PriestleyN.JacksonH. (2004). The assessment of executive functions: Coming out of the office. Brain Injury, 18, 1067-1081. doi:10.1080/02699050410001672387
46.
MarshH. W.BallaJ. R.McDonaldR. P. (1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103, 391-410. doi:10.1037/0033-2909.103.3.391
47.
McAuleyT.ChenS.GoosL.SchacharR.CrosbieJ. (2010). Is the Behavior Rating Inventory of Executive Function more strongly associated with measures of impairment or executive function?Journal of the International Neuropsychological Society, 16, 495-505. doi:10.1017/S1355617710000093
48.
MeltzerL.KrishnanK. (2007). Executive function difficulties and learning disabilities: Understandings and misunderstandings. In MelzterL. (Ed.), Executive function in education: From theory to practice (pp. 77-105). New York, NY: Guilford Press.
49.
MillerE. K.CohenJ. D. (2001). An integrative theory of prefrontal cortex function. Annual Review of Neuroscience, 24, 167-202. doi:10.1146/annurev.neuro.24.1.167
50.
MiyakeA.FriedmanN. P. (2012). The nature and organization of individual differences in executive functions: Four general conclusions. Current Directions in Psychological Science, 21, 8-14. doi:10.1177/0963721411429458
51.
MiyakeA.FriedmanN. P.EmersonM. J.WitzkiA. H.HowerterA.WagerT. D. (2000). The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognitive Psychology, 41(1), 49-100. doi:10.1006/cogp.1999.0734
52.
MüllerU.KernsK. (2015). The development of executive function. In LibenL. S.MüllerU.LernerR. M. (Eds.), Handbook of child psychology and developmental science, Vol. 2: Cognitive processes (7th ed., pp. 571-623). Hoboken, NJ: Wiley.
53.
MuthénL. K.MuthénB. O. (1998-2012). Mplus user’s guide (7th ed.). Los Angeles, CA: Author.
54.
NaglieriJ. A.GoldsteinS. (2015a). Assessment of executive function using rating scales: Psychometric considerations. In NaglieriJ. A.GoldsteinS. (Eds.), Handbook of executive functioning (pp. 159-171). New York, NY: Springer.
55.
NaglieriJ. A.GoldsteinS. (Eds.). (2015b). Handbook of executive functioning. New York, NY: Springer.
56.
OteroT. M.BarkerL. A. (2015). The frontal lobes and executive functioning. In NaglieriJ. A.GoldsteinS. (Eds.), Handbook of executive functioning (pp. 29-44). New York, NY: Springer.
57.
PelicanoE. (2010). The development of core cognitive skills in autism: A 3-year prospective study. Child Development, 81, 1400-1416.
58.
PenningtonB. F.OzonoffS. (1996). Executive functions and developmental psychopathology. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 37, 51-87. doi:10.1111/j.1469-7610.1996.tb01380.x
59.
ReynoldsC. R.KamphausR. W. (1992). Behavior Assessment System for Children (BASC). Circle Pines, MN: AGS.
60.
ReynoldsC. R.KamphausR. W. (2004). Behavior Assessment System for Children (2nd ed.). Bloomington, MN: Pearson. doi:10.1002/9780470479216.corpsy0114
61.
ReynoldsC. R.KamphausR. W. (2015). Behavior Assessment System for Children (3rd ed.). Bloomington, MN: Pearson.
62.
ReynoldsC. R.MacNeill HortonA. (2008). Assessing executive functions: A life-span perspective. Psychology in the Schools, 45, 875-892. doi:10.1002/pits.20332
63.
RoalfD. R.GurR. E.RuparelK.CalkinsM. E.SatterthwaiteT. D.. . . GurR. C. (2014). Within-individual variability in neurocognitive performance: Age- and sex-related differences in children and youths from ages 8 to 21. Neuropsychology, 28, 506-518. doi:10.1037/neu0000067
64.
RomineC. B.ReynoldsC. R. (2005). A model of the development of frontal lobe functioning: Findings from a meta-analysis. Applied Neuropsychology, 12, 190-201. doi:10.1207/s15324826an1204_2
65.
RothR. M.IsquithP. K.GioiaG. A. (2005). Behavior Rating Inventory of Executive Function–Adult version (BRIEF-A). Lutz, FL: Psychological Assessment Resources.
66.
RothR. M.IsquithP. K.GioiaG. A. (2015). Assessment of executive functioning using the Behavior Rating Inventory of Executive Function (BRIEF). In NaglieriJ. A.GoldsteinS. (Eds.), Handbook of executive functioning (pp. 301-332). New York, NY: Springer.
67.
RothR. M.LanceC. E.IsquithP. K.FischerA. S.GriancolaP. R. (2013). Confirmatory factor analysis of the Behavior Rating Inventory of Executive Function-Adult Version in healthy adults and application to attention-deficit/hyperactivity disorder. Archives of Clinical Neuropsychology, 28, 425-434. doi:10.1093/arclin/act031
68.
SadehS. S.BurnsM. K.SullivanA. L. (2012). Examining an executive function rating scale as a predictor of achievement in children at risk for behavior problems. School Psychology Quarterly, 27, 236-246. doi:10.1037/spq0000012
69.
Schermelleh-EngelK.MoosbruggerH.MüllerH. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8, 23-74.
70.
SchoemakerK.MulderH.DevovićM.MatthysW. (2013). Executive functions in preschool children with externalizing behavior problems: A meta-analysis. Journal of Abnormal Child Psychology, 41, 457-471.
71.
ShalliceT.BurgessP. W. (1991). Deficits in strategy application following frontal lobe damage in man. Brain, 114, 727-741.
72.
SilverC. H. (2014). Sources of data about children’s executive functioning: Review and commentary. Child Neuropsychology: A Journal on Normal and Abnormal Development in Childhood and Adolescence, 20, 1-13. doi:10.1080/09297049.2012.727793
73.
StraussE.ShermanE. M. S.SpreenO. (2006). A compendium of neuropsychological tests (3rd ed.). New York, NY: Oxford University Press.
74.
TaylorS. J.BarkerL. A.HeaveyL.McHaleS. (2013). Typical developmental trajectory of social and executive functions in late adolescence and early adulthood. Developmental Psychology, 49, 1253-1265.
75.
ToplakM. E.WestR. F.StanovichK. E. (2013). Practitioner review: Do performance-based measures and ratings of executive function assess the same construct?Journal of Child Psychology and Psychiatry, and Allied Disciplines, 54, 131-143. doi:10.1111/jcpp.12001
76.
TorralvaT.RocaM.GleichgerrchtE.BekinschteinT.ManesF. (2009). A neuropsychological battery to detect specific executive and social cognitive impairments in early frontotemporal dementia. Brain, 132, 1299-1309. doi:10.1093/brain/awp041
77.
WilloughbyM. T.BlairC. B.PekJ.The Family Life Project Investigators. (2013). Measuring executive function in early childhood: A focus on maximal reliability and the derivation of short forms. Psychological Assessment, 25, 664-670. doi:10.1037/a0031747
78.
WilloughbyM. T.WirthR. J.BlairC. B.The Family Life Project Investigators. (2012). Executive function in early childhood: Longitudinal measurement invariance and developmental change. Psychological Assessment, 24, 418-431. doi:10.1037/a0025779
79.
WilsonB. A.AldermanN.BurgessP. W.EmslieH.EvansJ. J. (1996). Behavioural Assessment of the Dysexecutive Syndrome (BADS). Suffolk, England: Thames Valley Test.
80.
YuC. Y. (2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continues outcomes (Doctoral dissertation). University of California, Los Angeles.
81.
ZelazoP. D.CarlsonS. M. (2012). Hot and cool executive function in childhood and adolescence: Development and plasticity. Child Development Perspectives, 6, 354-360. doi:10.1111/j.1750-8606.2012.00246.x
