Planning is an essential cognitive process of executive functions and is considered as one of the most important brain functions. Planning has been extensively studied in the field of neuropsychology, but there is a lack of computerized assessment tools of planning ability that are easily accessible to researchers and clinicians. The present study aims to validate a newly developed online spatial network test that is designed for both clinical and nonclinical populations. Exploratory factor analysis revealed a unidimensional factorial structure with moderate to high internal consistency in the test. Two-parameter logistic item response theory analysis showed acceptable item and model fit and no violation of the local independence assumption. The overall success rate exhibited a positive correlation with the performance of planning before attempting to solve the items. After correcting for attenuation, moderate to high correlations were found between the spatial network test and the International Cognitive Ability Resource 16 short form cognitive ability test and the automated perceptual maze test, respectively, demonstrating convergent, discriminant, and criterion validity. Future directions and potential applications of the spatial network test as an assessment tool to measure planning for researchers and clinicians are discussed in the end.
AllainP.NicoleauS.PinonK.Etcharry-BouyxF.BarréJ.BerrutG.. . . Le GallD. (2005). Executive functioning in normal aging: A study of action planning using the Zoo Map Test. Brain and Cognition, 57, 4-7.
2.
BanichM. T. (2009). Executive function the search for an integrated account. Current Directions in Psychological Science, 18, 89-94.
3.
BennettD. A.WilsonR. S.SchneiderJ. A.EvansD. A.BeckettL. A.AggarwalN. T.. . . BachJ. (2002). Natural history of mild cognitive impairment in older persons. Neurology, 59, 198-205.
4.
BentonA. L.ElithornA.FogelM. L.KerrM. (1963). A perceptual maze test sensitive to brain damage. Journal of Neurology, Neurosurgery & Psychiatry, 26, 540-544.
5.
BiggsN.LloydE. K.WilsonR. J. (1976). Graph theory, 1736-1936. Oxford, England: Oxford University Press.
6.
BlairC. (2006). How similar are fluid cognition and general intelligence? A developmental neuroscience perspective on fluid cognition as an aspect of human cognitive ability. Behavioral and Brain Sciences, 29, 109-125.
7.
BrennanM.WelshM. C.FisherC. B. (1997). Aging and executive function skills: An examination of a community-dwelling older adult population. Perceptual Motor Skills, 83, 1187-1191.
8.
BurtonL. J.MazerolleS. M. (2011). Survey instrument validity Part I: Principles of survey instrument development and validation in athletic training education research. Athletic Training Education Journal, 6, 27-35.
9.
CaiL.HansenM. (2013). Limited-information goodness-of-fit testing of hierarchical item factor models. British Journal of Mathematical and Statistical Psychology, 66, 245-276.
10.
CampbellD. T.FiskeD. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
11.
CellaD.GershonR.LaiJ. S.ChoiS. (2007). The future of outcomes measurement: Item banking, tailored short-forms, and computerized adaptive assessment. Quality of Life Research, 16, 133-141.
12.
ChalmersR. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1-29.
CondonD. M.RevelleW. (2014). The International Cognitive Ability Resource: Development and initial validation of a public-domain measure. Intelligence, 43, 52-64.
15.
CronbachL. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.
16.
Cuberos-UrbanoG.CaracuelA.Vilar-LilarR.Valls-SerranoC.BatemanA.Verdejo-GarcA. (2013). Ecological validity of the Multiple Errands Test using predictive models of dysexecutive problems in everyday life. Journal of Clinical and Experimental Neuropsychology, 35, 329-336.
17.
CuretonE. E.D’AgostinoR. B. (1983). Factor analysis: An applied approach. Hillsdale, NJ: Lawrence Erlbaum.
18.
CuttingL. E.MaterekA.ColeC. A.LevineT. M.MahoneE. M. (2009). Effects of fluency, oral language, and executive function on reading comprehension performance. Annals of Dyslexia, 59, 34-54.
19.
DaviesA. D. (1965). The perceptual maze test in a normal population. Perceptual and Motor Skills, 20, 287-293.
20.
DziubanC. D.ShirkeyE. C. (1974). When is a correlation matrix appropriate for factor analysis? Some decision rules. Psychological Bulletin, 81, 358-361.
21.
ElithornA. (1955). A preliminary report on a perceptual maze test sensitive to brain damage. Journal of Neurology, Neurosurgery & Psychiatry, 18, 287-292.
22.
EmbretsonS. E.ReiseS. P. (2013). Item response theory. New York, NY: Psychology Press.
23.
FordJ. K.MacCallumR. C.TaitM. (1986). The application of exploratory factor analysis in applied psychology: A critical review and analysis. Personnel Psychology, 39, 291-314.
24.
FriedmanS. L.ScholnickE. K.CockingR. R. (Eds.). (1987). Blueprints for thinking: The role of planning in cognitive development. Cambridge, England: Cambridge University Press.
25.
GierlM. J.HaladynaT. M. (2012). Automatic item generation: Theory and practice. New York, NY: Routledge.
26.
GierlM. J.LaiH.TurnerS. R. (2012). Using automatic item generation to create multiple-choice test items. Medical Education, 46, 757-765.
27.
GrafmanJ. (1989). Plans, actions and mental sets: Managerial knowledge units in the frontal lobes. In PerecmanE. (Ed.), Integrating theory and practice in clinical neuropsychology (pp. 93-138). Hillsdale, NJ: Lawrence Erlbaum.
28.
HansonB. A.BéguinA. A. (2002). Obtaining a common scale for item response theory item parameters using separate versus concurrent estimation in the common-item equating design. Applied Psychological Measurement, 26, 3-24.
29.
HarmanH. H.JonesW. H. (1966). Factor analysis by minimizing residuals (minres). Psychometrika, 31, 351-368.
30.
Hayes-RothB.Hayes-RothF. (1979). A cognitive model of planning. Cognitive Science, 3, 245-262.
31.
Holgado-TelloF. P.Chacón-MoscosoS.Barbero-GarcíaI.Vila-AbadE. (2010). Polychoric versus Pearson correlations in exploratory and confirmatory factor analysis of ordinal variables. Quality & Quantity, 44, 153. doi:10.1007/s11135-008-9190-y
32.
HollandP. W.WainerH. (1993). Differential item functioning. New York, NY: Psychology Press.
33.
HornJ. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179-185.
34.
HuL. T.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
35.
HumphreysL. G.MontanelliR. G.Jr. (1975). An investigation of the parallel analysis criterion for determining the number of common factors. Multivariate Behavioral Research, 10, 193-205.
36.
KaferK. L.HunterM. (1997). On testing the face validity of planning/problem-solving tasks in a normal population. Journal of the International Neuropsychological Society, 3, 108-119.
37.
KaiserH. F. (1970). A second generation little jiffy. Psychometrika, 35, 401-415.
38.
KaiserH. F. (1974). An index of factorial simplicity. Psychometrika, 39, 31-36.
39.
KaufmanA. S.KaufmanN. L. (2004). Kaufman Assessment Battery for Children–II (KABC-II). Circle Pines, MN: American Guidance Service.
40.
KirkD. B. (1973). On the numerical approximation of the bivariate normal (tetrachoric) correlation coefficient. Psychometrika, 38, 259-268.
41.
LautenschlagerG. J. (1989). A comparison of alternatives to conducting Monte Carlo analyses for determining parallel analysis criteria. Multivariate Behavioral Research, 24, 365-395.
42.
LedesmaR. D.Valero-MoraP. (2007). Determining the number of factors to retain in EFA: An easy-to-use computer program for carrying out parallel analysis. Practical Assessment, Research & Evaluation, 12(2), 1-11.
LoeB. S.RustJ. (2019). The Perceptual Maze Test revisited: Evaluating the difficulty of automatically generated mazes. Assessment, 26, 1524-1539. doi:10.1177/1073191117746501
45.
LuriaA. R. (1976). Cognitive development: Its cultural and social foundations. Cambridge, MA: Harvard university press.
MallowsC. L.OdlyzkoA. M.SloaneN. J. A. (1975). Upper bounds for modular forms, lattices, and codes. Journal of Algebra, 36, 68-76.
48.
Maydeu-OlivaresA.JoeH. (2006). Limited information goodness-of-fit testing in multidimensional contingency tables. Psychometrika, 71, 713. doi:10.1007/s11336-005-1295-9
49.
Maydeu-OlivaresA.JoeH. (2014). Assessing approximate fit in categorical data analysis. Multivariate Behavioral Research, 49, 305-328.
50.
MosherF. A.HornsbyJ. R. (1966). On asking questions. In BrunerJ. S.OlverR. R.GreenfieldP. M.HornsbyJ. R.KenneyH. J.MaccobyM.Sonstroem. . . A. M. (Eds.), Studies in cognitive growth (pp. 86-102). New York, NY: John Wiley.
51.
NaglieriJ. A.DasJ. P. (1997). Cognitive assessment system. Itasca, IL: Riverside.
52.
NorrisG.TateR. L. (2000). The Behavioural Assessment of the Dysexecutive Syndrome (BADS): Ecological, concurrent and construct validity. Neuropsychological Rehabilitation, 10, 33-45.
53.
OrlandoM.ThissenD. (2000). Likelihood-based item-fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24, 50-64.
54.
OrlandoM.ThissenD. (2003). Further investigation of the performance of S − X2: An item fit index for use with dichotomous item response theory models. Applied Psychological Measurement, 27, 289-298.
55.
OwenA. (1997). Cognitive planning in humans: Neuropsychological, neuroanatomical and neuropharmacological perspectives. Progress in Neurobiology, 53, 431-450.
56.
PhillipsL. H.KliegelM.MartinM. (2006). Age and planning tasks: The influence of ecological validity. International Journal of Aging & Human Development, 62, 175-184.
ReckaseM. (2009). Multidimensional item response theory. New York, NY: Springer.
60.
ReiseS. P. (1990). A comparison of item-and person-fit methods of assessing model-data fit in IRT. Applied Psychological Measurement, 14, 127-137.
61.
ReiseS. P.WallerN. G.ComreyA. L. (2000). Factor analysis and scale revision. Psychological Assessment, 12, 287-297.
62.
RevelleW. (2017). Psych: Procedures for personality and psychological research. Evanston, IL: Northwestern University.
63.
ReyA. (1941). L’examen psychologique dans les cas d’enctphalopathie traumatique [The psychological examination of cases of traumatic encephalopathy]. Archives de Psychologie, 28, 286-340.
64.
RustJ.GolombokS. (2014). Modern psychometrics: The science of psychological assessment. New York, NY: Routledge.
65.
ShalliceT. (1982). Specific impairments of planning. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 298, 199-209.
66.
ShalliceT. I. M.BurgessP. W. (1991). Deficits in strategy application following frontal lobe damage in man. Brain, 114, 727-741.
67.
TellegenA.WallerN. G. (2008). Exploring personality through test construction: Development of the Multidimensional Personality Questionnaire. In BoyleG. J.MatthewsG.SaklofskeD. H. (Eds.), The SAGE handbook of personality theory and assessment (Vol. 2, pp. 261-292). Thousand Oaks, CA: Sage.
68.
The International Cognitive Ability Resource Team. (2014). The project. Retrieved from http://icar-project.com/
69.
TsutakawaR. K.JohnsonJ. C. (1990). The effect of uncertainty of item parameter estimation on ability estimates. Psychometrika, 55, 371-390.
70.
van der LindenW. J.HambletonR. K. (2013). Handbook of modern item response theory. New York, NY: Springer Science.
71.
WainerH. (2000). Computerized adaptive testing. New York, NY: John Wiley.
72.
WardG.MorrisR. (2005). Introduction to the psychology of planning. In MorrisR.WardG. (Eds.), The cognitive psychology of planning (pp. 1-34). Hove, England: Psychology Press.
73.
WickhamH. (2016). ggplot2: Elegant graphics for data analysis. New York, NY: Springer.
74.
WilliamsB.OnsmanA.BrownT. (2010). Exploratory factor analysis: A five-step guide for novices. Australasian Journal of Paramedicine, 8, 93. doi:10.33151/ajp.8.3.93
75.
WilsonB. A.AldermanN.BurgessP.EmslieH.EvansJ. (1996). Behavioural assessment of the dysexecutive syndrome. Bury St. Edmunds, Suffolk, England: Thames Valley Test Company.
76.
YenW. M. (1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of Educational Measurement, 30, 187-213.
77.
ZwickW. R.VelicerW. F. (1986). Factors influencing five rules for determining the number of components to retain. Psychological Bulletin, 99, 432-442.
