Sage Journals: Discover world-class research

Abstract

Psychoeducational assessments provide clinicians with the foundation for diagnostic determinations and meaningful recommendations for youth with neurodevelopmental disorders. Due to the COVID-19 pandemic, the field shifted from seeing clients face-to-face (FTF) to conducting evaluations virtually via teleneuropsychology (TNP) in the client’s own home. The current study sought to determine the equivalence of cognitive and achievement scores obtained in the context of FTF versus TNP testing. Nineteen youth with diverse neurodevelopmental disorders completed the Woodcock Johnson-Fourth Edition (WJ-IV). Results suggest generally equivalent results, with the caveat that scores from measures of Fluid Reasoning and Auditory Processing were significantly different across testing modalities. Environmental factors, characteristics of the clients, administration challenges, technical considerations, and practice effects may have contributed to differences in scores. These findings have implications for assessment standardization, case conceptualization, and implementation of recommendations in the school setting when psychoeducational assessments are conducted virtually.

Keywords

youth neurodevelopmental disorder teleneuropsychology assessment

Introduction

A psychoeducational assessment is a comprehensive investigation of a student’s cognitive, academic, and psychological/behavioral functioning. Across academic settings, psychoeducational assessments are indispensable tools to identify areas of strength and need, inform diagnostic decisions, and determine appropriate interventions, accommodations, and supports. In Canada, there is a growing demand for psychoeducational assessments of youth with, or suspected of, neurodevelopmental disabilities, such as intellectual disability, autism spectrum disorder (ASD), attention-deficit/ hyperactivity disorder (ADHD), and learning disabilities. Yet, financial and geographical constraints, long wait times, and feasibility are consistent barriers to accessing psychoeducational assessment services (Bauer et al., 2012; Gander & Campbell, 2018; Harder et al., 2020; Ontario Psychological Association & Canadian Academy of Psychologists in Disability Assessment, 2020, p. 4; Salinas et al., 2020).

Psychological services have greatly benefited from advancements in technology. With careful consideration of accessibility and confidentiality, telehealth has flourished as an effective means to provide health care, from consultation to intervention, through electronic communication, such as video and telephone calls (Madigan et al., 2021). Telehealth encompasses diverse care provision, including clinicians seeking consultation from allied professionals, communicating and providing services to clients, and monitoring the well-being of clients (Wosik et al., 2020). More specifically, teleneuropsychology (TNP) is the practice of administering virtual versions and adaptations of neuropsychological batteries via audiovisual technology to aid diagnostic determinations and treatment planning of mental disorders (Bilder et al., 2020). Foundational practice recommendations for TNP were outlined by Grosch et al. (2011), and TNP literature has expanded since then. Evidence among adults suggests that TNP is a feasible, reliable, and valid option when compared to in-person testing (e.g., Brearly et al., 2017; Cullum et al., 2014). Although sparser, literature with pediatric populations highlights TNP as a feasible, reliable, and valid substitution or compliment to in-person testing (Ransom et al., 2020; Ruffini et al., 2022). Further, TNP assessment has received positive feedback from patients and caregivers (Parsons et al., 2022).

Practically, there are numerous advantages to virtual testing with youth. Indeed, TNP provides options to alleviate common barriers (Salinas et al., 2020). First, TNP assessments allow clinicians to target youth located in geographical regions exceedingly distant from assessment centers and communities lacking trained mental health professionals. Likewise, TNP assessment allows culturally and linguistically competent clinicians to extend the reach of their services. TNP can also reduce financial constraints and stress experienced by parents of assessed youth, including alleviating requirements for transportation, expenses associated with traveling, and time off work.

The use of virtual/remote testing by psychological practitioners has drastically increased in recent years. Specifically, in March 2020, the novel coronavirus (i.e., COVID-19) imposed physical distancing guidelines in Canada that continued to wax and wane over the course of the pandemic (Canadian Institute for Health Information, 2022). During this period, many provincial licensing and professional practice groups quickly moved to develop and distribute best practice guidelines on the administration of remote assessments that adhered to physical distancing guidelines (see the joint statement released by OPA and CAPDA in August 2020). Accordingly, many clinicians transitioned to providing TNP services, with the client completing testing within their own home without assistance from a trained technician.

The state of the literature on TNP assessment with youth, while promising, did not fully capture the type of evaluations being conducted with these distancing requirements in place. For example, Hodge et al. (2019) compared scores from the Wechsler Intelligence Scale for Children-Fifth Edition (WISC-V) administered via TNP and in-person and found a high-level of agreement among a sample of children with a Specific Learning Disorder with impairment in reading. Additionally, Wright (2018) did not find significant differences between scores from FTF administration of the Woodcock-Johnson—Fourth Edition (WJ-IV) and scores from virtual administration of the same assessment battery to school-aged children. However, the majority of TNP studies, including Hodge et al. (2019) and Wright’s (2018) work, have taken place in a satellite clinic and/or utilized a technician/proctor to configure equipment, manipulate test stimuli, and maintain a distraction-reduced environment (see Ruffini et al., 2022 for a review). Therefore, definitive conclusions about the applicability of TNP to assessments conducted without these environmental and technical supports cannot be drawn due to notable limitations in the existing literature.

To date, only one pediatric study has compared TNP to FTF testing without added support from a technician/proctor and in the client’s home (see Harder et al., 2020); however, this research investigated a very specific population, youth (10–17 years) with demyelinating disorders. In that study, Harder et al. (2020) found that cognitive and academic testing did not differ between FTF and TNP administration across a diverse neuropsychological battery, including comparable measures to the current study (e.g., Wechsler Intelligence Scale for Children-Fifth Edition; Wechsler Adult Intelligence Scale-Fourth Edition; Woodcock-Johnson-Third Edition Tests of Achievement). With similar methodology, there is a clear need to expand knowledge about whether TNP is comparable to FTF testing for a population commonly in need of psychoeducational assessments, namely, youth with diagnosed or suspected neurodevelopmental disorders.

The objective of this study was to determine the consistency of results obtained from FTF and TNP assessment using the Woodcock-Johnson-Fourth Edition (WJ-IV) for youth with neurodevelopmental disorders. Further, this study incorporated behavioral observations obtained during virtual testing to identify common challenges and concerns when conducting a TNP assessment. In the context of high demand for TNP and limited research, this exploratory study aimed to provide preliminary information on the utility of TNP assessment with a popular psychoeducation battery, the WJ-IV, conducted in the youth’s own home without benefit of a technician. This research aims to inform realistic recommendations for practitioners considering or currently engaging in this form of practice.

Method

Recruitment and Participants

This project was approved by the General Research Ethics Board (GREB) prior to data collection. Participants were recruited via convenience sampling from a transition program for grade seven and eight students who were identified as requiring support transitioning from elementary to secondary education. This is an innovative transition support program available to students already identified as having learning differences in the educational environment (who were already receiving accommodations and supports as outlined in an Individual Education Plan). Over a 2-year period (grades seven and eight), students and their families are provided with psychoeducation to understand the differences in expectations for students between the elementary and secondary systems, to learn more about the student’s unique learning needs and strengths, and to develop the self-esteem and self-advocacy skills required to access required supports. Students typically undergo an updated psychoeducational assessment as part of this program, though some evaluations were delayed in the cohorts included in this study due to the COVID-pandemic. Participants who had undergone a FTF assessment before the onset of the COVID-pandemic or who were waiting for a FTF assessment were invited to participate in a TNP assessment of approximately 3 hrs in length. Although counterbalancing was attempted, only four of the 25 students who accepted our invitation and completed TNP assessment were from the group who had undergone FTF testing first. Further, two participants who completed the TNP assessment first did not follow up to complete the FTF assessment. As such, a decision was made to focus the analyses on the 19 students who completed a TNP assessment first and FTF assessment second. Participants were compensated with a $25 gift card for completing the TNP assessment and their parents were provided with a summary of the assessment findings.

Participants ranged in age from 12 to 13 years and were diagnosed with, or suspected of having, a neurodevelopmental disorder(s) (i.e., learning disabilities, ADHD, and/or ASD). A majority of participants (n = 17) had undergone prior testing through their school board, community psychologist, or primary care physician and received a diagnosis of a neurodevelopmental disorder before entering the transition program. A minority (n = 2) of students were suspected of having a neurodevelopmental disorder(s) as indicated by their IEP and referral to the transition program; following their completion of the FTF assessment, appropriate diagnoses were communicated. Additionally, one participant was classified as having a mild intellectual disability (MID), an educational identification recognized by the Ontario Ministry of Education to capture students who require support in accessing the curriculum but do not meet criteria for a formal diagnosis of an intellectual disability. Comparable to the well-documented prevalence rates of neurodevelopmental disorders during childhood (Francés et al., 2022), there were a greater number of males than females in the current study. All students spoke English as a primary language and were of European descent. See Table 1 for a comprehensive review of sample demographics.

Table 1.

Demographic Characteristics.

Sample characteristics	M (SD)	Total (n)	%
Age	12.31 (0.48)
12		13	68.42
13		6	31.58
Gender
Male		12	63.16
Female		7	36.84
Neurodevelopmental disorder
LD		8	42.11
ADHD		4	21.05
ASD		1	5.26
MID		1	5.26
ADHD and LD		3	15.79
ASD and ADHD		2	10.53
Time interval between assessment conditions	8.55 (2.31)	19

Note. Time interval between assessment conditions is reported in months. LD = learning disability; ADHD = attention deficit/hyperactivity disorder; ASD = autism spectrum disorder; MID = mild intellectual disability.

Procedure

TNP assessment was completed by the participant via the participant’s own computer at home, with the assessor located remotely, in their own private office. Prior to the assessment, all interested participants (and their caregivers) were contacted over the phone by an examiner to review study information, including the purpose and nature of the assessment, and obtain informed consent. Consenting participants agreed over the phone to the need for appropriate technology, a quiet workspace with reduced distractions, and to have a caregiver or responsible adult accessible (somewhere in the house) if required. Participants were also reminded about this information on the date of their assessment. Consistent with typical psychoeducational assessments, participants were instructed that they were not permitted assistive devices, like calculators, and reminded to apply their best effort. Required response forms provided by the publishers, were either mailed ahead to participants or dropped off at their homes in a sealed envelope, and participants were provided with a postage-paid envelope to return their completed response forms to the examiner. During the TNP assessment, the examiner used the screen share function on Zoom to display the testing materials and participants were required to leave their camera on. The subsequent FTF assessment was completed in our clinic by either a licensed psychologist or supervised psychometrist under standard testing conditions, including one-on-one administration, a distraction-free environment, and the use of standardized assessment materials. Given COVID-19 guidelines, however, precautions, such as wearing masks and using a plastic barrier between the participant and examiner, were exercised. TNP assessments were administered during late 2020 (i.e., November to December) and early 2021 (i.e., January to April) and FTF assessments were completed in the latter half of 2021 (i.e., July to December) and 2022 (i.e., January to June). Clinical observations were reported as per protocol for each assessment condition. Due to participant’s availability, responsiveness, and in-person limitations of the pandemic, time between TNP and FTF testing ranged from approximately 7 months to nearly 17 months, with an average of approximately 8 months, 16 days between testing modalities (see Table 1).

Measures

Socio-Demographic Information

As part of the transition program referral package, caregivers provided demographic information regarding their youth, including, but not limited to, age, gender identity, and diagnostic history.

Psychoeducation Assessment

Comprehensive cognitive and academic testing was administered using measures from the Woodcock-Johnson- Fourth Edition (WJ-IV; Schrank et al., 2014), with age-level norms, and supplemented with the use of performance validity tests to assess participant’s effort and motivation. Specifically, the Memory Validity Profile (MVP; Sherman & Brooks, 2015) assessed performance validity during TNP assessment and the Medical Symptom Validity Test (MSVT; Green, 2004) assessed performance validity for FTF testing.

Statistical Analyses

Inductive content analysis was performed with qualitative data to determine themes among assessors’ behavioral observations during TNP testing (Elo & Kyngäs, 2008). To identify themes, two independent coders consulted until agreement was reached. This information was used to provide a more qualitative context for the interpretation of quantitative scores, as well as to help identity common challenges when conducting TNP with this population. For quantitative analysis, means and standard deviations (SD) were reported for each index of the WJ-IV. Intraclass correlation coefficients (ICCs) were computed to assess the agreement (i.e., test-retest reliability) between the two assessment conditions, FTF and TNP. ICC guidelines suggest coefficient values less than .5 indicate poor reliability, between .5 and .75 indicate moderate reliability, between .75 and .9 indicate good reliability, and values greater than .9 indicate excellent reliability (Koo & Li, 2016). Of primary interest to the current study, paired samples t-tests compared the means of WJ-IV index scores on FTF and TNP assessments to determine statistically significant differences among testing conditions. All analyses were conducted with participants’ standard scores on the WJ-IV. Of note, selection of assessment subtests was determined by the psychologist to suit the unique nature of the participant’s referral question. Thus, not all 19 participants completed the same WJ-IV-Tests of Achievement and analyses were performed with the largest possible n for each index. To test the relationship between time between TNP and FTF testing, correlations were examined; no associations were identified and therefore, results are reported without adjusting for varying time between tests. Given the multiple t-tests conducted, the authors also considered correcting for Type 1 error. However, as this study was experimental, it was felt that Type 1 error was less problematic than the risk of a Type II error. As such, no correction was made.

Results

All participants obtained passing scores on both the MVP and MSVT, suggesting that students likely put forth sufficient effort, despite differences in testing modality. However, the examiners’ behavioral observations highlighted common issues when administering TNP assessments with this population that may threaten the utility of in-home virtual testing and consistency of results. Examiners reported on distractions in the environment (e.g., loud background noises), personal factors (e.g., anxiety about being “on camera”), workspace and administration issues (e.g., lack of appropriate workspace, participant not adhering to directions to stop working on timed tasks), and technological challenges (e.g., issues with sound quality). See Table 2 for a description of behavioral observations noted by examiners.

Table 2.

Themes Identified From Qualitative Analysis During TNP Testing.

Theme	Essence	Behavioral observation
Environmental	Reflects environmental factors that provided distractions during administration.	Loud background noises
		Dog entered participant’s room
		Interruption(s) from people in the home environment
Personal factor	Reflects participant’s unique characteristics that imposed challenges during testing and due to the nature of virtual assessment, assessor was not able to provide redirection or assistance.	Anxiety about being “on camera” and therefore, camera was tilted to view only hands
		Fatigue
		Fidgeting
		Humming
		Distractibility
		Not wearing glasses as needed
Workspace/administration	Reflects challenges associated with administering the WJ-IV and therefore, interfered with standardization of the tasks.	Lack of appropriate workspace
		Difficulty knowing if the participant used tools, such as paper-and-pencil/calculator
		Participant(s) did not adhere to directions to stop working on timed tasks
		Used a calculator and other devices for assistance on tasks
Technical	Reflects technical difficulties encountered by the participants and/ or assessor during testing.	Poor sound quality
Technical		Parent sat next to participant during testing

Note. WJ-IV = Woodcock-Johnson-Fourth Edition.

According to Koo and Li’s (2016) guidelines and despite the concerns raised above, results suggested 12 of the 14 WJ-IV indexes demonstrated good agreement across testing conditions, ICCs ≥ 0.76. The remaining two WJ-IV indexes demonstrated moderate agreement across testing conditions, ICCs ≥ 0.50 (see Table 3). Therefore, test-retest reliability for in-person and virtual in-home testing of the WJ-IV was acceptable in the current study.

Table 3.

ICC Results for FTF and TNP Testing Modalities by WJ-IV Index.

	ICC	95% CI	n
General intellectual ability	0.96	[0.89, 0.98]	19
Comprehension knowledge	0.96	[0.49, 0.99]	19
Fluid reasoning	0.96	[0.90, 0.98]	19
Short-term working memory	0.92	[0.31, 0.98]	19
Cognitive processing speed	0.83	[0.59, 0.93]	19
Auditory processing	0.74	[0.33, 0.90]	19
Long-term retrieval	0.66	[0.09, 0.87]	19
Visual processing	0.90	[0.75, 0.96]	19
Cognitive efficiency	0.71	[0.27, 0.89]	19
Broad reading	0.51	[–0.32, 0.81]	18
Basic reading skills	0.96	[0.87, 0.98]	18
Mathematics	0.94	[0.83, 0.97]	14
Academic skills	0.96	[0.87, 0.99]	14

Note. ICC = intraclass correlation coefficients.

Further exploration of the data using paired samples t-tests generally failed to identify significant differences as a function of administration modality, ps ≥ .057 (see Table 4), except for two areas. Standard scores for the Fluid Reasoning index were significantly lower in the TNP condition (M = 93.05, SD = 19.14) compared to the FTF condition (M = 101.63, SD = 20.78), t(18) = 5.10, p < .001. Standard scores for the Auditory Processing index were also significantly lower in the TNP condition (M = 78.47, SD = 13.93) compared to the FTF condition (M = 87.16, SD = 13.10), t(18) = 3.07, p = .007. The individual subtests subsumed in the Fluid Reasoning and Auditory Processing Indexes were next explored. Within the Fluid Reasoning index, performance on both Number Series and Concept Formation subtests significantly differed as a function of condition; Number Series (M = 94.53, SD = 15.81) and Concept Formation (M = 92.95, SD = 18.91) TNP scores were significantly lower than the same subtest’s (M = 98.84, SD = 16.53; M = 103.00, SD = 20.02) FTF scores, t(18) = 4.59, p = < .001. Related to the Auditory Processing index, TNP standard scores on the Nonword Repetition subtest (M = 84.53, SD = 9.88) were significantly lower than those from FTF assessment (M = 92.84, SD = 12.68), t(18) = 2.67, p = .016. Further analysis was conducted to determine whether interval length between testing sessions could be confounding the results, but no association was found.

Table 4.

Paired Samples T-Tests.

WJ-IV Index	Virtual	In-Person	n	T	p	Cohen’s d
WJ-IV Index	M (SD)	M (SD)	n	T	p	Cohen’s d
General intellectual ability	89.73 (14.64)	91.47 (15.46)	19	1.23	.236	0.281
Comprehension knowledge	94.37 (12.64)	94.95 (11.86)	19	0.51	.615	0.117
Fluid reasoning	93.05 (19.14)	101.63 (20.78)	19	5.1	<.001	1.169
Number series	94.53 (15.81)	98.84 (16.53)	19	2.51	.022	0.577
Concept formation	92.95 (18.91)	103.00 (20.02)	19	4.59	<.001	1.053
Short-term working memory	91.63 (13.84)	89.00 (15.91)	19	–1.01	.326	–0.232
Cognitive processing speed	88.79 (11.85)	89.95 (12.04)	19	0.46	.649	0.106
Auditory processing	78.47 (13.93)	87.16 (13.10)	19	3.07	.007	0.703
Phonological processing	79.58 (18.77)	84.63 (12.66)	19	1.92	.071	0.441
Nonword repetition	84.53 (9.88)	92.84 (12.68)	19	2.67	.016	0.612
Long-term retrieval	98.89 (14.33)	97.53 (11.37)	19	–0.77	.45	–0.177
Visual processing	97.52 (10.38)	101.00 (13.84)	19	1.32	.204	0.303
Cognitive efficiency	88.32 (12.06)	86.78 (11.40)	19	–0.49	.631	–0.112
Broad reading	87.39 (14.12)	90.11 (15.11)	18	2.04	.057	0.481
Basic reading skills	86.94 (10.92)	89.06 (10.31)	18	1.87	.079	0.44
Mathematics	83.71 (15.50)	82.78 (16.30)	14	–0.53	.605	–0.142
Academic skills	80.71 (11.46)	80.71 (11.25)	14	0	1	0

Note. WJ-IV = Woodcock-Johnson-Fourth Edition; M = mean standard score; SD = standard deviation.

Discussion

Due to the COVID-19 pandemic’s physical distancing precautions, assessment services shifted rapidly to TNP in a client’s home and without the benefit of a technician. In this context, this exploratory study attempted to determine the reliability of results from FTF and TNP assessment using a common psychoeducational measure, the Woodcock-Johnson-Fourth Edition (WJ-IV), for youth with neurodevelopmental disorders. Behavioral observations yielded unique challenges including environmental, personal factors, workspace/ administration, and technical difficulties. Similar to Wright (2018)’s initial findings with a general school sample, quantitative results suggested equivalence of results for most cognitive and academic measures administered. However, in the current study, findings revealed lower scores on TNP measures of Fluid Reasoning (i.e., Number Series, Concept Formation) and Auditory Processing (i.e., Nonword Repetition) compared to the same measures delivered via FTF. Thus, consistent with previous literature (e.g., Harder et al., 2020), TNP in the client’s home appears to be a reliable procedure with the caveat that virtual testing may impose unique standardization challenges and scores on measures of Fluid Reasoning and Auditory Processing may differ based on modality of administration in youth with neurodevelopmental disorders.

The encouraging accessibility and flexibility of virtual psychoeducational assessments must be considered within the standardization challenges of TNP (Farmer et al., 2020; Hewitt et al., 2020). First, the client’s home can hinder a distraction-reduced environment and interfere with their ability to provide undivided attention during virtual testing due to the presence of family members and lack of confidential spaces (Sumpter et al., 2023). As evidenced in the current study, clinicians lack the ability to create a controlled and private setting when practicing TNP; family members and/or pets may enter the client’s testing setting, unexpected loud noises cannot be prevented, and generally, participants may lack adequate space to participate in virtual in-home testing.

Second, personal factors and their effect on performance must be considered. Although participants passed validity tests during TNP in the current study, notable experiences such as reported anxiety about being “on camera,” may have impacted motivation to perform to the best of their ability. Indeed, Hewitt et al. (2020) suggested performance anxiety associated with household members overhearing responses can impact testing. We also speculate that the lack of an in-person assessor closely watching participant’s testing and providing encouragement may have affected performance and motivation. In fact, providing encouragement and redirection in the face of resistance to testing for young people is common practice (Bracken & Nagle, 2017), yet difficult to provide during TNP due to the virtual separation. Consequently, typical encouragement practices during challenging tasks, such as Fluid Reasoning, may have differed based on environment and may have contributed to the discrepancy in scores obtained.

Workspace and/or administrative challenges also occurred during TNP testing for the current study. Assessors were not consistently able to visually confirm the tools participants used for testing (e.g., paper-and-pencil/calculator) and strictly enforce directions to stop working on timed tasks. These factors together could have inflated scores as the participant(s)’ testing practices were no longer comparable to conditions in which the test was standardized.

Finally, the use of technology for testing and its related impact on scores is pivotal to consider. Of note, assessments are limited to youth with access to appropriate technology, such as a computer or tablet, and sufficient internet bandwidth. Even when technologies are available for the delivery of test stimuli, visual and auditory output may be altered, and internet connectivity can waver. Thus, participants’ lower scores on Auditory Processing may be best explained by lack of audio clarity as this is a frequently cited challenge in the context of TNP among adults (Fox-Fuller et al., 2022) and, indeed, observed in the current study. Likewise, TNP puts demands on clients to access higher-order executive functions to independently set up technology and maintain the functionality of materials. Thus, Fluid Reasoning scores in the TNP condition may be more sensitive to participants diagnosed with ADHD and/or ASD as executive function impairment is commonly associated with these disorders (Craig et al., 2016). Moreover, severity of impairment and scope of independence related to diagnoses is crucial to consider when interpreting lower Fluid Reasoning scores.

Practice effects were also postulated as a possible reason for the difference of scores on Fluid Reasoning and Auditory Processing during FTF versus TNP testing. In this study, participants completed TNP first and FTF second and thus, participants may have improved their FTF Fluid Reasoning and Auditory Processing scores as they were previously exposed to the same testing material. However, previous research on test/re-test intervals ranging between 1 day and a few weeks identified practice effects for spatial and processing speed tasks but not measures of reasoning (Salthouse & Tucker-Drob, 2008). Likewise, the average testing interval between TNP and FTF in the current study exceeded publisher’s recommendations for time intervals between cognitive and achievement testing (Riverside Assessments, 2022). As such, while there may have been some contribution of practice effects on scores obtained, it is unlikely that this was the main determinant of score differentials.

Regardless of the cause for lower TNP scores compared to FTF scores on measures of Fluid Reasoning and Auditory Processing, until further research is conducted to clarify the underlying mechanisms, clinicians may wish to include additional FTF assessment to promote reliable measurement of skills and uphold standards of psychometric assessment. Clinicians can opt to supplement TNP assessment of fluid reasoning with additional FTF subtests from batteries such as the Wechsler Intelligence Scales for Children-Fifth Edition (WISC-V; Wechsler, 2014), Raven’s 2 Progressive Matrices Clinical Edition (Raven et al., 2018) and/or the Test of Nonverbal Intelligence Fourth Edition (TONI-IV; Brown et al., 2010). Additionally, assessment of Auditory Processing may be supplemented with FTF administration of subtests from the Comprehensive Test of Phonological Processing- Second Edition (CTOPP-2; Wagner et al., 2013), Test of Auditory Processing Skills- Fourth Edition (TAPS-4; Martin et al., 2018), and the NEPSY-II (Korkman et al., 2007). Without the ability to supplement with FTF testing, examiners may wish to interpret results from measures of Fluid Reasoning and Auditory Processing with some caution, with the understanding that results from TNP assessment in these areas may underestimate the client’s actual abilities.

Strengths, Limitations, and Future Directions

Delivering TNP to youth in an ecologically valid environment is a strength of the current study. Apart from Harder et al. (2020), TNP research has not compared in-person psychoeducational assessments to virtual psychoeducational assessments, in-home, without support from technicians. Next, the characteristics of participants can be viewed as a strength and limitation as youth with neurodevelopmental disorders are underrepresented in the literature. Prior TNP research with varying methodological procedures have primarily examined youth with language difficulties (e.g., Salinas et al., 2020), adult populations (e.g., Brearly et al., 2017), and individuals with highly specific conditions (e.g., youth with hearing loss; Werfel et al., 2021). However, the generalizability of the current study to populations outside of the narrow age group (i.e., 12–13 years old) and youth with neurodevelopmental disorders warrants caution. Further, a G*Power (Faul et al., 2007) sensitivity analysis, with 80% power and alpha at .05 (two-tailed), suggested the paired samples t-tests could not reliably detect effect sizes smaller than Cohen’s d = 0.68. Repeating this study with larger sample sizes will allow future researchers to increase the likelihood of detecting true effects between FTF and TNP conditions and promote the capacity for subgroup analyses among neurodevelopmental disorders. A final limitation of the current study involves the unique period of time in history, as assessments were administered during the global pandemic. During this period, youth were subjected to fluctuating restrictions, which may have contributed to varying levels of stress. In addition, COVID-19 assessment precautions, including masking, plexiglass barrier, and physical distancing, used during the FTF testing, may have impacted participant’s performance. Yet, the presence of an external variable that may impact a student’s learning and development is not uncommon to psychoeducational assessments. In fact, considering the dynamic interactions among young person’s unique contexts, such as COVID-19, when interpreting cognitive and academic data is integral to diagnostic assessment procedures (Stifel et al., 2020).

Future work should continue to examine the validity of TNP assessments with diverse clinical populations varying in ages and diagnoses as intersectional identities may interact to render TNP delivery more suitable for some populations over others. Likewise, future research must evaluate TNP with typically developing children to establish expectations for a wider range of pediatric populations. Forthcoming research should also replicate current findings and examine underlying mechanisms, like technology limitations and executive functioning abilities, to explain the differences between FTF and TNP standard scores on Fluid Reasoning and Auditory Processing subtests. Additionally, the field of TNP would benefit from comparing TNP and FTF results from other commonly and uncommonly administered measures of cognition and achievement with consistently timed intervals between conditions. Although attempted in this study but abandoned due to uneven sample sizes, use of a fully counterbalanced design would be ideal to reduce concerns regarding the possible impact of practice effects in research to come. Finally, future research procedures may benefit from standardized parental involvement protocols to mitigate disruptive behaviors and environments. Taken together, increasingly comprehensive studies will enhance knowledge regarding the utility and reliability of TNP for youth requiring assessment of clinical diagnoses in the convenience of their home.

Relevance to the Practice of School Psychology

Consistent with the scientist-practitioner model, school psychology benefits from integration of up-to-date scientific knowledge to inform efficacious practice (Shaw, 2021). Indeed, findings address questions raised regarding effective delivery of psychoeducational assessments in the education setting (Farmer et al., 2021) such that TNP assessment may be an appropriate tool for school psychologists on a case-by-case basis. Current results highlighted common challenges experienced when administering TNP assessment with youth with neurodevelopmental disorders, such as environmental factors, behavioral characteristics, administration difficulties, and technical considerations, which can be taken into consideration when determining the appropriateness of TNP for a specific client. Additionally, this research supports the general reliability of results obtained from TNP assessment to inform educational plans and accommodations for youth with this population. Further, results provide some guidance on areas where TNP may underestimate scores, providing suggestions on areas where additional FTF assessment may be warranted to supplement results. Taken together, informed decision-making in the delivery and application of TNP assessments will enhance assessment delivery as well as the student’s long-term capacity to access the education curriculum.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Beth Pollock

Allyson G. Harrison

Author Biographies

Julia Moreau is a second-year doctoral student in the Clinical Psychology program at Queen’s University. Under the supervision of local Clinical Psychologists, she provides psychoeducational assessments and therapeutic interventions for school-aged children in the surrounding area.

Beth Pollock is the Clinical Director of the Regional Assessment Resource Centre at Queen’s University. As a practicing Neuropsychologist, her work focusses on the assessment of children, youth and adults with neurodevelopmental disorders.

Allyson G. Harrison holds an appointment as an assistant professor in the Department of Clinical Psychology at Queen’s University in Kingston, Ontario. Her research is dedicated to issues of differential diagnosis in LD and ADHD.

References

Bauer

R. M.

Iverson

G. L.

Cernich

A. N.

Binder

L. M.

Ruff

R. M.

Naugle

R. I.

(2012). Computerized neuropsychological assessment devices: Joint position paper of the American Academy of Clinical Neuropsychology and the National Academy of Neuropsychology. The Clinical Neuropsychologist, 26(2), 177–196. https://doi.org/10.1080/13854046.2012.663001

Bilder

R. M.

Postal

K. S.

Barisa

Aase

D. M.

Cullum

C. M.

Gillaspy

S. R.

Harder

Kanter

Lanca

Lechuga

D. M.

Morgan

J. M.

Most

Puente

A. E.

Salinas

C. M.

Woodhouse

(2020). Inter organizational practice committee recommendations/guidance for teleneuropsychology in response to the COVID-19 pandemic. Archives of Clinical Neuropsychology, 35, 647–659. https://doi.org/10.1093/arclin/acaa046

Bracken

A. B.

Nagle

(2017). Clinical observation of preschool assessment behavior. In Bracken

A. B.

(Ed.), Psychoeducational assessment of preschool children (4th ed., pp. 95–110). Routledge.

Brearly

T. W.

Shura

R. D.

Martindale

S. L.

Lazowski

R. A.

Luxton

D. D.

Shenal

B. V.

Rowland

J. A.

(2017). Neuropsychological test administration by videoconference: A systematic review and meta-analysis. Neuropsychology Review, 27(2), 174–186. https://doi.org/10.1007/s11065-017-9349-1

Brown

Sherbenou

R. J.

Johnsen

S. K.

(2010). Test of nonverbal intelligence (4th ed.). Pro-Ed.

Canadian Institute for Health Information. (2022). COVID-19 intervention timeline in Canada. https://www.cihi.ca/en/covid-19-intervention-timeline-in-canada

Craig

Margari

Legrottaglie

A. R.

Palumbi

de Giambattista

Margari

(2016). A review of executive function deficits in autism spectrum disorder and attention-deficit/hyperactivity disorder. Neuropsychiatric Disease and Treatment, 12, 1191–1202. https://doi.org/10.2147/NDT.S104620

Cullum

C. M.

Hynan

L. S.

Grosch

Parikh

Weiner

M. F.

(2014). Teleneuropsychology: Evidence for video teleconference-based neuropsychological assessment. Journal of the International Neuropsychological Society, 20(10), 1028–1033. https://doi.org/10.1017/S1355617714000873

Elo

Kyngäs

(2008). The qualitative content analysis process. Journal of Advanced Nursing, 62(1), 107–115. https://doi.org/10.1111/j.1365-2648.2007.04569.x

10.

Farmer

R. L.

McGill

R. J.

Dombrowski

S. C.

Benson

N. F.

Smith-Kellen

Lockwood

A. B.

Powell

Pynn

Stinnett

T. A.

(2021). Conducting psychoeducational assessments during the COVID-19 crisis: The danger of good intentions. Contemporary School Psychology, 25(1), 27–32. https://doi.org/10.1007/s40688-020-00293-x

11.

Farmer

R. L.

McGill

R. J.

Dombrowski

S. C.

McClain

M. B.

Harris

Lockwood

A. B.

Powell

S. L.

Pynn

Smith-Kellen

Loethen

Benson

N. F.

Stinnett

T. A.

(2020). Teleassessment with children and adolescents during the coronavirus (COVID-19) pandemic and beyond: Practice and policy implications. Professional Psychology: Research and Practice, 51(5), 477–487. https://doi.org/10.1037/pro0000349

12.

Faul

Erdfelder

Lang

A. G.

Buchner

(2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/bf03193146

13.

Fox-Fuller

J. T.

Rizer

Andersen

S. L.

Sunderaraman

(2022). Survey findings about the experiences, challenges, and practical advice/solutions regarding teleneuropsychological assessment in adults. Archives of Clinical Neuropsychology, 37(2), 274–291. https://doi.org/10.1093/arclin/acab076

14.

Francés

Quintero

Fernández

Ruiz

Caules

Fillon

Hervás

Soler

C. V.

(2022). Current state of knowledge on the prevalence of neurodevelopmental disorders in childhood according to the DSM-5: A systematic review in accordance with the PRISMA criteria. Child and Adolescent Psychiatry and Mental Health, 16(1), 1–15. https://doi.org/10.1186/s13034-022-00462-1

15.

Gander

Campbell

(2018). Examining access to psychoeducational assessments by socioeconomic factors for children with behavioral referrals. Paediatrics and Child Health, 23(Suppl 1), e57. https://doi.org/10.1093/pch/pxy054.143

16.

Green

(2004). Green’s Medical Symptom Validity Test (MSVT) for Microsoft windows user’s manual. Green’s Publishing.

17.

Grosch

M. C.

Gottlieb

M. C.

Cullum

C. M.

(2011). Initial practice recommendations for teleneuropsychology. The Clinical Neuropsychologist, 25(7), 1119–1133. https://doi.org/10.1080/13854046.2011.609840

18.

Harder

Hernandez

Hague

Neumann

McCreary

Cullum

C. M.

Greenberg

(2020). Home-based pediatric teleneuropsychology: A validation study. Archives of Clinical Neuropsychology, 35(8), 1266–1275. https://doi.org/10.1093/arclin/acaa070

19.

Hewitt

K. C.

Rodgin

Loring

D. W.

Pritchard

A. E.

Jacobson

L. A.

(2020). Transitioning to telehealth neuropsychology service: Considerations across adult and pediatric care settings. The Clinical Neuropsychologist, 34(7–8), 1335–1351. https://doi.org/10.1080/13854046.2020.1811891

20.

Hodge

M. A.

Sutherland

Jeng

Bale

Batta

Cambridge

Detheridge

Drevensek

Edwards

Everett

Ganesalingam

Geier

Kass

Mathieson

McCabe

Micallef

Molomby

Ong

Pfeiffer

Silove

(2019). Agreement between telehealth and face-to-face assessment of intellectual ability in children with specific learning disorder. Journal of Telemedicine and Telecare, 25(7), 431–437. https://doi.org/10.1177/1357633X18776095

21.

Koo

T. K.

M. Y.

(2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012

22.

Korkman

Kirk

Kemp

(2007). Nepsy (2nd ed.). Pearson.

23.

Madigan

Racine

Cooke

J. E.

Korczak

D. J.

(2021). COVID-19 and telemental health: Benefits, challenges, and future directions. Canadian Psychology, 62(1), 5–11. https://doi.org/10.1037/cap0000259

24.

Martin

Brownell

Hamaguchi

(2018). Test of audition processing skills (4th ed.). Pro-Ed.

25.

Ontario Psychological Association & Canadian Academy of Psychologists in Disability Assessment. (2020). Guidelines for best practices in psychological remote/tele assessments version 1. https://www.psych.on.ca/getmedia/8d9c5cf3-e4f1-470b-8a51-3d603dbbc219/OPACAPDA-RemoteTele-Assessment-V9-revised.pdf

26.

Parsons

M. W.

Gardner

M. M.

Sherman

J. C.

Pasquariello

Grieco

J. A.

Kay

C. D.

Pollak

L. E.

Morgan

A. K.

Carlson-Emerton

Seligsohn

Davidsdottir

Pulsifer

M. B.

Zarrella

G. V.

Burstein

S. M.

Mancuso

S. M.

(2022). Feasibility and acceptance of direct-to-home tele-neuropsychology services during the COVID-19 pandemic. Journal of the International Neuropsychological Society, 28(2), 210–215. https://doi.org/10.1017/S1355617721000436

27.

Ransom

D. M.

Butt

S. M.

DiVirgilio

E. K.

Cederberg

C. D.

Srnka

K. D.

Hess

C. T.

M. C.

Katzenstein

J. M.

(2020). Pediatric teleneuropsychology: Feasibility and recommendations. Archives of Clinical Neuropsychology, 35(8), 1204–1214. https://doi.org/10.1093/arclin/acaa103

28.

Raven

Rust

Chan

Zhou

(2018). Raven’s 2 progressive matrices, clinical edition (Raven’s 2). Pearson.

29.

Riverside Assessments. (2022, June). WJ IV: How long before I can re-administer as assessment? Riverside Insights. https://support.riversideinsights.com/support/solutions/articles/70000585699-wj-iv-how-long-before-i-can-re-administer-an-assessment

30.

Ruffini

Tarchi

Morini

Giuliano

Pecini

(2022). Tele-assessment of cognitive functions in children: A systematic review. Child Neuropsychology, 28(6), 709–745. https://doi.org/10.1080/09297049.2021.2005011

31.

Salinas

C. M.

Bordes Edgar

Berrios Siervo

Bender

H. A.

(2020). Transforming pediatric neuropsychology through video-based teleneuropsychology: An innovative private practice model pre-COVID-19. Archives of Clinical Neuropsychology, 35(8), 1189–1195. https://doi.org/10.1093/arclin/acaa101

32.

Salthouse

T. A.

Tucker-Drob

E. M.

(2008). Implications of short-term retest effects for the interpretation of longitudinal change. Neuropsychology, 22(6), 800–811. https://doi.org/10.1037/a0013091

33.

Schrank

F. A.

McGrew

K. S.

Mather

(2014). Woodcock-Johnson IV. Riverside.

34.

Shaw

S. R.

(2021). Implementing evidence-based practices in school psychology: Excavation by de-implementing the disproved. Canadian Journal of School Psychology, 36(2), 91–97. https://doi.org/10.1177/08295735211000513

35.

Sherman

E. M. S.

Brooks

B. L.

(2015). Memory Validity Profile (MVP) professional manual. Psychological Assessment Resources Inc.

36.

Stifel

S. W. F.

Feinberg

D. K.

Zhang

Chan

M. K.

Wagle

(2020). Assessment during the COVID-19 pandemic: Ethical, legal, and safety considerations moving forward. School Psychology Review, 49(4), 438–452. https://doi.org/10.1080/2372966x.2020.1844549

37.

Sumpter

Camsey

Meldrum

Alford

Campbell

Bois

O’Connell

Flood

(2023). Remote neuropsychological assessment: Acceptability and feasibility of direct-to-home teleneuropsychology methodology during the COVID-19 pandemic. The Clinical Neuropsychologist, 37, 432–447. https://doi.org/10.1080/13854046.2022.2056922

38.

Wagner

R. K.

Torgesen

J. K.

Rashotte

C. A.

Pearson

N. A.

(2013). Comprehensive test of phonological processing–2nd ed. (CTOPP-2). Pro-Ed.

39.

Wechsler

(2014). Wechsler Intelligence Scale for Children (5th ed.). NCS Pearson, Incorporated.

40.

Werfel

K. L.

Grey

Johnson

Brooks

Cooper

Reynolds

Deutchki

Vachio

Lund

E. A.

(2021). Transitioning speech-language assessment to a virtual environment: Lessons learned from the ELLA study. Language, Speech, and Hearing Services in Schools, 52(3), 769–775. https://doi.org/10.1044/2021_LSHSS-20-00149

41.

Wosik

Fudim

Cameron

Gellad

Z. F.

Cho

Phinney

Curtis

Roman

Poon

E. G.

Ferranti

Katz

J. N.

Tcheng

(2020). Telehealth transformation: COVID-19 and the rise of virtual care. Journal of the American Medical Informatics Association, 27(6), 957–962. https://doi.org/10.1093/jamia/ocaa067

42.

Wright

A. J.

(2018). Equivalence of remote, online administration and traditional, face-to-face administration of Woodcock-Johnson IV cognitive and achievement tests. Archives of Assessment Psychology, 8(1), 23–35.

In-Person and In-Home Teleneuropsychological Assessments With Youth With Neurodevelopmental Disorders: What’s the Difference?

Abstract

Keywords

Introduction

Method

Recruitment and Participants

Procedure

Measures

Socio-Demographic Information

Psychoeducation Assessment

Statistical Analyses

Results

Discussion

Strengths, Limitations, and Future Directions

Relevance to the Practice of School Psychology

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iDs

Author Biographies

References