Abstract
Objectives:
Previous virtual care literature within the field of speech-language pathology has primarily focused on validating the virtual use of intervention programmes. There are fewer articles addressing the validity of conducting virtual assessments, particularly standardized assessment of oral language and literacy abilities in children. In addition, there is a lack of practical, useful recommendations available to support clinicians and researchers on how to conduct these assessment measures virtually. Given the recent rapid rise in virtual care and research as a result of the Coronavirus-19 pandemic, clinicians and researchers require guidance on best practices for virtual administration of these tools imminently. This article seeks to fill this gap in the literature by providing such recommendations.
Methods:
We (a) completed a narrative review of the extant literature, and (b) conducted semi-structured interviews with a group of 12 clinicians, students and researchers who had administered standardized language and literacy assessments with a variety of monolingual and multilingual school-aged children, with and without speech and language difficulties, in clinical and research settings. Six themes: candidacy for virtual assessment, communication and collaboration with caregivers, technology and equipment, virtual administration, ethics, consent and confidentiality, and considerations for bilingual populations were identified as a result of these two processes and were used to develop a set of recommendations to guide the use of standardized assessments in a virtual setting. In line with the Guidelines International Network, these recommendations were rated by group members, and reviewed by external stakeholders. A quasi-Delphi consensus procedure was used to reach agreement on ratings for recommendations.
Results:
We have developed and outlined several recommendations for clinicians and researchers to guide their use of standardized language and literacy assessments in virtual care, across six key themes.
Conclusions:
This article is one of the first to share practical recommendations for virtual assessment in the domain of oral language and literacy assessment for clinicians and researchers. We hope the current recommendations will facilitate future clinical research in this area, and as the body of research in this field grows, this article will act as a basis for the development of formal Clinical Practice Guidelines.
Keywords
Introduction
Virtual care, formerly or alternatively known as telepractice, telehealth, remote care or telemedicine, is any interaction between patient or client and a member of their circle of care, occurring remotely, using any form of communication or information technologies with the aim of facilitating or maximizing the quality and effectiveness of patient care. 1 There is no consistent and universally agreed-upon term for this service delivery model. For the purposes of this article, we will use the term most often used in Ontario, virtual care, as this is where these recommendations were developed.
The use of virtual care has been steadily growing, but saw a recent surge in 2020, in large part due to restrictions imposed by the Coronavirus-19 (COVID-19) pandemic, which obligated many healthcare professionals and researchers to begin working remotely. In the United States, primary care virtual visits increased by 50% from 2019 to 2020. 2 Similarly, Canada Health Infoway reports that in so far as 2021, 40% of primary care visits were conducted virtually, approximately double the tele-visits pre-pandemic. 3
Virtual care is advantageous for various reasons: clinicians can save time and provide care to a greater number of individuals, 4 clients who live in rural areas without clinics or those without transportation can access clinical services remotely, 5 those with physical disabilities that make travelling challenging, or those who feel more comfortable undergoing assessment in their own homes, also benefit from virtual care. 6 Furthermore, bilingual clients seeking clinicians who speak their language(s), and those seeking practitioners with knowledge in a specific area of practice may be more likely to find a suitable match when they can search a broader geographical area. 7 This is especially important when we consider that although approximately half of the world’s population is bilingual,8,9 only 6.5% of American Speech-Language-Hearing Association (ASHA)-registered Speech-Language Pathologists (S-LPs) report being bilingual, 10 and 73% of Canadian S-LPs report there being a lack of availability of other S-LPs who speak the language of their diverse client populations. 1
Despite these many advantages, a 2011 ASHA survey reported that only 2.3% of delivery of any speech-language pathology service was conducted via virtual care, and only 11% of clinicians had used virtual care. 11 This earlier reluctance to embrace virtual care could have been a result of perceived barriers to the provision of this type of service. In a 2016 paper, Sutherland provides a review of these barriers reported by clinicians in Australia. 12 Three categories of barriers were identified: financial, technological and cultural. Many clinicians felt that the upfront costs of providing virtual services were prohibitive, both for the clinicians and the communities they serve. They also reported that there was limited access to training in and familiarity with technology for clinicians and clients and that addressing this issue would require significant support from policy makers and government officials. In addition, there were concerns on behalf of clinicians that rural families would not embrace or be interested in accessing services virtually. This notion was not supported by research, which on the contrary suggested that these families were interested in this service delivery model, largely in part due to the lack of face-to-face clinics available in their communities. 13 Many clinicians, particularly those in urban areas, indicated that virtual care was a service that would be uniquely valuable to those in remote areas, and were minimally interested in pursuing this model.
Evidently, the widespread integration of virtual care accelerated in Spring 2020, with the intensification of the COVID-19 pandemic. In 2016, StatsCan reported that 4% of the population worked from home, while 32% still work remotely now in 2021. 14 Clinicians and researchers adjusted and modified their work as schools, clinics and laboratories began operating virtually during the pandemic, demonstrating that many of these barriers were often surmountable when no other options were available. In fact, a recent study from Sutherland in May 2021, collected feedback in the form of a questionnaire from 27 paediatric S-LPs, who conducted Clinical Evaluation of Language Fundamentals-5 (CELF-5) assessments virtually. Their qualitative reports suggested that the primary obstacle was overcoming technical issues in virtual assessment, but that otherwise virtual assessments were easy to conduct, and still allowed for connection with clients throughout the pandemic. 15 Now that working remotely is so commonplace, it is likely that virtual clinical assessment and research techniques will remain a trend in the future, especially considering that in 2021, a reported 82% of US employees and 80% of Canadian employees indicated interest in working from home at least some of the time after the pandemic ends. 14
Owing to the fact that virtual care is a relatively new service delivery model in both clinical work and research, there is still a subsequent paucity of literature related to its application. 16 Existing studies suggest that virtual care is a feasible, effective and appropriate alternative or addition to face-to-face practice for speech-language pathology clinicians and researchers. 16 Thus far, many studies have focused on the implementation of specific treatment programmes or therapy methods through virtual means.16–21 Several studies have validated the use of specific treatment programmes for adults and children in a virtual service delivery model across domains of practice. This includes but is not limited to instruction in reading and spelling for children, 22 speech sound intervention in children aged 6–10 years, 23 stuttering treatment for young children using the Lidcombe Program, 24 and for adolescents using the Camperdown Program, 25 voice intervention for adult patients with Parkinson’s using the Lee Silverman Voice Treatment LOUD intervention 26 and in the management of dysphagia in adults. 27 On the contrary, fewer studies have examined the validity and feasibility of conducting assessments, specifically standardized assessments online. 28 This is interesting, in part, because while the ultimate goal of intervention is the functional use of language in real-life scenarios, this contextual support could be more challenging to promote in a virtual care setting, where clinicians and clients are experiencing different circumstances. Standardized assessments, however, aim to evaluate the building blocks of skills like language and literacy in the absence of this contextual support. These assessments are designed to evaluate specific language competencies in isolation. Consequently, it is possible that standardized assessments may be well-suited to virtual administration.
Of the few studies pertaining to standardized assessment, most have focused on the validity and virtual administration of standardized assessment tools in the adult population, rather than the paediatric population. These studies have primarily focused on validity of assessments for virtual use and indicate high rates of agreement between online and in-person evaluation for standardized assessment of neurogenic communication disorders using the Boston Diagnostic Aphasia Examination in post-stroke adult patients, 29 non-standardized evaluation of oral motor skills, swallowing and communication ability in post-laryngectomy adult patients 30 and standardized assessment of linguistic skills in post-stroke adults using the standardized instrument, the Western Aphasia Battery. 31 There have been few studies that have evaluated validity of commonly used oral language and literacy assessments for virtual use in the paediatric school-aged population. 28
Of the published studies exploring validation of virtual use of standardized language and literacy assessments, there is some evidence to suggest that certain subtests of the CELF are valid (CELF-4) 32 and amenable (CELF-5) 15 for use in a virtual setting. In addition, the makers of the Test of Integrated Language and Literacy Skills (TILLS) conducted a study in 2020, which reportedly validates their product for virtual use, as Tele-TILLS. 33 However, this validation article is not made available on their site. Furthermore, many of these validation studies are based on research conducted or funded by the test makers themselves. These groups may have an inherent bias to validate their tools for virtual use, when this has been one of the only available models of service delivery throughout the pandemic. Regrettably, few other in-person tests are validated for virtual use. In response, many test makers have created online or virtual versions of their most common tests and recommend use of these products in a virtual setting. For example, Pearson indicates that the online Q-global version of the CELF-5 can be used reliably in a virtual setting. 34 However, many clinicians and researchers may not yet have access to or training in the use of these virtual materials. To address this issue, some test makers like Pearson, 35 which produces oral language measures such as the Expressive Vocabulary Test (EVT) 36 and Pro-Ed, 37 which develops the Comprehensive Test of Phonological Processing (CTOPP), 38 have issued No Objection orders, allowing S-LPs and researchers to use portions of their test materials virtually through non-public facing teleconference software, provided they follow their rules for administration, and in certain occasions, ask for their permission. 34
Despite the limited number of independent studies validating virtual standardized assessment for tests of oral language and literacy, regulatory bodies such as Speech-Language Pathology and Audiology Canada (SAC), 39 ASHA, 6 Royal College of Speech Language Therapists, 40 Speech Pathology Australia 41 and the Indian Speech and Hearing Association, 42 have authorized clinicians to proceed with assessment of these skills via virtual care. They are cognizant that as a result of the COVID-19 pandemic, clinicians are in a unique position where they are required to continue to provide accurate, timely and informative assessment through virtual care, while research into the validation of their assessment materials is ongoing. Permission to continue these evaluations is critical, as strong oral language skills and proficient literacy abilities have both been well-established as skills that are associated with later positive life outcomes, such as adolescent social and emotional skills and adult socio-economic status (SES) and acquired education level.43,44 In their work, researchers and clinicians commonly use standardized assessment measures to assess, identify difficulties and provide early intervention for mitigating potential oral language or literacy difficulties.
Oral language comprises knowledge of five areas: phonology (the system of speech sounds in a language); morphology (the smallest meaningful units of language and how they are combined); semantics (the understanding and use of words and phrases); syntax (the rules that governs how words and phrases can be combined); and pragmatics (the social norms that dictate how language is used with others in context). 45 S-LPs have extensive knowledge of these domains and play a vital role in the evaluation and management of all aspects of oral language, from birth to adulthood. This role often includes the use of standardized tests. 46 Because of their knowledge of oral language and its relationship with reading and writing, S-LPs are also regularly involved in the assessment and management of literacy difficulties in children as well. 47 Literacy is defined as ‘the ability to identify, understand, interpret, create, communicate, and compute using printed and written materials associated with varying contexts’. 48 It is broken down into three main skills, reading: the process of converting symbols in print into identifiable words, which includes the ability to decode, and to comprehend; writing: the process of expressing thoughts and ideas using printed symbols to make up individual words and sentences; and spelling: the ability to separate words into their individual sounds and link those sounds with specific letter or letter combination to create a word. 47 S-LPs assess and treat discourse level skills like reading comprehension and written expression, as well as word-level skills like phonological awareness or the ability to hear and manipulate parts in words. 47 Again, the role of the S-LP in literacy often includes using standardized tools to evaluate these component skills.
Given that standardized oral language and literacy assessments are often a critical piece of a comprehensive speech and language assessment from both clinical and research perspectives,49–51 and those using these tools may be required to conduct these assessments virtually, clinicians and researchers require updated guidelines to inform their virtual practice, so that they can continue to provide these valuable standardized assessments. Standardized tests have historically been created for use in in-person settings, and many components of the virtual testing process may be different or unfamiliar to clinicians and researchers who are used to in-person administration but have had to adapt their practice to current circumstances. Practice documents from regulatory bodies like College of Audiologists and Speech-Language Pathologists of Ontario (CASLPO) and SAC, as well as guideline papers published in academic journals, 29 have focused primarily on technical, licensure and ethical considerations, and less on the clinical and practical components of assessment, such as the role of the caregiver, organization of the testing space or manipulation of testing materials.5,39,52 Documents from other international regulatory bodies, such as the Telepractice page available on the ASHA website, provide practical resources dedicated to best practice in virtual care (telepractice), including a section specific to assessment. However, this assessment page only briefly discusses test modification, standardization and documentation of results, and does not address other topics like the role of the caregiver in assessment or specific administration recommendations. 53 In addition, the process for developing this ASHA assessment page is not described or available for review. A brief list of references is provided, but no information is given regarding the authors of the document, their qualifications, their experience with virtual care or how they decided what recommendations to include or exclude.
Scope of the recommendations
The objective of this article is to provide recommendations for using commonly assessed standardized tools in a virtual setting. These include receptive and expressive skill evaluations within the domain of oral-language and cognitive-linguistic skills as well as reading, writing and spelling. These recommendations are intended for clinicians and researchers who use standardized language and literacy assessments, within these domains, in the context of virtual care or research. To address the identified gap in the literature, the recommendations presented here consider the practical components of assessment – as opposed to technical considerations typically addressed by regulatory bodies and validation studies. They explore themes commonly discussed in the existing virtual care literature, as well as those that pertain to issues raised by group members engaging in virtual administration of assessments. The themes include candidacy for virtual assessment, 16 technology and equipment management, 54 virtual administration of test protocols, 55 ethics, consent and confidentiality, 55 in addition to communication and collaboration with caregivers and considerations for bilingual populations – the latter two themes emerged based on this study.
When considering previously published guidelines, these recommendations are unique and address gaps in the literature. We provide a resource that is useful for both clinicians and researchers, that is based on the best current available evidence, as well as lived clinical and research experience conducting assessments virtually. Furthermore, the development of these recommendations is reported in a transparent way and is replicable if needed. Finally, we have no bias to recommend the use of specific products or assessment tools in our article. Note: A preprint of this article has been uploaded to the medRxiv server. 56
Methods
Ethics statement
This study has been partially funded by the University of Toronto’s COVID-19 Student Engagement Award (RIS Human Protocol Number: 38608). No conflicts of interest, financial or otherwise, are declared by the authors or members of the guideline development group.
Composition of ‘recommendation development group’
The recommendation development group is comprised of 12 members, with varied experiences using virtual platforms to conduct standardized language and literacy assessment measures with school-aged children.
The initial group was comprised of 10 individuals conducting virtual assessments as part of a study examining online, as compared to in-person, assessments of standardized oral language and literacy tools in monolingual and bilingual children at the University of Toronto (Department of Speech-Language Pathology). It included a clinical speech-language pathologist with 6 years of experience working for a school board, 1 year of which was exclusively in a virtual service delivery model; a second-year research MSc student studying bilingual literacy development; two second-year clinical speech-language pathology students; and six research assistants from the University of Toronto. This group conducted standardized assessments on a group of typically developing English-speaking monolingual (n = 81) and linguistically diverse bilingual children (n = 99), aged 4–8 years (n total = 180). Typically developing children in this age group were assessed, rather than children with identified language and literacy disorders, as this was part of the inclusion criteria for this study. While clinicians do not regularly assess typically developing children, researchers often do, and this article is meant to serve both clinicians and researchers. Group members participating in this study evaluated expressive vocabulary (via the EVT-2), 36 phonological awareness (via the CTOPP-2) 38 and word and non-word reading (via the Woodcock Reading Mastery tests). 57 All tests were conducted in English. This ongoing tele-assessment study commenced in April 2020, after the COVID-19 pandemic and is ongoing.
To gather further clinical insight in the process of developing the recommendations, two additional clinical S-LPs each with over 15 years of experience of face-to-face practice experience, were recruited for participation, for a total of three clinical S-LPs. These clinicians, worked in an exclusively virtual setting for the 2020–2021 school year. As part of their clinical work, they conducted virtual standardized assessments on school-aged children with speech and language difficulties, either for the purpose of formal identification in the school system, or for the provision of specific programming for educators and caregivers. These clinicians used a variety of standardized assessment tools, which included but were not limited to the CELF-5, 58 the Tele-TILLS, 33 the Test of Auditory Processing Skills-4 (TAPS-4), 59 the CUBED Narrative Language and Dynamic Decoding Measures 60 and the EVT-3. 61
In summary, the final group consisted of 12 individuals with varying backgrounds and experiences administering standardized language and literacy assessments virtually. Several group members were novices, either new to research, like the research assistants, or new to clinical practice, like the speech-language pathology students. Others were more experienced in their respective fields, like the MSc student or the school-board clinicians. Some had experience evaluating typically developing students, like those participating in the study, and others primarily assessed children with suspected or identified language and literacy difficulties, like the practicing clinicians. A few members had more practice using assessment tools virtually, while others had primarily conducted assessments face-to-face. All members, novice or expert, researcher or clinician had experience conducting standardized assessments with school-aged monolingual and bilingual children. The diversity among the 12 group members ensures that a broad spectrum of experiences are reflected in this recommendation paper.
Development of recommendations
The following recommendations (see the ‘Results’ section) have been developed in accordance with the Guidelines International Network framework (GIN). 62 These recommendations are intended to act as a step towards the development of Clinical Practice Guidelines (CPGs) for virtual standardized assessment, and as additional research becomes available, should be revised, and updated. These recommendations were developed between August 2020 and June 2021. See Figure 1 (below) for the development process.
Process for development of the recommendations.
Team meetings
While conducting virtual assessments as part of the previously mentioned online literacy study, the 10 initial lab group members met weekly, or bi-weekly, to discuss issues relating to test administration and to provide feedback and advice to one another. These conversations between lab members were unstructured, and a variety of topics, including behaviour management, communication with caregivers, online scheduling, connectivity and technology troubleshooting, and test administration rules were addressed. These problem-solving and trouble-shooting discussions served as a starting point for the development of an informal shared document that provided practical, useful information for administering standardized assessments virtually. The group members subsequently determined that this information would be valuable for clinicians and researchers working in other settings beyond the lab, and as a result, the process of developing the current recommendation paper was initiated.
Development of key themes
The themes addressed in this article were developed from two sources. First, select themes were incorporated based on information from articles identified in a literature search, conducted in August–September 2020. For this narrative style review, two search concepts were used on MEDLINE and Embase databases: (1) Telemedicine (telepractice, telehealth, telemedicine and virtual care) and (2) Speech-Language Pathology (speech therapy and speech disorder). This search yielded a total of 176 articles, of which 129 were deemed irrelevant and removed, as they did not relate to language or literacy, along with 12 duplicates, leaving a total of 35 unique articles. Of these 35, 30 were excluded as they either discussed intervention or treatment in language and literacy rather than assessment; or were conducted on preschool or adult populations rather than with school-aged children; or were conducted face-to-face rather than virtually. Only five specifically addressed virtual language or literacy assessment and screening in a virtual setting with school-aged children,32,63–65 Three papers discussed issues pertaining to language assessment, and two examined validity and reliability of conducting literacy assessments online. See Table 1A in Supplemental Appendix A for a summary of these articles.
Subsequently, the authors identified an additional three papers from this narrative review, which provided general guidelines for virtual practice in speech-language pathology.16,54,55 Review of these guideline articles identified common themes, which were used to guide the development of the recommendations. See Table 1B in Supplemental Appendix A for a summary of these guideline papers.
Importantly, this narrative review revealed a lack of evidence to inform clinical practice regarding virtual assessment. The small number of articles was not felt to be sufficient to warrant a systematic review of the evidence, which would support the development of formal CPGs. Rather than wait for publication of additional papers on this topic, the authors decided to combine the findings of this narrative style review, with lived experiences of clinicians and researchers to create a set of recommendations that could be readily available during the COVID-19 pandemic.
Consequently, the themes outlined in this recommendation paper are primarily based on information from the three guideline papers identified in the narrative review, and the experience of the group members conducting assessments virtually. Specifically, the guideline papers lead to the identification of four themes – administrative, technical and ethical issues; 54 clinical considerations; 55 client candidacy; 16 and the team meetings with group members yielded two additional themes – communication and collaboration with caregivers and considerations for bilingual children. The combination of the narrative review and the team meetings yielded a total of six themes. These themes were derived from the best available scientific evidence, as well as the clinical expertise of the researchers and clinicians involved, both of which are described as critical components of evidence-based practice, according to ASHA’s position statement on evidence-based practice in communication disorders. 66 For a summary of the themes and the sources that inspired them, see Tables 1B and 1C in Supplemental Appendix A.
Individual interviews
Subsequent to the identification of the six themes, an interview script was developed and administered by the first author with each group member (see Supplemental Appendix B). The six themes, as previously stated, are: Candidacy for Participation in Virtual Assessment; Communication and Collaboration with Caregivers; Technology and Equipment Considerations; Virtual Administration of Standardized Assessments; Ethics, Consent and Confidentiality; and Special Considerations for Bilingual Populations. An additional section titled ‘other’ was included in the interview script to address any outstanding points or issues that group members felt did not align with previously identified themes. Members were also encouraged to contact the first author by email after the interview if they had any additional points for consideration. The first author had a previously established working relationship with all group members, whether through volunteer research at the lab for the initial 10 members, or through clinical work at the school board for the 2 clinical members. Group members were aware that they were participating in the development of recommendations for virtual standardized assessment and were familiar with the experience and the objectives of the first author.
First, group members from the lab with experience conducting standardized assessment virtually were purposefully selected for participation in the interview and development of the recommendations. Initially, members were asked as a group to participate virtually during a lab meeting, and following this general invitation, were sent a follow-up email by the first author confirming a time for their individual sessions. The interviews were conducted virtually over the course of a 2-week period, and typically lasted 30–40 min. The first author acquired verbal consent to record the sessions, and all sessions were audio and video recorded on Zoom. No other persons were present at the time of the recording and all interviews were conducted from the first author’s home.
The first author used the interview script, which addressed the previously identified six themes, to guide the conversation. Group members were asked to frame their responses in terms of challenges and advantages associated with each theme and were given as much time as they needed. Interviews were semi-structured, and the group members were provided the opportunity to discuss the advantages and challenges associated with each theme, as well as potential solutions they may have identified, and problems that persisted. All recruited members participated in the interviews. Upon completion of the interviews, it was determined that additional clinical input from S-LPs working virtually outside the lab would help strengthen the recommendations and render them more clinically significant. The first author recruited two additional clinical S-LPs from a school board for participation in individual interviews. These clinicians were selected because they had extensive in-person clinical experience working with school-aged children, and they were both currently working virtually for a school board in the 2020–2021 school year. They participated in the same interview process as the other group members. Each of the 12 members was interviewed only once and group members were not provided with a copy of these notes for review.
Analysing the interviews and compiling the recommendations draft
The first author listened to and took notes on all 12 interview audio files. Observations were recorded and noted in a table with challenges and advantages for each of the six themes. Subsequently, the third and fourth authors each independently listened to and evaluated six recordings each and noted their observations in their own notes document. The first author then integrated all noted observations into one document. See Supplemental File 1 for the rough notes from these interviews. This methodology is consistent with a deductive style analysis,67,68 as themes were identified prior to conducting the interviews and based off narrative review of existing literature, and discussions in lab meetings. The authors elected to take this approach for several reasons. First, presenting each member with an opportunity to address all themes ensured that every member had the chance to discuss and contribute equally to each section, and that there would be robust discussion across a wide variety of topics. Second, deductive analyses are useful when time and resources are limited. Given the rapid uptake of virtual service delivery as a result of the pandemic, the authors felt the need to make these recommendations readily available as soon as possible, and thus selected the most time-sensitive approach. To address some of the potential bias of structuring our interviews around themes, the authors elected to include an ‘other’ discussion point at the end of each interview. This provided each group member the option to add additional points that may not have felt aligned with a specific, previously identified theme.
Subsequently, the first, third and fourth authors collectively developed the first set of recommendations from the aggregate of the interviews. No direct quotes from the group members were used in the recommendations. Rather, the authors identified key points from the interviews for inclusion in the recommendations, based on how often a topic was discussed and its’ general relevance. The authors elected to take this approach to be consistent with existing recommendation papers from published scientific studies and from regulatory bodies that typically present broad general statements or directives, rather than direct quotes from specific clinicians or researchers.39–41,52–55 There was a great deal of overlap in responses to each theme from all group members, and only few minor or overly specific ideas. For example, nearly all group members mentioned the importance of a strong Internet connection, and the need to prepare for and plan reinforcement and breaks in an online testing session, so these points were included as recommendations in the first draft. However, overly specific, or infrequently mentioned points, like relabelling the names of audio files associated with specific tests for ease of recognition, were not included. The authors collectively used these notes to draft the first set of the recommendations, which were grouped into the previously identified six themes for ease of use and readability. This resulted in a total of 45 recommendations across the six themes that were directly derived from the interviews.
Decision-making process for ratings
This initial draft of the recommendations was then disseminated to all 12 group members, without qualifying descriptive adjectives, via a survey to allow for their rating. Members were instructed to rate each guideline, by choosing either ‘must’, ‘should’ or ‘consider’. They were informed that the ‘must’ rating referred to recommendations that are required/mandatory to be followed to comply with regulations from their regulatory governing bodies such as those from CASLPO. For example, the obligation to obtain informed consent for all aspects of virtual care, and the importance of determining whether a client is appropriate for this type of service delivery. 69 The ‘should’ rating referred to recommendations that should be followed whenever possible or feasible, to ensure best practice, but that are not necessarily mandated by a governing body. For example, suggesting the assessment take place in a quiet, separate room with a caregiver present is preferred, but not always possible for every family depending on their housing situation, and their other caregiver and work obligations. The ‘consider’ rating referred to recommendations that are merely valuable pieces of additional information for consideration that clinicians and researchers might find useful as they begin their virtual assessments. For example, if available, using two devices to allow for better observation of the child while simultaneously displaying testing information.
A Quasi-Delphi method was used to survey group opinion regarding the recommendations, with decision-making consultations between the first and second authors, along with the principal investigator, at the first and second rounds of survey. Previous studies, adopting the Delphi method, recommended 2–4 survey rounds and a minimum of 50%–80% consensus at each survey round. 70 Similarly, we aimed to conduct a minimum of two survey rounds, with a mid-level consensus threshold of 70% per round, until consensus was achieved. Recommendations were approved and adopted in instances where a minimum of 70% of group members agreed on their rating. Review of the first round of ratings indicated that the group members reached 70% consensus on 22 out of 45 recommendations. Subsequently, a second version of the survey was shared with group members, after consultation with the first and second authors as well as the principal investigator, regarding unresolved recommendations that did not achieve group consensus. In this second survey version, members were asked to rate the remaining unresolved recommendations again. In this version, only the top two choices chosen in the first survey were provided as options. For example, if an unresolved recommendation had a 50%:40%:10% rating for ‘must’, ‘should’ and ‘consider’, only the top-two rating options, ‘must’ (50% consensus) and ‘should’ (40% consensus), were sent to group members for re-rating. To ensure rating accuracy and avoid response bias, group members were not informed of agreement percentages per rating option as well as which of the top-two rating options had greater agreement. Review of the second survey ratings indicated the group had reached consensus on an additional 10 recommendations. All 12 group members participated in both rounds of recommendation rating. To decide on a final rating for the remaining recommendations, the first and second author, along with the principal investigator met to discuss and determine the final rating for the remaining 13 out of 45 recommendations. The final rating was determined when the three group members unanimously agreed. A breakdown of the rating of votes for survey 1 and survey 2 can be found in Table 2A and 2B, respectively in Supplemental Appendix C. After finalizing the ratings, the recommendations were complete and were shared with community stakeholders for review and feedback.
Peer review and stakeholder consultations
To obtain feedback on the quality of the recommendations, a draft was shared with four stakeholders, three additional school board S-LPs working in a virtual care setting and one additional researcher who reported using standardized assessment tools virtually in their work. The two school board S-LPs each had more than 5 years of clinical experience, and a full year of experience working in an exclusively virtual setting at the time of the consultation. The researcher was a fourth-year student in a combined MClSc/PhD programme with more than 3 years of lab experience, and recent direct experience conducting standardized assessments in a virtual setting with children. These external stakeholders were asked to evaluate the quality of the recommendations using an adapted version of the Appraisal of Guidelines for Research & Evaluation Instrument (AGREE-II). 71 We adapted the AGREE-II checklist, developed for intervention-based clinical guidelines, as there are no checklists specific to assessment guidelines development (regardless of assessment medium, whether online or in-person). The AGREE II is comprised of 23 items sorted into 6 categories: Scope and Purpose, Stakeholder Involvement, Rigour of Development, Clarity of Presentation, Applicability and Editorial Independence. Individual items are rated on a scale of 1–10 and the checklist is scored within each category using a percentage, with a score of 70% indicating a high-quality guideline. In our modified version of the AGREE II, we retained the six categories, but adapted select items within each to better reflect the practice to evidence-based approach used in the development of these recommendations. We also opted to emulate the International Centre for Allied Health Evidence Guideline Quality Checklist (iCAHE), 72 and use a yes/no rating system for each item, rather than a 10-point rating. This was done for ease of stakeholder use, and for ease of interpretation and scoring. See Table 3 in Supplemental Appendix D for the adapted and scored stakeholder checklist.
Given that the tool was modified, the results of the checklist were analysed qualitatively rather than quantitatively. All four stakeholders provided positive feedback across the 23 items in the form of their yes/no ratings. Three out of the four stakeholders rated all 23 items positively. One out of four stakeholders rated one item negatively, indicating that a procedure for updating the guidelines was not provided. This was the only negative rating on an item. As a result, the authors attempted to elaborate on the recommendation updating process in the final draft of the article. In addition, subjective feedback from the stakeholders indicated that the recommendations were ‘very readable’ and a ‘wonderful resource for clinicians and researchers alike’. Regarding next steps, one clinical stakeholder indicated that the process for updating the recommendations should be better outlined and more robust. However, given that this recommendations article is one of the first of its kind in the field of educational speech-language pathology, there is limited consensus or documented procedure for how to update recommendations in a practice guideline of this type. To provide researchers and clinicians with up-to-date recommendations, these guidelines will be continuously updated as the body of literature progresses (see the section ‘Recommendations expiration and updating’).
Results
Recommendations for virtual administration of standardized assessments
Candidacy for participation in virtual assessments
Communication and collaboration with caregivers
Technology and equipment
Administration of standardized assessments
Ethics, consent and confidentiality
Considerations for bilingual participants
Recommendations expiration and updating
As previously stated, we consider these recommendations as a starting point for future guidelines development and research into online assessment by clinicians and researchers. These recommendations will be updated as new themes emerge in future research investigating the in-the-field use of these recommendations by clinicians and researchers with typically and non-typically developing clinical populations. We anticipate that, as a result of the current COVID-19 pandemic, further research on tele-assessments will be conducted in the coming years. As such, we will conduct a preliminary literature search, within the next 2 years, to determine if there are sufficient tele-assessment studies, within the field of oral language and literacy research, to develop formalized CPGs – based on a systematic literature review. Additional searches may also be conducted on a bi-yearly basis, based on a need-based analysis of current literature. These updates will be developed by consulting the co-author leads in conjunction with a recommendation development group with a similar composition, as the current team, to ensure diversity of clinical and research experience conducting standardized assessments in virtual setting. These updates will be published to the following link: https://dataverse.scholarsportal.info/dataverse/virtualcare/ on the University of Toronto’s open access Dataverse.
Discussion
In recent years, virtual care has become more widespread. 3 Previously, the incorporation of virtual care was primarily driven by the desire to save time and better allocate resources and reach those in rural communities who lived without access to valuable medical and health services. However, widespread use of virtual care became necessary for clinicians and researchers alike with the outbreak of the COVID-19 pandemic and work-from-home orders, while guidelines to support virtual clinical and/or research decision-making are scarce. This article offers practical recommendations for virtual administration of oral language and literacy assessments to be used by clinicians and researchers.
Through the virtual administration of standardized tests, and in the development of these recommendations, group members in this study identified benefits and challenges associated with virtual assessment. Regarding the challenges, several group members referred to poor Internet quality and its effect on their ability to administer certain tests. In particular, tests that include a timing component such as the digit, letter, colour and object; rapid automatic naming subtests in the CTOPP-2, 38 or tests that require careful attention to specific sounds such as the repetition of non-words in the CTOPP-2 38 were most affected by poor Internet quality. Tests that require pictorial or word stimuli like the word and non-word reading tests and the EVT-2 36 were impacted by the size of device that the participant was using to view the stimuli. In our experience, some families did not have access to computers, tablets or monitors and attempted to participate in the assessment using their phone. Group members also reported that some families struggled to access assessment meeting links, particularly those who did not have experience navigating and using virtual communication software. See Supplemental File 1 for interview analysis notes from authors E.W., B.M. and C.F.
These issues are of particular concern for lower SES families who may lack the necessary Internet access, software or hardware.76,77 Even in developed countries, such as the United States, 17% of children have no access to a laptop or desktop computer.78,79 Furthermore, children belonging to lower SES households and marginalized Black, Indigenous and People of Colour communities are more likely to have limited or no access to a computer and/or Internet connection. 78 In Canada, children from lower income households are less likely to have access to their own Internet-enabled device as compared to higher income households; furthermore, as compared to higher income households children from lower income households are also more likely to only access the Internet through mobile devices rather than personal computers, which may influence assessment quality. 77 The first author experienced this firsthand, while assessing children virtually as a clinician at a school board. Children from low SES neighbourhoods were less likely to have a computer at home. They often used school-issued devices (in this case, Chromebooks), or when those devices were not working or available, their caregivers’ phones. On several occasions, the school did not have sufficient devices available for those who required them. Consequently, these students were not able to participate in assessment with the school S-LP, and importantly also not able to attend their virtual daily classes with their peers and educators, placing them at greater risk of falling behind. In addition, the school-issued devices did not allow students and their families to easily join meetings on the platform that the S-LPs were instructed to use (Microsoft Teams) as they were designed for classroom use and the Google Meet platform. This was problematic, as Google Meet is not regarded to be as secure in terms of its’ personal health information protection properties as Microsoft Teams. This created additional barriers to participation in assessment for low SES families, which had to be resolved through meeting invitation sharing through the classroom teacher. These studies and clinical anecdotes suggest that the populations who may have the greatest need to for oral language and literacy assessment, those who are marginalized, living in low SES or remote and rural communities, are often those who do not have access to the necessary tools to participate in such assessments. Prior research has demonstrated the negative influence of low SES on early oral language skills, such as vocabulary knowledge and subsequent literacy development.80,81 It is, therefore, important to continue providing in-person oral language assessments and intervention for children belonging to marginalized and low SES communities. Clinicians and researchers should keep in mind that virtual assessment might not be suitable to all segments of the populations, due to limitations associated with Internet and computer access.
Virtual assessment may be particularly effective for bilingual children living in a country where the heritage language is not the dominant societal/educational language. Given the lack of practicing bilingual S-LPs across Canada and the United States,10,82 tele-assessment enables access to S-LPs and language interpreters speaking the same heritage language – without additional transportation or geographical barriers. For bilingual children with limited or non-balanced proficiency in the societal/educational language, assessment across both heritage and societal languages can enable early identification of potential oral language and literacy issues as well as facilitate intervention in the dominant language.
This article has identified that further research is needed to validate virtual administration of standardized oral language and literacy assessments commonly used by clinicians and researchers. In addition, due to the limited-permission nature of the No Objection orders, it is uncertain whether publishers will indefinitely allow clinicians to conduct virtual assessments using these tests that were developed for in-person administration. These restrictions, along with a lack of standardized assessments validated for virtual use, limit the scope and type of assessments clinicians and researchers are able to conduct. For example, a commonly used tool for assessing phonological processing, the CTOPP-2, 38 has not been validated for virtual use. As a result, clinicians and researchers may be required to administer non-standardized assessments, or components of validated standardized measures, such as the Kaufman Test of Educational Assessment-3, 83 which require additional training and may not be as readily available in their place of practice. This article has identified that further research is needed to validate virtual administration of standardized oral language and literacy assessments commonly used by clinicians and researchers, such as the CTOPP-2, 38 EVT-3 61 and PPVT-5, 74 among others. Virtual assessments can also pose additional challenges specific to researchers. Virtual experiments may facilitate a larger sample size by limiting potential geographical location or transportation-related participation barriers. However, heterogeneity in terms of the type and quality of Internet-enabled devices accessible to the child, such as whether the child completes the assessment via a limited function and smaller mobile device as compared to a personal computer, may limit generalizability of research findings to children across diverse SES groups. Despite potential research-related challenges, virtual assessments can facilitate global cross-cultural speech-language research collaborations, while limiting potential cost and location-related assessment or research participation barriers.
Limitations
The current recommendations are an initial step towards the development of evidence-based virtual assessment guidelines. These recommendations were developed based on the experience of 12 individuals using three standardized measures with a select population. We are cognizant that feedback from four stakeholders on an adapted checklist cannot be generalized to indicate that these recommendations would be rated positively by all clinicians and researchers working with standardized assessments in a virtual setting. Future iterations of these recommendations, and other more formalized CPGs, should be reviewed and rated by clinicians and researchers working in a variety of settings and with diverse experiences to ensure their usefulness, readability and to determine their quality. It is important to note that all members and stakeholders who participated in this study had an interest in developing recommendations for virtual standardized assessments, which may have influenced the outcomes. This could theoretically be avoided in the future, through inclusion of members who have varied opinions on implementation of virtual practice. Ideally, reviewer and stakeholder groups should include clinicians and researchers with mixed opinions on virtual care, some with ample experience, as well as those who are new to this service delivery model. This would provide the opportunity to explore whether (1) the recommendations are aligned with what experienced clinicians and researchers already do in their practice and (2) if the recommendations are useful for new clinicians and researchers in establishing good practices for virtual standardized assessment of oral language and literacy tests. Finally, we recognize the limitations of using a deductive analysis to evaluate the interviews in this study. Although the authors attempted to mitigate some of the potential bias by including an ‘other’ section to provide an opportunity to address points that did not align with a given theme, it is possible that different responses might have been obtained from members, had the interviews not been structured in this format. Ultimately, this approach was chosen as it is less time-consuming than conducting an inductive thematic analysis, and the authors were motivated to share recommendations to their clinical and research colleagues in a timely manner for use during the COVID-19 pandemic.
Future directions
As additional research in the field of virtual language and literacy assessment becomes available, more formalized CPGs should be developed, that systematically evaluate and incorporate these studies into their recommendations. Beyond virtual care, there is also more generally a lack of CPGs available in the field of developmental speech-language pathology. Currently, those working in the medical sector, have access to select CPGs to inform certain aspects of their practice. Specifically, there are formal guidelines available for instrumental assessment of voice, 84 and acquired velopharyngeal dysfunction. 85 However, those seeking guidelines pertaining to intervention in the areas of speech, language and literacy development, either in person or in a virtual model, may have to rely on general position papers from regulatory bodies, which often do not provide concrete practice recommendations, or individual research studies, which are often based on specific populations, limiting practice generalizability. Those working as clinicians and as researchers in the field of speech-language pathology would benefit from the development and dissemination of additional CPGs addressing topics from developmental and medical sectors of the field. Consequently, future research should endeavour to evaluate the virtual administration of these types of assessments with children and adolescents with mild-to-severe speech and language difficulties and should expand the areas of oral language and literacy that are evaluated beyond expressive vocabulary, phonological awareness and word and non-word reading. In addition, further research should consider the effect of child attention, caregiver involvement and access to necessary technology as a factor of SES on the virtual administration of such tests, and others.
It should also be noted that the majority of available standardized tools were developed for in person-use with monolingual English-speaking children. These standardized norm-referenced assessments are a valuable tool but are not the singular method of evaluating the language and literacy skills of school-aged children, particularly for children from linguistically and culturally diverse backgrounds whose English language skills often differ from their monolingual English peers. It will be critical for future research to investigate the validity of and provide guidelines for the administration of other non-standardized and informal evidence-based assessments, such as language sampling analysis and dynamic assessment in a virtual setting, with both monolingual and multilingual children.86,87 As the body of research on virtual care continues to grow, researchers and clinicians must continue to collaborate on such studies to ensure that future guidelines are useful, realistic and helpful for all.
Conclusion
These recommendations cannot act as a basis for conclusive guidelines for all virtual clinical scenarios involving assessment of oral language and literacy skills. However, these findings provide a foundation for clinicians and researchers who are embarking on virtual assessment and who are seeking practical and useful suggestions to guide their practice and studies. They also provide a framework for future research into the feasibility and execution of online standardized assessments, as well as developing additional or revised guidelines.
An unexpected finding from this study and the development of these recommendations is that caregivers played a larger-than-expected role in the administration of the assessment tasks. Caregiver participation was critical in the management of behaviours, provision of reinforcement if required, translation of child responses from other languages, provision of interpretive feedback for the clinician or researcher as needed, as well as trouble-shooting any technical difficulties that arose during administration, particularly for younger participants. The caregiver was also required to assist if manipulatives were needed to complete tasks. Consequently, it may be valuable for future studies to consider whether there is an effect of caregiver participation on assessment results, as their involvement during the assessment may have unintended but significant effects on clients’ outcome. This notion should be further explored using a systematic approach, which also examines guidelines and recommended practices for parents involved in virtual assessments.
Second, the establishment of these recommendations has also illustrated that virtual assessment is a new skill to be learned by clinicians and researchers, similar to how standardized assessment is a new skill to be learned when clinicians are beginning their practice. Consistent, deliberate practice should be prioritized by those administering the tests to learn the nuances pertinent to administering the assessment virtually, as opposed to in-person. In anticipation of these challenges, academic institutions have begun providing instruction to clinical and research graduate students in the field of speech-language pathology in all aspects of virtual care, including virtual assessment. It will be critical for this practice to continue, to ensure that their graduates are qualified to practice or conduct research in the post- COVID era where virtual care has become ubiquitous.
Supplemental Material
sj-docx-1-smo-10.1177_20503121211050510 – Supplemental material for Towards development of guidelines for virtual administration of paediatricstandardized language and literacy assessments: Considerations for clinicians and researchers
Supplemental material, sj-docx-1-smo-10.1177_20503121211050510 for Towards development of guidelines for virtual administration of paediatricstandardized language and literacy assessments: Considerations for clinicians and researchers by Emily Wood, Insiya Bhalloo, Brittany McCaig, Cristina Feraru and Monika Molnar in SAGE Open Medicine
Supplemental Material
sj-docx-2-smo-10.1177_20503121211050510 – Supplemental material for Towards development of guidelines for virtual administration of paediatricstandardized language and literacy assessments: Considerations for clinicians and researchers
Supplemental material, sj-docx-2-smo-10.1177_20503121211050510 for Towards development of guidelines for virtual administration of paediatricstandardized language and literacy assessments: Considerations for clinicians and researchers by Emily Wood, Insiya Bhalloo, Brittany McCaig, Cristina Feraru and Monika Molnar in SAGE Open Medicine
Footnotes
Acknowledgements
The authors would like to acknowledge and thank the 12 group members who participated in the development and rating of these recommendations, as well as the 4 stakeholders who provided valuable insight and feedback on this article, prior to its submission for publication. We are grateful and appreciative of your contributions.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Ethical approval
This study has been approved by the Social Sciences, Humanities and Education Research Ethics Board at the University of Toronto (RIS Human Protocol Number: 38608).
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study has been partially funded by the University of Toronto’s COVID-19 Student Engagement Award (RIS Human Protocol Number: 38608).
Informed consent
Verbal informed consent was obtained from all subjects before the study. Verbal informed consent was obtained from all group members participating in recommendation development prior to the interviews. All group members were informed of the purpose of the interview (to help develop recommendations) and were given an option to decline participation in the interview process, Group members were also encouraged to and provided the opportunity to ask questions. Verbal consent was deemed sufficient as group members were colleagues and lab members voluntarily participating in the development of the guidelines, and the study posed no risk. Group members’ informed verbal consent is documented in audio recordings of the interviews. This was approved by the Institutional Ethics Board associated with this project.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
