Abstract
Objectives
To evaluate the screening performance of 6/6 and 6/12 vision cut-offs with an illiterate E-chart implemented by a public health nurse to test children for ocular abnormalities and uncorrected refractive error. The gold standard diagnosis is an eye examination performed by an ophthalmologist.
Setting
A cross-sectional population-based study was conducted among 2113 students' ages 6–7 and 13–14 years old in 70 Northern District Israeli schools.
Methods
Students were tested by nurses and ophthalmologists. A nurse examination was carried out using the illiterate E-chart for vision measurement. The medical examination included vision history, clinical eye examination, vision and retinoscopy testing. The Physician's evaluation of whether students needed a referral for diagnostic procedures, treatment and/or follow-up was recorded. Screening test's performance was determined using ophthalmologist's decision regarding referral as the gold standard. Detection rate (DR), false-positive rate (FPR), odds affected positive result (OAPR), positive predictive value (PPV) and negative predictive value (NPV) were estimated overall and by students' demographic characteristics.
Results
For vision >6/6 cut-off in at least one eye (eyes tested separately): DR – 71.9% (95% CI 65.8–78.7%), FPR – 22.8% (95% CI 17.9–28.9%), OAPR – 0.98:1 (95% CI 0.84:1–1.15:1), PPV – 52.7% (95% CI 45.4–61.2%), NPV – 90.9% (95% CI 88.7–93.1%). For 6/12 vision cut-off, namely vision 6/12 or worse in both eyes (tested separately): DR – 58.6 (95% CI 51.8–66.4%), FPR – 15.2% (95% CI 10.9–21.1%), OAPR – 1.13:1 (95% CI 0.94:1–1.35:1), PPV – 61.1% (95% CI 52.9–70.6%), NPV – 87.6% (95% CI 84.9–90.4%).
Conclusions
Vision-screening test performance measures are mild. It is suggested to change vision cut-off level that denotes vision abnormality from current policy of vision not equal 6/6 in both eyes (tested separately) to vision 6/12 or worse in both eyes (tested separately). This change will result in reduction of FPR from 22% to 15%, concomitant with an increase in false-negative rate from 28% to 41%. Students may be equally screened by either a senior or a less experienced nurse.
INTRODUCTION
Screening programmes for vision abnormality vary with respect to the screening tool, the professional who is carrying out the testing, the threshold for failure and the setting. The most common method to test vision and visual acuity (VA) is using the standard Snellen chart. 1 Screening techniques are likely to differ in detection rate (DR) and false-positive rate (FPR).
The purpose of screening at school age is to reduce the proportion of children who have a vision deficit that could be corrected by spectacles. Although vision is considered the most appropriate screening test to identify individuals with visual impairment due to uncorrected refractive errors, 2 only a small number of studies was identified that dealt with its performance measures among school children, and with the role of different professionals carrying out vision-screening tests. 1,3–16
The policy of the Israeli Public Health Service (PHS) of Ministry of Health (MOH) is to screen all students ages 6–7 years old, who have missed testing at pre-school (ages 5–6 years), and all students ages 13–14 years old for vision abnormality. The screening is delivered at school by PHS nurses using the Illiterate E-chart. The assumption underlying this policy is that early detection of vision abnormality by screening may lead to early intervention followed by optimal students' function. 17
The Northern District, the largest MOH Region in the country, populates 1.1 million residents, half of whom are Jews and half are Arab. The 270,000 school-aged children reside in five subdistricts. Many of the Arab students live in low socioeconomic-status rural areas.
In 2002–2003 we launched a Northern District Israeli Vision Screening Project aimed to assist policy decision-making by examination of various aspects of school children vision screening. Vision abnormality, using the same study sample as presented in this article, was found to be a significant health problem (unpublished data).
The main objective of the present study was to test the performance of the vision-screening test used by PHS nurses in Northern District Israeli school children. Emphasis was given to all the following measures of test performance: DR, FPR, odds affected positive result (OAPR), positive predictive value (PPV) and negative predictive value (NPV), using ophthalmologist examination and decision regarding need for further referral as the gold standard.
METHODS
The study commenced with a training programme for nurse screeners, as well as for eye medical examiners, delivered by a team of study personnel. It focused on issues such as strategies to obtain parents informed consent, study procedures and research forms.
Population
The population chosen for this study included students ages 6–7 (first graders) and 13–14 years old (eighth graders) attending the regular education system of the Northern District. A cluster sampling of 75 schools (35 for recruitment of first graders and 35 for recruitment of eighth graders), representing 13.9% of 541 eligible Northern District schools (elementary and middle schools) was used. Each school was represented by all students in one class randomly pre-selected. This strategy was in line with the intention to recruit 2000 students (1000 of each age group). The assumptions underlying sample size determination included 10% prevalence of vision abnormality (estimated upon Northern District VA results of the former year), and test's DR of 40%. The intention was to point estimate the DR with an error that would not exceed 10%. This meant a required sample size of 100 students with vision abnormality in each age group.
Leaflets were distributed to parents prior to study commence in three languages (Hebrew, Arabic and Russian), explaining study goals and procedures, requesting their consent. Study protocol was approved by the institutional review board (IRB) of the Haemek Medical Center,
Measurements
The nurse examination
The examination was carried out by 67 nurses using the illiterate E-chart for vision screening (supplier Sarel, MOH, Israel). The chart has a large ‘E’ at the top (denoting vision of 6/60), and six additional horizontal lines of letters, decreasing in size from top to bottom (6/60, 6/42, 6/30, 6/24, 6/18, 6/12 and 6/6). The bottom line denotes vision of 6/6. The potential range of vision that can be measured at 5 m is therefore 6/6 to 6/60 (=20/20–20/200). All children were always measured at 5 m. No alternative distance adopted in those with very poor vision. Nurse activities were carried out according to standard PHS procedures. 17 In brief, this included requesting the student, who sat 5 m distant to the chart, to cover each eye in its turn, and to specify the direction of the letter ‘E’ shown by the nurse in decreasing rows order. Exceptions to the usual PHS routine included the following: (1) students with spectacles or contact lenses were also tested (in the usual routine they were not considered candidates for vision screening), (2) all first graders were examined (not only those who missed the vision-screening test at age 5, prior to school entry), (3) at the end of each examination, report of vision results (or VA for those wearing eyeglasses) of each eye and a statement regarding whether student passed (or failed) the test, was handled to study coordinator, (4) a letter to parents reporting screening results was not sent by the nurse, but by physician. Vision-screening test was considered negative (successful) if the student, when tested for each eye separately, correctly identified in each eye four out of the nine E letters at the bottom (seventh) row of the chart, in which case student was recorded having vision (or VA) of 6/6 (=5/5).
The medical (ophthalmologist) examination
Examination was carried out at school, following nurse examination, at separate location. It was carried out by 10 opthalmologists from Haemek medical center (5 eye specialists and 5 residents, who have finished at least three years of residency training). The examination included vision history, a clinical eye examination and vision testing, using the illiterate E-chart. For students using spectacles, the corrected VA was recorded. If signed informed consent was available, the testing proceeded with delivery of eye drops (cycloplegia) for fundus and retinoscopy, 30–45 minutes later. Physicians recorded the findings and their clinical judgement regarding referral for further diagnostic procedures, treatment and/or follow-up. If positive, the referral reason and professional/clinic were stated and the child was classified as having a vision abnormality. Indicated reasons were one or more of the following: (1) myopia (retinoscopy: >1 [absolute value]), (2) hypermetropia (retinoscopy: >2), (3) cylinder (astigmatism; retinoscopy: >1), (4) corneal pathology, (5) lental pathology, (6) Retinal pathology, (7) other. At the end of study day the physicians, blinded until that moment to the nurses' results, received their reports. Letters to parents were then prepared.
Data analysis
The main data analysis included students wearing eye glasses at the time of screening (20 students ages 6–7 and 91 students ages 13–14). Vision-screening results were as determined by the nurse and vision abnormality as determined by physician.
Vision-screening abnormality
Prevalence of vision-screening abnormality, as determined by the nurse, was calculated by age group, using two different definitions: (1) the current PHS definition of a positive screening test: vision is worse than 6/6 (=5/5) in at least one eye (each eye tested separately), (2) vision equals to or worse than 6/12 (=5/10) in both eyes (each eye tested separately).
Retinoscopy
Refractive error types were classified as mild, moderate or high myopia, hypermetropia and/or cylinder (astigmatism) using criteria defined in Appendix A. Refractive error subtypes captured by criteria of vision abnormality were calculated.
Prevalence of vision abnormality
All students referred by the ophthalmologist for further evaluation, therapy or follow-up were considered prevalent cases of vision abnormality. Prevalence of vision abnormality and 95% CI were calculated by age group and vision-screening results.
Measures of vision-screening performance 18
The measures of vision-screening performance were determined using field physician's decision regarding student being a prevalent case of vision abnormality as the gold standard. DR (sensitivity, the proportion of affected individuals with a positive screening test result), FPR (1-specificity, the proportion of unaffected individuals with positive screening results), the odds of being affected given a positive result (OAPR) (the ratio of the number of affected to unaffected individuals among those with positive screening test results), PPV (the proportion of students with a positive test result who were classified by ophthalmologist as prevalent cases of vision abnormality) and NPV (the proportion of students with a negative test result who actually had no ophthalmologist determined abnormality), as well as 95% CI were estimated overall and by students' demographic characteristics.
To account for the cluster sampling of schools, estimation and comparison of vision (or VA) and of measures of screening performance was performed using logistic regression model adjusted for correlated data (SAS® GENMOD procedure with REPEATED statement). Compound symmetry (exchangeable) structure was assumed for the correlation among the students checked at the same school (class). This analytic approach implements the GEE methodology for correlated data. 19,20 Prevalence ratio (PR) and 95% CI were estimated by introducing a LOG link and a binomial distribution into the GENMOD procedure. Data analysis was performed using SPSS (version 13.0, SPSS Inc., Chicago, Illinois) and SAS (version 9.1, SAS Institute Inc., Cary, NC) software. P < 0.01 was considered statistically significant.
RESULTS
Study population
Study population included a sample of 2113 children in 70 regular education Northern District schools. This comprises 93.3% of 75 schools originally sampled for the study, and more than the 2000 students planed. Students surplus, compared with intended number, relates to the sampling scheme (all students in each class sampled). Compliance of school directors was 93.3% (70 out of 75), of parents – 68.3% (1444 out of 2113) and of school children 99.0% (2092 of 2113).
Flow diagram of study, in line with the suggestion of The Standard for Reporting of Diagnostic Accuracy (STARD) initiative 21 is presented in Figure 1. Of the intended sample, 131 students did not attend school at study day and seven were tested by a nurse only (two with abnormal vision-screening results according to both definitions) leaving school prior to ophthalmologist examination. Accordingly, 1975 students (93.5%) were tested by both a nurse and a physician. Of them 1862 students (94.3%) had data about decision made by field physicians regarding further referral, and were therefore eligible for measures of vision-screening performance analysis. The actual population, compared with intended sample included a 1–2% higher proportion of 13–14-year-old students, males, Arab and residents of Nazareth subdistrict (Table 1). As shown, students were evenly distributed by age and gender, and about two-thirds were Arab. Most students were residents of Acre and Yizre'el subdistricts. A comparison of study population to Northern District eligible subjects revealed slight (6%) over- representation of 13–14-year-old students.
Flow diagram of study by definition of vision-screening abnormality. *Abnormal result: (1) Vision >6/6 in at least one eye, (2) vision ≥6/12 in both eyes (tested separately); ** Reference standard: Ophthalmologist examinbation and decision regarding need for further referral;
Target condition present: vision abnormality (further referral for diagnosis, treatment or follow-up is required);
Target condition absent: normal vision (no further referral for diagnosis, treatment or follow-up is required). n
1 and n
2 refer to number of students according to the first and second definitions of abnormal vision screening, respectively
Demographic characteristics of intended and actual study population
*Chi square test
Vision-screening results
About two-thirds of study participants had normal vision (or VA) screening findings, as determined by the nurse (6/6 in both eyes tested independently) (Table 2). Higher screen positive results (according to both definitions) were noticed among 6–7-year-old students.
Distribution of vision-screening results by age group (n = 1862)
*% adjusted for clustering (school)
† Vision-screening result as determined by a nurse, exact result for each eye tested separately
‡6/18 or worse in at least one eye
Prevalence of vision abnormality
Of 1862 students, 420 (23.4%, adjusted for clustering) were referred by field physicians for further evaluation, therapy or follow-up. They were considered positive (prevalent) cases of vision abnormality. Non-referred students were considered negative cases. The prevalence of vision abnormality was higher among 13–14 as compared with 6–7-year-old students (28.6% versus 17.6%, respectively, P = 0.0005). A positive correlation (P for trend <0.0001) was noticed between vision-screening results and prevalence of vision abnormality (Table 3).
Prevalence of vision abnormality by vision-screening results and age group (n = 1862)
*As determined by a physician
†% adjusted for clustering (school)
‡PR adjusted for clustering (school)
§As determined by a nurse, exact result for each eye tested separately
¶6/18 or worse in at least one eye
Measures of vision-screening performance
Measures of vision-screening performance (overall and by student's age group and cut-off level of vision-screening abnormality) are presented in Table 4. As shown, if a worse vision cut-off level was chosen, the DR and FPR decreased and the OAPR and PPV increased. Since no differences in test performance measures were found when comparing students tested by a specialist with those examined by a resident, nor when comparing screening carried out by more versus less senior nurses, analysis of vision-screening performance measures, as described above, was carried out without split by physician's or nurse seniority.
Measures of vision-screening performance by age group and definition of vision-screening abnormality (n = 1862)
*Each eye tested separately
†% adjusted for clustering (school)
Refractive error subtypes captured by screening test
Vision-screening abnormality, defined as vision >6/6 in at least one eye, was found to capture about two-thirds of students who had myopia or astigmatism and 38% of students with hypermetropia. These proportions were reduced when cut-off level was changed to 6/12 or worse in both eyes, each eye tested independently (Table 5).
Refractive error subtypes captured by criteria of vision-screening abnormality (n = 1708)
*Each eye tested separately
†% adjusted for clustering (school)
‡With or without myopia or hypermetropia
Impact of screening at Northern District level
Based upon the 23.4% prevalence of vision abnormality, taking into account the DR (72%) and FPR (23%) using current definition of vision-screening abnormality (vision >6/6 in at least one eye), as well as the age, gender and ethnicity distribution of the Northern District, it implies that 3348 (95% CI 2941–3754) of 10,000 students would screen positive ([TP + FP]/ [TP + FP + TN + FN]). The screen positive Northern District rate, if cut-off level is changed to vision equals to 6/12 in both eyes (tested separately) and worse, would have reduced to 2535 (95% CI 2129–2942) of 10,000 students.
DISCUSSION
The main advantage of this study, beyond its large population-based sample size, is its design, which complies with most published methodological standards for the evaluation of diagnostic test research. 22 As stated by Harper et al., 23 in relation to ophthalmic diagnostic tests, evaluation of screening performance should comply with the same criteria as those used for evaluating diagnostic tests. The seventh standard (presentation of test reproducibility) was not met due to feasibility considerations. According to Harper, 23 only 10% of the studies complied with this standard. The study design enabled assessment of all four cells (TP, FP, FN and TN). Often, only subjects receiving positive results are referred for further testing. 12–15
According to the current PHS policy, children whose vision is less than 6/6 in at least one eye are referred for further assessment. The appropriateness of using this referral criterion, tested by others, 4,12–15 has not been previously tested in Israeli school children. Was our gold standard appropriate? One might argue that including physicians of varying levels of expertise, was problematic, yet no differences in vision abnormality rates were detected for the two groups (data not presented). Residents participated in this study not only due to too few specialists available but also since they are expected to comply with community needs.
A positive association was demonstrated between vision-screening results and prevalence of vision abnormality. Still, comparing vision findings of individual students, as recorded by the nurse, with that of the ophthalmologist, moderate performance measures were found. It should be mentioned that although physicians were blinded to the nurse findings, they were not blinded to their own vision test results. Thus, if vision had an important impact upon the physician's decision, biased (over estimated) performance measures might be introduced. The consequences of the relatively high FPR test performance results are unnecessary anxiety, as well as increased inconvenience and costs caused by follow-up of individuals falsely labelled as positive. The FN students may be unaware of vision abnormality or potential adverse effects this may have on academic performance, as well as a delay of a most probably simple therapeutic intervention. It should be mentioned in this context, however, that despite the fact that vision-screening programmes exist for over 100 years, 24 and although their benefits seem intuitive, the value of such programmes in junior and senior schools, 1 and recently also in pre-schools, has been questioned. 25,26 Comparison of our findings with those reported in the literature reveals different screening methodologies and results. 3,5,8,9
Physicians are often more interested in the predictive value of a test than in its DR and specificity (1-FPR). But what is an acceptable appropriate predictive value? As stated by Moskowitz, 27 each clinician must make that decision within the framework of his experience, the reported literature and the expected impact on the population screened. Unfortunately, only a small number of studies were identified regarding predictive values of vision-screening tools among school children. 3,4,6,16 The PPV in our study was 53% and the NPV was 91%. Higher PPV (96%) and lower NPV (78%) were reported using early-treatment diabetic retinopathy study (ETDRS) charts among children of ages 7–9 years old. 3 It is known that as the disease prevalence increases, the PPV increases and the NPV decreases. 28 This phenomenon was demonstrated when comparing PPV and OAPR, as well as NPV values between 13–14-year-old and 6–7-year-old students. The prevalence of vision abnormality was 1.73 times higher among 13–14-year-old students (unpublished data), while PPV values were higher and NPV lower. We believe that in our district, where the PPV of the test was 53%, the benefits of early detection of vision abnormality may outweigh the risks of being falsely labelled positive.
The project data support the notion to change the vision cut-off level that denotes screening failure to vision 6/12 or worse in both eyes (tested separately). The main reason is the FPR of 22% that would have been decreased to 15% if the vision cut-off level is changed. FN rates, however, will concomitantly increase from 28% to 41%. The importance of low FPR for relatively benign medical conditions (vision abnormality), in contrast to high DR required for severe medical problems (such as cancer) was also noted by others. 3 As presented, the ‘price’ of changing vision cut-off level will mainly affect (reduce) the proportion of myopic and astigmatic students detected by screening. The impact on detection of hypermetropia will be smaller.
The higher FPR among 6–7-year-old students than in the older children is important to consider when planning corrective actions and coping with limited resources. This in part may be due to lack of comprehension of the test in younger children. On one hand, if resources are inadequate, the above data support vision screening only among 13–14-year-old students. This recommendation was also recorded for additional arguments (unpublished data), as well as by others. 28 On the other hand, since vision abnormality was found among 18% of 6–7-year-old students, which in most cases could easily be rectified by appropriate spectacle prescription, we believe vision screening should be continued in both age groups, despite the poorer screening performance in the younger age group and the lower PPV. The lack of impact of nurse seniority on any of the performance measures means students could be screened by either a senior or by a less experienced nurse. This finding allows policy-makers to plan running the system with less costly staff.
Limitations
The Israeli population of the Northern District does not necessarily represent the Israeli population at large. In fact, the study population differed demographically (slightly however, no more than 1–2%) from the intended population. In order to overcome the potential biases in the Northern District estimation of specificity and PPV, attributed to over-representation of 13–14-year-old students and under-representation of Jews, these measures have been dealt with by direct standardization. A negligible impact was noticed. An insignificant impact was also demonstrated when ‘sensitivity analysis’ was carried out to cope with the missing data (120 students) of ophthalmologist decision regarding student's referral. As to the reason for using the E-chart for vision screening, rather than the validated ETDRS chart, 3,29 this is because of the inapplicability of ETDRS to Israeli 6–7-year-old students, who are not yet familiar with the English alphabet.
Further exploration of the following is suggested: (1) the reasons for mild vision-screening performance measures, especially among 6–7-year-old students. (2) factors related to being screened positive or having refractive errors or ocular abnormalities, (3) the potentially adverse effects of falsely diagnosing students as negative or as positive.
Footnotes
ACKNOWLEDGEMENT
This research was supported by administer general of Mrs Esther and Mr Chaim Koppel trust (Study number 5001).