Introduction

Amblyopia, affecting at least 2% of the global population, is a prevalent vision disorder that can lead to irreversible vision impairment if not addressed promptly.¹ Instrument-based vision screening, utilizing autorefraction or photorefraction, plays a pivotal role in detecting amblyopia risk factors (ARFs) and visually significant refractive errors (VSRE). The evolution of these technologies has prompted the American Association for Pediatric Ophthalmology and Strabismus (AAPOS) to establish standardized guidelines, shaping device development and application.² Recognizing the issue of excessive referrals for non-amblyopic, symmetric astigmatism, the 2013 guidelines was updated in 2021. This update aimed to enhance diagnostic precision by adjusting symmetric astigmatism and amblyogenic anisometropia criteria.³

Over the past decade, the Spot Vision Screener has become an important tool in pediatric eye care. Several studies, adapting to the revised AAPOS 2021 guidelines, have proposed updated referral criteria for the Spot device.^4,5 Nevertheless, the diversity in study populations has complicated the selection of appropriate local referral criteria, highlighting the need to compare and validate of these criteria within consistent population samples in order to make comparisons conveniently. However, this has not yet been reported. Therefore, this study seeks to refine and validate the Spot Vision Screener’s referral criteria for detecting ARFs, VSRE, and amblyopia, aligning with the AAPOS 2021 guidelines in a pediatric ophthalmology clinic setting.

Method

Study Population

This study selected children visiting the Maternal and Child Healthcare Clinic pediatric ophthalmology clinics in Yuhuatai District, Nanjing, China, from January 2022 to May 2023. Yuhuatai District, a region with medium socioeconomic status in Eastern China, is one of Nanjing’s 11 municipal districts. The inclusion criteria were: (1) children aged 12 months to 7 years; (2) undergoing both refractive screening and cycloplegic retinoscopy (CR). The exclusion criteria included: (1) inability to use atropine ophthalmic gel; (2) lack of essential medical information, such as children referred directly to higher-tier hospitals for severe strabismus or syndromes.

This study received approval from the Ethics Committee of the First Affiliated Hospital with Nanjing Medical University (No. 2017-SR-205), adhering to the Declaration of Helsinki principles. Written informed consent was obtained from each child’s parents or legal guardians. Children aged four years and above gave oral consent immediately before the examination.

Examinations

Qualified optometrists performed various examinations, including visual behaviour assessment, distance visual acuity using the linear HOTV logMAR VA chart (Good-Lite, Elgin, Illinois, USA), anterior segment and fundus examination, usage of the Spot Vision Screener (Welch Allyn, Skaneateles Falls, New York, software versions V3.0.04.07), evaluation of ocular alignment and motility, and refractive status post-cycloplegia.

For the Spot Vision Screener test, the device was positioned 1 meter away from the non-cycloplegic child under dim lighting. Both eyes were examined simultaneously as the child focused on a fixation target. This instrument measured spherical and cylindrical values ranging from −7.50D to +7.50D in 0.25D increments.

Following this, parents applied 1.0% atropine sulfate ointment (Shenyang Xingqi Eye Medicine Co, Ltd.) thrice daily for three days. Children returned for cycloplegic retinoscopy on the fourth day using a hand-held retinoscope. All children in this study had their medications reviewed before receiving 1% atropine, and received guidance and monitoring during the study to ensure patient safety. In our study, we did not observe any significant adverse effects related to 1% atropine.

Definition

The study adopted the 2021 AAPOS guidelines to define cycloplegic confirmatory examination failure levels: hyperopia > +4.00 D, anisometropia > 1.25 D, myopia < −3.00 D (children <48 months) or < −2.00 D (children ≥48 months), and astigmatism > 3.00 D (<48 months) or > 1.75 D (≥48 months).³ For children ≥ 4 years old, amblyopia was defined as a best-corrected monocular visual acuity of 20/40 (logMAR 0.3) or worse, or an inter-eye difference of 2 or more lines with ARFs present.^3,6 For children under 4, amblyopia diagnosis was based on response to monocular occlusion, fixation preference, and preferential looking with ARFs.

Data Analysis

The study divided children into two groups: those under four years and those four years or older. Excluding anisometropia, only right eye data were analyzed to prevent enantiomorphism bias. Data analysis utilized IBM SPSS V13.0 (Chicago, IL, USA), with P<0.05 indicating statistical significance. Descriptive data were expressed as mean ± standard deviation (SD). Chi-square analysis was used to compare the actual number of children receiving CR diagnosed with a visual problem with the number of children receiving SPOT screening who would have been referred for those same problems using the 2021 AAPOS cut-offs. Paired t-test analysis and Bland-Altman plots assessed differences and agreement between SPOT and CR.

The study evaluated the SPOT’s ability to identify several ARFs and VSRE through receiver operating characteristic (ROC) curves, using CR as a reference and comparing with the 2021 AAPOS age-based referral criteria. Optimal cut-off points were established to derive the “Nanjing referral criteria”. This new criterion’s Sensitivity (Se), Specificity (Sp), Youden index, positive predictive value (PPV), and negative predictive value (NPV) for detecting “ARFs & VSRE” and “ARFs & VSRE & amblyopia” were then compared with previously reported referral criteria.

Results

In this study, 1103 children, with an average age of 4.44 ± 1.33 years, participated. The cohort comprised 547 boys and 556 girls, including 347 children under four years and 756 aged four years or older. Applying the 2021 AAPOS guidelines’ referral criteria using CR, 308 children were diagnosed with refractive ARFs or VSRE. This group included 110 with hyperopia, 46 with myopia, 136 with astigmatism, and 94 with anisometropia. Several children exhibited more than one type of refractive ARF and VSRE. While applying the 2021 AAPOS guidelines’ referral criteria using SPOT, 434 children were diagnosed with refractive ARFs or VSRE. This group included 29 with hyperopia, 63 with myopia, 318 with astigmatism, and 81 with anisometropia. Several children exhibited more than one type of refractive ARF and VSRE. The study found that using the 2021 AAPOS cut-offs, comparing to CR, SPOT tended to underestimate hyperopia (χ² = 50.38, P < 0.001) and overestimate astigmatism (χ² = 91.87, P < 0.001). Examination using SPOT may lead to more children referred than examination (χ² = 32.24, P < 0.001) (Table 1).

Table 1 Comparison the Number of Children Diagnosed with a Visual Problem of SPOT and CR Using 2021 AAPOS Guidelines

The study revealed that compared to CR, SPOT tended to underestimate both spherical (+1.20 ± 1.32D vs +1.94 ± 1.88D, P < 0.001) but to overestimate cylindrical values (−1.70 ± 0.94D vs −1.20 ± 0.81D, P < 0.001). The 95% limits of agreement (LOA) showed a range of −1.48 to +2.95 D (bias of 0.74 D) for spherical values and −0.69 to +1.68 D (bias of 0.49 D) for cylindrical values between CR and SPOT (Table 2 and Figure 1).

Table 2 Comparison of SPOT and Cycloplegic Retinoscopy in 1103 Children

Figure 1 Bland-Altman plots of agreement between the Spot Vision Screener and cycloplegia retinoscopy (CR). The dots represent individuals. The horizontal imaginary green line represents mean difference values between the two methods. The two full green lines represent upper limit and lower limit of the 95% confidence interval (95% CI). (A) Bland-Altman plots of sphere values. (B) Bland-Altman plots of cylinder values.

Using ROC curve analysis, we established optimal cut-off points (Table 3): +2.13D for hyperopia, −2.88D for myopia, −2.88D for astigmatism, and 0.88D for anisometropia in children under four; +1.88D for hyperopia, −1.88D for myopia, −2.13D for astigmatism, and 0.88D for anisometropia in children aged four years or older. Since it is necessary to consider the performance of the criteria for each visual disorder, these cut-off points were combined to form the basis of the newly established Nanjing referral criteria. Given that the Spot screener measures spherical and cylindrical values in 0.25D increments, so the optimal cut-off points in practical application: +2.25D for hyperopia, −3.00D for myopia, −3.00D for astigmatism, and 1.00D for anisometropia in children under four; +2.00D for hyperopia, −2.00D for myopia, −2.25D for astigmatism, and 1.00 D for anisometropia in children aged four years or older.

Table 3 Characteristics of ROC Curves for ARFs and VSRE Compared with Cycloplegic Retinoscopy

We also calculated the performance of the Nanjing referral criteria for each visual disorder separately. As shown in Table 3, SPOT demonstrated the highest accuracy in detecting myopia in both age groups (< four years old: AUC=1.00, Youden index=0.98; ≥ four years old: AUC=0.96, Youden index=0.83). For hyperopia, SPOT showed similar accuracy in both age groups (< four years old: AUC=0.90, Youden index=0.63; ≥ four years old: AUC=0.90, Youden index=0.69). Detection accuracy for astigmatism was higher in children under four years old (AUC=0.95, Youden index=0.81) compared to those aged four years or older (AUC=0.88, Youden index=0.60). Conversely, SPOT was more accurate in detecting anisometropia in the older age group (≥ four years old: AUC=0.88, Youden index=0.68) than in the younger group (< four years old: AUC=0.81, Youden index=0.49).

Table 4 shows the performance characteristics of different reported referral criteria for detecting “ARFs & VSRE” and “ARFs & VSRE & amblyopia” based on the failure levels from AAPOS 2021 guidelines, including the Nanjing referral criteria, the Arnold referral criteria and the Peterseim referral criteria.^4,5 In detecting ARFs & VSRE, the Youden index of different criteria ranged from 0.08 to 0.53, in which the Nanjing referral criteria had the second-best Youden index (0.48) with a Se of 86.04% and a Sp of 62.26%. In detecting ARFs & VSRE & amblyopia, the Youden index of different criteria ranged from 0.09 to 0.50, in which the Nanjing referral criteria also had the second-best Youden index (0.45) with a Se of 82.49% and a Sp of 62.53%. The Peterseim referral criteria had the highest Se but the lowest Sp and Youden index. The Arnold referral criteria had the best Youden index and specificity in both two groups.

Table 4 Performance Characteristics of Different Referral Criteria of the Spot Vision Screener for Detecting ARFs, VSRE and Amblyopia

To enhance specificity, we analyzed misdiagnosed cases. Anisometropia was the primary misdiagnosis in children under four years (54.48%, 73/134), while astigmatism was more common in children aged four years or older (58.43%, 97/166). The point on the ROC curve that is closest to the top-left corner, representing the optimal cut-off between sensitivity and specificity, was selected as the specific threshold (< 4 years old: 1.13D for anisometropia; ≥ 4 years old: −2.38D for astigmatism, that is, < 4 years old: 1.25D for anisometropia; ≥ 4 years old: −2.50D for astigmatism). This adjustment led to the modified Nanjing criteria, achieving a Youden index of 0.52 with a specificity of 71.45% for detecting ARFs & VSRE, and a Youden index of 0.48 with a specificity of 71.67% for identifying ARFs, VSRE & amblyopia. (Table 4).

Discussion

Our study assessed the Spot Vision Screener’s efficacy in diagnosing “ARFs & VSRE” and “ARFs & VSRE & amblyopia” in 1103 Chinese children. We also evaluated existing referral criteria and introduced new criteria based on our findings.

The need to revise guidelines arose from the high over-referral rates for non-amblyopic, symmetric astigmatism. The 2021 AAPOS guidelines, acknowledging this issue, introduced meridional refractive power concepts and redefined failure levels and age group classifications. A previous population-based study compared the 2013 and 2021 AAPOS guidelines, noting similar referral rates in children aged 61 to 72 months.⁷

In our study, we observed that the SPOT underestimates mean spherical values (+1.20 ± 1.32 D) compared to CR (+1.94 ± 1.88 D). Additionally, the cylindrical value recorded by SPOT was higher (−1.70 ± 0.94 D vs −1.20 ± 0.81 D). These discrepancies, particularly the 0.50 D difference in cylinder value, are crucial as the 2021 guidelines use meridional refractive power for calculating hyperopia, myopia, and anisometropia, which differs from the previous guidelines that used half of the cylinder values for calculating spherical equivalent (SE). Thus, recalculating new referral criteria based on these updated guidelines is essential for effective vision screening.

ROC curves analysis demonstrated that SPOT performs well across all ARFs and VSRE (< four years old, 0.81 to 1.00; ≥ four years old, 0.88 to 0.96). Myopia displayed the highest AUC and Youden index, aligning with the guidelines for myopic VSRE (Figure 2 and Table 3). Given the increasing prevalence of myopia worldwide, especially in East Asia,^8,9 it’s important to screen for myopia as a VSRE, even though it is no longer considered an ARF.

Figure 2 Receiver operating characteristic (ROC) curve for refractive results obtained by comparison between the Spot Vision Screener and CR. Colorful bold lines represent the ROC curve, the imaginary line connected by dots is used for comparison and makes no sense. Black larger circle dots were used to create the Nanjing referral criteria. (A) Hyperopia < 4 years old; (B) Myopia < 4 years old; (C) Astigmatism < 4 years old; (D) Anisometropia < 4 years old; (E) Hyperopia ≥ 4 years old; (F) Myopia ≥ 4 years old; (G) Astigmatism ≥ 4 years old; (H) Anisometropia ≥ 4 years old.

We propose “Nanjing referral criteria” based on the 2021 AAPOS guidelines, which showed high sensitivity (86.04%) and satisfactory specificity (62.26%). We further developed a modified version with higher specificity to address the decrease in specificity, primarily due to anisometropia misdiagnosis in younger children and astigmatism in older children. The modified criteria improved specificity from 62.26% to 71.45% while maintaining reasonable sensitivity of 80.52%, addressing the balance between over- and under-referral (Table 4). While increase in sensitivity at a cost of reduction in specificity may appear marginal, in the context of Nanjing’s healthcare system, this balance is crucial. It’s a balance between identifying most of the true positives while minimizing the burden on healthcare systems and preventing anxiety for individuals who do not need further evaluation, which is vital in a region where follow-up resources may be limited.

Several studies have been published with revised referral criteria for different devices, including SPOT,^4,5 2WIN,⁴ Suresight⁷ and PlusoptiX^4,7 based on the AAPOS 2021 guidelines. Table 4 shows the performance characteristics of different referral criteria of SPOT for detecting ARFs, VSRE and amblyopia in the 1103 children. To the best of our knowledge, this is the first study to compare different referral criteria in the same large study population sample. The first previous study was carried out by Arnold et al and consisted of 755 children from 3 cohorts (30 < 4 years old, 725 ≥ 4 years old). The children in this study were older, with a mean age of 9 years old. Since the number of children < 4 years old (9 with ARFs) is low, the referral criteria for this age group in particular need further validation. We validated the accuracy of the Arnold referral criteria in both age groups in 1103 children aged 1–7 years. It had the highest Youden index (0.53 for ARFs & VSRE, 0.48 for ARFs & VSRE & Amblyopia) among different criteria, but it is worth noting that hyperopia in the Arnold referral criteria is defined according to spherical SE, which differs from the guidelines.⁴ Peterseim et al reported the other study, including 1090 children aged 6 months old to 13 years old with an ARFs prevalence of 24%.⁵ The Peterseim referral criteria had the most liberal threshold for each refractive error, causing the best Se and NPV but the lowest Sp, PPV and Youden index, suggesting that it may be more appropriate for older children with higher Se requirements. The accuracy of all referral criteria was mildly reduced after including amblyopia as a detecting target.

The results presented separately according to age groups can provide a practical reference for vision screening. The efficiency of screening could be improved by applying instrument- and age-specific pass/fail criteria based on the population characteristics, economic situation and frequency of screening. The comparative data on different referral criteria provided in this study can provide a reference for selection.

This study’s strengths lie in its validation of various criteria within the same demographic, achieving the most extensive sample size to date, and maintaining an appropriate age range that sufficiently covers children under 4 years old while excluding those over 10 years old, who are less susceptible to amblyopia. However, it does have limitations. Notably, it excludes children with severe strabismus or syndromes, often directly referred from maternal and child healthcare clinics to specialized eye hospitals. Additionally, the validation results obtained in a clinical setting may not fully translate to general population screenings. The study did not explicitly outline the limitations of the AAPOS 2021 guidelines. Fine-tuning the thresholds to reflect the local screening population’s characteristics and incorporating the comparative insights from this study could enhance the precision of the criteria. The specificity of 71% obtained with the new referral criteria is still not very high enough for a refractive screening. While considering the sensitivity of the referral criteria, further research is needed on technological instruments and other methods to improve the specificity of the referral criteria.

In conclusion, we propose the “Nanjing Referral Criteria” and its modified version with enhanced specificity for the Spot Vision Screener to optimize vision screening in children. It’s crucial for vision screening programs to evaluate desired sensitivity and specificity to select the most appropriate referral criteria. Vision screening programs can utilize this validated data to select referral criteria best suited to their local context.