Diagnostic tools can help schools more consistently and fairly match instructional resources to the needs of their students. To ensure the best educational outcome for each child, diagnostic decision-making systems seek to balance time, clarity, and accuracy. However, recent research notes that many educational decisions tend to be made using professional judgment alone. Judgments grounded on data, statistical models, and even informal prediction models, however, outperform those based on intuition alone. The purpose of this manuscript is to describe the theoretical basis for signal detection and methods for statistically evaluating diagnostic decisions in education. We make recommendations to help test developers and consumers apply this methodology to other diagnostic systems in education and interpret the use of signal detection methods for educational screeners and diagnostic tests.
ArdoinS. P.ChristT. J.MorenaL. S.CormierD. C.KlingbeilD. A. (2013). A systematic review and summarization of the recommendations and research surrounding Curriculum-Based Measurement of oral reading fluency (CBM-R) decision rules. Journal of School Psychology, 51, 1–18.
2.
BakerS. K.SmolkowskiK.KatzR.FienH.SeeleyJ. R.Kame‘enuiE. J.Thomas BeckC. (2008). Reading fluency as a predictor of reading proficiency in low-performing, high-poverty schools. School Psychology Review, 37, 18–37.
3.
BakerS. K.SmolkowskiK.SmithJ. M.FienH.Kame‘enuiE. J.Thomas BeckC. (2011). The impact of Oregon Reading First on student reading outcomes. Elementary School Journal, 112, 307–331.
4.
BrooksH. E. (2004). Tornado-warning performance in the past and future: A perspective from signal detection theory. Bulletin of the American Meteorological Society, 85, 837–843.
5.
BurkelR. H.ChiouC.-P.KeyesT. K.MeekerW. Q.RoseJ. H.SturgesD. J.. . . TuckerW. T. (2002). A methodology for the assessment of the capability of inspection systems for detection of subsurface flaws in aircraft turbine engine components (Final Report, DOT/FAA/AR-01/96). Washington, DC: Federal Aviation Administration, Office of Aviation Research, U.S. Department of Transportation.
6.
CohenJ. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.
7.
DanaJ.DawesR. M. (2004). The superiority of simple alternatives to regression for social science predictions. Journal of Educational and Behavioral Statistics, 29, 317–331. doi:10.3102/10769986029003317
8.
DawesR. M. (1962). A note on base rates and psychometric efficiency. Journal of Consulting Psychology, 26, 422–424.
9.
DawesR. M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34, 571–582. doi:10.1037/0003-066X.34.7.571
10.
FleissJ. L. (1981). Statistical methods for rates and proportions (2nd ed.). New York, NY: Wiley.
11.
Florida Department of Education. (2014). Florida Assessments for Instruction in Reading Aligned to the Language Arts Florida Standards: FAIR – FS Grades 3 through 12 Administration Manual. Tallahassee: Author. Retrieved from http://www.fldoe.org/core/fileparse.php/5423/urlt/312AdminManual.pdf
12.
GigerenzerG.GoldsteinD. (1996). Reasoning the fast and frugal way: Models of bounded rationality. Psychological Review, 103, 650–669.
13.
GoldbergL. R. (1970). Man versus model of man: A rational, plus some evidence, for a method of improving on clinical inference. Psychological Bulletin, 73, 422–432.
14.
GoodR. H.KaminskiR. A. (Eds.). (2002). Dynamic Indicators of Basic Early Literacy Skills (6th ed.). Eugene, OR: Institute for the Development of Educational Achievement. Available from http://dibels.uoregon.edu/
15.
GoodR. H.IIIKaminskiR. A.DeweyE. N.WallinJ.Powell-SmithK. A.LatimerR. J. (2011). DIBELS Next technical manual [draft]. Eugene, OR: Dynamic Measurement Group.
16.
GroveW. M.ZaldD. H.LebowB. S.SnitzB. E.NelsonC. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12, 19–30.
17.
Harcourt Educational Measurement. (2002). Stanford Achievement Test [SAT-10]. San Antonio, TX: Author.
18.
HarperR.ReevesB. (1999). Reporting of precision of estimates for diagnostic accuracy: A review. British Medical Journal, 318, 1322–1323.
19.
HedgesL. V.OlkinI. (1985). Statistical methods for meta-analysis. New York, NY: Academic Press.
20.
HintzeJ. M.RyanA. L.StonerG. (2003). Concurrent validity and diagnostic accuracy of the dynamic indicators of basic early literacy skills and the Comprehensive Test of Phonological Processing. School Psychology Review, 32, 541–556.
21.
JenkinsJ. R.HudsonR. F.JohnsonE. S. (2007). Screening for at-risk readers in a response to intervention framework. School Psychology Review, 36, 582–600.
22.
KahnemanD. (2011). Thinking, fast and slow. New York, NY: Farrar, Straus and Giroux.
23.
KatzD.FoxmanB. (1993). How well do prediction equations predict? Using receiver operating characteristic curves and accuracy curves to compare validity and generalizability. Epidemiology, 4, 319–326.
24.
KettlerR. J.Feeney-KettlerK. A. (2011). Screening systems and decision making at the preschool level: Application of a comprehensive validity framework. Psychology in the Schools, 48, 430–441. doi:10.1002/pits.20565
25.
KlooA.ZigmondN. (2008, February). Implementing progress monitoring in a really low achieving school among very low-skilled teachers. Paper presented at the 2008 annual Pacific Coast Research Conference, Coronado, CA.
26.
KraemerH. (1992). Evaluating medical tests: Objective and quantitative guidelines. Newbury Park, CA: SAGE.
27.
LynnS. K.BarrettL. F. (2014). “Utilizing” signal detection theory. Psychological Science, 25, 1663–1673. doi:10.1177/0956797614541991
MalhotraR.IndrayanA. A. (2010). A simple nomogram for sample size for estimating sensitivity and specificity of medical tests. Indian Journal of Ophthalmology, 58, 519–522. doi:10.4103/0301-4738.71699
30.
MartinS. D.ShapiroE. S. (2011). Examining the accuracy of teachers’ judgments of DIBELS performance. Psychology in the Schools, 48, 343–356.
31.
MasonS. J.GrahamN. E. (1999). Conditional probabilities, relative operating characteristics, and relative operating levels. Weather and Forecasting, 14, 713–725.
32.
McAlenneyA. L.CoyneM. D. (2015). Addressing false positives in early reading assessment using intervention response data. Learning Disability Quarterly, 38, 53–65.
33.
McGrathR. E. (2008). Predictor combination in binary decision-making situations. Psychological Assessment, 20, 195–205. doi:10.1037/a0013175
34.
MeehlP. E. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. Minneapolis: University of Minnesota Press.
35.
NelsonJ. M. (2008). Beyond correlational analysis of the Dynamic Indicators of Basic Early Literacy Skills (DIBELS): A classification validity study. School Psychology Quarterly, 23, 542–552.
36.
NeymanJ.PearsonE. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London, 231, 289–337.
37.
PepeM. S. (2003). The statistical evaluation of medical tests for classification and prediction. New York, NY: Oxford University Press.
38.
PetersonW. W.BirdsallT. G.FoxW. C. (1954). The theory of signal detectability. IRE Professional Group on Information Theory, 4, 171–212.
39.
PetscherY.KimY.-S.FoormanB. R. (2011). The importance of predictive power in early screening assessments: Implications for placement in the response to intervention framework. Assessment for Effective Intervention, 36, 158–166.
40.
RiceM. E.HarrisG. T. (2005). Comparing effect sizes in follow-up studies: ROC area, Cohen’s d, and r. Law and Human Behavior, 29, 615–620.
41.
RitchieS. J.BatesT. C. (2013). Enduring links from childhood mathematics and reading achievement to adult socioeconomic status. Psychological Science, 24, 1301–1308. doi:10.1177/0956797612466268
42.
RobertsR.LewinsohnP.SeeleyJ. (1991). Screening for adolescent depression: A comparison of depression scales. Journal of the American Academy of Child & Adolescent Psychiatry, 30, 58–66.
43.
SalviaJ.YsseldykeJ. E.BoltS. (2013). Assessment: In special and inclusive education. Belmont, CA: Wadsworth.
SeeleyJ. R.SticeE.RohdeP. (2009). Screening for depression prevention: Identifying adolescent girls at high risk for future depression. Journal of Abnormal Psychology, 118, 161–170. doi:10.1037/a0014741
46.
SilberglittB.HintzeJ. (2005). Formative assessment using CBM-R cut scores to track progress toward success on state-mandated achievement tests: A comparison of methods. Journal of Psychoeducational Assessment, 23, 304–325.
47.
SlovicP.RorerL.HoffmanP. (1971). Analyzing use of diagnostic signs. Investigative Radiology, 6, 18–26.
48.
StreinerD. L.CairneyJ. (2007). What’s under the ROC? An introduction to receiver operating characteristics curves. Canadian Journal of Psychiatry, 52, 121–128.
49.
SwetsJ. A. (1973). The relative operating characteristic in psychology. Science, 182, 990–1000.
50.
SwetsJ. A. (1988). Measuring the accuracy of diagnostic systems. Science, 240, 1285–1293.
51.
SwetsJ. A. (1996). Signal detection theory and ROC analysis in psychology and diagnostics: Collected papers. Hillsdale, NJ: Lawrence Erlbaum.
52.
SwetsJ. A.DawesR. M.MonahanJ. (2000). Psychological science can improve diagnostic decisions. Psychological Science in the Public Interest, 1, 1–26.
53.
WagnerR. K.TorgesenJ. K.RashotteC. A. (1999). CTOPP: Comprehensive Test of Phonological Processing. Austin, TX: Pro-Ed.
54.
YoudenW. J. (1950). Index for rating diagnostic tests. Cancer, 3, 32–35.
55.
ZhouX.-H.McClishD. K.ObuchowskiN. A. (2002). Statistical methods in diagnostic medicine. New York, NY: Wiley.
