Statistical Misunderstandings of the Properties of School Scores and School Accountability

Brody J. E. (2003, August 12). “Normal” blood pressure: Health watchdogs are resetting the risk. The New York Times, p. F7.

Campbell R. (2002, August 11). API's error margin leaves a lot to chance: Mathematical imprecision could lead to inaccurate interpretations. The Orange County Register. Retrieved January 5, 2005, from http://www.ocregister.com/features/api/text_version/index.shtml

Carlin B. P. , and Louis T. A. (2000). Bayes and empirical Bayes methods for data analysis (2nd ed.). New York: Chapman & Hall.

Council of Chief State School Officers (CCSSO). (2002). Making valid and reliable decisions in determining adequate yearly progress. A paper in the series: Implementing the State Accountability System Requirements under the No Child Left Behind Act of 2001. ASR-CAS Joint Study Group on Adequate Yearly Progress, Scott Marion and Carole White, Co-Chairs. Retrieved December 12, 2004, from http://www.ccsso.org/content/pdfs/AYPpaper.pdf

Deffendall L. (2003, October 19). No statistics are being left behind. The Lexington Herald-Leader.

Gladwell M. (2003, September 15). Making the grade. The New Yorker.

Kane T. J. , and Staiger D. O. (2002). Volatility in school test scores: Implications for test-based accountability systems. In and and Ravitch D. (Ed.), Brookings papers on education policy (pp. 235—269). Washington, DC: Brookings Institution.

Lehman E. L. , and Casella G. (1998). Theory of point estimation (2nd ed.). New York: Springer-Verlag.

Linn R. L. , Baker E. L. , and Herman J. L. (2002, Fall). Minimum group size for measuring adequate yearly progress. The CRESST Line, 1, 4—5.

Linn R. L. , and Haug C. (2002). Stability of school-building accountability scores and gains. Educational Evaluation and Policy Analysis, 24(1), 29—36.

Lynn R. (2003, December 18). Federal mandate: A complex statistical formula allowed them to meet their targets. The Salt Lake Tribune. Retrieved from http://166.70.46.216/2003/dec/12182003/utah/120969.asp

Marion S. , and Gong B. (2003, September). Evaluating the validity of state accountability systems. Paper presented at CCSSO, St. Louis, MO. Retrieved November 12, 2004, from http://www.ccsso.org/content/pdfs/CCSSO_Validity_BGSM03.pdf

No Child Left Behind Act of 2001, Pub. L. No. 107-110, 115 Stat. 1425 (2002).

Novak J. R. , and Fuller B. (2003, December). Penalizing diverse schools? Similar test scores, but different students, bring federal sanctions. PACE Policy Brief 03-4. Retrieved January 3, 2004, from http://pace.berkeley.edu/policy_brief_03-4_Pen.Div.pdf

O'Connell J. (2004, March 24). Fight for changes to NCLB. California Department of Education. Retrieved December 3, 2004, from http://www.cde.ca.gov/eo/ce/sl/nclbfight.asp

Rado D. , and Little D. (2003, September 28). Schools toying with test results: Some states meet standards with art of statistics. The Chicago Tribune, 1, 22.

Rogosa D. R. (2001, October). Year 2000 update: Interpretive notes for the academic performance index. California Department of Education, Policy and Evaluation Division. Retrieved April 1, 2004, from http://www.cde.ca.gov/ta/ac/ap/researchreports.asp

Rogosa D. R. (2002a, September). Commentaries on the Orange County Register series: What's the magnitude of false positives in GPA award programs? Application of OCR “margin of error” to API award programs. California Department of Education, Policy and Evaluation Division. Retrieved April 1, 2004, from http://www.cde.ca.gov/ta/ac/ap/researchreports.asp

Rogosa D. R. (2002b, October). Irrelevance of reliability coefficients to accountability systems: Statistical disconnect in Kane–Staiger “Volatility in School Test Scores.” CRESST deliverable. Retrieved December 5, 2004, from http://www-stat.stanford.edu/∼rag/api/kscresst.pdf

Rogosa D. R. (2002c, July). Plan and preview for API accuracy reports. California Department of Education, Policy and Evaluation Division. Retrieved April 1, 2004, from http://www.cde.ca.gov/ta/ac/ap/researchreports.asp

Rogosa D. R. (2002d, December). Accuracy of API index and school base report elements. California Department of Education, Policy and Evaluation Division. Retrieved April 1, 2004, from http://www.cde.ca.gov/ta/ac/ap/researchreports.asp

Rogosa D. R. (2003a, June). Confusions about consistency in improvement. CRESST deliverable. Retrieved December 5, 2004, from http://www-stat.stanford.edu/∼rag/api/consist.pdf

Rogosa D. R. (2003b, September). Four-peat: Data analysis results from uncharacteristic continuity in California student testing programs. California Department of Education, Policy and Evaluation Division. Retrieved April 1, 2004, from http://www.cde.ca.gov/ta/ac/ap/researchreports.asp

Rogosa D. R. (2003c, October). California's AMOs are more formidable than they appear. California Department of Education, Policy and Evaluation Division. Retrieved April 1, 2004, from http://www.cde.ca.gov/ta/ac/ap/researchreports.asp

Rogosa D. R. (2003d, November). The NCLB “99% confidence” scam: Utah-style calculations. CRESST deliverable. Retrieved December 5, 2004, from http://www-stat.stanford.edu/∼rag/nclb/utahNCLB.pdf

Rogosa D. R. (2004, January). Assessing the effects of multiple subgroups: A rebuttal to the PACE Policy Brief December 2003 “Penalizing diverse schools? Similar test scores, but different students, bring federal sanctions.” California Department of Education, Policy and Evaluation Division. Rejoinder: Being fair to NCLB. Retrieved December 5, 2004, from http://www.cde.ca.gov/ta/ac/ap/researchreports.asp

Rogosa D. R. (2005). A school accountability case study: California API awards and the Orange County Register margin of error folly. In and and Phelps R.P. (Ed.), Defending standardized testing (pp. 205—226). Mahwah, NJ: Lawrence Erlbaum Associates.

Sharon K. , and Tully S. (2002, August 16). State testing expert says API margin of error is insignificant: Leader who helped design index calls it as accurate as possible. The Orange County Register. Retrieved January 5, 2005, from http://www.ocregister.com/features/api/text_version/index.shtml

Sharon K. , Tully S. , and Campbell R. (2002, August 11). Test scores unreliable: Error margin means state can't precisely measure how schools are doing, but the cash still flows. The Orange County Register. Retrieved December 5, 2004, from http://www.ocregister.com/features/api/text_version/index.shtml

Utah State Office of Education. (2003). State of Utah consolidated state application accountability workbook. Plan Approved by U.S. Department of Education, June 10, 2003. Retrieved from http://www.ed.gov/admins/lead/account/stateplans03/index.html