Are More Stringent NCLB State Accountability Systems Associated With Better Student Outcomes? An Analysis of NAEP Results Across States

Abstract

This study developed a comprehensive measure of the stringency level of NCLB states’ accountability systems, including the strength of their annual measurable objectives, confidence intervals, performance indexing, retesting, minimum subgroup size, and the difficulty levels of proficiency standards. This study related accountability stringency in 2003 to student achievement and achievement gaps on NAEP math and reading tests from 2003 and 2005. The results were inconsistent across grades, subjects, and ethnic groups. An increase in accountability stringency, such as requiring faster progress, not allowing for retesting, and allowing a smaller minimum subgroup size were related to improved math achievement for fourth-grade Hispanic students. Not using confidence intervals was related to higher math and reading achievement for White and Hispanic students. However, accountability stringency did not have any positive effects and even negative effects on eighth-grade Black students.

Keywords

NCLB accountability stringency the strength of annual measurable objectives confidence intervals performance indexing retesting minimum subgroup size proficiency standards student achievement achievement gap NAEP

Get full access to this article

View all access options for this article.

References

Amrein

A. L.

Berliner

D. C.

(2002a). High-stakes testing, uncertainty, and student learning. Education Policy Analysis Archives, 10(18). Retrieved August 6, 2009, from http://epaa.asu.edu/epaa/v10n18/

Amrein

A. L.

Berliner

D. C.

(2002b). An analysis of some unintended and negative consequences of high-stakes testing. Tempe: Education Policy Studies Laboratory, Arizona State University.

Anagnostopoulos

(2006). “Real students” and “true demotes”: Ending social promotion and the moral ordering of urban high schools. American Educational Research Journal, 43(1), 5-42.

Betts

J. R.

Costrell

R. M.

(2001). Incentives and equity under standards-based reform. In Ravitch

(Ed.), Brookings papers on education policy: 2001 (pp. 9-74). Washington, DC: Brookings Institution Press.

Booher-Jennings

(2005). Below the bubble: “Educational Triage” and the Texas accountability system. American Educational Research Journal, 42, 231-268.

Braun

(2004). Reconsidering the impact of high-stakes testing. Education Policy Analysis Archives, 12(1). Retrieved August 6, 2009, from http://epaa.asu.edu/epaa/v12n1/

Carey

(2007, November 1). The Pangloss Index: How states game the No Child Left Behind Act. Education Sector. Retrieved August 6, 2009, from http://www.educationsector.org/usr_doc/The_Pangloss_Index.pdf

Carnoy

(2005). Have state accountability and high-stakes tests influenced student progression rates in high school? Educational Measurement: Issues and Practice, 24(4), 19-31.

Carnoy

Loeb

(2002). Does external accountability affect student outcomes? A cross-state analysis. Educational Evaluation and Policy Analysis, 24, 305-331.

10.

Chudowsky

(2005). States test limits of federal AYP flexibility. Washington, DC: Center on Education Policy.

11.

Darling-Hammond

(2004). Standards, accountability, and school reform. Teachers College Record, 106, 1047-1085.

12.

Dee

Jacob

(2009). The impact of No Child Left Behind on student achievement (NBER working paper 15531). Cambridge, MA: National Bureau of Economic Research.

13.

Desimone

Thomas

Hayes

Frisvold

(2005). Beyond accountability and average mathematics scores: Relating state education policy attributes to cognitive achievement domains. Educational Measurement: Issues and Practice, 24(4), 5-18.

14.

Diamond

J. B.

Cooper

(2007). The uses of testing data in urban elementary schools: Some lessons from Chicago. In Moss

P. A.

(Ed.), Yearbook of the National Society for the Study of Education: Vol. 106, Part 1. Evidence and decision-making (pp. 241-263). Chicago: National Society for the Study of Education.

15.

The Education Trust. (2005, October 19). Closing the achievement gap: 2005 NAEP reading and mathematics results show some gains, but slowing progress. Retrieved October 25, 2010, from http://www.edtrust.org/dc/press-room/press-release/closing-the-achievement-gap-2005-naep-reading-and-math-results-show-some.

16.

Figlio

D. N.

Rouse

C. E.

(2006). Do accountability and voucher threats improve low performing schools? Journal of Public Economics, 90, 239-255.

17.

Freudenberg

(2006, April 17). 2 million scores ignored in “No Child” loophole. Retrieved from August 10, 2008, from http://www.msnbc.msn.com/id/12357165/from/RSS/

18.

Grissmer

D. W.

Flanagan

Kawata

J. H.

Willianson

(2000). Improving student achievement: What state NAEP test scores tell us. Santa Monica, CA: RAND.

19.

Goertz

M. E.

(2005). Implementing the No Child Left Behind Act: Challenges for the states. Peabody Journal of Education, 80(2), 73-89.

20.

Hanushek

E. A.

Raymond

M. E.

(2003a). Improving educational quality: How best to evaluate our schools? In Kodrzycki

(Ed.), Education in the 21st century: Meeting the challenges of a changing world (pp. 193-224). Boston: Federal Reserve Bank of Boston.

21.

Hanushek

E. A.

Raymond

M. E.

(2003b). Lessons about the design of state accountability systems. In Peterson

P. E.

West

M. R.

(Eds.), No child left behind? The politics and practice of accountability (pp. 127-151). Washington, DC: Brookings Institution Press.

22.

Hanushek

E. A.

Raymond

M. E.

(2004). The effect of school accountability systems on the level and distribution of student achievement. Journal of the European Economic Association, 2, 406-415.

23.

Hanushek

E. A.

Raymond

M. E.

(2005). Does school accountability lead to improved student performance? Journal of Policy Analysis and Management, 24, 197-327.

24.

Jacob

(2005). Accountability, incentives and behavior: The impact of high-stakes testing in the Chicago Public Schools. Journal of Public Economics, 89, 761-796.

25.

Koretz

D. M.

Hamilton

L. S.

(2003). Teachers’ responses to high-stakes testing and the validity of the gains: A pilot study (CSE Technical Report 610). Los Angeles: CRESST/University of California.

26.

McLaughlin

D. H.

Bandeira

Mello

Blankenship

Chaney

Esra

Hikawa

. (2008a). Comparison between NAEP and state reading assessment results: 2003 (NCES 2008-474). Washington, DC: National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education.

27.

McLaughlin

D. H.

Mello

Blankenship

Chaney

Esra

Hikawa

. (2008b). Comparison between NAEP and State Mathematics Assessment Results: 2003 (NCES Rep. No. 2008-475). Washington, DC: National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education.

28.

McNeil

Valenzuela

(2000). The harmful impact of the TAAS system of testing in Texas. Cambridge, MA: Civil Rights Project at Harvard University.

29.

Mosteller

Boruch

(2002). Evidence matters: Randomized trials in education research. Washington, DC: Brookings Institution.

30.

New York State Department of Education. (2002). Understanding your school report card 2002: Guide to elementary and middle school Assessments. Retrieved October 25, 2010, from http://www.p12.nysed.gov/repcrd2002/k8guide.html

31.

No Child Left Behind Act of 2001, Pub. L. No. 107-110 (2002).

32.

Perna

L. W.

Thomas

S. L.

(2009). Barriers to college opportunity: The unintended consequences of state-mandated testing. Educational Policy, 23, 451-479.

33.

Peterson

P. E.

Hess

F. M.

(2006). Keeping an eye on state standards. A race to the bottom? Education Next, 3, 28-29.

34.

Porter

Linn

R. L.

Trimble

C. S.

(2005). The effects of the state decisions about NCLB adequate yearly progress targets. Educational Measurement: Issues and Practice, 24(4), 32-39.

35.

Raymond

M. E.

Hanushek

E. A.

(2003). High-stakes research. Education Next, 3(3), 48-55.

36.

Rogosa

D. R.

(2003). The NCLB “99% confidence” scam: Utah-style calculations. Retrieved August 6, 2009, from http://www-stat.stanford.edu/~rag/nclb/utahNCLB.pdf

37.

Rosenshine

(2003). High-stakes testing: Another analysis. Education Policy Analysis Archives, 11(24). Retrieved August 6, 2009, from http://epaa.asu.edu/epaa/v11n24/

38.

Serow

R. C.

(1984). Effects of minimum competency testing for minority students: A review of expectations and outcomes. Urban Review, XVI, 67-75.

39.

Shepard

(1990). Inflated test score gains: Is the problem old norms or teaching the test? Educational Measurement: Issues and Practice, 9(3), 15-22.

40.

Simpson

M. A.

Gong

Marion

(2006). Effect of minimum cell sizes and confidence interval sizes for special education subgroups on school-level AYP determinations (Synthesis Report 61). Minneapolis: University of Minnesota, National Center on Educational Outcomes. Retrieved August 6, 2009, from http://education.umn.edu/NCEO/OnlinePubs/Synthesis61.html

41.

Steele

C. M.

(1997). A threat in the air: How stereotypes shape intellectual identity and performance. American Psychologist, 52, 613-629.

42.

Steele

C. M.

Spencer

S. J.

Aronson

(2002). Contending with group image: The psychology of stereotype and social identity threat. In Zanna

M. P.

(Ed.), Advances in experimental social psychology (Vol. 34, pp. 379-440). New York: Academic Press.

43.

Stiefel

Schwartz

A. E.

Chellman

C. C.

(2007). So many children left behind: Segregation and the impact of subgroup reporting in No Child Left Behind on the racial test score gap. Educational Policy, 21, 527-550.

44.

U.S. Department of Education. (2006). No Child Left Behind. Growth model: Ensuring grade-level proficiency for all students by 2014. Retrieved August 3, 2009, from http://www.ed.gov/admins/lead/account/growthmodel/proficiency.html

45.

What Works Clearinghouse. (2008). What Works Clearinghouse procedures and standards handbook (Version 2.0). Washington, DC: Author. Retrieved September 29, 2010, from http://ies.ed.gov/ncee/wwc/pdf/wwc_procedures_v2_standards_handbook.pdf

46.

Watanabe

(2007). Displaced teacher and state priorities in a high-stakes accountability context. Educational Policy, 21, 311-368.