Restricted accessResearch articleFirst published online 2016-1
Do decision rules matter? A descriptive study of English language proficiency assessment classifications for English-language learners and native English speakers in fifth grade
English language proficiency assessments (ELPA) are used in the United States to measure annually the English language progress and proficiency of English-language learners (ELLs), a subgroup of language minority students who receive language acquisition support mandated and largely funded by Title III (NCLB, 2001). ELPA proficient and non-proficient classifications are determined by applying decision rules to combine the sub-domains of listening, speaking, reading, and writing in a conjunctive, compensatory, mixed or complementary manner in order that an ELP performance standard can be set. Although the ELP performance standard is used to set accountability objectives for federal reporting, it also is used to reveal students’ readiness for exit from English language services. This study operationalizes and tests the ELP performance standard for student-level decision making by describing to what extent students are classified as non-proficient under different models and rules and the effect of these differences on their eligibility for redesignation. Test performances from one state’s ELPA were gathered from a statewide sample of ELL (n = 875) and randomly selected sample of native English speaker students (non-ELL, n = 92) in fifth grade. Findings indicate sizable differences in non-proficient classifications for ELLs, non-ELLs, and a constructed subgroup of academically high-performing students. There were also observed differences in redesignation eligibility in all groups suggesting that choice of model and decision rule can extend the length of time even high-performing students spend in English language services. Discussion includes implications for validation of high-stakes classification systems.
AbediJ. (2004). The No Child Left Behind Act and English language learners: Assessment and accountability issues. Educational Researcher, 33(1), 4–14.
2.
AbediJ. (Ed.) (2007). English language proficiency assessment in the nation: Current status and future practice. Davis, CA: UC Davis School of Education.
3.
AbediJ. (2008a). Classification system for English language learners: Issues and recommendations. Educational Measurement: Issues and Practice, 27(3), 17–31.
4.
AbediJ. (2008b). Measuring students’ level of English proficiency: Educational significance and assessment requirements. Educational Assessment, 13(2/3), 193–214.
5.
AbediJ. (2013). Testing of English language learner students. In GeisingerK. F. (Ed.), APA Handbook of testing and assessment in psychology: Vol. 3, Testing and assessment in school psychology and education (pp. 355–368). Washington, DC: American Psychological Association.
6.
AbediJ.HermanJ. (2010). Assessing English language learners’ opportunity to learn mathematics: Issues and limitations. Teachers College Record, 112(3), 723–746.
7.
American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME] (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
8.
AugustD.E.ShanahanT.E. (2006). Developing literacy in second-language learners: Report of the National Literacy Panel on Language-Minority Children and Youth. Mahwah, NJ: Lawrence Erlbaum.
9.
BaileyA.L.CarrollP.E. (2015). The assessment of English language learners in the era of new academic content standards. Review of Research in Education, 39, 253–294.
10.
BachmanL.F. (2005). Building and supporting a case for test use. Language Assessment Quarterly, 2(1), 1–34.
11.
BachmanL.F.PalmerA. (2010). Language assessment in practice. New York: Oxford University Press.
12.
BaileyA.L. (2007). The language demands of school: Putting academic English to the test. New Haven, CT: Yale University Press.
13.
BaileyA.L.ButlerF.A. (2003). An evidentiary framework for operationalizing academic language for broad application to K-12 education: A design document. CSE Report 611. Center for the Study of Evaluation, National Center for Research on Evaluation, Standards, and Student Testing, Graduate School of Education & Information Studies,University of California, Los Angeles.
14.
BaileyA.L.ButlerF.A. (2004). Ethical considerations in the assessment of the language and content knowledge of U.S. school-age English learners. Language Assessment Quarterly, 1(2–3), 177–193.
15.
BaileyA.L.HeritageM. (2010). English language proficiency assessment foundations: External judgments of adequacy. Enhanced Assessment Grant US DOE Evaluating the Validity of English Assessments (EVEA) Project deliverable. Retrieved from www.eveaproject.com
16.
BaileyA.L.KellyK.R. (2013). Home language survey practices in the initial identification of English learners in the United States. Educational Policy, 27, 770–804.
17.
BaileyA.L.WolfM.K. (2012). The challenge of assessing language proficiency aligned to the Common Core State Standards and some possible solutions. Understanding Language Initiative, Commissioned Paper, Stanford University, CA. Retrieved from http://www.ell.stanford.edu
18.
California Department of Education (CDE). (2011). A comparison study of kindergarten and grade 1 English-fluent students and English learners on the 2010–11 edition of the CELDT (October25, 2011).
19.
CallahanR. (2005). Tracking and high school English learners: Limiting opportunities to learn. American Educational Research Journal, 42, 305–328.
20.
CarrollP.E.BaileyA.L. (2012, November). Examining conjunctive and compensatory classification models for an English language proficiency assessment: A descriptive study of English learners and native English speakers. Paper presented at the Conference to Honor Ronald Hambleton, University of Massachusetts, Amherst, MA.
21.
Castro-OlivoS. M.PreciadoJ.A.SanfordA.K.PerryV. (2011). The academic and social-emotional needs of secondary Latino English learners: Implications for screening, identification, and instructional planning. Exceptionality: A Special Education Journal, 19(3), 160–174.
22.
ChapelleC.A. (2012). Validity argument for language assessment: The framework is simple… . Language Testing, 29(1), 19–27.
23.
ChapelleC. A.EnrightM. K.JamiesonJ. M. (Eds.). (2011). Building a validity argument for the Test of English as a Foreign Language TM. New York: Routledge.
24.
ChesterM.D. (2003). Multiple measures and high-stakes decisions: A framework for combining measures. Educational Measurement: Issues in Practice, 22(2), 32–41.
25.
CookH.G. (2009). Annual Measurable Achievement Objective #2 (AMAO 2): A Report on a Meeting Comparing the Kentucky Core Content Test (KCCT) and ACCESS for ELLsTM to Establish an AMAO 2 Exit Criterion. Kentucky Department of Education.
26.
CookH. G.LinquantiR.ChinenM.JungH. (2012). National Evaluation of Title III Implementation Supplemental Report – Exploring approaches to setting English language proficiency performance criteria and monitoring English learner progress. Washington, DC: U.S. Department of Education, Office of Planning, Evaluation, and Policy Development, Policy and Program Studies Service. Retrieved from www.ed.gov/about/offices/list/opepd/ppss/index.html
27.
DouglasK.M.MislevyR.L. (2010). Estimating classification accuracy for complex decision rules based on multiple scores. Journal of Education and Behavioral Statistics, 35(3), 280–306.
28.
EstradaP. (2014). English learner Curricular Streams in four middle schools:Triage in the trenches. Urban Review, 46, 535–573.
29.
EstradaP.WangH. (2013, May). Reclassifying and not reclassifying English learners to fluent English proficient, year 1 findings: Factors impeding and facilitating reclassification and access to the core. Paper presented at the American Educational Research Association, San Francisco, California.
30.
Federal Register, 73(202), 61828–61844. October17, 2008.
31.
Faulkner-BondM.WaringS.ForteE.CrenshawR.L.TindleK.BelkapB. (2012). Language instruction educational programs (LIEPs): A review of the foundational literature. Washington, DC: U.S. Department of Education; Office of Planning, Evaluation and Policy Development; Policy and Program Studies Service. Retrieved from www2.ed.gov/rschstat/eval/title-iii/language-instruction-ed-programs-report.pdf.
32.
FlorezI.R. (2012). Examining the validity of the Arizona English language learners assessment cut scores. Language Policy, 11(1), 33–45.
33.
ForteE.PerieM.PaekP. (2012). Exploring the relationship between English language proficiency and English language arts. EAG EVEA project deliverable. Retrieved from www.eveaproject.com
34.
FrancisD.J.RiveraM.O. (2007). Principles underlying English language proficiency tests and academic accountability for ELLs.In AbediJ. (Ed.), English language proficiency assessment in the nation: Current status and future practice (pp. 13–31). Davis, CA: University of California Davis School of Education.
35.
GándaraP.RumbergerR.Maxwell-JollyJ.CallahanR. (2003, October7). English learners in California schools: Unequal resources, unequal outcomes. Education Policy Analysis Archives, 11(36). Retrieved on October 23, 2011 from http://epaa.asu.edu/epaa/v11n36/
36.
GarciaE.E.LawtonK.Diniz de FigueiredoE.H. (2010). Assessment of young English language learners in Arizona: Questioning the validity of the state measure of English proficiency. Los Angeles: Civil Rights Project, University of California. Retrieved from: http://civilrightsproject.ucla.edu/research
37.
HakutaK.ButlerY.G.WittD. (2000). How long does it take English learners to attain proficiency? Policy report, University of California Linguistic Minority Research Institute, University of California, Berkeley.
38.
HillL.WestonM.HayesJ. (2014). Reclassification of English-learner students in California. San Francisco, CA: Public Policy Institute of California.
39.
KaneM.T. (2006). Validation.In BrennanR. L. (Ed.), Educational measurement (4th ed., pp. 17–64). Westport, CT: American Council on Education and Praeger.
40.
KaneM.T. (2012). Validating score interpretations and uses. Language Testing, 29(1), 3–17.
41.
KaneM.T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.
42.
KiefferM.J.RiveraM.FrancisD.J. (2012). Research-based recommendations for the use of accommodations in large-scale assessments: 2012 update. Practical Guidelines for the Education of English Language Learners. Book 4. Center on Instruction.
43.
KimJ. (2011). Relationships among and between ELL status, demographic characteristics, enrollment history, and school persistence (CRESST Report 810). Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
44.
KimJ.HermanJ.L. (2010). When to exit ELL students: Monitoring success and failure in mainstream classrooms after ELLs’ reclassification. (CRESST Report 779). Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
45.
LinquantiR. (2001). The reclassification dilemma: Challenges and choices in fostering meaningful accountability for English learners (UC Language Minority Research Institute Policy Report 2001-01). Davis, CA: University of California, Davis.
46.
LinquantiR.CookH.G. (2013). Toward a “common definition of English learner”: Guidance for states and state assessment consortia in defining and addressing policy and technical issues and options. Washington, DC: Council of Chief State School Officers.
47.
LinquantiR.GeorgeC. (2007). Establishing and Utilizing an NCLB Title III Accountability System: California’s Approach and Findings to Date.In AbediJ. (Ed.), English language proficiency assessment in the nation: Current status and future practice (pp. 105–118). Davis, CA: University of California Davis School of Education.
48.
LlosaL. (2008). Building and supporting a validity argument for a standards-based classroom assessment of English proficiency based on teacher judgments. Educational Measurement: Issues and Practice, 27(3), 32–42.
49.
MahoneyK.S.MacSwanJ. (2005). Reexamining identification and reclassification of English language learners: A critical discussion of select state practices. Bilingual Research Journal, 29(1), 31–42.
50.
MitzelH.C.LewisD.M.PatzR.J.GreenD.R. (2001). The bookmark procedure: Psychological perspectives.In CizekG.J. (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 249–281). Mahwah, NJ: Lawrence Erlbaum.
51.
National Research Council. (2011). Allocating federal funds for state programs for English language learners. Panel to Review Alternative Data Sources for the Limited-English Proficiency Allocation Formula under Title III, Part A, Elementary and Secondary Education Act. Committee on National Statistics and Board on Testing and Assessment. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
52.
No Child Left Behind Act of 2001 (NCLB) (2001). Conference Report to Accompany H.R., 1, Rep. No. 107-334, House of Representatives, 107th Congress, 1st Session, December 13. Pub. L. No. 107-110, 115 Stat. 1425.
53.
PorterS.G.VegaJ. (2007). Overview of existing English language proficiency tests.In AbediJ. (Ed.), English language proficiency assessment in the nation: Current status and future practice (pp. 93–104). Davis, CA: UC Davis School of Education.
54.
RaganA.LesauxN. (2006). Federal, state, and district level English language learner program entry and exit requirements: Effects on the education of language minority learners. Education Policy Analysis Archives, 14(20). Retrieved on October6, 2011 from http://epaa.asu.edu/epaa/v14n20/
55.
RobinsonJ.P. (2011). Evaluating criteria for English learner reclassification: A causal-effects approach using a binding-score regression discontinuity design with instrumental variables. Educational Evaluation and Policy Analysis, 33(3), 267–292.
56.
SireciS.HanK.T.WellsC.S. (2008). Methods for evaluating the validity of test scores for English language learners. Educational Assessment, 13(3), 108–131.
57.
SlamaR.B. (2014). Investigating whether and when English learners are reclassified into mainstream classrooms in the United States: A discrete-time survival analysis. American Educational Research Journal, 51(2), 220–252.
58.
SolórzanoR.W. (2008). High stakes testing: Issues, implications, and remedies for English language learners. Review of Educational Research, 78(2), 260–329.
59.
SpolskyB. (2008). Language testing at 25: Maturity and responsibility?Language Testing, 25(3), 297–305.
60.
Stanford English Language Proficiency Test (2003). San Antonio, TX: Harcourt Assessment.
61.
“State A” Department of Education (2009a). “State A ELPA” 2009 Technical Report. Retrieved from “State A” website.
62.
“State A” Department of Education (2009b). Title III/Limited English Proficient (LEP) Accountability Plan. Retrieved from “State A” website.
63.
StephensonA.JiaoH.WallN. (2004). A performance comparison of native and non-native speakers of English on an English language proficiency test. San Antonio, TX: Harcourt Assessment. Retrieved from www.pearsonassessments.com
64.
StephensonA.JohnsonD.F.JorgensenM.A.YoungM.J. (2003). Assessing English language proficiency: Using valid results to optimize instruction. San Antonio, TX: Harcourt Assessment. Retrieved from www.pearsonassessments.com
65.
United States Department of Education, National Center for Education Statistics. (2014). The condition of education 2014 (NCES 2014-083).
66.
WalquiA.KoelschN.HamburgerL (2010). What are we doing to middle school English learners? Findings and recommendations for change from a study of California EL programs (Narrative Summary). San Francisco: WestEd.
67.
WangJ.NiemiD.WangH. (2007). Impact of different performance assessment cut scores on student promotion. (CSE [CRESST] Report 719). Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
68.
WolfM.K.FarnsworthT. (2014). English language proficiency assessments as an exit criterion for English learners.In KunnanA. (Ed.), The companion to language assessment (pp. 303–317). New York: Wiley-Blackwell.
69.
WolfM.K.FarnsworthT.HermanJ. (2008). Validity issues in assessing English language learners’ language proficiency. Educational Assessment, 13(3), 80–107.
70.
WolfM.K.HermanJ.L.DietelR. (2010). Improving the validity of English language learner assessment systems (CRESST Policy Brief No. 10–Full Report). Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
71.
World-Class Instructional Design and Assessment [WIDA] Consortium. (2013). ACCESS for ELLsTM. Madison, WI: Board of Regents of the University of Wisconsin System.
72.
XiX. (2010). How do we go about investigating test fairness?Language Testing, 27 (2), 147–170.
73.
XiongY.S.ZhouM. (2006). Structuring inequality: How California selectively tests, classifies, and tracks language minority students. California Policy Options, UCLA School of Public Affairs, UCLA. Retrieved from: http://escholarship.org/uc/item/98d66346
74.
ZhangB. (2010). Assessing the accuracy and consistency of language proficiency classification under competing measurement models. Language Testing, 27(1), 119–140.
