This article examines the intersection of psychornetrics and sociolinguists in the testing of English language learners (ELLs); it discusses language, dialect, and register as sources of measurement error. Research findings show that the dialect of the language in which students are tested (e.g., local or standard English) is as important as language as a facet that influences score dependability in ELL testing. The development, localization, review, and sampling of items are examined as aspects of the process of test construction critical to properly attaining linguistic alignment: the correspondence between the features of the dialect and the register used in a test, and the features of the language to which ELLs are exposed in both formal and instructional contexts.
Get full access to this article
View all access options for this article.
References
1.
AbediJ. (2002). Standardized achievement tests and English language learners: Psychometric issues. Educational Assessment, 8, 231—257.
2.
AbediJ. (2004). The No Child Left Behind Act and English language learners: Assessment and accountability issues. Educational Researcher, 33(1), 4—14.
3.
AbediJ., HofstetterC., and LordC. (2004). Assessment accommodations for English language learners: Implications for policy-based empirical research. Review of Educational Research, 74, 1—28.
4.
AbediJ., and LordC. (2001). The language factor in mathematics tests. Applied Measurement in Education, 14, 219—234.
5.
AbediJ., LordC., HofstetterC., and BakerE. (2000). Impact of accommodation strategies on English language learners’ test performance. Educational Measurement: Issues and Practice, 19(3), 16—26.
6.
Aguirre-MuñozZ., and BakerE. L. (1997). Improving the equity and validity of assessment-based information systems (CSE Tech. Rep. No. 462). Los Angeles: University of California, Center for the Study of Evaluation; National Center for Research on Evaluation, Standards, and Student Testing; Graduate School of Education and Information Studies.
7.
American Educational Research Association/American Psychological Association/National Council on Measurement in Education. (1999). Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association, American Psychological Association, and National Council on Measurement in Education.
8.
AllaloufA. (2003). Revising translated differential item functioning items as a tool for improving cross-lingual assessment. Applied Measurement in Education, 16, 55—73.
9.
AugustD., and HakutaK. (Eds.). (1997). Improving schooling for language minority students: A research agenda. Washington, DC: National Academy Press.
10.
BachmanL. F., LynchB. K., and MasonM. (1995). Investigating variability in tasks and rater judgments in a performance test of foreign language speaking. Language Testing, 12, 239—257.
11.
BialystokE. (Ed.). (1991). Language processing in bilingual children. Cambridge, UK: Cambridge University Press.
12.
BialystokE. (2001). Bilingualism in development: Language, literacy, and cognition. Cambridge, UK: Cambridge University Press.
13.
BolusR. E., HinofotisF. B., and BaileyK. M. (1982). An introduction to generalizability theory in second language research. Language Learning, 32, 245—258.
14.
BormuthJ. R. (1970). On the theory of achievement test items. Chicago: University of Chicago Press.
15.
BrennanR. L. (1992). Elements of generalizability theory. Iowa City, IA: American College Testing Program.
16.
BrennanR. L. (2001). Generalizability theory. New York: Springer-Verlag.
17.
BriggsD. C., and WilsonM. R. (2004, June.). Generalizability in item response modeling. Presentation at the International Meeting of the Psychometric Society, Pacific Grove, CA.
18.
BriskM. E. (1998). Bilingual education: From compensatory to quality schooling. Mahwah, NJ: Erlbaum.
19.
BrownJ. D., and BaileyK. M. (1984). A categorical instrument for scoring second language writing skills. Language Learning, 34, 21—42.
20.
ButlerF. A., and StevensR. (1997). Accommodation strategies for English language learners on large-scale assessments: Student characteristics and other considerations ((CSE Tech. Rep. No. 448). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.
21.
CamilliG., and ShepardL. (1994). Methods for identifying biased test items. Thousand Oaks, CA: Sage.
22.
CasanovaU., and AriasB. (1993). Contextualizing bilingual education. In AriasM.B., and CasanovaU. (Eds.), Bilingual education: Politics, practice, and research (pp. 1—35). Chicago: University of Chicago Press.
23.
ColeM. (1999). Culture-free versus culture-based measures of cognition. In and and SternbergR.J. (Ed.), The nature of cognition (pp. 645—664). Cambridge, MA: MIT Press.
24.
CookL. I., and Schimitt-CascallarA. P. (2005). Establishing score comparability for tests given in different languages. In HambletonR.K., MerendaP.F., and SpielbergerC.D. (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 139—169). Mahwah, NJ: Erlbaum.
25.
CoulmasF. (2005). Sociolinguistics: The study of speakers’ choices. Cambridge, UK: Cambridge University Press.
26.
Council of Chief State School Officers. (1992). Recommendations for improving the assessment and monitoring of students with limited English proficiency. Washington, DC: Author.
27.
CronbachL. J., GleserG. C., NandaH., and RajaratnamN. (1972). The dependability of behavioral measurements. New York: Wiley.
28.
CrystalD. (1997). The Cambridge encyclopedia of language (2nd ed.). Cambridge, UK: Cambridge University Press.
29.
CumminsJ. (1984). Bilingualism and special education: Issues in assessment and pedagogy. Clevedon, UK: Multilingual Matters.
30.
CumminsJ. (1996). Negotiating identities: Education for empowerment in a diverse society. Los Angeles: California Association for Bilingual Education.
31.
CumminsJ. (1999). Alternative paradigms in bilingual education research: Does theory have a place?Educational Researcher, 28, 26—32, 41.
32.
CumminsJ. (2000). Language, power and pedagogy: Bilingual children in the crossfire. Clevedon, UK: Multilingual Matters.
33.
CumminsJ. (2003). BICS and CALP: Origins and rationale for the distinction. In PaulstonC.B., and TuckerG.R. (Eds.), Sociolinguistics: The essential readings (pp. 322—328). London: Blackwell.
34.
De AvilaE. (1990). Assessment of language minority students: Political, technical, practical and moral imperatives. Proceedings of the First Research Symposium on Limited English Proficient Student Issues. Retrieved February 9, 2005, from http://www.ncela.gwu.edu/pubs/symposia/first/assessment.htm
35.
EdelskyC. (1990). With literacy and justice for all. Rethinking the social in language and education. London: Falmer Press.
36.
ErcikanK. (1998). Translation effects in international assessment. International Journal of Educational Research, 29, 543—553.
37.
FarrM., and BallA. F. (1999). Standard English. In and and SpolskyB. (Ed.), Concise encyclopedia of educational linguistics (pp. 205—208). Oxford, UK: Elsevier.
38.
FergusonA. M., and FairburnJ. (1985). Language experience for problem solving in mathematics. Reading Teacher, 38, 504—507.
39.
FerraraS., MacmillanJ., and NathanA. (2004, January). Enhanced database on inclusion and accommodations: Variables and measures (NAEP State Analysis Project Report to the National Center for Education Statistics). Washington, DC: NCES.
40.
FreedleR. O. (2003). Correcting the SAT's ethnic and social-class bias: A method for re-estimating SAT scores. Harvard Educational Review, 73, 1—43.
41.
GandaraP. (1997). Review of research on instruction of limited English proficient students. Davis: University of California, Linguistic Minority Research Institute Education Policy Center.
42.
GeneseeF. (1994). Introduction. In and and GeneseeF. (Ed.), Educating second language children: The whole child, the whole curriculum, the whole community (pp. 1—11). Cambridge, UK: Cambridge University Press.
43.
GreenfieldP. M. (1997). You can't take it with you: Why ability assessments don't cross cultures. American Psychologist, 52, 1115—1124.
44.
GrisayA. (2003). Translation procedures in OECD/PISA 2000 international assessment. Language Testing, 20, 225—240.
45.
GuerreroM. D. (1997). Spanish academic language proficiency: The case of bilingual education teachers in the U.S. Bilingual Research Journal, 21(1). Retrieved April 1, 2004, from http://www.ncela.gwu.edu/pubs/nabe/brj/v21.htm
46.
GuttmanL. (1969). Integration of test design and analysis. In Proceedings of the 1969 invitational conference on testing problems (pp. 53—65). Princeton, NJ: Educational Testing Service.
47.
HakutaK., and BeattyA. (Eds.). (2000). Testing English-language learners in U.S. schools: Report and workshop summary. Washington, DC: National Academy Press.
48.
HakutaK., Goto ButlerY., and WittD. (2000). How long does it take English learners to attain proficiency? (Policy Report No. 2000–1). Davis: University of California, Linguistic Minority Research Institute.
49.
HakutaK., and McLaughlinB. (1996). Bilingualism and second language learning: Seven tensions that define the research. In BerlinerD.C., and CalfeeR.C. (Eds.), Handbook of educational psychology (pp. 603—621). New York: Simon & Schuster Macmillan.
50.
HallidayM. A. K. (1978). Language as social semiotic: The social interpretation of language and meaning. London: Edward Arnold.
51.
HamayanE., and PerlmanR. (1990). Helping language minority students after they exit from bilingual/ ESL programs. Washington, DC: National Clearinghouse for Bilingual Education.
52.
HambletonR. K. (1994). Guidelines for adapting educational and psychological tests: A progress report. European Journal of Psychological Assessment, 10, 229—240.
53.
HambletonR. K., and JonesR. W. (1994). Comparison of empirical and judgmental procedures for detecting differential item functioning. Educational Research Quarterly, 18, 23—36.
54.
HambletonR. K., MerendaP. F., and SpielbergerC. D. (Eds.). (2005). Adapting educational and psychological tests for cross-cultural assessment. Mahwah, NJ: Erlbaum.
55.
HambletonR., and RodgersJ. (1995). Item bias review. Practical Assessment, Research & Evaluation, 4. Retrieved October 31, 2004, from http://PAREonline.net/getvn.asp?v=4&n=6
56.
HambletonR. K., SwaminathanH., and RogersH. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.
57.
HaugenE. (1966). Dialect, language, nation. American Anthropologist, 58, 922—935.
58.
HeubertJ. P., and HauserR. M. (Eds.). (1999). High stakes: Testing for tracking, promotion, and graduation. Washington, DC: National Academy Press.
59.
HivelyW., PattersonH. L., and PageS. H. (1968). A “universe-defined” system of arithmetic achievement tests. Journal of Educational Measurement, 5, 275—290.
60.
HofstetterC. H. (2003). Contextual and mathematics accommodation test effects for English-language learners. Applied Measurement in Education, 16, 159—188.
61.
HymesD. (1974). Foundations in sociolinguistics: An ethnographic approach. Philadelphia: University of Pennsylvania Press.
62.
KrashenS. D. (1996). Under attack: The case against bilingual education. Culver City, CA: Language Education Associates.
63.
LaCelle-PetersonM. W., and RiveraC. (1994). Is it real for all kids: A framework for equitable assessment policies for English language learners. Harvard Educational Review, 64, 55—75.
64.
LeeO. (1999). Equity implications based on the conceptions of science achievement in major reform documents. Review of Educational Research, 69, 83—115.
65.
LeeO. (2002). Promoting scientific inquiry with elementary students from diverse cultures and languages. Review of Research in Education, 26, 23—69.
66.
LeeO. (2005). Science education with English language learners: Synthesis and research agenda. Review of Educational Research, 75, 491—530.
67.
LeeO., and FraddS. H. (1998). Science for all, including students from non-English language backgrounds. Educational Researcher, 27, 12—21.
68.
LeeY. W. (2005). Dependability of scores for a new ESL speaking test: Evaluating prototype tasks. Princeton, NJ: Educational Testing Service.
69.
MacSwanJ., and RolstadK. (2003). Linguistic diversity, schooling, and social class: Rethinking our conception of language proficiency in language minority education. In PaulstonC.B., and TuckerG.R. (Eds.), Sociolinguistics: The essential readings (pp. 329—340). Maiden, MA: Blackwell.
70.
MessickS. (1989). Validity. In and and LinnR.L. (Ed.), Educational measurement (3rd ed., pp. 13—103). Washington, DC: American Council on Education, and National Council on Measurement in Education.
71.
MolloyH., and ShimuraM. (2005). An examination of situational sensitivity in medium-scale interlanguage pragmatics research. In T. Newfields, Y. Ishida, M. Chapman, & M. Fujioka (Eds.), Proceedings of the May 22—23, 2004, JALT Pan-SIG Conference (pp. 16–32. Retrieved September 8, 2005, from www.jalt.org/pansig/2004/HTML/ShimMoll.htm.
72.
MullisI. V. S., KellyD. L., and HaleyK. (1996). Translation verification procedures. In MartinM.O., and MullisI.V.S. (Eds.), Third international mathematics and science study: Quality assurance in data collection. Chestnut Hill, MA: Boston College.
73.
MuzziA. (2001). Challenges in localization. The ATA Chronicle, 30(11), 28—31.
74.
National Assessment of Educational Progress. (1996). Mathematics items public release. Washington, DC: Author.
75.
National Assessment of Educational Progress. (2000). Mathematics items public release. Washington, DC: Author.
76.
No Child Left Behind Act of 2001, Pub. L. No. 107–110, 115 Stat. 1425 (2002).
77.
PrestonD. R. (Ed.). (1993). American dialect research. Philadelphia: John Benjamins.
78.
PrestonD. R. (1999). Sociolinguistics. In and and SpolskyB. (Ed.), Concise encyclopedia of educational linguistics (pp. 65—70). Amsterdam: Elsevier.
79.
RickfordJ. R., and RickfordA. E. (1995). Dialect readers revisited. Linguistics and Education, 7(2), 107—128.
80.
RiveraC. (1984). Language proficiency and academic achievement. Clevedon, UK: Multilingual Matters.
81.
Ruiz-PrimoM. A., ShavelsonR. J., HamiltonL., and KleinS. (2002). On the evaluation of systemic science education reform: Searching for instructional sensitivity. Journal of Research in Science Teaching, 39, 369—393.
82.
SextonU., and Solano-FloresG. (2002, April). A comparative study of teachers’ cultural perspectives across different cultures: Preliminary results. Poster presentation at the Annual Meeting of the American Educational Research Association, Seattle, WA.
83.
ShavelsonR. J., and WebbN. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
84.
ShepardL., TaylorG., and BetebennerD. (1998). Inclusion of limited-English proficient students in Rhode Island's Grade 4 mathematics performance assessment (CSE Tech. Rep. No. 486). Los Angeles: University of California; Center for the Study of Evaluation; National Center for Research on Evaluation, Standards, and Student Testing; Graduate School of Education and Information Studies.
85.
SireciS. G. (2005). Using bilinguals to evaluate the comparability of different language versions of a test. In HambletonR.K., MerendaP.F., and SpielbergerC.D. (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 117—138). Mahwah, NJ: Erlbaum.
86.
Solano-FloresG., and BackhoffE. (2003). Test translation in international comparisons: A preliminary study. Mexico City, Mexico: Mexican Department of Education, National Institute for Educational Evaluation.
87.
Solano-FloresG., Contreras-NiñoL. A., and BackhoffE. (2005, April). The Mexican translation of TIMSS-95: Test translation lessons from a post-mortem study. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Quebec, Canada.
88.
Solano-FloresG., LaraJ., SextonU., and NavarreteC. (2001). Testing English language learners: A sampler of student responses to science and mathematics test items. Washington, DC: Council of Chief State School Officers.
89.
Solano-FloresG., and LiM. (2006). The use of generalizability (G) theory in the testing of linguistic minorities. Educational Measurement: Issues and Practice, 25, 13—22.
90.
Solano-FloresG., LiM., and SextonU. (2005). On the accuracy of teacher judgments of the linguistic and cultural adequacy of test items. Unpublished manuscript.
91.
Solano-FloresG., and Nelson-BarberS. (2001). On the cultural validity of science assessments. Journal of Research in Science Teaching, 38, 553—573.
92.
Solano-FloresG., SperoniC., and SextonU. (2005, April). The process of test translation: Advantages and challenges of a socio-linguistic approach. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Quebec, Canada.
93.
Solano-FloresG., and TrumbullE. (2003). Examining language in context: The need for new research and practice paradigms in the testing of English-language learners. Educational Researcher, 32(2), 3—13.
94.
Solano-FloresG., TrumbullE., and KwonM. (2003, April). The metrics of linguistic complexity and the metrics of student performance in the testing of English language learners. Paper presented at the 2003 Annual Meeting of the American Evaluation Research Association, Chicago, IL.
95.
Solano-FloresG., TrumbullE., and Nelson-BarberS. (2002). Concurrent development of dual language assessments: An alternative to translating tests for linguistic minorities. International Journal of Testing, 2, 107—129.
96.
StansfieldC. W., and KenyonD. M. (1992). Research of the comparability of the oral proficiency interview and the simulated oral proficiency interview. System, 20, 347—364.
97.
StevensR. A., ButlerF. A., and Castellon-WellingtonM. (2000). Academic language and content assessment: Measuring the progress of English Language Learners (ELLs) (CSE Tech. Rep. No. 552). Los Angeles: University of California; National Center for Research on Evaluation, Standards, and Student Testing; Graduate School of Education and Information Studies.
98.
TanzerN. K. (2005). Developing tests for use in multiple languages and cultures: A plea for simultaneous development. In HambletonR.K., MerendaP.F., and SpielbergerC.D. (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 235—263). Mahwah, NJ: Erlbaum.
99.
TunickL. (2003). Language translation, localization, and globalization. The ATA Chronicle, 32(2), 19—21.
100.
ValdésG., and FigueroaR. A. (1994). Bilingualism and testing: A special case of bias. Norwood, NJ: Ablex.
101.
van de VijverF., and PoortingaY. H. (1997). Towards an integrated analysis of bias in cross-cultural assessment. European Journal of Psychological Assessment, 13, 29—37.
102.
van de VijverF., and TanzerN. K. (1998). Bias and equivalence in cross-cultural assessment: An overview. European Review of Applied Psychology, 47, 263—279.
103.
WardhaughR. (2002). An introduction to sociolinguistics (4th ed.). Oxford, UK: Blackwell.
104.
WolframW., AdgerC. T., and ChristianD. (1999). Dialects in schools and communities. Mahwah, NJ: Erlbaum.
YooB., and ResnickL. B. (1998). Instructional validity, opportunity to learn and equity: New standards examinations for the California Mathematics Renaissance (CSR Tech. Rep. No. 484). Los Angeles: National Center for Research on Evaluation, Standards, and Student Testing.
