The role of response effects (tendencies to respond to factors other than item content) in the self-assessment of second language ability was investigated through a split-ballot procedure using positively- and negatively-worded ques tions and graded (i.e. level-specific questions). Results indicate that both an acquiescence effect (a tendency to respond positively regardless of item con tent) and overestimation were present. Although both effects were present at all levels of subjects, they were most evident for less experienced learners. Discussion focuses on explanation of these effects and implications for both test construction and pedagogy.
Get full access to this article
View all access options for this article.
References
1.
Alwin, D.F. and Jackson, D.J.1980: Measurement models for response error in surveys: Issues and applications. In Schuessler, K.F., editor, Sociological methodology, 1980. San Francisco : Jossey-Bass.
2.
Anderson, P.L.1982: Self-esteem in the foreign language: A preliminary investigation . Foreign Language Annals15, 109-115.
3.
Bachman, L.F. and Palmer, A.S.1989: The construct validation of self-ratings of communicative language ability. Language Testing6, 14-29.
4.
Barrows, T., Ager, S.M., Bennett, M.F., Braun, H.I., Clark, J.L.D., Harris, L.G. and Klein, S.F.1981: College students' knowledge and beliefs: A survey of global understanding. The final report of the global understanding project. New Rochelle, NY: Change Press ERIC ED 215 653.
5.
Bass, B.M., Cascio, W.F. and O'Connor, E.J.1974: Magnitude estimations of expressions of frequency and amount. Journal of Applied Psychology59, 313-20.
6.
Bates, R.J.1984: Educational versus managerial evaluation in schools. In Broadfoot, P., editor, Selection, certification and control: Social issues in educational assessment. London: Falmer Press.
7.
Blanche, P.1988: Self-assessment of foreign language skills: Implications for teachers and researchers. RELC Journal19, 75-93.
8.
Blanche, P. and Merino, B.J.1989: Self assessment of foreign-language skills: Implications for teachers and researchers. Language Learning39, 313-40.
9.
Blue, G.M.1988: Self-assessment: The limits of learner independence. ELT Documents131, 100-18.
10.
Borgatta, E.F. and Bohrnstedt, G.W.1980: Level of measurement once over again. Sociological Methods and Research9, 147-60.
11.
Bradburn, N.M. and Sudman, S.1979: Improving interview method and questionnaire design. San Francisco: Jossey-Bass.
12.
Brumfit, C.1984: Communicative methodology in language teaching: The roles offluency and accuracy. Cambridge: Cambridge University Press.
13.
Cannell, C.F., Miller, P.V. and Oksenberg, L.1981: Research on interviewing techniques. In Leinhardt, S., editor, Sociological methodology, 1981. San Francisco: Jossey-Bass .
14.
Carr, R.A.1977: The effects of specific guidelines on the accuracy of student self-evaluation. Canadian Journal of Education2, 65-77.
15.
Chapman, L.J. and Bock, R.D.1958: Components of variance due to acquiescence and content in the F Scale measure of authoritarianism. Psychological Bulletin,55, 328-33.
16.
Clogg, C.C.1984: Some statistical models for analyzing why surveys disagree . In Turner, C.F. and Martin, E., editors, Surveying subjective phenomena, Vol. 2. New York : Russell Sage Foundation.
17.
Converse, J.M. and Presser, S.1986: Survey questions: Handcrafting the standardized questionnaire . Beverly Hills: Sage.
18.
Currall, S.C. and Kirk, R.E.1986: Predicting success in intensive foreign language courses . Modern Language Journal70, 107-13.
19.
Davidson, F. and Henning, G.1985: A self-rating scale of English difficulty: Rasch scalar analysis of items and rating categories. Language Testing2, 164-79.
20.
Davison, M.L. and Srichantra, N.1988: Acquiescence in components analysis and multidimensional scaling of self-rating items. Applied Psychological Measurement12, 339-51.
21.
DeLameter, J.1982: Response-effects of question content. In Dijkstra, W. and van der Zouwen, J., editors, Response behaviour in the survey-interview . New York: Academic Press.
22.
Dickinson, L.1987: Self-instruction in language learning. Cambridge: Cambridge University Press.
23.
Duncan, O.D.1984: Rasch measurement: Further examples and discussion. In Turner, C.F. and Martin, E., editors, Surveying subjective phenomena, Vol. 2. New York: Russell Sage Foundation.
24.
von Elek, T.1985: A test of Swedish as a second language: An experiment in self-assessment. In Lee, Y., Fok, A., Lord. R., and Low, G., editors, New directions in language testing. Oxford: Oxford University Press.
25.
Elliott, L.L.1961: Effects of item construction and respondent aptitude on response acquiescence. Educational and Psychological Measurement21, 405-15.
26.
Ferguson, N.1978: Self-assessment of listening comprehension. IRAL16, 149-56.
27.
Garhart, C. and Hannafin, M.1986: The accuracy of cognitive monitoring during computer-based instruction, ERIC ED 267 768.
28.
Gove, W.R.1982: Systematic response bias and characteristics of the respondent . In Dijkstra, W. and van der Zouwen, J., editors, Response behaviour in the survey-interview. New York: Academic Press.
29.
Hamilton, L.C.1981: Self-reports of academic performance: Response errors are not well-behaved. Sociological Methods and Research10, 165-85.
30.
Hilton, T.L., Grandy, J., Kline, R.G. and Liskin-Gasparro, J.E.1985: Final report: The oral language proficiency of teachers in the United States in the 1980s - An empirical study. Princeton, NJ: Educational Testing Service.
31.
Hippler, H-J.1987: Response effects in surveys. In Hippler, H-J., Schwarz, N. and Sudman, S., editors, Social information processing and survey methodology. New York: Springer-Verlag.
32.
Horwitz, E.K., Horwitz, M.B. and Cope, J.1986: Foreign language classroom anxiety. Modern Language Journal70, 125-32.
33.
Howard, G.S.1981: On validity. Evaluation Review5, 567-76.
34.
Janssen-van Dieten, A-M.1989: The development of a test of Dutch as a second language: The validity of self-assessment by inexperienced subjects. Language Testing6, 30-46.
35.
Klockars, A.J. and Yamagishi, M.1988: The influence of labels and positions in rating scales . Journal of Educational Measurement25, 85-96.
36.
LeBlanc, R. and Painchaud, G.1985: Self-assessment as a second language placement instrument . TESOL Quarterly19, 673-87.
37.
Lessler, J.T.1984: Measurement error in surveys. In Turner, C.F. and Martin, E., editors, Surveying subjective phenomena, Vol. 2. New York: Russell Sage Foundation .
38.
Levine, M.G., Haus, G.J. and Cort, D.1987: The accuracy of teacher judgement of the oral proficiency of high school foreign language students. Foreign Language Annals20, 45-50.
39.
Lichtenstein, S. and Newman, J.R.1967: Empirical scaling of common verbal phrases associated with numerical probabilities. Psychonomic Science9, 563-64.
40.
Loughrin-Sacco, S.J.1990: Inside the 'black box' revisited: Toward the integration of naturalistic inquiry in classroom research on foreign language learning . Polylingua1, 22-26.
41.
Magnan, S.S.1989: Do spoken French and grammatical control improve with course work?French Review63, 16-30.
42.
Newstead, S.E.1988: Quantifiers as fuzzy concepts. In Zétényi, T., editor, Fuzzy sets in psychology. Amsterdam: Elsevier Science.
43.
Newstead, S.E. and Collis, J.M.1987: Context and interpretation of quantifiers of frequency . Ergonomics30, 1447-62.
44.
Oller, J.W. Jr and Perkins, K.1978: Language proficiency as a source of variance in self-reported affective variables. In Oller, J.W. Jr, and Perkins, K., editors, Language in education: Testing the tests. Rowley, Mass: Newbury House.
45.
Olson, M.A. and Martin, D.1980: Assessment of entering student writing skill in the community college, ERIC ED 235 845.
46.
Oskarsson, M.1981: Subjective and objective assessment of foreign language performance. In Read, J.A.S., editor, Directions in language testing: Selected papers from the RELC Seminar on "Evaluation and Measurement of Language Competence and Performance'; Singapore, April 1980. Singapore: SEAMEL Regional Language Centre.
47.
Oskarsson, M.1989: Self-assessment of language proficiency: Rationale and applications. Language Testing6, 1-13.
48.
Ostrom, T.M.1987: Bipolar survey items: An information processing perspective . In Hippler, H-J., Schwarz, N. and Sudman, S., editors, Social information processing and survey methodology. New York: Springer-Verlag.
49.
Pohl, N.F.1982: Using retrospective pre-ratings to counteract response-shift confounding. Journal of Experimental Education50, 211-14.
50.
Ray, J.J.1983: Reviving the problem of acquiescent response bias. Journal of Social Psychology121, 81-96.
51.
Reed, K.X.1988: Expectation vs ability: Junior college reading skills, ERIC ED 295 706.
52.
Reyna, V.F.1981: The language of possibility and probability: Effects of negation on meaning. Memory & Cognition9, 642-50.
53.
Schmitt, N. and Stults, D.M.1985: Factors defined by negatively keyed items: The result of careless respondents?Applied Psychological Measurement9, 367-73.
54.
Schuman, H. and Presser, S.1981: Questions and answers in attitude surveys: Experiments on question form, wording, and content. New York: Academic Press.
55.
Schwartz, N. and Hippler, H-J.1987: What response scales may tell your respondents. In Hippler, H-J., Schwarz, N. and Sudman, S., editors, Social information processing and survey methodology. New York: Springer-Verlag.
56.
Tourangeau, R.1987: Attitude measurement: A cognitive perspective. In Hippler, H-J., Schwarz, N. and Sudman, S., editors, Social information processing and survey methodology. New York: Springer-Verlag.
57.
Trott, D.M. and Jackson, D.N.1967: An experimental analysis of acquiescence. Journal of Experimental Research in Personality2, 278-88.
58.
Turner, C.F.1984: Why do surveys disagree? Some preliminary hypotheses and some disagreeable examples. In Turner, C.F. and Martin, E., editors, Surveying subjective phenomena, Vol. 2. New York: Russell Sage Foundation.
59.
Turner, C.F. and Martin E.1984a: Measurement and error: An introduction. In Turner, C.F. and Martin, E., editors, Surveying subjective phenomena, Vol. 1. New York: Russell Sage Foundation.
60.
Turner, C.F. and Martin E.1984b: Question and answer process. In Turner, C.F and Martin, E., editors, Surveying subjective phenomena, Vol. 1. New York: Russell Sage Foundation .
61.
Turner, C.F. and Martin E.1984c: Variability in survey measurements of subjective phenomena: Empirical results. In Turner, C.F. and Martin E., editors, Surveying subjective phenomena, Vol. 1. New York: Russell Sage Foundation.
62.
Upshur, J.1975: Objective evaluation of oral proficiency in the ESOL classroom . In Palmer, L. and Spolsky, B., editors, Papers on language testing 1967-1974 . Washington, D.C.: TESOL.
63.
Velleman, P.F. and Hoaglin, D.C.1981: Applications, basics, and computing of exploratory data analysis. Belmont, CA: Duxbury.
64.
Wesche, M.B., Morrison, F., Ready, D. and Pawley, C.1990: French immersion: Postsecondary consequence for individuals and universities. Canadian Modern Language Review46, 430-51.
65.
Wickens, T.D.1989: Multiway contingency tables analysis for the social sciences . Hillsdale, N.J.: Lawrence Erlbaum .
66.
Wilkinson, L.1987: SYSTAT: The system for statistics. Evanston, IL: SYSTAT, Inc.
67.
Winkler, J.D., Kanouse, D.E. and Ware, J.E. Jr1982: Controlling for acquiescence response set in scale development. Journal of Applied Psychology67, 555-61.
68.
Wyer, R.S. Jr and Hartwick, J.1980: The role of information retrieval and conditional inference processes in belief formation and change. In Berkowitz, L., editor, Advances in experimental social psychology , Volume 13, New York: Academic Press.
69.
Yule, G.Yanz, J.L. and Tsuda, A.1985: Investigating aspects of the language learner's confidence: An application of the theory of signal detection. Language Learning35, 473-88.
