Begging the questionnaire: instrument effect on readers' responses to a self-report checklist

Abstract

This study investigates the validity of having English as a second language (ESL) test-takers select from a checklist the strategies they believe they use when answering multiple-choice reading comprehension questions. Five groups of tertiary-level ESL male and female students studied the same reading passage and questions, and four groups each sequentially used a different version of a self-report checklist of reading strategies. The checklist was manipulated to establish whether it exercised an effect on users. The most popular strategy, established empirically, was first repositioned on, then removed from, the checklist. It was observed that students then selected an alternative from the modified list rather than volunteer in an open category information about how they had arrived at their answers. A similar finding was noted when the next most popular strategy was also removed from the list. The fifth group responded without the aid of the checklist, and were asked to explain their answering strategies in their own words. This group tended to use words and phrases different from those on the checklist. It was concluded that self-report checklists exercise an instrument effect on users' behaviour and thus can invalidate the data collected. Implications are suggested for test-item validation.