Sage Journals: Discover world-class research

Abstract

Although questionnaire surveys are widely applied in second-language research, there is still little research on the application of questionnaire among early career researchers. Problems such as the lack of details concerning why the questionnaire survey is adopted, how the questionnaire is constructed, and how the survey is administered and reported are prevalent in student teachers’ research proposals or theses, which greatly impairs the quality of their academic work. However, few studies have been conducted to address these problems. This study aims to explore the real situation of the questionnaire survey application by examining student teachers’ perceptions of a questionnaire survey and their application of this research method to their theses. Using a mixed-method approach, the study collected quantitative and qualitative data from 138 student teachers, including 103 fresh undergraduates and 35 post-graduates. Quantitative data analysis used descriptive statistics, while qualitative data analysis used semi-structured interviews and content analysis. The study found that: (1) student teachers had some basic but unsystematic perceptions of the questionnaire survey. They had insufficient awareness of the rules of writing good questions, piloting questionnaires, sampling strategies, doing reliability, and validity analysis, etc. (2) Most student teachers developed questionnaires based on existing questionnaires without explaining reasons and adjustments. They introduced questionnaire design in their theses in an incomplete way, with piloting, selecting sample strategies, and statistical analysis methods often missing. Thus, some pedagogical implications can be drawn to improve student teachers’ awareness and competencies of the questionnaire survey.

Keywords

questionnaire survey perceptions application student teachers teacher education

Introduction

Due to its main strength such as being easy to construct and analyze the data, especially with the help of computers and some sophisticated data processing software, a questionnaire survey has become a popular scientific research method. Moser and Kalton (1971) said that nine out of ten social studies were conducted with a questionnaire. Xu and Li (2018) also discovered that the questionnaire survey was the most frequently used quantitative method in second-language teacher studies in China and abroad. However, the world is full of people who believe everyone who can write English and has common sense can produce good questionnaires (Oppenheim, 1992). The fact is that aside from badly designed questionnaires, other flaws in questionnaire surveys such as insufficient information about the questionnaire construction, lacking reliability and validity analysis, and inappropriately selecting data analysis methods can be frequently noticed in students’ research proposals or theses. As Gillham (2008) points out, no single method has been so abused as the questionnaire survey, asserting that good research cannot be built on poorly collected data. Worse still, despite the wide application of questionnaire surveys in second-language research, there wasn’t enough awareness of the theories of questionnaire design (Dörnyei & Taguchi, 2010).

Foreign language education research cannot do without data collection methods among which questionnaire surveys are one of the most important means of data collection. Though many foreign language teachers long to apply questionnaire surveys to their teaching and research practice, most of them did not have the idea of how to design a high-quality questionnaire or how to adapt others’ questionnaires to their research contexts due to the lack of systematic training of research methods (Qin, 2009). In contrast with in-service teachers, student teachers are supposed to have better acquaintance with a questionnaire survey which is an essential part of methodology commonly included in their teacher education program. They are also more likely to apply it to their foreign language education research to fulfill the requirement of thesis writing. It is evident that their knowledge and competencies of research methods such as questionnaire surveys are crucial to the development of scientific literacy and teacher competency. However, empirical research on how they perceive and apply these research methods is scarce. This study aims to investigate student teachers’ perceptions and the application of a questionnaire survey. The research questions are addressed as follows:

What are the student teachers’ perceptions of a questionnaire survey in foreign language education research?

How well do the student teachers apply questionnaire surveys in their theses?

Literature Review

Questionnaire Surveys in Foreign Language Education Research

The popularity of questionnaire surveys in foreign language education research has been widely recognized, and many scholars have examined its application in foreign language research (e.g., X. S. Gao, 2004; Perry, 2005). The first full-length volume devoted to questionnaires in L2 research was published in 2002, describing how to construct questionnaires, administer questionnaires, and process questionnaire data (Dörnyei & Taguchi, 2010). Kormos (2003) commented that except for Brown’s (2001) work, Dörnyei’s work is an excellent resource book for experienced and beginning researchers as well as undergraduate and graduate students in research design courses. Tuleja et al. (2011) claimed that using a questionnaire across cultures requires extra effort in translating it into the target language(s) and culture(s) because a good questionnaire developed in one language/culture may not necessarily “travel well” across cultures due to differences in meaning and interpretation. Rowley (2014) offered novice researchers lots of advice on questionnaire design, distribution, data analysis, and presentation. Larson-Hall and Plonsky (2015) is one of the few empirical studies to explore the deficiencies in the research on questionnaire surveys, claiming that most foreign language research reports questionnaire data built on other research fields’ criteria, like psychology, education, or linguistics, instead of the field its own.

In China, there similarly exists a widespread application of questionnaire surveys in foreign language research but there are relatively limited studies on this research method. Although many researchers have studied the status and trend of research methods in foreign language research, such as Y. H. Gao et al. (1999) and Wen and Lin (2016), a questionnaire survey is only roughly referred to in these studies. Qin (2009) first presented a monograph, introducing in Chinese the features and application scope of a questionnaire survey in foreign language teaching research, including each stage of questionnaire survey design and administration through considerable cases. At the same time, some scholars focus attention on specific aspects of the questionnaire survey, such as the statistical methods (Bao, 2012), the main problems existing in the application of questionnaire survey employed in core journal papers in foreign language research (Zheng & Wang, 2014), and similarities and differences in questionnaire survey application in foreign language education research in and outside China (e.g., Huang, 2014; Zhang, 2016). Xu and Li (2018) found that there are some deficiencies in the questionnaire surveys applied in second language teacher studies in China, such as the lack of reliability and validity, the lack of pilot tests, and the inappropriate use of sampling strategies. However, there is a lack of studies that target how student teachers perceive questionnaire surveys, and how well they apply them in their studies. Even though many studies were clustered around the perceptions or attitudes of student teachers or pre-service teachers (Fernändez-Portero, 2022; Gan et al., 2020; Rafiq et al., 2022; Rosas-Maldonado et al., 2021; Soo & Goh, 2017), there is still an unfilled gap in identifying whether the pre-service teachers are fully equipped with the knowledge and competencies of conducting questionnaire surveys. Hence, this paper responds to this challenge by exploring the status quo of student teachers’ perceptions and the application of a questionnaire survey in foreign language education research.

Analytical Framework

The process of a questionnaire survey can be composed of three stages, namely, constructing questionnaires, administering questionnaires, and processing questionnaire data according to the time sequence (Dörnyei & Taguchi, 2010). Aside from these stages, Qin (2009) included other specific steps such as designing questionnaire items, piloting, and questionnaire evaluation. French and Che (2012) introduced skills and techniques on how to increase the response rate, how to implement a survey, how to design the survey tool itself, etc. Rowley (2014) gave novice researchers advice and support on designing good questionnaires, maximizing their response rate, and undertaking appropriate data analysis. By integrating the most important elements and items contained in these works, the current study adopts an analytical framework that includes three major stages: questionnaire construction, questionnaire administration, and questionnaire processing and reporting. Questionnaire construction involves a series of steps and procedures, including deciding on general features, like main parts, question types, rules of writing questionnaire items, etc. Questionnaire administration includes piloting questionnaires, sampling strategies, and strategies to increase the response rate. The final part of questionnaire processing and reporting involves procedures in processing questionnaire data, including doing reliability and validity analysis, statistical analysis, and summarizing and reporting questionnaire data. The three stages and nine main elements in a questionnaire survey comprise the framework (see Table 1) for categorizing and analyzing student teachers’ perceptions and application of questionnaire surveys.

Table 1.

Stages and Main Elements in a Questionnaire Survey.

Stages	Main elements
Questionnaire construction	The main parts of a questionnaire
	Question types in the questionnaire
	Rules of writing questionnaire items
Questionnaire administration	Piloting questionnaires
	Sampling strategies
	Strategies to increase the return rate
Questionnaire processing and reporting	Doing reliability and validity analysis
	Statistical analysis
	Summarizing and reporting questionnaire data

Methodology

Research Participants and Context

One hundred thirty eight student teachers took part in the study, of which 35 were postgraduates and 103 undergraduates, 7.97% were male and 92.03% were female. Both groups have completed academic subject courses in their earlier BA and MA studies and would graduate in the summer of that year. All the research participants of the present study came from the College of Foreign Languages in a university in A Province, China, which is a renowned, distinctive teaching-oriented, and research-oriented university, that aims to serve and lead teacher education in A Province. Starting in 1998, the College developed a teacher education program featuring research-based learning, integrated curriculum, cooperative learning, and humanistic outcome (RICH). Under the guidance of the RICH teaching rationale, students are expected to improve their subject knowledge, teaching skills, and humanistic quality. The program consists of studies in academic disciplines and pedagogical studies, including microteaching and teaching practice. The program also provides courses such as Academic English and Thesis Writing to prepare for bachelor’s and master’s theses. However, students are not necessarily well-equipped with specific research methods such as questionnaire surveys since the methodology is only taught and learned as one small part of the course. Therefore, in the present study, it will be important to examine how the student teachers perceive and apply questionnaire surveys to their foreign language education research.

Research Methods

This study adopted both quantitative and qualitative methods: questionnaire, content analysis, and interview. The questionnaire was mainly used to examine the general situation of student teachers’ perceptions of questionnaire surveys, and content analysis was to look into the application of this research method in their theses, while the interview was applied to seek a deeper understanding of participants’ perceptions and application of questionnaire survey.

Questionnaire

Due to its effectiveness in eliciting factual, attitudinal, and behavioral data or characteristics of a specified population (Dörnyei & Taguchi, 2010; French & Che, 2012), a questionnaire survey was deemed appropriate for this study. Since there was no already-made questionnaire related to the perceptions of questionnaire surveys, the questionnaire in this study was designed according to the analytical framework based on the descriptions of the questionnaire survey in the two books by Qin (2009) and Dörnyei and Taguchi (2010). The questionnaire was divided into two parts: perceptions of the questionnaire survey and background information questions. The former was composed of four sections: a general understanding of questionnaire survey, perceptions of questionnaire construction, perceptions of questionnaire administration, and perceptions of questionnaire processing and reporting. There were 17 items in total. The questionnaire was designed in Chinese and was distributed to them through online administration and one-to-one administration. Both administrations were conducted during a period after the participants had finished their thesis defense to ensure that they not only had a deep impression of doing research but were willing to spare time to answer the questionnaire. To increase the response rate, certain strategies such as sending red packets to the WeChat group (a widely used social platform in China), offering some desserts, and informing participants of the research purpose and process in advance, etc. were adopted during the administration.

Content Analysis

Content analysis was adopted to analyze the participants’ theses in which a questionnaire survey was applied as the main research method. With the consent of the participants, 10 theses were selected to study their application of the questionnaire survey, including five theses from undergraduates and five from postgraduates. The analysis was made according to the three stages of a questionnaire survey: questionnaire construction, questionnaire implementation, and questionnaire processing and reporting.

Interview

A semi-structured interview was run after the questionnaire survey to collect more qualitative data. The interview outline consisted of questions regarding respondents’ general understanding of the questionnaire survey, questionnaire construction, questionnaire administration, and summarizing and reporting questionnaire data. The interview was recorded and transcribed for analysis with the participants’ permission.

Four participants including two undergraduates (one boy and one girl) and two postgraduates (both girls), who had answered the questionnaire, agreed to be interviewed one by one. The average time each interview lasted was 20 min and every interviewee was provided with a small gift as a sign of appreciation at the end of the interview.

Data Collection and Analysis

Data were collected voluntarily between March and June 2018 from student teachers at the university in A province, China, utilizing random sampling. Confidentiality and autonomy were explained to participants at the beginning of data collection. The questionnaire was piloted before it was formally distributed. Before piloting the questionnaire, it was handed out to three postgraduate students and completed in our presence. Based on their feedback, some details, including the overall appearance of the questionnaire, clarity of questions, and instructions were adjusted. Then with the help of a head teacher, 32 students who majored in Educational English at another Normal University and would graduate in June of the same year took part in the questionnaire piloting online and the reliability and validity of the questionnaire were analyzed. The Cronbach’s alpha coefficient was between .5 and .6, which was accountable for the attitudinal questions, the number of items, and the piloting participants (Qin, 2003). Based on the piloting, questions without differentiation were deleted, and some options and expressions were adjusted according to the participants’ suggestions and answers. Then the questionnaire was formally distributed to the participants of the study.

Altogether 168 questionnaires were collected, but 30 of them were invalid because all the answers to different questions were the same. Hence, there were 138 valid questionnaires, 103 from undergraduates and 35 from postgraduates. The item reliability analysis showed that Cronbach’s Alpha coefficient of the final questionnaire was .672, indicating the data were reliable. Quantitative data analysis used descriptive statistics which informed the frequency and percentage of the student teachers’ perceptions of the questionnaire survey.

Content analysis was used to identify the key information from qualitative data. The researchers analyzed the qualitative data using content analysis and discussed the data analysis to achieve a synthesis in the data interpretation. First, 10 student teachers’ theses in which the questionnaire survey was applied as the main research method were selected. The coding scheme for idea generation was adopted from the descriptions of the questionnaire survey by Qin (2009), Dörnyei and Taguchi (2010), French and Che (2012), and Rowley (2014). Based on the three stages and nine main elements of a questionnaire survey presented in Table 1 in the “Analytical Framework,” the parts of the research design, findings, and questionnaire appendix in the 10 theses were carefully studied and coded. Then all the codes were categorized and the typical examples of data were selected so that some thematic patterns were identified and summarized (Bogdan & Biklen, 2007). Finally, the inductive content analysis of the transcribed interview data revealed the major aspects contributing to the student teachers’ perceptions and application of the questionnaire survey.

Findings

Student Teachers’ Perceptions of a Questionnaire Survey

The first research question concerning student teachers’ perceptions of a questionnaire survey can be answered from four aspects: general understanding of questionnaire survey, perceptions of questionnaire construction, perceptions of questionnaire administration, and perceptions of questionnaire processing and reporting.

General Understanding of Questionnaire Survey

Based on a descriptive analysis of the questionnaire, all the participants had some general but unsystematic understanding of the questionnaire survey. 63% of them had the awareness of giving specific reasons in their theses for selecting and applying the questionnaire survey. The interview data showed that all the interviewees were familiar with the questionnaire survey, but their understanding was limited. As two student teachers commented:

A questionnaire survey is a method in which questionnaires are used to collect information from one group and it is widely applied…It is simple to collect a large amount of data…But it is difficult to increase the return rate… (Student A)

I know the questionnaire survey, but I do not have a systematic understanding. With the emergence of online questionnaires, their application becomes convenient and widespread … (Student C)

Another comment echoed this point:

I generally know questionnaire surveys without theoretical knowledge.… Its application is convenient but the data collected from it may be invalid because of unreal answers… (Student D)

Perceptions of Questionnaire Construction

Constructing questionnaires involves deciding on the main parts of a questionnaire, question types, and rules for writing questionnaire items. As for the student teachers’ perceptions of the main parts in a questionnaire, the data showed that title, general instruction, specific instruction, and items were regarded as essential parts, with percentages of 87.7%, 83.3%, 74.6%, and 84.8% respectively. 68.8% of the respondents considered final thanks necessary in components of a questionnaire. However, only 31.9% and 15.2% of them reported that it was necessary to provide results feedback and researchers’ contact, indicating that these two parts were mostly overlooked by the participants.

The participants’ perceptions of question types in a questionnaire were also explored. The statistics showed that open questions, single-choice questions, and multiple-choice questions were more familiar to the respondents, with percentages of 81.9%, 87%, and 89.9% respectively. 61.6% of the respondents chose rank order items and 44.9% chose numeric items. The least known question types were the scaling techniques such as the Likert scale (26.1%) and semantic differential scale (15.9%).

Rules such as writing short and simple items, avoiding ambiguous or loaded words or sentences, and avoiding negative constructions, can help to improve the quality of questionnaires. To study the respondents’ perceptions of these rules, seven statements were designed in the questionnaire, and the respondents were asked to identify which of the following statements were inappropriately designed.

(1) I do not use English phrases which I am not familiar with.

(2) I learn some useful words, phrases, and sentences from the text, television, and movies. If I put these words, phrases, or sentences in English writing, I have an accomplishment sense.

(3) I feel shy and uneasy when I talk with others in English.

(4) I often go to the English corner to practice speaking English.

(5) Do you think that our school cannot offer elective courses in English movie appreciation?

(6) Learning English is very important.

(7) Do you think that the 3M foreign language teaching model is effective?

It can be noticed that none of these statements follow the rules of writing questionnaire items: (1) is a structure of double negative; (2) is too long; (3) is a double-barreled question; the word “often” in (4) is a non-specific word; (5) should be written neutrally; (6) is a question that is likely to be answered in the same way by everybody; (7) has a terminology “3M” without considering respondents’ knowledge. According to the questionnaire data, only (5) could be recognized to be inappropriate by more than half of the respondents (62.3%), which indicated that the respondents were more aware of the rule of writing question items neutrally. The percentages of the respondents who could recognize the rules of writing question items in (1), (2), (3), (4), (6), and (7) were 50%, 41.3%, 29%, 26.1%, 39.1%, and 34.1% respectively. This proved that most participants generally did not know enough about writing effective questionnaire items.

Besides questionnaire data, the interview data also revealed their lack of awareness in this respect, as two student teachers reported:

I do not know the basic rules of writing questionnaire items, and if some, avoiding grammar mistakes and ambiguous meaning… (Student A)

Rules such as avoiding double negative sentences, and explaining certain technical terms in advance. But I have no systematic understanding of writing questions… (Student D)

The results of student teachers’ perceptions of questionnaire construction are more clearly presented in Table 2.

Table 2.

Student Teachers’ Perceptions of Questionnaire Construction.

Stage	Main elements	Items (%)
Questionnaire construction	The main parts of a questionnaire	Title (87.7)
		General instruction (83.3)
		Specific instruction (74.6)
		Items (84.8)
		Final thanks (68.8)
		Promise to provide results feedback (31.9)
		Researchers’ contact (15.2)
	Question types in the questionnaire	Open questions (81.9)
		Single-choice questions (87)
		Multiple-choice questions (89.9)
		Rank order items (61.6)
		Numeric items (44.9)
		The Likert scale (26.1)
		Semantic differential scale (15.9)
	Rules of writing questionnaire items	(1) Avoid the structure of double negative. (50)
		(2) Avoid too long questions. (41.3)
		(3) Avoid double-barreled questions or ask one question at a time. (29)
		(4) Use specific words in the questions. (26.1)
		(5) Avoid leading questions. (62.3)
		(6) Avoid questions that are likely to be answered in the same way by every participant. (39.1)
		(7) Avoid terminologies in the questions. (34.1)

Perceptions of Questionnaire Administration

Piloting questionnaires, sampling strategies, and strategies to increase the return rate are the three main elements in questionnaire administration. In terms of the piloting questionnaire, the questionnaire data showed that 39.9% of the respondents had no awareness of piloting before formal administration. One interviewee declared that he had no idea of the necessity to pilot questionnaires. This proved the respondents’ lack of understanding of the importance of piloting questionnaires before administering them.

In terms of sampling strategies, random sampling was the best well-known strategy, with which 93.5% of the respondents were familiar. Cluster sampling, purposive sampling, and quota sampling were less known, with percentages of 46.4%, 42.8%, and 30.4% respectively. Snowball sampling (14.5%) and convenience sampling (13%) were the least known strategies, indicating that the respondents didn’t have a wide repertoire of sampling strategies. As one student teacher said in the interview:

I know little thing about sample selecting methods… (Student A)

As for strategies to increase return rate, seeking permission (87.7%) and stressing confidentiality (79.7%) were stressed. 39.1% of the respondents attached importance to emphasizing questionnaire instructions and 32.6% were aware of the strategy of announcing the questionnaire survey in advance. Compared with the above tactics, the presence of administrators and promising feedback on results were noticed by fewer participants, with a percentage of 23.9% and 20.3% respectively. It suggested that student teachers had limited knowledge about how to increase the response rate, which was also confirmed in the interview.

I notice the importance of increasing participants’ responses, but I have no idea how to achieve the goal… (Student B)

During administration, I am not familiar with some details, like things I need to do before delivering questionnaires and when participants were filling them out… (Student D)

The results of student teachers’ perceptions of questionnaire administration are presented in Table 3.

Table 3.

Student Teachers’ Perceptions of Questionnaire Administration.

Stage	Main elements	Items (%)
Questionnaire administration	Piloting questionnaires	No awareness of piloting before formal administration (39.9)
	Sampling strategies	Random sampling (93.5)
		Cluster sampling (46.4)
		Purposive sampling (42.8)
		Quota sampling (30.4)
		Snowball sampling (14.5)
		Convenience sampling (13)
	Strategies to increase the return rate	Seeking permission (87.7)
		Stressing confidentiality (79.7)
		Emphasizing questionnaire instructions (39.1)
		Announcing the questionnaire survey in advance (32.6)
		Presence of administrators (23.9)
		Promising feedback on results (20.3)

Perceptions of Questionnaire Processing and Reporting

The stage of questionnaire processing and reporting consists of procedures in processing questionnaire data, including doing reliability and validity analysis, statistical analysis, and summarizing and reporting questionnaire data.

The questionnaire data showed that 76.1% of the respondents had an awareness of reliability and validity analysis. However, when it comes to specific methods to test questionnaires’ reliability, such as inter-rater reliability, test-retest reliability, parallel form reliability, split-half reliability, and Cronbach alpha coefficient, 47.1% of the respondents reported that they did not know any method and Cronbach alpha coefficient was relatively more popular with a percentage of 18.1%. The other four methods were almost unknown to the respondents: 16.7%, 13.0%, 2.2%, and 2.9% respectively. In terms of questionnaire validity including content validity, construct validity, predictive validity, and concurrent validity, 50% of the respondents reported that they did not know anyone and content validity (38.4%) was relatively more familiar to them. The percentages for construct validity, predictive validity, and concurrent validity were 7.2%, 2.2%, and 2.2% respectively. It was evident that most participants were not familiar with the different types of validity, let alone the specific methods to test them. The lack of knowledge of reliability and validity analysis was also confirmed by the interview data.

Because questionnaires have been applied by other scholars, I think there is no need to test the reliability and validity… (Student A)

I know the necessity of reliability and validity analysis, but I don’t know the details … (Student B)

In terms of statistical procedures, the data showed that 57.2% of the respondents had no idea of selecting appropriate analysis methods such as T-tests, ANOVAs (Analysis of Variance), and correlation analysis, indicating their poor knowledge of statistical data analysis methods. The result was congruent with what the participants reported in the interview.

Analyzing questionnaire data is a big obstacle because I have no idea of choosing appropriate statistical analysis methods… (Student B)

I just know simple methods for data analysis and I feel confused to choose complicated statistical analysis methods… (Student C)

As for summarizing and reporting questionnaire data, 84.1% of the respondents agreed that questionnaire content should be reported; 79% stressed the description of participants’ basic information; 76.1% realized the importance of covering questionnaire limitations; 67.4% thought that the specific procedures of questionnaire data processing and reporting should be provided; and 63.8% paid their attention to the process of questionnaire construction, indicating that questionnaire construction was the least important thing to be considered in the stage of data reporting.

Table 4 clearly shows the results of student teachers’ perceptions of questionnaire processing and reporting.

Table 4.

Student Teachers’ Perceptions of Questionnaire Processing and Reporting.

Stage	Main elements	Items (%)
Questionnaire processing and reporting	Doing reliability and validity analysis	Awareness of reliability and validity analysis (76.1)
		Inter-rater reliability (16.7)
		Test-retest reliability (13.0)
		Parallel form reliability (2.2)
		Split-half reliability (2.9)
		Cronbach alpha coefficient (18.1)
		Content validity (38.4)
		Construct validity (7.2)
		Predictive validity (2.2)
		Concurrent validity (2.2)
	Statistical analysis	No idea of selecting appropriate analysis methods such as T-tests, ANOVAs (Analysis of Variance), and correlation analysis (57.2)
	Summarizing and reporting questionnaire data	Questionnaire content (84.1)
		Participants’ basic information (79)
		Questionnaire limitations (76.1)
		Procedures of questionnaire data processing and reporting (67.4)
		Process of questionnaire construction (63.8)

Student Teachers’ Application of the Questionnaire Survey

The answer to the second research question pertaining to student teachers’ application of the questionnaire survey can be divided into three sections according to the three stages of the questionnaire survey: constructing questionnaires, administering questionnaire survey, processing and reporting questionnaire data.

Constructing Questionnaires

This paper intended to analyze the participants’ construction of questionnaires from three aspects: the developing process, the main parts of questionnaires, and the rules of writing question items.

For the developing process, seven participants mentioned in their theses that their questionnaires were constructed based on existing questionnaires, as could be seen from the following descriptions:

the questionnaire is modified based on the…

the questionnaire is based on the…

the questionnaire is designed by other scholars…

This study employs a questionnaire adapted from…

the questionnaire is designed by referring to a ready-made relevant questionnaire…

the questionnaire is developed and validated by other scholars…

Two theses didn’t provide the questionnaire development process and the remaining one stated that the questionnaire was designed by the writer based on the theoretical framework. In the seven theses that contained information on the developing process, the adapting process was just described in one single sentence with little detailed information. Only one stated the steps with more details:

Considering the different backgrounds of studies and participants’ performance after a two-month classroom observation, there are a few changes that inevitably need to be made in the content of the questionnaire. To be specific, in the original version…

As for the main parts of the questionnaires, four questionnaires in the theses lacked an appropriate title with just “questionnaire for teachers” or “questionnaire for students.” Two questionnaires missed general instructions. Eight questionnaires failed to give specific instructions. Only three theses showed thanks at the end of the questionnaires. All 10 questionnaires did not provide researchers’ contact and results feedback. This result indicated that the participants had an incomplete understanding of the main parts of the questionnaires.

As for the rules of writing question items, examples of violation of the rules were evident throughout the questionnaires in the 10 theses, which showed that the participants paid little attention to the rules while constructing questionnaire items.

(1) Double-barreled Questions:

In English class, I positively answer questions and actively participate in classroom activities;

Learning English makes me happy, and my spoken English has made great progress;

Teachers should cultivate students’ open minds to foreign cultures, and make them actively involved in foreign cultures.

(2) Ambiguous or unclear questions:

I often practice English writing or speaking;

I keep up with important new technologies;

I am afraid of talking about English, so I rarely open my mouth to talk in English.

(3) Questions with loaded words or biased sentences:

Do you have a good learning habit of taking English notes to memorize language points?

Aren’t you willing to help your classmates during cooperative learning in English class?

Which aspects of your English ability have been improved by this kind of learning method?

(4) Questions without differentiating answers:

Do you agree that learning English is a way to know another culture?

Dictation plays a role in English learning;

Do you think technology is important in English teaching?

(5) Questions containing acronyms, abbreviations, jargon, or terminologies

I know what the goal is like under a “comprehensive vision.”

Administering Questionnaire Survey

Administration of the questionnaire survey was analyzed including piloting questionnaires, selecting samples, and strategies to increase the quality and quantity of participants’ responses.

Among the 10 theses, seven theses referred to nothing about the piloting questionnaire, and the other three theses applied the piloting, among which two just described the process without analyzing the validity and reliability of the questionnaire. Only one thesis gave more details on the piloting questionnaire as follows.

The questionnaire is tested among ××students and 30 of them fully complete it. The results are analyzed by SPSS 20.0 and Bartlett’s Test and KMO value are conducted, indicating the questionnaire is valid, and the value of Cronbach’s alpha shows the questionnaire is reliable. That is to say, the questionnaire is appropriate to conduct factor analysis, and the questions in the questionnaire are interrelated.

As regards participants’ selecting samples during their application, none of the 10 theses introduced sampling strategies directly and they just mentioned participants’ basic information or reasons for which certain subjects were chosen. For instance:

Since the author has the internship here and owns access to the school, those subjects are selected as research subjects…

The writer teaches ×× parallel classes in a high school as an intern teacher…

In terms of strategies to increase the quality and quantity of participants’ responses, only two theses talked about some strategies, such as offering a small reward, delivering in the researcher’s accompany, reading general instructions before handing out questionnaires, and stressing anonymity. In the interview, two participants expressed their lack of knowledge about how to improve the questionnaire return rate.

I notice the importance of increasing participants’ questionnaire responses, but I have no idea how to achieve the goal… (Student B)

During the questionnaire administration, I want to know what I need to do before handing out questionnaires when participants are filling them out, as well as after they return questionnaires… (Student D)

Processing and Reporting Questionnaire Data

Participants’ application of processing and reporting questionnaire data includes steps of processing data, choices of data analysis methods, and report of questionnaire survey results.

It was found that most of the 10 theses gave little information about data processing and only one thesis included some details in the sentence “After being scored and calculated, all the results are analyzed by SPSS 20.0….”

In terms of questionnaire data analysis, most of the 10 theses used descriptive statistics with the frequency distribution of the questionnaire data. Only one thesis applied an independent sample T-test to compare the difference in the same variable between two groups of samples. Other inferential statistics, such as correlation analysis, one-way ANOVA, and simple linear regression were paid little attention to. The lack of knowledge about statistical analysis methods was also confirmed in the interview.

The data analysis in my study is mainly showing the frequency in a bar chart or a simple form, which is just a simple statistical analysis method… (Student C)

As for the questionnaire survey results report, only one thesis showed its results in running text, and the other nine theses displayed their research results in the form of tables, bar charts, and pie charts. However, many problems existed in results reporting, including having no specific title for a table, directly showing tables or figures generated from the software, representing too much information in tables without enough explanation, representing in an inappropriate layout format, etc.

Discussion

Both quantitative and qualitative data revealed that student teachers had some basic but inadequate and unsystematic understanding of the questionnaire survey. Most student teachers developed questionnaires based on existing questionnaires without justifying and explaining adjustments, which is consonant with the findings in Qin (2009) and Larson-Hall and Plonsky (2015). Apart from the deficiencies discovered in the questionnaire surveys applied in second language research such as the lack of reliability and validity, the lack of pilot tests, and the inappropriate use of sampling strategies (Xu & Li, 2018), this study also revealed other shortcomings in student teachers’ perceptions and application of questionnaire surveys, such as the limited knowledge about the main parts of the questionnaires, the rules of constructing questionnaire items, increasing the response rate, and statistical data analysis methods. These findings indicate that student teachers’ insufficient and unsystematic knowledge about questionnaire surveys greatly impairs their competence in constructing and conducting questionnaire surveys in foreign language education research.

One possible factor concerns the curriculum design of the teacher education program at the University. Though it is a teaching-oriented and research-oriented university, the former is more prominent than the latter. Since many courses aim at helping student teachers develop their teaching competencies, courses designed for developing their research method awareness and scientific literacy are insufficient. For student teachers at this College, there is only one research-related course Academic English and Thesis Writing which intends to prepare students with the necessary knowledge and skill to write their theses. According to what the participants said in the interview, this course is provided in the second semester of the third year during their 4 years of college study, with only two credits and 32 class hours in total. Due to the limited class time and too much content covered in the course, more attention is paid to how a thesis can be written from the abstract to the references. Therefore, systematic learning of questionnaire surveys and other research methods seemed impossible. Although teacher educators at the College have taken some measures to solve the problem such as combining online and offline learning, more time and effort are spent on choosing a research topic, and writing a research proposal or literature review rather than specific sections on research methods such as constructing and administering questionnaires. It was noticeable that systematic learning of questionnaire survey theories was not paid sufficient attention to in both online and offline learning in the teacher education program. The lack of systematic training in research methods contributed a lot to teachers’ knowledge about and competence in designing a high-quality questionnaire (Qin, 2009).

Another factor that may have contributed to this result may be that teacher educators did not have enough awareness of the importance of research methodology and did not give student teachers enough opportunity and support to apply what they know to their thesis writing. As novice teachers, they usually lack routines and expert teachers’ knowledge structure (Enow & Goodwyn, 2018; König et al., 2019), making it difficult for them to adapt their knowledge to real education research contexts. Hence, many researchers (e.g., Ball, 2000; Pantic & Wubbels, 2010; Struyven & Meyst, 2010) highlighted the importance of integrating knowledge with skills in professional tasks and situations. Combining theoretical learning and applying it in practice is the best way to learn and develop competencies (Shuanghong et al., 2021). Only by encouraging student teachers to apply their knowledge of questionnaire surveys to different research contexts can they learn the complex details and techniques involved in questionnaire construction, administration, and data analysis. Thus, their understanding of this research method could be deepened and strengthened during the process of learning by doing.

Conclusion and Practical Implications

This study shows that although the participants generally know about the questionnaire survey, their perceptions were not comprehensive and systematic. In their theses, most participants did not include a full explanation of the questionnaire construction process. Specific instruction, researchers’ contact, results feedback, and final thanks were always ignored in the main parts of their questionnaires. Violations of rules of writing question items were noticeable in the questionnaires included in their theses. They had an awareness of the importance of selecting samples. piloting questionnaires, doing reliability and validity analysis, and reporting questionnaire data, but they did not know the detailed steps and techniques. Both the quantitative and qualitative data prove that more research and more training provided by universities for students are urgently needed.

Based on the above findings, many pedagogical implications can be put forward to improve student teachers’ perceptions and application of questionnaire surveys in foreign language education research.

Firstly, some adjustments in the curriculum design are necessary. More courses concerning research methodology should be added to the teacher education program in normal universities to train student teachers’ awareness and competencies in conducting studies properly and scientifically. More credits or class hours and more systematic instruction should be provided to broaden their understanding of research methods. As was suggested by Gan et al. (2020) and Jose and José (2022), teacher education programs need to be updated to provide student teachers with opportunities and requirements for developing both an intellectual understanding and skill demonstration or knowledge transformation. In the case of the questionnaire survey, not only the general content information about the questionnaire survey, but also the specific skills and techniques including the main parts of the questionnaire, question types, rules of writing question items, sampling strategies, strategies to increase the response rate, steps of doing reliability and validity analysis, and statistical analysis methods should be covered in the teaching syllabus. In addition, more research-based projects such as workshops and learning communities can be organized to encourage student teachers to integrate their theoretical knowledge into their research practice, as is proved that the connection between theory and practice through investigation is one of the characteristics of good practice in teacher education provision (Linda, 2022).

Secondly, teacher educators can play a better role in facilitating teacher candidates’ acquiring competencies and enabling them as active learners through group work and teaching practices (Shuanghong et al., 2021). Since teachers are key agents and also gatekeepers to determine what enters and gets into the classroom (Lee & Hwang, 2022), they should take the initiative to innovate teaching content and teaching methods, create an interactive and collaborative learning environment, design tasks to encourage theory-practice integration and provide timely feedback to give student teachers positive learning experiences, which in turn can influence how they perceive and conduct foreign language education research in their teaching practice and future classrooms (Rosas-Maldonado et al., 2021).

Thirdly, student teachers themselves should take the initiative and empower themselves with the basics of the questionnaire survey. Reading more methodology books, attending more research-related lectures and workshops, applying theories to their teaching practice, etc. would all help them generate well-designed questionnaires and yield data-proven results.

Lastly, research participants should be encouraged to take on their responsibility in treating questionnaires seriously and providing their genuine opinions or answers while filling out questionnaires. They should realize that they can also contribute to scientific discoveries through their serious attitude and sincere collaboration with the researchers.

Limitations and Future Research

The findings of this study are based solely on student teachers’ perceptions and the application of a questionnaire survey. Teacher educators, curriculum designers, or other parties’ perspectives would help to verify the results. Besides, in comparison with the total number of participants involved in the questionnaire, the number of theses that were investigated in the current study is insufficient due to the inaccessibility of more willing participants. A larger sample size for content analysis would help to provide more insights into the process of the questionnaire survey application. In addition, the present study was conducted in China only, so the findings cannot necessarily be generalized to other countries. However, as the questionnaire survey conducting skills focused on in the present study relate to challenges that might occur in other contexts as well, implications on student teachers’ perceptions and application of questionnaire survey in the present study may be of interest in other national and international contexts and teacher education programs as well. Therefore, it could be worthwhile for future research to transfer our empirical approach to different cultural contexts. Future studies could also follow up on student teachers’ development of knowledge and competencies of questionnaire surveys from undergraduates to postgraduates by doing some longitudinal studies, which is likely to open up a new relevant research field in foreign language teacher education.

Supplemental Material

sj-docx-1-sgo-10.1177_21582440241245621 – Supplemental material for Student Teachers’ Perceptions and Application of Questionnaire Survey in an EFL Context

Supplemental material, sj-docx-1-sgo-10.1177_21582440241245621 for Student Teachers’ Perceptions and Application of Questionnaire Survey in an EFL Context by Pingping Hu and Honglei Xu in SAGE Open

Footnotes

Acknowledgements

The authors wish to sincerely thank all the participants for their active engagement in this research process.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethical Approval

This is not applicable since no ethics committee is available.

ORCID iD

Pingping Hu

Data Availability Statement

The data that has been used is confidential.

Supplemental Material

Supplemental material for this article is available online.

References

Ball

D. L.

(2000). Bridging practices: Intertwining content and pedagogy in teaching and learning to teach. Journal of Teacher Education, 51(3), 241–247. https://doi.org/10.1177/0022487100051003013

Bao

(2012). A survey on the usage of statistical analysis method in foreign language teaching research in China. Foreign Language World, 148(1), 44–51.

Bogdan

R. C.

Biklen

S. K.

(2007). Qualitative research for education: An introduction to theory and methods (5th ed.). Allyn & Bacon.

Brown

J. D.

(2001). Using surveys in language programs. Cambridge University Press.

Dörnyei

Taguchi

(2010). Questionnaire in second language research: Construction, administration, and processing (2nd ed.). Routledge.

Enow

Goodwyn

(2018). The invisible plan: How English teachers develop their expertise and the special place for adapting the skills of lesson planning. English in Education, 52(2), 120–134. https://doi.org/10.1080/04250494.2018.1438119

Fernändez-Portero

(2022). Measuring preservice foreign language teachers’ attitudes toward inclusive education through a newly developed scale. Foreign Language Annals, 55, 1188–1211. https://doi.org/10.1111/flan.12660

French

Che

M. F.

(2012). Designing and using surveys as research and evaluation tools. Journal of Medical Imaging and Radiation Sciences, 43, 187–192. https://doi.org/10.1016/j.jmir.2012.06.005

Gan

Z. D.

Liu

F. L.

Yang

C. C. R.

(2020). Student-teachers’ self-efficacy for instructing self-regulated learning in the classroom. Journal of Education for Teaching, 46, 120–123. https://doi.org/10.1080/02607476.2019.1708632

10.

Gao

X. S.

(2004). A critical review of questionnaire use in learner strategy research. Prospect, 19(3), 3–14.

11.

Gao

Y. H.

L. C.

(1999). The development tendency of applied linguistics research methods in China and Western Countries. Foreign Language Teaching and Research, 118(2), 8–16.

12.

Gillham

(2008). Developing a questionnaire (2nd ed.). Continuum.

13.

Huang

H. F.

(2014). The comparative analysis of questionnaire survey in second language at home and abroad. Huazhong Normal University.

14.

Jose

José

(2022). Integrating augmented reality in language learning: Pre- Service teachers’ digital competence and attitudes through the TPACK Framework. Education and Information Technologies, 27, 12123–12146. https://doi.org/10.1007/s10639-022-11123-3

15.

König

Bremerich-Vos

Buchholtz

Fladung

Glutsch

(2019). Pre-service teachers’ generic and subject-specific lesson-planning skills: On learning adaptive teaching during initial teacher education. European Journal of Teacher Education, 43(2), 131–150. https://doi.org/10.1080/02619768.2019.1679115

16.

Kormos

(2003). Questionnaires in second language research: Construction, administration, and processing by Zoltan Dörnyei. TESOL Quarterly, 37(4), 773–774. https://doi.org/10.2307/3588226

17.

Larson-Hall

Plonsky

(2015). Reporting and interpreting quantitative research findings: What gets reported and recommendations for the field. Language Learning, 65(S1), 127–159. https://doi.org/10.1111/lang.12115

18.

Lee

Hwang

(2022). Technology-enhanced education through VR-making and metaverse-linking to foster teacher readiness and sustainable learning. Sustainability, 14, 4786. https://doi.org/10.3390/su14084786

19.

Linda

l. V.

(2022). Best practice in teacher education: What is research telling us? Journal of Education for Teaching, 48(3), 271–273. https://doi.org/10.1080/02607476.2022.2075189

20.

Moser

C. A.

Kalton

(1971). Survey methods in social investigation. Heinemann.

21.

Oppenheim

A. N.

(1992). Questionnaire design, interviewing, and attitude measurement (new ed.). Pinter.

22.

Pantic

Wubbels

(2010). Teacher competencies as a basis for teacher education – Views of Serbian teachers and teacher educators. Teaching and Teacher Education, 26(3), 694–703. https://doi.org/10.1016/j.tate.2009.10.005

23.

Perry

F. L.

Jr. (2005). Research in applied linguistics. Lawrence Erlbaum Associates Publishers.

24.

Qin

X. Q.

(2003). Quantitative data analysis in foreign language teaching research. Huazhong University of Science and Technology Press.

25.

Qin

X. Q.

(2009). Questionnaire survey in foreign language teaching. Foreign Language Teaching and Research Press.

26.

Rafiq

K. R. M.

Yunus

M. M.

Susiati. (2022). Re-envisioning technological pedagogical content knowledge and online teaching readiness of English for foreign language pre-service teachers in language teacher education. Frontiers in Psychology, 13, 927835. https://doi.org/10.3389/fpsyg.2022.927835

27.

Rosas-Maldonado

Durán-Castro

Martin

(2021). The socio-emotional influence of past teachers on novice English teachers’ beliefs. Profile: Issues in Teachers’ Professional Development, 23(2), 215–230. https://doi.org/10.15446/profile.v23n2.90809

28.

Rowley

(2014). Designing and using research questionnaires. Management Research Review, 27(3), 308–330.

29.

Shuanghong

Hannele

Vilhelmiina

Leila

(2021). Finnish student teachers’ perceptions of their development of 21st century competencies. Journal of Education for Teaching, 47(5), 638–653. https://doi.org/10.1080/02607476.2021.1951602

30.

Soo

R. S.

Goh

H. S.

(2017). Pre-service English teachers’ reticent beliefs towards oral participation in EAP classrooms. Asia-Pacific Education Researcher, 26(3–4), 155–162. https://doi.org/10.1007/s40299-017-0336-3

31.

Struyven

Meyst

M. D.

(2010). Competence-based Teacher Education: Illusion or reality? An assessment of the implementation status in Flanders from teachers’ and students’ points of view. Teaching and Teacher Education, 26(8), 1495–1510. https://doi.org/10.1016/j.tate.2010.05.006

32.

Tuleja

E. A.

Beamer

Shum

Cham

E. K.

(2011). Designing and developing questionnaires for translation tutorials. IEEE Transactions on Professional Communication, 54(4), 392–405.

33.

Wen

Q. F.

Lin

(2016). The application tendency of applied linguistics research methods from 2001 to 2015. Modern Foreign Language, 39(6), 842–852.

34.

J. F.

(2018). Retrospection and reflection on ESL teacher research method home and abroad (2000–2017). Journal of PLA University of Foreign Language, 4, 87–95.

35.

Zhang

X. Q.

(2016). Analysis of the quality of results reporting of L2 questionnaire surveys in and outside China. Central China Normal University.

36.

Zheng

X. M.

Wang

Y. S.

(2014). How to apply questionnaire survey properly in foreign language research—Based on the analysis of articles in China Foreign Language CSSCI Journals (2008–2013). Technology Enhanced Foreign Language Education, 158(4), 8–13.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB