Assessing Multinational Interest in STEM: Implementing a Comparative Survey Research Study in China

Abstract

Abstract

This paper introduces the design and implementation of a comparative survey research project, amistem [Assessing Multinational Interest in stem (Science, Technology, Engineering, and Mathematics)] in China, the U.S., and Australia that is being conducted by researchers at Indiana University. We begin with an overview of how survey research has developed in China during the past two decades, especially in the field of higher education. This analysis provides the backdrop for considering the challenges we faced in “internationalizing” the stem survey research project. Part two of the paper illuminates the difficulties of implementing the study in multiple contexts, including China. Part three of the paper examines challenges related to the translation and cultural adaptation of the u.s.-based survey for the Chinese context. Our conclusions shed light on the contexts, processes, challenges and possibilities of conducting multi-national survey research in China, as well as on ways to insure that multi-national surveys are culturally appropriate and comparable.

Introduction

This article presents a case of the challenges and potential benefits, from development and design to implementation and analysis, of conducting cross-cultural research. The focus is amistem, Assessing Multinational Interest in stem (Science, Technology, Engineering, and Mathematics), a survey research project conducted in China, the U.S., and Australia by researchers at Indiana University. An examination of how survey research on student learning in higher education has developed in China during the past two decades provides the backdrop for explaining the “internationalization” of amistem-China and what we learned from the process. Discussion of the difficulties of implementing the study is followed by a summary of the challenges associated with the linguistic and socio-cultural translation of the u.s.-based English language survey for the Chinese context. Our conclusions shed light on the difficulties and possibilities of conducting multi-national survey research in China, as well as on ways to insure that multi-national surveys are culturally appropriate and comparable.

The Context for amistem Implementation: Survey Research in Chinese Higher Education Literature, 1990 to 2012

During the past two decades, scholars in China have increasingly used survey methodology to study dimensions of teaching and learning in higher education. To understand this research context and the challenges it may suggest for cross-national projects like amistem, we investigate the characteristics of Chinese survey research on higher education teaching and learning. In our discussion of sampled studies, we summarize 1) the target population, sampling techniques and rationales of the surveys; 2) the key themes and questions of the studies; 3) considerations of survey validity and credibility; and 4) conclusions drawn from the studies, including social and policy implications. Our sample is taken from a comprehensive search of cnki 1 academic papers using survey research published between the years 1990 to 2012. Twenty-nine papers representing 26 projects were generated in this category. Less than 1% of published papers on higher education in the cnki database used survey research methodology, a fact that raises interesting comparative questions about research traditions, training and priorities.

Themes and Content Areas of Survey Research on Student Learning

The psychological state of college students, language and distance learning, employment issues, and college students’ opinions on current educational policies are the main areas of investigation in our sample. In the domain of the psychological state and development of college students, several research papers focus on the motivations and aspirations of academic success among college students based on students’ self-reported data. For example, one research paper draws on survey data from 2007 students in 13 universities to investigate the relationship between procrastination and academic success. 2 The researchers studied the characteristics and factors related to types of procrastination, as well as the relationship of procrastination to students’ learning attitude, aptitude, motivation and institutional characteristics, such as reputation of university and student year in college, gender and college entrance examination score. Funded by Ministry of Education social science grants, research projects like this one have the capacity of sampling students from various institutions yielding significant results that take into account both institutional and individual characteristics. The method of using self-reported survey data proves an effective choice for studying students’ psychological states and development, which is in the subjective domain of human development, and allows researchers to identify in students’ responses what may be unique Chinese characteristics.

A large portion of our sample projects fall into the domain of psychology and language learning. Questions studied include student suicide and psychological pressure, sometimes with a focus on specific groups of students. 3, 4, 5 In the domain of teaching and learning, distance education and student learning outcomes and satisfaction are prominent research topics. Most online surveys address issues such as why respondents chose distance education or what aspects of distance education attracts participants, rather than probing deeply into the processes of teaching and learning. In other words, the focus of the projects is whether and in what areas distance education is needed, rather than about how to improve learning outcomes through distance education, which is a crucial area that calls for more research.

Over 30% of the survey research papers in our pool are designed to respond to contemporary social and educational issues and policies at the national level. For example, during the past two decades, the Chinese government initiated state-aided student loan policies or need-based tuition remission policies for students from low-income families. Some survey-based longitudinal research projects traced the influence of these policies, not only in relation to learning outcomes, but also in order to investigate the social-psychological dimensions of students’ development as part of the influence of these policies. 6 Survey research projects in this category provide important empirical evidence to policy making and implementation processes both at the national and institutional levels, and in so doing have the potential to further the process of creating, critically assessing and communicating “information that is useful in understanding and improving policies.” 7

In contrast to research on national level policies, just over 10% of the papers focus on institutional policy and administrative reforms, ranging from cafeteria renovation to the privatization of dormitory management. Such studies provide important information on student opinions on university reforms, and as such reflect one of the most important purposes of research, that is, to inform practice, especially institutional change.

Student employment is another area to which survey research notably contributes. Most of the surveys in our sample take the form of voluntary online surveys across regions and institutions, with the surveyed population being college students, especially seniors. Questions include categories of career aspiration, relationship between college education and career development, and satisfaction level regarding career development services. Interesting examples from some surveys ask students to rate the level of importance of factors influencing student success on the job market, including academic achievement, family background, appearance (professional image), and overall presentation. These options clearly parallel distinct patterns of hiring priorities and practices in contemporary China. 8, 9, 10

Notably, survey research as a methodology has contributed to our understanding of the processes of teaching and learning, especially from 1990 to 2000, when collecting teaching evaluations was not a common practice in Chinese universities. Surveys in this category serve a significant function of providing systematic feedback on teaching, although most findings in these projects are not generalizable.

Concerns and Challenges of Survey Research

Review of the studies in our sample brought into sharper focus several issues that could potentially affect the validity and credibility of research findings. First, given constraints of funding, some research projects generated findings from a small sample size, lessening the validity of statistical models being used. In some papers, convenience samples came from only one university or only a few classes. Survey research within a single institution does not have the capacity to generate findings that are applicable to a larger population, although qualitative research at one institution can provide meaningful findings on teaching and learning, and provide the springboard for survey questions or test taken-for-granted assumptions about teaching, learning and student and faculty experiences.

Meanwhile, in most projects that drew samples from multiple higher education institutions, participants were from tier one universities and did not represent the diversity of Chinese higher education. We expect that researchers will turn more of their attention to student populations in second and third tier universities, whose experiences and learning outcomes are critical for assessing pedagogical and policy reforms.

Additionally, in most projects in our sample researchers considered student age, grade and major as analytical categories when examining correlations with individual learner characteristics. Adding to such studies family background, including socio-economic background, parents’ professions, and geographical origin could more comprehensively reflect individual dimensions crucial to understanding student learning. On the other hand, some surveys in our data pool include an item asking participants to choose whether they are from urban or rural areas. Questions such as this one may be problematic in projects with a small sample size, in which students’ identities are difficult to sufficiently protect. The question of whether and when to include items referencing student background raises important ethical concerns for broad discussion.

Our sample of studies also raise a question of the relationship between the researcher and the researched. To the best of our knowledge, there is no equivalent structure and process in China to what u.s. institutions call an Institutional Review Board for Protection of Human Subjects (irb). Survey research requires researchers to maintain relative distance from the subjects of research. Such distance is crucial when investigating issues such as quality of teaching and students’ levels of satisfaction regarding different aspects of their universities. A small number of papers in our data pool report findings on teaching quality that were collected by instructors from students in their own classes. Whether such surveys result in “genuine” findings is an important question. Ensuring that data remain unidentifiable is a fundamental requirement of most survey research.

Finally, two critical issues emerged from our overall analysis of studies in our pool with direct application to our implementation of amistem. First, the return rate of most surveys was very high; the lowest was 88%, and over half of our studies reported a 100% return rate. Such admirable rates seem to be a result of surveys being administered before or after class sessions by instructors, which calls into question another foundational principle of survey research, that it is voluntary. Second, pilot studies to test and then refine survey items were a step missing in most of the projects we analyzed.

Discussion and Reflection

Through our review, we identified significant survey research projects on learner characteristics, attitudes and aptitude, and how these impact student learning outcomes, topics and findings relevant to the amistem project. Less research focuses on the processes of learning and the interactions between teachers and learners, which suggests the need for new approaches to such questions. Furthermore, survey research has been relatively well funded, especially at the national level, in the Chinese higher education arena. Adaptations of internationally recognized surveys, such as the u.s.-based National Survey on Student Engagement, have increased during the past decade and seem to have had a positive impact on the development of survey research in China, as well as broadened and deepened capacity for institutional research.

Some of this type of survey research has been inspired by practical issues identified in the teaching and learning process, whereas a majority of the projects started with certain theoretical models or established surveys in the U.S., U.K., or Germany, but with significant revision to better “fit” the Chinese student population and context. Survey research in this category has the potential to significantly contribute to learning theories by bringing to transnational research data from Chinese students’ population. Providing the structures and opportunities for cross-cultural collaboration and data sharing is one of the key goals of our stem project, and we turn next to some of the challenges associated with that process.

Gaining Access for amistem: The Challenges of Collecting Survey Data in Cross-National Contexts

In the u.s. there has been substantial rhetoric about the need to increase the general science literacy of the populace as well as the number of individuals pursuing degrees and careers in science, technology, engineering and mathematics (stem). The u.s. is not alone in expressing these goals, which have become a signature of global education and economic reform policies. 11, 12, 13 For example, key education policy documents in China claim, “Education development has vastly enhanced the quality of the entire nation, and stimulated innovation in science, technology and cultural prosperity . . .” 14

In an effort to understand the nature of experiences that generate and maintain interest in stem, we conducted a survey in 2012 of university students and faculty across the u.s. The results yielded a number of interesting findings: including support for the notion that many individuals report first becoming interested in stem at early ages (i.e., before u.s. 6th grade) and that women and men point to different factors as key to the generation and maintenance of their interests. We were left wondering how American student experiences compared with those from other nations. As a first step toward finding out, we set out to revise the 2012 survey and to secure participation from universities in China and Australia to provide data with which we could make robust comparisons.

To facilitate participation we decided that for the first “foreign” stages of the study, it made sense to work with institutions where we already had contacts. Thus, in this large-scale “pilot” we sought universities in China and Australia where our School of Education or University had established collaborative research agreements. The rest of this section outlines our experiences in inviting and convincing those universities and students to participate in the study. Our intent is to share our experiences with those who might want to conduct similar work in cross-cultural settings.

Getting Noticed

In order to access students at universities we needed some point of entry. In the u.s. we have access to data on the degree production of colleges and universities and used these data to select a random sample of schools to solicit after stratification on geography, size, control (i.e., public or private) and primary degree level (i.e., 2-year or 4-year). For each school selected we took the approach of contacting administrators at the institution to inform them of the study and ask their willingness to participate. These initial contacts were made via paper mailing or via email. After we decided to make a round of phone calls to the institutions we quickly realized that these initial contacts may have never been read by those we intended as most responded that they never received the mailings we sent. We are not totally sure why this would occur, but this partially explains why we do not get any form of response from some schools.

Survey Fatigue

Once we are able to get the attention of an administrator and solicit their participation, the most common response we hear is: “We’re interested in the study and the results, but our students are already over-surveyed and thus we shouldn’t participate and make things worse.” This is akin to the nimby (“Not in my backyard”) issue in society related to the placement of necessary facilities that none of us want near our homes (e.g., sanitation or sewage facilities). In this case it’s nami (“Not at my institution”). To be clear, it rarely seems that the administrators just don’t want to be hassled. Essentially what we hear from administrators is that they agree that the work is important, but it is going to have to be done elsewhere. These administrators usually seem to understand the irony of this situation and often point the finger at themselves saying that it is internal evaluation surveys that are causing the problem. Regardless of the source, all agree that the problem is getting worse. This trend seems to be confirmed by anecdote 15 and research. 16

While we generally sense that smaller institutions are less likely to give us a NAMI response and to be interested in participating, this is not an absolute. In terms of conducting survey research, this situation can make things extremely difficult to create any sort of large dataset.

Thankfully, some institutions seem to be forward thinking and have figured out a solution. We have come across a few schools who are attempting to mitigate the potential for survey fatigue by randomly assigning students to groups each year and having a quota on the total number of surveys each group (or student) can be solicited for. This means that a given student sees no more than a handful of survey requests each year, but that his/her peer may get requests for different surveys based on the sampling. Researchers request access to one of the sets and once the sets are assigned for a semester, no more solicitations will be entertained. On the positive side, this can provide access to a randomly selected sample that may have been off limits previously. Conversely, this approach provides access to a sample that may not match the ideal sample. Based on the prevalence of fatigue, we think many of the larger higher education institutions will likely go the route of the quota system to control the number of studies students are asked to participate in.

Scheduling

Even if we get the opportunity to speak with administrators and they are willing to have their school participate in the study, another hurdle to overcome can be scheduling. Our first two attempts to survey schools in the u.s. came in spring or the 2nd term of the academic year. Simply put, this is not a good time to try to get a survey out to students and faculty. While the later part of the academic year may be desirable to gauge change or to evaluate experiences of the preceding months, many others want to do the same and so a survey logjam is created. This is exacerbated by the fact that many institutional evaluation efforts occur at the same time for the same reasons. This confluence makes many administrators back out of participating. We often heard that the “ideal” window of time for new surveys is likely between the beginning of the year, when students are asked to complete many surveys about expectations, and the end of the first term, when students and faculty are busy with exams/work. In our current campaign we are on a timetable where we will survey students from the middle of the first semester (October) through early in the second semester (March).

As with the quota system mentioned above, some schools have figured out that a centralized control of on-campus survey efforts is important. It seems that this task generally falls to offices of institutional research, if they exist. At a visit with one such manager, he unfolded a large schedule on which all of the survey efforts at his institution were marked. There was not much “space” on the schedule, and he was concerned about the amount of surveying going on that he was unaware of. While this approach may limit collecting data at the same time across institutions, the ability to get in the official queue probably means better chances for success.

Ethics Review

When the stars have aligned, and a university is willing to participate and has an opening in their schedule, the next hurdle may be passing their own ethics review or approval for human subjects research. A lengthy discussion of the need to have human subjects research approved in the u.s. is beyond the scope of this paper. There are federal guidelines for what types of practices are acceptable in research and as noted above most institutions that conduct research have internal boards (irbs) that review research proposals to make sure their procedures will abide by these norms and regulations. Usually, studies approved by one irb are accepted by other institutions through reciprocity. However, exceptions to this exist. The current survey work is considered exempt from regular review because it is considered low risk. Yet, at a number of institutions we have had to complete various levels of documentation (including new irb protocols) and review to be allowed to collect data. We have not let this deter us from trying to get these institutions to participate, but this inevitably delays the process and in one case caused us to miss our survey window.

Conducting research internationally adds another layer of complexity. u.s. researchers still need to gain irb approval for studies conducted outside the country, but this approval does not supersede any local-level approval necessary to do work. In our current research, this means that we did not need to seek any form of ethics approval at the universities included in China, but we needed to submit protocols and seek approval from Ethics Review Boards in Australia. As we plan to expand the study further we are certain to face varying levels of administrative review.

Building a Sample

In our study we are collecting data from current undergraduate and graduate students, researchers and faculty/academic staff. When schools agree to participate, we need to discuss how a sample of participants will be created. Our primary goal is to collect data from a variety of institutions so we do our best to be flexible and work with the preferences of each school. What this means is that we are willing to accept a range of sampling options. At one end of the spectrum, the ideal end, schools provide us with contact information for students and faculty in our selected disciplines and we manage the distribution of all contacts. This gives us the opportunity to personalize solicitations, which has been shown to improve participation, and to calculate accurate response rates. Some institutions are willing to provide us names to conduct a census of all eligible members; others prefer to select a random sample of individuals. Schools generally understand our desire to have access to this information but they also have concerns about sharing contact details. Thus, we also include a set of schools that do not provide us with contact information but instead they distribute a general survey link to students and staff on our behalf.

Spam Filters

When all approvals are granted and someone at the participating institution has agreed to provide contact information, it is possible to move on to soliciting participants. There is a rich literature on the best methods for soliciting survey participants, but due to budget constraints and pragmatism, email is the only logical method for our study. While it is possible to automate distribution of such a large volume of messages, coordinating the sending of initial solicitations and reminders is no trivial task. Our survey software company (Qualtrics) manages distribution of messages, which is quite helpful. These messages are handled on the company’s servers, with the chance to specify a Reply-to address. Because these servers send out enormous volumes of messages each day, the company has gone through efforts to have their servers ‘whitelisted’ by many organizations so that emails originating from them are not blocked by institutional servers or email programs. Qualtrics now provides reporting of the number of undeliverable messages that are “bounced,” which really helps users to make more robust estimates of response rate and sampling bias. The whitelisting is definitely helpful in the U.S., but it is of questionable value internationally.

In the u.s. we have sent more than 10,000 email solicitations over the last few years, with less than 1% of those rejected by spam filters. Our recent disbursement went smoothly to institutions in Australia, but we were not so lucky with distribution in China. For the initial distribution we sent emails to approximately 3,000 students and faculty. Within 12 hours we had 1,900 bounced messages in our email inbox! Over the next few weeks we worked to diagnose why messages were being rejected and attempted to remedy the situation. What we realized was that if we sent messages in small batches and with a low throttle (number of messages delivered per hour), we could get many of the messages through. However, if too many messages hit the servers at the intended institutions within a limited period of time, some form of alarm would be set off and any further messages would be rejected. Coupling small batches of emails with a low throttle rate and an individualized subject line for each message we were able to eventually get messages through to nearly all individuals.

Collecting Data

Once you’ve wended your way through the myriad obstacles mentioned previously, the final hurdle involves getting individuals to complete the survey you spent so much time developing (and maybe translating). When we first initiated discussions with colleagues in China and told them about our plans to conduct an online survey, we commonly were told that the norm in China was paper surveys and not online surveys. Many told us that it would be best if we printed the survey and then had faculty tell their students to complete the questionnaire. While this goes against the requirement of voluntary completion of surveys outlined in our IRB, we were assured that we would get a very high response rate if we adopted this practice. The realities of trying to distribute and collect large numbers of paper surveys in China was logistically and financially daunting. Despite these warnings, we forged ahead with the plan to collect surveys online.

Before we went forward with the survey distribution, a member of the project team (Adam) met with a former mentor who is a methodologist and Dean of a Faculty of Education in China. We discussed the plans for the project and, as if on cue, he commented that paper surveys are de rigueur for collecting data in China. After discussing the difficulties with our approach, we came up with a plan where if online data collection faltered we would change course and go with a short paper version of the survey that students and faculty could complete quickly in a classroom or office environment. This short version would include core survey items and a space where respondents could provide their email contact if they were open to completing more items. We piloted this version during the summer of 2013 and of 50 students who completed the survey, approximately 45% of respondents offered their contact information.

Improving Participation

In our most recent round of surveying in the u.s. we sought to evaluate the effect incentives have on improving participation. This is another area where there is a rich literature. 17, 18, 19 We decided to design an experiment where we tested different levels of incentive on participation for students and faculty. For schools where we had contact information, we randomly assigned individuals to an incentive group (e.g., No Incentive, $15, $30, $45, etc.). We included a single line in the email solicitation indicating that upon completion participants would be entered into a random drawing to win a gift card.

Evaluation of the results indicated that there were small positive gains in response based on incentive. We measured response in two ways: survey initiation and the proportion of survey items completed. The most notable difference in initiation across incentive amounts and respondent status (i.e., undergraduate, graduate, faculty) was during the initial solicitation, with small but noticeable improvements in initiation rate. This occurred for both undergraduates and graduate students, but not for faculty. When evaluating the proportion of items completed, there was a modest gain in completion between No Incentive and any incentive across all groups. Interestingly, there was a general increase across level, with undergraduates completing about 75% of the potential items, graduate students completing just over 80% and faculty averaging greater than 85%. For faculty, there was a small trend where higher incentives were associated with more items completed.

We decided to conduct a similar experiment in China, but based on the prior results we excluded the No Incentive group from the study and adjusted the incentives for relative value. We found no substantive difference in survey initiation (8%) or completion (6%) across groups (RMB 40, 80, 120 and 160). While our results indicate a minimal gain for including an incentive, the gain is likely not worth the effort.

While we planned to do a similar experiment in Australia to determine if results would hold, we discovered an extra hurdle that dissuaded us from doing so. In completing the paperwork for Ethics approval, we determined that any form of incentive would require us to obtain a permit from the Australian Capital Territory to administer the incentive draw for the survey. Given the lack of major gains in response tied to incentives, and the external permits necessary, we decided to forego running the experiment in Australia.

One other major issue came up with our consideration of improving participation. While in the u.s. it is assumed that any university students and faculty will have ready access to broadband internet connections, this is not the case in China. Many of our students and others we met in China alerted us to the fact that students usually pay for internet access on a monthly basis and many only purchase plans that limit their access to websites within China. While we thought of ways to address this situation, we did not arrive at a simple answer. Unfortunately, there’s no easy way for us to gather information from potential respondents about how this affected their ability to participate.

Translation and Adaption of stem Instruments

In addition to the steps to successful survey distribution above, translation plays a central role in multilingual survey projects and goes hand-in-hand with survey adaptation. 20 Below we provide detailed illustrations of the process of translating and adapting the English survey questionnaire into Chinese. We raise challenges that emerged from translation and adaptation and explain the measures we took to ensure the accuracy and comparability of the Chinese version of the survey. Such measures demanded of our research team strong commitment to self-reflexivity, that is attending to how and what we learn when we compare and think across cultures.

Envisioning Cross-Cultural Survey Research as a Scholarship of Mutual Engagement 21

Team Translation Approach

Informed by the literature on cross-cultural survey translation, 22, 23 we adopted a team translation approach that is particularly useful in addressing the challenges of cross-cultural translation. Our team initially consisted of six bilingual Chinese doctoral and master’s students with one student specializing in Science Education and the rest specializing in Comparative and International Education. All team members were female. At a later stage of survey refinement, when the team worked on revisions to address issues emerging from the pilot survey conducted at three Chinese universities, a Chinese male doctoral student majoring in Science Education joined the team. Including one student from Taiwan, all students were from different provinces of China. Before studying at Indiana University, team members majored in diverse undergraduate and master’s programs, including Education, English Literature and Nursing. One team member obtained her master’s degree in Chinese Language and Literature from Singapore. Team members also had various levels of experience in survey design, translation and implementation. The intention in creating a team of such diversity was that its mix of skills, experience and disciplinary expertise could produce an optimal survey version. Dr. Adam Maltese, the survey’s creator, oversaw the team’s work and served as a consultant.

The procedure of team translation was time-consuming, complex, and iterative. Two translators worked independently to produce two versions of the Chinese translation. Two reviewers were appointed based on their previous experience with survey research. By comparing the two versions, a reviewer marked items on which the two translators largely disagreed and the reviewer herself disagreed with both translators. The marked items were discussed by the whole team in order to produce the most accurate translation possible. Based on group discussions, some survey items were revised, deleted, or added to reflect Chinese social and educational contexts. We also purposely kept some items for multi-national comparisons and to capture the phenomenon of increasing student mobility, experience, and understandings of teaching and learning around the globe. 24

Challenges of and Issues in Translation and Adaption

Language-Driven Adaption

The first challenge we faced in survey translation was whether we should translate the acronym stem into Li Gong Ke (Science and Engineering Track, 理工科). Li Gong Ke is commonly used in China to indicate the disciplines in Math, Science and Engineering. The term can also be extended to include disciplines in Medicine and Agriculture in which students are usually required to learn advanced math and science knowledge. The team decided to keep stem in the Chinese version of the survey for two reasons. First, we wanted to emphasize the four areas stem represents: Science, Technology, Engineering and Mathematics. Second, with increasing communication and cooperation between Chinese and American higher education institutions, the acronym is understood by a number of Chinese undergraduate students and faculty. At a team meeting, for example, one member reported that through her personal experience with faculty members at one of the pilot universities, a top tier research university in China, she found that faculty knew what “stem” meant. However, other team members wondered whether the acronym might confuse faculty and students at second and third tier Chinese institutions. The team addressed this concern by taking the following measures. While the acronym stem was kept in the Chinese version of the survey, an explanation of the acronym was given at the beginning of the survey. In addition, we spelled out stem through questions to help respondents quickly understand its meaning. For instance, we included, “Did you enter college with an intention to (or did you previously) select a major in science, technology, engineering or mathematics (stem)?”

Socio-Cultural Adaption

Socio-cultural adaptation of cross-cultural surveys is generally more difficult than translation involving linguistic expression. Although globalization has facilitated the interaction among nations, each society still has unique political, social, cultural and educational systems. Accordingly, cultural practices in one society might not have an equivalent in another society. The following examples illustrate the challenges the team faced when adapting English survey questions to fit the Chinese cultural context, as well as the approaches the team adopted to solve those challenges.

First, clear differences between the American and Chinese higher educational systems are reflected in professional ranking systems, flexibility in choice of majors by students, and teaching styles. The professional ranking system in u.s. higher educational institutions is differentiated into tenure and non-tenure tracks, and the major differences between a lecturer and an assistant professor are that the former is not eligible for tenure and usually has no research obligations. However, the tenure system does not exist in most higher education institutions in mainland China, where professional ranks have been differentiated into four levels in an ascending order: teaching assistant, lecturer, associate professor and professor. Such systematic differences posed challenges for the team when translating the English question “What is your primary occupation?” where both “lecturer” and “assistant professor” were offered as response options. At the team meeting, some members suggested removing “assistant professor” from the option section. However, the title of assistant professor is familiar to Chinese faculty; particularly those who obtained doctoral degrees from u.s. higher education institutions consistently refer to themselves as “assistant professor” in Chinese (zhu li jiaoshou 助理教授). Also, the team concluded that it would be interesting to find out how many participating faculty members would choose the option of “assistant professor.” 25 After long discussion, the team decided to keep both options in the Chinese survey.

The team also revised some questions to reflect the influence of the Chinese higher education system on students’ choices and teaching styles. For example, Chinese high school seniors are required to select majors when they fill out their college application forms. Switching majors is also relatively difficult at Chinese universities and colleges, although a few universities have begun to implement policies that allow a small number of students to change their major field of study. For most students who intend to change their majors, the best time for doing so is when they apply for graduate school. Therefore, the question in the English language survey, “As an undergraduate how aware are/were you of the job market for employment related to the discipline of your intended major?” might cause confusion among Chinese students, some of whom might think they were being asked whether they understand the job market of their desired majors. After discussion, the team changed “the intended major” to “current major” and translated it accordingly. Similarly, we deleted the word “clickers” in a question that asks whether teachers use quizzes or clickers to provide students with immediate feedback in class, since clicker devices have not been commonly used in lecture halls in Chinese universities and colleges.

Second, we adapted and added some questions to reflect the social structure of Chinese society. When translating demographic questions, we deleted a question that asks respondents’ race, because race has not been a salient issue in Chinese demographic categorization systems and general public perceptions of difference. At the same time, we added a question asking respondents which provinces they are from since there are salient differences in economic and educational development among Chinese provinces and regions.

Meanwhile, cross-cultural translation requires translators to have a deep knowledge of cultural practices in both societies. Although all translators that made up our team have studied in the u.s., they are not necessarily familiar with children’s and school cultures in the u.s. Therefore, Dr. Maltese’s u.s. cultural knowledge was critical to the team’s ability to understand and accurately translate the meaning of cultural practice in the u.s. into questions that would make sense to Chinese respondents. For example, among the options provided for the question: “Who was most responsible for sparking your initial interest in stem?” is “famous scientist or science personality”. Team members were puzzled by the term “science personality”. To explain the term, Dr. Maltese used the example of Mr. Bill Nye, who was host of “Bill Nye the Science Guy,” a popular science television programs for children aired in the 1990s. This explanation facilitated the team’s translation. Although there is no similar program in China, the team came up with the translation “science figures in television or other mass media” (电视等大众文化中的科学人物) to best capture the meaning of “science personality”.

Keeping Items for Multinational Comparison

Because amistem currently includes participants from universities and colleges in the U.S., China and Australia, we purposely kept in the survey some items for multinational comparison even though they seemed somewhat unsuitable for the Chinese context. Furthermore, we wanted to capture trends related to increasing numbers of students receiving their educations in multiple countries. For example, one survey question asks respondents whether they are completing a Pre-Medical School program. Unlike medical schools in the u.s. and Australia, medical schools in China offer bachelor’s degrees. However, it is possible that some American students who are currently studying in Chinese universities have had such an experience. Also, Chinese students who study at Indiana University Bloomington campus participated in the pilot survey, and some of whom might be enrolled in the Pre-Medical School program. Similarly, for comparative purposes we categorize pre-collegiate schooling into three periods: grade 1-4, 5-8 and 9-12.

Handling Sensitive Issues

A few questions in the last part of the survey, which is an optional section, ask students about their sexual orientations and the gender of their parents. Because LGBT issues have not been openly discussed at most Chinese universities and colleges, some team members questioned whether the questions should be included in the Chinese version. The team decided to keep the questions since the survey would be conducted primarily online, which gives students privacy to answer such questions. Preliminary analysis of the data collected from the pilot survey conducted in three Chinese universities showed that among 104 respondents who answered the sexual orientation question, 13 students indicated that they did not consider themselves “straight (不是男女同性恋).” 26 In addition, when translating the question: “Were both of your parents/guardians the same gender?” the team encountered a similar challenge. “Parents” in Chinese, “fu mu” (父母), already implies two genders: “father and mother”; a literal translation, that is, whether “father and mother” have the same gender, is obviously awkward. However, no other colloquial word or phrase in Chinese replaces “fu mu” in a gender neutral way.

Revisions after the Pilot Survey: The Phenomenon of Moderatism

Survey translation, adaptation and modification is perhaps above all an iterative process, 27 as illustrated by one of the more intriguing discussions among the survey team about what we now call the phenomenon of “moderatism.” After the pilot survey, a new doctoral student joined the survey team and enriched the team’s discussion and understanding of the art and science of translation with his skills and expertise. Based on preliminary analyses of the results of the pilot survey, he raised concerns about the appropriateness of the translations for two statements: “You see yourself as a math person” and “You see yourself as a science person.” He pointed out that the current translations, “您认为您是数学达人” and “您认为您是科学达人,” might confuse respondents, leading them to understand that they were being asked whether they were math or science geniuses. Chinese cultural norms privileging modesty may prompt respondents to give negative answers to the two statements, even if they indeed are math or science persons. The significant difference in the proportion of respondents who chose “no” (strongly disagree and disagree) to the two statements between those in China and in the u.s. lent support to his proposition that Chinese respondents might value moderation when assessing their possible responses on a particular item. While around 62% of Chinese respondents chose “no” to the statement of “you see yourself as a math person,” only 34% of American respondents did so. For the statement of “you see yourself as a science person,” 47% of Chinese respondents chose “no,” only 28% of American respondents did so. Based on this discussion, the team revised the two statements into “you are good at math/science,” (您擅长数学/科学) and await findings that might further help us understand whether moderatism should be an important consideration in understanding how Chinese learners respond to questions about achievement and ability.

The translation process illustrated above demonstrates that a cross-cultural and diverse team approach to survey translation and adaptation is an effective way to avoid bias and achieve accuracy. Also, bringing new members into the team after a pilot survey will facilitate the revision process, because translators close to the project may not be able to identify subtle translation errors after working on the survey for such a long time. When reflecting on the translation process, Dr. Maltese stressed that a high trust relationship built between translators and the survey designer is critical to ensure genuine communication between the two sides, especially in the case where the survey designer has no direct or deep knowledge of the target language and culture.

In the end, we did not anticipate the extent to which members of the translation team would learn about each other’s educational systems and what is taken for granted when we speak of teacher, learner, and classroom. The extent to which the translation team would provide participants with a significant experience in becoming globally-aware and culturally-sensitive scholars was one of the happy unintended consequences of this study.

Conclusion

Our primary purpose in this paper has been to introduce and share our experiences in preparing for and implementing the amistem survey project, with the goal of extending our comparative research to other interested institutions. Through sharing our experiences in “internationalizing” amistem, we invite higher education researchers to engage in discussions regarding the roles of culture and context in international survey research, especially in the areas of learning, teaching, and engagement, for our ever-more-global community of higher education scholars. Cross-national and cross-cultural educational research should draw attention to various knowledge systems, interaction patterns, and communication styles, including those with whom researchers work and the purposes to which their research is applied. At our most optimistic we believe that collaborative and transnational projects like amistem can promote “education that fosters transcultural communicability and encourages constructive, creative action.” 28

As implied in part two and three of this paper, it is perhaps ironic that multi-national and cross-cultural survey research is becoming more difficult and time consuming, not to mention costly, to conduct in the current technological environment in which it can be so easy to get surveys in front of potential respondents. For example, each of the project phases discussed above, from translation to getting noticed to sampling, adds total time to the process of getting access and collecting data. There are many points along the research chain where a single person determines whether or not the effort can proceed. Nevertheless, it is still possible to conduct such work. In fact, it is essential for work like this to proceed to better inform institutions and policymakers about the myriad experiences of diverse students and faculty and to lay a foundation for greater collaboration on transnational and comparative survey research.