Abstract
This article reports results from research on cultural models, and assesses the effects of computers on data quality by comparing open-ended questions asked in two formats—face-to-face interviewing (FTFI) and computer-assisted, self-interviewing (CASI). We expected that for our non-sensitive topic, FTFI would generate fuller and richer accounts because the interviewer could facilitate the interview process. Although the interviewer indeed facilitated these interviews, which resulted in more words in less time, the number of underlying themes found within the texts for each interview mode was the same, thus resulting in the same models of national culture and innovation being built for each mode. Our results, although based on an imperfect research design, suggest that CASI can be beneficial when using open-ended questions because CASI is easy to administer, capable of reaching more efficiently a large sample, and able to avoid the need to transcribe the recorded responses.
Keywords
Introduction
The advent of the computer has had a significant effect on social science research. All stages of the research process, including data gathering, analysis, interpretation, and results reporting are influenced to a greater or lesser degree by the use of a computer. Even though in many of these stages it is the researcher who uses the computer, the computer can also be used by the research subjects. In computer-based interviews, for example, it is possible for both interviewer questions and respondent answers to be recorded on the computer. The research presented in this article is an offshoot of a larger research program comparing New Zealand innovation culture with that of selected European nations. Our original intent was to conduct only face-to-face interviews (FTFI) for use in cultural modelling, but the research evolved as we considered ways to reduce the costs associated with conducting face-to-face interviews in multiple European nations. Thus, we decided to explore the use of computer-assisted, self-interviewing (CASI) because it would allow for more efficient collecting of our European data. In this article we explore the use of computers in research and consider the effects on data quality of computer-assisted, self-interviewing (CASI) compared to face-to-face interviewing (FTFI) when using open-ended questions of a nonsensitive nature.
For open-ended questions of the nature presented in this paper, FTFI has long been the standard because there is scope for important interactions between the subject and interviewer. If necessary, the interviewer can help guide respondents through the interview process. Furthermore, interviewers can ask questions to clarify unclear responses as well as ask follow-up questions to delve deeper, eliciting fuller accounts of respondent beliefs, attitudes, and feelings. The use of CASI does not allow for clarifying or follow-up questions. What a respondent chooses to write is what a researcher obtains for analysis. However, CASI is not without its benefits. By using CASI, a researcher can reach more efficiently a larger sample of respondents. Instead of face-to-face interviews conducted over weeks or months, a researcher can gather respondents in a computer lab on a single evening. By using CASI, a researcher also can avoid the cost and time of transcribing recorded responses.
This article will provide a theoretical background for the research, which will be followed by a description of the research design and presentation of the findings. We expected that for our nonsensitive topic, FTFI would generate a richer source of data because the interviewer could facilitate the interview process. Results showed that although FTFI produced more words in less time, the number of underlying themes found within the texts for each interview mode was the same, which resulted in the same models of national culture and innovation being built for each mode.
Theoretical Background
When comparing FTFI and CASI there are important theoretical and methodological issues to consider, such as the mediation of computers with respect to communication and the inherent differences between oral and written modes of discourse and their effects on data analysis. To date, much of the research comparing data quality between CASI and FTFI has been for closed-ended questions of a sensitive nature. Compared to FTFI, CASI has been more successful for gathering data about sexual behaviour (Ghanen, Hutton, Zenilman, Zimba, & Erbelding, 2005; Smith et al., 2009; Tideman et al., 2007) and drug use (Perlis, Des Jarlais, Friedman, Arasteh, & Turner, 2004). This evidence suggests that CASI respondents are more willing than their FTFI counterparts to report illicit, socially undesirable, or potentially embarrassing behaviours. In other words, CASI can help mitigate response effects, that is, where respondents systematically refuse to answer certain questions, under-report socially undesirable information, over-report socially desirable information, give moderate responses, or agree with the interviewer. Kiesler and Sproull (1986) reasoned that computer-based surveys are unique because they contain less social context information – the basis for many response effects. In research settings that are both impersonal and anonymous, we can expect respondents to become more self-centred and relatively unconcerned with social norms and with the impression they give others in such settings. If these processes occur with topics of a non-sensitive nature then it is possible for CASI to generate fulsome responses. However, with topics that are non-sensitive in nature there may be fewer response effects and the use of CASI and FTFI may lead to similar results, at least for closed-ended questions.
For research on topics of a less sensitive nature, the mode of data collection has an important influence on the quality of data. As such, researchers have been justifiably concerned with data collection and mode effects on data quality (De Leeuw, 1992). There are two main classes of mode effects (De Leeuw, 1992) of relevance for comparing FTFI and CASI. These are differences in (1) media-related factors and (2) information transmission. Regarding media-related factors, FTFI and CASI differ on factors related to the social conventions inherent in their means of communication. Face-to-face communication for the gathering of information is a relatively routine occurrence for people (e.g., talking with doctors, bosses, employees, etc.). Although computers are used increasingly for information gathering, it is arguable that their use is less routine for the average person than a face-to-face encounter, particularly for older generations. Therefore, familiarity with FTFI means that it should produce data uninfluenced by respondent concerns about the mode. The second media-related factor is the locus of control. The locus of control concerns who is in control in the interview. In a face-to-face interview the locus of control is shared between the interviewer and interviewee. The flow of communication is determined by both parties. In a computer-assisted, self-interview the locus of control is firmly in the hands of the interviewee, who is in charge of pace and can easily skip questions. CASI, therefore, can be expected to lead to poor data quality if questions are ignored. The third media-related factor of relevance is “the ability of the medium to convey sincerity of purpose. The personal contact in a face-to-face situation gives an interviewer far more opportunities to convince a respondent of the legitimacy of the study in question” (De Leeuw, 1992, p. 15). In the case of CASI, there is less opportunity to communicate trust and legitimacy. All these factors have a bearing on the quality of data and suggest that data quality with CASI may suffer in a number of ways.
FTFI and CASI differ markedly with respect to information transmission. Verbal, nonverbal, and paralinguistic communication are possible with FTFI but not with CASI, where all communication is via the printed word. Oral and written communication involve different forms of thought and expression (Kvale, 1996, p. 166); thus, one would expect to find differences in interview responses. Furthermore, because of the many ways to convey information via FTFI, issues can arise when researchers attempt to analyze the transcripts of oral interviews. Transcripts lack contextual clues (nonverbal and paralinguistic communication markers) that help make sense of the interview. Transcripts of oral communications are decontexutalized because a “living” conversation has been frozen in written language. Kvale (1996) contends that “if one accepts as a main premise of interpretation that meaning depends on context, then transcripts in isolation make an impoverished basis for interpretation” (p. 167). Accordingly, CASI transcripts may be hard to interpret. Another way in which the transmission of information differs between the two modes is how stimuli are presented. In the case of CASI, the stimuli (e.g., the questions) are presented visually. In the FTFI encounter, the primary presentation of stimuli is auditory, although visual stimuli may also be used. It may be that the visual presentation of questions in CASI offers an advantage because the questions are ever present and can be easily referred back to as the respondents make their responses. Another distinction is the temporal order of stimuli presentation. In the case of FTFI, the interviewer determines the question order and respondents generally cannot go back and forth between questions. With CASI, depending on the program used, respondents may be able to go back and forth between questions and amend their answers. This process may facilitate or encourage respondents to answer the questions.
Researchers are familiar with FTFI and expect that skilled interviewing will yield rich data. Researchers are less familiar with CASI and are likely to have misgivings about the richness of the data generated. The literature shows that human expression varies according to the mode of communication used in the research. In essence, there are quite different social processes occurring during the conduct of FTFI and CASI. With CASI and the use of written communication we can expect that this mode influences what respondents say and how they say it. Accordingly, the nature of the data obtained is likely to be different. But does this difference in data quality mean that the data analysis necessarily produces different results? We expected that FTFI would yield richer data and that the cultural models derived from FTFI would differ from those derived from CASI.
Method and Design
As previously stated, the goal of our larger research programme was to devise cultural models of New Zealand innovation. Cultural models are those presupposed, taken-for-granted models of knowledge and thought that are used in the course of everyday life to guide a person's understanding of the world and his or her behaviour (D'Andrade, 1984). They are also the constructed representations made by researchers in order to describe shared knowledge and perceptions used by groups of people in their daily lives (Blount, 2002; Cooley, 2003). Cultural models systematically draw on personal discourse—the representations, practices, and performances through which meanings are produced, connected into networks, and legitimised (Gregory, 2000). Discourse analysis allows researchers to get the insider's perspective on respondent knowledge, thought, and word meaning.
By first analyzing and then organizing key themes identified via discourse analysis, cultural models of the world can be built (Blount, 2002; Strauss & Quinn, 1997). According to Blount (2002):
Once a text is created from discourse, one works ‘backwards,’ asking questions about how the text was created, in effect asking what the conceptualizations are upon which the text is based. The conceptualizations are the raw materials of the analysis. They reflect the agent's underlying mental models, the framework with which the world is engaged. The reconstructed mental models of an individual constitute the cognitive architecture upon which the discourse is generated. (p. 9)
The tasks of discourse analysis and subsequent cultural modelling are to identify the key components of thought and to serialize, embed, and hierarchically organize them into a coherent model. In this sense, the approach uses abductive or retroductive logic (Blaikie, 1993) in which research begins “by describing these activities and meanings and then deriving from them categories and concepts that can form the basis of understanding or an explanation of the problem at hand” (Blaikie, 1993, p. 163).
To obtain the data necessary to formulate our models of New Zealand innovation culture we obtained respondent discourse via both computer interviews and face-to-face interviews. Respondents were recruited by contacting local high schools. A payment of $450NZD was provided to schools in exchange for soliciting adult volunteers and providing a venue. We selected two schools within Christchurch, New Zealand, one from the lower income bracket (New Zealand Decile 1–3) and one from the mid-tier income bracket (New Zealand Decile 4–7). Contact with prospective schools was made by telephone and school representatives involved with fundraising were sought. The school representative was directed to source prospective participants, an even number of men and women if possible, from people involved with the school or who lived in the local area. The local area was defined as the suburb in which the school was located, and even though our protocol was open to the solicitation of non-parents, all those interviewed were parents or grandparents of students attending the school. School representatives obtained parent participation by contacting school groups already involved in fundraising endeavours for activities such as sports and study abroad trips.
Our sampling goal was to obtain 20 participants from each of the two high schools—10 for each interview mode at each school, thus providing a total of 20 for each interview mode. Consensus analysis, unlike more conventional statistical methods, allows for very small sample sizes to reach statistical significance. In addition, our initial sample size goals were tentative estimations of a sample size thought necessary to achieve informational redundancy. As the topic of interest was generalized culture and innovation culture we expected that there would be wide agreement within a given society on these topics. Had information redundancy not been achieved with this sample size, further sampling would have been conducted.
We were not able to randomly assign all participants to either of the two modes because of the difficulty in finding participants who were available on the day arranged for CASI. Participants with open schedules were assigned randomly to a computer session or a face-to-face interview session. The non-random assignment of participants to a research mode applied to approximately half of the participants in each case. At each school, CASI was held on one evening, and those participants only available on the night in which the computer session was held completed the CASI. Those participants unable to participate in the computer session because of scheduling conflicts completed FTFI at the best available alternative time. Based on the availability of participants, FTFI was scheduled during afternoons and evenings over a two-week period. A two week period was needed because participant schedules proved to be busy and only one interviewer was available to conduct the interviews.
The research design is limited in that there was an incomplete random assignment of participants to the two modes. However, it seems likely that participants who were not available for the CASI evening were otherwise similar to those that were available, likewise for participants not available for FTFI. In support of this claim, Table 1 shows the numbers of men and women, their ages, and their reported incomes under each treatment for each of the two schools. The data shows that most participants were women, most were in their forties, and incomes were spread across the broad income bands, although four participants who participated in a face-to-face interview did not declare their income.
Socioeconomic Characteristics of the Sample
The qualitative interview portion of our research took on average one and a half hours and was scheduled in advance at a designated time and place (on-site at the schools) and outside of normal daily activities. The researcher began each interview by clarifying its purpose and explaining that participants would be asked questions about New Zealand culture and national identity. Participants were assured that there were no right or wrong answers to any of the questions, and we asked that they speak freely about their beliefs and opinions. The face-to-face interviews were recorded and later transcribed while the computer-assisted interviews were saved as Word files. Because the research consisted of only one interview with each participant and participants were asked to discuss an area in which they could be considered knowledgeable (e.g., their perceptions of NZ culture and national identity), we were advised by the university human subjects board that the research was exempt from the need to seek formal human subjects' approval according to New Zealand regulations.
In order to analyze the discourse obtained during the process (either CASI or FTFI), each interview text, managed as a Word file, was imported into NVivo 7 and coded according to key words and phrases. These data were then inductively analyzed for patterns, structure, and linkages of themes. The resulting cultural models demonstrate how participants perceived New Zealand culture in general, and New Zealand innovation culture in particular.
The differences between the FTFI results and CASI results were compared by four means: (1) Evaluating differences in the quantitative characteristics of the interview data, such as word count, time, and number of non-responses, using multivariate analysis of variance (MANOVA). (2) Quantitatively comparing the number of items listed (a measure of answer completion and recall ability) for questions in which respondents were asked to list five or more examples of an item (e.g., cultural symbols, important historical events, and important figures in science and technology, etc.) using MANOVA. The questions asking for lists were divided into three groups based on three themes: cultural elements, national identity elements, and innovation elements. (3) Comparing the prevalence of discourse themes between the two modes. (4) Qualitatively evaluating answer quality.
Results
Three multivariate analyses of variance (MANOVAs) were conducted and the initial analysis compared word counts, interview length, and missing data. 1 The result of the first analysis indicated that the two modes differed reliably on mean word count and interview length, but not on missing data (Hotelling's T2 = 184.30, F(3, 39) = 58.44, p < 0.05). Because of these differences, we considered it prudent to use word count and interview length as covariates in the subsequent analysis that followed. For the second MANOVA, the administration mode (FTFI/CASI) was used as the independent variable and the mean number of items listed within the cultural, national identity, and innovation element groups served as the dependent variables. The result of this analysis indicated that the three dependent variables showed no significant differences between the types of data collection (Hotelling's T2 = 1.27, F(3, 39) = 0.62, n.s.).
Characteristics of FTFI and CASI
significantly different at p <0.001. Unless otherwise specified, all tests adopted a probability level of p < 0.05.
The main themes present in participant interviews did not vary significantly between modes and resulted in the same cultural models of innovation culture being built. To provide quantitative evidence of the similarities, Table 3 shows the themes which were significant (mentioned more than in passing) in more than five interviews across either mode. The number of FTFI and CASI interviews in which the theme has a significant presence is documented and the table shows very little variation in theme prevalence across modes. For example, themes prevalent in 20 or more CASI responses were also prevalent in 20 or more FTFI responses. These themes included landscape, clean and green, lifestyle, multicultural, smallness, sport, and place in world.
Number of CASI and FTFI Interviews with Significant Mention of Key Theme
Note. For sake of brevity, this table contains only those themes mentioned by five or more respondents.
Although similar in thematic content to FTFI responses, it should be noted that CASI responses were often more direct and tightly-focused. This is not unexpected as oral and written modes of communication are inherently different and one would expect a written response to lack the repetitions, incomplete sentences, and digressions that are often found in oral communication (Kvale, 1996). In response to the question “How do you think New Zealand is identified internationally?” the following face-to-face response was typical:
Some countries, I guess it depends on where you come from. I have a lot of international students and they regard NZ as very expensive and that for many things like technology it is extremely expensive in NZ compared to their countries. Common commodities like power are extremely expensive compared to other countries. When I went to America and went into a store, he thought I was from England and he said where you from and I said NZ and he said what language do they speak. When I said we speak English and we are below Australia, he said, ah, you wear grass skirts. I said no we don't. He was an Asian man. That was his perception of NZ. When I lived in Dubai, thirteen years ago, their perception of NZ was that it wasn't like part of the world. It had seven wonders. It had extremely beautiful scenery, lovely weather, beautiful culture and lovely people.
In contrast, the typical CASI response was much shorter and to the point:
A great place to live. A clean green environment with 50 million sheep and 4 million people. A country with wonderful scenery and hospitable people. A small country down in the Pacific just below Australia (Hmm…is it a state of Australia?)
Although differing in length and directness, much of the substance of these responses is similar. The codes used when analysing this FTFI response included: NZ as expensive, NZ as backwater, Landscape as significant, the friendly kiwi, and Culturally unique. The codes used when analysing the CASI response included: Landscape as significant, The friendly kiwi, NZ as small, Agricultural NZ, Clean and green, and NZ as backwater. Three of the codes were shared between the FTFI and CASI responses. Both sets of responses highlight the view of New Zealand as a far away, beautiful nation with friendly people and the belief that the international community knows little about New Zealand.
Responses to another question further illustrate the direct and tightly focused CASI response. In response to the question, “What are aspects of New Zealand history that you feel played an important part in the formation of NZ national identity?” the following face-to-face response was typical:
I think you've still got to go back to the Maori wars because it played a part. With the treaty it actually formed NZ. That would be the big one for identity. I think not just the Maori wars but the first and second world war when troops sent across. People realized that there were NZ as well as Australian troops. A lot thought we were all one country but they then realized it was two separate countries and they were called ANZACS. NZ played a big part in both wars. Because of what they did they were New Zealanders. Because of what they did, people said oh he did that, oh he's a New Zealander. It was recognition for NZ. This country was the first to give women the vote and that was quite a big one. With women being able to get the vote you had a women's point of view.
In comparison, the typical CASI response was more direct while sharing similar content themes. For example, one CASI response listed the following items:
Whaling Ships- bringing European settlers to the country
Treaty of Waitangi- Signed treaty of some Maori tribes and representatives from England. This treaty still plays a major role in the day-to-day running of this country
Kate Sheppard- empowering women from an early perspective in historical terms, this is evident even now as women are a strong hold in most families with positive and negative consequences. Women in New Zealand being the first in the world to get the vote.
World Wars- changed gender roles, men got to see more of the world as we are so isolated. Made us part of the international community. Got us recognized. Linked us with Australia- the ANZACS.
Immigration- changing the cultural make up of any populated area, once again with positive and negative consequences. Racial tensions.
The FTFI response produced six codes: Significance of Maori wars, Multicultural, Significance of World Wars, Relationship to Australia, Place in world, and First in world. A number of these codes were shared with the CASI response, which produced eight codes: Early settlement, Multicultural, First in world, Significance of World Wars, Relationship to Australia, Place in world, Immigration, and Racial tensions
Discussion and Conclusion
Our research programme began with FTFI in New Zealand but evolved to use CASI in selected European countries. This development led to research questions about the suitability of CASI compared to FTFI. Would CASI provide data of sufficient depth and quality to enable us to generate cultural models, and, if it did, how would the models compare to those generated from FTFI data? With our New Zealand data we were able to explore these questions. The literature shows that human expression is mediated by the mode of communication used in research. The outcomes of using FTFI and CASI are uncertain because each mode has advantages and disadvantages. FTFI is more likely to yield rich data since the interviewer mediates the process. On the other hand, we also expected that in the written mode of CASI, respondents would be more in control of the interview process and, thus, have an opportunity to express themselves well, even in the absence of an interviewer. Taking into account the above considerations, we expected that FTFI would be more suitable for our goal of developing cultural models of innovation culture.
Our results showed that the FTFI transcripts had more than double the number of words than transcripts obtained using CASI. In addition, the time required to record the face-to-face-interview was shorter so the interviews yielded more words per unit time. Although these results might suggest richer material was obtained via FTFI, the discourse analysis did not find richer material. The same themes dominated in both modes of interview and identical cultural models were built from the data.
CASI, although lacking the support of an interviewer, was for our purposes just as effective as FTFI in providing data on which to build cultural models of innovation. As the literature indicates, CASI respondents have relatively more control of the interviewing process compared to FTFI. It appears that this control meant that they were able to provide responses that, while succinct, were sufficiently rich enough to provide an excellent base on which to develop cultural models of innovation. Furthermore, CASI respondents were able to present their thoughts in a focussed manner in part because they were able to keep track of the questions. FTFI respondents often asked to have the question repeated.
Overall, we concluded that CASI methods appear not to have a significant effect on the substance of the results even though the nature of the data varied depending on the mode of the interview. It may be that our results reflect the rather modest ambitions of the questions, which were not particularly demanding or exhaustive explorations of complex topics. When exhaustive detail on specific issues is needed, then FTFI may be advantageous. However, the results indicated that CASI may not suffer from serious drawbacks when used with open-ended questions where modest levels of detail are needed. This finding indicates that, subject to further research to corroborate our findings, CASI can be used with open-ended questions where the research objective is to seek qualitative understanding or clarification of a topic. Given that CASI is quick to administer, is capable of reaching efficiently a large sample, and avoids the need to transcribe recorded responses, it offers several potential advantages for research using open-ended questions.
Footnotes
1
Although the sample size is small for a multivariate analysis, this approach was adopted in order to avoid the use of the corrections that would be necessary for controlling the familywise α-level inflation inherent in multiple univariate analyses. Furthermore, the use of the T2 technique limited the division of an already small sample into just two groups, with multiple dependent measures. Because there were no missing data, there were multiple dependent measures, and each level of the independent variable had at least 20 observations, it was felt that this would provide sufficient power for the analysis.