When “Normal” Becomes Normative: A Case Study of Researchers’ Quotation Errors When Referring to a Focus Group Sample Size Study

Abstract

In 2011, we published a review exploring how researchers report and justify their focus group sample sizes. We concluded that sample sizes vary widely and that most researchers give no explanation for their sample size. The aim of our 2011 study was to describe practice rather than develop guidance. However, after our study was published, we noticed that new researchers were using our information about typical sample sizes as justification for their own sample size. In other words, practice that we had presented as typical or “normal” but generally lacking in justification was being used as normative. The current study aims to explore the misrepresentation of descriptive information as normative. Specifically, we map this type of quotation error in references to our 2011 study. Using Google Scholar, we identified all articles referencing our study. We then extracted quotations where the researchers had referred to our study and categorized these as follows: (a) quotations where the researchers had used the descriptive information from our study to justify their sample size and (b) quotations where the researchers had referred to our study for other purposes or where the purpose was unclear. We assessed 205 articles that had referred to our 2011 study. We identified the type of quotation error we were interested in, namely the misrepresentation of descriptive information as normative, in 50.7% of the included articles. Our study shows very high rates of one type of quotation error: the misrepresentation of descriptive information about focus group sample size as normative. Researchers referring to other researchers’ work carry most of the responsibility for ensuring that they do this appropriately. However, the authors of the research being referred to also need to consider how they can make their results clearer. We offer suggestions as to how this might be achieved.

Background

In 2011, we evaluated a set of research studies to explore how researchers reported and justified their focus group sample sizes (Carlsen & Glenton, 2011). An earlier review had reported wide variation in focus group practice and inadequate reporting (Twohig & Putnam, 2002), and we were interested in discovering whether the situation had improved since then. To map current practice, we extracted data on the sample sizes used in 220 studies and on any explanation authors gave for this number. We concluded that sample sizes still varied widely and that most authors gave no explanation for their sample size.

The aim of our study was to describe practice rather than develop guidance. We therefore presented the results of our study in descriptive rather than normative terms (i.e., we described what we had discovered to be typical or “normal” practice and described if and how this practice had been justified by the study authors but did not present recommendations about sample size). While our study conclusions included recommendations, this was in terms of encouraging improvements in reporting. In addition, we called for more evidence-based guidance regarding sample size. In no part of our study, did we recommend specific sample sizes.

Soon after our study was published, authors of new focus group studies began to refer to it. However, we quickly noticed several instances where our results were being misrepresented. One particular type of misrepresentation stood out: Authors were using our information about typical sample sizes as their justification for their choice of sample size. In other words, practice that we had presented as “normal” but generally lacking in justification, was now being used as “normative.”

Inaccurate referencing of studies can have a number of implications. It can lead to displeasure among the original authors (De Lacey, Record, & Wade, 1985) and can undermine trust among readers (De Lacey et al., 1985; Jergas & Baethge, 2015). But perhaps the most serious consequence of inaccurate referencing is “the difficulty in correcting a major inaccuracy that may well become ‘accepted fact’” (De Lacey et al., 1985). As the number of articles referring to our study increased and the same misrepresentation continued to occur, we became concerned that our study was actually worsening people’s focus group reporting. Ironically, while our study had highlighted many researchers’ failure to justify their sample size decisions, it appeared that the same study had now become a justification in itself. By simply counting researchers’ sample size practice, we were potentially entrenching this practice. We therefore decided to explore how widespread this type of referencing error was and to consider whether we could have done anything differently to avoid this situation.

Referencing Errors

When discussing and evaluating referencing errors, De Lacey distinguishes between citation errors (errors that could prevent immediate identification of the source of reference, for instance, because of misspelling of author names) and quotation errors (errors that deal with the accuracy of statements made in regard to another author’s work; De Lacey et al., 1985). The prevalence of quotation errors in medical journals has been mapped in a number of studies and further synthesized in two systematic reviews. These show quotation error rates of 20% (Wager & Middleton, 2008) and 25.4% (Jergas & Baethge, 2015), respectively.

Quotation errors can further be categorized according to the seriousness of their implications. De Lacey (De Lacey et al., 1985) distinguishes between trivial errors (quotations in which errors of transcription did not alter or obscure the meaning of the quoted source), errors that are slightly misleading (quotations that misled or could mislead, but the errors were not sufficiently serious to destroy or fundamentally to alter the meaning of the source), and serious errors (quotations that seriously misrepresent or bear no resemblance to the original source).

Some authors also distinguish between misquotation of narrative statements and misquotation of numerical data (Awrey et al., 2011; Goldberg et al., 1993). One reason for this distinction is that quotation errors in references to numerical results may be easier to determine. However, quotations of both words and numbers can be accurate in the sense that they exactly duplicate the words or numbers of the original study but can still be grossly misleading if taken out of context. The selective reporting of outcomes from clinical trials by systematic review authors is one such example. Assessing the extent to which quotation errors have occurred therefore requires a certain level of judgment, regardless of whether the finding is narrative or numerical.

The focus of this study is the misrepresentation of descriptive information as normative. We have defined this as a quotation error. We would also argue that this error is a serious one as it seriously misrepresents the original source. The findings that are now being misrepresented are numerical in the sense that researchers are misrepresenting our results regarding average sample sizes. However, the misinterpretation of these numbers lies in the meaning of the numbers and their role as descriptive rather than normative information. Whether the researchers have represented these numbers correctly or not is therefore not relevant here.

We are not aware of any taxonomy of quotation errors. Nor have we come across discussions of this particular type of quotation error. By exploring this quotation error further, we hope to learn more about ways in which research can be misrepresented and how these types of errors might be avoided. This misuse of previous research can ultimately lead to the establishment of “accepted facts” and can undermine the quality of qualitative research.

Aim

The overarching aim of this article is to explore one particular type of quotation error: the misrepresentation of descriptive information as normative. Specifically, we will explore this particular type of quotation error by mapping its presence in references to our 2011 study. In addition, we will discuss why authors might have made this type of quotation error and what we and other authors can do to avoid similar errors in the future.

Method

Searching and Including Articles That Referred to Our Study

We identified our 2011 study (Carlsen & Glenton, 2011) through Google Scholar (https://scholar.google.no) on September 7, 2017. We then selected all articles that were listed as having referred to our study. From this list of articles, we excluded duplicate articles; articles published in languages other than English, Norwegian, Swedish, or Danish; and articles that Claire Glenton or Benedicte Carlsen had coauthored. After examining the full text versions of each article, we further excluded articles that made no reference to our own article, despite it appearing in Google Scholar’s list; as well as articles where the authors had listed our study in the reference list but not actually used it in the main text. Finally, we excluded articles that we were unable to retrieve the full text version of.

Data Extraction and Analysis

In each of the included articles, we extracted quotations where the authors had referred to our own study. We also extracted text immediately preceding or following the quotation if it helped understand the context in which our study had been used.

Claire Glenton and Benedicte Carlsen independently categorized all quotations into one of two groups: (a) quotations where the authors had used the descriptive information from our study to justify their focus group and (b) quotations where the authors had referred to our study for other purposes or where the purpose was unclear.

For category (b), we noted how the authors were using the study but did not attempt to quantify these uses. We resolved disagreements through discussion. However, where we disagreed about whether the quotation belonged in category (a) or not, we gave this quotation the “benefit of the doubt” and categorized the quotation as unclear. When our study was cited more than once in an article, all quotations were checked, but if more than one quotation error was found, only one was recorded.

Results

Two hundred forty-six articles were listed as having referred to our study in Google Scholar (appendix available on request). From this list, we excluded 12 duplicate articles; 14 articles that were published in languages other than English, Norwegian, Swedish, or Danish; three articles where we were coauthors; and 10 articles that did not refer to our study at all, despite appearing in Google Scholar’s list, or where our study appeared in the reference list but was not cited in the main text. In addition, we excluded two articles because we were not able to find the full text. This left us with a total of 205 articles (see Figure 1).

Figure 1.

Flow diagram.

Using “Normal” as Normative

We identified the type of quotation error we were interested in, namely the misrepresentation of descriptive information as normative, in 104 (50.7%) of the 205 included articles.

In some of these quotations, authors stated directly that they had decided on the number of focus groups or the number of focus group participants because this was within the range that our study had reported:

Once the survey was complete, potential focus group interviewees were sought out, and a group of 10 volunteers from the survey study agreed to participate in the focus group. This sample size is appropriate and adequate since it falls within Carlsen and Glenton’s (2011) methodological study of sample-size reporting in focus group studies. (Study #46)

Based on Carlsen and Glenton’s (2011) findings from a review of FGD in research that the median number of participants was eight, it was decided that eight participants would be selected to participate in each FGD. (Study #56)

In other cases, this normative use of descriptive information about sample size was implied rather than stated directly:

Both the number of groups and the number of students included were small but within the range used in similar research (Carlsen & Glenton, 2011). (Study #52)

The misinterpretation of our descriptive information as normative was particularly evident when authors stated or implied that we regarded particular sample sizes as, for instance, “adequate,” “appropriate,” “recommended,” “suitable,” “ideal,” or “used successfully”:

We conducted three focus groups to collect data, a number recommended in recently published studies (Lid & Malterud, 2012; Carlsen & Glenton, 2011). (Study #19)

The authors acknowledge the small sample size of participants. Ideally, focus groups should comprise between 4 and 12 participants (Kitzinger, 1995; Carlsen & Glenton, 2011). (Study #26)

Although the number of focus groups was limited to four, this is within the range used successfully in previous studies (Carlsen & Glenton, 2011), with no new themes emerging by having a greater number of focus groups (Mason 2010). (Study #59)

Another variation of this quotation error was where authors stated that they had chosen a particular sample size because this was sufficient to reach saturation, with reference to our study:

Sample size was determined based on the principles of saturation, which suggest that, with as few as four discussions, no additional information will be obtained (Carlsen & Glenton, 2011). (Study #66)

Five focus groups were conducted for this research. Leading five separate focus groups has been found to be adequate to reach a saturation point, namely, the point where groups provide only repetitive information (Carlsen & Glenton, 2011). (Study #68)

Conducting six separate focus groups has been found to be adequate in reaching a point of saturation, that is, a point after which no more new information is retrieved from the interviews (Carlsen & Glenton, 2011; Onwuegbuzie et al., 2009). As such, six focus groups were conducted, at which point no new information appeared in the discussions, so no further focus groups were planned. (Study #83)

In our 2011 study, authors who had justified their focus group sample size often did so with reference to the principle of saturation either theoretical saturation (Glaser & Strauss, 1967) or the simpler concept of data saturation (Strauss & Corbin, 1990). In both cases, researchers are expected to collect and analyze data through an iterative process until saturation has been achieved. It is therefore not possible to predetermine sample size. Despite being a common concept in qualitative research environments, several of the authors had failed to use this concept correctly, claiming saturation but also predetermining their sample size, an inconsistency we pointed to in our study. It was therefore particularly ironic that our study that had pointed to the misuse of the saturation concept is now serving as a new source of the continued misuse of this concept.

Use of the Study for Other Purposes or Unclear Use of the Study

We did not identify the quotation error of interest in the remaining 102 articles. In many of these articles, the authors had referred to our study when describing or defining concepts or methods such as qualitative research, focus groups, saturation, purposive sampling, transferability, and accountability, for instance:

In qualitative research design approach, the aim is to explore a topic in depth (Carlsen & Glenton, 2011). (Study #106)

In many articles, authors had also correctly represented the main results of our study by referring to the wide variation and poor reporting of focus group sample sizes, for instance:

Reviews indicate that qualitative researchers demonstrate a low level of transparency regarding sample sizes and the underlying arguments for these (Carlsen & Glenton, 2011; Mason, 2010). (Study #141)

In the remaining articles, it was unclear why the authors had referred to the original study as there was no apparent link between the quotation and the study, for instance:

Purposive sampling contains a nonprobability sampling that allows the inclusion of specific components or subjects in a study. It ensures that the components will have certain features relevant to the study (Carlsen & Glenton, 2011). (Study #134)

We came across a number of other types of quotation errors and citation errors in this group of articles. For instance, several articles gave the wrong numbers when referring to our numerical results. However, we did not attempt to analyze or quantify these errors as the focus of this study was on one specific type of quotation error.

Discussion

We found that over half of the articles referring to our 2011 study misrepresented the study’s results by using descriptive information as normative guidance. While we only quantified this type of quotation error, the number is still much higher than the numbers reported in systematic reviews for quotation errors in general (Jergas & Baethge, 2015; Wager & Middleton, 2008). In addition, other types of quotation errors are likely to have been made that fall outside the scope of this study. This includes situations where one paper references another paper that has misrepresented our study but without mentioning that our study is the original source.

Assessments of quotation errors are often subjective and require judgment, particularly in this case, where the study results referred to involved both numerical and narrative results. Our assessments may also have been influenced by the fact that we were assessing other people’s use of our own published work. While we would have preferred that the numbers were lower and that more researchers had correctly represented the results of our study, we may also have been overly critical in our assessments of how other researchers had used our study. It is therefore possible that another, more independent assessor may have come to slightly different results. (In an attempt to address this limitation, the underlying data we assessed in this study are also available upon request.) We do, however, feel confident that the frequency of this type of quotation error is high enough to cause concern.

If this error continues, this could represent the beginning of a myth about recommended sample size in focus group studies. Other researchers have documented similar situations where poor referencing has led to the establishment of myths and beliefs despite a lack of empirical evidence to support them. For instance, Rekdal (2014) describes how poor referencing led to the “urban legend” that a decimal point error caused people to believe that spinach is an excellent source of iron. Similarly, Harzing (2002) describes how inappropriate referencing led to a myth regarding expatriate failure rates.

Why Do Quotation Errors Occur and Why Was Our Study Misrepresented?

As researchers, we all make referencing errors. In fact, during the writing of this article, we discovered one of our own in the original 2011 study. One reason for these errors may be that we do not read the study properly; a situation encouraged by lack of time, pressure to publish, and the sheer volume of literature that is available to us. In fact, we may not even read the study at all, relying instead on other people’s references to it. As researchers, we may also see what we want to see in other people’s research.

With regard to our own study, we suspect that many researchers have only read the abstract and have probably read even this part poorly, selecting the numbers describing typical sample sizes, while ignoring the text that puts these numbers into context. By the very act of counting a phenomenon and reporting what most researchers do, we may also have encouraged a type of cognitive bias referred to as the “bandwagon effect” (Leibenstein, 1950), where people are more likely to adopt practices and beliefs that have already been adopted by others. This bias may make it particularly easy for people to interpret descriptive information normatively. Another circumstance that may have encouraged the misrepresentation of our study is the absence of properly described, consistent guidance in the literature on focus group sample size (Carlsen & Glenton, 2011). It is therefore perhaps not surprising that researchers looking for guidance have come across our study and chosen to use it as they do.

In addition, we are familiar with the pressure many qualitative researchers are under to prespecify exact sample sizes in research proposals, particularly in research environments with more quantitative research traditions. While this is in conflict with the iterative approaches usually seen as more appropriate for qualitative data collection, our own experiences suggest that a lack of prespecified sample sizes can be interpreted as a sign of poor research by funders with quantitative backgrounds.

We would argue that it is good practice to justify one’s choice of methods, both for qualitative and quantitative methods of data collection, although these justifications should reflect the research aim and epistemology. It is also reasonable to support these justifications through references to others’ work. However, our use of references can quickly become ritualized and serve more as a symbol of rigor than as substantive support. The use of references as symbol rather than substance is likely to increase the frequency of quotation errors. The need to be brief, particularly in research journals more used to publishing quantitative studies, may also lead us to rely on references rather than offer our own lengthier descriptions and justifications. We would argue, however, that a well-described and thoughtful justification with no supporting references is preferable to a brief description with references that actually have little meaning.

What Can Be Done to Prevent This Type of Quotation Error?

In their systematic review of quotation accuracy in medical journal, Jergas and Baethge (2015) summarized their own and other authors’ suggestions for how quotation errors might be avoided. Several of these suggestions focus on increased reference checking and target the behavior of the referring authors, their journal editors, and peer reviewers.

We agree that authors referring to other people’s work carry the main responsibility for ensuring that they reference it appropriately. Editors and peer reviewers also carry some responsibility for ensuring some level of quality. But it is also important to think through what we as study authors can do to avoid being misrepresented. Most importantly, we have a responsibility to ensure that our results as presented as clearly as possible. We probably also need to manage our expectations regarding other researchers’ willingness to read our articles carefully, from start to finish. With the assumption that many researchers will only read the article’s abstract as our starting point, we suggest a number of approaches that might help.

First of all, assume that readers will not distinguish between main points and minor points and that minor points may be given more attention than they deserve if they are included in the abstract. Make sure, therefore, that the abstract focuses on the main points of your study.

Secondly, assume that readers will need help in distinguishing between descriptive and normative information. Make it clear whether your main results are descriptive or normative and integrate this information as closely as possible into the main results.

Thirdly, assume that numbers are likely to be extracted and separated from the accompanying text that explains them. Were we to publish the 2011 study again, we would not have included numbers at all in the abstract but would instead have stuck to text describing the main points. Consider doing the same if you think your numerical results could be taken out of context.

Finally, consider these issues in your study title as well. The title of our original study was “What about N? A methodological study of sample size reporting in focus group studies.” This title may have given the reader an expectation that we were going to offer guidance on sample size. Were we to publish the study again, we would have tried to flag the main messages of the study to the reader, for instance, by including terms such as “wide variations” or “poor reporting.”

As Jergas and Baethge (2015) point out, there is a lack of evidence about the actual impact of measures targeting quotation errors. Our own suggestions are no exception. However, more attention needs to be given to what we ourselves can do to avoid being misrepresented and to ensure that the results of our work are used as intended.

Conclusion

Our study shows very high rates of one type of quotation error: the misrepresentation of descriptive information about focus group sample size as normative. Researchers referring to other researchers’ work carry most of the responsibility for ensuring that they do this appropriately. However, the authors of the research being referred to also need to consider how they can make their results clearer. We have offered suggestions as to how this might be achieved.