Abstract
Reviewing Mixed Methods Research Manuscripts
Our work as guest coeditors for this special issue was greatly assisted by our referees. In this issue, we presented six of more than 1 million scholarly journal articles that were published this year. Indeed, according to Ware (2006), There are approximately 23,000 scholarly journals in the world, collectively publishing 1.4 million articles a year. The number of articles published each year and the number of journals have both grown steadily for over two centuries, by about 3% and 3.5% per year, respectively. The reason is the equally persistent growth in the number of researchers, which has also grown at about 3% per year and now stands at around 5.5 million. (p. 3)
As noted by Fischer (2011), in general, reviewers should provide reviews that are constructive, comprehensive, thorough, balanced, diplomatic, open minded, respectful of the author’s ideas and property rights, and prompt. Even more importantly, reviewers always should strive to maximize nonmaleficence (i.e., not causing harm to the authors); beneficence (i.e., working for the benefit of authors); (social) justice (i.e., providing reviews based on universal principles and rules, in an impartial and warranted manner in order to guarantee fair and equitable treatment of all authors); fidelity (i.e., demonstrating faithfulness, loyalty, and commitment to the journal); professional competence (i.e., writing reviews that are within the reviewer’s set of skills and knowledge of the topic explored, methodology and methods used, and the results reported); integrity (i.e., being fair, honest, and respectful of author’s ideas, procedures, and findings); scholarly responsibility (i.e., adhering to best practices through reviews that are warranted and transparent and that involve self-reflection and meta-cognition on the part of the reviewer); social responsibility (i.e., demonstrating awareness of the social dimensions of the author’s topic); and respecting rights, dignity, and diversity (i.e., striving to eliminate bias for misrepresenting authors’ works and not discriminating authors based on exceptionalities of their topics; cf. Onwuegbuzie & Frels, 2016)—which, if all elements are followed, would yield a reviewer who is meta-ethical, which implies that he or she adheres to virtue ethics (i.e., referring to the character of the reviewer—as opposed to the standards of excellence for the journal—providing the impetus for ethical reviews) and pragmatic ethics (i.e., using the standards set by the journal editor under the assumption that the editor is progressing morally in line with the progression of scientific knowledge).
Over the years, a few researchers have examined the factors that determine reviewers’ appraisal of manuscripts submitted to journals for review for possible publication. For example, in Gilliland and Cortina’s (1997) study, 116 reviewers from five rehabilitation counseling-related journals completed a 63-item questionnaire design to extract information on common reasons for manuscript acceptance or rejection. These reviewers identified 13 manuscript characteristics that were deemed to be important determinants of reviewer acceptance or rejection, which represented either negative influences or positive influences. Respectively, the eight negative influences were direct replications that added little to theory development (51%); topics that present material well outside the mainstream of the field (47%); manuscripts that represent pilot studies with little evidence of generalizability (43%); manuscripts that are too lengthy (36%); application of inappropriate analysis (e.g., parametric test for ordinal data; 35%); manuscripts containing only secondary analysis of data presented by others (32%); experimental data with no control group (29%); and studies lacking statistical significance, whether they are based on either new or currently popular theories (28%). Contrastingly, respectively, the five positive influences were manuscripts representing a new, original theory (33%); author analyzing interval data appropriately (29%); manuscript with content of interest to the field but differing in content from those traditionally published in the journal (23%); new statistical methods, including data collection techniques (20%); and author’s reputation (20%).
However, all the studies examining factors that determine reviewers’ appraisal of manuscripts submitted to journals for review for possible publication have involved the appraisal of monomethod manuscripts that represented the quantitative research tradition only or the qualitative research tradition only. Indeed, to date, no studies exist examining factors that determine reviewers’ appraisal of manuscripts submitted to journals that represent the mixed methods research tradition. Therefore, for this editorial, we decided to investigate these factors. In particular, our primary goal was to begin the conversation among members of the mixed methods research community regarding what makes a quality review by using the appraisal of manuscripts of our special issue reviewers to develop a framework for comprehensively reviewing mixed methods research manuscripts.
Method
Research Design
Our study involved combining quantitative and qualitative approaches within a case study—yielding what Onwuegbuzie (2015) referred to as a mixed methods case study (MMCS)—specifically, a qualitative-dominant MMCS. The case was bounded by the manuscripts submitted to the two International Journal of Qualitative Methods (IJQM) mixed methods research special issues. These two special issues yielded 20 manuscripts that we selected as guest coeditors (i.e., criterion sampling; Onwuegbuzie & Collins, 2007) from the 70 proposals submitted, based on the quality and fit (i.e., it [potentially] provided a compelling example of how mixed methods research informs and enhances qualitative research—consistent with our call for the special issue). From the 20 sets of authors whose proposals were accepted, 16 submitted full manuscripts within the 5-month deadline for submission.
After receiving the submissions, we sought three reviews for each manuscript as part of a quadruple-blind peer-review process (i.e., during the initial review process, [a] the peer reviewers were not aware of author identification, [b] each author(s) was not aware of the identity of the reviewers of the manuscript, [c] the action editor was not aware of author identification at the time that the editorial decision was made, and [d] the action editor was not aware of the reviewers of the manuscript at the time that the editorial decision was made)—with the exception of the final manuscript that was reviewed for which the action editor was aware of the identity of the author(s), which yielded a triple-blind review. It is noteworthy that for each manuscript, we selected at least one reviewer who was a recognized mixed methods research expert. Together, the reviewers provided the authors feedback that was greater than the sum of the individual reviews.
Across the 16 submitted manuscripts, 45 reviews of the initial submissions were obtained, which became the final sample for our MMCS study. Each of these 45 individual reviewers provided her or his own feedback based on six criteria (i.e., relevance, quality of information, quality of writing, conforming to American Psychological Association [APA] guidelines, adherence to ethical standards, and suggestions for improvement). One of the editors then compiled the feedback together and pointed out the general themes to the author while also giving the authors access to the individual reviews. We were unusually fortunate as editors that all our reviewers were recognized experts in their fields, timely in their contributions to this process, and extraordinarily helpful in their encouragement and critiques. Indeed, several of our authors commented on the mentoring that they received via the reviewers’ feedback, and we are grateful to our reviewers for providing the learning experience for all of us.
Analysis
We conducted a sequential mixed methods analysis (Onwuegbuzie & Teddlie, 2003) to analyze the data. This analysis involved us conducting a qualitative analysis followed by a quantitative analysis. Specifically, we used constant comparison analysis (Glaser, 1965) to analyze qualitatively all of the reviewers’ comments. This analysis involved coding chunks of words, by each reviewer, into meaningful units of words, phrases, sentences, or paragraphs that described the contents of the segmented data. These codes represented the underlying themes within each meta-theme, which were identified a posteriori (Constas, 1992). Our codes and locus of typology (i.e., theme) development were investigative, arising from our own constructions (Constas, 1992). After the coding process, we transformed both the meta-themes and themes into numerical data that could be analyzed qualitatively—a process known as quantitizing (Miles & Huberman, 1994; Onwuegbuzie & Teddlie, 2003; Sandelowski, Voils, & Knafl, 2009; Tashakkori & Teddlie, 1998).
Once we had quantitized the data, we conducted a descriptive analysis of the quantitized meta-themes and themes, which involved determining the frequency (i.e., by counting the number of reviewers who contributed to each theme and meta-theme) and prevalence rates (i.e., by determining the proportion of reviewers who contributed to each theme and meta-theme), for the full set of reviewers (i.e., n = 45) as well as for the reviewers who had mixed methods research expertise (i.e., mixed methods reviewers; n = 17) and the reviewers who did not have mixed methods research expertise but who had expertise in other areas such as in quantitative research or qualitative research (i.e., nonmixed methods reviewers; n = 28). Notably, all our mixed methods reviewers were leading mixed methodologists in the field with (a) on average, these reviewers having more than 16 years of academic experience (i.e., M = 16.47, SD = 12.24); (b) slightly more than one half of them (52.9%) beginning their careers as mixed methodologists before the publication of the landmark first edition of the Handbook of Mixed Methods Research (Tashakkori & Teddlie, 2003); (c) nearly one third of them (i.e., 29.4%) being authors/coauthors of mixed methods research textbooks; (d) nearly one half of them (i.e., 47.1%) being authors/coauthors of methodology (i.e., quantitative, qualitative, and/or mixed research) textbooks; and (e) nearly one half of them (i.e., 47.1%) being authors/coauthors of a chapter in one or both editions of the Handbook of Mixed Methods Research. In addition to a descriptive analysis, for the mixed methods reviewers, we conducted a correspondence analysis to factor the meta-themes with their associations in at least a two-dimensional map (Onwuegbuzie, Leech, Dickinson, & Zoran, 2010). A correspondence analysis is a multivariate and visual technique for conducting a quantitative analysis of emergent themes or meta-themes (Michailidis, 2007). Onwuegbuzie and Combs (2010) referred to a correspondence analysis as representing a crossover mixed analysis, whereby we used the analysis types associated with one tradition (i.e., quantitative analysis: descriptive analysis and correspondence analysis) to analyze data associated with a different tradition (i.e., qualitative data: emergent meta-themes). We used QDA Miner 4.1.23 (Provalis Research, 2015) to conduct our correspondence analysis.
Results
Ideally, consistent with our qualitative dominant MMCS design, in our results section, we had intended to present an array of rich quotations directly from the voices of the reviewers. However, due to our very short time frame for writing our editorial, we did not have time to obtain the permission of all 45 reviewers as well as from all the 16 sets of authors to include these quotations. Also, each quotation that we would have provided likely would have revealed the identity of the manuscript to which we were referring to. And we did not want to give any impression that we were singling out any particular set of authors for criticism. Thus, in providing the following findings, we have omitted the quotations that were present in earlier drafts of this editorial. However, after we have obtained permission from the reviewers and authors, we do intend to present these quotations in a follow-up work such that reviewers can hear the voices of the reviewers and have a sample of the quality of the reviews that we received, for which we are so appreciative.
Meta-Themes That Represented a Limitation
The constant comparison analysis yielded the following six meta-themes that each represented a limitation of one or more of the 16 manuscripts: lack of warrantedness, lack of justification, writing issues, lack of transparency, lack of integration, and philosophical issues. Each of these meta-themes will be described in subsequent sections. Table 1 displays the frequency and prevalence rates pertaining to the meta-themes extracted from the reviewers’ comments to the 45 submitted manuscripts for all reviewers and as a function of type of reviewer. It can be seen from this table that for the mixed methods reviewers, lack of warrantedness was the most prevalent meta-theme, with slightly more than three quarters (i.e., 76.5%) of these reviewers being classified under this meta-theme. That is, mixed methods reviewers were more likely to provide a criticism of a manuscript that was classified as representing lack of warrantedness than any other meta-theme. In contrast, for the nonmixed methods reviewers, lack of justification was the most prevalent meta-theme, with more than two thirds (i.e., 71.4%) of these reviewers being classified under this meta-theme.
Frequency and Prevalence Rates Pertaining to the Meta-Themes Extracted From Reviewers’ Comments to Manuscripts Submitted for the Two International Journal of Qualitative Methods Special Issues in Mixed Methods Research.
Figure 1 illustrates the reviews of the 17 mixed methods reviewers mapped, via correspondence analysis, onto the space that displays the six emergent meta-themes. This figure shows how the reviews of the mixed methods reviewers related to each other in regard to these six meta-themes. In particular, the lack of warrantedness and philosophical issues meta-themes exclusively were represented by conceptual, theoretical, or methodological manuscripts. In contrast, the lack of transparency meta-theme exclusively was represented by empirical manuscripts. Finally, the lack of integration, lack of justification, and writing issues meta-themes were represented by both conceptual/theoretical/methodological manuscripts and empirical manuscripts.
Correspondence analysis plot of the six emergent meta-themes associated with limitations as a function of genre of manuscript. E = empirical manuscript; M = conceptual/theoretical/methodological manuscript.
Meta-theme 1: Lack of warrantedness
This meta-theme, representing the most prevalent meta-theme, echoes the reporting criteria as described in the seminal document developed by the Task Force on Reporting of Research Methods in American Educational Research Association (AERA) Publications and adopted by the AERA Council in 2006, wherein it is stipulated that reports of empirical research should be warranted inasmuch as adequate evidence should be provided to justify the results and conclusions (American Educational Research Association [AERA], 2006). Thus, the lack of warrantedness meta-theme refers to any criticism made by a reviewer regarding inappropriate, inadequate, or missing evidence. This meta-theme contained the following seven themes: (a) lack of evidence, (b) insufficient findings, (c) old/inadequate sources used, (d) lack of definition, (e) no/inadequate reference to the most recent mixed methods research literature, (f) reference list errors, and (g) citation errors. The lack of evidence theme refers to the author making a statement anywhere in the manuscript that was clearly biased, overly judgmental, nonscholarly, or the like, or, even if the statement was reasonable, lacked citations. The insufficient findings theme, as the term suggests, arose when the author omitted one or more important findings that were needed to address one or more research questions and/or to test one or more hypotheses, or one or more findings that were presented lack sufficient depth. The old/inadequate sources used theme implies that citations were provided; however, a significant proportion of them were either dated or were not appropriate for the claim made. The lack of definition theme means that one or more terms were introduced without any definition and/or explanation. The no/inadequate reference to the most recent mixed methods research literature theme occurred when the author did not provide the most up-to-date citation(s) from the mixed methods research literature at one or more of the 12 components of a primary research report identified by Onwuegbuzie and Frels (2016): problem statement, literature review, theoretical/conceptual framework, research question(s), hypotheses, participants, instruments, procedures, analyses, interpretation of the findings, directions for future research, and implications for the field. The reference list errors theme occurred when the author provided one or more references in the reference list that contained an error of omission or commission. And, as noted by APA] (2010, p. 180), Because one purpose of listing references is to enable readers to retrieve and use the sources, reference data must be correct and complete …. Authors are responsible for all information in their reference lists. Accurately prepared references help establish your credibility as a careful researcher.
Finally, the citation errors theme reflects a failure “to make certain that each source referenced appears in both places [text and reference list] and that the text citation and reference list entry are identical in spelling of author names and year”; APA, 2010, p. 174). Disturbingly, citation errors are rampant in manuscripts of all genres—with as many as 91.8% of authors committing one or more citation errors (Onwuegbuzie, Frels, & Slate, 2010). Interestingly, Onwuegbuzie, Waytowich, and Jiao (2006) reported that manuscripts submitted to the Research in the Schools journal that contain more than three citation errors are approximately 4 times more likely (odds ratio = 4.01; 95% confidence interval = 1.22, 13.17) to be rejected than are manuscripts with three or less citation errors. Consistent with this finding, Onwuegbuzie, Frels, et al. (2010) documented that manuscripts with more citation errors are statistically and practically (Cohen’s [1988] d = 0.45) significantly less likely to be accepted for publication than are manuscripts with less citation errors. Thus, citation errors are not only errors that ethically must be avoided, but also their presence is predictive of unsatisfactory manuscripts to some degree.
Table 2 displays the frequency and prevalence rates of themes pertaining to the lack of warrantedness meta-theme themes extracted from the reviewers’ comments to the 45 submitted manuscripts for all reviewers, and as a function of type of reviewer. It can be seen from this table that the lack of evidence theme was the most prevalent for both mixed methods reviewers and nonmixed methods reviewers.
Meta-Theme 1: Lack of Warrantedness: Themes Extracted From Reviewers’ Comments.
Meta-theme 2: Lack of justification
This meta-theme was pertinent when the author did not make clear the importance of the study in some way. This meta-theme contained the following six themes: (a) Underdeveloped, (b) Lack of significance, (c) Did not advance qualitative research, (d) Did not advance mixed methods research, (e) Lack of rationale, and (f) Lack of purpose statement. The underdeveloped theme refers to a manuscript containing a conceptual, theoretical, or methodological essay wherein the underlying assumptions, ideas, beliefs, propositions, theories, schemas, models, hypotheses, or the like have not been explicated sufficiently. Or, in the case of a mixed methods research study, this theme implies that one or more of the elements of the inquiry (e.g., conceptual framework, theoretical framework) have not been discussed sufficiently. Whereas some reviewers pointed to elements of the manuscript that were underdeveloped, in some cases, the whole manuscript was described as being underdeveloped, with the reviewer using a word like “superficiality.” In many instances here, the reviewer criticized the manuscript for not being true to the special issue theme of enhancing qualitative research through mixed methods. The lack of significance theme arose when the author did not make clear the (educational) significance of the study or the conceptual/theoretical/methodological essay. The did not advance qualitative research theme was particularly pertinent for this set of manuscripts because the theme of the special issue was “How mixed methods informs and enhances qualitative research.” Similar to the did not advance qualitative research theme, the Did not advance mixed methods research occurred when the value added for conducting a mixed methods research study over a monomethod research study was not apparent.
The lack of rationale theme occurred when authors did not provide a rationale either for their study or for their conceptual/theoretical/methodological essay. With respect to empirical studies, criticism associated with this theme also arose when the author did not provide a rationale for mixing. Finally, the lack of purpose statement theme occurred when authors did not specify the purpose either of their study or of their conceptual/theoretical/methodological essay. Table 3 displays the frequency and prevalence rates of themes pertaining to the lack of justification meta-theme extracted from the reviewers’ comments to the 45 submitted manuscripts for all reviewers, and as a function of type of reviewer. It can be seen from this table that the undeveloped theme was the most prevalent.
Meta-Theme 2: Lack of Justification: Themes Extracted From Reviewers’ Comments.
Meta-theme 3: Writing issues
This metatheme, which pertains to the quality and effectiveness of the authors’ writing, contained the following six themes: (a) Lack of clarity, (b) Lack of consistency, (c) Lack of structure, (d) Repetition, (e) Inappropriate word usage, and (f) Lack of transitions. Specifically, comments relating to lack of clarity occurred when one or more sections of the manuscript was not clearly written. Also adversely impacting the clarity of the manuscript was lack of consistency, which, interestingly, was only pointed out by the nonmixed methods reviewers (i.e., n = 5). This theme was apparent when there were two or more statements that contradicted each other. For instance, one author presented information in the abstract that was not consistent with information provided in the body of the manuscript. In terms of the lack of structure theme, the reviewer making this observation tended to criticize the author’s introduction section and/or method section. With regard to the introduction section, the author who was criticized did not use the following standard format: literature review—theoretical framework—rationale—purpose statement—research questions—hypothesis—educational significance. That is, the rationale, purpose statement, research questions, and educational significance, respectively, were not presented at the end of the introduction section—after the literature review section—such that in the introduction section, the literature review identified the knowledge base, the rationale identified a significant gap in the literature, and the purpose statement delineated how the author(s) attempted to fill this gap. Alternatively, the author could have presented the rationale, purpose statement, and research questions, respectively, early on in the introduction section (e.g., first or second paragraph). This would be followed by a section entitled such as “Review of the Related Literature,” which contained the literature review, theoretical framework, and educational significance. In any case, the author’s lack of structure adversely affected the flow of the manuscript as well as the ease with which the logic of the article could be followed. That is, this lack of structure prevented the introduction section from flowing in a maximal way. With respect to the method section, the author who was criticized did not use the standard participants—instruments—procedure—analysis format.
The repetition theme, as its label suggests, came to the fore when the author unnecessarily repeated information at various points of the manuscript. The inappropriate word usage theme occurred when the author used an inappropriate word to describe a concept. One example of this was an author mistakenly using the words “methodology” and “method” interchangeably. Yet, these terms are very different. Whereas a methodology can be defined as a broad approach to scientific inquiry with general preferences for certain types of designs, sampling logic, analytical strategies, and so forth, methods include specific strategies and procedures for research design, sampling, data collection, analysis, and the like.
Finally, the lack of transitions theme occurred when the author did not adequately link sentences. In particular, in these instances, the author did not use link words to make these connections. Yet, link words/phrases are very useful for connecting ideas and, hence, for connecting sentences and paragraphs. Interestingly, analyzing manuscripts that were submitted over a 3-year period (i.e., 2011–2014) to the journal Research in the Schools (RITS), for which one of the special issue editors serves as an editor, via the use of QDA Miner 4.1 and WordStat 6.0, Onwuegbuzie (2016) documented that the dimension labeled as add information/provide similarity was the most commonly used (by 71.6% of authors), followed by the dimension labeled narration (by 60.8% of authors), and the dimension labeled sequence previous ideas (also 60.8% frequency). The remaining nine dimensions were used by less than 50% of the authors. The three most common link words, respectively, were finally (52.7%), similarly (52.7%), and additionally (51.4%). All other link words/phrases were used by one third of the authors or less. Even more compelling was Onwuegbuzie and Frels’s (2016) finding that the following dimensions statistically significantly and practically significantly predicted whether or not a manuscript was rejected by the editor: add information/provide similarity, narration, and provide an emphasis. More specifically, manuscripts that contained one or more link words/phrases that were classified as add information/provide similarity were 1.75 (95% confidence interval [CI] = 1.09, 2.79) times less likely to be rejected than were their counterparts, manuscripts that contained one or more link words/phrases that were classified as narration were 1.32 (95% CI = 1.01, 2.31) times less likely to be rejected than were their counterparts, and manuscripts that contained one or more link words/phrases that were classified as provide an emphasis were 1.75 (95% CI = 1.07, 2.86) times less likely to be rejected than were their counterparts. Table 4 displays the frequency and prevalence rates of themes pertaining to the writing issues meta-theme extracted from the reviewers’ comments to the 45 submitted manuscripts for all reviewers, and as a function of type of reviewer. It can be seen from this table that the lack of clarity theme was by far the most prevalent.
Meta-Theme 3: Writing Issues: Themes Extracted From Reviewers’ Comments.
The six themes representing the writing issues meta-theme, each play a role in adversely affecting the readability of a manuscript. Interestingly, Onwuegbuzie, Mallette, Hwang, and Slate (2013) provided evidence that readability plays an important role with regard to the quality of manuscripts. Using the Flesch Reading Ease (RE) and Flesch-Kincaid Grade Level (GL)—two commonly used and easily accessible (e.g., via Microsoft Word) readability formulas, among many other findings, Onwuegbuzie et al. (2013) observed that (a) manuscripts with Flesch RE scores between 0 and 30 are 1.64 more times less likely to be rejected than are manuscripts with Flesch RE scores greater than 30, and (b) manuscripts with Flesch-Kincaid GL scores of 16 and above are 4.55 times less likely to be rejected than are manuscripts with Flesch-Kincaid GL scores less than 16. Interestingly, with regard to the Flesch RE scores, the findings of Gazni (2011) and Metoyer-Duran (1993) were remarkably consistent with those results reported by Onwuegbuzie et al. (2013). Specifically, Gazni (2011) examined the relationship between readability estimates (i.e., Flesch RE scores of abstracts only) and citation rates for articles published between 2000 and 2009 from the five institutions (e.g., Harvard) that secure the largest number of citations. Based on an analysis of approximately 260,000 abstracts, spanning 22 disciplines, Gazni derived a statistically significant, negative relationship between text difficulty and citation rates; in other words, the more difficult the text, the more it was cited. Additionally, the Flesch RE scores, which ranged from an average of 12.6 in pharmacology/toxicology to an average of 25.6 in mathematics, all fell into the very difficult range or college grade level (i.e., 1–30), the highest level of text difficulty.
Metoyer-Duran (1993) examined whether readability estimates differed significantly among published, accepted, and rejected manuscripts and abstracts from College and Research Libraries during the 1990–1991 period. This researcher ascertained that the readability estimates of manuscripts accepted for publication were significantly different from the readability estimates of manuscripts rejected for publication. For example, the mean Flesch RE score was 28.04 for accepted manuscripts (i.e., within the 1–30 range) and 30.77 (i.e., outside the 1–30 range) for rejected manuscripts. Thus, manuscripts that were accepted for publication contained text that was more difficult than was the text in manuscripts that were rejected. The findings of Onwuegbuzie et al. (2013), Gazni (2011), and Metoyer-Duran (1993) regarding the predictability of the readability of manuscripts make the writing issues meta-theme particularly noteworthy.
Meta-theme 4: Lack of transparency
This meta-theme also echoes the reporting criteria as described by the authors of AERA’s (2006) seminal document, wherein it is stipulated that reports of empirical research should be transparent inasmuch as reporting should make explicit the logic of inquiry and activities that led from the development of the initial interest, topic, problem, or research question; through the definition, collection, and analysis of data or empirical evidence; to the articulated outcomes of the study (AERA, 2006, p. 33). According to the standards, alongside being warranted, “Reporting that takes these principles into account permits scholars to understand one another’s work, prepares that work for public scrutiny, and enables others to use that work” (AERA, 2006, p. 33). In the context of the IJQM reviewers, the lack of transparency meta-theme refers to any criticism made by a reviewer regarding missing information about the methods used such as lack of information regarding the sample size, sampling scheme, research design, elements of the data collection process, and/or the analysis process. This meta-theme contained the following six themes: (a) insufficient procedures, (b) lack of sampling clarity, (c) lack of conclusion, (d) insufficient explanation of analysis, (e) poor discussion of table/figure, and (f) did not provide directions for future research. Specifically, comments relating to insufficient procedures theme, as the label suggests, occurred when the author did not provide adequate procedural information for the reviewer to assess what had been undertaken. Similarly, the lack of sampling clarity theme emerged when the author failed to provide sufficient information regarding the sample size and sampling scheme for all phases of the mixed methods research study (i.e., type of random sampling scheme [e.g., simple random sampling] or type of purposive sampling scheme [e.g., convenience sampling, criterion sampling]; see, for e.g. Collins, Onwuegbuzie, & Jiao, 2007; Kemper, Stringfield, & Teddlie, 2003; Onwuegbuzie & Collins, 2007; Teddlie & Yu, 2007) and/or the relationship between the quantitative and qualitative samples (i.e., mixed sampling design; Onwuegbuzie & Collins, 2007). The lack of conclusion theme, which, interestingly, was only pointed out by the mixed methods reviewers (i.e., n = 4), arose when an author did not provide a conclusion to their study or to their conceptual/theoretical/methodological essay. On the two occasions when reviewers criticized a manuscript as containing an insufficient explanation of analysis, the authors made only scant mention of the analysis used. As such, it was not possible for these reviewers to assess the appropriateness of the analysis techniques used. One of the authors did not even refer to the tables and figures, let alone discuss them, thereby leaving it for the readers to interpret these visual representations purely by themselves, which, in turn, affected the clarity of the results that were presented. Thus, this error of omission was classified under the poor discussion of table/figure theme. As noted by the authors of APA (2010), “In the text, refer to every table and tell the reader what to look for. Discuss only the table’s highlights…” (p. 130). Finally, the did not provide directions for future research theme, which arose on one occasion, emerged when the author did not provide readers with any recommendations for follow-up studies in the discussion section. Table 5 displays the frequency and prevalence rates of themes pertaining to the lack of transparency meta-theme extracted from the reviewers’ comments to the 45 submitted manuscripts for all reviewers and as a function of type of reviewer.
Meta-Theme 4: Lack of Transparency: Themes Extracted From Reviewers’ Comments.
Meta-theme 5: Lack of integration
This meta-theme was pertinent when the author did not (sufficiently) integrate the quantitative and qualitative components but, instead, presented the quantitative and qualitative components in a separate manner. This meta-theme contained the following five themes: (a) lack of mixed methods framework, (b) lack of validity/legitimation discussion, (c) inappropriate research design, (d) lack of application, and (e) lack of linking research question(s) to research design. The lack of mixed methods framework theme dealt with the author not using one of the numerous existing mixed methods research frameworks (e.g., designs, typologies, and models) to frame their studies. Each of the more than 40 books published on mixed methods research has a unique framework that could have been used by the authors, as do numerous journal articles and book chapters. For example, between them, the first edition (Tashakkori & Teddlie, 2003) and the second edition (Tashakkori & Teddlie, 2010) of the Handbook of Mixed Methods Research contained typologies for several stages of the mixed methods research process, including the purpose (Newman, Ridenour, Newman, & DeMarco, 2003), research questions (Plano Clark & Badiee, 2010), research design (e.g., Nastasi, Hitchcock, & Brown, 2010), and data analysis (e.g., Onwuegbuzie & Combs, 2010). Yet, the authors who were flagged by the reviewers did not use any of these frameworks. Nor did they develop their own framework. Further, three sets of authors’ manuscripts contained a lack of validity/legitimation discussion about their findings, despite the fact that there are several mixed methods-based validity/legitimation frameworks in existence. Indeed, Heyvaert, Hannes, Maes, and Onghena (2013) identified the following 13 validity frameworks that they called critical appraisal frameworks (CAFs): Alborz and McNally (2004); Bryman, Becker, and Sempik (2008); Caracelli and Riggin (1994); Creswell and Plano Clark (2007; see also Creswell & Plano Clark, 2011); Dellinger and Leech (2007); Dyba, Dingsøyr, and Hanssen (2007); Greene (2007); O’Cathain, Murphy, and Nicholl (2008); Onwuegbuzie and Johnson (2006); Pluye, Gagnon, Griffiths, and Johnson-Lafleur, (2009); Pluye, Grad, Dunikowski, and Stephenson (2005); Sale and Brazil (2004); and Teddlie and Tashakkori (2009). From these 13 CAFs, Heyvaert et al. (2013) generated the following 13 headings that group similar criteria: criteria for qualitative part of the study; criteria for quantitative part of the study; criteria for mixing and integration of methods; rationale for mixing methods stated; theoretical framework; research aims and questions; design; sampling and data collection; data analysis; interpretation, conclusions, inferences, and implications; context; impact of investigator; and transparency. This points to the lack of general awareness of quality criteria for mixed methods research.
The inappropriate research design theme, as the label suggests, reflects the two sets of authors who did not specify a design that was consistent or compatible with the underlying research questions and procedures used. In contrast, the lack of application theme referred to a conceptual essay in which the authors did not illustrate the applications or implementation of the framework used. Finally, the lack of linking research question(s) to research design theme represents a set of authors who did not show the connection between the research question and selected research design—for example, as outlined by Onwuegbuzie and Leech (2006) and Plano Clark and Badiee (2010). Table 6 displays the frequency and prevalence rates of themes pertaining to the lack of integration meta-theme extracted from the reviewers’ comments to the 45 submitted manuscripts for all reviewers and as a function of the type of reviewer.
Meta-Theme 5: Lack of Integration: Themes Extracted From Reviewers’ Comments.
Meta-theme 6: Philosophical issues
This meta-theme was pertinent when the author did not (adequately) discuss the underlying research philosophy. This meta-theme contained the following two themes: (a) mislabeling of philosophy and (b) no clear specification of self-philosophy. The mislabeling of philosophy theme refers to the author not showing adequate knowledge and understanding of the underlying research philosophy. Indeed, one reviewer noted to the author that he or she or they clearly lacked knowledge of mental models associated with mixed methods research. Another author was criticized for use of the term “positivism” instead of “postpositivism.” Now, there are two major types of positivism: (a) classical positivism, which was introduced by Auguste Comte (French philosopher) and (b) logical positivism, which originated in the Vienna Circle, a group of European scholars, during the 1920s and 1930s. As stated by Yu (2003, p. 9), “When many authors discuss the relationship between positivism and research methodology, the context is situated in logical positivism rather than classical positivism.” According to Yu (2003), classical positivism represents a single movement, characterized by a philosophy that scientific inquiry should be empirical, which yielded antirealism and instrumentalism. In contrast, Hacking (1983) identified the following six major themes of logical positivism: (a) emphasis on verification, (b) proobservation, (c) anticause, (d) downplaying explanation, (e) antitheoretical entities, and (f) antimeta-physics. Indeed, logical positivism was discredited as a viable research philosophy after the Second World War. Thus, when authors refer to logical positivists, it is very likely that this reference represents a mischaracterization of contemporary quantitative researchers—as was the case here. Rather, a significant proportion of quantitative researchers has postpositivistic leanings, whose ontology (i.e., nature of reality) is that understanding of reality is constructed; epistemology (i.e., nature of knowing) is that findings are probably objectively obtained using primarily quantitative methods; axiology (i.e., role of values in inquiry) is that research is influenced by values of researchers; rhetoric (i.e., language of research) is that a formal writing style using an impersonal voice predominates; and methodology (i.e., process of research) that stems from a deductive logic in which research is influenced by theory/hypothesis (cf. Johnson & Onwuegbuzie, 2004).
The no clear specification of self-philosophy theme came to the fore when the reviewer criticized the author(s) for not making clear her or his or their own research philosophy. Another criticism falling under this theme was the author(s) providing a research philosophy that contradicted other statements made in the manuscript. The third and final criticism surrounded the author(s) seemingly unwittingly shifting her or his or their philosophical position during the manuscript. Table 7 displays the frequency and prevalence rates of themes pertaining to the philosophical issues meta-theme extracted from the reviewers’ comments to the 45 submitted manuscripts for all reviewers and as a function of type of reviewer.
Meta-Theme 6: Philosophical Issues: Themes Extracted From Reviewers’ Comments.
Meta-Themes That Represented a Strength
The constant comparison analysis also yielded 10 themes that each represented a strength of the manuscripts: clear writing, enhance qualitative research, useful framework, well grounded, significant, evidence provided, identification of appropriate philosophy, thorough analysis, well designed, and creative. Because all of these themes represent the opposite of one of the themes (i.e., what we call a theme antonym) discussed in previous sections—for example, the enhance qualitative research theme represents the opposite of the did not advance qualitative research theme under the lack of justification meta-theme—we assume that these 10 strengths-based themes are self-explanatory and do not need further explication. Table 8 displays the frequency and prevalence rates pertaining to the themes extracted from the reviewers’ comments to the 45 submitted manuscripts for all reviewers and as a function of type of reviewer. It can be seen from this table that clarity in writing was by far the most common reason for a manuscript receiving praise, just as it yielded a very common reason for a manuscript to be criticized if this clarity was absent.
Meta-Theme 7: Strengths: Themes Extracted From Reviewers’ Comments.
Discussion
Learning to review journal manuscripts represents a self-guided journey, typically involving minimal mentoring and support, with little or no explicit guidelines (Lu, 2012). Despite the fact that a significant proportion of academic faculty members formally review journal manuscripts, and despite the increasing body of literature describing aspects of the peer-review process, it is surprising that few standards and criteria have emerged that characterize an optimal review. Consistent with our assertion here, Lu (2012) made the following observation: ‘What a good review is’ is not an easy question to answer, considering in the literature there is a lack of clarity among scholars in articulating ‘what high-quality or good research might be’. As a management journal editor claimed, peer review is inherently subjective because judgements about knowledge are filtered through a personal lens, which ‘alters individual referees’ understanding and shapes their thinking in an idiosyncratic fashion’. However, we know only snippets about quality judgements; for example, there is high agreement between reviewers when it comes to rejection and in identifying a paper of high quality. But such studies rarely went deeper. (p. 56)
As can be seen from the findings from the MMCS presented in this editorial, together, the reviewers offered the authors feedback that was greater than the sum of the individual reviews—for which we are extremely grateful to the reviewers. The major finding yielded from these 45 special issue reviewers is the emergence of the following six meta-themes (and 32 themes) that characterized their criticisms: lack of warrantedness (i.e., containing 7 themes), lack of justification (i.e., containing 6 themes), writing issues (i.e., containing 6 themes), lack of transparency (i.e., containing 6 themes), lack of integration (i.e., containing 5 themes), and philosophical issues (i.e., containing 2 themes). The utility of this finding is that these six meta-themes and their associated 32 themes may be used to inform reviewer sheets that contain criteria used by reviewers to assess mixed methods research manuscripts submitted to journals for review for possible publication. This emergent thematic structure could lead to the development of up to 32 assessment items for reviewers of mixed methods research manuscripts such as the reviewer sheet that we have developed from this structure and have presented in the appendices (see Appendix A for partial reviewer sheet and Appendix B for a list of the 32 items). It can be seen from this reviewer sheet that all 32 items are stated in a positive direction as a means of rewarding authors for containing these elements (i.e., positive reinforcement) rather than penalizing authors for omitting these components (i.e., punishment). We believe that this reviewer sheet not only is potentially useful for reviewers by providing them with explicit items that characterize a quality manuscript to evaluate as well as for their editors who, subsequently, would be the recipients of quality reviews but also is potentially useful for authors of mixed methods research manuscripts by providing them with explicit guidelines for developing these manuscripts. Further, we believe that such a reviewer sheet would be helpful for college-level instructors of mixed methods research courses, mentors, advisors, thesis/dissertation chairs/supervisors and other committee members, as well as authors of future mixed methods textbooks and other mixed methods works, and even writers of future editions of style guides such as the APA Publication Manual. Importantly, we expect that this reviewer sheet also would be helpful for mixed methods practitioners who can be defined as those “with theoretical and practical knowledge of three methodologies (i.e., qualitative, quantitative, and mixed methods)” (Poth, 2012, p. 315).
Although our reviewer sheet stems from the voices of our 45 talented reviewers, and these voices indubitably have led to what we deem as 16 quality published articles across our two IJQM special issues, this reviewer sheet (and any other sheets that subsequently are developed from the present or future findings) should be subjected to validation/legitimation studies. In particular, future research is needed to determine the quality of reviews that our newly developed reviewer sheet generates. Once validated/legitimated, we contend that such an evidence-based reviewer sheet should make it easier for reviewers to provide what Fischer (2011, p. 227) refers to as value-added reviews. Such improvement in quality of reviews, in turn, would improve the quality of mixed methods research articles. And little is more crucial to the advancement of both science and the field of mixed methods research than the establishment of optimal reviews via the information gatekeepers—in this case, reviewers of mixed methods manuscripts.
Footnotes
Appendix A
First Page of 32-Item Reviewer Sheet.
| Warrantedness | |||
| For each of the following statements, please indicate the extent to which you agree or disagree, according to the scale below. | |||
| 1 = strongly disagree 2 = disagree 3 = agree 4 = strongly agree | |||
|
There is adequate support claims and findings made that stem from the extant literature. |
|||
| 1 | 2 | 3 | 4 |
| Comment: | |||
|
The findings are presented clearly and adequately. |
|||
| 1 | 2 | 3 | 4 |
| Comment: | |||
|
The sources used are both adequate and sufficiently recent. |
|||
| 1 | 2 | 3 | 4 |
| Comment: | |||
|
All new or unconventional terms and concepts are adequately defined. |
|||
| 1 | 2 | 3 | 4 |
| Comment: | |||
|
Adequate reference is made to the most recent mixed methods research literature. |
|||
| 1 | 2 | 3 | 4 |
| Comment: | |||
|
The manuscript does not contain any reference list errors. |
|||
| 1 | 2 | 3 | 4 |
| Comment: | |||
|
The manuscript does not contain any citation error. |
|||
| 1 | 2 | 3 | 4 |
| Comment: | |||
Appendix B
List of 32 Items on Reviewer Sheet.
| Warrantedness |
|
There is adequate support claims and findings made that stem from the extant literature. The findings are presented clearly and adequately. The sources used are both adequate and sufficiently recent. All new or unconventional terms and concepts are adequately defined. Adequate reference is made to the most recent mixed methods research literature. The manuscript does not contain any reference list errors. The manuscript does not contain any citation error. |
| Justification |
|
The manuscript is sufficiently developed. The educational significance is stated clearly. The manuscript adequately advances qualitative/quantitative research. The manuscript adequately advances mixed methods research. Both the rationale of the study and rationale for mixing quantitative and qualitative approaches are stated clearly. The purpose statement is stated clearly |
| Writing quality |
|
The writing style of the manuscript is clear. The manuscript does not contain an inconsistent information. The manuscript contains an appropriate structure. The manuscript does not contain any repetition. The manuscript contains appropriate word usage throughout. The manuscript contains appropriate transitions between sentences and paragraphs. |
| Transparency |
|
All the most important procedures are described clearly (if applicable). The sampling design, sampling scheme(s), sample size, and size of all observational units are specified clearly (if applicable). The analysis is described clearly (if applicable). All tables and figure are referred to and explained sufficiently (if applicable). Directions for future research are provided adequately. A clear and substantive conclusion is provided. |
| Integration |
|
The mixed methods research framework is presented clearly and adequately Each research question is linked explicitly to the research design used. An appropriate mixed methods research design is used The application of the manuscript is presented clearly and adequately (if applicable) Discussion of the validity/legitimation issues is presented clearly and adequately. |
| Philosophical lens |
|
All research philosophies that are presented are labeled accurately and adequately. The research philosophy used by the researcher/author is specified and described clearly and accurately. |
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.