Sage Journals: Discover world-class research

Abstract

Arguably, quality assurance is more complicated in qualitative studies than in quantitative studies. Several procedures for quality assurance are available, among which the audit procedure as proposed by Akkerman, Admiraal, Brekelmans, and Oost. In this article, we reflect on this procedure based on our own experiences as well as based on a review of studies in which the audit procedure was employed. More specifically, we discuss (1) the choice for an auditor and the relationship between auditee and auditor and (2) the function of the audit. We propose that future auditees (a) explicitly report on the auditee–auditor relationship, (b) explicitly report on the function of their audit, and (c) have their audit trail documents available for review. With this methodological position paper, we aim to contribute to the current call to make social science studies and their conclusions more transparent and thereby to enhance the quality of qualitative studies.

Keywords

audit quality procedure qualitative methods qualitative approach qualitative data analysis

What Is Already Known?

This article provides a review of the audit procedure (a method for quality assurance) as it has been used by researchers in the last decade, and extends the procedure by suggesting ways to increase transparency of the audit procedure.

What This Paper Adds?

With this methodological position paper, we aim to contribute to the current call to make social science studies and their conclusions more transparent and thereby to enhance the quality of qualitative studies.

Introduction

In scientific studies, researchers aim to develop knowledge or theories by formulating research questions or hypotheses and systematically collecting and analyzing data that are relevant to the research questions or hypotheses. Based on these analyses, conclusions are drawn in terms of answers to the research questions or in terms of confirming or rejecting hypotheses. Historically, within scientific studies, a distinction between quantitative and qualitative methods is made (e.g., Neuenschwander, 2013). Even though the functionality of this distinction is disputed by some (e.g., Allwood, 2012; Sale, Lohfeld, & Brazil, 2002), when comparing quantitative to qualitative studies, it is generally considered more difficult to ensure the validity of conclusions in qualitative studies given the lack of standardized strategies and procedures and the relatively large role of the researchers’ interpretation in drawing conclusions (Anfara, Brown, & Mangione, 2002; Chowdhury, 2015; Finlay, 2002; Weiner-Levy & Popper-Giveon, 2013). Several methods or strategies have been put forward to overcome this difficulty (e.g., Koch, 2006; Lietz, Langer, & Furman, 2006; Morse, Barrett, Mayan, Olson, & Spiers, 2002; Shenton, 2004; Stige, Malterud, & Midtgarden, 2009), among which is the audit procedure set out by Akkerman, Admiraal, Brekelmans, and Oost (2008). This procedure is aimed at ensuring the quality, mainly of qualitative scientific research in terms of visibility (Are decisions explicated and communicated?), comprehensibility (Are these decisions substantiated?), and acceptability (Are the decisions acceptable according to the standards, values, and norms in the particular research domain?). The audit procedure offers researchers (the auditee) a comprehensive methodology to leave a trail of choices and decisions made, which can subsequently be checked by a second party (the auditor).

As the audit procedure appeared to be the most specific and elaborate procedure for ensuring quality in qualitative studies, together with the second author of this article (who acted as auditor), I carried out the procedure as was set out by Akkerman et al. (2008; as auditee) while conducting a study in the domain of educational sciences (De Kleijn, Meijer, Brekelmans, & Pilot, 2015). While the audit process was rather time-consuming, indeed in our view it has substantially increased the quality and transparency of the study’s conclusions. The article is structured as follows: We (a) briefly summarize the audit procedure, (b) describe the two issues we encountered with the audit procedure and how other researchers dealt with them, (c) describe the setting of our own study and the choices we made concerning the audit procedure, and (d) provide additional guidelines for conducting and reporting on the audit procedure, based on our experiences. With this article, our goal is to provide new insights into the audit procedure and to offer recommendations to increase its value, which could expand our understanding and use of the audit procedure. We thereby aim to contribute to the debate about ensuring the validity of qualitative studies in the social sciences.

Brief Description of the Audit Procedure

The audit procedure (Akkerman, Admiraal, Brekelmans, & Oost, 2008) provides guidelines for the auditee, who keeps an audit trail of the research study, and for the auditor, who checks the research for visibility, comprehensibility, and acceptability. The main aim of the procedure is “to link interpretations and conclusions in the final research document to the data, through all steps in-between” (Akkerman et al., 2008, p. 270). It is a further elaboration of the audit procedure for naturalistic research as set out by Halpern (1983), Lincoln and Guba (1985), and Schwandt and Halpern (1988). The elaboration entails a specification of the object of the audit procedure, namely, the process of data gathering and the process of data analyses, with a clear distinction between both. Also, Akkerman et al. (2008) argued that their approach is not only useful for naturalistic research but for all kinds of complex research.

To engage in the audit procedure, the auditee creates an audit trail, which documents the entire research process including the processes of data gathering and data analysis. The creation of the audit trail thus starts at an early stage in the research process and includes the following five components: (1) a start document in which the conceptual framework, planned methods, and expected results are presented as well as a reflection on the researchers’ position in the study, (2) a final document, such as the journal article, (3) the raw data, (4) the processed data, such as coded records or summaries, and (5) a process document that covers a systematic report of the actions undertaken and the associated results in terms of data gathering and analysis (Akkerman et al., 2008).

The audit procedure then consists of seven stages. The first stage is an orientation to the audit procedure, in which the auditee and the auditor negotiate and agree upon goals, roles, and rules of the audit. The second stage concerns an orientation to the study, in which the auditee explains the audit trail in order for the auditor to become familiar with the study. The third stage is determining the auditability of the study, in which the auditor determines the completeness, comprehensibility, and utility of the audit trail. The fourth stage concerns negotiating the contract. Auditee and auditor establish a time line, the format of the outcomes, and the criteria for renegotiation. The fifth stage is the actual assessment stage, in which the auditor assesses the research process in terms of the criteria visibility, comprehensibility, and acceptability. During the sixth stage of renegotiation, the auditor presents the findings and discusses discrepancies with the auditee. In this stage, the auditee and auditor determine what actions will follow, for instance, redesigning the research process, adjusting the auditor report, or modifying the agreement. The seventh and final stage is the auditor writing the final auditor report on the quality of the study.

With respect to the auditor report, Akkerman et al. (2008) described three main components. First, the report should contain a summary of the specific audit procedure. The second component is a description of the quality of the study with respect to visibility, comprehensibility, and acceptability. Third, an assessment scheme is included (see Table 1), in which the conclusions of the auditor are systematically summarized.

Table 1.
Assessment Scheme as Proposed by Akkerman et al. (2008).

Audit Trail Components Quality

Visibility Comprehensibility Acceptability

Data gathering Planned

Realized

Data analysis Planned

Realized

Choosing to Perform the Audit Procedure

Value of performing the audit procedure

Several researchers who have applied the audit procedure have reflected on its benefits. For instance, the importance of the audit procedure for enhancing transparency and acceptability (Akkerman, Bronkhorst, & Zitter, 2013) and for enhancing the quality of small-scale studies (Den Brok, Van Eerde, & Hajer, 2010) are acknowledged. Additionally, the advantages of the audit procedure for extending the focus from the quality of the research outcomes to quality of the research process (Reynolds et al., 2011) are described as well as the fact that the audit requires clarification and self-reflection on the researcher’s part (Carcary, 2009). In line with this, Strijbos and De Laat (2010), in reflecting on the results of their qualitative analyses, acknowledged that they could have applied the audit procedure to validate their findings.

However, with all the demands placed on the audit procedure and reporting about it, some have argued that the audit can become a whole study in itself (Efimova, 2009). For instance, Poortman and Schildkamp (2012) criticized the audit procedure because the enormous scope of an audit report results in a lack of transparency as readers of a research paper cannot judge the quality of the performed audit themselves. They argue that it is unacceptable that qualitative research on the one hand is “too complex to be judged by standardized methods or procedures,” but that on the other hand “readers should trust that the ‘auditor’ has made sure that the quality is in order” (Poortman & Schildkamp, 2012, p. 1730).

Although the audit procedure in itself indeed is a complex procedure, when it is followed through and reported on in a transparent way, it helps readers judge the value of the audit as a method of quality assurance. We therefore take the stance that conducting an audit procedure such as the one set out by Akkerman et al. (2008), is par excellence a way of developing standardized methods and procedures for quality assurance of qualitative studies. We do agree that transparency is highly important and therefore suggest that when the audit procedure is applied, authors have the documentation available online or upon request (e.g., Hawkins, Coffee, & Soundy, 2014; Poria, Reichel, & Brandt, 2010). This way readers can judge the quality of a study for themselves and also learn from it. Interestingly, this would be in line with the trend of open data in quantitative studies (Asendorpf et al., 2013; Wicherts & Bakker, 2012), since several studies have indicated that in experimental studies transparency and reproducibility are also often lacking (Bakker & Wicherts, 2011; Wicherts, Borsboom, Kats, & Molenaar, 2006).

In sum, even though conducting the audit procedure requires time and effort from both the auditee and auditor, most researchers acknowledge its value in terms of increasing and establishing quality in qualitative research.

What is missing in the description of the audit procedure?

Given the value of the audit procedure and the first author’s aim to conduct a qualitative study in a transparent and acceptable way, the first author decided to perform the audit procedure as a means to enhance the quality of her research study. Even though the audit procedure is documented thoroughly (Akkerman et al., 2008), two questions were encountered right at the start of performing the audit procedure. The first author had to decide who would act as auditor, and once an auditor was found (the second author of this article), they together had to decide at which point during the research study the audit procedure would take place, and thus what the specific role of the auditor would be. In our opinion, these two issues were open to multiple interpretations and remained unclarified. Therefore, to make a decision, we performed a review of how other researchers had answered these two questions. In How Other Researchers Have Used and Reported on the Audit Procedure section, we describe the results of this review. In Context of Our Own Study and Experiences With the Audit Procedure section, we describe the choices we made concerning how to perform the audit procedure.

How Other Researchers Have Used and Reported on the Audit Procedure

In this section, we address two specific aspects regarding the audit procedure: the choice of the auditor and the function of the audit procedure. For both, we address (a) the specific question we encountered and (b) how other researchers dealt with this issue. For part (b), we selected all research papers that referred to the paper by Akkerman et al. (2008) to indicate they had performed the audit procedure. We included peer-reviewed papers written in English that were published up to September 2016. This led to the identification of 26 articles, which are marked with an asterisk () in the reference list. These articles were read and summarized concerning the two specific aspects of the audit procedure outlined above.

The choice and role of an appropriate auditor

With respect to the auditor’s identity, Akkerman et al. (2008) suggested that “an auditor must be an expert in the research domain, has to be able to assess a study from a methodological perspective, and should have an independent opinion about the research” (Akkerman et al., 2008, p. 272). It makes absolute sense that the auditor should have knowledge of the domain and should be independent, but when we decided to perform the audit procedure we struggled with how “independent” the auditor can and should be. Akkerman et al. (2008) reflected on the auditor’s independence by saying that if the criteria of being an expert in the domain, understanding the methodological perspective, and having an independent opinion about the research are met, “supervisors of the researcher could act as auditor as they probably have time and are willing to evaluate the quality of the research” (Akkerman et al., 2008, p. 272). In our view however, a supervisor by definition does not have an independent opinion about the research as she or he was probably involved in the decision-making during data gathering and data analyses, and has a considerable stake in getting the study published as she or he would be a coauthor.

Reviewing the papers in which the audit procedure was applied, we found that some of them provided some information about the auditee–auditor relationship. Korthagen, Attema-Noordewier, and Zwart (2014), Van Kan, Ponte, and Verloop (2013), and Van Rijswijk, Akkerman, Schaap, and Van Tartwijk (2016) explained that the auditor was an independent researcher, and Dobber, Akkerman, Verloop, and Vermunt (2012), Baartman, Prins, Kirschner, and Van der Vleuten (2011), Ehlen, Van der Klink, and Boshuizen (2016), and Van der Wal, De Kraker, Kroeze, Kirschner, and Valkering (2016) specified that the auditor was an external researcher. On the other hand, Bronkhorst, Baartman, and Stokking (2012), Gallardo-Echenique, Marqués-Molías, Bullen, and Strijbos (2015), and Hawkins, Coffee, and Soundy (2014) reported that the second author was the auditor. Fluijt, Bakker, and Struyf (2016) indicated that two colleague researchers reviewed the quality of the study separately. Interestingly, Bulanda, Szarzynski, Siler, and McCrea (2013) explicitly stated that the “externality of auditors is no guarantee of authenticity, and an ‘insider’s perspective’ also has much to offer” (Bulanda, Szarzynski, Siler, & McCrea, 2013, p. 282). Bulanda et al. (2013) therefore involved three other researchers in the triangulation process in the audit procedure who also coauthored the paper. Lastly, some researchers gave a description of their analysis stages in their paper and seemed to believe that this constituted the audit trail of which the quality could be judged by the reader (Popadiuk, 2010; Poria et al., 2010). In the other papers that used the audit procedure, no mention was made of the auditor’s identity (e.g., Bronkhorst, Meijer, Koster, Akkerman, & Vermunt, 2013; Dobber, Akkerman, Verloop, Admiraal, & Vermunt, 2012; Poria et al., 2010; Poria & Beal, 2016; Van der Zande, 2010).

The function of the assessment and renegotiation stage

During the assessment stage of the audit procedure, the auditor assesses the quality of the study, after which the auditor presents the findings to the auditee and together they determine what actions will follow. With respect to these two stages, they can be interpreted to be intended to establish* whether the conclusions of the study were visible, comprehensible, and acceptable enough in the eyes of the auditor. Another interpretation would be that they are intended to actually increase visibility, comprehensibility, and acceptability of the study based on the auditor’s assessment.

In Akkerman et al.’s (2008) paper, we found support for both interpretations (i.e. establishing and increasing quality). For instance, the fact that the final document is suggested to be the starting point for the audit, which suggests that the document is not assumed to be changed based on the audit results. Also, Akkerman et al. (2008) wrote that “the auditor has to evaluate if these results and conclusions are in line with the (often invisible) processes of data gathering and analysis, and if these processes are acceptable according to general standards” (pp. 266–267), which also suggests establishing quality as the main goal. On the other hand, they also stated that depending on the agreements in the contract, “the auditor report may also include feedback about how to adjust the research process in case it is evaluated having too little quality,” which supports the interpretation of increasing validity.

Most of the 26 studies that applied the audit procedure seem to have done so for establishing the quality of the study. For instance, the only thing Poria, Reichel, and Brandt (2011) reported was that they conducted an audit trail to ensure the results’ credibility. Several other studies also indicated that the auditor concluded, assessed, or judged that the study met quality criteria for scientific research such as reliability, validity, visibility, comprehensibility, and acceptability (Akkerman, Admiraal, & Simons, 2012; Akkerman, Admiraal, Simons, & Niessen, 2006; Dobber, Akkerman, Verloop, Admiraal, et al., 2012; Dobber, Akkerman, Verloop, & Vermunt, 2012; Koopman, Bakx, & Beijaard, 2014; Leeferink, Koopman, Beijaard, & Ketelaar, 2015; Swinkels, Koopman, & Beijaard, 2013; Van der Zande, 2010; Van Kan, Ponte, & Verloop, 2013). Interestingly, Van der Zande (2010) explicitly mentioned the use of a “summative” audit (p. 120).

On the other hand, some studies also reported how the audit procedure had impacted the final results and conclusions, suggesting the use of the audit procedure for increasing its quality (Baartman, Prins, Kirschner, & Van der Vleuten, 2011; Bronkhorst, Baartman, & Stokking, 2012; Bronkhorst, Meijer, Koster, & Vermunt, 2011). The authors of these studies indicated that the audit procedure revealed small differences between the auditee and the auditor, which in turn led to minor alterations in the analyses on which both the auditee and auditor agreed (Baartman et al., 2011; Bronkhorst et al., 2012). In one case (Bronkhorst et al., 2011), the audit report was used as a starting point for the analyses and the discussion section.

Context of Our Own Study and Experiences With the Audit Procedure

Context of our study

The study for which our audit was performed concerned the question how thesis supervisors in higher education adapt their supervision to master students’ needs (De Kleijn et al., 2015). It was an exploratory, qualitative study spanning several months, in which group discussion meetings, individual interviews, and logbooks were used as instruments. In the sections below, we describe the choices we made regarding the choice of auditor and the function of the audit.

The choice and role of an appropriate auditor

The audited study was part of the PhD project of the first author and thus carried out with guidance from a supervision team. The choice for an auditor was guided by two principles, namely, that the auditor had to be someone with expertise on the topic of the study, and that the auditor had an independent opinion that was not jeopardized by for example being a part of the supervising team of the auditee. The auditee asked the second author to be the auditor based on her research in a related field of study, namely, teacher supervision of collaborative student activities in secondary education (e.g., see Van Leeuwen, Janssen, Erkens, & Brekelmans, 2013). The auditor was a colleague of the auditee within the same research department. One could argue that working in the same department does not ensure an independent opinion, and we contemplated this question before starting the audit procedure. We decided that given the facts that the auditor was not involved in the supervision of the auditee’s PhD project, and that the topics of their research were related yet distant enough for the auditor to be a critical reader, the auditor would be independent enough to take on this role.

During the auditing process, we (i.e., auditor and auditee) experienced that the auditor played an important role in quality improvement of the research paper (see The function of the audit procedure subsection). The auditor read into the study, critically reflected on the analyses, and came up with suggestions for improvement, which the auditee in turn followed. By doing so, the auditor fulfilled nearly the same tasks as would a second or third author. No agreements had been made beforehand concerning a possible coauthorship. During the audit procedure, the auditee reasoned that adding the auditor as a coauthor could be justified by the substantial contribution the audit procedure made to the paper’s quality and that coauthorship would do the auditor justice for her time and effort. On the other hand, even though the auditor had no expectation of coauthorship and this coauthorship came into question only after we successfully completed the audit, we anticipated that it would jeopardize the audit’s objectivity in the reviewers’ and readers’ eyes. After all, adding the auditor as a coauthor would mean that the auditor has a stake in getting the study published, which could have influenced her judgment of the study’s quality. For this reason, we eventually decided not to include the auditor as coauthor.

The function of the audit procedure

Concerning the function of the audit procedure, as explained one of our earliest questions was when to start. Following the seven stages of the audit procedure (Akkerman et al., 2008), we started by negotiating about the goals, roles, and rules of the audit. The auditee approached the auditor when she had recently finished data collection. This gave us the opportunity to not just establish the validity of the study, but also to increase its validity by using the auditor’s feedback for the further stages of analyses and reporting on the study. We made agreements to audit the research process at three points, namely, after data gathering, after data analysis, and when the whole study was written down, specifically focusing on the conclusions that were drawn from the study. At every stage, the auditee prepared the audit trail up to that point, the auditor assessed these documents, the outcomes of the assessment were discussed, the auditee processed the feedback, and the auditor wrote down the final auditor report. In other words, we followed the stages of the audit procedure in three consecutive cycles.

By means of the auditor’s comments, the auditee was led to critically appraise the choices she made and to elaborate on the chosen methodology. The auditor reports led to concrete points not only for enhancing the transparency of the final manuscript but also to suggestions for enhanced analyses of the data. In particular, there was a large role in the study for the concept of adaptivity, which means thesis supervisors adjust their strategies according to specific student characteristics. A recurring point of discussion between auditor and auditee was whether or not to separate how supervisors monitor student characteristics and the subsequent supervising strategies they employ. The fact that the auditor was a researcher in a related field helped to shape constructive discussions about how to proceed with the analyses. In the final stage of auditing, the manuscript of the performed study was judged to be sufficient concerning its visibility, comprehensibility, and acceptability. The final audit trail of the study included all of the elements mentioned above, for example the initial negotiation between auditee and auditor, as well as all the materials for each of the three stages of auditing. In the published manuscript (De Kleijn et al., 2015), it is stated that the audit trail is openly available to readers upon request.

Additional Guidelines for Conducting and Reporting on the Audit Procedure

In the sections below, based on what other researchers have reported as well as based on our own experiences with the audit procedure, we present our reflections on and recommendations for the choice of the auditor and the function of the audit procedure.

Choosing the auditor

In our review of studies, a number of examples were encountered of auditee–auditor relationships; the auditor could be an external researcher, a collaborating researcher, or a more distant yet direct colleague (as in our case). We think two aspects are important for choosing an auditor. First of all, the auditor should have a minimal level of knowledge or expertise concerning the study’s topic, enabling the auditor for example to judge the suitability of the methodological choices.

The other aspect is that the auditor should have an adequate measure of objectivity. Because this is sometimes a difficult criteria to assess (as it was in our case), we suggest that when choosing an auditor, researchers may ask themselves whether the designated person is able to engage in critical reflexivity (Creswell & Miller, 2000) of the study that is being audited. This means the auditor should be someone who is able to pinpoint and question the auditee’s assumptions, beliefs, and biases that may have shaped their decisions during the research process. This criterion assumes the auditor is knowledgeable yet objective enough to challenge the auditee’s choices, leading to a valid decision on the study’s visibility, comprehensibility, and acceptability. In our opinion, the suggestion by Akkerman et al. (2008) that a member of the supervision team can act as auditor is therefore questionable, since the supervision team usually shapes the assumptions and beliefs the auditee uses to conduct his or her studies. Although both the auditor and the supervising team provide feedback to the auditee, the core difference is that the judges the appropriateness and transparency of the choices that are made, whereas the supervising team also shapes the choices themselves. Thus, although the goals of the auditor and the supervising team overlap—to ensure authenticity and trustworthiness of the study—their roles in the process differ.

Another question related to the choice of an auditor is whether the auditor should be included as coauthor of a paper once it has been audited and submitted for publication. In our opinion, coauthorship of the auditor can in some cases be defendable, as long as the documentation of the audit trail would make clear what the auditor’s input was. Another possibility would be to separate the audit’s formative function from its summative function by involving two different auditors for each function, or to transfer the summative phase to journal reviewers. Authors could then, along with their paper, submit the audit documents for the reviewer to establish whether the analyses and conclusions are indeed visible, comprehensible, and acceptable. However, this would increase the work load of voluntary reviewers even further, which might not be desirable. Therefore, open-access audit documents might be suitable as well, so that all readers can take on the auditor role and review the auditee’s documents. This would be in line with the general call for more transparency in social science research and review processes (Wicherts, Kievit, Bakker, & Borsboom, 2012).

Our experience with the audit procedure at least showed that, especially when it concerns a formative audit, it is good for future applicants of the audit procedure to be aware of the large role or involvement of the auditor. We therefore think it is wise to establish agreements about possible coauthorship right at the beginning of the audit procedure, which in hindsight we also should have done ourselves.

For future applications of the audit procedure, we suggest that researchers explicitly address their relationship with the auditor, the arguments for choosing the specific auditor, and the role the auditor fulfilled. Doing so would enable reviewers and readers to judge the appropriateness and objectivity of the auditor.

The function of the audit procedure

As noted in The function of the assessment and renegotiation stage subsection, the audit procedure can and has indeed be interpreted to be intended for two separate goals: to establish whether the conclusions of a study are visible, comprehensible, and acceptable enough or to actually increase visibility, comprehensibility, and acceptability of the study based on the auditor’s assessment. In our view, the two interpretations of increasing or establishing quality can be linked to the concept of assessment, and more specifically to the distinction between formative and summative assessment. An assessment, no matter the object that is being assessed, is “a judgement that encapsulates all the evidence up to a given point” (Taras, 2005, p. 468). As explained by Taras (2005), both formative and summative assessments in essence boil down to making a judgment according to particular standards. The difference between them is that summative assessment is an evaluation or judgment that often incorporates an aspect of finality, for example, when the evaluation is used for the purposes of certification (Sadler, 1989), whereas formative assessment means that judgments about the quality are used to shape and improve the object that is under evaluation. Formative assessment includes feedback that points out the difference between the quality or level of the work being assessed and the standard to which it is compared, as well as suggestions for how the required standard may be achieved (Taras, 2005).

When translated to the audit procedure, we may speak of a formative audit and a summative audit procedure. For both of these, the audit procedure in essence is an evaluation of the study that is being audited at a certain time point, based on the evidence that the auditee provides to the auditor. The difference between the two functions is that a summative audit constitutes a judgment that the auditee can report in the study as evidence of its quality. A formative audit means the judgment of the auditor can be used to improve the study that is being audited, so the auditor’s judgment should contain feedback for reaching the standards of visibility, comprehensibility, and acceptability. In The choice and role of an appropriate auditor subsection, several examples were discussed where the audit procedure served a formative function. Bronkhorst, Meijer, Koster, and Vermunt (2011) even explicitly reported on their “formative” (p. 1123) use of the audit procedure; they used the auditor’s report as a starting point for their analyses and their discussion section.

Given the possibilities for increasing a study’s quality, we would prefer using the auditor report formatively during completion of a study to a strictly summative function of the audit after a study is carried out and reported. However, given practical restrictions in terms of time and money, researchers can also opt for a summative audit. Based on how other researchers used the audit procedure, our own experience with the audit procedure and the current call for more transparency in social science research, we would propose two additional guidelines for future auditors and auditees. First, we would recommend that the auditor and auditee discuss beforehand (i.e., during the orientation phase) whether the function of the audit will be summative and/or formative so that both parties know what will (not) be undertaken based on the auditor report. Second, we suggest that for the sake of transparency, auditees always explicitly report on the function of the audit procedure.

Reporting on the audit procedure

In the 26 papers we reviewed, the amount of information available about the audit procedure varied considerably. The audit procedure is meant to increase the validity of a qualitative study, but this can only be the case when it is clear how the audit procedure was performed. In other words, the description of the audit procedure should be transparent and not solely rely on the reader’s willingness to trust the authors or the auditor (Morse et al., 2002; Poortman & Schildkamp, 2012). This means that certain decisions and characteristics concerning the audit procedure should be explicitly addressed in qualitative research papers. We therefore propose that future auditees have their audit documentation available online or upon request, that they explicitly address their relation to the auditor and the arguments they used to select their auditor, and that they explicitly address the function of their audit.

As we take the position that it is important to keep refining and developing procedures to assure the quality of qualitative studies, in Table 2, we provide a template that could be incorporated into the methods section of research papers that employ the audit procedure.

Table 2.
Suggested Audit Table to Be Included in Studies That Have Performed an Audit Procedure.

Audit Characteristics

Relation with the auditor^a Coauthor/internal/external

Arguments supporting the auditor’s expertise and independence (open field)

Function of the audit Formative^b/summative/both

^aIn case of multiple auditors (e.g., a formative and summative auditors), both relationships can be described. ^bIn case of formative auditing, we would also suggest briefly reflecting on the adjustment to the study based on the audit procedure.

Conclusion

In this methodological position paper, our goal was to provide new insights into the audit procedure and to offer recommendations to increase its value. Based on our experience with the audit procedure and a review of studies in which the audit procedure was applied, we addressed two issues that might need more attention in future studies: the choice for and role of the auditor and the formative and/or summative function of the audit. With this article, we hope to contribute to the current call to enhance the quality of studies in social science by making their conclusions more transparent. Our suggestions help achieve this goal because (a) researchers are encouraged to consider the quality of their work in the process of data gathering and data analyses, (b) journal reviewers can use the audit documents as part of their review of qualitative papers, and (c) readers can request the audit documents and judge for themselves whether they agree with the auditor’s judgment and the auditee’s decisions.

Audit Characteristics
Relation with the auditor^a	Coauthor/internal/external
Arguments supporting the auditor’s expertise and independence	(open field)
Function of the audit	Formative^b/summative/both

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Akkerman

Admiraal

Brekelmans

Oost

(2008). Auditing quality of research in social sciences. Quality & Quantity, 42, 257–274.

*Akkerman

Admiraal

Simons

R. J.

(2012). Unity and diversity in a collaborative research project. Cultural Psychology, 18, 227–252.

*Akkerman

Admiraal

Simons

R. J.

Niessen

(2006). Considering diversity: Multivoicedness in international academic collaboration. Cultural Psychology, 12, 461–485.

Akkerman

Bronkhorst

L. H.

Zitter

(2013). The complexity of educational design research. Quality & Quantity, 47, 421–439.

Allwood

C. M.

(2012). The distinction between qualitative and quantitative research methods is problematic. Quality & Quantity, 46, 1417–1429.

Anfara

V. A.

Brown

K. M.

Mangione

T. L.

(2002). Qualitative analysis on stage: Making the research process more public. Educational Researcher, 31, 28–38.

Asendorpf

J. B.

Conner

De Fruyt

De Houwer

Denissen

J. J. A.

Fiedler

… Wicherts

J. M.

(2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–119.

*Baartman

L. K.

Prins

F. J.

Kirschner

P. A.

Van der Vleuten

C. P.

(2011) Self-evaluation of assessment programs: A cross-case analysis. Evaluation and Program Planning, 34, 206–216.

Bakker

Wicherts

J. M.

(2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43, 666–678.

10.

*Bronkhorst

L. H.

Baartman

L. K.

Stokking

K. M.

(2012). The explication of quality standards in self-evaluation. Assessment in Education: Principles, Policy and Practice, 19, 357–378.

11.

*Bronkhorst

L. H.

Meijer

P. C.

Koster

Akkerman

Vermunt

J. D.

(2013). Consequential research designs in research on teacher education. Teaching and Teacher Education, 33, 90–99.

12.

*Bronkhorst

L. H.

Meijer

P. C.

Koster

Vermunt

J. D.

(2011). Fostering meaning-oriented learning and deliberate practice in teacher education. Teaching and Teacher Education, 27, 1120–1130.

13.

* Bulanda

J. J.

Szarzynski

Siler

McCrea

K. T.

(2013). ‘Keeping it real’: An evaluation audit of five years of youth-led program evaluation. Smith College Studies in Social Work, 83, 279–302.

14.

Carcary

(2009). The research audit trial – Enhancing trustworthiness in qualitative inquiry. The Electronic Journal of Business Research Methods, 7, 11–24.

15.

Chowdhury

M. F.

(2015). Coding, sorting and sifting of qualitative data analysis: Debates and discussion. Quality & Quantity, 49, 1135–1143.

16.

Creswell

J. W.

Miller

D. L.

(2000). Determining validity in qualitative inquiry. Theory into Practice, 39, 124–130.

17.

De Kleijn

R. A. M.

Meijer

P. C.

Brekelmans

Pilot

(2015). Adaptive research supervision – Exploring expert thesis supervisors’ practical knowledge. Higher Education Research and Development, 34, 117–130.

18.

Den Brok

Van Eerde

Hajer

(2010). Classroom interaction studies as a source for teacher competencies: The use of case studies with multiple instruments for studying teacher competencies in multicultural classes. Teachers and Teaching: Theory and Practice, 16, 717–733.

19.

*Dobber

Akkerman

Verloop

Admiraal

Vermunt

J. D.

(2012). Developing designs for community development in four types of student teacher groups. Learning Environments Research, 15, 279–297.

20.

*Dobber

Akkerman

Verloop

Vermunt

J. D.

(2012). Student teachers’ collaborative research: Small-scale research projects during teacher education. Teaching and Teacher Education, 28, 609–617.

21.

Efimova

(2009). Passion at work: Blogging practices of knowledge workers. PhD thesis, University of Twente, Enschede, the Netherlands.

22.

*Ehlen

C. G.

Van der Klink

M. R.

Boshuizen

H. P.

(2016). Unravelling the social dynamics of an industry–school partnership: Social capital as perspective for co-creation. Studies in Continuing Education, 38, 61–85.

23.

Finlay

(2002). Negotiating the swamp: The opportunity and challenge of reflexivity in research practice. Qualitative Research, 2, 209–230.

24.

*Fluijt

Bakker

Struyf

(2016). Team-reflection: The missing link in co-teaching teams. European Journal of Special Needs Education, 31, 187–201.

25.

*Gallardo-Echenique

E. E.

Marqués-Molías

Bullen

Strijbos

J. W.

(2015). Let’s talk about digital learners in the digital era. The International Review of Research in Open and Distributed Learning, 16, 156–178.

26.

Halpern

E. S.

(1983). Auditing naturalistic inquiries: The development and application of a model. PhD thesis, Indiana University, Bloomington.

27.

*Hawkins

Coffee

Soundy

(2014). Considering how athletic identity assists adjustment to spinal cord injury: A qualitative study. Physiotherapy, 100, 268–274.

28.

Koch

(2006). Establishing rigour in qualitative research: The decision trail. Journal of Advanced Nursing, 53, 91–100.

29.

*Koopman

Bakx

Beijaard

(2014). Students’ goal orientations and learning strategies in a powerful learning environment: A case study. Studies in Educational Evaluation, 43, 186–196.

30.

*Korthagen

F. A.

Attema-Noordewier

Zwart

R. C.

(2014). Teacher–student contact: Exploring a basic but complicated concept. Teaching and Teacher Education, 40, 22–32.

31.

*Leeferink

Koopman

Beijaard

Ketelaar

(2015). Unraveling the complexity of student teachers’ learning in and from the workplace. Journal of Teacher Education, 66, 334–348.

32.

Lietz

C. A.

Langer

C. L.

Furman

(2006). Establishing trustworthiness in qualitative research in social work implications from a study regarding spirituality. Qualitative Social Work, 5, 441–458.

33.

Lincoln

Y. S.

Guba

E. G.

(1985). Naturalistic inquiry. Newbury Park, CA: Sage.

34.

Morse

J. M.

Barrett

Mayan

Olson

Spiers

(2002). Verification strategies for establishing reliability and validity in qualitative research. International Journal of Qualitative Methods, 1, 13–22.

35.

Neuenschwander

(2013). Qualitas and quantitas: Two ways of thinking in science. Quality & Quantity, 47, 2597–2615.

36.

Poortman

C. L.

Schildkamp

(2012). Alternative quality standards in qualitative research? Quality & Quantity, 46, 1727–1751.

37.

*Popadiuk

N. E.

(2010). Asian international student transition to high school in Canada. The Qualitative Report, 15, 1523–1548.

38.

*Poria

Beal

(2016, 4). An exploratory study about obese people’s flight experience. Journal of Travel Research. Epub ahead of print. doi:10.1177/0047287516643416

39.

*Poria

Reichel

Brandt

(2010). The flight experiences of people with disabilities: An exploratory study. Journal of Travel Research, 49, 216–227.

40.

*Poria

Reichel

Brandt

(2011). Dimensions of hotel experience of people with disabilities: An exploratory study. International Journal of Contemporary Hospitality Management, 23, 571–591.

41.

Reynolds

Kizito

Ezumah

Mangesho

Allen

Chandler

(2011). Quality assurance of qualitative research: A review of the discourse. Health Research Policy and Systems, 9, 1–10.

42.

Sadler

D. R.

(1989). Formative assessment and the design of instructional systems. Instructional Science, 18, 119–144.

43.

Sale

J. E.

Lohfeld

L. H.

Brazil

(2002). Revisiting the quantitative-qualitative debate: Implications for mixed-methods research. Quality & Quantity, 36, 43–53.

44.

Schwandt

T. A.

Halpern

E. S.

(1988). Linking auditing and metaevaluation: Enhancing quality in applied research. Thousand Oaks, CA: Sage.

45.

Shenton

A. K.

(2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for Information, 22, 63–75.

46.

Stige

Malterud

Midtgarden

(2009). Toward an agenda for evaluation of qualitative research. Qualitative Health Research, 19, 1504–1516.

47.

Strijbos

J. W.

De Laat

M. F.

(2010). Developing the role concept for computer-supported collaborative learning: An explorative synthesis. Computers in Human Behavior, 26, 495–505.

48.

*Swinkels

M. F. J.

Koopman

Beijaard

(2013). Student teachers’ development of learning-focused conceptions. Teaching and Teacher Education, 34, 26–37.

49.

Taras

(2005). Assessment—summative and formative—some theoretical reflections. British Journal of Educational studies, 53, 466–478.

50.

*Van der Wal

M. M.

De Kraker

Kroeze

Kirschner

P. A.

Valkering

(2016). Can computer models be used for social learning? A serious game in water management. Environmental Modelling Software, 75, 119–132.

51.

*Van der Zande

(2010, 12 2–3). Empowering teachers to teach socioscientific issues: The role of teacher identity in teaching. In Boerwinkel

D.J.

Waarlo

A.J.

, Proceedings of the Second Invitational Workshop on Genomics Education. Paper presented at Genomics for Decision Making, Utrecht (117–124). Utrecht: CD-B Press.

52.

*Van Kan

C. A.

Ponte

Verloop

(2013). How do teachers legitimize their classroom interactions in terms of educational values and ideals? Teachers and Teaching, 19, 610–633.

53.

Van Leeuwen

Janssen

Erkens

Brekelmans

(2013). Teacher interventions in a synchronous, co-located CSCL setting: Analyzing focus, means, and temporality. Computers in Human Behavior, 29, 1377–1386.

54.

*Van Rijswijk

Akkerman

Schaap

Van Tartwijk

(2016). Past perceptions and future expectations: Sensed dis/continuity at the start of teacher education. Teaching and Teacher Education, 58, 99–108.

55.

Weiner-Levy

Popper-Giveon

(2013). The absent, the hidden and the obscured: Reflections on ‘dark matter’ in qualitative research. Quality & Quantity, 47, 2177–2190.

56.

Wicherts

J. M.

Bakker

(2012). Publish (your data) or (let the data) perish! Why not publish your data too? Intelligence, 40, 73–76.

57.

Wicherts

J. M.

Borsboom

Kats

Molenaar

(2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61, 726–728.

58.

Wicherts

J. M.

Kievit

R. A.

Bakker

Borsboom

(2012). Letting the daylight in: Reviewing the reviewers and other ways to maximize transparency in science. Frontiers in Computational Neuroscience, 6, 1–9.

Audit Trail Components		Quality
Audit Trail Components		Visibility	Comprehensibility	Acceptability
Data gathering	Planned
	Realized
Data analysis	Planned
	Realized

Reflections and Review on the Audit Procedure

Abstract

Keywords

What Is Already Known?

What This Paper Adds?

Introduction

Brief Description of the Audit Procedure

Choosing to Perform the Audit Procedure

Value of performing the audit procedure

What is missing in the description of the audit procedure?

How Other Researchers Have Used and Reported on the Audit Procedure

The choice and role of an appropriate auditor

The function of the assessment and renegotiation stage

Context of Our Own Study and Experiences With the Audit Procedure

Context of our study

The choice and role of an appropriate auditor

The function of the audit procedure

Additional Guidelines for Conducting and Reporting on the Audit Procedure

Choosing the auditor

The function of the audit procedure

Reporting on the audit procedure

Conclusion

Footnotes

Declaration of Conflicting Interests

Funding

References