The Case of Contradictions: How Prolonged Engagement,Reflexive Journaling,and Observations can Contradict Qualitative Methods

Abstract

Interviews and focus groups are common methods for conducting qualitative research. They provide in-depth descriptions of a phenomenon by listening to and analyzing participants' words. Yet, when a researcher immerses themself in an environment for a prolonged period participants are able to act and think naturalistically rather than in a formal interview setting. Prolonged engagement allows the researcher to blend into the participants' environment, therefore increasing the likelihood that the activities that occur in the presence of the researcher do not differ from the activities that occur without the researcher present. This prolonged interaction can provide more in-depth data including observations and address questions of credibility, rather than only conducting an interview or focus group. However, the observations can also bring forward contradictions participants display that differ from the interview or focus group conversation. By spending 3 months in Ghana with the International Agricultural Education Fellowship Program conducting monitoring and evaluation efforts, using a mixed-methods approach, contradictions arose from observations and reflexive journaling that would not have been found without the prolonged engagement. The purpose of this study is to provide methodological insight by examining how contradictions from interviews and focus groups arise when the researcher is immersed in the participants' environment and highlight the importance of prolonged engagement, observations, and reflexive journaling to qualitative credibility.

Introduction

Interviewing and participant observation continue to be some of the most known representatives of qualitative research studies (Bogdan & Biklen, 1982; Foley et al., 2021; Green & Thorogood, 2014; Smit & Onwuegbuzie, 2018). The researcher imbeds themselves in the context of those they plan to study where they earn participants’ trust, come to know them personally, and use a detailed written record of what they hear and observe (Bogdan & Biklen, 1982; 2007). Qualitative research involving interviews and focus groups allows the researcher to capture, analyze, and understand participants' experiences through their perspectives (Denzin & Lincoln, 2005; Seidman, 2006).

In academic settings, interviewing is a qualitative approach frequently used in phenomenological research (Bhar, 2019). In phenomenological qualitative research, the participants provide a first-person description of their lived experience (Bhar, 2019). Guided by Heidegger (1927, 2011), the notion of examining the everyday experiences of participants as human beings in their social contexts can be explored through phenomenological research. All participants are part of a shared experience, although the researcher recognizes the same experience cannot be replicated (Merriam & Tisdell, 2016). In phenomenological interviews the researcher is responsible for ensuring the participant is at ease and sets an appropriate environment (McCracken, 1988; Thompson et al., 1989). Further, the ability to be reflexive must be at the forefront of the researcher’s mind to accurately explore and interpret the meaning of the phenomenon (Horrigan-Kelly et al., 2016). Although there are differing definitions of reflexivity, Olmos-Vega et al. (2022) incorporated the numerous descriptions into one definition, stating “reflexivity is a set of continuous, collaborative, and multifaceted practices through which researchers self-consciously critique, appraise, and evaluate how their subjectivity and context influence the research processes” (p. 2). Checking one’s preconceptions, or being reflexive, is an ongoing process and is vital when crafting the research question throughout the interview process and during data analysis (Horrigan-Kelly et al., 2016; Olmos-Vega et al., 2022).

One way to integrate a reflexive mindset is through prolonged engagement and thick description (Berger, 2015; Olaghere, 2022). When a researcher is able to have prolonged engagement with their participants the researcher develops a relationship and co-constructs meanings with said participants (Barusch et al., 2011). Prolonged engagement involves a sufficient amount of time to gain the appropriate representation of the full breadth of participants’ voices. Researchers are able to understand the culture on a deeper level and check for misinformation or anomalies (Glesne & Peshkin, 1992; Lincoln & Guba, 1985). Reflexivity is essential to represent participants according to their own conditions (Glesne, 1999; Watt, 2007). Prolonged engagement serves as great importance as interviewing and participant observation have received critiques on if participants’ views and actions are the same outside of the interview setting (Atkinson & Coffey, 2002; Hammersley, 2007).

An additional consideration to interviewing include Douglas’ (1976) four main problems in understanding social reality including misinformation, evasion, lies, and fronts. Douglas (1976) describes the importance of gaining trust and diving deeply into the direct personal experience of the participants’ world. By simply interviewing a participant, the representation they provide during this small exchange may or may not contain the whole truth of their social reality (Miller & Dingwall, 1997).

When combined, interviewing, participant observation, and prolonged engagement provide an opportunity to collect relevant and potentially valid qualitative data (Reeves et al., 2013). These common qualitative methods require the use of credibility and trustworthiness measures, such as previously recommended measures by Lincoln and Guba (1985), Kvale (1995), and Morse et al. (2002).

Credibility in Qualitative Methods

Credibility in qualitative research continues to increase in common research conversation and remains a central concept in methodological literature (Kvale, 1995; Lub, 2015). Therefore, various qualitative researchers address the question of credibility within qualitative research (Kvale, 1995; Lincoln & Guba, 1985; Morse et al., 2002). One of the most well-known works in qualitative credibility includes Lincoln and Guba’s (1985) development of trustworthiness in qualitative research (Lub, 2015). Their quality criteria include methods to address credibility, dependability, confirmability, transferability, and authenticity (Lincoln & Guba, 1985).

Credibility refers to the trustworthiness of the research’s findings as well as the conceptual evaluation of the original data and if readers believe the authors' interpretations (Kyngäs et al., 2020; Lincoln & Guba, 1985). Lincoln and Guba (1985) describe three activities to address credibility including prolonged engagement, persistent observation, and triangulation. Dependability concerns assessing the quality and consistency of the data collection, data analysis, and theory generation processes and determining if it is stable over time and under varying conditions (Kyngäs et al., 2020; Lincoln & Guba, 1985). Lincoln and Guba (1985) suggest the same audit of the process and product of research can be used for both dependability and confirmability.

Confirmability measures how well the research findings are supported, or confirmed, by the original data. The researcher examines if the findings are shaped by the respondents, or if the researcher’s interests or biases are reflected in the results (Kyngäs et al., 2020; Lincoln & Guba, 1985).

Transferability examines if the research findings can be applicable, or transferred, to other contexts and if results hold true to other settings or groups of participants (Kyngäs et al., 2020; Lincoln & Guba, 1985). This differs from generalizability, as generalizability requires a representative sample to generalize to the general population (Lincoln & Guba, 1985). Although the conventional form of transferability may include the expectation to have precise conclusions of external validity, in qualitative research transferability requires the researcher to provide a thick description so the interested reader can make conclusions about the possibility of a transfer (Lincoln & Guba, 1985).

Lastly, authenticity is defined as the extent that researchers fairly and sufficiently demonstrate a range of realities. The research involves various citations to connect the results and original data (Kyngäs et al., 2020; Lincoln & Guba, 1985). The strategies Lincoln and Guba (1985) suggest to address credibility, dependability, confirmability, transferability, and authenticity occur at various phases of the research process including throughout the inquiry, data collection, data coding and analysis, and during the presentation of the findings (Guba & Lincoln, 1994; Lincoln & Guba, 1985). Moreover, Lincoln (1995) added peer debriefing and member checking to the original list of activities to address credibility. Therefore, focusing on prolonged engagement, persistent observation, peer debriefing, triangulation, and member checking to evaluate the rigor of qualitative research, strengthen its credibility, and gain authentic data.

Other researchers suggest further credibility in qualitative research, including Kvale (1995), who outlined three approaches to credibility in the postmodern context including validation as investigation, communication, and action. The first approach, investigation, suggests credibility is a craft that includes questioning, checking, and theorizing throughout the phenomena (Kvale, 1995). Kvale (1995) described the second approach, communication, as putting an emphasis on observation and including a communication concept and conservation about the observations. The third approach, action, included a pragmatic concept of credibility that relies on observation and interpretations with a plan to act upon and make changes to the conditions investigated (Kvale, 1995). By combining these three approaches credibility in qualitative research involves the researcher having a questioning and critical attitude (Kvale, 1995). A later publication by Kvale and Brinkmann (2015) further describes how credibility also includes the credibility of the researcher themselves.

In addition to Lincoln and Guba (1985), Guba and Lincoln (1994), Lincoln (1995), and Kvale (1995), Morse et al. (2002) suggests reconsidering the verification strategies from Lincoln and Guba (1985) to ensure rigor. Their strategies include investigator responsiveness, methodological coherence, appropriate sampling, an active analytic stance, and saturation (Morse et al., 2002). These strategies move the responsibility of rigor from the external judges to the investigator and require the investigator to attend to rigor throughout the entire research process (Morse et al., 2002). Morse et al. (2002) claim if the question of credibility is only taken into account at the end of the study, the researcher may miss serious threats to credibility and reliability that are too late to change. Therefore, they recommend verification throughout the research process instead of post hoc to proactively address credibility threats (Morse et al., 2002).

The application of these credibility measures takes time and effort (Lub, 2015). Credibility in qualitative research goes further than solely academic reflection, and there may not be a single set of assessment criteria for qualitative research (Hammersley, 2007; Lincoln & Guba, 1985; Lub, 2015). Additionally, the conversation of triangulation, or the combination of two or more aspects to increase one’s capability to interpret findings is also of note as researchers have described its use as an alternative to validation (Denzin, 2012; Denzin, 1970; Flick 2007; Thurmond, 2001). Although the debate on qualitative credibility standards continues, the use of certain methods continues to be prevalent in qualitative research, and various credibility procedures are used within these methods. In this study, we found particular credibility procedures to be vital as they produced contradictory findings from common qualitative methods.

Methods

This paper draws from a larger research study using mixed methods to examine the impact of long-term international experiences in agricultural education. The study was also used for monitoring and evaluation purposes for the United States Department of Agriculture Foreign Agricultural Service funded grant program. Further, the study conducted was a part of a master’s thesis. Three other researchers involved in proposal, but one researcher conducted data collection and analysis with the support of other researchers. The study used a convergent parallel mixed methods design where the qualitative and quantitative components are collected concurrently, analyzed separately, then converged to confirm or disconfirm findings (Creswell & Plano Clark, 2011). The population included a census of eight fellows, n = 6 female and n = 2 male, ages 22 to 27, from the International Agricultural Education Fellowship Program (IAEFP). Most participants (n = 7) were United States Citizens, and one participant (n = 1) was a dual citizen in the United States and Nigeria. They lived and worked in rural Ghanaian communities for 10 months as agricultural educators, 4-H (a youth leadership program) advisors, and agricultural extension agents. All participants completed every part of the interview including the interviews, focus groups, and surveys. The program required the fellows to be United States citizens, have an undergraduate degree in an agriculturally related field, some experience in the agricultural youth organizations focusing on leadership such as 4-H or FFA, and participate in a 7-week training (Elliot & Redwine, 2020). The training included four components on interior formation, strategies for teaching pedagogy, cultural awareness, and Ghanaian agriculture (Dado, 2022). The fieldwork took place in the Eastern and Central regions of Ghana where the qualitative components included semi-structured one-on-one interviews, focus groups, monthly open-ended survey questions, and analyzing various IAEFP artifacts such as documents and training materials. Another focus group occurred in the United States upon the completion of the fellowship. The monitoring and evaluation plan was executed on the timeline as planned in the grant narrative.

The qualitative components were phenomenological in nature due to all participants being a part of the IAEFP, they each had a shared experience that cannot be replicated in the exact way (Merriam & Tisdell, 2016). The specific data collection methods of the one-on-one interviews and initial focus groups took place in January and February of 2022, in the seventh month of their 10-month experience. The one-on-one interviews gave the participants a confidential space to share their experiences thus far in the program with the researcher. Interviews were semi-structured in nature, including a set of structured questions, but they flowed like a conversation including follow-up questions or any information the participant wished to express. All qualitative components were conducted in English as all participants were English speakers. The interviews occurred at each of the fellows’ communities in a space of their choosing over a 3-week period. After each of the one-on-one interviews was completed, we conducted two focus groups with four participants in each the following day after the final one-on-one interview. The focus groups were held at the program house in Koforidua, Ghana during the February monthly meeting with all participants. Because the focus group was after the individual interviews, it allowed the researchers to execute peer debriefing and edit questions based on the one-on-one interviews. The focus group allowed for group reporting and explored comradery between the fellows. Additionally, we conducted another focus group with seven participants in June when the participants completed their fellowship and returned to the United States. The second focus group gave participants the opportunity to either confirm what they had previously discussed and adjust or add to what they had previously said. Thus, including member checking into the research protocol. Open-ended survey questions on monthly surveys from September 2021 through May 2022 where participants could support their quantitative responses by explaining their answers.

Before the participants left for their experience, my engagement with them began prior to their departure as we engaged them in their initial training in the United States. In addition to the interviews and focus groups in-country, we remained in Ghana with the participants for 3 months. This began with nearly 2 weeks spent among the participants whilst they received content training. We visited them at their homes within their community placements, as well as living at the program house among them when they gathered for three to 4 days monthly for training and reflection, and attended several events and competitions associated with their role in their program. The observations occurred during the weekend sessions, visitations to each site, and other program events. After the participants returned to the United States, we continued my engagement with them in a final focus group. Their comfort with the researchers was evident as conversations flowed easily and more findings were confirmed by having the final focus group. This prolonged engagement with the subjects was vital for both my further understanding of them as individuals and the context of their situations.

The final piece that contributed to qualitative data was a reflexive journal. The reflexive journal included all events that occur throughout the study and allows the researcher to reflect on all activities (Wallendorf & Belk, 1989). During data analysis, the reflexive journal assisted in cross-checking the data collected (Wallendorf & Belk, 1989). From the first time we engaged with the participants in July 2021 to the final focus group the following June 2022, we kept a reflexive journal based on my observations. The journal included a table of contents that linked the date to the journal entry. The first entry was when we were first introduced and could write down their names and demographics such as their age, where they were from, their educational background, etc. Then as we moved to the 3 months in Ghana, we were able to add more frequent journal entries. These entries included the observations from visiting communities, observations from the conversations at the program house, and the themes we saw emerging from the interviews and focus groups. This reflexive journal was also vital for the qualitative analysis as it assisted with coding into categories and narrowing themes found within the semi-structured interviews and focus groups. Further, the long-term nature of the reflexive journal allowed the researchers to build rapport with the participants because of the reflections documented as well as with data collection and analysis.

To ensure confidentiality, participant numbers were used throughout the research process. This included participant numbers for the interview and focus group transcriptions and within field notes. Additionally, only the researchers associated with the study and listed on the Institutional Review Board Approval had access to the data that was stored on a password-protected computer. Further, participants signed informed consent documents and were aware they could retract any statements or completely remove themselves from the study at any time.

Credibility

For this study, to address credibility and trustworthiness within qualitative research we followed Lincoln and Guba’s (1985) and Lincoln’s (1995) trustworthiness measures to ensure confidence. Specific strategies included a reflexive journal, audit trail, field notes, data saturation, transcription, member checking, and debriefing (Lincoln & Guba, 1985; Lincoln, 1995). Through the use of these strategies, we were able to triangulate findings with the multiple methods of data collection and credibility measures to add to the depth of my data collection and analysis (Denzin, 2012; Flick, 2007).

To establish credibility and authenticity we kept a reflexive journal from the time of designing the study through data collection. This reflexive journaling served as the collection tool of informal conservation data, as informal conservations were not recorded and happened randomly throughout the experience. To continue dependability and confirmability, we kept careful documentation through data collection and analysis and performed an audit trail through deliberation with a fellow researcher. We wrote careful notes during and after interviews and focus groups and asked follow-up questions in interviews (Lincoln & Guba, 1985). Therefore, we took comprehensive field notes which address credibility and transferability. To further credibility and transferability, we reached data saturation by conducting focus groups, interviews, and open-ended survey questions, as well as documenting observations. We also recorded and verbatim transcribed the interviews and focus groups through Otter, a transcription software, to ensure authenticity and credibility (Lincoln & Guba, 1985). The final component we conducted to ensure trustworthiness included member checking and peer debriefing to establish themes and deliberate, this addressed credibility, dependability, and confirmability (Lincoln & Guba, 1985). Member checking allowed for increasing the credibility of the findings due to the participant’s ability to “check” what they previously said, and either added to their statements or modified their statements (Koelsch, 2013). Throughout the member checking, participants mostly added to their thoughts or explained the reasoning further, participants did not disagree with their previous comments. Peer debriefing was critical to analyze how the qualitative themes evolved and emerged throughout the prolonged data collection. The frequent peer debriefs exposed aspects of the findings that may have been undiscovered if data was not scrutinized by multiple researchers (Nowell et al., 2017).

Findings

Using the constant comparative method by Glaser and Strass (1967), through analyzing and debriefing the qualitative data, six themes and 28 sub-themes were identified. All the codes and themes from every component of qualitative data collection were the same including interviews, focus groups, and open-ended questions on the monthly questionnaire. The themes include programmatic adjustments, international development, challenges, personal and professional growth, personal positivity’s, and contradictions (Dado, 2022). Although the other themes heavily describe the participants' thoughts about the IAEFP and skills gained through the experience, the contradictions theme exposes how their words in the interviews, focus groups, and open-ended questions contrasted with their everyday words and actions. This theme came directly from the reflexive journaling and observational data we kept through my prolonged engagement. Reflexive journaling of observations occurred throughout data collection and closely aligned with the timeline of interviews and focus groups; thus, contradictions cannot be explained through history or maturation. These findings increase the credibility of the research further due to the member checking and prolonged engagement that could confirm or disconfirm the findings. This theme included three subthemes, contradicting monthly meetings, contradicting desire for feedback, and contradicting expectations.

Contradicting Monthly Meetings

Participants participated in monthly meetings with their cohort and during the interviews and focus groups they described how they would like to have more structured monthly meetings with a clearly defined purpose. They mentioned ideas to be more productive including increased training on agricultural topics, professional development, lesson plan sharing, and spending less time doing reflections. However, when a more structured schedule was implemented in February and March based on these recommendations, participants expressed displeasure and requested multiple changes to the schedule so they could have more free time to go to the market, do laundry, or do other daily activities they usually do in their community. This contradicted what they described within their interviews, focus groups, and survey questions as participants expressed disapproval for the implementation of what they directly asked for. In the final focus group after the completion of the fellowship, they continued to express their desire for more productive and structured monthly meetings, even though they had verbally responded negatively when a structured schedule was given to them.

Contradicting Desire for Feedback

Participants mentioned they had a desire for increased feedback on their classroom instruction and 4-H advising. The program coordinator visited each fellow monthly, but participants expressed hope for more feedback to better their teaching skills via their interviews. As participants mentioned this concern early on in data collection for the monitoring and evaluation component of the grant program, the program coordinator was able to address the participants' concerns. However, when the program coordinator provided an opportunity for participants to record their teaching and receive feedback from a pedagogical expert, multiple participants, who had specifically requested this feedback, contradicted themselves when they asked if it was required, and expressed not wanting to complete the recording. Their objections to the opportunity to receive feedback were a direct contradiction of their descriptions in the formal data collection both during their fellowship and within the final focus group.

Contradicting Expectations

Aligning with their desire for more feedback, participants expressed a desire to know the quantitative expectations of the program from funding stakeholders. For example, they wanted to know how many classes they were supposed to teach, or how many students they should have in their 4-H club as a benchmark goal. However, when numbers were given to them after this feedback, for example, how many students they should bring to the 4-H camp, they expressed frustration and aimed to change the expected number. Further, participants frequently had conversations where they would express that they “didn’t like being told what to do” or being told how they should perform their teaching and 4-H club. Participants mentioned how they enjoyed their independence and appreciated the program’s flexibility. This contradicts their statements of wanting to have specific benchmarks and clearer expectations.

Prolonged Engagement, Reflexive Journaling, and Observations

The findings for the theme, contradictions, would not have been revealed without prolonged engagement, reflexive journaling, and observations. The contradictions theme contributed greatly to the overall study by providing more context to the mixed methods research and future program adjustments. The use of prolonged engagement, reflexive journaling, and observations proved useful to address credibility, but it also deeply contributed to the research findings as well.

Swain and King (2022) suggested informal conversations pose equal validity and have congruent epistemological status as formal conversations in interviews and focus groups. The findings from this study validate that conclusion as well; there is an advantage to increasing credibility through observations and informal conversations. However, there is the threat of memory that has the possibility of contamination and degradation (Swain & King, 2022). This threat can be mitigated through reflexive journaling and reflecting on the researcher’s own bias (Swain & King, 2022; Symon & Cassell 2012). In this study, if the informal conversations that came from prolonged engagement, observations, and reflexive journaling did not occur, the qualitative findings would not have resulted in as rich and descriptive outcomes.

The year-long prolonged engagement and observation of the participants within this study allowed researchers to observe varied and frequent contradictions between what the participants stated in their interviews and focus groups, and what they stated and did outside of the interviews and focus groups. Participants would often make statements and act in direct contradiction and opposition to their wants, thoughts, critiques, etc. Stated in their interview times and focus groups. Although this led us to question the credibility of the original focus groups and interviews, because we had the credibility components, we could critically analyze the data and debrief with fellow researchers to confirm the findings to thus add the theme of contradictions.

Additionally, the rapport with participants born of prolonged engagement assisted in the overall mixed methodology research. In the convergent parallel mixed methods design, the qualitative and quantitative components were analyzed separately, then compared on a side-by-side comparison (Creswell & Plano Clark, 2011). The side-by-side comparison allowed the qualitative components to explain some quantitative components (Creswell & Plano Clark, 2011). Specifically, the observations could explain some of the quantitative findings, specifically if there was an outlier in responses. Such as when participants quantitatively answered that their cultural awareness decreased from 1 month to the next, they qualitatively explained how they struggled to adapt to a cultural norm they frequently experienced that month. Subsequently, the prolonged engagement was critical to the credibility of the mixed methodology.

Conclusions and Recommendations

Within this study, the qualitative methods of interviews and focus groups provide rich descriptions of a phenomenon, yet without prolonged engagement, reflexive journaling, and participant observation, the juxtaposition within the results would not have been revealed. The complex issue of credibility has been in conversations within the qualitative research field for decades (Kvale, 1995; Lincoln & Guba, 1985; Lub, 2015; Morse et al., 2002). This data is evidence of the necessity of specific credibility measures and activities to most accurately describe a phenomenon. Further, as the data was also used for monitoring and evaluation purposes, the prolonged engagement, reflexive journaling, and participant observation provided a more accurate examination for monitoring and evaluation purposes of the grant program. Due to the descriptive nature of the findings, more accurate changes were made to the grant program for future years. Additionally, it also influenced the instruments and timeline for data collection in future grant programs. We conclude the credibility measures deeply influenced the qualitative findings as well as the overall monitoring and evaluation components of the grant program.

In future mixed methods research and phenomenological qualitative methods, we recommend placing prolonged engagement, reflexive journaling, and participant observation including prolonged engagement as top priorities. These activities are vital for credibility, yet also to increase the depth of the research findings. To reveal the full story in phenomenological research, the researcher should be completely immersed in the research process (Barusch et al., 2011; Flynn & Korcuska, 2018; Lincoln & Guba, 1985). Further, through the contradictions that appeared in the interviews and focus groups, we find member checking vital and encourage the use of member checking in qualitative research. Finally, when possible, we recommend increased time in data collection and designing a study to increase the prolonged engagement as much as possible. While this could include being actively involved with the participants throughout the entire research process, if this is not possible, we recommend including a second time to collect data after a sufficient time period has passed. In this study, when the participants were able to do a second focus group after they completed their fellowship, they could add to or edit what they have previously explained which contributed to the findings and either confirmed or disconfirmed what they had previously stated.

Although these findings were incredibly useful for the continuation of this grant program, we see these findings having applicability in social science research and evaluation. This research provides clear evidence of the value of prolonged engagement, reflexive journaling, and observations. While we recognize the additional monetary costs and time associated with long term engagement, this study demonstrates the value of including the costs of prolonged engagement in grant and program budgets to obtain the most relevant and high-quality data. This study adds validity to why funders should also be willing to include the costs of prolonged engagement as it creates more trustworthy monitoring and evaluation findings. The evidence furthers the conversation surrounding credibility in qualitative methods and can influence study designs.