Abstract
Longitudinal qualitative methodologies have gained increased attention in health and social science research as an appropriate methodology to explore participants’ experiences of change over time but is a challenging methodology to implement. We reflect on the methodological aspects of a novel longitudinal ‘mixed’ mixed-methods study using quantitative and qualitative methods with both individuals and groups, employing various participatory approaches. We focus on key learnings from our study including the use of a structured recruitment process to minimise loss to follow up, and issues of reflexivity when researchers and participants have a similar background. In addition, like many large mixed-methods longitudinal studies we experienced challenges determining sample size and managing and analysing a large dataset. There is a need for establishing clear guidelines for determining sample size, addressing reflexivity and managing large datasets. The lessons learned from this study will add rigour to the literature on conducting mixed-methods longitudinal studies.
Introduction
The purpose of undertaking research is to describe a situation, test a hypothesis or answer a research question, with the aim that research findings should as closely as possible reflect the ‘true’ situation. Qualitative and quantitative methods are complementary approaches to achieving these results. Where quantitative methods seek to be representative of a population and provide numeric answers with clear margins of error, qualitative methodologies seek a deeper understanding which, while not representative of a larger population, may be able to understand a more complex truth (Grossoehme & Lipstein, 2016; Guetterman et al., 2015) Studies which use mixed-methods take advantage of the complementary qualities of both methodologies to answer research questions comprehensively from different perspectives, and can provide innovative approaches to investigate complex issues (Fetters et al., 2013; Guetterman et al., 2015). The use of a variety of qualitative data collection methods, for example observations, participatory techniques, individual interviews, can provide different qualitative perspectives on the research question. In recent years there has been a growing interest in the use of mixed methods studies, specifically in health research (Plano Clark et al., 2015; Tariq & Woodman, 2013).
Longitudinal qualitative research aims to prospectively explore the lived experience of change, describing how and why change occurs, within social and cultural dimensions (Calman et al., 2013). This is a challenging methodology to employ if its strengths are to be fully realized (Plano Clark et al., 2015). Longitudinal qualitative data collection is iterative, continuously drawing on the themes emerging from the data (Carduff et al., 2015). Such research is time consuming, costly, and recruitment and retention of participants over long time periods may be challenging, particularly if the topic is sensitive or the population vulnerable. In addition, large datasets are generated, and analysis is complex with multiple dimensions which can take both a cross-sectional and longitudinal focus (Grossoehme & Lipstein, 2016; White et al., 2012). This complexity increases when both qualitative and quantitative methods are used, with the added challenge of integrating the findings of the two methodologies, and it is common for quantitative and qualitative results of mixed methods studies to be presented separately (Fetters et al., 2013). Little has been written about the challenges of using mixed methods in a longitudinal study and there is little guidance about how to design and implement these studies (Plano Clark et al., 2015; Schumacher et al., 2021).
In this paper we reflect on a longitudinal mixed-methods study employing a variety of data collection tools and interactive methods to prospectively explore the lived experiences of women working in the informal economy at defined time points during pregnancy, after having a baby, and after returning to work, to understand how these women managed the competing demands of caring for a newborn baby and earning a living to support their household and family. Our methodology was similar to the fully longitudinal mixed methods design described in the methodological review by Plano Clark et al., with both qualitative and quantitative data collected at all time points over the follow-up period (Plano Clark et al., 2015). However, we also conducted a series of FGDs with cohort participants after completion of the follow up period, which deviates from the methodology proposed by Plano Clark. In addition, our methodology was primarily qualitative, so that quantitative data collection methods were used to inform and provide context to the qualitative data but we did not have sufficient sample size for quantitative data to be presented separately. Thus, the methodology employed in this study was a variation from established methodologies for longitudinal mixed methods designs.
Methods
During the analysis and writing up of the study results, we reflected on our experiences. We used a novel approach by prospectively employing a variety of quantitative and qualitative data collection methods, using both individual and group research activities with our cohort of participants, and a variety of participatory techniques during a series of FGDs conducted on completion of the follow up. We collected quantitative data about household food insecurity, feeding practices and postnatal depression among our cohort participants at each time point. Key members of the research team [authors] met at regular intervals to identify and discuss the challenging aspects of undertaking a longitudinal mixed methods study, which are highlighted in this paper. There are important lessons for researchers who wish to use similar methods to gain an in-depth understanding of the lived experiences of vulnerable groups and explore complex research questions from multiple perspectives. To understand these methodological questions, it is necessary to describe an overview of the methodologies employed in the study. We adopted a mixed methods prospective design, based on Bronfenbrenner’s established Ecological Model (Bronfenbrenner, 1979) and adapted by the researchers (Figure 1).The theoretical framework recognises that health is influenced by the interaction between the individual, family, community and the wider physical and social environment. The framework allowed us to understand the range of factors that influence child health and wellbeing, and portrays these as concentric layers of influence, to show that changes at one level effect all levels (Bronfenbrenner, 1979). The adapted framework aimed to describe factors that influence a child’s health and wellbeing, particularly including the informal work environment. The framework provided a well-defined pathway to explain and link the variables measured during data collection. Theoretical Framework.
Summary of Data Collection Methods Employed.
The researchers identified quantitative factors from the theoretical framework that could not be reliably assessed using qualitative methods alone. Quantitative variables were selected to provide additional context for the qualitative data. We did not seek to undertake a quantitative study, but rather we aimed to provide a clear frame for our qualitative data. Quantitative data was collected on employment, sociodemographic data, feeding practices and mental health. We used the Household Food Insecurity Access Scale (HFIAS) (Coates et al., 2007) and the Edinburgh Postnatal Depression Scale (EPDS) to measure household food insecurity and maternal depression. We used an administered questionnaire to define quantitative variables at each time point. Administered questionnaires were used due to the low literacy rates of study participants.
Sampling and Recruitment
Our estimated sample size was 20 participants. This was deemed sufficient to allow for collection of rich data and to reach data saturation, based on our qualitative research experience. We recruited 24 participants to allow for loss to follow-up (LTFU) given the long follow-up period. Informal workers were recruited at the antenatal clinic in two clinics in Durban, South Africa, during the last trimester of pregnancy and followed up until after they had returned to work. Women attending the antenatal clinic were assessed for eligibility by trained field workers, and all eligible pregnant women were invited to participate.
To provide participants with adequate opportunity to withdraw from the study before enrolment and thereby prevent LTFU, we employed a three-stage recruitment process that included 1) screening for eligibility at the antenatal clinic, 2) a field visit by researchers to explain the study methodology and obtain written informed consent, and 3) a further field visit to collect baseline data. When participants completed all three stages, they were allocated a study number and formally enrolled in the cohort.
Data Collection
The data collection team consisted of two researchers with a Masters level of education [authors]. The researchers were both African, female, of a similar age to participants, and were familiar with participants’ cultural context. Although one researcher lived in the same community as participants, no participant was personally known to researchers. Each researcher was allocated a number of study participants to follow-up throughout the study. Focus group discussions (FGDs) were conducted jointly by the two researchers.
Longitudinal studies focus on change over time. Our study aimed to prospectively capture experiences at points of change in the participants’ circumstances rather than at fixed time intervals. Data collection was therefore closely managed, with bi-weekly telephone calls to participants to identify times of change and schedule interviews when required.
In depth interviews (IDIs), administered questionnaires, field notes were used to collect qualitative and quantitative data, with participants at five major transition points over the follow-up period: during pregnancy, within 2-weeks post-delivery, before and after returning to work, and when the child was left in childcare (Figure 2). Quantitative data was collected before each IDI using proprietary software on tablet computers and uploaded to a centralised server in real time. IDIs were used to collect qualitative data about sociodemographic, childcare practices, and living conditions. After completion of the follow up period FGDs were conducted using participatory techniques to explore participant’s experiences from a different perspective in a group context. Data Collection Timeline.
To ensure that the data collection process was iterative and adapted to emerging themes, the research team met regularly to review findings and identify topics to be addressed in follow up interviews. On completion of all the IDIs participants were brought together to participate in a series of four FGDs. To promote development of relationships between participants, participants remained in the same group for the FGDs. Each of the FGDs employed a particular participatory technique to explore a relevant topic (Table 1). The sequence of participatory methods used during FGDs was designed to be incremental in building trust, starting with a ranking exercise and progressing towards more sensitive topics like photovoice and journey with my baby (Figure 2). The methods are described in more detail in other published manuscripts (Horwood et al., 2021; Luthuli et al., 2020; Mapumulo et al., 2021).
Data Management
Transcribing and quality control of transcripts were carefully planned and tracked throughout the study period to ensure data privacy and quality. Audio recordings were downloaded, appropriately labelled, and stored in password protected folders. Audio-recordings were sent in batches to be transcribed and translated and tracked to ensure that they were returned. Final transcripts were similarly labelled and stored. A proportion of transcripts were quality controlled by researchers by listening to the audio-recordings and comparing with the written transcripts.
Quantitative data was quality controlled in real time to check for errors and ensure that data was complete.
Data Analysis
SPSS was used to analyze the quantitative data. Background characteristics, feeding and childcare practices were presented as frequencies. For the EPDS high risk of depression was determined by a score of >13. Household food insecurity was assessed using the HFIAS and categorized into none, mild, moderate or severe food insecurity based on the established scoring system (Coates et al., 2007). Only participant codes were used throughout the study, this allowed linking of individual participants’ responses across the quantitative data, IDIs and FGDs and provided anonymity throughout the study. For the qualitative data we used NVIVO12 to undertake a thematic analysis based on a priori themes. Priority was given to themes that the research team wanted to explore during the analysis.
Ethical Considerations
Working with vulnerable populations required adequate renumeration and support. Participants received financial compensation of R150 (approx. $9.60 USD) for each interview or FGD to compensate participants for the time spent away from work. Participants who scored high in the depression assessment and participants who were found to have moderate or severe food insecurity were referred to the study counsellor. The study counsellor provided basic support and referred participants to the appropriate support services within their communities, this included primary care services for participants at risk of depression, and social services for those with food insecurity. For the researchers, debriefing sessions were available with a research psychologist to assist them with coping mechanisms for difficult issues that arose during data collection
Results
In the results we describe how the study methods transpired compared with the planned methods, and describe the successes, challenges and learning points from study implementation.
Minimal Loss to Follow up
In total, 41 participants were screened and eligible to participate in the study, to achieve our sample size of 24 participants. The three-step recruitment process was completed with each participant until 24 participants were enrolled in the study. Eleven women failed to complete the recruitment process, suggesting that the three-step recruitment succeeded in excluding many participants who were not motivated to participate over the long term. The remaining six eligible women were placed on a reserve list to replace any participants who were LTFU early in the study. Early LTFU was clearly defined in the study protocol, as participants who completed the baseline interview, but not the post-delivery interview. One participant withdrew from the study before the post-delivery interview and was replaced by an alternative participant on the reserve list. During the follow up period, three participants did not complete data collection because they moved away from study site, thus LTFU was very low.
Compensation Payments Were an Incentive to Participate
Providing payment to compensate for time away from work was a strong incentive to participate and may have been a reason for the low LTFU. However, this also posed a challenge because for some women this payment may have been an undue incentive, for example one participant pretended that she had returned to work so that she could have a field visit and receive the payment. We removed the data from this interview and excluded her from further participation in the study. It is an ethical requirement to ensure that participants are appropriately compensated for their time, however, depending on the study methodology, this needs to be balanced against incentivising participants to provide responses that will allow continued participation.
Importance of the Researcher/Participant Relationship
The depth and richness of any data set was determined by the relationships built between researchers and study participants (Guillemin & Heggen, 2009). In our study, building rapport and trust between participants and researchers helped to elicit rich data during interviews and FGDs, particularly where topics were sensitive. The researchers met participants at locations and in spaces where they felt most comfortable as the first phase of building trust. Using the same group of participants for most of the series of FGDs allowed trust to be built between the participants.
For many participants, being involved in this study provided a support system and an outlet to speak about challenges, and it is likely that being able to tell their stories had a therapeutic effect for participants (Batty, 2020). During the study period, some participants shared experiences that were highly emotive including domestic abuse, witchcraft, and food insecurity. However, we remained clear on the difference between conducting qualitative research and therapy, where the latter includes giving advice which is inappropriate in a research setting. Due to the strong connections between the participants and researchers, participants were able to honestly open up about their experiences allowing researchers to explore and understand participants’ lives.
Multiple Data Collection Techniques
This study included a number of novel approaches including collecting qualitative and quantitative data at times of change, conducting individual and group data collection activities with the same cohort, and using a variety of participatory qualitative techniques during a series of FGDs. This methodology was extremely successful in obtaining rich data about a series of sensitive topics including women’s relationships with the child’s father, complex and sometimes abusive family dynamics, and risk-taking around childcare choices. In particular, the use of both individual and group discussions to explore similar topics yielded different insights into women’s experiences. At the time of the FGDs participants had longstanding relationships with researchers, and they quickly developed relationships with each other over the series of FGDs. As a result, constructive discussions were held between participants providing novel insights into participants’ lived experiences from the perspective of co-participants. Group members were able to challenge each other and work together to suggest solutions to concerns expressed during FGDs.
We collected a lot of quantitative data during the series of IDIs. As a result, the number of participants was small and the only purpose of the quantitative data was to provide context for the qualitative data. For example, we presented data about challenges and delays to obtaining a child support grant in the context of household food security measured at the same time points (Luthuli et al., 2022), photovoice data about child care choices combined with quantitative data about childcare used by mothers (Horwood et al., 2021), and experiences of infant feeding together with quantitative data about how planned and actual feeding practices changed at each time point (Luthuli et al., 2020). We acknowledge that the mixed methods approach for our study could have been strengthened if we had made clearer links between the qualitative and quantitative data, so that findings from the quantitative questionnaires could have guided the qualitative interviews. For example, when participants reported food insecurity or high depression scores during the quantitative data collection, this should have led directly to a series of relevant questions in the IDIs. We did this successfully with feeding practices but more qualitative data about food insecurity and experiences of depression or low mood would have strengthened the richness of the data set. Since quantitative and qualitative data were collected at the same visit, this would have required data collectors to immediately adapt the focus of the IDIs based on responses to the quantitative interviews.
Longitudinal Data Collection as an Iterative Process
In order to focus on the longitudinal nature of the data and describe individual participants’ experiences of change over time, it was important that researchers picked up the dominant themes from previous interviews across the series of interviews. To achieve this, researchers intended to review transcripts concurrently with data collection activities. However, this was challenging because the time taken to complete the transcripts was much longer than anticipated. The timing of interviews was often unpredictable, and interviews were scheduled at short notice, so that transcripts were frequently not available at the time of the next interview. In this situation researchers listened to audio recordings of the previous interview and reviewed field notes from telephone conversations. Regular telephone correspondence strengthened the methodology by maintaining researchers contact with participants and providing opportunities to engage with participants on significant life changes between interviews.
It was also important that data collection was iterative not only with individual participants but across the whole cohort. Thus, themes and topics that arose in some interviews could be highlighted for inclusion in the discussion guides for other participants, if appropriate. This was achieved by regular meetings between researchers to reflect on any new or unexpected themes that had arisen during discussions. However, this was time consuming and was not always achieved or sustained, so that opportunities for identifying new themes and making the most of the longitudinal nature of the study may have been lost.
Data Analysis
We used a thematic analysis but realised quickly that we had to focus on those key themes that were important for the study objectives to avoid being overwhelmed with data. We found using an analytical framework assisted in guiding the data analysis.
After initial coding, the research team came together to identify key themes from the coding framework and developed a coding tree to work with. Our team had a variety of skills and experience, and we divided the coding between team members. However, one challenge was that the analysis became fragmented as different team members worked on different interviews and on different topics.
Discussion
Longitudinal mixed methods research in the health field is evolving and is a methodology where innovative approaches can be employed, but it is important to reflect on the learnings if the methods are to be improved and methodological rigour maintained (Carduff et al., 2015; Plano Clark et al., 2015; Schumacher et al., 2021). Longitudinal qualitative research has a long-established place in social science disciplines and has attributes that make it very suitable for health research. However, given the relatively small number of such studies where mixed methods are used, there is a need for methodological literature aimed at discussing the complexities of conducting health-related longitudinal mixed methods research (Carduff et al., 2015; Plano Clark et al., 2015). This paper adds to the literature on the use of longitudinal mixed methods research. In this discussion, we aim to review several topics that were a challenge for us when undertaking this study, and where we suggest further guidance is needed to improve the rigour and methodology of research. These key topics were: estimating sample size; managing reflexivity during data collection; managing analysis of large qualitative datasets.
Estimating Sample Size
Data saturation is most often used in qualitative studies to determine when a sample size is complete. However, it is difficult to understand how researchers employ this method in practice, and most qualitative research reports do not explicitly state how researchers determined when they reached a sample size saturation (Hennink & Kaiser, 2022; Vasileiou et al., 2018). For example, how do researchers determine with certainty that no new themes emerged in a particular interview or FGD, and how many interviews are required with no new themes before data collection ends. There are other constraints, such as cost and time that are used to determine when saturation is reached (Robinson, 2014). Using data saturation to determine sample size is inappropriate in a longitudinal study because sample size must be pre-determined and LTFU over the study period must be considered.
Onweugbuzie and Leech (2007) suggest researchers find studies of similar design to determine sample size, while (Bernard, 2013) suggest 10–20 people are sufficient to explore and understand the lived experiences of participants when conducting a qualitative study. In addition, structured techniques have been proposed to calculate when saturation is reached (Guest et al., 2020). In reality a sample size is often a “thumb suck,” with many researchers aiming to balance the parameters of obtaining enough data to reach saturation with having too much data, making the analysis difficult to manage (Fugard & Potts, 2015). This is the approach we took, using our experience to estimate the numbers required to achieve this balance, but we suggest that a more systematic approach would have assisted us to collect a more manageable quantity of data. A review of sampling methods employed suggests that these vary widely, including some that are not compatible with the conduct of qualitative research, but there is little guidance on how to operationalise the concept of saturation (Hennink & Kaiser, 2022). There is even some suggestion that all a priori sample size estimations are incompatible with concepts underpinning qualitative research, which is in essence adaptive and emergent (Sim et al., 2018). If rigour in longitudinal qualitative health research is to be achieved, we suggest that clear guidelines regarding sample size for qualitative sampling be developed. Further, researchers would be assisted by step-by-step guidance on operationalising the concept of reaching data saturation (Sebele-Mpofu, 2020).
Managing Reflexivity during Qualitative Data Collection
Reflexivity or the process of researchers reflecting on their role as part of the research process and how they were impacted by the research, is a key component of qualitative research (Palaganas et al., 2017). It is an ongoing process of reflecting on how researchers’ viewpoint, gender, location, and assumptions influenced the study results (Hesse-Biber & Piatelli, 2007). While the importance of considering reflexivity is frequently highlighted, it is rarely systematically addressed in the writing up of qualitative health research (Malterud, 2001; Palaganas et al., 2017). We suggest that more guidance on how to address reflexivity would assist researchers to maintain rigour in qualitative data collection and analysis.
In our case this reflection focused on how to balance building rapport with participants without overstepping healthy researcher/participant boundaries. In quantitative research health professionals are taught to maintain a professional boundary to prevent emotional distress (Lonne, 2003), however qualitative researchers are trained to do the opposite, to build rapport by disclosing personal information with participants to encourage participants to share their thoughts and emotions (Batty, 2020; Ceglowski, 2000; Goodwin et al., 2003). Due to the similarities of the backgrounds and the longitudinal nature of this study, researchers provided support and developed ongoing relationships with participants and sometimes, as Batty (Batty, 2020) suggests, found it difficult to maintain a distance from the participants concerns, which left the researchers feeling overwhelmed and emotionally exhausted (Dickson-Swift et al., 2008; Wilkes & Elmir, 2011).
To ensure researchers are not overwhelmed with participants demands, a clear exit strategy should be communicated with participants so they are prepared for the transition and closure of the study (Dickson-Swift et al., 2008) and Batty (Batty, 2020), suggest setting clear, defined boundaries at the beginning of the study, focusing on consent, rapport, clarity on therapy versus research, leaving the research relationship, and the management of professional boundaries. It is important that researchers undertaking long term data collection roles receive training in how to deal with these situations, that there are clear protocols to guide the limitations of their role, and, most importantly, that the support provided to the researchers is adequate to address these concerns. Support for researchers should be scheduled regularly rather than relying on researchers to identify concerns themselves.
Managing Data Analysis of a Large Qualitative Dataset
Due to the vast amount of quantitative and qualitative data collected, data analysis was challenging and time consuming. Data reduction is common among qualitative studies due to the large amount of data collected, however, there is no standardized approach to analysing large datasets. Though we used a theoretical framework to guide emerging themes, the study produced too much for our team to analyse.
The literature suggests various techniques to make the data analysis easier. Instead of searching for themes in interview transcripts, structured coding can be used to code answers to specific questions asked that related to the research question. Pezalla suggests dividing the interview questions into themes during data collection in order to organize long transcripts by theme (Pezalla et al., 2012). Fugard recommends determining how many times the researcher wants to observe a specific theme for it to be considered noteworthy in the analysis (Fugard & Potts, 2015), while RADaR is a systematic rapid qualitative data analysis technique used to reduce the amount of data collected to produce a concise analysis (Watkins, 2017).
We learned the importance of having data reduction plan at the beginning of the study and to document the data reduction process. A more systematic sampling process could have also reduced the volume of data. Additionally, it is important to have a research team that is diverse in research background and skills in order to get varied views and perspectives on the data collection and data analysis process.
Conclusion
As longitudinal mixed methods research is more commonly employed in the health field, it is important that robust methods are used. We highlight methods used in our study that were particularly successful in obtaining rich data, particularly employing a variety of data collection techniques with individuals and groups in our cohort. We also highlight areas where clear guidance is lacking for researchers aiming to implement this method, including determining sample size, addressing reflexivity and managing large qualitative data sets, all of which are integral components of conducting longitudinal mixed methods research. We suggest that clear guidance is required so that the potential of this research methodology can be realised without compromising research rigour.
Footnotes
Acknowledgements
The authors would like to thank Dr Nigel Rollins from the World health Organization for his strong support for this study. We would like to thank Ethekwini health district for allowing recruitment of participants in the clinics. We also thank the study counsellor Cleo Phewa for her contribution. Our particular thanks go to participants for their contributions.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The funding for this study was provided by the World Health Organization and Grant id 2018/845490. The funders did not participate or play a role in the design of the study, data collection, data analysis or reporting of the results.
