Abstract
Over summer, I had the opportunity to observe one of the new iRobot vacuum cleaners in action. Chief among the advertised benefits of these appliances is the artificial intelligence system. This enables the vacuum cleaner to build a “mind map” of the building so that, over time, it becomes increasingly better at navigating the floors. This capacity to navigate successfully around furniture and other obstacles improves over time because information generated from successive circuits of the floor refines the map’s detail. Once the mapping is complete, the machine can be programmed to do selected rooms, selected routines, or the whole floor. This enables the vacuum to clean floors while people are out of the house. The vacuum sends texts confirming the job is done or reporting problems. The machines are fascinating to watch for a little while; it is diverting to observe this little pod pottering about on the floor and wonder how many times it will try to get under the couch (it will never fit) and watch as it determinedly pushes so hard it gets stuck and then sends a text asking to be rescued.
The house in which I was observing had a glass wall. Watching the vacuum cleaner trying, unsuccessfully, to deal with this invisible obstacle prompted reflections on the differences between the brave new world promised by the use of algorithms and counting-based systems, and qualitative approaches to knowledge generation. In this sense, observing the vacuum on the other side of the glass wall illustrated some of the differences between nomothetic and idiographic ways of knowing.
Of course, the machine could not “see” the glass wall to begin with, and it thus needed the algorithms to “discover” it. However, even after several months of mapping, the vacuum cleaner appeared to not have noticed the wall. It did not seem to ever “learn” that there was a permanent obstacle that would continue to frustrate its efforts no matter how much energy it expended in trying to push its way through. Its algorithm did not apparently include accumulating the knowledge that this repeated inability to pass through the invisible wall held meaning worth retaining. This was odd as it had developed an intimate connection with a nearby chair. On every circuit, it consistently diverted to clean around each leg and under the chair as well, demonstrating an astonishing level of dedication. Indeed, it would continue to seek out the chair legs to circle, even when the chair was removed. Sometimes if the chair was not there, it would go “off-piste” down the hall, instead of to the kitchen, as instructed, apparently confused by the disappearance of something it had mapped. These unpredictable actions seemed almost human, and it was tempting to anthropomorphize and attribute free will to talk about its unhealthy attachment to the chair along with its stubborn refusal to recognize the wall. The chair, which was often moved to other parts of the house, seemed to form an anchor worth remembering, while the glass wall remained out of scope and not relevant to the mapping process despite its permanence.
I could stand on the other side of the glass wall to the vacuum cleaner and observe everything clearly. I could, for instance, observe that it was not that good at getting into the corners. I could also walk around the wall and become part of life on the other side, something the vacuum cleaner could not seem to master. Finally, I could, from either side, observe the wall itself: I could see it, and see through it, at one and the same time. Being able to see the vacuum cleaner on the other side of the glass wall trying to break through and at the same time to actually also see the glass wall itself prompted me to think about the important roles that qualitative methods play in the generation of knowledge.
It is this richness of knowledge created by careful, systematic observation from within (emic or insider research), one step removed (etic, or outsider research) and by looking at the wall (using critical reflective processes) that qualitative research offers. This distinctive contribution to the advancement of knowledge lies in the capacity to trace subtle patterns and pick up and unpick nuanced threads that explain why human behavior deviates from averaged norms, so that strategies and responses can take account of variability as well as commonality. Their flexibility, capacity to take multiple standpoints, and use multiple strategies enable us to see and know in many different ways. Alongside this, the critiques of qualitative methods as not relevant to scientific inquiry because they lack validity, reliability, and generalizability seem to reflect the iRobot carefully mapping the floor on the other side of the glass wall, oblivious to the floor it could not “find,” the corners it could not reach nor to deal efficiently with the mutability of the chair.
Others have also noted this curious juxtaposition of confidence that quantitative methods will tell us what to see alongside the critique that qualitative methods do not meet the standards of a scientific enterprise and that they are not a legitimate way for generating new knowledge (Sale & Thielke, 2018). For instance, in an open letter to the BMJ editors on their decision to not publish papers reporting qualitative research (Greenhalgh et al., 2016, p. 3), the impact on outcomes for patients when surgical safety checklists are used is cited. The authors observe: Far from being a simple “technical” procedure, the [surgical safety] checklist demands new forms of cooperation and communication between surgeons, anesthetists, and nurses. Depending on a host of contextual factors, safety checks may substantially disrupt team routines and be resented rather than welcomed. When (and to the extent that) the checklist is treated as a tick-box exercise, it will fail to generate benefits [that clinical trials have identified] and may even lead to harms. From the policy maker’s perspective, qualitative studies of the professional, organizational, and political context of nationally driven checklist-based patient safety initiatives can help explain both successes and failures.
Combining qualitative and quantitative approaches is not straightforward. This journal, IJQM, is one of the few that explicitly encourages reporting mixed methodologies, recognizing the value of such work and the importance of methodological accounts of their use. However, many journals are reluctant to publish mixed-methods research, and reviewers often lack familiarity with both methodologies and thus can struggle to fairly assess such papers. Sometimes the challenge of combining both approaches is as mundane as being able to compress two methodologies and their related findings into journal word-length restrictions. As a result, researchers sometimes have to report qualitative and quantitative results separately, diluting the power of the approach and undermining its central purpose. More technical in nature are challenges with effectively drawing together the broad, averaging conclusions achieved from the large-scale patterns quantitative methods generate (as reflected in the confident message from the vacuum cleaner that “cleaning is complete”) with the more subtle knowledge we gain from qualitative methods (as reflected in our certain knowledge that the vacuum cleaner “did not get into the corners,” nor see the [to us] very obvious blockage represented by the wall that prevented it from completing the job). At a deeper level, there may be epistemological differences that make it challenging to draw together the knowledge generated by these different methodologies. Despite the challenges, there is much to be gained from combining these techniques, given that there is a clear understanding of what each technique offers; what type of information and explanation each can provide.
A recent project I was involved in illustrates the benefits of utilizing mixed methods at all stages of the research process. Our purpose was to trace patterns of change in outcomes for a large cohort of vulnerable youth as they moved through adolescence and into young adulthood. We wanted to trace patterns in changed circumstances for these youth during this life stage and to link these changes to factors such as the quality of their relationships, neighborhoods, education, and a range of other experiences. We had a particular interest in the role of psychosocial services in the lives of these youth and, accordingly, they were selected partly on the basis of having multiple service involvement. The project began with a large-scale, longitudinal survey repeated 3 times at approximately annual intervals, followed by 3 years of qualitative interviews that created individual life histories for a subset of youth (Sanders & Munford, 2017). Both surveys and interviews were administered/conducted individually.
Thinking about the different contributions qualitative and quantitative methods can make, we noticed two things when undertaking the survey. The first was the amount of talking and explaining by youth that occurred around the completion of the survey. Young people’s explanations of why they chose various options were very valuable data, but it wasn’t captured in the survey at all. This meant that if we had used the survey as the only data gathering tool for the first 3 years, we would have missed all this detail. Fortunately, as qualitative researchers, we had developed a case-summary template that interviewers completed after each interview, and so, this information was captured (Sanders et al., 2014). This information was useful not only as data, it was also useful in terms of keeping in touch with youth and finding them for successive interviews.
Related to this, the second thing we noticed as we were completing the three rounds of surveys was that qualitative techniques were the most effective strategies for keeping in contact with youth between interview rounds. The relational tools that are part and parcel of every qualitative researcher’s repertoire provided a framework for building the picture of the young person’s world that generated a social map enabling us to find them again and again. It quickly became apparent that if we adopted standard survey methodologies for tracking and retaining the youth, we would have a very high attrition rate. In a very real sense then, the success of the survey depended heavily on our capacity to adopt qualitative techniques both to capture the rich data that the survey process stimulated and to keep youth in the study so that we had a robust data set. We developed the first iteration of the PARTH model (Sanders & Munford, 2017) to describe our retention strategies. This was subsequently modified as a framework that could inform psychosocial interventions with youth (Sanders & Munford, 2019; www.youthsay.co.nz). Much like the vacuum cleaner analogy, if we had stuck with just the robot, we would not have cleaned the room properly.
While in many ways this project was quite a traditional piece of research, in that its primary data collection strategies were interviews (both survey and semistructured qualitative interviews), our intention was to generate data that would stimulate change in social policies and interventions with vulnerable young people leading to better outcomes. From the outset, we wanted to generate information that could be used to implement change. The prospects are poor for youth who become involved with multiple service systems in Aotearoa/New Zealand. By and large, interventions do not substantively improve their chances of achieving successful outcomes as adults (Department of Prime Minister and Cabinet, 2019; Johnson, 2018; McLeod & Tumen, 2017; Mental Health and Addiction Inquiry Panel, 2018; Ministry of Social Development, 2016; Ministry of Youth Development, 2017).
A key purpose, therefore, was to learn about what aspects of interventions did make a difference and for which youth. This implementation focus meant that throughout the project, our attention was not only on completing the research and writing the more traditional academic papers but also on what type of data was most likely to have the greatest impact on practitioners and policy makers, those who make decisions about service specifications that shape service delivery, and those who do the intense work of supporting youth.
With this aim in mind, from the longitudinal survey data, we were able to demonstrate that, on average, youth achieved better outcomes when they experienced psychosocial service interventions delivered in ways consistent with Positive Youth Development principles (Sanders et al., 2017). This pattern applied irrespective of the particular type of service they received. This was a powerful finding. It demonstrated, statistically, that positive relational practices by professionals opened pathways to better outcomes for the most vulnerable youth in our communities. We thought that this piece of analysis alone would produce a game-changing impact in social service delivery.
The policy community places a high priority on “evidence,” and accordingly, we anticipated that our solid statistical models, published in respected international journals, would demonstrate the importance of factoring relational quality into their service development work. We therefore presented these findings to a range of key policy makers, attending their forums and also meeting individually with them. However, disappointingly, this approach turned out to be largely ineffective. It seemed that the concept of encouraging positive relational practices was difficult to translate into policy. The idea that relationship building could and should be a central feature of interventions with vulnerable youth did not gain traction. Much like the glass wall for the vacuum cleaner, relationships seemed to be the invisible obstacle that stopped interventions from being successful and that very invisibility made it difficult for policy practitioners to pin this idea down and capture it in service specification language.
Perplexed by this response, we turned our attention to practitioners. We experienced a more enthusiastic response here, but it seemed practitioners also struggled to take the patterns in the survey data and apply them to their own practice. At this point, we began using the qualitative findings to tell a larger story of young people’s experiences of psychosocial interventions in the context of their lives and to situate the statistical models as illustrative of how experiencing positive relational practices influenced outcomes across a population. Much like the observer watching the vacuum cleaner on the other side of the glass wall, the qualitative data provided the larger picture within which the vacuuming occurred. In doing this, we drew not only on the data (qualitative and quantitative) but also the experience of doing the research. In this process, we began to adapt the PARTH model, which we initially developed to explain to other researchers ways to improve their retention rates as a model for relationship building in interventions with youth. This approach was highly effective, and practitioners seemed to immediately relate to the stories not only of young people’s experiences of interventions but also researchers’ experiences of locating and interviewing them. In the end, it was this layered story drawing on both types of data and the experiences of relationship building needed to retain youth in the research that demonstrated how practice could be made more effective.
In sum, the learning here was that changing the approaches professionals take to working with vulnerable youth required a richly textured and layered story, something that neither statistics nor qualitative findings alone could tell.
The glass wall analogy also seems useful when thinking about doing youth research. For instance, while all researchers were young once, we are “on the other side of the wall” as adult researchers. This means that even when we may think we understand what young people are saying and what we are seeing, we cannot behave as if the wall is not there. In Aotearoa/New Zealand, this idea of seeing but not necessarily knowing can be expressed in terms of Ako: the Māori concept of being simultaneously a learner and a teacher. This reminds us that in listening to a young person and gathering data from them, our learning is an interactional dialogue where shared meaning is created, regardless of the methodology employed. This is a complex reciprocal process of interaction to negotiate shared meaning, as Giddens has noted: Social actors give meaning to existing social and natural conditions, but they also try to exert their influence to change these conditions…human activity is bound up both with interpretation and power. (Giddens, cited in Coenen & Khonraad, 2003, pp. 439–440)