Abstract
What are the challenges of turning data subjects into research participants—and how can we approach this task responsibly? In this paper, we develop a methodology for studying the lived experiences of people who are subject to automated scoring systems. Unlike most media technologies, automated scoring systems are designed to track and rate specific qualities of people without their active participation. Credit scoring, risk assessments, and predictive policing all operate obliquely in the background long before they come to matter. In doing so, they constitute a problem not only for those subject to these systems but also for researchers who try to study their experience. Specifically, we identify three challenges that are distinct to studying experiences of automated scoring: limited awareness, embeddedness, and ongoing inquiry. Starting from the observation that coming to terms with one's position as a data subject constitutes a form of learning in its own right, we propose a research strategy called critical companionship. Originally articulated in the context of nursing research, critical companionship invites us to accompany a data subject over time, paying critical attention to how the participant's and the researcher's inquiries complicate and constitute each other. We illustrate the strengths and limitations of this methodology with materials from a recent study we conducted about people's credit repair practices and sketch a set of sensibilities for studying contemporary scoring systems from the margins.
Introduction
As the number of studies diagnosing discrimination, inequality, and bias in computer systems grows (Benjamin, 2019; Noble, 2018), researchers have set out to understand how people experience and cope with being measured, tracked, and judged by seemingly intelligent machines. Rather than trying to open “black boxes” by deciphering mysterious algorithms, auditing for data bias, or studying developers and engineers, these scholars have been interested in the everyday experiences of challenging these systems in different areas of life, including social services (Eubanks, 2018), insurance (Cevolini and Esposito, 2020), household assistants (Pridmore and Mols, 2020), and music recommendations (Siles et al., 2020). A key concern behind these efforts to decenter the study of big data systems has been the imbalance of power between sorters and sortees—“those who are able to extract and use un-anticipatable and inexplicable (…) findings and those who find their lives affected by the resulting decisions” (Andrejevic, 2014: 1683). A better understanding of the latter, it is argued, promises to increase the agency and voice of people at the margins of these systems and also helps us move beyond a dominant but narrow focus on technology as the starting point for our inquiries (Kennedy, 2018; Pink et al., 2017; Taylor, 2017).
Yet capturing and representing the experiences of data subjects—people who are subject to persistent tracking, scoring, and analysis through data-driven systems (Couldry and Yu, 2018: 4479)—is not an easy feat. Some challenges are generally known, such as the thorny epistemic problem of soliciting experiences that would not otherwise have been articulated (Ziewitz, 2017) or the practicalities of working with geographically distributed participants (Star, 1999). Other challenges, however, are more specific to the case of automated scoring. For one, these systems are designed to “sink” into the background (Bowker and Star, 1999: 237), becoming a defining but subliminal feature of our lives. Further, even if a data subject does become aware of their position, they usually have a hard time making sense of the organizations and machines that are widely seen as inscrutable in ways that even their creators do not fully understand (Burrell, 2016: 2–3). Methodologically, this situation therefore raises an intriguing question: how to capture the experience of people who may not fully be aware of their position and who themselves are figuring out their situation vis-à-vis the system?
In this paper, we propose a methodology for studying the experiences of people who are subject to automated scoring systems and explore the strengths and limitations of that methodology through an empirical example. Drawing on work in feminist science studies and healthcare research, we develop the idea of critical companionship (Titchen, 2001: 80) and adapt it to the case of automated scoring. Originally conceived in the field of nursing research as a form of training on the job, critical companionship gives us a handle on the partly interlocking “learning journeys” of researcher and participant. Whereas participants are trying to make sense of consequential scoring systems, researchers are trying to make sense of how participants experience that process. By (self-)critically accompanying data subjects over time, we aim to make productive this relationship and turn potential frictions into an analytic resource in its own right. Empirically, we explore the strengths and limitations of this methodology through the example of a study we conducted in the specific context of consumer credit scoring. Drawing on a mix of diaries, diary-interviews, and fieldnotes, we reflect on our own experience of accompanying ten participants as they were trying to repair a subpar credit score over the course of one year. We conclude by identifying a set of sensibilities for studying with data subjects.
While we explore and demonstrate the methodology of critical companionship in the specific case of credit scoring, there are of course many other instances of scoring systems that operate in different ways. Predictive policing (Jefferson, 2018) and psychometry-based marketing (Han, 2017), for example, are even more opaque and inaccessible to data subjects than credit scoring. These differences are analytically important and should inform ongoing investigations of “securing through algorithms” (Amoore and Raley, 2017). However, our focus in this paper is on how the methodology of critical companionship can help us analyze those moments in which a data subject becomes aware of being scored and tries to act upon this knowledge. In this sense, credit scoring is a useful setting for exploring these ideas.
From data systems to data subjects
To appreciate the challenges of studying the experiences of data subjects, it helps to have a closer look at their peculiar role within contemporary data systems. For the longest time, the relationship between humans and computer systems has been studied through the figure of the user, a concept bridging the design and use of digital technologies (Hyysalo and Johnson, 2015). Yet this figure is no longer adequate to capture the experiences of people in a data-driven world. As new computational techniques for tracking and evaluating conduct are becoming more and more pervasive, people are increasingly subjected to constant monitoring that does not depend on direct participation (Chan, 2019b; Zuboff, 2019). In fact, most operators of these systems would prefer that those affected were not aware of their position, trying to protect their scores from outside interference. The figure of the data subject nicely captures this dual role of people acting as both resources and targets for an automated scoring system.
Not surprisingly, these developments have caused a number of concerns. Not only do data subjects have to contend with ongoing surveillance and exposure. They also are affected by a range of racial, social, and economic biases that are perpetuated by these systems (Benjamin, 2019; Noble, 2018). In fact, most scoring systems have far-ranging consequences for people's life chances as citizens, consumers, patients, students, workers, and so on (Fourcade and Healy, 2013), often putting them in a position of precarity (Duffy, 2020). The situation is exacerbated by the fact that data subjects do not have access to these systems, which are widely seen as opaque and inscrutable (Burrell, 2016). Since formal recourse and redress are often not available, the few remaining ways to influence and optimize one's image tend be portrayed as troublemaking and a way to “game the system” (Ziewitz, 2019). In this sense, data subjecthood can be regarded as a key feature of “data colonialism,” a new form of colonialism that is appropriating human life through its conversion into data (Couldry and Mejias, 2019: xix).
Against this backdrop, a number of initiatives are under way to better understand the role of data subjects, mostly focusing on a particular system or technology. In the case of credit scoring, for example, most work to date has relied on statistical analyses of datasets (Fellowes, 2006; Nelson, 2010) or quantitative surveys (Arya et al., 2013; Levinger et al., 2011). More recently, researchers have turned to more interpretive methods, analyzing discourse in discussion forums (Deville, 2016; DuFault and Schouten, 2020; Mackenzie, 2017) or conducting ethnographic studies of marginalized groups and financial inclusion advocates (Kear, 2017). Beyond that, more mundane forms of sense-making, folk theories, and everyday experiences of credit scoring have remained under-explored, even within scholarship on the Quantified Self movement (DuFault and Schouten, 2020). Our research builds on and expands this work not only by exploring the methodological challenges of following the everyday experiences of data subjects, but also by taking seriously the role of the researcher in this process.
Pitfalls for empirical inquiry
From a methodological point of view, the peculiar position of the data subject poses a number of specific challenges. Not only do researchers face the usual problems of studying temporally and geographically distributed practices (Star, 1999: 383) or capturing people's experiences in technologically mediated settings (Kazmer and Xie, 2008), they also have to account for the obliquity of automated scoring and the asymmetries of power that result from it. Three challenges stand out in this regard: limited awareness, ongoing inquiry, and embeddedness.
First, data subjects may not be aware of their predicament and only learn that they are being scored during particular events. For example, a tenant may not be aware they have a tenant score until a landlord rejects their application on that basis. Even Facebook users who interact with personalized ads and news feeds on a daily basis are not necessarily aware that they are being algorithmically targeted (Eslami et al., 2015: 153, 2016; Rader and Gray, 2015: 178). In an analysis of online credit forums, Dufault and Schouten (2020: 302) found that it usually takes a “precipitating incident” or negative event to trigger such awareness. The problem, then, is that any study focusing on the experiences of data subjects will inevitably prompt participants with information that is likely to transform their understanding of their own position vis-à-vis the system. Yet, if such moments of realization are an important aspect of data subjecthood, then they should be captured as a part of the phenomenon and not be dismissed as a methodological inconvenience.
Second, even if participants are generally aware of the existence of a score, making sense of it is not a singular event. Shrouded in technological complexity and corporate secrecy, automated scoring systems tend to be inscrutable to an extent that even their creators cannot explain their operation (Burrell, 2016: 2–3). People thus conduct their own investigations, coming up with folk theories about the system (Bucher, 2017), acquiring expertise (Chan, 2019a), or learning how to be an “ethical” participant (Ziewitz, 2019). These interactions between the system and its subjects are constantly developing and mutually constitutive, resulting in cycles of what Rona-Tas (2017: 56) called “enhanced performativity.” In other words, living with and making sense of scoring systems involves an ongoing process of observation and inquiry in its own right—a process that runs alongside the researcher's investigations. An adequate research strategy will therefore have to take into account the duality of investigations by both the researcher and their participants.
Third, the experience of scoring systems is inextricably linked with the experience of circumstances that the score purports to measure. For example, a person struggling with a subpar credit score will also struggle with a range of other issues, such as unpaid bills, credit card debt, or family obligations. Furthermore, experiences of scoring differ widely among data subjects. For example, automated systems have been shown to reinforce and perpetuate systemic racism, economic inequality, and other forms of historical oppression (Jefferson, 2018; O’Brien and Kiviat, 2018). Many scoring systems also operate as self-reinforcing feedback loops that are locking people into particular segments of society. More than other forms of digital engagement, the lived experience of data subjects thus remains deeply embedded in their lifeworlds, requiring us to take a broader view of what it means to study the experiences of data subjects.
Taken together, these challenges make clear that people's experiences of automated scoring are not a readily observable phenomenon. We thus suggest that we expand our analytic repertoire to capture how the inquiries of researchers and data subjects can complement and complicate each other in productive ways. Turning data subjects into research participants is not only a practical but also an epistemic challenge because confronting data subjects with their subjecthood invariably influences their reactions and positionality—a challenge that is generally known but exacerbated in the case of automated scoring. If these systems are most effective when they disappear, then how can we document this liminal experience not as a state of being but as a process of becoming—of becoming aware and learning about an otherwise obscure technology?
Towards a strategy of critical companionship
The research strategy developed here takes as its starting point the observation that methods are performative (Law, 2004: 56). That is, data are never really “raw,” but shaped by the particular circumstances of their capture, reflecting longstanding asymmetries of power and assumptions in the field (Collins, 2000; Harding, 1987). This observation is particularly relevant when working with data subjects, who are likely to rethink and redefine themselves in interactions with both the scoring system and the researcher. Rather than trying to erase this influence from our research, we propose to take it seriously and turn it into an analytic resource. In doing so, we found the notion of companionship particularly helpful. What if we looked at data subjects not as phenomena for distanced observation, but as companions on a shared journey of inquiry?
The notion of companionship has been deployed across a range of academic fields. One useful theoretical articulation can be found in the work of Donna Haraway (2004: 12), who developed a dynamic and emergent notion of companionship, understood as “co-constitutive relationships in which none of the partners pre-exist the relating, and the relating is never done once and for all.” Although Haraway is mostly interested in human-animal relations, the idea of co-constitutive relationships has proven useful in a range of cases, including humans and robots (Turkle, 2006), stories and listeners (Frank, 2010), and, more recently, humans and data (Lupton, 2016). Understood as “less a category than a pointer to an ongoing becoming with” (Haraway, 2008: 16), a focus on companionship invites us to consider how also the researcher and participant are “co-evolving,” to use Lupton's (2016: 4) term, as parts of their respective projects.
The phenomenon of researchers and research subjects facing related epistemic challenges is of course not new. In ethnographic research, for example, scholars have been increasingly confronted with a situation in which “our subjects are themselves engaged in intellectual labors that resemble approximately or are entirely indistinguishable from our own methodological practices” (Holmes and Marcus, 2008: 2). In these cases, it is helpful to think of inquiry as “an analytical relationship in which we and our subjects—keenly reflexive subjects—can experiment collaboratively with the conventions of ethnographic inquiry” (p. 2). Yet, while such collaboration may be an obvious choice in highly professionalized domains like engineering, finance, art, and architecture, the case is different when it comes to people less used to this kind of work. Compounding their already precarious situation, data subjects, for example, not only lack expertise or prior training that could guide their research into scoring systems, but also tend to have less resources to engage in research. A co-constitutive inquiry following the logic of companionship must therefore take into account the imbalances of resources and power between the different actors (Cooky et al., 2018: 2). The stakes and risks are disproportionately higher for the data subject as the structurally disadvantaged party. We thus suggest that researchers approach companionship with a critical awareness of this difference—a form of critical companionship.
The idea of critical companionship was first developed in the field of nursing research. Titchen and McGinley (2003) use the term in clinical training to describe “a helping relationship based on trust, high challenge and high support, in which an experienced practitioner accompanies a less experienced practitioner on a learning journey” (p. 115). For example, a highly skilled physiotherapist may team up with a less experienced colleague and facilitate a form of “research on the job,” paying particular attention to embodied and embedded practices. While this notion nicely captures the tension between companionship and criticality, we take it one step further and adapt it to the world of automated scoring. In contrast to the formal organization of a hospital, for instance, there is no clear hierarchy of knowledge in our case. Researcher and data subject are both confronted with an inscrutable technology so that neither tends to be in a position to tell the other how it's done. Hence, while the overall idea of challenge and support continues to be useful, we are interested in how these interactions play out in the particular context of an automated scoring system. In the remainder of this article, we will explore the strengths and limitations of this methodology through our experiences with a recent study we conducted of credit repair practices in Upstate New York.
Critical companionship in action: Credit score repair in upstate New York
The following study emerged from our interest in how ordinary people, i.e. people with no prior expertise in automated scoring, experience and cope with subpar credit scores. In the United States, consumer credit scores are used in a variety of situations to predict the creditworthiness of people (Fourcade and Healy, 2013; Lauer, 2017). Increasingly, these predictions do not just concern financial credit as in the case of loans, mortgages, and insurance, but also forms of social credit as in the case of job applications, leases, and even online dating (Rona-Tas, 2017). Traditionally calculated on the basis of a person's credit history as reported to the three credit bureaus Experian, TransUnion, and Equifax, credit scoring products are offered by a range of specialized providers, such as FICO, and the credit bureaus themselves. These scores are typically distributed on a curved scale from 300 to 850 with higher scores indicating higher creditworthiness.
Consumer credit scoring is a particularly fitting case for testing critical companionship. People have limited awareness of the system's operation until they are confronted with a negative result; the system has been designed to become embedded in the lives of those it claims to measure without their active participation; it operates on timescales that are difficult to anticipate. To address these challenges, we proceeded in three stages. In Stage 1, we focused on making companions, identifying an initial set of individuals who were concerned about their credit score through a mix of targeted promotion, a screening survey, and an initial round of semi-structured interviews. In Stage 2, we focused on managing relationships with a subset of Stage 1 participants and accompanied them over a year through participant diaries, diary-interviews, and fieldnotes. Finally, we concluded the study by parting ways in Stage 3 with a round of more open-ended conversations with Stage 2 participants (Figure 1). Our goal in this paper is therefore not to report on the substantial findings of the study, but to reflect on our experience of using the idea of critical companionship as a methodological approach.
Project overview.
Stage 1: Making companions
The first step was to initiate relationships with people who might be interested in having us accompany them on their respective credit score repair journeys. To achieve that goal, we wanted to establish trust between participants and ourselves through an initial round of semi-structured interviews. Conducting interviews would allow us to obtain a basic understanding of respondents’ situations. It would also enable us to build a pool of people who might be willing to take part in the next stage of the project. In our experience, three aspects were particularly important to this process: the careful calibration of recruitment strategies, a close engagement with participants’ own plans and reasons for the interview, and the tendency among participants to portray credit score repair as a journey of discovery.
First, our main goal in recruiting was to strike a balance between two extremes. On the one hand, we were trying to avoid so-called datapreneurs, i.e. people who were semi-professionally engaged in managing their credit scores (DuFault and Schouten, 2020). On the other hand, we needed to identify participants who were at least minimally concerned about their score. In addition, we learned early on that talking about debt was generally considered a taboo in the United States. As the organizer of a local program for financial literacy noted, there tends to be frustration and a certain fatalism in how people talk about their credit score: “They don't get the opportunities to improve their financial situation because they have a bad credit score, and because they don't have a good credit score, the opportunities for improving their financial situation are minimal” (Fieldnotes, 6 June 2018).
In order to address these issues, we decided not to recruit exclusively on online credit forums such as CreditBoards or Facebook Groups. Instead, we focused on our local community in Upstate New York, where we could meet participants in person. To cast the net wide, we designed a poster with the prompt: “Are you concerned about your credit score?” While the word “credit score” broadly indicated our interest in forms of automated scoring, the prompt was kept reasonably open (“concerned”) not to give off the impression that taking part required a specific set of skills. We posted the flyer in public places, including credit unions, employment agencies, housing services offices, shopping malls, grocery stores, and on university campus. Electronic copies were circulated on local online forums like Craigslist. The flyer directed respondents to a short screening survey, in which they indicated why they were concerned about their credit score and what range that score fell in, including an option for “I don't know my score.”
In our experience, the recruiting process was largely successful, resulting in a pool of volunteers that was surprisingly diverse. Out of 55 respondents, 37 had credit scores below 700 whereas only eight had credit scores above 700; ten respondents said they did not know their score. 1 In selecting participants for the initial round of interviews, we aimed to maintain a balance of respondents in this distribution. Ultimately, we scheduled interviews with 22 participants. Among these, five did not know their score, 11 had credit scores below 700, and six had credit scores above 700. In our initial interviews, we tried to obtain a basic understanding of participants’ experiences with credit scoring. The topic guide included questions such as when and how participants first learned about their credit score; what changes they had encountered in their score since they had started paying attention; what they did to improve their score, including interactions with intermediaries like credit bureaus and credit repair organizations; and how they would describe the idea of a credit score to someone else. 2 The interviews were primarily oriented towards building trust and laying a foundation for a longer-term engagement. Overall, we found that participants with whom we had been able to develop good rapport were more likely to continue with the study.
A second insight concerned the need to pay attention to our interlocutors’ own plans and reasons for participating in the study. Already at this early stage, it was striking to see how some participants approached the interview with particular ideas of how it might be useful for their own projects. Olivia 3 , for example, who eventually participated in all three stages of the study, was a single parent living in Section 8 housing with her adult son, who recently started college. 4 Since they rarely talked about financial issues, she wanted to use the interview as an occasion for engaging with her son on the subject. 5 During the interview, however, her son rarely spoke and seemed disinterested while Olivia broke repeatedly into tears. As interviewers, we became aware of the emotional complexities of talking about credit scoring and the guilt participants felt in contending with their financial situations. Our attempt to respond with empathy and a non-judgmental attitude was an important element of building these relationships.
Finally, the interviews further confirmed our expectation that dealing with a subpar credit score was an ongoing learning process for our participants as much as it was for ourselves. Asking participants about their first encounter with their score was particularly instructive. Take, for example, Jennifer, a single parent with four children, two of whom are currently in college: I knew that I had a credit score, but I didn't know what it was. I was about 26, the first time I learned what my credit score was. I wanted to buy a house. I didn't know I had bad credit. A real estate agent ran my credit report and he said, ‘Well, you’ve got a lot of work to do in order to buy a house.’ At that point in time, I was so embarrassed. My poor credit score was a 430. (laughs) I was young, I didn't have credit cards, but it was phone bills and doctor bills and things like that. That's when I learned.
So, I went online to annualfreecreditreport.com. Checked it out… found everyone that I owed money to. […] I had about $20,000 in debt and I did pay the bulk of it. A little at a time. When I got my income taxes [refunds], I took a big chunk of money and paid down a lot of debt. Any time I had extra money, if I had little small bills, I’d pay those. My goal was to buy a house, because I lived in public housing with my children, and I didn't want them to grow up there. So, in about two years, that's how long it took me to pay off the [debt]. I was […] as aggressive as I could be. […]
I was approved for a mortgage when I was 28. With four children. Single parent. Full time job, full time student. That's all I know is, is how to survive (Jennifer, first interview, 11 August 2018).
Like Jennifer, most of our participants presented their experiences as a journey of discovery, trying to figure out the meaning and workings of their scores. Usually triggered by a major life event like buying a house, looking for a job, or requiring costly medical treatment, these inquiries tended to take up a considerable amount of time.
Although the initial conversations produced important insights, the accounts collected tended to be painted in broad strokes. When we probed participants for further details, they often could not remember what they specifically did to improve their scores. Who had they talked to? How had these conversations changed their understanding of the problem? How had they reconciled their everyday financial practices with concerns about the score? It became clear that answering these questions required more sustained engagement.
Stage 2: Managing relationships
The transition to Stage 2 marked the beginning of a longer-term relationship with ten participants from our initial pool. Given our focus on credit repair practices, we decided to engage exclusively with participants whose credit score was less than 700. In order to collaborate effectively over time, we found the diary/diary-interview method to be particularly helpful—a method that combines participant diaries with follow-up interviews (Zimmerman and Wieder, 1977; see also Czarniawska-Joerges, 2007: 66). Our interlocutors welcomed the opportunity to check in regularly and reflect on their ideas as they were already engaged—to varying degrees—in their own research. Conversely, we had a chance to stay in touch and follow up on the occasion of particular events. For the diary, we designed a password-protected online form to solicit monthly entries between October 2018 and September 2019 (Table 1).
Questions on diary entry form.
In managing these relationships, we encountered three particularly noteworthy dynamics: the mutual adjustment of inquiries through the medium of diaries and diary-interviews, the challenges of working through shared emotions and affect, and the thorny politics of expertise in researcher-participant relationships.
A first dynamic concerned the need to mutually accommodate our respective circumstances and adjust the research design accordingly. For example, having learned about our interlocutors’ personal commitments, we had anticipated that filling in the form should not take more than half an hour. Yet, as it turned out, the arrangement did not work for everyone. Whereas one participant sent us two entries every other month, others got in touch more often. Similarly, we had initially set a minimum of 1000 characters for a response. After one respondent copy-pasted the words “1000 characters is an awful lot to fill especially for the first entry in this diary hopefully I will be more accustomed to this for future entries but perhaps this minimum could be reduced somewhat?” until the count was reached (Colin, Diary entry, 17 October 2018), we decided to remove the character requirement. Questions that we thought were self-explanatory needed further elaboration. While most responses became more detailed over time, two participants left our study at the beginning of 2019. We tried to contact them several times to get a sense of why they quit, but they did not respond. Moments like these made us realize that critical companionship requires a consistent effort from both parties over time and is susceptible to failure.
A second dynamic concerned the need to work through shared emotions and affect. Especially the open-ended question at the bottom of the form became an unexpected outlet for the mixed emotions our participants experienced. Some used it to express their anger: It is EXTREMELY difficult making your score go up by a few points—when one thing makes them plummet in the double digits. It also doesn't make sense that you lower your debt by $100 and your score moves up 2 points—but you increase your debt by $10 and your score lowers by 15 points. (Melissa, Diary entry, 29 October 2018, emphasis in the original).
Others used it to talk about the instability of their finances and lives: Time and money are luxuries of envy. Whoever says money doesn't make one happy is full of untruths. Money brings security which brings peace of mind and happiness. The circumstances change, but stability is happier and healthier than going without. (Olivia, Diary entry, 22 September 2019).
Sharing these moments allowed us to unpack the affective dimension of credit scoring while being active listeners for our participants.
The diary/diary-interview approach was particularly valuable in this regard. Consider the excerpts from Benjamin's diary in Figure 2. In the Stage 1 interviews, we had observed that participants oscillated frequently and rather abruptly between not caring and caring deeply about their scores, entertaining seemingly conflicting sentiments. Yet, as Benjamin's entries show, this oscillation made more sense when followed over time. When he felt in control of his financial situation, he was hopeful about improving his score. When he lost control, he felt frustrated. During our follow-up conversations, we asked Benjamin further about his conversations with the credit counsellor and the loan specialist, the process of applying for a credit card through a gas station, the score builder program, and the mysterious collection item. These conversations were elaborate and provided us with insights that would have been difficult to get at otherwise.
Excerpts from diary entries.
Finally, a third dynamic concerned our own role as researchers in the relationship. Although we had indicated in the consent form that we were not qualified to provide legal or financial guidance, it was tempting for our interlocutors to regard us as experts just by virtue of our university credentials. Some questions were informational, such as when Olivia (Diary entry, 16 October 2018) asked about a resource we had talked about: “learning how to read my credit report is of high priority. […] I had found a link somewhere through the FICO website to learn how to do this yet am not finding that resource at this very moment now. Would you be able to redirect me to it?” Other questions were more specific. On one occasion, for example, Danielle (Diary entry, 23 July 2019) was wondering: “if the bank denies me my request for an auto loan, how will that impact my credit score?” Questions like these put us in a difficult position. Answering them would have jeopardized our goal of documenting kinds of recourse that participants relied upon; in the worst case, we might even harm our interlocutor by spreading inaccurate or misleading information. On the other hand, not responding to these questions was also not an option, not least because these conversations allowed us to explore the tactics and solutions that our interlocutors used to solve their problems. Following the idea of critical companionship, we therefore tried to find a sweet spot between credit counselling and mere emotional support. This involved treating the efforts of our interlocutors not just as data for our research, but as occasions to care for their success and learn from them as contributory partners for our research. In building companionship with our interlocutors, we continuously emphasized our own lack of expertise when talking with participants, adding caveats, and asking questions rather than suggesting a particular solution. At the same time, we also regularly reviewed and discussed concerns amongst ourselves and—in case of doubt—consulted secondary sources on credit repair including talking with a counsellor at a local credit union to avoid potential misunderstandings.
That said, it was surprisingly difficult to keep that balance, and at times our conversations did have unintended consequences. Consider the following excerpt from a follow-up conversation with Colin (Follow-up on Diary, 29 January 2019), whose credit score had dropped after a complicated divorce: Interviewer: One of the things that you mentioned is that […] my credit score is calculated in part by how much of my available credit I’m using. […] The more cards that I have with zero balance, the better it is for my overall credit score. […] Have you also thought of actually applying for more credit cards so that there are some cards which have zero balance that you don't use? Colin: Um, no. Any offers I’ve gotten, like the pre-qualified ones, they’re still charging an annual fee. […] I’m just not gonna pay for having a credit card. […] I didn't actually think about [your question], but hopefully more cards would show more credit and less used. But that's something I didn't think about doing. Interviewer: Fair enough. I just asked you because I thought that is what you meant, when you said, ‘The more cards that I have with zero balance…’ Colin:... yeah, but the more I have paid down to zero, then yes- Interviewer: Ah, okay. Colin: So, I can show that these… the better I am with it, but I’m not interested in, I guess, expanding debt. I think that’d probably lower my credit score if I could ever get on there.
As the episode makes clear, conversations about credit repair involve a good amount of speculation. In this case, our initial summary of Colin's thoughts had misconstrued his statement as a plan to obtain additional credit cards to reduce his credit utilization ratio. Colin then clarified his point and even went as far as saying that he had not considered this as a strategy. Yet, while the misunderstanding was immediately resolved in our conversation, the question must have lingered on in Colin's mind. For only three months later we received a diary entry, in which he talked about getting approved for a credit card with a $10,000 limit and a 15-month 0% balance transfer offer (Colin, Diary entry, 25 April 2019).
The example shows how a seemingly innocuous question can influence the course of action a participant may take. While Colin initially confirmed that he would not be applying for another credit card, he eventually transferred his existing debts to a new one and was also able to save on paying interest. We can speculate that this strategy had worked for Colin because he had a mid-650 s credit score at the time. For someone with a lower credit score, this strategy may well have backfired because they could have been rejected and their score would have dropped as a result.
In other words, the line between asking and advising can become inevitably blurred—an observation that speaks to a larger tension in our work, namely the challenge of caring for participants without undermining their autonomy. The scene further shows how our interlocutors were neither passive in these conversations nor were they simply following suggestions. Rather, they were charting their own credit repair trajectories and often used us as a sounding board for questions and concerns without us realizing. Managing this process in the case of credit scoring was likely easier than it would have been in other cases, in which information about scoring practices and models is not available to the same extent. In such situations, companionship will likely become more about identifying strategies for navigating the unknown rather than discussing a potential course of action as in Colin's case. In any case, the mutual shaping of the two inquiries (of researchers and their interlocutors) was central to managing relationships at this stage of the process. 6
Stage 3: Parting ways
In October 2018, we began to organize the exit interviews with the remaining eight participants. The goal of this final stage was to tie up loose ends and reflect together on the process. Broadly speaking, we were interested in how the process changed participants’ understanding of the credit scoring system; what they saw as their main disappointments and achievements; what aspects of the system they found still unclear and what advice they had for someone starting out; and how they would change the system if they could. We concluded the one-hour conversation with a discussion of our shared experiences and how they could be improved in future studies.
The exit interviews revealed a number of instructive insights into practicalities of critical companionship. To begin with, participants reported that the regular diaries and conversations had changed the role of credit scoring in their daily lives. As Maurice (Final interview, 31 October 2019) observed, “It's almost like when you have a log of your daily activities, there's a heightened awareness around it, it affects you even tangentially.” The need to follow up and generate another diary entry was experienced as motivating by participants. Danielle (Final interview, 28 October 2019), for instance, appreciated the sense of accountability that “really made me stay focused and on top of my credit, and that was an intended consequence that I was looking for.” Especially the length of the study was seen as consequential, making participants realize how long it took them to improve their score. The key is “to think about long-term goals and where you want to be,” Colin (Final interview, 30 October 2019) noted, “Otherwise you will just shoot yourself in the foot.” For most participants, the study became less about improving their score per se and more about tracking their bills and everyday expenses. Some were able to follow budgets they had made, others struggled to keep their expenses under control. One participant decided to file for bankruptcy to resolve their financial situation. Overall, however, most participants experienced some improvement in their credit scores. 7 “It is still a mystery to me how the score is calculated, but now that I know where it is and […] the things in general I have to do, [… I] feel better about myself” (Colin, Final interview, 30 October 2019). While participants thus went away with some sense of achievement, it is important to remember that the closing of the study did not mean closure for participants. A common theme in exit interviews was the sense that it had only just begun and that participants wanted to continue to improve their credit score and finances. “Just because the study is over doesn't mean [you] stop trying to work on your credit,” Danielle (Final interview, 28 October 2019) observed.
A final theme emerging from these conversations concerned the role of “feedback,” as participants called it in the interviews. While some were happy with the open-ended questions, others felt we could have provided better and more frequent comments on their diaries. Although we always were available for phone calls, email messages, or texts, our participants did not use this option frequently. Rather, they felt that future iterations of the study would benefit from more organized and formal follow-ups with in-depth questions. With hindsight, our tendency to avoid extensive feedback warrants further thought. As already discussed, we started with the goal of limiting our influence in order not to interfere too much with what we had thought of as participants’ “normal” lives. We also were concerned about providing bad advice. At the same time, we noticed that it often was impossible to keep this distance.
That said, our relationship with participants was not extractive in the sense of learning without giving back. By engaging in these relationships, we were listening to their efforts and helped them reflect on their predicament. Our conversations were not only about credit scoring; they also encompassed financial struggles, finding a new job, looking for a place to live, and other life events. Although our conversations occasionally turned into strategies, they mostly served as a source of motivation and support for our participants. Detachment, the interviews showed, was not conducive to building critical companionship. By the end of our study, we were not just researchers studying the impact of subpar credit scores on other people's lives, we actually had become invested in their success and cheered them on as they continued on their journey.
Sensibilities for studying with data subjects
What can these observations tell us about critical companionship as a research strategy? To conclude this paper, we will now extract a set of sensibilities for studying with data subjects. The notion of a sensibility is particularly helpful here because, unlike methodological rules and recipes, sensibilities acquire meaning in the empirical locations they are used and require us continually to reassess our own role as researchers. 8 In other words, sensibilities are particularly useful for dealing with ambiguous situations and dilemmas, in which there is no obvious course of action.
A first sensibility concerns the need to work with a non-binary notion of awareness. As we have seen, data subjects do not routinely think about their data subjecthood but may experience it in particular moments of breakdown and repair. Once a scoring practice has revealed itself, data subjects tend to embark on a process of inquiry, rethinking their position vis-à-vis the system. But even then, the score is usually only one aspect of their considerations and embedded in a range of other practices and demands. From a researcher's perspective, it is therefore important to account for this sporadic salience of scoring practices in the lives of those they claim to represent. As the case of credit score repair has shown, this challenge starts with the wording of recruiting materials and the specific situations in which participants are approached. It also favors a longitudinal approach that allows researchers to accompany participants on their journeys, leaving open the possibility that scoring practices may not actually be the main concern that drives someone's behavior. Moving from a binary notion of awareness to an understanding of awareness as a situated practice opens the possibility of a more differentiated view of how data subjects experience the opacity and obliquity of contemporary scoring systems.
A second sensibility revolves around the need to learn from epistemic friction and recognize how participants’ and researchers’ inquiries are inextricably linked. As we have seen, the line between asking and prompting is inevitably blurred in practice. One person's investigation becomes another person's data. The researcher learns from the interlocutor what it means to cope with and depend on automated scoring. The interlocutor uses the researcher as an expert, a counselor, or a sounding board. If we are conceptualizing data subjecthood as both a process and a practice, then clichés like detached observation or complete surveillance would not get us closer to our goal. Rather, critical companionship requires that we find a balance between these methodological extremes, not unlike previous work in STS that has combined, for instance, ethnographic and experimental sensibilities. Lucy Suchman's (2007: 121–122) study of human-computer interaction at Xerox Parc, for example, was explicitly designed to engage two colleagues in the operation of a photocopier in order to provoke a situation in which the interactions could be observed and surfaced issues that would not otherwise have been available. This phenomenon of “inducing trouble for analytic purposes” (Schwartz and Jacobs, 1979: 272) is similarly useful in a field like credit score repair, where activities are undertaken by a single person—a situation that is exacerbated by societal taboos about financial hardship and the perceived complexity of computationally generated scores. In this sense, critical companionship does rely on forms of provocation, but it also recognizes that the resulting interactions are revealing if they are analyzed as such.
Finally, a third sensibility concerns the need to navigate the ethics of companionship. Our study was a constant reminder of the many structural inequalities that characterized the relationship between researcher and participant. Whereas for Jennifer understanding credit was a question of “survival,” we could pursue the project without immediate consequences. Whereas for Danielle the struggle for financial stability never stopped, we were able to conclude the project and even benefit from writing up our findings. Whereas Colin shared experiences that helped us gain a better understanding of his reasoning, we were wary of giving feedback out of fear that we might be misleading him. In other words, the stakes involved were unequally distributed, forcing us continually to reconsider our privilege in the relationship. In order to account for this imbalance, we found a number of considerations to be particularly helpful. First of all, we realized that it was our responsibility to manage expectations. One reason participants were drawn to the study was the hope of working with real experts in the field who helped them solve their problems. Being open and upfront about one's own role and abilities should therefore be an important part of any such inquiry. Examples of some useful strategies included putting our own experience in context and making clear how unreliable our thoughts might be. We further found it useful to have a set of resources at hand that we could recommend to participants. In our case, for instance, we had developed a relationship with a local community-based credit union who offered affordable courses in financial literacy. Another way of dealing with the imbalance was to reimburse participants for their time and effort. Especially individuals in precarious financial situations appreciated this aspect as a (modest) token of appreciation. All in all, we found it helpful to see these ethical considerations not only as a problem, but also as an opportunity. Being aware of our responsibilities allowed us to be critical companions, who may not have all the answers but who can still make a contribution as active listeners or sounding boards.
Conclusion
Studying the lived experiences of data subjects is not an easy feat. It means inverting our lens from what is readily available (such as websites, systems, operators, and designers) to what is marginalized and hidden by design (such as the experiences and practices of people subject to these systems). Data subjects, we have learned, are often not supposed to be aware of their position and face significant asymmetries of power, expertise, and information. Following earlier work by scholars in the fields of critical data studies, STS, and healthcare research, we suggest a methodology called critical companionship that conceptualizes researcher and participant as companions on two partly overlapping journeys of inquiry. While not without its risks, studying with data subjects allows us to tackle the “constitutive paradox” (Spivak, 1988: 13) of studying marginalized communities, namely the fact that our work both represents and constitutes our subjects in problematic ways. 9 In focusing on critical companionship, we foreground this paradox and turn it into a resource for analysis. Critical companionship is thus best understood as a strategy for managing the interlocking epistemic projects of researcher and data subjects.
How can we bring to life the sensibilities of critical companionship and turn the challenges of studying (with) data subjects into an opportunity? If we take the challenges of limited awareness, embeddedness, and ongoing inquiry seriously, then how we, as analysts, articulate experiences of and struggles with these systems will be deeply shaped by our presence. There are no generic everyday experiences “out there” to be enumerated; there are only ongoing inquiries conducted by participants on how to cope with their predicament. As the case of credit score repair has shown, these inquires have varied levels of sophistication, ranging from self-trained datapreneurs to unsuspecting data subjects with limited awareness of their situation. Participation in these journeys cannot be accomplished through some simple rule or recipe. Rather, we need a set of sensibilities that can guide us over time and that continuously alert us to our own involvement in the process and the resulting responsibility. Sensibilities like working with a non-binary notion of awareness, recognizing epistemic friction, and navigating ethical dilemmas provide us with a useful starting point.
Somewhat ironically, documenting the experience of data subjects is much easier for automated systems than for human researchers who need to build relationships and trust and hold themselves accountable for a similar degree of access. Whether and to what extent this strategy will work in other settings is thus an open question. The ideas and sensibilities presented here may well be helpful beyond the case of credit scoring when an individual is tracked and traced. Predictive policing and insurance scoring may be very different in their respective levels of opacity and in the nature of the stakes for those involved. Eventually, however, the usefulness of critical companionship will depend on the specific circumstances of the study and how it is adapted to generate relationships that are both supportive and productive. One way or another, research strategies like critical companionship provide a much-needed counterpoint to the increasingly ubiquitous technologies of tracking and surveillance that have been governing our lives. While it is tempting to start by focusing on systems and their implications, inverting this relationship to study algorithmic systems from the margins will be a key challenge going forward.
Footnotes
Acknowledgments
We thank our research participants for their time, patience, and, well, companionship. The paper received generous feedback at the “The Hustle Economy: Race, Gender, and Entrepreneurship” workshop at the Data & Society Research Institute. Many thanks to Kyla Chasalow as well as Agnieszka Leszczynski, Matthew Zook, and three anonymous reviewers for invaluable comments on earlier versions of the paper.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by a National Science Foundation CAREER award (#1848286) and a Small Grant from the Cornell Center for Social Sciences.
