Data ambiguity and clinical decision making: A qualitative case study of the use of predictive information technologies in a personalized cancer clinical trial

Abstract

Personalized medicine aims to tailor the treatment to the specific characteristics of the individual patient. In the process, physicians engage with multiple sources of data and information to decide on a personalized treatment. This article draws on a qualitative case study of a clinical trial testing a method for matching treatments for advanced cancer patients. Specialists in the trial used data and information processed by a specifically developed drug-efficacy predictive algorithm and other information artifacts to make personalized clinical decisions. While using high-resolution data in the trial was expected to provide a more accurate basis for action, sociomaterial engagements of oncologists with data and its representation by artifacts paradoxically hindered personalized clinical decisions. I contend that the engagement between human discretion, ambiguous data, and malleable artifacts in this non-standardized trial produced moments of contradiction within entanglement. Sociomaterial approaches should acknowledge such conflicts in further analyses of medical practice transitions.

Introduction

Dominant evidence-based medicine allocates patients with already manifested symptoms to existing treatment protocols and paths for managing disease. This approach was developed in an attempt to standardize healthcare work and the conduct of health professionals. ¹ Personalized medicine wishes to reverse this approach, and while making use of evidence as well, it tailors the treatment to the specific needs and characteristics of the individual patient. Supporters of personalization aim to change medical practice from a reactive to a preventive one, using fine-grained data, to achieve better health outcomes. ² Starting from advances in genomic and other “omic” sciences, the tailored treatment will encompass as many information sources as possible, including life style, behavior, the environment to which the individual is exposed, the molecular make-up, and population-based genetic information. Information technologies are therefore crucial for integrating all these data sources into clinically meaningful information. ³ The nature of clinical decision-making in personalized medicine changes as well, as the ultimate goal is to customize a decision to each patient while data source constantly evolves. ⁴

This article focuses on clinical decision-making in personalized medicine as a collaborative process by which physicians make sense of data through interaction among themselves and with digital information artifacts, some of which process data and others represent it. ⁵ This process is explored here in a qualitative analysis of a personalized cancer clinical trial. The trial intended to show that a personalized matching of cancer therapies can achieve better outcomes than standard treatment protocols and thus can be applied in early treatment lines.^6,7 To this purpose, processes of devising a personalized treatment for each participating patient were guided by a host of information sources: genomic (DNA alterations) and gene-expression (differential expression of genes measured in quantities of RNA molecules transcribed from them) data, clinical state and history, and risk factors of each individual patient. Computerized artifacts processed these data to match molecular alterations and drugs, indicating also which drugs are most likely to be effective for the patient. The ultimate clinical decision, however, was still vested with the oncologists, not with the artifacts.

As will be shown here, despite personalized medicine’s best intentions, the engagement of oncologists with data did not always produce more precise decisions. The case study demonstrates how, as non-standardized personalization departs from “traditional” evidence-based medical practice, ⁸ biomolecular and computerized innovations challenge accepted practices of decision-making such that they deviate or even defy routine. As this approach gains wider hold, at least in the field of cancer medicine, it signifies not only a transition in how decisions are made, but attests to the obstacles facing the wider attempt to overhaul medical practice. ⁹ On this background, I therefore seek to understand how clinical decision-making was affected by working with the data within a setting aimed to implement a personalized approach to treatment?

Related work and contribution

In focusing on decision-making processes, the artifacts and the data which artifacts processed in the case study can be viewed as supporting these decisions. Literature on clinical decision support systems (DSS) in health informatics research is ample. Pertaining to the case presented here, there are studies which focus on supporting personalized or molecularly guided decisions. Literature in this field acknowledges the critical need for DSS in personalized settings. Such systems are necessary to handle the ever-increasing quantities and variety of complex data required to tailor decisions.^10–12 Over time, systems based on patient-specific genomic assays, to be integrated with infrastructural platforms such as electronic medical records (EMR) and into clinical workflows, will facilitate the fulfillment of the “bench-to-bedside” promise of personalized medicine. In personalized medicine, these systems are thus perceived as enablers, not as mere standardized tools for knowledge accumulation and distribution. ¹³

Simultaneously, studies in this field have also pointed out the obstacles and hurdles the development and implementation of systems to support personalized decisions still face. Most of the obstacles discussed pertain mainly to the context into which such systems should be implemented, not only to systems as such. This context is not unique to personalized setting, although the volume and complexity of personalized data may amplify the effects of factors such as the perceived complexity of data,^10,12 underlying uncertainty attributed to wrong probabilistic reasoning by clinicians, ¹⁴ additional demands on physicians’ time, incomplete or conflicting data, the rapidly changing scientific evidence and recommendations, ¹⁵ and users’ limitations in understanding data analysis techniques and their representation. ¹⁶ On this background of research on decisions support systems in personalized medicine, this article sheds light on an actual instance where personalized clinical decisions were made and carried out. Hence, the analysis brought here offers an in-depth qualitative perspective not common in this strand of literature.

The types of obstacle indicated above mean that it is not solely a matter of better technical design that impedes the deployment of DSS in personalized medicine. The personalized healthcare context comprised of human and non-human actors determines whether the engagement of actors with DSS will produce better outcomes. In line with the analysis brought forth here, sociomateriality has served to as a framework for interpreting such engagements in information systems research. It emphasizes the intertwining and entanglement of humans, technology, and the material aspects of organizational settings. The basic assumption contends that entities in an assemblage such as a clinical trial do not have inherent characteristics but acquire them through emergent, relational, and temporary interactions within a specific context that provides shared meaning.^17–21 While certain strands in sociomateriality literature see humans and material surroundings as indistinguishable, other scholars have called for re-balancing human- and object-oriented perspectives to overcome sociomateriality’s ontological vagueness. ²²

While sociomateriality is characterized by holistic assumptions, ²² this article highlights challenges, tensions, and contradictions discerned in instances where humans and data representations disentangle. Tensions of this kind have been discussed in interpretations of the materiality and the sociotechnical nature of health information systems. A key and recurrent issue these studies have underscored is the discrepancy between work itself and its representation in data in health information systems. This gap is felt particularly in how clinicians relate to the tools meant to streamline information and support treatment decisions, as expressed in resistance, workarounds, adaptations, and limitations in knowledge production.^23–26

This article discusses similar tensions. Data, artifacts, and their representations can stand apart from the articulation work of clinical decision-making and can potentially confound it. However, many of the aforementioned studies discuss processes of standardization of medical practice and decision-making. Thus, a further contribution is made by showing how the work of formulating a personalized treatment enhanced the discretion of oncologists, while they engaged with the non-standardized setting of the trial and with patient-specific data represented by tailor-made artifacts. Practitioners acting in the setting of the trial, though adhering to certain conventions and constraints, worked beyond the “trajectory” ²⁷ of managing complex disease. They worked more on data—less on people—to contemplate decisions. Ironically, while personalized data were meant to provide a sound basis for decision, the ambiguity of data itself coupled with the way in which oncologists used the tools available to them, confounded processes of making personalized clinical decisions.

Method

The case on which I focus here is the WINTHER ²⁸ personalized cancer clinical trial that took place between 2013 and 2016. This multi-site, non-randomized clinical trial tested the effectiveness of a method for matching drugs for advanced cancer patients based on their DNA or RNA alterations. I took a single case-study strategy, which allows for an in-depth qualitative interpretation of the relations between components within the case.^29–31 Here, I focus on decision-making processes in the trial, not on the subsequent treatment of patients. The latter aspect was not investigated for ethical reasons and the focus on the engagement of physicians with computerized artifacts did not necessitate disturbing patient privacy.

The main source to trace clinical decision-making processes in the trial consisted in recordings of on-line sessions of a Clinical Management Committee (CMC; the trial’s term for so-called Molecular Tumor Board ³² ). The committee included oncologists, trial administrators, and bioinformaticians. Some 18 h of CMC sessions were recorded and transcribed. In most of these sessions, specific cases were discussed and decisions made. I was therefore able to follow decision-making instances as they took place. To understand the rationale of practitioners, semi-structured interviews were conducted with three out of six of the principal investigator oncologists, ²⁸ with the two trial administrators and with the two bioinformaticians. In addition, numerous trial documents were collected. Access to all these materials was gained upon a confidentiality agreement with the WIN Consortium managing the trial. All identifying details were omitted, including names, cancer, and drug types.

To support customized trial clinical decisions, computerized artifacts intended to process patients’ data were developed and maintained by two bioinformaticians and a biologist who also functioned as one of the trial administrators. The bioinformaticians were also responsible for processing the molecular data of each patient and feeding them into an on-line portal later used by oncologists. The portal presented practitioners with several types of data and information and allowed for documenting decisions. The portal included several components organized by cases:^28,33

1. A clinical datasheet for each patient, entered by each patient’s treating oncologist.

2. A report on genomic (DNA) alterations identified by advanced sequencing technologies for each patient in a sample of her tumor or metastasis and matched with existing molecular-targeted drugs.

3. A list of drugs ranked according to their predicted efficacy score. The list was based on two underlying tables. The first is a list of human-relevant, cancer-related drugs, which could also be experimental compounds, along with genes associated with each drug. The list was derived from the Comparative Toxicogenomics Database (CTD). ³⁴ The second is an extensive table produced for each patient, containing a list of genes found in two matching samples: one taken from the tumor (the same sample as above) and one from a normal tissue. The table posits the expression of every gene (RNA) in both samples and calculates the discrepancy between them. A specifically developed algorithm filters the list of gene expression for the genes associated with each drug and produces a predicted efficacy score for that drug. The score is a weighted average of the discrepancy between the expression of the genes associated with the drug in the cancerous sample compared with the normal sample. The ranked drug table displayed on the portal was specific for each patient. Doctors could drill down into the detailed results of each drug, see the genes with which the drug is associated, and access the table of raw gene-expression data for each patient.

According to the trial’s designated workflow, oncologists would convene in each on-line session, assisted by bioinformaticians and trial administrators, to formulate a personalized treatment for each patient. The case would be presented by the patient’s treating oncologist starting by reviewing her clinical history and current state. The oncologist then reviewed the genomic alterations’ report of the patient. If alterations were identified and matched with existing drugs that could be accessed at the treating center, the oncologist, after discussion, would decide on that course of action (a path termed arm A). For about half of the patients, no such alternations were found. In that case, the oncologists then went on to review the table of ranked drugs and discussed the results produced by the efficacy algorithm. Following this path (termed arm B) eventually allowed the oncologists to tailor a treatment for every participating patient.

The collected materials were scrutinized by employing content analysis techniques. Textual sources were examined using thematic analysis. Transcriptions of CMC sessions and interviews were examined by using a content analysis technique specific to conversation and speech, which is called “Membership Categorization Analysis” (MCA). MCA explores how parties to a conversation enact their world and actions through the use of categories and attributions, drawing on social vocabularies and conventions.^35,36 Actual application of the technique to the analysis of transcriptions was performed on the ATLAS.ti 7 qualitative data analysis software. ³⁷ The context in which CMC sessions took place could be inferred by analyzing trial documents, interviews, and information accessed on the portal.

Results

The articulation work needed to reach a decision in the case study was not new or unique. Neither is the contingent, non-routine character of healthcare work mediated by artifacts.^38,39 The “personalized” framework, however, differentiates this particular setting from others. Within it, accepted categories such as cancer histology and treatment lines were partially abandoned. Computerized artifacts developed for the trial were intended to streamline the work of formulating decisions, and attempts were made to standardize the workflow and the representation of processed data. Yet, the effect was that of enhanced emphasis on the knowledgeable, “uncontrolled” discussion of human actors negotiating personalized information. As one oncologist related to the arm B algorithm,

We don’t even know what we’re debating. [The algorithm] just gave us an answer, and that’s it. Now it increases uncertainty because we’re people, and we’re saying, well, how did it come up with that? … So I think, in order to quiet the debate, it would be better if the scoring tool was like this, it says, this is your drug, and this is why this is your drug. So we would understand. So it would make us happier. (Interview, Paris, 29 June 2015)

That is, despite their knowledge and experience, oncologists had to choose between drugs upon cues given to them by computerized artifacts. As willing participants in this personalized experiment, they were still aware of the limitations of the data analyzed and displayed, as well as of their own limitations in deciphering the meaning of results, as demonstrated in the following sections.

More information: human knowledge and the malleability of digital artifacts

Molecular data specific to patients, while reflecting the complexity of cancer, are not sufficient in providing oncologists in the trial with information and knowledge necessary to support personalized decisions. In the following example, taken from one of the on-line sessions, the molecular biologist requested that more substantive information on the function of the genes targeted by drugs in arm B be added to the portal:

“Molecular Biologist:

I think that we should see if we can put some more valuable information, which should be good, because here it is not so easy, if you recall the drug you have, just the issue of the drug, but I think it will be good that we have the differential gene-expression and the target which are [???]. Do you think it is easy to put?

Bioinformatics lab assistant:

Yes it is … I can put the target gene symbol. But is it a ligand or …

Molecular Biologist:

No, it’s fine. I mean if you have the gene symbol its fine, we can understand what it is and the fold change [the difference in gene expression of the gene between the two aforementioned samples].” (CMC 04/11/2013)

This example resonates with the need to better “understand” “why” these specific drugs are offered. The demand for more comprehensive information is expressed here in two main ways. First, as in the example above, is the perceived need to provide additional context. The result is the request to modify the representation of data on the portal. Second, knowledge embodied in the experience and expertise of oncologists is not always compatible with how the trial was set up. Oncologists were informed with technological and scientific advances in the field. Such advances go beyond the trial’s initial definitions of drug pools. This results in pressures not only to alter the artifact mediating data but also to open the knowledgebase on which the trial rests. In the example below, after reviewing arm A results, oncologists seek a solution for the patient discussed in the results of arm B. One of the oncologists expresses dissatisfaction with the result:

“Oncologist 1:

You know, none of the [immunotherapeutic] agents never come up. Is there a reason for that?

Oncologist 2:

I think they weren’t put in the database [of drugs]. Are they in the database?

Bioinformatician:

… to be sure …we didn’t update the list, we didn’t change the list of drugs, so any drug that came after the trial began is not in our database.

Oncologist 1:

But I think [immunotherapeutic] agents, they never come up and they’re such powerful agents.” (CMC 30/03/2015)

The database referred to here is the pool of drugs derived from the CTD as mentioned above. The ability and will to alter the artifacts to display more information and to expand the knowledge on which the trial is based act against external conventions of how a trial should be conducted. This can potentially undermine the credibility of trial outcomes. Such pressures are almost inherent to medical work. However, personalization based on molecular data renders these characteristics of work much more contingent. The result is a willingness to “personalize” the trial itself by altering underlying data, mediating artifacts and the way to work with these two entities. A division of labor emerged regarding this issue. While oncologists stuck to their role as clinicians and users of information, bioinformaticians acted as mediators of the meaning of biomarkers, as in the first passage above, and as gatekeepers of trial conventions, as in the second passage. Thus, as artifacts and knowledgebases were malleable but did not have the ability to independently evolve, changing the ways these artifacts work depended on differing human understandings.

Uncertain clues: ambiguity and flexibility in the use of data

Not only do physicians regard data and the ways it is represented in a flexible manner, molecular data itself challenges the physicians’ ability to reach a decision. In the following interaction, the vagueness of information as presented in the gene-expression ranked drugs table of arm B brings the treating physician, Oncologist 3, to question the very logic of the trial:

“Oncologist 3:

… and anyway I admitted [the patient] to the trial so now I need to go through the data from the trial, and there’s nothing suggesting to treat with mTOR [gene] inhibitor, so I won’t treat [the patient] with mTOR inhibitor.

Molecular Biologist:

Well, I mean, [drug 1] or [drug 2] is in a position number 15, and the score is not very high but it’s anyway in the top 15, so this is allowed in second line and this is classic.

Oncologist 2:

Well, based on recent reports, immunotherapy has the best chance in somebody like this, compared to pretty much anything. Is there any way you can give [the patient immunotherapy] in combination? That has the highest report response, also responds fast.

Oncologist 3:

I will work in this line, that is going to be opened in 3 or 4 months, but it’s for first line patients, and … so to give an immunotherapy, that we don’t have any data from the genomics and from the gene-expression assessment … I think if [the patient] didn’t get into the trial, I wouldn’t try to get [the patient] immuno, but [the patient] is in the trial so I need to give [the patient] something based on the trial, right?” (CMC 29/09/2014)

During discussion, one of the oncologists suggests treating the patient with a molecularly targeted mammalian Target Of Rapamycin (mTOR) inhibitor. However, the rules of the local treatment center prevent oncologist 3 from administering this drug to a patient at this advanced stage of the disease. More importantly, the data presented do not support this choice. The physician has therefore to follow the suggestion of the algorithm. The preferred course of action, as mentioned by oncologist 2, would be administering immunotherapy, yet the ranked drug table points to two other drugs ranked in low positions. If the physician could have anticipated this situation, he says, he would not have put this patient on the trial in the first place. The fact that data do not provide a clear indication as to the preferred course of action both hampers the decision and casts doubt in the eyes of oncologists over the structure of the trial and its benefit to patients. The rigid framework of the trial (which, as seen above, is guarded by bioinformaticians) is meant to render the results scientifically credible. Yet it posits the continuing preference for human decision-making ⁴⁰ against representations of data. While this is characteristic of the case study and cannot be viewed as inherent to personalized medicine, at play here are larger medical paradigmatic forces that create local tensions.

Molecular data and disruption of clinical decision-making

The position expressed by the oncologist quoted at the outset sheds light on instances in which oncologists do not readily accept the results of data processed by artifacts. Limited trust in results converges with artifacts and knowledge malleability and data ambiguity to undermine confidence in decisions. This is demonstrated in the following example, where oncologist 3 questions the results in arm A, which leads consistently to a certain treatment for a certain type of caner:

“Oncologist 3:

But I think that 60 percent of the patients in the trial are KRAS mutations, suggesting automatically that each patient will go to arm A and get, let’s say [a KRAS targeted drug].

Oncologist 2:

That’s a good question, I don’t think it should, but is there data for MEK inhibitors in [the patient’s cancer’s] KRAS?

Oncologist 4:

I think there is a problem of the trial, so we committed to do these matchings … I’m not sure, there’s conflicting data, so I would agree [that] in [a type of] cancer they haven’t shown anything, but in these tumor types … I think with this patient we’ll have the same situation, if you can get [a KRAS targeted drug] that’s fine, that would be arm A, and if not the options in arm B …

Oncologist 3:

What are we deciding in this case? Molecular Biologist, what do you think?” (CMC 29/09/2014)

The recurrent pattern of arm A results raises the dilemma of how to treat the patient: whether with DNA-matched drugs (arm A) or with RNA-matched drugs (arm B). Such a dilemma is an expression of making personalized decisions based on indeterminate data in an organizational framework that strives to maintain both the flexibility of personalization and clinical trial standards. Such a tension is demonstrated in the following example, where the treatment indicated by the data originating from molecular analyses can be allocated to both arms:

“Oncologist 1:

The real question will be whether this is actually an arm A or arm B patient, because the PDGFR beta mutation is clearly being targeted by arm A and the VGFR is arm B, and you’re getting both in one drug, so the choice for the patient is excellent, but there’s a bit of a conundrum for the study that we’ll have to work out in the end.

Bioinformatician:

May I request when we call a decision, you make a note of it.

Oncologist 1:

Yes, I think that there’s going to have to be some retrospective review that includes the idea of what happens if you targeted both arm A and arm B, did you get it in the hands with that.” (CMC 07/07/2014)

This matter of classification is not merely a discursive one, since what is at stake here is the trial as a proof-of-concept for the personalized approach more generally. In the passage above, the drug offered by the molecular analysis fits into both categories of genomic- or transcriptomic-guided treatment. The dual result represented by technology must be reckoned with by non-technological reasoning. Oncologist 3, reflecting later on the first case above, explicates the rationale guiding the physicians in this process:

“When you have more data, maybe you can use them in a more logical way that connects also to the disease type on arm B. Sometimes you look at arm B and look at the numbers and you see they are really over-over expressed. [So] you say, ok, let’s instead of doing the obvious … Drug1[for example], doesn’t inhibit the KRAS [DNA mutation], it inhibits a pathway with KRAS. That’s the whole point, KRAS activates a million other things … OK, so we held back the MEK [mutation] a bit [with Drug1], but we didn’t inhibit all the other things it activates, you see?” (Interview, 12/01/2015)

On the one hand, since the trial procedure invested the final decision with human actors, under these circumstances, their discretion was enhanced by the need to deal with the indeterminacies created by data and its representations. Yet on the other hand, a need for “some retrospective review” is felt. As oncologist 3 attested, “There’s no structured aspect in the teleconferences, except for who’s presenting the case and then each one talks. Maybe something more structured should be built in, more with quality control, on people’s knowledge…” (Interview, 12/01/2015). Moreover, Oncologist 3 contends that “Cancer is a moving target, it changes all the time” (Interview, 12/01/2015). This echoes the view of molecular data as disruptive technology that challenges conventional cancer medicine ⁴¹ in the sense that clinical action has to constantly adjust to the changing multiple molecular characteristics of disease. The discursive notion of “disruption” is used by proponents of personalized medicine to denote these effects on existing medical approaches, yet this use has practical consequences. ⁴² As reflected above, two types of logics are experienced as disrupted: the “logical way” of connecting disease to gene expression results on arm B (i.e. clinical–human interpretation of data) and the digital and statistical logics of processing data and representing results (embodied in the trial’s artifacts and the workflow designated for using them). The convergence of these logics ends in a “conundrum.” This localized disruption, again, takes place within the larger perceived disruption to “traditional” approaches.

Conclusion

The WINTHER trial, as a case study, puts in its center, by its procedure and stated goals, the engagement of physicians, data, digital artifacts, and representation of results. By doing so, the trial realizes the perception of molecular technologies and DSS as “enablers” of the personalized medical promise. Yet in this localized and situated realization, obstacles and challenges rise from that same engagement. Paradoxically, if humans made clinical decisions, not technologies and data, the very interaction of both renders decisions imprecise.

This reasserts a critical–realistic understanding of sociomaterial processes in healthcare. Tensions as those demonstrated here attest to possibilities of disentanglement. ⁴³ Although human and material agencies are both constitutive of personalized medical work practices, the WINTHER trial shows this is not always the case, and thus, the functional assumptions of agential sociomateriality should be qualified. These tensions arise partly from the formal constraints of a clinical trial conflicting with its own content of testing an unstandardized treatment method. More importantly, at work here is the incongruency between the vision of a broader contextual personalized medicine and the lack of context in its practical realization.

Lack of context in the trial can be seen first in the focus on molecular types of data, while not considering other sources of information that personalized medicine originally aims to encompass. Data itself are represented by artifacts in the trial without biological context, leading to difficulties in interpreting knowledge by physicians, who wonder “how did it [the arm B algorithm] come up with that?” Lack of certain aspects of context, coupled with working on ambiguous data in the process of tailoring decisions, and less with patients, deprives the physicians of the embodied experience necessary to clear-up the uncertainty in decision-making. ⁴⁴ The analysis presented here suggests therefore that to complete the full cycle of personalized medical work, the micro-level instabilities and tensions require further examination, in regard to the personalized approach itself and the digital artifacts it employs, as both make further strides into clinical practice.