James Jurin and the avoidance of bias in collecting and assessing evidence on the effects of variolation

James Jurin – a brief introduction

James Jurin (1684–1750) was an 18th-century physician and polymath. ¹ As a scholar at Trinity College, Cambridge, he became acquainted with the work of Sir Isaac Newton, of which he became an enthusiastic advocate. After leaving Cambridge, he travelled with Mordecai Carey, later Bishop of Killala and Achonry (in Ireland), to continental Europe. They attended the lectures of Hermann Boerhaave at Leiden, then Europe’s leading medical school. Jurin later returned to Cambridge to study medicine. Thereafter, he practised medicine while continuing to develop his interest in science and mathematics, corresponding with many of Europe’s leading scientists (Voltaire referred to ‘the famous Jurin’ in the Journal des Sçavans, cf. http://gallica.bnf.fr/ark:/12148/bpt6k56605t/f603).

Under the pseudonym ‘Philalethes Cantabrigiensis’, Jurin engaged in a pamphlet war (the ‘Analyst controversy’) with Bishop Berkeley. Berkeley had argued that Newton’s account of the calculus was incoherent, which prompted Jurin’s defence of Newtonian mathematics. Jurin was held in high regard as a physician (though his treatment for Robert Walpole, Britain’s first prime minister, was controversial). He became president of the Royal College of Physicians shortly before he died in 1750. This was facilitated by his position as Secretary of the Royal Society (1721–1727).

In this article, I examine how Jurin’s work, particularly his work on variolation, introduced or developed several mechanisms that allow him to make a far more reliable assessment of a medical intervention than had been hitherto undertaken, and which were subsequently influential.

Jurin’s study of variolation

Variolation was the practice of inoculating a person, usually a child, with pus or scab from a smallpox blister to give them a mild case of smallpox. That gave protection against potentially fatal naturally occurring smallpox thereafter. In the early 18th century, this practice was referred to as (smallpox) inoculation, but later became known as ‘variolation’ to distinguish it from Jenner’s inoculation using cowpox (‘vaccination’) in 1796.

Variolation had been introduced to Britain in 1721 by Lady Mary Wortley Montagu, who had encountered the practice in Istanbul, where she had been the British ambassador’s wife. It was quickly adopted by a small number of enterprising physicians, including Thomas Nettleton, a Yorkshire doctor. ² Nettleton wrote to the Royal Society describing his experience of variolation. ³ He was concerned to establish two propositions. First, that the illness produced by variolation really was a mild case of smallpox (and so can be expected to confer the same subsequent immunity to smallpox as that resulting from naturally acquired smallpox). And second, that the risk of death from variolation is less than that from naturally acquired smallpox. Nettleton drew up a table showing the numbers of individuals who had contracted smallpox in several towns in Yorkshire, Lancashire and Cheshire and of those, how many had died. This showed that nearly one-fifth of those with natural smallpox had died, whereas of the 61 individuals who had been variolated, none had died.

Nettleton’s letter was received by Jurin in his role as Secretary of the Royal Society, who arranged for it to be published in the Society’s Philosophical Transactions, which had previously published reports on variolation as it was practised in the Ottoman Empire. ⁴ Jurin decided to embark on his own study of the value of variolation and, particularly, its safety. Was an individual who underwent variolation more or less likely to die than one who did not and so who risked a natural infection? Since it was appreciated that what is at issue is risk or chance, it was clear to Jurin, as it had been to Nettleton, that what was needed was more information about the number of those who had been variolated and the proportion of those who died, along with data on mortality from natural smallpox – as Nettleton noted in his letter, ‘I am very sensible you will require a great number of Observations, before you can draw any certain Conclusions’.

Jurin was perfectly placed, intellectually and socially, to undertake such an investigation, since he was both a doctor and a mathematician and was well connected through his role in the Royal Society. He used his position to solicit information from across Britain about individuals who had been inoculated with smallpox. Jurin specified the details that he required (name, age, manner of inoculation; whether the inoculation was successful in producing smallpox, and if so after how long; whether the patient survived or died; and so on).

Jurin published his results first in a letter addressed to Dr Caleb Cotesworth, a Fellow of the Royal Society and of the Royal College of Physicians, and physician to St Thomas’s Hospital, printed first in January 1722/23 in the Philosophical Transactions of the Royal Society, and then reprinted as a separate pamphlet. (The given month ‘January 1722/23’ is the month we would call January 1723 [‘New Style’] but was given at the time as January 1722 [‘Old Style’], the difference resulting from moving the start of the new year from 25 March to 1 January, which in England occurred in 1752.)

The pamphlet included reports from Jurin’s correspondents regarding inoculations carried out in Massachusetts and in Pembrokeshire. In 1724, Jurin published a further pamphlet, entitled ‘An Account of the Success of Inoculating the Small Pox in Great Britain. With a Comparison between the Miscarriages in that Practice, and the Mortality of the Natural Small-Pox’. This was dedicated to the Princess of Wales in acknowledgment of her support for inoculation (encouraged by Lady Mary Wortley Montagu). Jurin updated the pamphlet three times, with additional data and discussion concerning inoculation in 1724, 1725 and 1726. A final version was prepared, covering the years 1727 and 1728, by John Gaspar Scheuchzer, like Jurin a polymath and a Fellow of the Royal Society and of the Royal College of Physicians.^5–9

Jurin notes the controversy surrounding inoculation and claims that his aim is simply to furnish the facts relevant to the debate in an impartial way. He does indeed provide the data in as objective a manner as possible. It is nonetheless clear that Jurin regards the case for inoculation as very strong. Jurin’s aim is to gather and present data to quantify three risks: (1) the risk of death due to inoculation among people who are inoculated; (2) the risk of death from natural smallpox among uninoculated people; and (3) the risk of death among people who contract natural smallpox.

The data to answer (1) (the risk of death from inoculation) were obtained from Jurin’s correspondents, as described above. In his Letter to the Learned Caleb Cotesworth, Jurin’s data imply that the mortality from smallpox inoculation is at most 1 in 91 – the ‘at most’ here reflects the fact that deaths following inoculation are not unequivocally the result of the inoculation rather than (for example) natural smallpox acquired contemporaneously with the inoculation. Jurin chose to assume that these deaths should be attributed to the inoculation in order that its opponents should not question the data used. In later publications, details were given of the fatal cases so that the reader could make their own assessment of the cause of death. Tables were given showing the different calculations one would make depending on how many deaths one attributed to the inoculation. In Scheuchzer’s final tally, 845 persons had smallpox by inoculation, of whom 17 were suspected to have died as a result (i.e. just over 2%).

The data needed to answer (2) were obtained in the first place from the bills of mortality, which tabulated deaths and their causes. Jurin gathered data going back to 1667 (leaving out 14 years where smallpox and measles were not distinguished). This gave a mortality rate in the population at large of 1 in 14 (just over 7%). By the time of Scheuchzer’s 1729 edition, it was possible to provide useful data on deaths during the years since the introduction of inoculation, so that deaths from inoculation were being compared with contemporary deaths from natural smallpox. This attributed 1 in 12 deaths to smallpox (i.e. 8.3%).

Jurin was also concerned to calculate the third risk, the risk of dying from smallpox if one contracted the disease. This was in part to show that inoculated smallpox was much less dangerous than naturally acquired smallpox. Another reason was that the lifetime risk of dying from smallpox, (2), underestimates its dangers and so underestimates the benefit of inoculation. That is because many of those who are listed in the bills of mortality are infants and very young children, who will not have been exposed to smallpox at all. Since the risks and benefits of inoculation concern older children and adults, Jurin wanted to calculate a risk of dying from smallpox for those who were exposed to smallpox. So Jurin removes from his figures deaths from conditions that affect children up to the age of two at most (this is a remarkable 39% of the total). The risk of dying from smallpox for the remainder is almost 12%. This therefore is also a lower bound on the risk of dying from smallpox if one contracts the disease naturally. Jurin provides another route to the latter risk, by gathering data from physicians such as Nettleton, who surveyed households in local towns to find out how many had contracted smallpox in the previous year and how many had died from it. The figure he calculates from these data is a risk of almost 19%. By 1729, the accumulated data give a risk of 16.5%.

Measures taken by Jurin to avoid bias

To answer some scientific questions, scientists are able to devise experiments that deliver clear results that unambiguously favour a particular hypothesis and rule out relevant alternative hypotheses – what Bacon ¹⁰ called an ‘experimentum crucis’. In other cases, the evidence can be varied, unclear and ambiguous, especially in healthcare: Does the patient really have this disease? Did they really recover? Can we be clear when they recovered? etc. One of the obstacles to gaining knowledge in such cases is the fact that they are liable to biases. Biases occur when researchers or their methods are disposed to treat the evidence in such a way that is not conducive to the truth. Bias in the assessment of a healthcare intervention can be especially problematic, because, as mentioned, the evidence is often unclear; because there are entrenched opinions; because healthcare is related to other social and economic matters; and because health and disease are emotive issues. All these were present in the assessment of variolation. It was not always obvious whether an inoculation had ‘taken’, producing a genuine case of smallpox, or whether the resulting case was a mild or severe one, or whether a death was caused by the variolation. This made the data especially susceptible to biased interpretation. Many people had strong prior opinions on variolation, e.g. that it was attempting to interfere with the will of God, or that it was a suspicious foreign, feminine practice. The idea of deliberately giving a child a case of smallpox was understandably alarming, and anxiety could turn to anger if the case was severe or led to death – the evidence of such cases would be far more salient than that of others. It was thus important for Jurin to avoid bias in his research. The avoidance of bias meant that he could rely on his results – he could know whether variolation was safer than the risk of natural smallpox; and it meant also that he could present results that would be persuasive to others, results that were as close as possible to being as persuasive as those from an experimentum crucis.

Quantification

The use of numerical data to understand health and disease was pioneered by John Graunt, ¹¹ whose work analysing the bills of mortality was a model for others, such as Edmond Halley, who produced the first statistical life tables. The work of Nettleton, Jurin and Scheuchzer on variolation seems to have been the first to use quantitative evidence to assess a medical intervention. Since the question concerned a comparison of risks, it could not be answered by a qualitative approach. Indeed, the concept of ‘chance’ was itself one that had only in recent decades been given a mathematical treatment, generating a quantitative rather than merely qualitative concept. The replacement of imprecise qualitative reports by numerical data meant that questions could be addressed which had previously not been addressed, or which, if answered, were liable to unconscious bias or lack of comparable comparison groups.

For example, because smallpox could scar or even blind those who survived a bad case, it was important that inoculation gave only mild cases of the disease (but genuine cases nonetheless). Many doctors held that the risk of a severe eruption could be reduced by the use of laxatives. Sir William Watson ¹² undertook a series of experiments to test this hypothesis and also to test different types of material used in the inoculation. ¹³ But what counts as a ‘severe’ eruption? If the experiment were to be repeated elsewhere, how could one know that other researchers would regard the same cases as equally severe by their standards? Watson eschewed qualitative assessment in favour of counting the number of smallpox pustules, thereby ensuring comparability of cases while also reducing the room for observer bias.

Tables

Jurin was clear in writing to his correspondents that he required information concerning all those inoculated, thereby avoiding bias arising from the selective use of data. The principal results of this research were laid out in tables to permit easy calculation of the risks in question. The use of tables for presenting data was already well established in Britain, ¹⁴ having been used liberally by Graunt. Tables have many advantages, such as the concise and easily accessible presentation of information. When tables present the collation of numerical data as evidence in the assessment of interventions, they have the further important benefit of reducing the danger of bias. A table helps avoid confirmation bias, whereby people focus only on evidence supporting an already favoured hypothesis, ignoring counter-evidence. And it avoids bias from the availability heuristic, whereby people form judgment about a complex matter on the basis of evidence that comes easily to mind (e.g. because it is dramatic, or recent, etc.). Presented in summary tabular form, no case exerts more influence on the conclusion than any other; in presenting all the available data, ‘unfavourable’ data are as likely to be included as favourable data.

Transparency and open data

The presentation of data in tables does not eliminate all biases. The inoculators who sent their results to Jurin could have introduced biases in their reports. To minimise bias from this source, Jurin took care to obtain detailed case reports rather than just the numbers. This enabled him to make uniform judgment (e.g. as to whether an inoculation had succeeded in giving the patient smallpox). He even asked for names (which were not published) so that any controversial cases could be investigated later. Jurin advertised the fact that he retained all his original data, which would allow particular cases to be examined in more detail or to be followed up if necessary. This was important, for example, in finding out whether someone who had natural smallpox had previously been inoculated.

Imprecision, borderline cases and uncertainty are especially liable to lead to bias. Jurin handled these by publishing the details of such cases so that readers could form their own judgements rather than rely on his. For example, it was not always clear whether a smallpox death following inoculation was due to that inoculation or due to a concurrent natural infection. So Jurin not only published the details of such cases but also produced tables that gave different risks depending on how many of the cases of smallpox one regarded as caused by the inoculation. Jurin may have been especially keen to be transparent to avoid charges of bias. He was acutely aware of the effect of confirmation bias and the availability heuristic on public opinion when it came to the controversial practice of variolation. As one correspondent, Edward Edlin, wrote to Jurin, ‘If anything goes amiss or seems to do so, the world presently sings of it with all the Aggravation imaginable, but on the other hand many Successful Experiments are I believe buryed in silence’. ⁸ Jurin was therefore very active in pursuing cases that placed the safety of variolation in doubt.

Blank forms

Another 18th-century innovation that promoted the reliability of data and of the inferences drawn from them was the introduction of the blank form. A template, with headings and questions already printed and space for respondents to write in their observations, was particularly helpful where research required input from many collaborators. In his correspondence with inoculators, Jurin requested detailed case studies and had to follow-up many reports for important missing information.

In 1723, Jurin pursued another project that aimed to relate weather and health. He resurrected and adapted a much earlier (1667) proposal of Robert Hooke’s for the collection of meteorological data. In collecting information from across Europe and North America, Jurin gave his correspondents detailed instructions about the data they were to collect and how it was to be presented, along with a specimen form that they could copy. ⁸ Half a century later, in 1780, a very similar project was undertaken within France and its colonies by Félix Vicq d’Azyr and the Société Royale de Médecine. ⁸ Vicq d’Azyr was able to use the bureaucracy of the Société and of the French state to ensure a greater degree of engagement than Jurin had been able to achieve, and he was aided by the use of a form that had dates and times printed in advance, making it difficult for participants to avoid making and recording the required observations. In 1731, Francis Clifton had proposed a standard tabular form for recording medical case histories. While his innovation was not widely adopted in the medical profession, the idea that standard forms and tables would promote the uniform, comparable and unbiased collection of data was nonetheless established. In his ‘An Attempt to improve the Evidence of Medicine’, George Fordyce ¹⁵ promoted the use of a standard table for recording patient histories, with benefits both to patients and to science. In 1795, Johann George Christoph Siebold was doing likewise at the Julius Hospital in Würzburg. ¹⁶

Three decades later, in Paris, Pierre Charles Alexandre Louis noted Fordyce’s proposal in his influential work on bloodletting. ¹⁷ Louis remarked that whereas Fordyce had the general practitioner in mind, which raised difficulties in the promotion and use of his scheme, Louis himself had the advantage of working in a hospital where it was easier to enforce such uniformity. The use of blank forms and standardised processes of collecting data can be seen as components of a broader process of the systematisation of record taking, collation and publication, originating with books of unconnected patient histories and leading to carefully constructed case series, which, argue Hess and Mendelsohn, ‘became a basic operation of medical knowing’. ¹⁶

Conclusion

The 18th century saw several developments that made the collection of evidence in clinical medicine more systematic and so less liable to bias and other forms of error that would undermine the reliability of inferences drawn from that evidence. ¹⁴ Jurin’s work on variolation, inspired by Nettleton, and carried forward by Scheuchzer, Watson and others, appears to have been the first example of the careful use of numerical data to assess a medical intervention. That work gave a reliable answer to the question of the relative safety of variolation compared with natural smallpox, an answer that could not have been gained by using a purely qualitative approach. An important reason why a qualitative approach is unable to give a reliable answer to questions such as this is that it is liable to bias (conscious or unconscious). An important and distinctive feature of Jurin’s work is the care he took to minimise particular sources of bias. Following Jurin, as Tröhler ⁹ shows, the use of numerical approaches and the avoidance of bias became well established in medicine in 18th-century Britain. Jurin has a strong claim, it may therefore be argued, ¹⁸ to have established clinical medicine as a science.