Out of the box

Abstract

Our world is being challenged by the wealth of content produced by digitization – up to now we have been thinking about the volume of information and our own cognitive limits. The rush of digitization and computerization has increased our ability to apply formulae to data, but not our ability to reason and argue. From the written word we might have been aware of the explosion in data in the significance of ‘big data’, a phenomenon which undermines the certainty of our own world in publishing and user-generated content such as the web. This article is based on the schisms in data analysis and the different philosophies of the frequentists and the Bayesians. Their philosophies illuminate a challenge from ‘big information’ – a challenge of information morality and reasonable doubt. Instinctively, we use information to remove doubt without recognizing that increasing information also changes reasonableness; the morality of information is determined by the ratio of signal to noise. ‘Smoking gun’ evidence is only compelling in that you are in no doubt as to the crime. As information increases, we assume it becomes harder to conceal the truth but forget that information itself produces falsehoods – we increase both the signal and the noise. In information poverty the essential skill is that of husbandry; in information excess the essential skill is becoming that of interpretation.