Abstract
I suppose that the first time that I became fully aware of the threats and opportunities offered by modern systems of data capture was during the first Iraq war when I worked in Ministry of Defence (MoD). That war was technologically a first in several different ways. It was the first to be fought with infrared night vision and the first to be fought with global positioning system (GPS). These two innovations were transformational because for the first time it was possible to rendezvous and fight at night in the desert – an enormous advantage for the allies over the Iraqis who did not have these capabilities.
The third big technological difference was that this was the first war to be fought with extensive satellite surveillance. However, what should have been a great advantage to the Allies turned out to be of virtually no help at all. The reason was that we had not yet learned to manage the deluge of information that literally poured down on us from the satellites. It comprised a time-series of enormously detailed images, and the only means that we had of dealing with it was the time-honoured approach to air-reconnaissance, namely, printing them out as photographs and looking at them!
The result was rooms full of frustrated photoanalysts surrounded by metre-high piles of beautifully detailed air photographs that the teams had no means of relating to each other or to anything else! In the time available, there had been simply no time to find the proper way of dealing with the data. In the end, nearly pensioned-off reconnaissance aircraft were summoned from the United Kingdom, and we were back to World War II aerial photography and the military got something that they could use! The lesson is that however good your data, unless you have a way of handling it properly, it is of no value.
For many purposes, civil or military, once the first image has been obtained, all we need beyond that is to be made aware of changes. This implies a capability to compare successive satellite passes over the same area and sound the alert if something has changed since the last pass. This is not as easy as it sounds, as no two satellites tracks are ever exactly the same, and the lighting conditions and, hence, the shadows may be different, not to mention cloud-cover variation. The challenge therefore is to develop software that can manage these variations and, for example, to be alerted to the fact that some fields are not growing the crops for which a particular subsidy was claimed!
Such problems may sound incredible in these days of digital maps and imagery that we can beam down to phones and pads and zoom in and out to almost any degree of detail that we want. The reality is, however, that as our ability to acquire observations improves so does the challenge of managing it. To add some scale to the problem, each day, roughly 9000 articles are added to Wikipedia, 145 billion emails are sent, 4500 million Google searches are made and 30,000 megabytes of data are recovered in the Sloan digital sky survey. And on and on! Clearly, there are archiving challenges, but there are enormous advantages for whoever can recover it, analyse it and exploit it efficiently.