‘Happy failures’: Experimentation with behaviour-based personalisation in car insurance

Abstract

Insurance markets have always relied on large amounts of data to assess risks and price their products. New data-driven technologies, including wearable health trackers, smartphone sensors, predictive modelling and Big Data analytics, are challenging these established practices. In tracking insurance clients’ behaviour, these innovations promise the reduction of insurance costs and more accurate pricing through the personalisation of premiums and products. Building on insights from the sociology of markets and Science and Technology Studies (STS), this article investigates the role of economic experimentation in the making of data-driven personalisation markets in insurance. We document a case study of a car insurance experiment, launched by a Belgian direct insurance company in 2016 to set up an experiment of tracking driving style behavioural data of over 5000 participants over a one-year period. Based on interviews and document analysis, we outline how this in vivo experiment was set-up, which interventions and manipulations were imposed to make the experiment successful, and how the study was evaluated by the actors. Using JL Austin’s distinction between happy and unhappy statements, we argue how the experiment, despite its failure not to provide the desired evidence (on the link between driving style behaviour and accident losses), could be considered a ‘happy’ event. We conclude by highlighting the role of economic experiments ‘in the wild’ for the making of future markets of data-driven personalisation.

This article is a part of special theme on Insurance Personalization. To see a full list of all articles in this special theme, please click here: https://journals.sagepub.com/page/bds/collections/personalizationofinsurance

Introduction

Insurance companies are developing health and car insurance policies that make use of Big Data technologies, ranging from new analytical technologies such as predictive modelling and Machine Learning to new data generation devices such as telematics, wearable health trackers or smartphone sensors, known as the Internet of Things (Spender et al., 2019). In the past few years, the American-based start-up health insurer Oscar gained a lot of attention in the news media as a hegemonic example of these contemporary developments in the insurance industry (McFall, 2019; Macmillan, 2015; Nusca, 2015). Another early adopter in this space is Vitality, which has partnered with Apple to offer discounts on the Apple Watches to consumers, providing benefits and rewards in the form of services and discounts for adopting a healthy lifestyle (French and Kneale, 2009; Gröger, 2014; Krempl, 2015). Both Oscar and Vitality are exemplars of ‘behaviour-based personalisation’ in insurance, involving practices aimed at ‘the personalisation of products and goods where markets and services are increasingly focused on the behaviour and lifestyle of actors, particularly in insurance markets’ (Meyers and Van Hoyweghen, 2018a).

The attention these examples receive shows that the ‘disruptive’ potentials of Big Data and predictive modelling in insurance prompt strong expectations, ranging from high hopes to major fears. Big Data comes with the promise of reducing insurance costs, more accurate pricing and the personalisation of risk to support healthy lifestyles and safe driving behaviour (Geneva Association, 2018). These promises of Big Data in insurance are characterised by the idea of ‘Pay As You Live’ (Ernst and Young, 2016), where the personalisation of insurance policies is based on the behaviour of individuals. At the same time, the promise of Big Data in insurance is associated with worries regarding issues of discrimination, unaffordable premiums, and a decline in solidarity (Lyon, 2018; O’Neil, 2016; Peppet, 2011). As such, Big Data is considered to be a real game-changer for ‘insurance-as-we-know-it’, an industry that has always relied on large numbers of data for the selection and assessment of risk (Hacking, 1990). Private insurance differentiates between risk groups through risk selection where clients are charged a premium that statistically reflects the level of risk they bring into the insured pool according to the principle of ‘actuarial discrimination’. This principle implies that private insurance charges higher premiums, or even exclude applicants, if the risk (e.g. medical conditions, age, profession, disability) clients represent is statistically higher than average. The emergence of new types of data and analytical tools poses important technical, societal and regulatory challenges for insurance (Blassime et al., 2019; Ewald, 2012). How these challenges will play out in European insurance markets, however, is not (yet) clear.

The potential of Big Data technologies cannot ‘just’ be applied to insurance but requires the making of new insurance markets. In Europe, many car, life and health insurers have begun to develop Big Data-enabled personalisation through building marketing campaigns, offering smartphones and providing other showcases to explore personalised pricing in insurance (McFall, 2019). In car insurance, for example, insurers try to launch Usage-Based car Insurance (car UBI) products (‘telematics insurance’), where the client’s driving behaviour is constantly monitored for calculating driving style dependent premiums and/or services. However, the development of these novel insurance products occurs more slowly than anticipated by the industry (Swiss Re, 2017). One major obstacle in the insurance industry for employing these new types of personalised data, such as driving style data, is regulatory restrictions in regard to antidiscrimination and data protection. Buying and selling insurance in the EU is governed by national contract laws, yet subject to European antidiscrimination and data protection regulations (Gellert et al., 2013; Henckaerts et al., 2019, Marelli et al., 2020). As stipulated in Belgian contract law, insurers are allowed to differentiate prices if they can provide ‘objective evidence’ of the differences in losses between risk groups, while not violating antidiscrimination law (Fontaine, 2017). This means that insurers have to provide actuarial evidence on the relation between particular risk factors (e.g., age, profession, type of car) and the experience of loss (e.g., the occurrence of car accidents) in order to differentiate prices.

The same requirement applies to Big Data-driven personalisation in insurance (Drechsler and Benito Sánchez, 2018). Insurers need to provide actuarial evidence on the relation between, e.g., driving style and the experience of loss before they are allowed by law to discriminate on the latter basis. Yet without collecting these data in real-time situations it is impossible to build an evidence base for the use of telematics data in car insurance. This means that insurers currently find themselves in a catch-22 where the usefulness of these data is unclear until they are proven to be useful. ¹

To move beyond this paradoxical situation in insurance, insurers have started to launch economic experiments to provide solid evidence on the usefulness of Big Data-enabled behaviour-based data. This attempt to tackle this particular catch-22 is also referred to as ‘the quest for the holy grail in insurance’ (interview, Innovation Manager, Reinsurance Company, 14 July 2015). Through experiments with behaviour-based personalisation, insurers hope to find this ‘holy grail’. In this article, we investigate the role of economic experimentation in the making of new insurance markets enacting Big Data-enabled personalisation. We document a case study of a car insurance experiment, launched by a Belgian direct insurance company in 2016 to set up a ‘driving style study’ of 5000 participants over a one-year period in order to establish evidence for the relationship between driving style and the company’s loss ratios.

In our argument below, we first discuss how the role of market-making and experimentation has been theorised in sociology of markets and Science and Technology Studies (STS). We claim that this approach offers an insightful take on studying the practices of data-driven personalised insurance markets. After presenting our methods and empirical materials, we outline the main findings of our study, reporting on the experiment’s background and aim, the various ways in which the study was set up, its manipulations and interventions, and the final outcomes of the study. Drawing on Austin’s distinction between happy and unhappy statements (Austin, 1962), we reflect on how to evaluate the ‘success’ or ‘failure’ of this economic experiment. As such this case study highlights the role of economic experimentation ‘in the wild’ in the making of future Big Data-enabled markets.

Making markets through experimentation

To study how practices of economic experimentation contribute to the making of future insurance products and markets, we use insights from STS and the sociology of markets. This approach has focused on the ‘constructedness’ of markets (Callon, 1998, 2017), stressing the performativity of economics and market devices in the enactment of markets (Mackenzie, 2008). This in turn led to a research program investigating the socio-technical practices of ‘markets in the making’ (Callon, 2007a) without pre-assuming the stability of these market practices (Ariztia, 2018; Neyland and Milyaeva, 2016).

Within this research program, ‘market devices’ are central to the construction of markets and their actors, generating properties of markets that play an important role in the making of economic markets (Muniesa et al., 2007). Devices such as websites, and blogposts, for example, perform expectations on future markets (Meyers and Van Hoyweghen, 2018b). We are interested here in two more, interrelated market devices, namely ‘market research’ and ‘economic experiments’ (Muniesa and Callon, 2007). Considerable attention has been paid to market research which informs economic actors on a state of affairs while at the same time generating opportunities to intervene in markets (Muniesa, 2014). The best-known forms of market research tools are focus groups (Lezaun, 2007), opinion polls (Igo, 2011) and consumer tests (Teil and Muniesa, 2005). Economic experiments are forms of market research to test and at the same time perform characteristics of new economic objects (Tironi and Laurent, 2014). Muniesa and Callon (2007) formulate this as follows:

[E]xperimental activities are research activities in the sense that they aim at observing and representing economic objects, but also – and quite explicitly – in the sense that they seek to intervene on these economic objects: to seize them, to modify and then stabilize them, to produce them in some specific manner. To experiment is to attempt to solve a problem by organizing trials that lead to outcomes that are assessed and taken as starting points for further actions. Experimentation is action and reflection. (Muniesa and Callon, 2007: 163, emphasis added)

The value of economic experiments is to observe the reaction of potential clients to newly developed products in situations quasi-representative of ‘real’ market-situations. These experimental situations enable markets, as for example in insurance, to test product features without having to meet all the requirements of fully fledged insurance products. Similarly, economic experiments contribute to the production of new knowledge on markets and products without having to fulfil all the requirements demanded by academic standards of economics: economic experiments, in other words, produce ‘wild economics’ (Callon, 2007a). Economic experiments are knowledge practices that (may) employ standards and procedures to produce and evaluate (knowledge on) ‘economic things’ which differ from the standards used by academically ‘confined economists’. Confined economic research (such as actuarial mortality tables based on historical population cohorts, Turnbull, 2017) produces evidence through the controlled and standardised manipulation of variables in stable test environments. Such levels of stability, control and standardisation are not always obtained in economic experiments, due to different priorities, material constraints and standards deemed necessary by economic actors. This, however, does not imply that ‘economics in the wild’ does not produce evidence. Its evidence is ‘different’ because the main objective of the economic experiment is to be useful for economic decision making on future markets (Camic et al., 2011).

Muniesa and Callon (2007: 164) propose to focus on three elements in the study of these experimental economic practices, namely: the site of experimental practices, the manipulation imposed on the object of study and the forms of demonstration employed. Laboratory experiments, the so-called golden standard of experimental research, are the most confined type of economic experimental practices. These experiments are conducted in a highly controlled setting (site), the interventions are calibrated and purified (manipulations), and the resulting evidence has a high internal validity (demonstration). In economic experimentations ‘in the wild’, these boundaries are less clear-cut: It is harder to find a good test site where uncontrollable environmental characteristics are not too disturbing (site), to make sure that experimental interventions are as clean as possible (manipulations) and that the observations of change can be attributed to the experimental set-up (demonstration). By presenting a case study of experimental practices of behaviour-based personalisation in car insurance, we aim to contribute to a better understanding of how these economic experiments ‘in the wild’ act as ‘action and reflection’ for the making of future insurance markets.

Methods and materials

To study the role of experimentation for the making of Big Data-enabled personalisation in insurance markets, we conducted an empirical case study as part of a wider-ranging research project on behaviour-based personalisation in European insurance. By studying experimental practices in car insurance rather than in health insurance, we follow a ‘detour’ made by the insurance industry itself. Car insurance is considered a ‘harmless’ safe space to experiment with behaviour-based personalisation and seen as a way of prototyping applied to the more sensitive field of health insurance in a later stage (Meyers, 2018c). The case study documents an economic experiment by a Belgian direct insurance company, hereafter called Royal Direct. The experiment consisted of setting up a scientific study, the driving style study (in Dutch: ‘rijstijlstudie’), which was launched on 25 January 2016 and aimed at tracking the driving behaviour of 5000 participants through smartphone sensors in return for a 20% premium discount on the premium. The case was selected because it offered a unique opportunity to study empirically how an economic experiment with personalisation in insurance comes about. The case acts as an exemplary case (Flyvbjerg, 2011; Swanborn, 2010) highlighting elements that are observable in other practices of experimentation. The driving style study involved the collaboration of five partners: the insurance company Royal Direct, Makecents – a technology start-up providing driving style and lifestyle profiles based on smartphone data, SafeDriveSave – a company specialised in providing driving-style coaching and ‘sustainable behavioural change’, Satellite – a communication and design company, and STAT, the data analytics team of a Belgian university.

To study how the economic experiment in car insurance was performed, we conducted semi-structured interviews and performed a document analysis. The interviews were held before, during, and after the experiment by the first author between March 2015 and March 2017. We interviewed the different actors involved in the driving style study, consisting of CEOs, commercial directors, creative directors, technology directors, managing directors and VPs Mobility, with educational backgrounds in psychology, engineering, photography and computer science. Some of these actors were interviewed twice. The interviews were held in Dutch and recorded and transcribed ad verbatim. One respondent did not give permission to be recorded. A total of 10 interviews were retained for analysis. The names of the interviewees were pseudonymised to their professional title, and the companies were given fictitious names. We also performed a document analysis of written sources consisting of publicly available information from the company’s website on the driving style study. This material included a general ‘introduction’ into the ‘driving style study’, a list of Frequently Asked Questions, information on the premium reduction when participating in the study, an advertisement video, as well as a list of advertorials published in a Belgian newspaper. We analysed these materials (interviews and documents) abductively (Tavory and Timmermans, 2014), including a ‘back-and-forth’ analysis between the research literature, the data collected for the specific case study, and the data of the broader research project (Meyers, 2018c). In the following sections, we outline the results of our empirical case study. First, we present the context in which the ‘driving style study’ was set up, as well as the original experiment’s aims and objectives. Next, we shift the attention to how the choice for smartphone sensor technology had particular consequences for the way the experiment was set up, and we also show the various manipulations and interventions imposed by the experiment on, for instance, on the objects of study. Finally, we discuss the results of the study and reflect on the evaluation of the experiment and on whether or not it was a failure.

Results

Setting the site: Looking for the holy grail to enlarge the playing field

In Europe, the most developed insurance markets for car UBI are Italy and the UK (Swiss Re, 2017). These insights from such established foreign markets cannot be simply applied in Belgium due to different insurance legislation, road infrastructures and risk portfolios. Belgian insurance providers needed to establish ‘actuarial evidence’ on the relation between driving style behaviour and loss. If prior to 2016, many Belgian insurance companies launched some test cases and marketing campaigns, mostly targeted to young drivers (see, for instance: AXA, 2013; Baloise, 2015; DVV, 2014), these had not resulted in clear evidence yet. In January 2016, Royal Direct, a Belgian direct insurance provider, announced the launch of its driving style study, aiming to track 5000 clients over a period of at least one year, in exchange for driving style coaching and a 20% premium discount. Until then, Royal Direct was best known for its ‘kilometerverzekering’ (kilometre insurance), a product of UBI launched in 2006. This insurance policy has differential premiums based on the self-reported distance driven. By analysing the data from this risk portfolio, including loss statistics and the self-reported distance driven per year, an evidence base had been built over the years (interview, Commercial Director, Royal Direct, 31 March 2016). Royal Direct managed to attract new customers through their kilometerverzekering indeed, but as this product was mostly interesting for people ‘who drive less than average’ (interview, Commercial Director, Royal Direct, 31 March 2016), the company came to realise that it had reached its market potential. Looking to increase its market share, the idea of setting up the driving style study gradually took shape:

To answer our growing ambitions, we were looking for new ways to attract the other half of the playing field too, that is, those drivers who really drive a lot, or at least more than average. Over time it quickly became clear that the solution would be telematics. (interview, Commercial Director, Royal Direct, 31 March 2016)

To enlarge its ‘playing field’, Royal Direct got interested in testing the waters of the new hype of telematics and personalised pricing. But Belgian insurers needed to comply with the law in providing actuarial evidence for personalised pricing in car insurance. In order to move beyond ‘pure guesswork’, Royal Direct decided to set up a real-life study, to try to establish this evidence of a link between driving style and loss statistics:

Everything you do before you have observed 100,000 driving hours is pure guesswork. The driving style study is a way of dealing with this problem. Instead of starting with differential premiums based on driving style – on which we had no actuarial base whatsoever – we decided to offer some discounts as a reward for participating in the study, so we could start to collect these measured driving hours along the way. (interview, Technology Director, Satellite, 16 April 2016, author’s emphasis)

The driving style study performed a twofold goal: generating data (to demonstrate a relation between style of driving and loss ratios) and attracting new clients. The only requirement to participate in the study was the purchase of a car insurance product with Royal Direct, or, if already a client, to link the client’s policy to the study’s platform. The platform consisted of a smartphone app and an online profile in order to generate data, visualise driving style indicators (breaking, accelerating, cornering, speed and phone usage) and to provide driving style coaching. Participating in the study was rewarded with a 20% discount on the premium. Once a client had chosen to participate in the driving style study, the client/participant could download the driving style study app and a beacon was sent to the client/participant by mail. The beacon was a small device to put into the insured car, connecting the client’s smartphone app through Bluetooth. The beacon, used to identify when the driving style study participant is driving the car, activated the smartphone sensor’s tracking software.

The driving style study was a first visible step towards a Belgian car insurance market enacting behaviour-based personalisation. The prospective Belgian market can be considered a ‘site of experimentation’ with the driving style study acting as an in vivo experiment (Tironi and Laurent, 2014) into the future market of data-driven personalisation in insurance. In order to establish the actuarial evidence for personalised pricing, the study had to be set up in real-time, ‘along the way’, without the guarantees of stability and controllability as in ‘confined’ insurance actuarial studies. In setting up the study, the experiment became a form of ‘trials of strength’ (Latour, 1987), where premature evidence based on (possibly false) assumptions was put to the test to become more ‘resistant’ by accumulating new data:

We drew a banana shaped figure, based on very raw and limited data, because no insurer knew what the correlation was between driven kilometres and loss ratio. At the time, we did not perform a study beforehand. We just drew our banana. In the end, the banana was wrong, it happened to be a linear curve. But nowadays we are one of the few in the insurance industry who know what the relation really is, and that is taken into account in our premiums today. We think the same will happen with the driving style study as well, because the logarithms are becoming more ‘right’ when more data comes in. (interview, Commercial Director, Royal Direct, 31 March 2016)

These ‘in vivo’ characteristics of the experiment made the driving style study a ‘risky event’ (Harraway, 1997) when it comes to making it successful, as we will illustrate in more detail below.

The choice of smartphone technology and its ramifications

In order to establish the relation between driving style and car accident losses, the experimental set-up had to be designed to facilitate the generation of the driving style data. The driving style study was not conducted in the stable ‘confined’ environments of, e.g., a laboratory or mortality statistics, but in a real-time, dynamic in vivo setting of Belgian bumpy roads, daily traffic jams, and lived, distracted Belgian car insurance clients. Describing this set-up clarifies which options were taken, including their consequences for how the experiment unfolded. A first decision had to be made on the type of driving style tracking technology to be used. In general, there are three types of technological solutions in the European car UBI market to measure and visualise driving behaviour: black boxes, dongles, and smartphones. While black boxes (devices installed beneath the car’s bonnet) and Dongles (devices plugged into the OBD connector) are considered more reliable and more accurate for measuring driving style, they are still quite expensive to be exploited. This is why Royal Direct decided to work with smartphone sensors, where the data can be captured by a piece of technology already owned by the drivers themselves:

There are some technological constraints of smartphone sensors. It is clear that the built-in black box and the dongles are technically more correct. But the question is how technically correct you want to be. Today, I know virtually nothing about the driving behaviour of my policyholders: Nothing! [English in original, authors]. If the use of this smartphone technology allows me to know 90% of their driving behaviour, that will be 90% more than I know today. If I can increase this to 100% with a black box, it will provide only a little more insight while it will cost much more. So, we are not looking for the technically perfect measurement. And that is the reason we went for the smartphone sensor data. (interview, Commercial Director, Royal Direct, 31 March 2016)

The accuracy of the data to be generated was not an absolute prerequisite in the choice of the technology. The costs of the data collection were an important parameter too. The data of smartphone sensors (including the driving style profiles) was seen here as ‘good enough’ (Kitchin, 2014; Mayer-Schönberger and Cukier, 2013) and considered as valuable when comparing to a situation where no experiment at all would be conducted.

The choice for smartphone tracking technology also affected other elements in the design, such as the choice in routing algorithms, use of visualisation and calibration techniques, and the clients’ anticipated reactions. Since smartphones have a limited battery power, Makecents – the start-up company generating the data from the smartphone sensors – had to be cautious to avoid ‘demanding too much’ (interview Chief Product Officer and Founder, Makecents, 11 March 2015) from the participant’s smartphone battery. GPS tracking is an energy intensive process since it measures dynamic, real-time data, which are dependent on environmental conditions. In order to save energy, the smartphone’s location of the smartphone was therefore tracked on an interval basis. The trajectory between measured locations was simulated by means of routing software (interview, Technology Director, Satellite, 16 April 2016):

[We] provide reliable measurements. That does not mean that we have to measure every second of every trajectory. It is for instance hard to locate a device through GPS when dealing with tunnels or broad-leaved trees right after it has rained. (interview, VP Smart Mobility, Makecents, 10 May 2016)

A concrete example of such a measurement bias was the situation at the railway station in the Belgian city of Leuven, where an underground tunnel connects the city’s ring road:

If you drive around the railway station of Leuven, typically a speeding violation will be registered, because the GPS tracking software assumes you are driving above the ground, while actually driving through a tunnel. Suppose it measures your location right before you enter the tunnel, and you leave the tunnel only five seconds later [see Figure 1., authors], the software tool assumes that you drove all the way round [see Figure 2., authors] and that you finish a trajectory which normally takes five minutes in only five seconds. This constitutes a strong speeding violation. […] This involved such a measurement bias that we cannot afford to punish people based on these things. (interview, Partner and Founder, Satellite, 7 April 2016)

Figure 1.

Route taken by driving in tunnel (actually driven).

Figure.2. Route taken by driving above ground (assumed by the driving style study’s routing algorithm).

For these reasons, the developers tried to find a way to hide the measurement’s (in)accuracies for the study’s participants. Once the drivers’ data was extracted and ‘cleaned’ by Makecents, these data were not translated into one singular ‘driving score’. For each of the measured variables (acceleration, breaking, cornering, speed and phone usage), an ‘event’ was constituted instead whenever a threshold was exceeded. These events were expressed as an average per 100 kilometres:

In the driving style study app, events are measured and counted, but they are expressed as for instance the number of breaking events, or the number of acceleration events per 100 km. Things can go wrong but after a while these biases will be averaged out towards a correct measurement. (interview, VP Smart Mobility, Makecents, 10 May 2016, authors’ emphasis)

Unlike in other experimental practices of behaviour-based personalisation in car insurance (using black boxes and dongles) (Meyers and Van Hoyweghen, 2018a), the specific events (when and where they occurred) could not be consulted by the individual clients/participants. Royal Direct considered the instalment of such an immediate feedback infrastructure to be too risky, as the measurement biases (due to the smartphone technology) could become a source of distrust for the customers/participants during their participation in the study:

Technically, we could have built a dashboard where events can be consulted but the problem is that because of the technology we use, we could be wrong in one out of ten cases. […] The whole reputation of your application would implode if things go wrong, … and they will go wrong. So we chose very explicitly to make it impossible for the client to consult past drives. This ensures that it will remain reliable. After all, if you have to admit that you are wrong in one out of ten cases, this would cause your reliability to drop instantly. (interview, Commercial Director, Royal Direct, 31 March 2016, authors’ emphasis)

Communicating ‘inaccurate information’ about the users’ driving style measurements (e.g., by wrongly reporting speed violations or assuming different routes than actually driven) was considered as detrimental to the trust of clients/participants in the experiment. To retain the clients’ trust in the experiment, specific technologies were ‘inscribed’ (Akrich, 1992) to prevent clients from running away (from the experiment, as well as from the company). The trust of customers is always important in market situations (Van Hoyweghen, 2014), and particularly in this experiment, as it was an in vivo ‘real-time’ event, Royal Direct was directly dependent on its clients/participants to continue producing driving style behavioural data.

The empirical material above exemplifies how the choice of the smartphone technology contributed to the specific forms of demonstration and intervention in the driving style study. The decision to go for the ‘cheap option’ of the smartphone sensors, which was considered to be ‘reliable enough’ (interview, Commercial Director, Royal Direct, 31 March 2016), had ramifications for the further design of the experimental set-up. This required the manipulation of other technologies and devices to make the smartphone technology ‘fit’ in the experiment. To cope with data quality concerns, individual drivers were given no access to scores. This manipulation did not only include data generation and analysis technologies, but also entailed the (further) moulding of the study’s objects themselves, that is, the real-life economic actors as drivers, as we will illustrate next.

Gradually unfolding driving style – Keeping the object of study triggered but alive

The driving style study also included a specific ‘driving style coaching’ part. Driving style coaching is a recurring theme in experimental practices of behaviour-based personalisation, as it allows individuals to adjust their behaviour to a better drive or lifestyle (Meyers and Van Hoyweghen, 2018a). This form of ‘experimental manipulation’ was set up in such a way that it aimed at finding a balance in the attention of the users. When participants started the experiment and were offered the driving style study app, they did not receive access to all of its functionalities (e.g. measured indicators, driving style advice) from the very beginning. These functionalities were in fact presented gradually, one after the other. For all of the measured parameters – breaking, accelerating, cornering, speed and phone usage – a metric was calculated. These metrics were not made visible from the start (as would be the case with other car UBI products). There were good reasons to make these parameters not immediately visible:

You could say on day one: ‘dear client, or dear driver, read this information and everything will be fine.’ But that is not how it works. We chose first of all to make the participants feel at ease with the driving style parameters. (interview, Managing Director 2, SafeDriveSave, 12 May 2016)

During the first months, the parameters were made visible only one by one, in order to keep the clients/participants ‘triggered’ to continue to use the app and at the same time be attentive to the road:

The big challenge in such a project […] is to keep the participant triggered, if you are going to monitor solely through online technologies. People are not capable of changing their driving behaviour overnight. Driving behaviour is too complex for that. […] When it comes to driving behaviour there are a lot of aspects to be taken into account, step by step. Doing it at once is not smart because it would increase maybe the risk of accidents because people do not pay attention to the road. In the first phase you have to raise the awareness among the participants that they have a driving style, that it can be measured, and that it is relevant for their safety. (interview, Managing Director 2, SafeDriveSave, 12 May 2016)

To raise the chances of a successful experiment, participants had to learn first of all to know that they actually have a driving style. But this learning (through the app, the parameters and the coaching) can be (too) distractive, so there is a risk, literally, of killing the object of study. To prevent this, specific technologies were again inscribed to keep the participants’ attention to their driving, while making them acquainted with their driving style behaviour. ‘Step by step’-devices may stimulate learning while preventing actors of becoming too distracted. This is why the participants exposed to the measured driving style parameters – breaking, accelerating, cornering, speed and phone usage – only ‘one by one’ (interview, Managing Director 1, SafeDriveSave, 25 April 2016). During this first period, no coaching was performed. After this, ‘reference scores’ for the driving style parameters were communicated to the individual clients. These scores were however not presented in relation to other users. This provided the opportunity for participants to start adapting their driving style behaviour already. This change in behaviour was called the ‘mere-measurement effect’ (interview, Managing Director 2, SafeDriveSave, 12 May 2016, English in original). Once the participants became acquainted with all these parameters, a second period in the experiment started, in which the individual scores were visualised in relation to those of peers. Finally, only after another round of further acquaintance with all the devices, the actual coaching started:

After having established trust in these new functionalities, we are going to set some goals. Next, we start to expect some goals for the different subgroups. And by then we find ourselves in the last three months of the year already. And during those last three months, we are going to actively coach the participants. (interview, Managing Director 1, SafeDriveSave, 25 April 2016)

The practices of driving style coaching, which can be considered as ‘experimental interventions’, were used to help the participants realise that they have a particular driving style, and to offer participants personalised advice to generate ‘sustainable behavioural change’, as was the mission of the company SafeDriveSave. Again, we see how particular technologies are inscribed in the experiment to mould the object of study. The gradual implementation of the coaching trajectory was intended to keep the participants ‘interested’ in the driving style study, to make them ‘aware’ of their driving style behaviour and to allow them to adapt this behaviour in such a way that they were not abruptly distracted from their attention to the road. This continued ‘entanglement’ in relation to the app by its users was crucial for the driving style study in order to generate driving style data. This again shows how the objects of the experiment needed to be manipulated to make the experiment ‘work’. After all, without participants, the experiment could not succeed in finding the ‘holy grail’ of personalised pricing in insurance – in establishing a link between driving style and car accident statistics.

Action and reflection at work – The driving style study as a ‘happy failure’?

The driving style study by Royal Direct aimed at tracking 5000 policyholders during a one-year period in order to build evidence on the relation between driving style and loss experience, the so-called ‘holy grail’ of Big Data-enabled personalisation in insurance. From the start, this was seen as an ‘ambitious goal’ by the partners (interview, Partner and Founder, Satellite, 7 April 2016). A year after the study’s launch, we contacted the partners again to find out about the final results. To our surprise, the study’s goal of recruiting 5000 participants was not reached and the data could not be analysed by the University data analytics team STAT. In hindsight, the study’s aim was characterised as a ‘mission impossible’ (interview, Partner and Founder, Satellite, 7 April 2016) in convincing 5000 car drivers to participate. In the end, only 243 participants were recruited in the study (interview, Commercial Director, Royal Direct, 4 January 2017), which was considered ‘a total failure’ by some of the actors (e.g. interview, Partner and founder, Satellite, 17 March 2017).

One of the reasons mentioned for this low number of participants was that the communication about the study focused too much on its marketing aspects (gaining clients by offering 20% discounts) rather than on the study’s scientific part. This communication strategy made that the driving style study did not stand out as something unique to which people wanted to contribute, as it seemed to be ‘just another’ cheap insurance product, in a market abundant with price-based marketing campaigns. This small number of participants was considered problematic because ‘massive amounts of data’ (interview, Technology Director, Satellite, 16 April 2016) were needed to train the algorithms and to build reliable models based on the study-generated data. To provide statistical evidence and give objective proof to legitimate price differentiation, the study needed to measure, at least, 100,000 driving hours:

To make the actuarial link between all sorts of measurement variables and how many car accidents occur, because that is what this study is all about, they [Royal Direct, authors] needed to measure 100,000 driving hours to make a statistical link. (interview, Technology Director, Satellite, 16 April 2016)

Moreover, without the input of the university data analysts, it was not possible to make the generated data ‘validated and cross checked’ (interview, Commercial Director, Royal Direct, 4 January 2017, English in original). This way, the driving style study did not succeed in producing the ‘holy grail’ of telematics for the Belgian market. Does this mean that the involved actors considered the investments in the driving style study as completely wasted?

The STS-inspired sociology of markets has been strongly influenced by American pragmatist philosophy, with authors such as William James, John Dewey and John Austin (Callon, 2007a; Meyers and Van Hoyweghen, 2018b; Muniesa, 2011). The pragmatist philosopher JL Austin (1962), who originally coined the notion of ‘performativity’, ² made a distinction between happy and unhappy statements or performatives. He did this to go beyond the positivistic idea that statements can be only either true or false:

[T]o be ‘true’ or ‘false’ is traditionally the characteristic mark of a statement. […] Besides the uttering of the words of the so-called performative, a good many other things have as general rule to be right and to go right if we are to be said to have happily brought off our action. What these are we may hope to discover by looking at and classifying types of cases in which something goes wrong and the act […] is therefore at least to some extent a failure: the utterance is then, we may say, not indeed false but in general unhappy. (Austin, 1962: 12–14, emphasis in original)

Just like performative utterances, the success or failure of economic experiments is not limited to their capacity of producing truth statements. Economic experiments are performative market devices that contribute to the making of future markets by producing knowledge on economic markets and products, but also by developing and testing future economic products or ‘economic things’ (Muniesa, 2014). In a way similar to Austin’s remark on qualifying statements as either happy or unhappy, Muniesa and Callon (2007) argue that economic experiments can be seen as performative ‘action and reflection’. In their view, economic experiments should not (only) be evaluated on their capacity to produce standardised forms of economic knowledge. It is relevant, then, to could ask: was the driving style study a happy or an unhappy experiment?

Our above discussed empirical observations demonstrate how the driving style study should not be seen as a total failure despite its incapacity to produce the holy grail (that is, evidence on the relation between driving style and loss ratios). First, by setting up a novel ‘site of experimentation’, the driving style study provoked an intervention in the Belgian insurance market. Unlike before, premium discounts were offered now as a reward for participating in the driving style study, thereby aiming to attract new customers for Royal Direct. Secondly, through the set-up of the experiment, a driving style data generation device and a platform were developed. With the driving style app, an infrastructure was built that made (constitutive) choices on how to generate driving style data (e.g. the choice for smartphone sensors, measuring with intervals), how to visualise driving style parameters (e.g., averages of the last 100 km, unveiling parameters one by one), how to take into account possible measurement errors in order to build trust (e.g., by not enabling to consult previous drives or showing particular ‘events’). Thirdly, the experiment established how to offer driving style coaching, ensuring the attention of the user to realise sustainable behavioural change while not risking to distract the driver’s attention to the road (e.g. by gradually unfolding his/her driving style). This infrastructure can be used again for future initiatives by Royal Direct to enact behaviour-based personalisation. Finally, the experiment developed a platform for gathering driving style data and exploring the possibilities of establishing ‘sustainable behavioural change’ through driving style coaching. This coaching intervened in the participants’ habits of driving, trying to get their attention while not distracting them. Overall, the driving style study ‘made a difference’ (Mackenzie, 2008) and was ‘not unhappy’ since it consisted of, and resulted in, ‘practices of action and reflection’.

The way economic experimentation opens up opportunities for ‘action and reflection’ was also made explicit by the involved actors. Most obviously, the action and reflection of experimentation is done by producing evidence. As mentioned, STAT, the University Data Analytics team, was not even activated to perform scientific analyses of the driving style data (to find actuarially relevant evidence of the relation between driving style and car accident losses). However, SafeDriveSave wanted to know whether their coaching had some effect on the driving behaviour of the participants:

Satellite had a quick look at the data. They were very curious and they showed us one or two graphs which demonstrated that the driving behaviour of people using the app had improved. I cannot really say that their behaviour improved significantly, but it improved. […] This is not validated, not cross-checked [English in original, authors], whatsoever. It is a very, very basic graph that would indicate that the driving style study works [authors’ emphasis], at least a little bit. (interview, Commercial Director, Royal Direct, 4 January 2017)

This further exemplifies that, although no sufficient data was collected to prove the link between driving style behaviour and car accident losses, the involved partners acknowledged the experiment’s ‘happiness’ in producing some link between driving style tracking/coaching and behavioural change, which indicates that the experiment ‘worked’ after all. They considered these results as instructive for the future exploration of behaviour-based personalisation in insurance. Furthermore, another company was consulted to ‘reflect’ on what Royal Direct could learn from the driving style study. With the help of a Dutch consultancy firm, hereafter called PromissoryInsight, an ‘innovation trajectory’ was set up after the study to ‘learn from the experiment’ (interview, Commercial Director, Royal Direct, 4 January 2017). A first ‘innovative idea’ that the consultancy firm found was ‘not to be afraid of younger drivers’. This is quite an ‘obvious’ idea in the field of car insurance, even according to the Commercial Director of the Royal Direct himself:

We arrived at the idea, very obvious – but most good ideas are obvious – that the whole insurance industry is acting like an ostrich with regard to young drivers. […] Our idea, then, was: let’s turn this around. If we have to be different than others, let us make it known; ‘Young people, come to us! No problem, you won’t pay excessively’. There are some conditions, however, such as that your parents have to come with you. We will develop a new campaign targeted to young drivers. But an insurance product which links the premium to driving behaviour will probably not be offered any time soon. (interview, Commercial Director, Royal Direct, 4 January 2017)

The original study’s ambition was to build a telematics product with differential premiums based on driving style behaviour, so that the ‘whole playing field’ (interview, Commercial Director, Royal Direct, 31 March 2016) could be targeted (rather than merely who drive driving less than average, for which the kilometerverzekering had been advantageous). While this original idea had to be dropped after the experiment, the scope of the original experiment was adjusted, from the ‘whole playing field’ to ‘young drivers’. At the same time, the site of the original experiment was shifted, from personalised pricing to niche targeting. In other words, reflection by the actors on the experiment resulted in some ideas of a new car insurance marketing campaign targeted to young drivers.

Moreover, the ‘innovation trajectory’ crystallised the need to focus even more on the usability of future personalised insurance products. For Royal Direct, these products will have to be easily accessible in terms of their design. Even the smallest constraint might develop into a hurdle for the successful development of behaviour-based personalisation:

We have to learn from our experience. Probably, we will get rid of the beacon because we learned what is really important: usability. The smallest constraint imaginable is already a huge problem in reality. So one has to design insurance products which are easy to use. And the beacon was developed to be easy to use, but apparently it was not. (interview, Commercial Director, Royal Direct, 4 January 2017)

Finally, the experiments’ infrastructure could still be used as an add-on to their car insurance products, to enable the continuation of data collection while stressing that a safe driving style is an important value to Royal Direct: ‘The idea is that we will offer our clients to make use of our app – fully free of charge. Using the app will have no further consequences whatsoever. I think quite a lot of people want to do this.’ (interview, Commercial Director, Royal Direct, 4 January 2017).

The above shows how the original diagnosed ‘failure’ of not turning the ‘driving style study’ into a success story (in terms of producing scientific evidence) was mobilised by the actors themselves to reflect on the company’s future plans. The PromissoryInsight’s innovation trajectory acted here as a market device to generate reflection on the experiment, in order to develop ‘innovative ideas’ for the company’s future products. This would at least contribute to rendering the economic experiment as one that is ‘not unhappy’.

Conclusions

The emergence of new types of Big Data and analytical tools to personalise insurance prices, services and products, poses important technical, societal and regulatory challenges (Blassime et al., 2019; Geneva Association, 2018). How these challenges will play out in practice, however, is not (yet) clear. In order to understand the possible consequences of Big Data in insurance, one cannot merely rely on the characterisations of the potential of technologies, nor on the raised expectations on the future of markets. The empirical analysis of the practices of markets-in-the-making is key when it comes to moving beyond the hypes and fears on the impact of Big Data technologies in insurance.

This article investigated the role of economic experiments in the making of these novel markets of data-driven personalisation. In documenting the trajectory of an experiment in car insurance in the Belgian insurance sector, we outlined how this in vivo experiment was set up, which interventions and manipulations were imposed to make the experiment successful, and how the study was evaluated by the actors. Despite all of the actors’ efforts (of intervention and manipulation), the driving style study did not turn into a success story. The original aim of the experiment was to establish evidence for the link between driving style behaviour and car accident losses, the so-called quest for the ‘holy grail’ in data-driven personalisation in insurance. This announced aim was not reached in the experiment, exemplifying the difficulties of recruiting, persuading and containing human study objects to participate in an economic experiment ‘in the wild’ (Callon, 2007a). We then employed John Austin’s characterisation of ‘happy’ and ‘unhappy’ performatives (Austin, 1962) to demonstrate that the study’s failure to produce evidence did not make the economic experiment a total fiasco. It was ‘happy’ in different ways (Callon, 2007a; MacKenzie, 2008).

The driving style study contributed to producing elements of a new, not-yet ‘fully realised’, insurance product enacting behaviour-based personalisation. In the unfolding of the experiment, important ingredients for the making of behaviour-based personalisation in insurance came to the fore. First, the choice for smartphone technology to generate data on driving behaviour had important ramifications on the experiment. Smartphone data was considered to be ‘reliable enough’, taking into consideration the lower cost to produce these data. This shows that Big Data technologies and practices of economic experimentation may impose requirements on data quality which differ from traditional knowledge practices. Yet, the low(er) reliability and accuracy of smartphone sensor data urged the developers of the driving style study cautiously to find ways of establishing trust in the technology. This was done by not enabling the customer/participants to check where or when the technology reported ‘events’, i.e., the exceeding of thresholds for the measured variables. In exploring the potentiality of these new technologies, the driving style study intervened in the Belgian insurance market by calibrating the sensors of the participants’ smartphone devices and by studying how clients reacted to the newly generated ‘economic things’ (Muniesa, 2014).

The above ingredients exemplify that ‘wild economics’ (Callon, 2007a), in our case the entanglement of the experimental situation and the insurance product attached to it, demands different – not better or worse – requirements from its data than confined forms of experimental research. This implies that it is harder, if not impossible, to obtain established forms of evidence as to be found in laboratory experimental research, such as in randomised control trials (Hogle, 2016; Hogle and Das, 2017). Traditional laboratory experimental situations, in economics and other sciences, are characterised by the ‘rarefaction of actors engaged in experimentation’ (Muniesa and Callon, 2007: 170). In replicating real-life situations in a purified way, having the most important characteristics in common while leaving out all other nuisances, it is tried to generate a high internal validity. The experimental practices of the driving style study, an exemplary case of ‘economics in the wild’/‘in vivo experimentation’, are not dealing with hypothetical situations, as ‘distinct’ from the real world. They are embedded in real-life practices of driving behaviour, in actual Belgian traffic rather than on a secluded circuit. This ‘wild experimentation’ was needed in order to generate data on actual Belgian driving behaviour.

Secondly, the driving style study provided an opportunity to test and experiment with ways of ‘coaching’ drivers towards a safer style of driving (Meyers, 2018c). Behaviour-based personalisation feeds on the idea that people can be nudged towards the ‘right’ direction, an idea that is at the core of behavioural economics (Thaler and Sunstein, 2009). Behavioural economics has pointed out how ‘real humans’ are less ‘rational’ and more ‘flawed’ than the fictional homo economicus of neoclassical microeconomics (Giocoli, 2013; Yeung, 2016). However, the ways in which individuals are ‘predictably irrational’ (Ariely, 2009) can be remedied when individuals are ‘placed in the right environment’ (Thaler, 2015). With the driving style study app, insurance clients were actively encouraged to change their behaviour. The drive style app tested here with personalised feedback infrastructures to make drivers accountable for their driving ‘choices’ (Meyers and Van Hoyweghen, 2018a). Moreover, with these coaching practices, one might witness a change in the role of insurance from providers of products to providers of services, where insurers act as personal coaches who ‘help individuals helping themselves’ (Tritch, 2007). Insurance markets are experimenting here with incentivising ‘choice architectures’ (Callon, 2007b; Thaler and Sunstein, 2009) in which economic actors are most likely to act autonomously. This highlights the role of economic experiments in the making of novel markets of Big Data-enabled personalisation. The article shows that in behaviour-based personalisation, practices of experimentation play a constitutive role in building infrastructures that make choices on how to collect which data, and how and when to enact driving style advice.

Although the experiment failed to produce evidence on the link between driving style and car accident risks, insightful knowledge was produced on what behaviour-based personalisation could look like in future insurance markets. Market actors are forced to experiment to build future (insurance) products and market conditions. The question remains if and how insurance providers will be capable of moving beyond the catch-22 of behaviour-based pricing and find ‘the holy grail in insurance’ by producing actuarial evidence that would make both regulators, insurers and clients ‘happy’.