SENHANCE: A Semantic Web framework for integrating social and hardware sensors in e-Health

Abstract

Self-reported data are very important in Healthcare, especially when combined with data from sensors. Social Networking Sites, such as Facebook, are a promising source of not only self-reported data but also social data, which are otherwise difficult to obtain. Due to their unstructured nature, providing information that is meaningful to health professionals from this source is a daunting task. To this end, we employ Social Network Applications as Social Sensors that gather structured data and use Semantic Web technologies to fuse them with hardware sensor data, effectively integrating both sources. We show that this combination of social and hardware sensor observations creates a novel space that can be used for a variety of feature-rich e-Health applications. We present the design of our prototype framework, SENHANCE, and our findings from its pilot application in the NutriHeAl project, where a Facebook app is integrated with Fitbit digital pedometers for Lifestyle monitoring.

Introduction

The benefits from systematically coupling self-reported data (or human observations) with traditional hardware sensor observations (such as readings from a heart rate monitor, a fitness tracker/pedometer, glucose-level indicators, etc.) are numerous. A human, as opposed to a hardware sensor may reveal information that only exists in the user’s mind and may otherwise be impossible to obtain. He/she is also capable of providing information about future situations or intent that can be valuable to health professionals and hard to obtain via traditional sensing means. Human involvement is also particularly useful in sensing various processes in complex personal, social and urban spaces where traditional sensors suffer from gaps in spatiotemporal coverage, limitations in making complex inferences, inability to adapt to dynamic and cluttered spaces, and aesthetic and ergonomic problems. ¹ A very simple example of the above is the inherent difficulty of a fitness tracker to determine the type of physical activity performed which regularly leads to the need for user input. In addition, the ability to compare the perceived by a human versus the perceived by a sensor observation can be very useful in cases where the sensor is treated as ground truth. For example, this can be the user’s perceived duration of an event (‘I jogged for 45 min’) compared to the sensor’s more accurate reading (‘physical activity duration: 30 min’). If done in an efficient and systematic way, integrating self-reported data with hardware sensors provides applications with much higher degrees of context awareness, further shortening the gap between digital and physical worlds and overall providing a much better service. ²

Web 2.0 and the proliferation of user-generated content (UGC) marked a new era for the Web. Many people now have a ‘digital life’ where they produce digital footprints on various Web services and increasingly so in Social Networking Sites (SNSs). Facebook alone is fast approaching 1 billion users. ³ In SNSs, users make observations that are extremely varied: from their opinion on what the weather is like, to what news is trending at the moment, to how many miles they ran today during their daily workout routine. In health care, this translates to people tracking their workout routines, posting reviews of their medical treatments and raising awareness about certain health conditions. ⁴

In a way, SNSs provide an endless source of self-reported data and research shows that online self-reporting can be as valid and reliable as other means (e.g. paper-based). ⁵ In addition, these observations are bundled and augmented by a wealth of metadata such as social data (friends, connections, group memberships, etc.) that can play multiple roles: aid the research itself in reaching conclusions about the user and his/her environment, establish context or even aid in calculating user trust. ⁶ Although lucrative because of their abundance and their accompanying metadata, these obser-vations are mostly unstructured and usually provided for different purposes. The proposed SENHANCE (Sensors ENHANCEd) framework uses Social Network Applications (SNApps), such as those provided by the Facebook Developer Platform, to get the best of both worlds – structured human observations that are fit-for-purpose as well as social data and other useful information present on the Social Networking platform. Furthermore, it employs Semantic Web (SW) technologies in order to systematically include data from SNSs as a sensor (i.e. a Social Sensor) in a way that is implementation-agnostic and applicable to many e-Health scenarios. A SNS can, thus, be treated as a repository of Human Sensor observations, where the sensor is the human entity that owns the SNS account. These observations are then available to be integrated with hardware sensor data, along with users’ social data that can be consumed by health professionals who place high value in self-reported data.

In the sections that follow, we present the design of our prototype monitoring framework, SENHANCE, and show how it can be used for Lifestyle monitoring in its pilot application in the NutriHeAl project, where a Facebook app is integrated with Fitbit digital pedometers. Section ‘Background and related work’ deals with related research in the area. Section ‘Gathering and modelling data from SNSs’ discusses the modelling of SNApps as Social Sensors while section ‘System architecture’ presents the overall system architecture of SENHANCE. Sections ‘Pilot implementation’ and ‘Advantages of SENHANCE’ deal with the pilot design, implementation and results, and, finally, sections ‘Future work’ and ‘Conclusion’ present our conclusions and suggestions for follow-up work.

Background and related work

The concept of using humans as sensors has been an important topic of research for many years. According to Srivastava et al., ¹ there is a long history of information gathered via human intelligence in the defence and security, social, behavioural and medical sciences. Self-reported data in Healthcare is, of course, not a new concept; it has been an essential component of health research since its inception. In Nutrition and Dietetics, for example, food and physical activity diaries are one of the most important tools in gathering patient data.^7,8 In Bonomi and Westerterp, ⁹ subjective methods (‘direct observations, diaries, activity logs, recall and questionnaires’) are defined as very popular methods for quantifying the selected variable (in that case, physical activity) due to their relatively low cost and the added value of contextual information provided by the user. This is especially true in large-scale studies, where cost and ease of deployment can become a very important factor in the overall success and results of the study.

In Paul and Dredze, ¹⁰ Twitter is used in public health research by tracking a range of tweets about ailments, medication and symptoms. Similar work can be found in Lee et al. ¹¹ and Szomszor et al. ¹² which report on tracking Flu outbreaks. In another scientific domain, researchers show how Social Media, and specifically Twitter, can be used for real-world event detection such as sports games and earthquakes.^13,14 We believe that treating Social Media as data repositories is a young scientific research area that, given the rapid popularity of websites like Facebook and Twitter, will become even more important in the coming years.

On the other hand, using sensors for health research has increasing popularity due to the ever-descending cost of sensory device deployment. This is not only true for scientific research applications^15,16 but for consumer-grade electronics as well. The latest iPhone from Apple, a device that has become almost pop-culture in the past few years, includes a separate motion co-processor that is ‘designed specifically to measure motion data from the accelerometer, gyroscope and compass present on the device’ with minimal battery drain (Apple iPhone; http://www.apple.com). An ‘infographic’ available online ¹⁷ shows 16 different ‘slots’ for body-worn sensors on the human body, with most of the suggested sensors being consumer-grade. In our own past work, we have used low-cost digital pedometers that automatically transmit data to a central server with great success.^18,19 All this indicates a trend towards a wealth of sensory data being available in the coming years from multiple sources and with rapidly increasing frequency.

Researchers, mostly in the Pervasive Computing community, have combined the two sources – User/Social and Hardware – in the past for different reasons. SenseFace ²⁰ provides user context from body sensor networks and ‘multimedia information contained within the social network space’ by developing a custom social network stack. S-Sensors ²¹ takes a different approach by using twitter to publish Sensor observations. SocialFusion ²² fuses mobile, social and sensing input streams to generate context awareness. In Rosi et al., ² authors point out that while current social services are application specific, ‘future social services should be constructed on the basis of common and reusable tools and mechanisms’.

To the best of our knowledge, our work presents the first attempt to develop a complete framework to deal with integrating Social and Hardware Sensors in an abstract, implementation-agnostic, fashion. We show how this can be achieved by utilising SNApps and SW technologies, as discussed in the sections that follow.

Gathering and modelling data from SNSs

A basic concept in SENHANCE is using SNS Applications (SNApps) as Social Sensors for gathering data. As mentioned before, data on SNS are mostly unstructured and usually provided for different purposes. SNApps are applications (usually Web-based) that are linked to SNSs and can guide a user in providing structured data while making the users’ social data available. SNApps are very prominent, for example, in Facebook, where they are provided via the Facebook Developer Platform (FDP; http://developers.facebook.com). A Web developer can build an application that could be as simple as an HTML Form, ‘interface’ it with Facebook via the FDP and offer it to the public or a selected audience. Users then use these applications for various reasons within Facebook. For example, over 1 million users (Facebook Health Fitness Apps; http://www.facebook.com/appcenter/category/healthfitness) use health and fitness apps such as Nike+ Running, which tracks running routines, and MyFitnessPal, which helps users keep a food journal.

SNApps may have been made popular by platforms such as Facebook but can be easily built upon any Social Network that provides Application Programming Interface (API)/data access. They also have the benefit of being able to ‘traverse’ a social graph by advertising themselves or being inadvertently advertised by users, thus expanding the potential user base of an e-health scenario (intervention, data collection, etc).

After setting up a SNApp, data on SNSs and thus data that can be collected via a SNApp can be seen as a twofold structure (Figure 1): Data describing the social network structure (Social Data) and data describing the UGC. ²³ A further categorisation of UGC can be made by distinguishing between active UGC (status updates, tweets and data from SNApps) and passive UGC, data that ‘lives’ on the platform. Passive UGC can be defined as information the user keeps on the SNS platform, usually in the forms of variables on a user profile, such as his/her Age, Gender and Location and possible implementation-specific variables (in more specialised SNSs such as health-oriented SNSs) such as height and weight. An interesting characteristic of passive UGC is that it can exist in the platform independent of a user’s active UGC activity. A user that creates a profile on a SNS only for networking purposes and never posts or interacts with other members (a ‘lurker’) may still keep useful information in his profile, especially if that forms part of a sign-up procedure.

Figure 1.

Examples of available SNS data and their categorisation.

It is critical for e-Health frameworks such as SENHANCE to collect and store data in an implementation-agnostic fashion, in order to create flexible and reusable data components that are ‘maximally adaptive’. ²⁴ Data should also be machine-understandable in order to facilitate integration and reduce human burden. ²⁵ SW technologies and Resource Description Framework (RDF; http://www.w3.org/RDF/) in particular provide meaningful information that can be better understood by ‘things’ as well as humans compared to current protocol descriptions. ²⁶ Ontologies, which are typically represented on the SW via Web Ontology Language (OWL; http://www.w3.org/TR/owl-features/), clarify the domain’s structure of knowledge and enable knowledge-sharing. ²⁷ SW technologies as a whole are part of ‘Medicine 2.0’. ²⁸ The modelling of UGC and Social Data from SNSs in SENHANCE is based on the above technologies (RDF/OWL) and the methodology that follows.

Modelling active and passive UGC

For the purposes of this framework, a SNApp is modelled as a sensor, that is, a Social Sensor. This way, data from multiple SNApps as well as true hardware sensors can be homogenised and integrated. The term Social Sensor has been used before in the literature^2,13,29 but is used herein to indicate the gathering of observations on a SNS from a human entity acting as a sensor. For example, in a scenario where users are submitting physical activity data via a Facebook application, they are acting as Human Sensors in the sense that they self-report the activity they performed as an observation. The value of a user profile variable is defined in a similar way. The Facebook application acts as a Social Sensor in the sense that it is responsible for detecting, annotating and aggregating these human observations. We believe it is necessary to make this distinction between human and social sensors because of accountability; it is the human who produces (senses) the event and whose credibility is in question in regard to data quality (DQ).

When it comes to ontology design in the SW, re-inventing the wheel by creating a new ontology is not only time-consuming but also discouraged. Re-using existing ontologies promotes the SW vision and creates the necessary conditions for linked data. In this regard, the Semantic Sensor Network (SSN) ontology ³⁰ is used to describe the sensors, what they measure and the result of these observations. SSN provides sensors with semantic interoperability and a basis for semantic reasoning. SSN can be seen as complimentary to OGC’s Sensor Web Enablement ³¹ (SWE) activities which is a long-standing community standard, in the sense that SWE is intended to provide standardisation at the syntactic and service levels and does not explicitly address semantic level interoperability. ³² Thus, by using SSN as the base for sensor observations, the common issue of heterogeneity between health sensors is addressed by creating reusable, shareable SW components that can exploit the benefits of OWL.

Let $\mathrm{P}=\{{\mathrm{p}}_1,{\mathrm{p}}_2,\dots ,{\mathrm{p}}_{\mathrm{k}}\}$ represent the set of k properties that can describe a user’s ${\mathrm{u}}_{\mathrm{i}}\in \mathrm{U}$ state, such as age, weight, height, blood sugar level, steps taken and so on. Then, ${({\mathrm{u}}_{\mathrm{i}},{\mathrm{p}}_{\mathrm{j}})}_{\mathrm{t}}$ denotes a time-stamped by t sequence of observations (values) made in a particular time period of T time units where $\mathrm{t}\in [0,\mathrm{T}]$ . For example, ${({\mathrm{u}}_1,{\mathrm{p}}_2)}_5=190$ , where ${\mathrm{p}}_2=\mathrm{steps\; taken}$ . SENHANCE does not differentiate as to the origin of these observations – they may come from the user profile or the SNApp, depending on the implementation. The above concepts are mapped to SSN OWL constructs as follows: a user ${\mathrm{u}}_{\mathrm{i}}$ is a ssn:Sensor (as a human sensor) and a ssn:FeatureOfInterest (as a target of sensing) with instances ${\mathrm{u}}_{\mathrm{i}}\in \mathrm{U}$ . A property ${\mathrm{p}}_{\mathrm{j}}$ is a ssn:Property with instances ${\mathrm{p}}_{\mathrm{j}}\in \mathrm{P}$ , and an observation ${({\mathrm{u}}_{\mathrm{i}},{\mathrm{p}}_{\mathrm{j}})}_{\mathrm{t}}$ is an ssn:Observation.

Because SSN does not dictate how the result of an observation (ssn:SensorOutput) is defined, the semantic representation of the set of acceptable values is implementation specific (e.g. using the semantic web for Earth and environmental terminology (SWEET) ontology ³³ or custom-built domain ontologies), as is the implementation of a time-reference mechanism for t. This flexible mechanism of representation allows for the creation of implementation-agnostic and reusable components, which agrees with the chosen representation of a user’s social data and the SW vision in general.

Modelling social data

Let $\mathrm{U}=\{{\mathrm{u}}_1,{\mathrm{u}}_2,\dots ,{\mathrm{u}}_{\mathrm{n}}\}$ be a group of n SNS users. In Social Network theory, a graph $\mathrm{G}=(\mathrm{V},\mathrm{E})$ where V is a finite set of vertices and E is a finite set of edges such that $\mathrm{E}\subseteq \mathrm{V}\times \mathrm{V}$ can be used to model the social network that describes U. ³⁴ Furthermore, the matrix $\mathrm{M}:={({\mathrm{m}}_{\mathrm{i},\mathrm{j}})}_{\mathrm{n}*\mathrm{n}}$ where ${\mathrm{m}}_{\mathrm{i},\mathrm{j}}=\{\begin{array}{c}1|({\mathrm{u}}_{\mathrm{i}},{\mathrm{u}}_{\mathrm{j}})\in E\\ 0|\mathrm{otherwise}\end{array}$ and $\mathrm{n}=\left|\mathrm{V}\right|$ , associated with that graph, is used to describe the relationships between individuals (reciprocal or not).

To describe this social space semantically, SENHANCE uses FOAF, ³⁵ one of the most popular SW Ontologies. By modelling social connections via FOAF, the extensibility of the RDF language and the expressive ontological notations of OWL can be leveraged to define more precise notions of relationships where needed. Extensions to FOAF that more clearly define relationships and tie strengths can be custom-built or found as published ontologies^36–38 which further proves the power of FOAF as a starting point for modelling social data. In addition, FOAF profiles can be generated and gathered from other services, be they SNApps or SNSs in general in order to integrate social data for monitored users from various sources. Many people already keep a personal FOAF profile on their website or on a shared hosting space and there are also SNSs that export FOAF. ³⁹

A tie ( ${\mathrm{m}}_{\mathrm{i},\mathrm{j}}=1$ ) is modelled as a foaf:knows relationship between ${\mathrm{u}}_{\mathrm{i}}$ and ${\mathrm{u}}_{\mathrm{j}}$ , which are both instances of foaf:person. A social group ${\mathrm{SG}}_{\mathrm{i}}$ (a community, group of interest, etc.) that consists of users ${\mathrm{u}}_{\mathrm{i}}\in \mathrm{U}$ is modelled as a foaf:group that has as foaf:member one or more foaf:person. This provides an abstract approach towards a user’s social data that can be further refined if needed.

System architecture

Figure 2 presents an overview of the SENHANCE framework, a semantic framework that systematically integrates SNApps as Social Sensors in an implementation-agnostic e-Health application. The SENHANCE architecture comprised a number of hardware and/or social sensors that create output in the form of artefacts, an abstracted notion of a typical sensor output. These artefacts are forwarded to a semantic adapter entity which is responsible for converting them into semantic artefacts, expressed in RDF. The semantic adapter follows the mappings presented in section ‘Gathering and modelling data from SNSs’ for the incoming data and generates the appropriate RDF with the help of external ontologies such as OWL-TIME (OWL-TIME; http://www.w3.org/TR/owl-time/), Semantically-Interlinked Online Communities (SIOC; http://www.sioc-project.org/), PROV Ontology (PROV-O; http://www.w3.org/TR/prov-o/) and implementation-specific ontologies. These observations are subsequently stored in a Semantic (triples) Database that can be queried via SPARQL (SPARQL Query Language for RDF; http://www.w3.org/TR/rdf-sparql-query/), the query language of the SW, to provide ‘smart’ complex queries that are agnostic of sensor type.

Figure 2.

An overview of the SENHANCE framework.

The SENHANCE semantic adapter

At the heart of every SENHANCE instance is at least one SENHANCE Semantic Adapter. This is a software entity responsible for transforming incoming data (sensor artefacts) into proper RDF for inclusion in the database. Section ‘Gathering and modelling data from SNSs’ presented the basic mappings of common elements that appear in such a scenario.

For Hardware sensors, due to the abstract nature of SSN but, at the same time, its very complete approach to sensor device and observation descriptions, inserting new sensory data in SENHANCE is very straightforward. A Sensor Registration Module is responsible for registering new Sensor Classes (owl:Class) as well as new Sensor Instances of these classes and a Sensor Observation Module is responsible for inserting new observations into the database. Depending on the granularity required by the implementation, a Sensor Class can be as simple as a few triples denoting its parent Class and its observed property, or as complex as adding ssn:MeasurementCapability, hardware characteristics and so on. A Sensor Instance, respectively, is a collection of triples that determine the class of the sensor, its ssn:Platform and its ssn:featureOfInterest. Finally, a Sensor Observation includes information about the sensor which produced it, its result (type and value) as well as the observation result time. The SSN Specification ³⁰ has more information about these basic building blocks, a basic example of which can be seen in Figure 3, assuming a digital pedometer sensor.

Figure 3.

A new Hardware Sensor Observation in SENHANCE.

For Social Sensors, the unique approach of SENHANCE allows them to be included in the system in the same way as hardware sensors. The main difference is that an observation that originates from a SNS is derived from data that are provided for a different purpose, like data from a SNApp. Figure 4 shows how a SENHANCE adapter can annotate an observation using SIOC to describe the original content, SSN to describe the end result (an ssn:Observation) and PROV-O to link the two together (original and derivative). Social data that may accompany the SNApp data are mapped using FOAF, as described in section ‘Gathering and modelling data from SNSs’. Thus, a new observation from a SNApp looks like the example in Figure 5.

Figure 4.

Mapping a SNApp observation using the SENHANCE Semantic Adapter.

Figure 5.

A new Social Sensor Observation in SENHANCE.

It is worthwhile to note that SENHANCE is a ‘generic’ Semantic Information System, which means that no assumptions are made about the ontology in compile-time. ⁴⁰ As a result, mapping new elements or data into the model is as straightforward as adding new triples to the database. The next time the reasoner runs (as a result of a query for example), these new data will be taken into account automatically.

Querying SENHANCE

With the expressive capabilities of SPARQL, a SENHANCE user can query the system for simple data retrieval or even traverse a social graph to get results from socially connected participants. For example, Query A (the left part of Figure 6), which is agnostic of the type of sensor that provides the data, could easily be expanded into Query B, which returns the physical activity data of both John Smith and all his social network connections.

Figure 6.

Example SPARQL Queries.

Due to the fact that all SENHANCE data are expressed in RDF and described by the ontologies mentioned in section ‘Gathering and modelling data from SNSs’, data can easily be linked between different semantic databases by using Federated SPARQL queries. At the time of writing, SPARQL Federation is a W3C Recommendation (SPARQL Federated Query; http://www.w3.org/TR/sparql11-federated-query/), but implementations already exist. ⁴¹ In its core, it enables SPARQL queries to retrieve data from different SPARQL endpoints (services). For example, given the existence of multiple SENHANCE monitoring instances, a SPARQL query can show the physical activity of a user from both instances. Respectively, given the existence of a semantic weather data endpoint for a location, a SPARQL query can show the physical activity of a user on a specific time period as well as show the weather data for this period, at the same time (i.e. they will appear as results of the query).

Pilot implementation

Pilot setup

To test the SENHANCE framework in realistic conditions, a group of 49 participants (n = 49, Avg. Age = 24.2 ± 6.8 years) took part in two rounds of a pilot, part of the NutriHeAl (NH) project, an on-going inter-disciplinary collaboration between the Department of Electrical & Computer Engineering of AUTH, Greece and the Department of Nutrition of TEITHE, Greece. The specific work package aims to explore the feasibility of integrating self-reported data from a SNS with true hardware sensor data as well as assess the usefulness of Social Data in related research. Participants were randomly sampled from the two universities’ student and staff population, resulting in a diverse young adult (ages 20–49 years) sample.

All users were equipped with a Fitbit Zip (Fitbit Zip; http://www.fitbit.com/zip) digital pedometer that continually tracks the wearer’s steps as long as it is worn, without user intervention. The device automatically converts data from an accelerometer into steps, using algorithms patented by its creators, and then uploads the data into the Fitbit server when in range (~6 m) of a Bluetooth USB Dongle and accompanying software. Each user was supplied with one pedometer and one USB Dongle in order to upload the data from their home or work computer. Data were made available to the SENHANCE instance via the Fitbit API, in 1- and 5-min intervals.

In addition to hardware sensing, the users were asked to disclose information about their activities and other variables (age, gender, weight, height and automatically calculated values such as body mass index (BMI) and Basal Metabolic Rate (BMR)) on Facebook via a Facebook app. The app, which was built for the purposes of this pilot and is partly based on our previous work,^18,42 is entitled ‘NutriHeAl Activity Diary’ (NutriHeAl Activity Diary; http://apps.facebook.com/activdiary) and is built on current Web technologies (HTML5, PHP and JavaScript/AJAX). When using the app, users are presented with a profile page, where they enter their basic information (contact details, age/height/weight), and a weekly calendar (Figure 7 (left)) where they can add their daily activity. The app allows the user to select one of the two pre-defined activities (Work and Sleep), add a custom Physical Activity (such as ‘Running’, ‘Jogging’, etc.) or add a more generic Other Activity (such as ‘Going out’ or ‘Shopping’). The user then specifies the duration of the activity via ‘From’ and ‘To’ fields.

Figure 7.

Snapshot of the NutriHeAl Facebook App.: (left) calendar view and (right) results.

Pilot participants were asked to wear their pedometer all day for a period of 4 weeks and record their physical activity on the NH Facebook app daily. Each user’s motivational goal was at least 10,000 daily steps and an activity diary that was as complete as possible. At the end, provided that users have submitted a filled-in weekly physical activity diary, reports were released on a participant basis, accessible only by the specific participant and the overseeing nutritionist (using a username/password combination). These reports consisted of (a) an overview that showed how close to the 10,000 daily steps goal a user was each day and (b) daily graphs that the user is able to ‘augment’ by pressing a button and retrieving the data from the Facebook app (Figure 7 (right)).

Pilot results

Data were gathered over the course of two rounds of 4-week (minimum) pilots. For each round, a SENHANCE instance was deployed with the appropriate adapter for Sensor (Fitbit Zip) and SNApp (NutriHeAl Activity Diary). The prototype SENHANCE system developed uses Python scripts for converting and forwarding data to a Sesame Triplestore. The Triplestore runs on Apache Tomcat and uses OWLIM-Lite (OWLIM; http://www.ontotext.com/owlim) 5.4 for reasoning, which supports SPARQL 1.1 and federated queries.

The NutriHeAl SNApp converted the activity duration into 5-min intervals (one observation every 5 min), to align with the FitBit data. This was purely a design consideration, as the 1-min granularity was considered a small window for inference in this scenario. An interface to the SENHANCE instance that collects the data was developed, which produces the result reports.

At the end of the pilot, 44 users (dropout of 5 users) reported a total of 6548 named and time-stamped activities. The users also automatically provided data about their Facebook Social circle (friends). The Fitbit Zips, respectively, reported step activity for an average of 33 days per user. Figure 8 shows an overview of the pilot as well as the rich RDF data collected via the SENHANCE adapters.

Figure 8.

SENHANCE Pilot Sample and Results.

It is worth noting that gathering data through an SNS may yield varying degree of success, namely because of two factors. First, it is possible that in an emergency or important situation users do not use social media and thus only hardware sensor observations will be available. Second, the reliability and data density of SNApp data can greatly vary and may not yield data in the way that is expected. Even though these issues were not overly present in the NutriHeAl pilot, mainly due to application design and user motivation, they still need to be taken into account by each implementation of this framework.

Another distinction that needs to be made when deciding on how or when to implement SENHANCE, as well as interpreting SENHANCE data, is between population-level and individual-level inference. On the population level, data collection can be significantly less invasive, have a higher threshold for error and, also, a lower data granularity needed for reaching conclusions. The data collected in the NutriHeAl pilot are better suited for individual-level inference, and data collection methodologies will need to be adapted in the case of population-level studies (e.g. modify the self-reporting intervals and/or the hardware sensor report granularity).

Advantages of SENHANCE

The combination of social and hardware sensor observations provided in a SENHANCE instance creates a novel space that can be used for a variety of feature-rich e-Health applications. Due to the fact that this article mainly discusses the architecture and design of the SENHANCE framework, we do not focus on specific outcomes of the pilot in regard to lifestyle/physical activity patterns, but rather present three advantages of the framework.

Seamless integration of social and hardware sensors

Because everything in SENHANCE is expressed in a common semantic language, it is possible to seamlessly integrate the two sources via SPARQL queries that request data from all sensors and thus obtain a more complete version of an event. These operations can also be made automatic by using OWL/RDF inference (semantic reasoners and rule engines). This integration is application- and implementation-agnostic; SENHANCE and the use of OWL/RDF constructs and SPARQL queries provide full customisation, data transparency and the possibility for data reuse.

In the case of the NutriHeAl pilot experiment, the primary outcome of integrating social and hardware sensor data is the added contextual information about the physical observation (steps) that is provided by the user. The pedometer (the hardware sensor), by itself, has no knowledge about the type of activity performed by the user and the user (via the social sensor) cannot accurately assess the intensity level of the activity performed (in steps). By comparing and cross-referencing the two sources, sensory data provided by the pedometer can be augmented in order to provide a clearer image of the activity performed.

For example, the right part of Figure 7 shows a ‘spike’ in user steps around 08:30–09:00 which, by itself, is just raw data. On the other hand, the left part of Figure 7 shows the user having reported 08:30–09:00 as ‘Walk’. In SENHANCE, the two sources can be integrated by requesting data from the time interval 08:30–09:00 from both sensors and get the more complete version of the activity as a result: ‘Walk, 08:30–09:00 m, 1920 steps’. Data can be retrieved via simple SPARQL queries such as the Query in Figure 9 and presented to the user according to application needs.

Figure 9.

Sample query for all SENHANCE sensors on user. Query Result from TopBraid Composer ME.

With further analysis and more evolved algorithms, patterns can be discovered for the whole sample and results can be drawn about an activity in general. For example, by utilising a basic activity detection algorithm (steps >500 for a period of ⩾10 min) in a subset of this specific dataset, 85 per cent of the activities detected were consistent with reported activities of users and could be cross-referenced. These results, which are very beneficial for Healthcare professionals in the Nutrition, Dietetics and Physical Activity domains are out of the scope of this article and are further discussed in Pagkalos et al. ⁴³

It is important to note that while several other applications (including commercial ones) may have similar integration functions, data obtained through SENHANCE are machine-understandable and easily reusable due to the very detailed semantic markup, which is a major benefit for e-Health systems.

Utilisation of social data

Another benefit of SENHANCE is the possibility to utilise the social data that accompanies other, implementation-specific SNApp data. By integrating social and hardware sensors, SENHANCE provides an augmented dataset along with the necessary semantic annotations to fuel further research. In the NutriHeAl project pilot, each participating user disclosed their list of Facebook friends, and these social data can be analysed using classic Social Network Analysis (SNA) techniques. Figure 10 shows the results of a minor analysis using three different SNA metrics that were applied to the network of the NutriHeAl pilot participants (Round 1) and can also be used in other e-Health use-cases: Community Detection (shown by coloured groups), Tie Strength (shown by size and label of connecting edge) and Influential persons in Network (shown by size of a node).

Figure 10.

Social Network Analysis of the NutriHeAl Pilot, Round 1, using Gephi 0.8.2 (Gephi: The Open Graph Viz Platform, https://gephi.org/).

The above SNA techniques are just a subset of the possibilities in a dataset where social data are available and are presented here to display the possibilities of a SENHANCE instance. For example, Tie Strength can result in a better understanding of the person-to-person connections in the pilot, and Community Detection can be used to further understand how users may affect one another when being monitored by a Dietician or when wearing a wearable sensor. The seminal work in Scott ⁴⁴ as well as recent research efforts in e-Health^45,46 present even more opportunities for utilising SNS data.

Semantic database linking

The semantic representation of data in SENHANCE not only makes integration between local data available, but also allows for integrating data from other semantic databases. This can be done via SPARQL Federated Queries or by downloading the data and merging them with the local SENHANCE instance if data are not time-sensitive. In the case of the NutriHeAl pilot, the original dataset was enhanced by including obesity statistics per age group and gender from data acquired by the Hellenic Medical Association for Obesity. ⁴⁷ This allows for queries that can return, for example, both actual and statistical data, such as the Query of Figure 11 which returns the BMI as measured by the NutriHeAl SNApp and the statistical BMI relative to Age and Gender from the HMAO database. ⁴⁷

Figure 11.

BMI as measured by the NutriHeAl SNApp and statistical BMI from the HMAO database. Query result from TopBraid Composer ME.

The possibilities of linking with external databases are abundant; a semantic database with weather information can augment the data per day in order to see whether there are correlations between activities performed and weather conditions, data from other experiments based on SENHANCE (or similar platforms that export semantic data) can be integrated and so on. OWL Inference mechanisms such as owl:equivalentClass/Property and owl:sameAs ⁴⁸ can also be used to link ontologies at the semantic level, sometimes called a ‘semantic join’.

For example, suppose that a different semantic database also includes results from digital pedometers but does not use the same Class name as a SENHANCE instance, making queries that look for ‘nutriheal:DigitalPedometers’ null. By simply defining the DigitalPedometer as an Inferred class (Figure 12), a sensor that ssn:observes nutriheal:StepsTaken is automatically inferred to be a digital pedometer, making our query interoperable. Because it is unrealistic to assume that everybody will use nutriheal:StepsTaken to refer to the steps taken by a user (‘that would require some grand design, which is contrary to the spirit of the web’ ⁴⁸ ), owl:sameAs can be used to semantically link this instance to another database’s instance (e.g. example:Steps). This mechanism can also be used to link an object to more detailed information in other databases such as, for example, linking a local instance of :BloodPressure to the DBPedia entry :Blood_Presure (DBPedia entry; http://dbpedia.org/resource/Blood_pressure).

Figure 12.

Examples of using owl:equivalentClass and owl:sameAs to ‘semantically join’ ontologies.

Future work

A SNApp can provide useful health-monitoring data and essentially act as a Social Sensor for many e-Health applications. Nevertheless, human self-reported data always carry with them the risk of uncertain DQ. In our pilot, trustworthiness of the users was assumed due to expected self-gains. Unfortunately, this is not always the case. Therefore, we are currently exploring how to best implement DQ assessment techniques such as treating sensor data as ground truth where available and maintaining user reputation tables based on Beta Reputation algorithms that take into consideration the self-reported data’s accordance to sensor data. Because of the seamless integration of Hardware and Social Sensors, SENHANCE makes these kinds of computations significantly easier.

In addition, the current implementation of SENHANCE requires a straightforward, albeit handmade work to integrate each sensor in the framework. Automatic recognition and description of each sensor, similar to context-recognition techniques presented in Hatala et al., ⁴⁹ Henricksen et al., ⁵⁰ Hervás and Bravo ⁵¹ and Rodríguez et al. ⁵² and performed via OWL inference, would contribute to the framework’s applicability in even more scenarios, especially in real-time and dynamic environments.

Finally, even though the expressiveness of SPARQL introduces many benefits, it can also act as a limiting factor for health applications where a more natural-language-style of querying may be of more use. We are, for that purpose, developing a monitoring dashboard with pre-configured queries that will make a SENHANCE instance more accessible. Many researchers have attempted to design more user-friendly interfaces for SPARQL Queries^53,54 which can be used over the SENHANCE interface to provide a better user experience. More work is needed though in this regard, in order to make applications as user-friendly as possible.

Conclusion

This work argued the use, modelling and implementation of SNApps as Social Sensors in e-Health and presented the SENHANCE framework, an original architecture for their integration with hardware sensor data. Based on the latest SW technologies, SENHANCE offers a complete framework for integrating these two data sources in an abstract and implementation-agnostic fashion. While several other applications (including commercial ones) may have similar integration functions, this work presents, to the best of our knowledge, the first attempt to perform integration at this scale using a framework approach based on OWL/RDF. Data obtained through SENHANCE are machine-understandable and easily reusable, which is a major benefit for e-Health systems.

The pilot implementation of SENHANCE in the NutriHeAl project, where a Facebook app and Fitbit Zip digital pedometers are used for Lifestyle Monitoring, presented a clear example of the usefulness of the semantically rich data collected using this framework and identified three key benefits in e-Health research: seamless data integration, social data utilisation and semantic database linking. With the variety of properly annotated data provided through this framework and the powerful capabilities of SW technologies, this innovative approach can, therefore, serve as the basis for a multitude of feature-rich e-Health applications.