Large-Scale Real-Time Semantic Processing Framework for Internet of Things

Abstract

Nowadays, the advanced sensor technology with cloud computing and big data is generating large-scale heterogeneous and real-time IOT (Internet of Things) data. To make full use of the data, development and deploy of ubiquitous IOT-based applications in various aspects of our daily life are quite urgent. However, the characteristics of IOT sensor data, including heterogeneity, variety, volume, and real time, bring many challenges to effectively process the sensor data. The Semantic Web technologies are viewed as a key for the development of IOT. While most of the existing efforts are mainly focused on the modeling, annotation, and representation of IOT data, there has been little work focusing on the background processing of large-scale streaming IOT data. In the paper, we present a large-scale real-time semantic processing framework and implement an elastic distributed streaming engine for IOT applications. The proposed engine efficiently captures and models different scenarios for all kinds of IOT applications based on popular distributed computing platform SPARK. Based on the engine, a typical use case on home environment monitoring is given to illustrate the efficiency of our engine. The results show that our system can scale for large number of sensor streams with different types of IOT applications.

1. Introduction

With the rapid advances in wireless sensor data collection and communication, increased number of IOT data is increasing explosively, which builds many massive wireless sensor networks (WSNs). It is predicted that within the next decade billions of devices (Cisco predicts that the number of the Internet connected devices will be around 50 billion by 2020) [1] will generate myriad of real-world data for many applications and services in a variety of areas such as smart grids, smart homes, e-health, automotive, transport, and environmental monitoring. Such stunning massive and widespread data can help us to observe the surroundings, learn patterns, and have a better understanding of the world, which will construct a more intelligent world.

To make full use of the senses to implement deeper web intelligence, a natural next step would be to unify and process IOT data by existing mature web infrastructure and protocols. Presently, many efforts have been put on this area from the Internet of Things (IOT) to Web. The W3C founded the Web of Things Community Group aiming at accelerating the adoption of Web technologies such as semantic technologies as a basic for enabling services for the combination of IOT with rich descriptions of web data and the context in which they are used. The industry also initiates the standards oneM2M (http://www.onem2m.org/), whose goal is to develop technical specification which is used to address the need for a common IOT service layer by reusing existing web standards and protocols, including RDF, HTTP, and Restful.

However, the characteristics of IOT data, including heterogeneity, variety, volume, and real time, pose a series of challenges to effectively organize, publish, and process the sensor data. The Semantic Web technologies are viewed as a key for the development of IOT. Figure 1 shows the generic functional model of oneM2M for supporting semantics in the specification of oneM2M study on abstraction and semantics enablement [2]. To be specific, it serves the following several purposes: First, the abstraction and semantics layer provide us with a good way to resolve the problems of interoperability and integration within this heterogeneous world of IOT data by defining and reusing some standard semantic concepts. Then, the Semantic Web provides a seamless interface to facilitate the interactions of IOT data and the other existing web of data such as Linked Data [3], DBpedia [4], LinkedGeodata [5], and various kinds of data from Web Services. At last, the service layer provides an interface for various IOT applications by semantic processing technologies, including semantic mash-up, query, and reasoning.

Figure 1

Generic functional semantic model for supporting IOT applications.

Currently, most of the existing works aim at the annotation and definition of various WSNs by providing corresponding description ontology. For example, ontologies such as the W3Cs Semantic Sensor Network (SSN) ontology have been developed, offering a number of constructs to formally describe not only the sensor resources but also the sensor observation and measurement data [6]. However, there has been little work focusing on the background processing of large-scale streaming IOT data. In the paper, we present a large-scale real-time semantic processing framework for IOT applications. The proposed framework efficiently captures and models different scenarios for various IOT applications. We have implemented an elastic streaming engine based on popular large-scale distributed computing platform SPARK [7]. Based on the engine, a typical use case on home environment monitoring is given to illustrate the efficiency of our engine. The results show that our system can scale for large number of sensors streams with different types of IOT applications.

The remainder of this paper is organized as follows. Section 2 outlines related work. In Section 3, we introduce the semantic processing framework and processing engine for IOT applications. Section 4 describes a typical use case in home environment monitoring. Section 5 presents our experiments and results. Finally, we conclude the work in Section 6.

2. Related Work

To the best of our knowledge, our framework and system are the first work addressing various semantic processing tasks for large-scale streaming IOT data, including IOT semantic mash-up, semantic query, and semantic reasoning. There is some related work as follows.

2.1. IOT Modeling and Ontology

One key research topic in IOT is to represent the “things” by standard vocabularies and schemas. Semantic Sensor Web (SSW) is a technology in which sensor data is semantically annotated for interoperability and also provides contextual information for situational knowledge [8]. Many works have proposed semantic model for representing sensors and data. Ontologies such as the W3C's SSN ontology have been developed [6]. These ontologies provide metadata for numerical, spatial, temporal, and other semantic objects. Similar works for sensor metadata description also include Sensor Data Ontology (SDO) [9] and SensorML [10].

These works mainly focus on semantic annotation for the interoperability of IOT by defining a unified and standard ontology, paying no attention to the high-level semantic IOT applications.

2.2. Semantic IOT Applications

Gyrard proposes a semantic-based Machine-to-Machine Measurement approach (M3) to automatically combine, enrich, and reason about IOT data to provide promising cross-domain IOT applications, such as naturopathy application based on multiple datasets [11]. The approach also presents a hub for cross-domain ontologies and datasets. References [12, 13] apply the IOT in the generic agriculture and healthcare context management. SSEO [14] is developed to enable semantic indexing, machine-processable event detection, and data exchange for smart space modeling. Other applications include CONON [15], CoOL [16], and CoBrA [17].

Most of the work aims at building IOT applications in a specific domain, failing to provide a generic semantic IOT processing framework. And they also did not deal with some important challenges for IOT data, such as the real time and scalability.

2.3. Stream Processing for Semantic Data

There are several semantic stream data processing engines, including Streaming SPARQL [18], C-SPARQL [19], and CQELS [20]. Streaming SPARQL extends SPARQL to process data streams. C-SPARQL defines an extension of SPARQL whose distinguishing feature is the support of continuous queries, that is, queries registered over RDF data streams and then continuously executed. CQELS is a native and adaptive query processor for unified query processing over Linked Stream Data and Linked Data.

However, most of these works only focus on the semantic query for Linked Stream Data, ignoring other common demands of IOT applications, including semantic mash-up and reasoning. What is more, the systems are designed to run on a single machine, while our system goes beyond that and specifically focuses on the common scalability issues for IOT applications.

3. Proposed Framework and Elastic Processing Engine

In this section, we propose the large-scale real-time semantic processing framework for IOT applications and elaborate the elastic processing engine to explain how it provides the capabilities for performing various IOT applications.

3.1. Framework

Figure 2 shows the architecture of our semantic processing framework. In general, it consists of five parts: physical entities layer, abstract entities layer, window-based data stream layer, virtual entities layer, and elastic semantic engine layer.

Figure 2

Large-scale real-time semantic processing framework for IOT applications.

3.1.1. Physical Entities Layer

Physical entities layer is located in the lowest layer of the framework, which is responsible for collecting raw sensor data in real time. Every physical entity represents a tangible element that can be sensed by sensors that are deployed in the oneM2M Field Domain environment and that is not specific to a particular IOT application in this environment. According to the oneM2M project standardization, every kind of sensors is to be organized by logical entity (AE) and common services entity (CSE), which provide application logic and common services, respectively.

3.1.2. Abstract Entities Layer

Abstract entities layer is responsible for receiving and implementing the abstraction for the physical devices by the semantic annotation of proxy software. The abstraction layer aims at hiding the complexity of devices and environments by providing a standard format to represent devices. So from the view of upper layer, all the heterogeneous physical sensors can be seen as unified data streams.

3.1.3. Window-Based Data Stream Layer

The layer focuses on extracting related data streams into the windows according to the demands of upper applications. In the real-world IOT applications, data takes the form of continuous streams instead of the form of finite datasets stored in a traditional repository. This is the case for traffic monitoring, environment monitoring, disaster management, telecommunication management, manufacturing, and many other domains. Every sensor corresponds to a window with a certain size.

3.1.4. Virtual Entities Layer

Virtual entities layer aggregates the related window data required by every virtual entity. Virtual entity is a new resource created by multiple window data streams, which is used to accomplish an application service. For our latter use case, if user in a home requests the service for Discomfort Index (DI), a new virtual entity will be generated through aggregating corresponding home appliance sensors (such as temperature, heater, and air cleaner sensors). Then we can get the service of DI by the virtual entity.

3.1.5. Elastic Semantic Engine Layer

The elastic semantic engine layer is the key of the architecture. The layer is responsible for receiving of outer requests, creating of corresponding virtual entities, interacting with static web of data, and real-time returning of continuous results. Our work mainly focuses on the layer. We will discuss it in detail in the next section.

3.2. Elastic Streaming Processing Engine

Our elastic semantic streaming processing engine provides various common IOT services by constructing corresponding virtual entities, including IOT semantic mash-up, query, and reasoning. Every virtual entity aggregates the RDF streams from corresponding several windows. A window extracts the latest elements from the sensor stream. Besides the streaming IOT data, some applications need auxiliary background knowledge, such as Linked Open Data, DBpedia, and LinkedGeodata.

In this part, related definitions are first given; then we elaborate on the 3 functional modules of the engines.

Definition 1 (RDF stream (S)).

The basic data unit for RDF stream is a quad ( $〈s,p,o,t〉$ ). A RDF stream is defined as an ordered sequence of pairs, where every pair is constituted by multiple basic data units, denoted as $D_i^j$ , where i represents the identification (ID) of certain sensor and j is the timestamp. For example, $S_i=\{D_i^{\mathrm{0}},D_i^{\mathrm{1}},D_i^{\mathrm{2}},\dots ,\hspace{0.17em}\hspace{0.17em}i\in \mathbb{N}\}$ denotes the RDF stream data of the ith sensor.

Definition 2 (window (W)).

A window is a subset of the RDF streams given a time range t. $W_i(t)=\{D_i^{\mathrm{0}},D_i^{\mathrm{1}},D_i^{\mathrm{2}},\dots ,D_i^{t-\mathrm{1}},\hspace{0.17em}\hspace{0.17em}t\in \mathbb{N}\}$ denotes the RDF stream data of the ith sensor within the latest t logical time units. For example, $W_{\mathrm{0}}(3)$ can represent the data of ${\mathrm{S}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{o}\mathrm{r}}_{\mathrm{0}}$ within the last 3 seconds.

Definition 3 (virtual window (V)).

Virtual window aggregates the related data needed by a virtual entity, which includes the corresponding window and background knowledge $B=(s,p,o)$ . A virtual window can be denoted as $V_i=\{W^i,B^i\}$ , where $W^i$ represents the set of windows aggregated by the virtual window $V_i$ and $B^i$ is the set of background knowledge bases needed by $V_i$ .

3.2.1. IOT Semantic Mash-Up

Semantic mash-up is one of the most basic demands in IOT domain since many IOT applications rely on the task. It provides functionalities to support new services by aggregating multiple disperse resources. For example, “compute the indoor air quality index (AQI) of a room” is a typical mash-up application, which needs to accomplish the task based on various window data sources including PM2.5, O₃, and CO.

The mash-up task is formalized via the concepts of filtering and recombination. Given a set of RDF streams $S=\{S_{\mathrm{0}},S_{\mathrm{1}},S_{\mathrm{2}},\dots ,S_N\}$ , N is the number of sensors streams. A filtering is a function $\upsilon :S\to T$ which maps the primitive RDF streams to a triple set $T=\{(s,p,o)\}$ . A recombination is a mapping $\gamma :T\to T^{\prime }$ which transforms T to $T^{\prime }$ based on the mash-up formulas. Take indoor AQI for example; T represents the triples containing the concentration of related sensors. γ denotes the mapping formula for computing the individual AQI.

3.2.2. IOT Semantic Query

IOT semantic query is a common function for IOT applications. It enhances the IOT discovery mechanism, to allow locating and linking resources or services based on their semantic information, such as “get the temperature of the ${\mathrm{r}\mathrm{o}\mathrm{o}\mathrm{m}}_{\mathrm{0}}$ ” and “get the all rooms whose PM2.5 ≥ 80.”

The query task is formalized via the concept of mapping. We denote as I, B, L, and V, respectively, the domains of IRIs, blank nodes, literals, and variables which are all disjoint. We also define $T=(I\cup B\cup L)$ . A mapping μ is a partial function $\mu :V\to T$ which gives the bindings for all the variables of a query. Evaluation occurs when an input graph pattern (denoted as P) in the query is matched against a virtual window (V). P is a set of triple patterns $t=(s,p,o)$ such that $s,p,o$ ∈ $(V\cup T)$ . We then define $\mathrm{d}\mathrm{o}\mathrm{m}(\mu )$ as the subset of V where μ is defined (i.e., the domain of μ) and use the notation $\mu (x)$ to refer to the bindings of variable x in μ.

3.2.3. IOT Semantic Reasoning

Reasoning is a mechanism to derive new implicit knowledge from semantically annotated data and to answer complex user query. It can be implemented as a piece of software to be able to infer logical consequences from a set of asserted facts or axioms. Many IOT services belong to the application type. For example, we can infer the Human Comfort Index based on the temperature and humidity, the dangerous level of gas leaking, and so on.

The reasoning task for IOT applications can be seen as the process of applying the reasoning rules in IOT data to derive new facts. We denote as W and B, respectively, the windows of sensor streams and background knowledge. F represents a set of facts that we want to γ contains a set of rules $\gamma =\{R_{\mathrm{1}},R_{\mathrm{2}},R_{\mathrm{3}},\dots \}$ . Thus the reasoning task is formalized as $\delta :(W,B)\stackrel{\gamma }{\to }F$ .

4. Use Case: Home Environment Monitoring

In this section, we give a common IOT use case. It is designed to facilitate the smart real-time monitoring to the home environment. We first give an overview of the use case; then the data model is presented. At last, three concrete application examples are introduced.

4.1. Overview

Nowadays, people are paying much attention to the environmental problems since we are facing a series of serious environmental pollution types, such as smog disaster and water pollution. To deal with these challenges, governments deploy lots of outdoor monitoring stations to capture and publish real-time environmental information to the public. However, there is limited work in the indoor environmental monitoring due to a lack of sensor devices and processing infrastructure.

With the popularity of smart home appliances (e.g., heater, air conditioner, humidifier, and air cleaner) equipped with environment sensors (e.g., sensors for temperature, humidity, CO₁, CO₂, and VOC), large volumes of data from all aspects of indoor environment are available, which makes it possible to implement various home monitoring applications including emergency detection and indoor air quality index. Presently, many commercial companies such as Huawei, Cisco, Intel, and Telecom are planning to deploy and develop related hardware and software infrastructure to provide similar services.

Our use case considers the scenario: suppose a number of households in a city have installed relevant smart appliances; they want to get a series of environment monitoring services from the supplier, including Indoor Air Pollution Index (API), Indoor Sensor Discovery, and Human Comfort Index ( $I_{\mathrm{H}\mathrm{C}}$ ). The three cases correspond to the above three kinds of IOT applications: IOT semantic mash-up, IOT semantic query, and IOT semantic reasoning.

4.2. Data Model

As the paper mainly focuses on the background streaming data processing for IOT applications, we do not create a complex ontology to semantically annotate all the various IOT data. Conversely, we design a simple concept model for the home environment monitoring scenario (see Figure 3). The model captures three types of resources: home, room, and sensor. The label under the resource denotes its URI. “p” is the namespace of the properties. Every sensor entity has three properties: type, value, and time.

Figure 3

The simple concept model for home environment monitoring.

Figure 4 shows a snapshot of the stream knowledge graph based on the concept model. Every home contains multiple rooms (living room, bedroom, kitchen, and so on). Every room is equipped with 15 kinds of sensors, including temperature, humidity, illumination, volume, PM10, PM2.5, O₃, CO, SO₂, and NO₂.

Figure 4

A snapshot of the stream knowledge graph.

4.3. Scenarios

4.3.1. Streaming Semantic Mash-Up

Outdoor AQI is available to us for years, while little attention is paid to indoor AQI, which is also important both for customers and for device suppliers. For customers, the indoor AQI can help them keep track of the latest situation of the house and the top air pollutants. For suppliers, the AQI data can help them monitor their devices and have a better understanding of the needs of different customers.

The indoor AQI task is a typical IOT semantic mash-up application since it needs to integrate multiple window data sources and combine them to complete a specific work. The indoor AQI task is composed of two phases. The first step is to compute the individual AQI (IAQI) for a pollutant. Equation (1) gives the computing formula for ${\mathrm{I}\mathrm{A}\mathrm{Q}\mathrm{I}}_{\mathrm{p}}$ based on [21]. The parameters in the equation are relevant to the specific pollutant. Related pollutants include PM10, PM2.5, ${\mathrm{O}}_3$ , CO, SO₂, and NO₂. Then these IAQIs will be sorted based on (2). The final result will be displayed by a sequence of IAQIs in a descending order, which will keep customers informed of the noticeable pollutants. For more details, readers can refer to [21]. Consider

$$\begin{array}{cc}& \begin{array}{c}{\mathrm{I}\mathrm{A}\mathrm{Q}\mathrm{I}}_{\mathrm{p}}=\frac{\mathrm{I}\mathrm{A}\mathrm{Q}{\mathrm{I}}_{\mathrm{H}\mathrm{i}}-\mathrm{I}\mathrm{A}\mathrm{Q}{\mathrm{I}}_{\mathrm{L}\mathrm{o}}}{\mathrm{B}{\mathrm{P}}_{\mathrm{H}\mathrm{i}}-{\mathrm{B}\mathrm{P}}_{\mathrm{L}\mathrm{o}}}\left(C_p-\mathrm{B}{\mathrm{P}}_{\mathrm{L}\mathrm{o}}\right)+\mathrm{I}\mathrm{A}\mathrm{Q}{\mathrm{I}}_{\mathrm{L}\mathrm{o}},\end{array}\end{array}$$

(1)

$\begin{array}{cc}& \mathrm{A}\mathrm{Q}\mathrm{I}=\max \left\{\mathrm{I}\mathrm{A}\mathrm{Q}{\mathrm{I}}_{\mathrm{1}},\mathrm{I}\mathrm{A}\mathrm{Q}{\mathrm{I}}_{\mathrm{2}},\mathrm{I}\mathrm{A}\mathrm{Q}{\mathrm{I}}_{\mathrm{3}},\dots ,\mathrm{I}\mathrm{A}\mathrm{Q}{\mathrm{I}}_n\right\}.\end{array}$

(2)

4.3.2. Streaming Semantic Query

The streaming semantic query provides us with a basic function to discover and query the IOT resources in real time. We implement the following 6 query examples to illustrate its applications (Table 1). All streaming queries are showed in Appendix A.

Table 1

IOT streaming query examples.

Query	Goal
Q1	Query the type and value of all sensors in a specific room
Q2	Query the value of PM2.5 sensor in a specific room
Q3	Query the value of all CO sensors and corresponding room ID
Q4	Query the value of all SO2 sensors and corresponding sensor ID
Q5	Query all the room ID and sampling time with humidity sensor
Q6	Query the temperature and sampling time in a specific room

4.3.3. Streaming Semantic Reasoning

Reasoning is ubiquitous in the IOT environment. We can derive corresponding conclusions if certain data streams trigger reasoning rules. For example, we can get warnings if some pollutants' concentration exceeds normal range, such as temperature and CO. Other reasoning examples include inferring the health status and sleep quality of a person based on some wearable devices such as smart band.

Here we give the example of reasoning the human comfort level. The application will help us to keep track of the conditions of indoor rooms, and automatically adjust the indoor environment to prevent heatstroke or cold in time. Specifically, two kinds of sensor streams (temperature and humidity) will be first integrated to compute the Human Comfort Index according to (3). Then based on reasoning rules in Appendix B, the level of human comfort will be derived. Consider

$$\begin{array}{cc}& \begin{array}{c}{I}_{\mathrm{H}\mathrm{C}}=T-\mathrm{0.55}\left(\mathrm{1}-H_R\right)\left(T-\mathrm{58}\right)\end{array}.\end{array}$$

(3)

5. Experiment and Evaluation

In this section, we introduce the experiments and evaluations. First the experimental environment is briefly presented including the configuration and data. Then extended experiments are performed to evaluate the system's functionality and scalability.

5.1. Experiment Setup

Configuration. The experiment is implemented on a SPARK cluster with three machines. Each node has 16 GB DDR3 RAM, 8-core Intel Xeon E5606 CPUs at 2.13 GHz, and 1.5 TB disk. The nodes are connected by the network with the bandwidth of 1000 M/s. All the nodes use CentOS6.4 with the software types JDK-1.7.0, Scala-1.10.1, and SPARK-0.9.0.

Data. The experimental data is generated by our stream data generator whose schema is based on the concept model in Figure 3. The main parameters of the generator are R and T, denoting the number of homes and sampling time, respectively. The number of homes is in proportion to the number of sensors denoted by $N_s$ ( $N_s=15\ast 5\ast R$ , where 15 and 5 represent the number of sensors in a room and rooms in a home). Sampling time stimulates the rate of sensor stream. The average data size generated by a sensor within a sampling time is 0.5 (KB). Data generator and all the source codes are available in (https://github.com/hualichenxi/Semantic-IOT-Engine/tree/master/Experiment).

5.2. Functional Evaluation

For functional experiment, we first briefly introduce the implementation of the elastic streaming semantic engine. Then based on the proposed engine, 3 kinds of representative IOT applications are presented to illustrate the functional characteristics of our system.

We built our elastic streaming processing engine based on the efficient in-memory cluster computing framework SPARK, which provides us with rich data abstraction and operation abstraction to meet the needs of various IOT applications. For the data model, we use the DStream (Discretized Stream) to model the window streams. For the processing model, operators provided by SPARK such as “filter” and “map” are translated into the processing primitives to effectively implement the different IOT scenarios.

Presently, we have preliminarily implemented the IOT mash-up subsystem, query subsystem, and reasoning subsystem. Every subsystem acts as a module of our elastic semantic processing engines. Once an IOT application requests service, corresponding subsystem will be activated to run a SPARK job to return continuous results. Figure 5 shows partial running results of the previous 3 use cases. For more results, readers can access (https://github.com/hualichenxi/Semantic-IOT-Engine/tree/master/Experiment).

Figure 5

The snapshot of indoor environment monitoring scenarios.

5.3. Scalability Evaluation

For scalability experiment, we use the AQI use case to illustrate the performance of our engine by increasing the number of sensors streams. During the process of SPARK streaming execution, we will write a total delay (TD) into the log file after the data in a time slice has been processed completely. The parameter records the total time from receiving window data to output final results. In our experiment, the time slice (D) is set as 5 seconds and we will run the program for 300 seconds. That is to say, 60 TD will be written into the log file.

Figures 6 and 7 show the trend of the processing time (TD) in single node and cluster with varied sensors. For single node experiment, the number of sensors is varied from 15,000 to 150,000. For cluster experiment, the number of sensors is varied from 75,000 to 750,000. The two figures only show the processing time for parts of sensors so that we can recognize the broken line well. From both figures, in the beginning of executing a SPARK job, TD is not stable (0~50) for all varied sensors. After a while, TD will stay in a comparatively stable level. Here we choose these TD in the last 250 s and compute their average value denoted as $\mathrm{T}{\mathrm{D}}_{\mathrm{S}\mathrm{A}}$ . To capture the fluctuation of the processing time, we also compute the standard deviation σ of $\mathrm{T}{\mathrm{D}}_{\mathrm{S}\mathrm{A}}$ .

Figure 6

Processing time (TD) with increased sensors in single node.

Figure 7

Processing time (TD) with increased sensors in cluster.

Equation (4) computes the system's throughput Q (MB/s): 0.5 is the average data size generated by a sensor in a sampling time, $N_S$ is the number of sensors, and $\mathrm{T}{\mathrm{D}}_{\mathrm{S}\mathrm{A}}$ is the processing time. Consider

$$\begin{array}{cc}& Q=\frac{\mathrm{1}}{\mathrm{1024}}\times \frac{\mathrm{0.5}{N}_s}{\mathrm{T}{\mathrm{D}}_{\mathrm{S}\mathrm{A}}},\end{array}$$

(4)

$\begin{array}{cc}& \text{sizeup}=\frac{\text{computing\hspace{0.17em}\hspace{0.17em}time\hspace{0.17em}\hspace{0.17em}for\hspace{0.17em}\hspace{0.17em}processing\hspace{0.17em}\hspace{0.17em}}m\times \text{data}}{\text{computing\hspace{0.17em}\hspace{0.17em}time\hspace{0.17em}\hspace{0.17em}for\hspace{0.17em}\hspace{0.17em}processing\hspace{0.17em}\hspace{0.17em}data}}.\end{array}$

(5)

Tables 2 and 3 show the execution results in single node and cluster. Figures 6 and 7 show the throughput and processing time with increased sensors. We can conclude the following results from the tables.

Table 2

Throughput and sizeup in single node.

$N_s$	TD (s)	TD/D	σ	Q (MB/s)	Sizeup
15,000	0.308	6.17%	0.06584	23.759	1
30,000	0.566	11.32%	0.08999	25.879	1.836
45,000	0.635	12.70%	0.09619	34.605	2.06
60,000	0.727	14.54%	0.11524	40.305	2.36
75,000	0.751	15.02%	0.11416	48.769	2.44
90,000	0.821	16.42%	0.10602	53.518	2.66
105,000	0.956	19.12%	0.11979	53.635	3.10
120,000	1.218	24.37%	0.13368	48.092	3.95
135,000	1.785	35.70%	0.19057	36.930	5.79
150,000	2.440	48.81%	0.24599	30.012	7.92

Table 3

Throughput and sizeup in cluster.

$N_s$ (k)	TD (s)	TD/D	σ	Q (MB/s)	Sizeup
37,500	0.438	8.76%	0.08675	41.825	1
75,000	0.543	10.85%	0.08929	67.474	1.24
112,500	0.566	11.32%	0.10169	97.025	1.29
150,000	0.575	11.49%	0.10676	127.452	1.31
187,500	0.623	12.46%	0.13582	146.922	1.42
225,000	0.627	12.54%	0.13051	175.160	1.40
262,500	0.753	15.07%	0.14888	170.142	1.72
300,000	0.991	19.83%	0.17464	147.753	2.26
337,500	2.519	50.37%	0.56164	65.431	5.75
375,000	3.294	65.89%	0.78245	55.583	7.52

Firstly, we can get the correlation among relevant variables: ${\mathrm{T}\mathrm{D}}_{\mathrm{S}\mathrm{A}}$ , Q, and $N_s$ . From Figures 8 and 9, we can see that ${\mathrm{T}\mathrm{D}}_{\mathrm{S}\mathrm{A}}$ and Q increase with the increasing numbers of sensors. But when the ratio of ${\mathrm{T}\mathrm{D}}_{\mathrm{S}\mathrm{A}}$ to D reaches a certain value, the throughput will decrease rapidly. This is because the computing capability of the system will not be able to catch up with the rate of the input data stream. As a result, the delay caused by the last time slice will have an effect on the next process and the system will become more and more unstable. The evident increasing of σ also reflects the result that ${\mathrm{T}\mathrm{D}}_{\mathrm{S}\mathrm{A}}$ has a larger fluctuation. The correlation analysis can help us control the rate of input stream and set proper time slice to accommodate the ability of the system.

Figure 8

Throughput and TD with increased sensors in single node.

Figure 9

Throughput and TD with increased sensors in cluster.

Secondly, Tables 2 and 3 show that our system achieves high throughputs: more than 53 MB/s and 175 MB/s in single node and cluster. Benefiting from the system's elastic processing ability, it can concurrently process more than 300,000 sensor streams efficiently.

At last, the tables show that our system achieves excellent scalability. For both single and cluster configuration, the sizeup (see (5)) of m times input is much less than m. Particularly for the cluster, when the input stream increases by 6 times ( $N_s=\mathrm{262,500}$ ), the processing time only increases by less than 2 (1.72) times. The results mean that the TD increases much more slowly than input data size and our system works better in processing larger input stream. At the same time, the two tables also show that the processing capability of cluster is much better than single node. For example, when the number of input stream sensors is 150,000, the execution time in cluster is 0.575, compared to 2.440 in single node. The best throughput in cluster (175 MB/s) is more than 3 times the one in single node (53 MB/s). It proves our system achieves good flexibility and elastic scalability: it can adapt to various different application scenarios and requirements by adding computing nodes.

To sum up, the results demonstrate excellent scalability regarding both the size of input stream and number of nodes.

6. Conclusion and Future Work

To effectively process massive streaming IOT data, the paper presents a large-scale real-time semantic processing framework for various IOT applications. According to the framework, we have implemented an elastic streaming engine based on popular large-scale distributed computing platform SPARK. Based on the engine, a typical use case on home environment monitoring is given to illustrate the efficiency of our engine. The results show that our system can scale for large number of sensors streams with different types of IOT applications. For future work, we are planning to deploy spatial semantic support in our distributed semantic engine to process all kinds of location-based IOT applications such as taxi service and parking service.