Abstract
In recent decades, the main data traffic of mobile communication is based on the RF technology. Even though the efficiency from optimization for utilization or reuse is applied, there is a limitation of increased traffic demand of RF communication. Visible light communication (VLC) is a new technology that can cooperate with RF to address this limitation. The standard for VLC was published in 2011 as IEEE 802.15.7, which defined the specifications for the MAC layer and the PHY layer. This standard was one of the first for this technology. The decoded operation from emitting light at the receiver in IEEE 802.15.7 is mainly based on the photodetector. However, with the development of image sensors (matrix of photodiodes) on smart devices, the amendment of IEEE 802.15.7 specification is considered. The extension will focus mainly on image sensor communications, named optical camera communications (OCC). In this paper, we analyze the performance of a camera communications system based on different types of image sensor architecture. Then, we suggest OCC-ID, a new application for broadcasting service using camera communications and the cloud model. The proposed architecture is a typical deployment scenario of camera communications.
1. Introduction
The rapid development of semiconductors has greatly changed the lighting technology, namely, LEDs. LEDs are being increasingly used because of their advantage of energy consumption. Compared with the traditional incandescent bulb, another advantage of LEDs is their switching ability. LEDs can switch the light intensity level at very high frequency. Thus, LEDs are promising for application in communication technology based on visible light communication (VLC). This new technology is a combination of illumination and communication. The limitations of VLC are data rate, communication range, and interference. However, to date, VLC has demonstrated remarkable achievement in terms of data rate. In the near future, VLC will play an important role in wireless communication generation. Compared with RF communications, communication technology based on VLC has advantages of directional transmission, safety, security, and high bandwidth.
The traffic demand on communication and multimedia has increased dramatically. The number of smart devices, which are powerful in terms of processing, memory, display resolution, and camera capacity, is increasing daily. Camera communication is a good business trend in VLC. One of the major advantages of camera communication is the inheritance of existing architecture. Compared with VLC, there is no extended cost on hardware at the receiver. The main cost is data-embedded board at the transmitter and built-in camera controlling application. Optical camera communications (OCC) has higher directional communication than traditional VLC. Unlike RF communication systems, the transmitters in OCC are light sources from LEDs, digital signage, display devices, or other light-emitting devices. The receiver is an application using image data from a camera image sensor. It can be a webcam, digital camera, CCTV camera, or specific image sensor.
In camera communication, the most considerate limitation is the data rate. With the commercial camera, the image frame rate is approximately 30 or 60 fps. Thus, the data rate cannot support high-speed communication. The solution for throughput enhancement can be obtained by MIMO technology [1]. However, this technique depends on the image sensor architecture. Based on the structure and data processing, the image sensor has two basic catalogues: global shutter and rolling shutter. The image construction process of the two techniques is different; therefore, the performances on data rate and topology deployment also vary. The application and service of camera communications are based on short data transmission. In this paper, we propose one new broadcasting application service-based cloud computing topology for camera communications technology. The architecture of the proposed model is shown in Figure 1. The service can be considered a typical application for camera communication. The rest of this paper will detail the analysis of the OCC system and then describe the proposed scenario model of OCC-based cloud service. In Section 2, we provide an overview of the existing related research on OCC. Section 3 presents our proposed broadcasting service-based OCC and cloud architecture. The evaluation of the performances of the system based on two typical image sensor architectures is described in Section 4. Finally, Section 5 concludes this paper.
OCC-ID service.
2. Related Works
The application of VLC was developed a long time ago. However, the main goal of VLC is a new communication system that can be used in a lighting system. With the advantage of rapid switching, LED is an important factor for the birth of VLC. Communication and illumination are combined for a wireless communication system. Research on VLC has a long history. One of the first applications can be considered as voice transmission using light waves from Alexander Graham Bell in 1880 [2]. In early 2000, Keio University proposed LED illumination and communication for a home network. These early studies are regarded as the first motivation for VLC research. The first standard for VLC was initiated by the Japan-based visible light communication consortium (VLCC) [3]. In addition to the VLCC standard, other standards are also available, such as the Visible Light ID system standard from Japan in 2007, Infrared Data Association [4] from USA in 2011, hOME Gigabit Access project [5] from the European Union in 2008, and VLCA [6] from Japan in 2014. However, for a full proposal of VLC, PHY and MAC specification was published by IEEE as 802.15.7 [7]. The IEEE 802.15.7 standard also describes the scenario and the use of VLC for wireless personal area networks, localization applications, and illumination control issue in lighting systems.
Another issue on the receiver for VLC is image sensor communication. This technique is also known as camera communication. When the term camera is used, most people think about an image, picture, and video. Actually, the original function of a camera is for entertainment as multimedia data. With the development of computers, especially in processing and storage, many applications of camera-based computer vision, such as tracking, object detecting, object recognition, and localization, have been proposed. In recent years, with the development of semiconductors and wireless communication, cameras and smart mobile devices have improved. Applications based on cameras and smart mobile devices will be a promising business in the future. Communication is a relatively new issue for cameras from the last decade. The architecture of OCC is shown in Figure 2. The communication issue can be applied for road-to-vehicle communication systems or data broadcasting [8, 9]. The camera-vehicle communication system [10], which includes an LED and high-speed camera, is applied for safety issues in traffic systems. The proposed OCC system provides reference architecture of ITS service with a hierarchical transmission scheme. Digital data will be modulated by OOK modulation on LED array for transmitter. The receiver is the high-speed Complementary Metal-Oxide Semiconductor (CMOS) camera. The “undersampled scheme for frequency shift on-off keying” modulation [11] operates for both rolling shutter and global shutter camera. The scheme is the nonflickering issue for visible light transmitter. For a global shutter image sensor, the “undersampled frequency shift on-off keying” scheme requests high synchronization between the transmitter and the receiver. This requirement is achieved by detecting the subsampled aliased frequencies and phase of light intensity to decode transmission bits. VLC-based CMOS camera systems [12, 13] use LEDs as a transmitter and smartphones with built-in cameras as a receiver. The light source of the LED transmitter is modulated by an enhancement of the on-off keying scheme using Manchester coding. The receiver is an Android camera application. The camera uses a rolling shutter image sensor based on CMOS. The captured image includes multisampling signals of the LED transmitter as pixel bands. Using a simple image processing algorithm, these bands can be reconverted to original binary data.
OCC architecture.
Another well-known technique that can be applied on cameras is barcode. The concept of barcode includes linear code, 2D Matrix, and 2D Quick Read. Barcodes are now widely deployed in various applications such as advertisements, newspapers, product management, social networks, signage, education, and retail. Given the development of smart mobile devices, the barcode has potential for a market's booming growth because the scanning technology is based on a camera. The barcode will be increasingly useful and important to businesses and consumers. Realizing the importance of the barcode and the use of mobile devices, many researchers have looked into the usability and acceptance of barcode-enabled mobile applications. A barcode system comprises three main components, namely, barcode generator, barcode reader or barcode scanner, and database. The generated code is unique for the considered target. It can be a series of numbers, an identification code, or a URL link. Barcode scanners have to be able to read the black-and-white zebra lines or 2D code on surface of objects and send the detected information to a processing terminal such as a computer that can identify the related information of that target in the database. In terms of core topology, the barcode technique is similar to the RFID system. However, the barcode has more advantages with regard to vision management issues. Compared with the barcode and RFID systems, the proposed OCC-ID inherits the visibility advantages of barcode, free movability of RFID, and full illumination of VLC technology.
3. Proposed OCC-ID Architecture
The proposed architecture of OCC-ID is presented in Figure 3. The main contributions of the proposed architecture are camera communications for broadcasting service and cloud architecture for data management. In this topology, the broadcasting information is controlled by a cloud service and displayed on lighting devices such as LEDs, digital signage, sighting boards, or lighting systems. The broadcasting information will be embedded invisibly in a lighting system. The service will not break out the illumination function of the existing light system. The communication function is detected only by a camera. Traditional broadcasting services are used for multimedia data, in which a customer can receive full data on the broadcasting channel. However, as discussed in the previous sections, camera communications have limitations on data rate so the communication topology, which is activated on a long connection, will not show the possibility of practical deployment. The link connection of OCC should be found on short data transmission. In our proposed system, we will embed small information that contains the link address. This address can be called the ID link, which is stored and managed by a cloud server. A customer uses a camera to catch the ID link from the lighting system and then downloads the full data from the embedded link from the data server based on high-speed RF technology, such as WiFi, 3G, 4G, or LTE connection interface. The proposed scenario is a broadcasting service with centralized data control. By applying the cloud model, the broadcasting data link can be managed and updated easily.
OCC-ID architecture.
The OCC-ID system has three main components: transmitter, receiver, and cloud computing infrastructure. The operation flow of service is illustrated in Figure 4. The communication function is embedded on the lighting or display system as a transmitter by an extra module. The broadcasting information will be sent by modulated light. The database of index link and control function is processed by cloud service. The transmitter and cloud infrastructure are connected by wires or wireless connection. The selected technology for link connection must support high-speed multimedia data, such as video streaming. The receiver includes two functions, namely, OCC receiver and link redirect. The ID string from the OCC connection is crucial for data retrieval from the cloud server. The data link will pop up on the receiver terminal. A demonstration of this process is described in subsequent sections.
OCC-ID operation flow.
4. OCC-ID Camera Communication Performance Evaluation
4.1. Camera Communication for OCC-ID
The camera communication technology operates as VLC. The light source signal is absorbed and converted to photons in one image. The architecture of OCC is shown in Figure 2. The digital transmission bit will be defined by one or several light transactions, depending on the modulation scheme. The modulation technique can be classified into two main categories, namely, OOK modulation and subcarrier OOK modulation. Two modulation schemes operate based on changing the state of on-off light source with specific strategy. They can be applied for the flickering condition in illumination at the transmitter. This is one of the basic requirements for VLC. The light illumination intensity should be at a constant level by human eye detection; thus, the illumination of the light intensity cycle must satisfy the eye response. For flickering and dimming [14] in VLC, research and specifications are conducted differently from modulation, RGB color balanced, or coded signal. The subcarrier OOK modulation is the typical technique used for indoor and outdoor applications.
At the receiver, the on-off state of light source will be recorded as group pixel of RGB color for “on” state or background for “off” state in one image. The image construction process depends on the image sensor characteristic. The structure of a camera image sensor can be classified into two main technologies: CCD and CMOS. Following these technologies, the operation of an image sensor is also defined by global shutter and rolling shutter. The imaging process of a rolling shutter image sensor and global shutter image sensor is shown in Figure 5. In a rolling shutter image sensor, image pixels are captured row by row so that different pixel array lines in an image are exposed at different times to absorb the light intensity through the sensor. Given the exposure time delay between rows of array pixels, the rolling shutter mechanism can record the changing status of an object relative to time. For a digital image, this ability causes motion blur such that the objects appear broken. However, in OCC, this mechanism is an important advantage. A series of light sources will be captured and stored in one image. For global shutter, all pixel sensors open and close the shutter simultaneously. Therefore, all pixels of image will obtain light information from sensors simultaneously and over the same duration. This mechanism can provide exposure capability for capturing moving objects. For multimedia imaging, this technique can be useful to obtain a high quality picture. For OCC, global shutter can obtain only one state of light source in one image.
(i) Frequency Shift OOK Modulation. A typical modulation scheme for the nonflickering problem in OCC is frequency shift OOK. An example of the structure of frequency shift OOK for OCC is shown in Figure 6. Digital bit is defined by a specific frequency of on-off light source or the state of light in phase modulation. With frequency subcarrier, at least two discrete samplings of camera are required to detect the pattern frequency. In general, the on-off frequencies of “0” bit and “1” bit are represented as
Subcarrier frequency shift OOK.
Phase frequency shift OOK.
(ii) OOK Modulation. In OOK modulation, the operation differs. The transmission bit will be coded directly to a light source, so all light signaling is important for decoding. The sampling process of OOK modulation is shown in Figure 8. The communication region of interest in the captured image will be exacted and decoded to bit stream. The disadvantage of traditional OOK modulation is flickering. However, this problem can be solved with enhancement schemes such as Manchester code and PPM. Similarly, with frequency shift OOK, global shutter and rolling shutter can be applied for OCC system with the same decoding process. The main advantage of OOK modulation is data rate.
OOK for OCC.
4.2. Performance Evaluation
The image signaling operation of frequency shift OOK on the rolling shutter image sensor is shown in Figure 9. The digital bits code the on or off light signal, and these bits are presented by a group of color pixels. Depending on the structure of the image sensor, the number of discrete light status bits that are captured differs. With global shutter, one image can only contain one, but multiple images can be obtained with rolling shutter. The image signaling of frequency shift OOK for the global shutter camera is shown in Figure 10. For OOK modulation, the process is similar to the frequency shift OOK; the only difference is data rate performance. Figure 11 shows the imaging process of the rolling shutter camera for OOK modulation. Based on the performance of frequency shift OOK with global shutter, some issues for OCC need to be considered. In terms of synchronization, the maximum bit rate of light source that a camera's exposure can capture a full decoded bit needs to be determined. How can we guarantee that the sampling frame will obtain a full bit pattern in frequency shift OOK with a rolling shutter image sensor? For OOK modulation with a global shutter, how can the sampling occur inside the pulse of light signal? The scheduling for sampling should guarantee the captured time of a camera. In practice, the frame rate of the camera is varied due to mechanical structures and light sensitive of image sensor. For example, with different environmental conditions, the frame rate of the camera will fluctuate. In Figure 12, the frame rate of camera fluctuation depends on the background light condition. The camera will attempt to change the exposure value to obtain an in-focus image. Thus, if the background is too dark or too bright compared with the transmitted light source, the camera needs more time to absorb photons to the sensors.
Frequency shift OOK in rolling shutter camera.
Frequency shift OOK and OOK in global shutter camera.
OOK in rolling shutter camera.
Frame rate variation.
Due to the developed business architecture, OCC is mainly deployed on broadcasting topology, so the full-duplex protocol for synchronization is not easy for implementation in commercial product. Most of researches on synchronization issue are considered on transmitter design or receiver sampling scheduling [9, 11, 15, 16] for asynchronous transmission. About the communication range, the image sensor of camera will not get photon directly. The photon from light source must go through one lens with a defined focal length. So compared with visible light communication based photodiode, OCC has more directed LOS. The camera can focus on ROI with low interference. Beside, by using lens, the camera can zoom to ROI at different distance easily. Most of cameras have zoom function to change the focus and zoom distance. So the communication distance depends on the camera lens.
The performance of an OCC system based on rolling shutter and global shutter is evaluated with an implemented scenario, as shown in Figure 13. The receivers are webcam for rolling shutter image sensor and smart NI camera for global shutter image sensor. The transmitter is 15 w LED controlled by the Arduino board and power control driver. The modulation scheme is applied based on frequency shift OOK, OOK, and OOK with Manchester code. Frequency shift OOK and OOK with Manchester code are applied for nonflickering issues of VLC. The configuration of the transmitter, receiver, and communication protocol with different setting modes is shown in Table 1. For OCC-ID service, the transmitter, LED, repeats broadcasting on an 8-bit ID string on a visible light optical channel for one image link from server. The receiver, camera, detects the ID and sends a request link from the server through a WiFi connection. The linked information of embedded ID will show in the receiver devices. A demonstration of the implementation of the OCC-ID service is shown in Figure 14. On the smartphone screen, the upper area is the image of the received signal. The lower part is the linked data of broadcasting ID.
Implementation configuration.
Experience scenario.
OCC-ID demo.
We also evaluate the performance of OCC system based on OOK modulation and frequency shift OOK modulation. Figures 15, 16, 17, and 18 show the data rate performance of the implemented OCC system. With frequency shift OOK of given configuration and rolling shutter camera performance in Figure 15, there are two bits embedded in one image frame. The number of embedded bits can be increased with higher symbol rate. However, to fit with global shutter camera, the frequency separation must satisfy the haft cycle period at the second sampling. With global shutter image sensor, the data rate is fit with the camera frame rate because it requires two image frames for decoding one communication bit as the result in Figure 16. This mode also requests two frames for synchronization which operate at over image sensor shutter speed, 10 kHz. The same principle, OOK, and global shutter image sensor performance in Figure 17 can get one bit for one sampling frame. The data rate can achieve 28 bits per second at frame rate of 30. Different from global shutter and frequency shift OOK mode, the synchronization session of OCC based global shutter and OOK mode is two Manchester code “0” bits. Given the reference from the phase of synchronization bit, the receiver can define the phase of sampling bit. Rolling shutter camera has advantages on the data rate performance compared with global shutter camera in both frequency shift OOK and OOK modulation. In Figure 18, with the rolling shutter camera and OOK modulation scheme, the OCC system can achieve more than 500 bps. For the lower frame rate configuration of basic frame rate configuration, 30 fps, the channel coding is applied to keep the data synchronization and illumination. Compared with other communication systems under commercial use, data rate is one of the most considered limitations of OCC technology. In RF communication, MIMO is a promising solution for data rate. However, with OCC, MIMO depends on image sensor architecture. For a global shutter image sensor, MIMO can be applied by using parallel data in LED arrays. At the receiver, the LED sources are separated and decoded independently by an image processing algorithm. With rolling shutter image sensor, the MIMO technique is a challenge issue because of the pixel construction process. There is a restriction on the distance between light source and image sensor for generating the pixel roll. The light source must cover whole vertical pixel of image.
Frequency shift OOK with rolling shutter camera.
Frequency shift OOK with global shutter camera.
OOK with global shutter camera.
OOK with rolling shutter camera.
5. Conclusion
Optical camera communication has great potential in future wireless communication because of its advantages of VLC and business trend. Currently, it is in the consideration of IEEE 802.17.5 revision. However, issues for OCC about synchronization, data rate, and interference still need to be addressed. Synchronization is an important issue because the receiving signal from a camera is the discrete image of a transmitter without feedback information and camera communication applications are based on broadcasting topology. Rolling shutter shows more advantages than global shutter in terms of synchronization. In this study, we present and evaluate the performance of two image sensor techniques, namely, rolling shutter and global shutter. The performances of the two image sensors and consideration issues will play important roles in standard contribution. Finally, we propose a new service application for OCC based on cloud architecture, named OCC-ID. It can be considered as invisible ID by taking the advantages of visible light communication, camera communication, and cloud computing technology. The ID is embedded in optical channel by OOK or frequency shift OOK modulation. The receiver uses camera to decode the embedded information from LED and, then, is redirected to the broadcasting link from cloud server based on detected ID. The OCC-ID system shows the advantage on dynamic content management compared with traditional ID system.
Footnotes
Competing Interests
The authors declare that they have no competing interests.
Acknowledgments
This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (no. 2013057922).