Abstract
In traditional Chinese medicine, the growth situation of the surface of nails reflects the physiological condition of the human body. Diagnosis by nail can effectively predict and prevent disease. Human nails have a high degree of uniqueness, and it can be used for biometric recognition. In this work, microscope sensor was used to capture the clear image and segment the lunula and nail plate effectively through image preprocessing. Fingernails’ image is managed as the identity authentication. Histogram of oriented gradients and local binary patterns are used to capture the characteristic value. It uses support vector machine and random forest tree for classification. The performance of each feature extraction algorithm was analyzed for the two classifiers and the deep neural network algorithm was used comparatively. Furthermore, the security and privacy of the Internet of Things is still a challenge. This work uses the highly anonymous blockchain technology to effectively protect data privacy and manage each user’s data through the blockchain, in which any change or manipulation can be recorded and tracked, and the data security is improved. Therefore, this article presents a nail analysis management system with the use of microscopy sensor and blockchain.
Keywords
Introduction
Nails are mainly composed of keratin and protein, and they are the basic structure of the outer layer of the skin. In addition to the nutrients they receive, their growth also depends on diseases, drugs, trauma, age, gender, diet, inheritance, and external environmental factors, such as chemical pollution effects. All of these factors will affect the actual growth rate of nails. In traditional Chinese medicine, nails reflect the health state of the human body. Nails are important clues to chronic diseases or diseases associated with systemic visceral organs. Healthy nails are ruddy, tough, curved, smooth, shiny, and their roots are shown as half-moon gray. In Chinese medicine, it is believed that lunula is the representative of human body’s vital essence, so it is also named as the health circle. Any change in the color or size of the lunula may reflect the unhealthiness of the body. Chaikan and Karnjanadecha 1 propose a nail diagnostic system, as well as methods of predicting disease and infection through the analysis of the appearance of blood capillary, microcirculations, and hemorrhages under the nail plate. Devi and Banu 2 sort out the related research work of nails in recent years, and the methods used include image contrast adjustment and magnification, 3 feature measurement, 4 image enhancement, 4 Laplacian skeleton algorithm, 5 and skeleton and binary algorithms. 6 Human nails have a high degree of uniqueness, and it can be used for biometric recognition. Sultana et al. 7 propose innovative biometric features, including nails, which can be used for biometric authentication. Easwaramoorthy et al. 8 design a system process to extract features only from the nail plate, so as to achieve a robust identification effect. In addition to applications in medicine and biometric authentication, nail beauty and make-up in the consumer market is also a great application space. Fujishima and Hoshino 9 propose a virtual nail art system, which develops a nail chip transformation method using artificial coordinates with a new parameter definition method and the depth direction. Consumers can easily try nail chips without damaging their nails. In our work, an experiment was conducted to capture clear nail images through microscope. It put forward a method of image preprocessing in an attempt to segment the different parts of the nail: lunula and nail plate. In addition, the results after segmentation will be stored in the management system, so as to analyze and record changes in the nails easily. And nails can be used for biometric authentication.
Internet of Things (IoT) consists of a large number of devices, which generates processes and exchanges large amounts of security data and privacy-sensitive information. Therefore, it is the target of various network attackers. 10 As for the energy consumption and processing costs, the traditional security methods are often expensive for the IoT. In order to protect user privacy, existing methods usually send noisy data or incomplete data, which may potentially hinder some IoT applications to provide personalized services. 11 As a result, IoT requires lightweight, scalable, and distributed security and privacy protection. However, nail image represent personal privacy data. When the data are stored in the system, it must provide a set of management mechanisms, so as to achieve the access method for data security and privacy to provide mutual trust. With the emergence of Bitcoin in 2008, 12 blockchain becomes the core technology and provides a solution to the problem of trust. Due to its distributed, secure, and private nature, blockchain can overcome the above challenges. The blockchain technology is based on the decentralized peer-to-peer network, which uses the cryptology principle and timing data, combined with consensus mechanism. It uses them to protect the coherence and continuity of the nodes in the distributed database, so as to verify the information and track it in real time, but it is difficult to be tampered and cannot be masked. Furthermore, through a high degree of anonymity technology, it creates a private, efficient, and secure shared value system. The characteristics of blockchain make it highly valued and applicable in the financial industry. In recent years, blockchain not only is used in Fintech, but also develops quickly in the application of IoT as referred in previous works.13–23 Our work uses blockchain to enhance the privacy of nail data, and it is difficult to expose the information through the certification of the blockchain. The certification of the blockchain can enhance the correctness and trustworthiness of the record. Finally, since blockchain is highly anonymous, users can choose to share de-individualized information, so as to provide more research data to medical institutions or research units.
The main contribution of this article is to propose a nail analysis management system. It uses the microscopy sensor to capture the nail image and puts forward a nail image preprocessing algorithm, so as to segment the lunula and nail plate effectively. Lunula and nail plate images help the researchers to do further analysis of the users’ health or disease. And nail images can be used for identity management. As for the biometric authentication of nails, our work uses two characteristic values: histogram of oriented gradients (HOG) and local binary pattern (LBP), and compares the accuracy of different algorithms under support vector machine (SVM), random forest tree, and deep neural network. Nail images are personal privacy data. When the data are stored in the system, the system uses the blockchain technology to protect the privacy and correctness of the data, so as to improve users’ trust in the system. In addition, through blockchain technology, any modification records of data are able to be tracked. A visual sensor network consists of wireless nail visual microcopy sensors that can sense and capture images for analysis and application. Therefore, we designed this architecture to meet the management and application of a large number of visual sensors.
The rest of the article is organized as follows. In section “Related works,” the related works are discussed. Then, the proposed system is presented in section “The proposed system.” In section “Experimental results,” we will demonstrate our experimental results. Finally, the conclusions are presented.
Related works
Image preprocessing of fingernails
Tsuda et al. 24 find that the distribution of pixels in the nail and skin areas is different and propose the nail detection system using the distribution differences in nail color pixel. The system takes the color of nails to do the calibration process. And then, it extracts the nail area. Tsuda et al. 25 propose a symmetrical extraction method based on the reflex characteristics of the nail and the position of the finger in contrast of the light source. It improves the accuracy of the segmentation of nails. Kumuda and Dinesh 26 propose a fairly nice method for segmentation of fingernail patterns and dividing the nail into three parts: lunula, distal free edge, and nail plate. The segmentation method is divided into two stages. During Phase 1, the color image is converted to grayscale image and contrast-limited adaptive histogram equalization (CLAHE) is used to enhance image contrast and morphological operations for noise corrections. During Phase 2, blob correction and watershed labeling methods were used for fingernail pattern segmentation. Experimental results have a significant effect. However, this approach does not apply to the experimental dataset herein. Beacuse the photosensitive components of the acquisition system are different, as well as the affluence of light and image quality, this approach is not applicable. Therefore, in our work a method of image preprocessing was proposed in an attempt to segment the different parts of the nail: lunula and nail plate. In order to maintain the image quality of nails, in this work a microscope was used to capture the nail images.
Biometric authentication using fingernails
Easwaramoorthy et al. 8 design a system process to extract features only from the nail plate. Since the edge of the nail plate is not continuous, a pentagonal structure is removed from the nail plate using the semantic points mediation technique, and the textural properties of the region of interest (ROI) and the nail plate are obtained based on this structure. Principal component analysis (PCA), independent component analysis (ICA), and Haar wavelet and scale invariant feature transform (SIFT) are used for feature extraction. Naive Bayes and SVM are used as the identification category. Barbosa et al. 27 introduce the transient biometrics, which is defined as the biometric identification technology that relies on biometric features that have been shown to change over time. The nail image constitutes a temporary biometric feature. The proposed solution can be divided into three phases. Phase 1 involves image segmentation and preprocessing. Phase 2 uses the LBP feature 28 to extract the biometric identification signature. Phase 3 uses Bayesian classifier to conduct matchmaking of the biometric signature. However, before the biometric recognition process, manual landmark needs to be done on the nail plate. Therefore, in our work, we proposed an efficient image preprocessing method; two characteristic values were used: HOG and LBP, and the accuracy of different algorithms under SVM, random forest tree, and deep neural network was compared.
Blockchain for IoT
Nakamoto 12 replaces the trusted electronic payment system with the cryptographic proof, which allows any parties reaching consensus to be able to trade directly without the need for the trusted third parties to participate. Through an end-to-end distributed time-stamp server, the system generates time-based calculation proof of transactions and then solves the problem of double payment. In order to achieve a distributed time-stamp server on an end-to-end basis, a Proof of Work (POW) system similar to the Hashcash technology 29 is required. POW includes scanning a value. Taking SHA-256 Secure Hash Algorithm for example, the hash value starts from one or more 0. On average, as the number of 0 rises, the amount of work required will increase exponentially, and the verification can be done by performing a hash calculation. In the time-stamp network, the work proof can be done by increasing the number of nonces in the block until the nonce value of the enough number of 0 of the hash corresponding to the block is provided. The block cannot be changed unless a certain amount of work is completed. When the subsequent blocks are linked later, if you want to change the blocks, you need to complete the work of all the blocks after that. In order to modify the previous block, the attacker must re-complete the block and the work of all the blocks after that, and catch and exceed the work of the non-attacker node. In the context of the IoT, it is not suitable to use the blockchain directly. There are several major challenges, such as resolving the high resource requirements for POW, long delays in transaction confirmation, broadcast transactions of low scalability, and blocks to the whole network. Therefore, Dorri et al. 13 propose a lightweight blockchain method to be applied in the smart home environment and provide distributed security and privacy to the device. In addition, taking the limited resources of most devices of the IoT into account, during the experiment, as for the proposed method, it has a significant effect on measuring the energy consumption. Samaniego and Deters 14 evaluate the blockchain technology applied in the cloud and fog as the hosting platforms. It uses blockchain to manage device configurations and store sensor data and micro-payments. IoT devices need to communicate and synchronize with each other. However, under the circumstance that thousands or tens of thousands of IoT devices are connected, there may be some restrictions and problems during synchronization while using the current server–client model. Therefore, Huh et al. 15 use the blockchain to construct the IoT system. The blockchain is used to control and manage IoT devices and Ethereum16–18 is chosen as the development platform for the blockchain. Ethereum provides the function of smart contract and can easily manage the configuration of the IoT devices and build the key management system. Christidis and Devetsikiotis 19 verify the blockchain and smart contracts for IoT. In addition, it puts forward the two-point blockchain–IoT integration: (1) promoting the sharing of services and resources to create a service market between devices and (2) allowing the present time-consuming work flows to be automated in an encrypted and verifiable manner. Furthermore, it is concluded that the blockchain–IoT integration is powerful and can lead to significant changes in multiple industries, paving the way for new business models and novel distributed applications. Other related research about the application of the blockchain in IoT is shown in previous works.20–23
The proposed system
Figure 1 shows the system architecture proposed in this article. Users are scattered in different areas, and a microscopy visual sensor is used to capture nail images. The nail image is transferred to the smart device via USB. The image is preprocessed in the smart device, and the processed image is transferred to the data management center. Visual sensor and smart device are collectively referred to as a device node. Data management center is responsible for receiving and managing the image data and further realizes the nail recognition function. Data management center is connected with zero or more device nodes. And multiple data management centers are connected to form a fingernail network that manages the nail information of the entire network together. In other words, the entire network of nail information, called ledger, has numerous copies, distributed on the data management center in the blockchain (fingernail network).
The system architecture.
Figure 2 shows that user 1 wants to upload the nail image to the data management center. The red point represents a device node, which is currently used by user 1, and issues the request for uploading images. At this point, the device node sends the information to the data management center. After the data management center receives the message, it will indicate that its ledger has added the image data of user 1. Each data management center connected to the fingernail network will receive the message, record the transaction to the ledger, update the current data, and pass on the message to the nearby data management centers.
The flow of the fingernail transaction request message.
In order to be able to trade in the blockchain (fingernail network), you need a wallet, which allows you to store and trade your data. Only through wallet your data transactions can be conducted; therefore, each wallet is protected by the special encryption method with the use of a pair of unique private key and public key. Therefore, when user 1 wants to upload the data, data management center 1 will use its private key to encrypt the transaction message and send it to the network. And then, each data management center will use the public key for decryption, so as to confirm that it is sent by data management center 1. After the encryption is completed, it will produce an electronic signature, and it will be used to confirm the source and authenticity of the transaction message. The content of the electronic signature is a string of text, which is composed of transaction messages and private key, so it cannot be used in other transaction information. If you change any of the characters in the transaction message, the electronic signature will change, so it is difficult for the hacker to change your message, as shown in Figure 3.
The flow of safe transaction in the blockchain.
Fingernail image preprocessing
As the nail cells are produced, the nails are compressed, advancing forward, appearing flat and translucent. Because the capillaries are visible, the nail plate appears pink. And the half-moon lunula is in the roots of the nails. Varying from person to person, not everyone has lunula and not every finger has lunula. The foremost part of the nail is called the free edge, as shown in Figure 4. This article proposes a method of image preprocessing. The implementation process is divided into two steps: step 1—capture the outline of the finger and extract the ROI of the nail plate; step 2—use the K-means clustering method 30 to obtain the nail plate’s clustering image and smoothen the clustering image, so as to make the lunula and nail plate easier to distinguish.
The structure of nail.
While using the microscope to capture the nail image, there will exist shadow in the image. In order to minimize its impact, the shadow will be removed and the outline of the finger will be extracted. First, the original image will be converted to hue–saturation–value (HSV) color space through color conversion. Among them, V is the information of brightness and the fixed V value makes the shadow to be eliminated from the original image. And then, it will be converted back to the image of the RGB color space. Through the Otsu thresholding way, 31 it will produce the binary image. The binary image is used as the mask to compare with the original image and extract the outline of the finger, as shown in Figure 5. After getting the outline of the finger, it will extract the nail plate ROI on the finger. First, the outline of the finger is sharpened through the Laplacian filter and histogram equalization is carried out. 32 And then, through finding the largest circle, a bounding box is obtained and the nail plate ROI is extracted. Figure 6 illustrates the results of the experiment.
The flow of finger contour extraction.
The flow of extracting nail plate ROI on fingers.
From the image of the nail plate, the lunula needs to be segmented from the nail plate. Our work uses K-means clustering to group the image colors. At the beginning, we convert the nail plate to the HSV color space through color conversion and perform color grouping, but the effect is not good. Therefore, we convert the nail plate to the gray scale and perform K-means grouping, which can be categorized under three groups. From the experimental process, we find that the outline has a clear view, but there is too much noise. This article designs a method to smoothen the grouped image. Noise correction is carried out for the group image through morphological operations. The steps are as follows: erosion, dilation, and adaptive threading of the image, 33 and the steps are repeated three times. And then, the median blur 33 is performed to get the smooth image. Finally, use the search method of angular linear to compensate the disappeared points on the edge by using the minimum bounds of lunula and nail plate, and make the lunula and nail plate can be segmented out. Figure 7 shows the experimental results, and the obvious difference between the nail plate and lunula can be found.
Segmentation on nail plate.
Biometric authentication using fingernails
The nail surface itself is streaked and rugged. Under the influence of light or external factors, it will produce the unique texture. Since the nail plate has such a rich texture, in this work, we explore two reliable features, which are HOG and LBP, to capture the texture description on the surface of the nail plate. HOG and LBP use SVM 34 and random forest tree 35 as the classifier for biometric identification. HOG technology has become an important location detection method for local regions, and it has been widely used in features of object detection, which is initially used for pedestrian detection. 36 LBP can be considered as a method to summarize the local gray-level structure. LBP is widely used in facial recognition 37 and texture classification. 38 Therefore, HOG and LBP have a high degree of recognition ability for texture analysis.
The latest development in texture classification is the use of convolutional neural network. 39 In this work, we design a deep network which is shown in Figure 8. The function of the classification model is to resize the preprocessed nail image to 64 × 64. The activation function adopts a Rectified Linear Unit (ReLU), 40 and the activation function of the former layer of the output layer adopts Softmax. 40 The loss function adopts cross-entropy. 40 The optimizer selects stochastic gradient descent and performs 50 epochs. 40
The proposed structure of convolutional neural network.
Define the data structure of the block and create a blockchain
The data structure of the block contains five fields: (1) Hash—the hash ID of the block; (2) Previous Hash—record the hash ID of the previous block; (3) Time-stamp—the setup time of the block; (4) Transaction—the contents of the transaction; and (5) Root—the starting point of the block. Each block has a unique number, which is generated using the SHA2 41 algorithm, and it is called the block hash ID. The contents of the transaction are adding, deleting, and modifying the nail image data.
The first block of any blockchain system (Block 0) is called Genesis Block. It uses Merkle Tree 42 when building the blockchain. Merkle Tree, also known as Hash Tree, is the binary tree that stores hash values. The value of the Merkle Tree’s leaf node is the cell data of the dataset or the hash of the cell data. The value of the non-leaf node is the hash value of all its child nodes. Any block scattered in the system has a number. This number is the order in which the blocks are generated. The way in which the blocks are generated can be designed by the system developers of the blockchain. As for Bitcoin’s blockchain system, the block generation process is known as mining. Therefore, if we want to achieve a blockchain, we should establish the Genesis Block first. Genesis Block will have its own hash value, and this value is generated by the hash algorithm, so it is also called hash ID. Using the algorithm, it will produce Block 1 through calculation. Block 1 will also have its own hash ID, and at the same time it will use the previous hash field to connect to the previous block. The previous block is the Genesis Block (Block 0). Similarly, Block 2 and more blocks are generated. These blocks are connected with the previous hash field, and this chain is the blockchain, as shown in Figure 9. Each block can be used to keep accounts, and it will be a complete ledger when the entire blockchain is connected together. Therefore, blockchain is also called distributed ledger. Each transaction will add a block. This method is used in this work to record the nail information of each user.
Example of the blockchain.
Difficulty of blockchain
As we all know, Bitcoin’s mining difficulty is very high. It means generating new blocks is a very difficult task. Bitcoin designs mining to be very difficult, and there is a very important reason in fact: to avoid the arbitrary generation of blocks. In order to generate new blocks, there will be the so-called difficulty, and the main purpose is to determine the generation conditions for the new hash value. Before generating the new blocks, you must calculate the block hash of the block first. Therefore, how to determine whether the hash value is usable is an important design mechanism. SHA2 is used to generate hash values. Whether the new block can directly use this hash value depends on whether there are enough zeros in front of the hash value. After the new block is produced, there is a basic principle that the difficulty must be more and more. For example, the difficulty of the previous block is at least two zeros; as for the next new block, the hash can only be successfully established after producing the hash value with three zeros in the front. Our work simplifies this function. The difficulty of the previous block should have at least two zeros, and the hash of the next new block must produce the hash value with at least three zeros, and so on.
Experimental results
In order to evaluate the system architecture proposed by us, we use an UPMOST (UPG622) DMC microscope camera sensor and an Android smart phone to integrate into a nail analysis capture device (device node) through the USB transmission interface. The data management center uses the i7-6700 quad-core processor plus 32 GB RAM. It is equipped with an NVidia GTX1080 display adapter, as shown in Figure 10. The benefits of using the graphics processing unit (GPU) is that it can increase the efficiency of deep neural network inference and the generation of new blocks (transaction record). Using virtual machine technology in this machine, it creates four data management centers to simulate the distributed management mechanism of the account book.
The setup of experimental environment.
Experiment 1: fingernail image preprocessing
It uses a USB microscope to connect to the Android phone and takes photos of the five fingernails (thumb, pointer finger, middle finger, ring finger, and little finger) of four men as shown in the left part of Figure 10. There are 20 nail images, including those with lunula and without lunula, which are shown in Figure 11. The issue of light’s influence is very important in the nail image. During capturing the images, light emitting diode (LED) lights are used to overcome the variation in the nail plate caused by the light, and it can clearly show the details of the nails, such as free edge, cross striation, and vertical striation. However, it still leads to the presence of shadow in the image. Therefore, our work removes the shadow part first, so as to enhance the segmentation efficiency. The experimental results are shown in Figure 12. From the experimental results, the lunula and nail plate in the five fingers can be distinguished. However, during the process of differentiation, the boundary compensation method uses the straight line for connection, and it is not natural enough in some cases. In the future, we will improve the boundary compensation method, and taking the situation of free edge into account, we will segment the overall structure of the nail integrally, and it will be applied in more places.
Dataset of the experiment.
Experimental results of the fingernail images.
Experiment 2: biometric authentication
Since the sample training requires a lot of nail data, our work downloads from the Visual Computing group website of Norges Teknisk-Naturvitenskapelige Universitet (NTNU) 43 with the use of the dataset. The Transient Biometric Nails Dataset (TBND) v2 dataset consists of three subsets, and there are a total of 93 fingernails for each subset, which are collected on different dates. The first subset D01 is composed of images obtained on the first acquisition day. The second subset D02 is composed of images obtained the next day after the first acquisition. The third subset D30 is composed of images obtained 30 days after the first acquisition.
In our work, we use the samples D01 and D02 for training. D30 is used for testing. As the nail grows slowly, we use D30 for testing, and it can prove that the nail is able to be used for biometric authentication. All the images have been previously processed by the algorithm we designed. Our work captures the characteristic values of HOG and LBP, and uses SVM and random forest tree algorithm to evaluate the accuracy under the settings of different parameters, as shown in Table 1. The training accuracy is generally quite high. However, the testing accuracy is only 63%–64%. Through checking the dataset, we find that the reason may be that data collection is not conducted in a consistent environment. Therefore, it results in a great variability in data.
Classification performance.
SVM: support vector machine; HOG: histogram of oriented gradients; LBP: local binary pattern.
As for the diversity of data, in order to make use of the limited training data to the best, this article uses the existing data to produce more training data through rotation, moving, scaling, and shearing. Each training dataset produces 1000 training data. In other words, there were only 186 training data first, but it increased to 186,000 now. The data training and testing with 50 epochs are conducted by using our neural network structure, and the results are shown in Figures 13 and 14. Training accuracy gradually converges to about 98%. Testing accuracy goes up and down, and it is more than 60% on the 28th, 42nd, and 45th epochs. Among them, the accuracy of the 45th epoch is 70.56%. In the future work, we should collect more information in a consistent environment, so as to improve the accuracy.
The training accuracy.
The testing accuracy.
Experiment 3: simulation of blockchain
Our work uses node.js to simulate the process of blockchain transactions. Using the virtual machine technology, it creates four data management centers to simulate the distributed management mechanism of the account book. During the process of generating new blocks (mining), we simplify the difficulty of blockchain and the workload of POW. Therefore, the generation of a block can be completed in 1 s. The synchronization of the account book in four scattered data management centers is also completed based on the mechanism. Since the simulation of the blockchain is conducted in the data management center, it will not cause bottlenecks to the resource-limited IoT device. Therefore, we think that the architecture we proposed is suitable for use in distributed visual sensor networks.
Conclusion
This study proposes a nail analysis management system. It uses microscope to capture the nail image and proposes a nail image preprocessing algorithm, so as to segment the lunula and nail plate effectively. The lunula and nail plate images help researchers to do further analysis of the users’ health or disease. And the nail images can be used for identity management. As for the biometric authentication of nails, it compares the accuracy of different algorithms. In addition, nail images are personal privacy data. When the data is stored on the system, the system uses blockchain technology to protect the privacy and correctness of the data, so as to improve users’ trust in the system. Through blockchain technology, any modification records of data are able to be tracked. Therefore, an architecture is proposed in this article to meet the management and application of a large number of visual sensors. In the future work, we would like to cooperate with hospital research centers to do the health analysis through fingernails.
Footnotes
Handling Editor: Tien-Wen Sung
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was financially supported by the Ministry of Science and Technology of Taiwan (under grants nos 104-2221-E-006-119-MY3, 103-2221-E-006-145-MY3, and 104-3115-E-194-001).
