Non-destructive acoustic screening of pineapple ripeness by unsupervised machine learning and Wavelet Kernel methods

Abstract

In a pineapple exporting factory, manual lines are usually built to screen fruits of non-ripen hitting sounds from millions of undecided fruits for long-haul transportation. However, human workers cannot concentratedly listen and make consistent judgments over long hours. Pineapple screening becomes arbitrary after approximately an hour. We developed a non-destructive screening device aside from the conveyor sorter to classify pineapples automatically. The device makes intelligent judgments by tapping a sound source to the skin of pineapples and analyzing the penetrated sounds by wavelet kernel decomposition and unsupervised machine learning (ML). The sound tapping relies on the well-touch of the skin. We also design several acoustic couplers to adapt the vibrator to the skin and pick high-quality penetrated sounds. A Taguchi experiment design was used to determine the most suitable coupler. We found that our unsupervised ML method achieves 98.56% accuracy and 0.93 F1-score by using a specially designed thorn-board for assisting tapping sound to fruit skin.

Introduction

The export statistics of agricultural products show that pineapples export nearly 50,000 and 2 billion metric tons and NT dollars, respectively. ¹ The most important parts of the pineapple supply value chain are storage and transportation management after harvest Ripening at harvest affects fruit quality directly, transportation, and storage life, ² so the correct and accurate judgment of the ripeness of the fruit is crucial to the processing technology after harvest Previous studies conduct a shipping experiment of fresh pineapples for export in Hawaii and posit that the ripeness of the fruit is the most appropriate when the shipping time is 6–12 days. ³ To match the shipping time and adjustment of supply and demand in different markets, farmers need to choose pineapples of different maturities for storage and transportation and hope that the best sales period of pineapples can be achieved in different markets.

During pineapple production, a professional and easy method should be in place to judge the growth and maturity of each pineapple and adjust the production parameters. Based on the suitable planting cycle of the fruit tree, a critical part after harvest is accurately determining the internal state of the fruit at all stages from fruit to maturity to decide later processing method. The problem that currently plagues the industry is that determining the internal state of the fruit is challenging. The intuition of senior staff can only judge it. Developing experienced harvesting operators is complex, and the judgment in the field lacks feedback about the actual life span on the exhibition shelf. This study dedicates to the development of key technologies for quickly judging the maturity of exported pineapples with acoustic wisdom technology, which can be applied to the judgment of the internal state of the fruit. This study intends to help pineapple exporters to judge the remaining life of pineapples quickly and accurately and increase market competitiveness.

Many people used acoustic pulse methods for the non-destructive evaluation of the hardness of pineapple fruits with a wide range of internal ripeness. ⁴ but the non-destructive ultrasonic methods for quality evaluation of fruit have typically been unsuccessful by the complex structure of fruits. ⁵ The traditional method of identifying the maturity of pineapples is based on farmers’ experience with grading and sub-packing. However, this method is costly in labor and requires experienced personnel to judge. Moreover, too many factors can cause errors. The traditional method can hardly be certified internationally. ⁶ To establish better quality assurance and reduce labor costs, using sophisticated equipment effectively and verifying with traditional classification experience.

Previous studies use many methods for non-destructive research on the quality of pineapples, including X-ray computed tomography imaging ⁷ to classify pineapples. The correlation between the intensity of X-ray tomographic images and fruit maturity is used to identify internal defects in pineapple pulp. The results show that this imaging method can effectively evaluate the internal quality of fruits during storage. Previous studies also use near-infrared light^8,9 to perform near-infrared calibration analysis of soluble solids inside pineapples. Multiple linear regression and modified partial least square regression are used to regress the calibration samples to conduct a non-invasive assessment of the quality of pineapples. However, limited by the cost of equipment and analysis benefits, the above methods for determining pineapple quality are only applicable for laboratory analysis, and they are not suitable for automated determination and sorting of pineapple maturity on site.

Food industry applications were becoming common in recent years using food sorting, classification, and prediction of the parameters, quality control, and food safety by AI technology. ¹⁰ Therefore, it was important to monitor the freshness of food products and crops in the context of the food chain by intelligent packaging systems in the food factory.^11–14

The judgment of pineapple maturity strongly affects the sales volume of pineapple in the market, so developing and using non-destructive acoustic screening methods actively to sort pineapple maturity is crucial. Figure 1 summarizes standard methods and reasons to judge the ripeness of pineapples. Regarding non-destructive acoustic fruit screening methods, previous studies demonstrate that the fruit can be predicted by striking it and then distinguishing the sound by human ears. Such a method can be used to predict the internal quality of agricultural products according to the relationship between fruit resonance frequency and hardness. ¹⁵ Artificial neural networks (ANN), and logic programming were one of the AI variety of algorithms ¹⁶ and it has been used as a tool in aiding real complex problem-solving in the food industry. ¹⁷ A study has been conducted on the various ANN applications in food process modeling where it has only highlighted the food process modeling using ANN. ¹⁸ Acoustic intelligence technology with artificial intelligence (AI) is based on machine learning (ML) technology mainly to allow the machine to have the same thinking logic and behavior mode as human beings. The system is trained through numerous sound waves to define pineapples’ maturity classification rules and characteristics. Finally, according to the above rules and features, pineapples with unknown maturity are classified, and pineapple maturity based on AI is achieved. Therefore, the sound recognition technology we developed is based on the experience of traditional pineapple percussion sound with sensitive electronic sensors, fluid dynamics, and probability decision theory. This technology aims to allow pineapple exporters to easily and quickly judge the maturity of the fruit in the field or on the conveyor belt after a large number of harvests through an affordable instrument and estimate the remaining lifespan of the currently growing fruit. In this way, different sales activities can be launched according to the production and sales, selecting storage and transportation characteristics. For example, according to the saleable period or sweetness, consumers can choose the characteristics they like to promote the sales.

Figure 1.

Reasoning flow for the classification process.

Well-renowned Machine learning classifiers (support vector machine (SVM), logistics regression (LR), naive Bayes (NB), and decision tree (J48) have been used for accurately diagnosing cardiovascular disease (CVD) outcomes and provide significant assistance to cardiologists. ¹⁹ It was showed Machine learning can use data for learning by studying algorithms and their constructions and making predictions by specific inputs. ²⁰ Therefore, the supervised learning model can successfully classify and regression based on empirical risk minimization. ²¹

The literature uses different mathematical models for ML to develop different learning models and study fruit maturity. Acoustic emission in wavelet multi-resolution decomposition and statistical hypothesis test can be used to test the ripeness of watermelons effectively and inexpensively, and the accuracies of training and testing reach 91.67% and 91.76%, respectively. ²² In addition, to classify the sweetness of pineapples, previous studies use electronic noses (gas sensors) to optimize ML parameters and classify pineapple aromas. PCA (principal component analysis) signal processing and mother wavelet are used in the research to reduce the noise of electronic nose signals. The optimal parameters are obtained by the signal processing method to find the characteristics of each signal, and three levels of pineapple sweetness are successfully classified by optimizing the parameters. When using the k-nearest neighbor algorithm, the highest accuracy is 82%. ²³ For the classification of banana maturity stages, previous studies propose a novel convolutional neural network (CNN) architecture. After taking pictures of bananas of different maturity levels, a large amount of image data is used for ML. The training database of CNN architecture includes the original image, positive image, negative image, and volume label of the original image as the input data. Then, the vector is normalized and provides the classification of banana ripening stages. ²⁴ The above studies suggest that we can develop an exclusive AI model by building different databases for ML to obtain the optimal solution to the problem.

Methods

To judge the ripeness of pineapple from the tipping sounds, we set up a sound injection experiment, as shown in Figure 2. Excitation sounds are generated from a computer and injected into the fruit by a vibrator, mediated by an acoustic coupler to ensure that vibrating energy is well-received for a digitalized acquisition device. This study tries to develop an algorithm to distinguish between raw and ripen pineapples and find an acoustic coupler that preserves the signals with the highest efficiency.

Figure 2.

Experiment setup. Excitation sounds are injected to the fruit by a vibrator and mediated by an acoustic coupler to ensure vibrating energy to be well-received for a digitalized acquisition device.

Before introducing the algorithm, we demonstrate the difficulty of sound recognition in this application. A traditional frequency analysis, like Fourier Transform, has been used for many applications. However, they can only be applied to periodic waves. That is, those waveforms can be perfectly represented by sine and cosine functions. For example, in Figure 3, we list two waveforms of raw and ripen pineapples. We can barely find any difference between the two kinds of fruit states in the time domain and frequency domain waveforms. Without resorting to advanced methods, conventional FFTs are challenging to provide a solution.

Figure 3.

The comparison between raw and rippen pineapples. The upper two time scale waveforms represent raw (upper) and rippen (lower). The lower two spectral power distributions represent raw (left) and rippen (right). Without further analaysis, both time domain and frequency domain waveforms can barely show the difference between the two kinds of fruit states.

Wavelet kernel decomposition

The sounds collected in this study contain substantial noise that must be removed, so wavelet kernel decomposition is used to remove the noise. Wavelet conversion has better time resolution at high frequencies and better frequency resolution at low frequencies, and these characteristics meet the resolution requirements for signal analysis at high and low frequencies.

In the audio analysis of the study, understanding the principle of wavelet transformation is necessary to remove the influence of noise. Mathematically, a Morlet wavelet (or Gabor wavelet) is a wavelet composed of a complex exponent (carrier) multiplied by a Gaussian window (envelope). This wavelet is related to human perception, including vision and hearing. According to Morlet's original form, the mother wavelet is defined as, ²²

$${\psi }_{a,b}(t)={\displaystyle \frac{1}{\sqrt{a}}\psi \left({\displaystyle \frac{t-b}{a}}\right),}$$

(1)

where the parameter a is the scaling factor. When $|a|<1$ , the mother wavelet is compressed, with a small degree of support on the time axis, and corresponds to high frequencies in that the mother wavelet becomes narrower and changes faster. When $|a|<1$ , the mother wavelet becomes wider and changes slowly, so it corresponds to the low frequency. $\mathrm{b}$ is the translation parameter, which is used to determine the position of the mother wavelet. According to the uncertainty principle,

$$\mathrm{\Delta }t\mathrm{\Delta }\omega \underset{\_}{>}{\displaystyle \frac{1}{2},}$$

(2)

where t is time, and $\omega$ is the angular frequency ( $\omega =2\pi f$ , f is the instantaneous frequency).

When the time resolution is high, the frequency resolution decreases. On the contrary, when the frequency resolution is high, the time resolution decreases. When the mother wavelet or window function is broader, the value of Δt is larger, so wavelet transformation is conducted. The frequency and time axis can be observed at the same time. The time resolution is better when the frequency is high, and the frequency resolution is better when the frequency is low. The fine or coarse components of the signal can be separated. In addition, in wavelet theory, fewer wavelet coefficients can be used to approximate a function, and when the signal is denoised or compressed, the process does not cause apparent damage to the signal. As shown in Figure 4, although the two waveforms of pineapples cannot be well-distinguished in the time domain, their wavelet transformations show significant differences. The ripen one (in the right panel of Figure 4) shows a clear ridge at the location of 300 Hz.

Figure 4.

Although the waveforms of raw (left) and ripen (right) pineapples cannot be well-distinguished in time domain, their wavelet kernel transformations show significant differences. The ripen one (right) shows a clear ridge at the location of 300 Hz.

Acoustic emission Ml (AEML) method

ML achieves AI by learning from past data and experience and finding its operating rules. It involves training the machine to identify the operating mode through sample training rather than programming with specific rules. The AEML method is a resource tool for AI's prediction and diagnosis system based on acoustic signals. It uses acoustic big data information technology to establish an AI learning model. Specifically, this method uses information from acoustic data, such as audio streams, acoustic images, acoustic structure parameters, sound wave features, and other signals. After obtaining the data, using AI data learning technology, the input data can be automatically analyzed to identify the type of noise and estimate the best control, design, and recommended predictive classification results (Figure 5). Acoustic sensors (e.g. microelectromechanical microphones) are now widely used, so applying the AEML method to the production line of quality management can save the cost of automatic product diagnosis. Compared with the traditional acoustic methods, the AEML method linked with the AI model can better identify and analyze the acoustic signal data of each analysis sample and perform related quality inspection and classification tasks accordingly. In addition, the AI technology used by the AEML method is not complicated. For example, CNN and recurrent neural network models can be used to make AI models. After the model is trained, it can be applied to analyze the information of each analysis sample of the production line.

Figure 5.

Conceptual framework based on the learning method of AI through acoustic spectrum processing.

The AEML method can effectively identify classification tasks related to noise, so it is also suitable for health monitoring of detection aids. When the parts of the acoustic coupler produce abnormal vibrations, it usually reflects the quality defect of the internal components. At this time, the acoustic information characteristics and the AI classification model can determine the internal defect of the acoustic coupler.

The signal recognition procedure of this study is shown in Figure 5. Because it is difficult to have a large number of labeled waveforms from human experts, the use of supervised learning methods may not be possible. On the other hand, the states of raw and ripen are not two clear-cut states. Ripeness is a state with smooth and gradually changing status. Human experts use their subjective judgment to bisect the status; therefore, human bias may dictate the success of the recognition process.

This study first uses distance metrics in wavelets to measure the distance between two arbitrary signals. As shown in Figure 6, the coherence values of the cross-wavelet power spectrum are used to be the metric between two waveforms. We then perform singular value decomposition (SVD) of the correlation matrix to extract the principal component. With the principle component analysis, the classification algorithm can achieve superior performance.

Figure 6.

The cross-wavelet power spectrum with accepted p-values. The coherence values are used to be the metric between two waveform.

For the selection of acoustic coupler, the device is set up on a stable and level platform in a quiet environment. Then the pineapples are placed on the acoustic coupler for recording. After all the acoustic couplers corresponding to the experimental subject are recorded, all audio files are transferred to the computer and analyzed for each acoustic coupler. The above actions are repeated until the best experimental acoustic coupler is found.

This study uses contact-based, sensitive electronic sensors and acoustic multi-frequency technology to set the pineapple maturity determination field. The experimental body (pineapples) is placed on the acoustic coupler. A desktop computer connects the speaker and matches the resonant speaker to play a stable beat (160 bpm) and set a moderate volume. Subsequently, a mobile phone is used to record the sound. The computer plays the sound. Human ears distinguish the sound changes produced for testing different pineapples to find out the rules and obvious similarities and confirm them. With the echo of fruit beating as the basis, a technology for judging the maturity of pineapple is developed. The development of this technology allows pineapple exporters and farmers to understand the concept of scientific harvesting, judge the maturity of the fruit simply, accurately, and quickly, and classify the quality of pineapple more accurately to reduce the investment of the workforce. For example, according to the saleable period or sweetness, consumers can choose the fruits they like to promote the sales of pineapples.

The experiment collected 124 sound record files with 10 s for each file. Not every recording was successful. While searching for proper acoustic coupler materials, most recordings cannot exceed the quality of machine classification. Therefore 70% of data must discard due to the coupler problem. Therefore, we excluded those unlabeled records. All the sound records have been split into small sound frames of 30-msec. After a preprocessing step, we obtained 3475 sound frames. Among the frames, 2780 unlabeled frames have been sent to perform clustering, and 695 labeled frames are kept for the final testing (verifying the correctness of our algorithm).

After statistical analysis, if the red indicator (raw pineapple) and blue indicator (ripe pineapple) fall above and below maturity level 3, respectively, the judgment implies unreasonable. After screening, the judgment on raw pineapple or ripen pineapple falls within a reasonable range.

Material matching

The collected pineapples are used to determine fruit maturity. One goal of this study is to find the best acoustic coupler by sound discrimination. Therefore, the study employs different factors (e.g. material, angle, and contact area) for repeated experiments and listening to compare the pros and cons of each acoustic coupler. Upon considering all factors and related analysis, the best acoustic coupler is selected. After preliminary discussion and judgment, the following five acoustic couplers are selected for comparison (Figure 7).

Metal bookshelf  +  hot melt glue: Commercially available iron bookshelves are processed by hot melt glue through the surface holes to increase the contact area with the experimental body, and the speaker is fixed behind the iron bookshelf.

Metal bookshelf  +  pinning bean board: A plastic pinning bean board is placed on the surface of a commercially available iron bookshelf, and then the speaker is fixed on the back of the iron bookshelf.

Display board  +  pinning bean board: A plastic pinning bean board is placed on the surface of a commercially available plastic display board, and then the speaker is fixed on the back of the display board.

Folder  +  pinning bean board: A plastic pinning bean board is placed on the surface of a commercially available hard-shell folder. A cardboard box is set outside to fix the folder with a speaker fixed on the back.

Pinning bean board  +  screw: The speaker is placed on a platform, the pinning bean board is on it, and then the screws are tightened.

Figure 7.

Five acoustic coupler devices: (a) Metal bookshelf  +  hot melt glue, (b) Metal bookshelf  +  pinning bean board, (c) display board  +  pinning bean board, (d) folder  +  pinning bean board, (e) pinning bean board  +  screw.

Results and discussion

In this study, when evaluating the correctness of pineapple maturity, the Taguchi analysis method is used to calculate the signal-to-noise ratio (SNR), where the signal refers to the critical element, which is the average of the quality characteristics. The noise is the unnecessary part. This analysis is used to measure the variance; the closer the target is, the smaller the value. When the SNR value is maximized, the expected loss is the smallest

Considering the closeness between the acoustic coupler and the pineapple and the calculation result of the SNR, the larger the SNR value, the smaller the expected loss. By using the calculated SNR value, the SNR value of the screw pinning bean board (Figure 8) as the acoustic coupler is the largest (Figure 9). Therefore, this study determines that the screw pinning bean board is the optimal acoustic coupler, which also obtains the optimal judgment results in the experiment.

Figure 8.

Improved acoustic coupler: Screw-locking pinning bean board.

Figure 9.

Performance of STD and SNR for different acoustic coupler devices.

With the clustering results in the l eft panel of Figure 10, we can further apply them to the classification of adaBoosting, as shown in the right panel of Figure 10. The classification result achieves an accuracy rate of 98.56%, with type I and type II errors of (9/374 and 1/321) and a 0.93 F1 score.

Figure 10.

The clustering (left) and classification (right) trees.

Compared to the results of 91.76% ²² and 82%, ²³ our method obtain a significantly better result.

Conclusions

Non-destructive fruit detection for determining the remaining life of the fruit was carried out in this study. The device makes intelligent judgments by tapping a sound source to the skin of pineapples and analyzing the penetrated sounds by wavelet kernel decomposition and unsupervised machine learning. The best acoustic coupler was selected using the Taguchi experiment design to adapt the vibrator to the skin and pick high-quality penetrated sounds. We found that our ML method achieves 98.56% accuracy and 0.93 F1-score by using a specially designed thorn-board for assisting tapping sound to fruit skin.

In addition, related sensors can classify and identify agricultural products during production and marketing, collect information, and transmit the information to the computer through network integration. Farmers can then know the growth situation, pest degree, yield, and maturity of agricultural products timely and accurately. In using such information, the relevant units can adjust the production and marketing at an appropriate time point. The invention and application of the above technologies is an excellent opportunity for domestic agriculture to be upgraded again. Facing foreign competitors, only the application of agricultural IoT can make domestic agriculture stand out. In such a way, farmers no longer depend on the weather. With the help of professional technology, farmers’ income can be improved, and more people will like to engage in agricultural production.

In the future of the agricultural Internet of Things (IoT), using a computer is necessary to interpret and analyze the maturity of crops through the sensor designed in this research to adjust sales. The research and development of this innovative technology are based on the data collected by the IoT components. Scientific decisions are then analyzed and made, including the non-destructive detection and risk decision optimization of the time-varying sudden wave resistance analysis method.