Efficient and robust approaches for three-dimensional sound source recognition and localization using humanoid robots sensor arrays

Abstract

Efficient and robust sound source recognition and localization is one of the basic techniques for humanoid robots in terms of reaction to environments. Due to the fixed sensor arrays and limited computation resources in humanoid robots, there comes challenge for sound source recognition and localization. This article proposes a sound source recognition and localization framework to realize real-time and precise sound source recognition and localization system using humanoid robots’ sensor arrays. The type of the audio is recognized according to the cross-correlation function. And steered response power-phase transform function in discrete angle space is used to search the sound source direction. The sound source recognition and localization framework presents a new multi-robots collaboration system to get the precise three-dimensional sound source position and introduces a distance weighting revision way to optimize the localization performance. Additionally, the experiment results carried out on humanoid robot NAO demonstrate that the proposed approaches can recognize and localize the sound source efficiently and robustly.

Keywords

Auditory perception sound source recognition and localization cross-correlation multi-robots collaboration system humanoid robot

Introduction

Humanoid robots have been designed to interact with people and react to environments. As a representation of its intelligence, auditory perception is essential for its recognition of environmental changes. For example, humanoid robots in RoboCup competition need to recognize the whistle blown by referee as a signal to start the game. In this situation, the technology of sound source recognition and localization (SSRL) is employed to classify if the audio type is whistle and localize the accurate sound source position to avoid misidentification of whistle from other fields. Due to the fixed sensor arrays in humanoid robots and the fact that a sound event occurs at an arbitrary direction in three-dimensional (3D) space with noise and reverberation, there comes challenge for SSRL. In addition, a lot of techniques like visual recognition need to be carried out simultaneously with the given computation resources limited humanoid robots. As a result, the computationally efficient and robust real-time 3D SSRL method is increasingly required.

In the last few years, many SSRL algorithms have been applied to intelligent robots.^1
–3 Confronted with sound recognition, frequency domain analysis is frequently adopted.⁴ Detected principal frequency components are classified to predefined frequency domain space to determine whether it is the certain audio type. Sensor arrays in humanoid robots, however, are usually fixed and equipped with low sample rate microphone. Thus, false recognition often occurs. To solve the problem, a new sound source recognition method which is not limited to frequency discrimination has been adopted in this article.

Despite the sensor arrays differ between various applications, commonly used sound source localization (SSL) cues are inter-channel time difference (ICTD) and inter-channel level difference. In Rascon and Meza⁵ and Argentieri et al.,⁶ a survey on SSL in robotics is presented. The typical ways using ICTD such as generalized cross-correlation phase transformation (GCC-PHAT)^7
–9 and steered response power-phase transform (SRP-PHAT)^10,11 estimate the sound source by time difference of arrival (TDOA) feature. Some multiple signal classification (MUSIC)-based sound localization ways are presented in Takeda and Komatini,¹² Birnie et al.,¹³ and Hoshiba et al.¹⁴ The basic principle of MUSIC algorithm is to decompose the covariance matrix of the output data, obtaining signal subspace and noise subspace. And the incident direction of the signal is estimated by using the orthogonality of the two subspaces. However, these methods are often hard to meet strict performance requirements in humanoid robots, for example, limited computation resource, noisy environment, and the requirement of reliability and efficiency.

Some applications are implemented on robots equipped with fixed microphone array configurations in Valin et al.,¹⁵ Lee et al.,¹⁶ Grondin and Michaud,¹⁷ and Bustamante et al.¹⁸ In Valin et al.,¹⁵ the mobile robot can localize the sound source over a range of 3 m and with a precision of 3°. In Athanasopoulos et al.,¹⁹ an application on humanoid robot NAO is presented, which reaches a precision of 5° to locate the sound source at the same elevation as the microphone array’s horizontal plane. Furthermore, with more computation resources, a deep neural network for SSL is introduced in He et al.,²⁰ Zhang and Wang,²¹ Yalta et al.,²² and Sun et al.²³

The methods mentioned above are still unable to meet the requirements in RoboCup soccer robot competition. In RoboCup SPL League, all teams must use NAO humanoid robots manufactured by Soft-Bank Robotics. The match is held on a total area of length 10.4 m and width 7.4 m. The whistle may be blown from same field or neighboring field as robots. So the strict requirements for distance range and positioning accuracy are challenging for SSRL algorithm. In 2019, a directional whistle challenge was held by organizing committee to investigate the possibilities of localizing the point where the referee whistle is blown. The details can be found on https://spl.robocup.org/technical-challenges-2019/. This article will take RoboCup SPL League competition and whistle challenge environment as one of the testbench of the methods proposed.

In this article, we propose an SSRL framework addressing the SSRL in humanoid robots. A cross-correlation-based recognition way is given to classify the audio type and a multi-robots collaboration system is used to locate the sound source. The cross-correlation-based recognition way takes prerecorded reference signal as basic clue rather than the frequency domain feature to distinguish the audio type. The single robot sound source direction estimator (SSDE) estimates the source direction by searching maximum SRP-PHAT function in discrete angle space while 2D multi-robots SSL algorithm and 3D SSL determine the source position. The SSRL aims at resolving the SSRL problem under indoor and outdoor circumstances. Figure 1 shows the structure of SSRL.

Figure 1.

The SSRL structure. SSRL: sound source recognition and localization.

The contribution of this article is included as follows: (1) A robust and anti-noise sound recognition algorithm based on cross-correlation feature. (2) Improved SRP-PHAT function and simplified discrete angle space search with less computation cost. (3) Multi-robots collaboration localization system attaching distance weighting revision for more precise localization result. (4) Real-time application on humanoid robots NAO equipped with microphone arrays like human.

The rest of the article is structured as follows. In the second section, we discuss sound recognition algorithm in SSRL, and the cross-correlation algorithm with noise filter is explained. We introduce SSL algorithm including 2D SSL and 3D SSL in the third section. Enhanced SRP-PHAT function and distance weighting revision is also presented in the third section. Experiments and result analysis are shown in the fourth section. Finally, in the fifth section, the article is concluded with a discussion and possible future research directions.

Sound source recognition algorithm

Considering the existence of noise and reverberation mixed in sampled audio data, the misidentification may occur if we classify the audio type by only detecting the principal frequency components.

The main procedure of sound source recognition algorithm in this article is demonstrated in Algorithm 1. The audio data are acquired and sampled from microphone in robot’s head. After transferring the time-domain signal to frequency-domain signal using fast Fourier transform, a cross-correlation function between the signal and prerecorded reference signal is calculated to identify the certain audio type if its value is over the presetting threshold.

Algorithm 1.

Sound source recognition.

Preprocessing: Noise filter

The noise in audio data origins from two aspects: the external noise from noisy environment and the internal noise caused by the processor’s ventilation fan, gyroscopes, and so on. It is essential to reduce the noise using noise filter before the recognition.²⁴ A band-pass filter is carried out in our algorithm and the upper-lower limits are determined by the audio type. For instance, the frequency of whistle is 2.5–3.5 kHz in most cases, so we can set the limits of band-pass filter $f_{L} - f_{H}$ as 1.5–4.5 kHz to filter out the noise. The noise filter procedure is executed before calculating the cross-correlation function to suppress the noise interference effectively.

Cross-correlation function

The audio type identification cue in SSRL is cross-correlation function^25
–27 between the sampled audio data segments and prerecorded reference sound source.

Given two sound source signal $x_{1} (n)$ and $x_{2} (n)$ , while $x_{1} (n)$ is the audio segments to be analyzed and $x_{2} (n)$ is the reference audio segments. The sampling frequency is both f and the number of samples is N.

For two continuous signals, the cross-correlation function is defined as equation (1)

R (τ) = E [x_{1} (m) x_{2} (m + τ)]

When we process the audio data in humanoid robots, the discrete sampling is carried out firstly. The cross-correlation function of two discrete signals is defined as equation (2), and the length of $R (τ_{n})$ is $2 N - 1$

R (τ_{n}) = \sum_{n = 0}^{n = N - |τ_{n}| - 1} x_{1} (n) x_{2} (n + τ_{n})

However, the method below to calculate the cross-correlation is time-consuming in humanoid robots with limited computation resources. Hence, we need to do the calculation in frequency domain.

According to Wiener–Khinchin theorem, the auto-correlation function of a wide-sense-stationary random process has a spectral decomposition given by the power spectrum of that process

\begin{array}{l} P (ω) = \int_{- \infty}^{+ \infty} R (τ) e^{- j ω τ} d τ \\ R (τ) = \frac{1}{2 π} \int_{- \infty}^{+ \infty} P (ω) e^{j ω τ} d ω \end{array}

where $P (ω)$ is the cross power spectrum of x ₁ and x ₂

P (ω) = \int_{- \infty}^{+ \infty} \int_{- \infty}^{+ \infty} x_{1} (t) x_{2} (t + τ) d t \cdot e^{- j ω τ} d τ

It can be simplified by the exchange integral property and the shift property of Fourier transform

P (ω) = F_{1}^{*} (ω) F_{2} (ω)

Thus, the cross-correlation function can be presented as

R (τ) = \frac{1}{2 π} \int_{- \infty}^{+ \infty} F_{1}^{*} (ω) F_{2} (ω) e^{j ω τ} d ω

And its discrete representation is

R (τ_{n}) = \frac{1}{N} \sum_{w = 0}^{N - 1} F_{1}^{*} (w) F_{2} (w) e^{j \frac{2 π}{N} w}

For any audio data segment from microphones, we calculate the cross-correlation function between it and the reference sound. The signal is identified as type T if the ratio of maximum value of cross-correlation function $max (R (τ_{n}))$ to maximum value of reference sound auto-correlation function $max (R^{'} (τ_{n}))$ is larger than the presetting threshold of type T

\frac{max (R (τ_{n}))}{max (R^{'} (τ_{n}))} > {threshold}_{T}

The empirical value of threshold is usually set to be 0.4 in practical use. As a result, we can deduce the audio type accurately based on the cross-correlation feature. The cross-correlation-based way can be more robust compared with principal frequency components detection-based way.

SSL algorithm

In this section, the proposed SSL algorithm is described. It consists of a single robot SSDE and multi-robots collaboration SSL algorithm. In “Single robot SSDE” section, a single robot SSDE will be introduced. “Multi-robot 2D SSL using distance weighting revision” and “Three-dimensional sound source localization” sections give more detailed introduction about 2D SSL algorithm and 3D SSL algorithm.

Single robot SSDE

As is depicted in Figure 2, the humanoid robot NAO is equipped with four microphones. The array configuration is analogous to human-ears as they are distributed on left and right sides. An important cue for SSL is the TDOA,^2,28,29 but it is not enough to locate the sound source based on the TDOA model between two microphones because the cone of confusion will lead to a mirror-image position. The classical GCC-PHAT estimates the angle by TDOA under the assumption of far-field.³⁰ As Figure 3 shows, the angle of sound arriving at mic1 and mic2 is α, and the difference of arrival distance is $c Δ T$ , so we can infer the direction of sound source according to the geometric relationship based on TDOA and distance between microphones. In this section, we propose a single robot SSDE using all the four microphones to distinguish the source direction.

Figure 2.

The microphone configuration in humanoid robot NAO and the straight sound source propagation path, the actual path may be more complicated.

Figure 3.

GCC-PHAT angle estimation. GCC-PHAT: generalized cross-correlation phase transformation.

The theoretical basis of SSL is the TDOA model of each microphone pairs. When the sound source S_i is close to the microphone pairs, the sound wave received by microphone M_i can be considered as a polar wave. Under the polar wave propagation assumption, time of arrival of each microphone can be estimated as the ratio of sound-to-microphone distance to sound speed c in air

Δ T = \frac{‖S_{i} - M_{i}‖}{c}

The coordinates that describe the source direction are shown in Figure 4.

Figure 4.

The coordinates used in SSDE. 2D position and 3D position in the figure are hypothesis points as the height is assumed to be the height of a referee in a robot soccer game. SSDE: sound source direction estimator.

Let the height of source be h, SSDE describes the source direction using azimuth α and elevation β with relative to robot’s coordinates. The SRP-PHAT feature is unique for each direction in discrete angle space, so the direction can be deduced by a set of microphone signals.

Given $X_{M} (n)$ as the signal segments received by the $M th$ microphone and $q (x, y, z)$ as the assumption sound source position, the SRP-PHAT feature function³¹ is defined as equation (10)

\hat{P} (q) = \sum_{l = 1}^{M} \sum_{m = l + 1}^{M} {\hat{R}}_{l m} [τ_{l m} (q (x, y, z)]

where ${\hat{R}}_{l m} [τ_{l m} (q (x, y, z))]$ is the generalized cross-correlation function between the lth and $m th$ microphone signal as

{\hat{R}}_{l m} (τ_{n}) = \frac{1}{N} \sum_{w = 0}^{N - 1} \frac{F_{l}^{*} (w) F_{m} (w)}{|F_{l}^{*} (w) F_{m} (w)|} e^{j \frac{2 π}{N} τ_{n}}

where $F_{m} (w)$ is the fast Fourier transform of $X_{m} (n)$ . $τ_{l m} (q (x, y, z))$ is the TDOA between the lth and mth microphone signal. Let r_l and r_m represent the position of microphone, c is the sound propagation speed in air (usually 343 m/s), then $τ_{l m} (q (x, y, z))$ is given by

τ_{l m} (q (x, y, z)) = \frac{(| | q - r_{m} | | - | | q - r_{l} | |)}{c}

Therefore, the estimation of real sound source position can be presented as equation (13)

{\hat{q}}_{s} = arg max_{q \in Q} \hat{P} (q)

As is mentioned before, it is time-consuming if we search in the 3D space coordinates point by point. Then, we propose SSDE to deal with the problem by discretizing the space Q to discrete azimuth-elevation angle space $Q^{^{'}} (α, β)$ .

As the sound source height is assumed to be h, we can get $τ (q^{^{'}} (α, β))$ by using $q^{^{'}} (α, β)$ . Given r_o as the head chain origin position in world coordinates, the 3D coordinates of $q^{^{'}} (α, β)$ can be approximated as equation (14)

\begin{array}{l} q_{x}^{​^{'}} = r_{o} (x) + \frac{h \times sin α}{tan β} \\ q_{y}^{​^{'}} = r_{o} (y) + \frac{h \times cos α}{tan β} \\ q_{z}^{​^{'}} = r_{o} (z) + h \end{array}

Now we are able to write down the SSDE in discrete angle space as equation (15)

\begin{array}{l} {\hat{q}}_{s} (α_{s}, β_{s}) = arg max_{q^{'} \in Q^{'}} \hat{P} (q^{​^{'}} (α, β)) \\ \hat{P} (q^{​^{'}} (α, β)) = \sum_{l = 1}^{M} \sum_{m = l + 1}^{M} {\hat{R}}_{l m} [τ_{l m} (q^{​^{'}} (α, β)] \\ τ_{l m} (q^{​^{'}} (α, β)) = \frac{(| | q^{​^{'}} - r_{m} | | - | | q^{​^{'}} - r_{l} | |)}{c} \end{array}

It has to be noted that the height assumption may bring some estimation error but the searching method in discrete angle space reduces the computation load so we can realize the real-time processing in humanoid robot NAO. In “Multi-robot 2D SSL using distance weighting revision” and “Three-dimensional sound source localization” sections, we will revise the 3D position to improve the result precision. The SSDE algorithm is concluded as Algorithm 2.

Algorithm 2.

Sound source direction estimator.

Besides, the localization resolution in discrete angle space is limited by the humanoid robot’s microphone array and sampling frequency since we use TDOA as our localization cue.³² Suppose the maximum frequency as $f_{max}$ , and the distance between the lth and mth microphone is $d_{l m}$ . The maximum time delay which can be measured is $\frac{d_{l m}}{c}$ , so $\frac{d_{l m}}{c} \times f_{max}$ kinds of angle can be distinguished under the maximum frequency. Given M microphones, the possible maximum resolution can be

Δ θ = \frac{360^{\circ}}{\prod_{l = 1}^{M} \frac{d_{l m}}{c} \times f_{max}}, m = l + 1

The distance between microphone arrays and the sampling frequency are the main factors that limit the SSDE resolution. As a result, the discretization interval should be set larger than Δθ in angle space.

Multi-robot 2D SSL using distance weighting revision

SSDE provides the estimated sound source direction relative to single robot. SSRL introduces a multi-robot collaboration SSL algorithm using SSDE. As is shown in Figure 5, a 2D position can be estimated by crossing the azimuth angles and distance weighting revision. After combining with the elevation angle, the height can be revised and we can get a more precise 3D position. The 2D SSL algorithm based on distance weighting revision will be presented in this section.

Figure 5.

An approach for multi-robot 2D SSL using distance weighting revision. (The blue point is estimated position as the average of three cross points, the green point is the ground truth of sound source, and the red point is the revised estimated position.) SSL: sound source localization.

It needs to be stated that the robots rely on Wi-Fi communication. After each robot finish processing the sound signal, the angle information will be sent to the master robot who is responsible for executing 2D SSL and 3D SSL algorithm in real time.

To illustrate 2D SSL, let’s assume there are R humanoid robots NAO (R = 3 in this article) with initial pose $L_{r} = [x_{r}, y_{r}, θ_{r}]$ , $r = 1 ... R$ . For each robot, the relative sound source direction $q_{r} (α_{r}, β_{r})$ is calculated using SSDE, where $α_{r}$ is the azimuth and $β_{r}$ is the elevation.

Intersect R robots’ azimuth angle rays, and we can get $C_{R}^{2}$ intersections P_i , $i = 1 ... C_{R}^{2}$ . After taking average, an estimated 2D position is achieved as equation (17)

P_{uncorrected} = \frac{1}{C_{R}^{2}} \sum_{i} P_{i}

When the sound source distance is far greater than the microphone interval, we can analyze the sound propagation by assuming microphones are configured at the head chain origin point. For angle resolution sector region, the larger the radius, the longer the arc will be $(L = r \times α)$ . All points in the arc will be regarded as the same angle in current angle resolution. That is to say, in the same angle resolution zone, the closer the microphone is to the sound source, the smaller the arc length L and the smaller the area it represents. Correspondingly, the small shifting of sound source $Δ q$ will be reflected in small deviation of angle Δα. Thus, closer distance to microphone means higher credibility of the identified sound source direction.

Using this criterion, 2D SSL revises the estimated 2D position by distance weighting revision. For each robot, the distance between it and uncorrected sound source position is calculated as d_r , $r = 1, ..., R$ . Then, we select the closest one to revise the position. The corrected position can be acquired by rotating the uncorrected pose to the azimuth ray of the closest one

P_{2 D} = P_{corrected} = L_{r_{i}} + d_{​_{r_{i}}} \cdot [\begin{matrix} cos (α_{r_{i}} + θ_{r_{i}}) \\ sin (α_{r_{i}} + θ_{r_{i}}) \end{matrix}]

where r_i is the index of the robot with closest distance to sound source.

In 2D SSL, not only all robots’ sound direction information is combined into uncorrected position, but also the closest robot distance with corresponding angle is used to correct the position. The corrected 2D position combines the information of distance, and it is more reliable than the uncorrected one. 2D SSL in this section provides the 2D estimated position of sound source, and we will infer the 3D estimated position in the next section.

Three-dimensional sound source localization

This section aims at deducing the 3D estimated position of sound source. Once we get the 2D position from 2D SSL, we can combine it with the elevation and work out the height of sound source to revise the height assumed before. Figure 6 shows the 3D SSL coordinates.

Figure 6.

3D SSL coordinates. (The dark green point is the ground truth 3D sound source location, and the dark red point is the estimated 3D position. The green point, red point, and blue point have the same meaning as described in Figure 4.) SSL: sound source localization.

The main procedure is described as follows:

Given two vectors with fixed starting points $\vec{n}$ and $\vec{β}$ , where $\vec{n}$ is the normal vector of horizontal plan with staring point as $P_{corrected}$ and direction as any normal vector. And $\vec{β}$ is the elevation ray vector with starting point as $L_{r_{i}}$ and direction as elevation direction $\vec{β_{r_{i}}}$ . Intersect $\vec{n}$ and $\vec{β}$ , and we can get $P_{cross}$ , which can be assumed as the 3D estimated sound source position $P_{3 D}$ .

The whole 3D SSL algorithm is concluded as shown in Figure 7. SSDE works out azimuth and elevation using acquired microphone signal firstly. Combining angle information and robots’ position, 2D position is deduced by 2D SSL. Next, the elevation angle is used to revise the height and get 3D position by 3D SSL.

Figure 7.

3D SSL algorithm structure. SSL: sound source localization.

The main procedure is described in Algorithm 3.

Algorithm 3.

Sound source localization.

Experiments and analysis

We perform two experiments and one application to evaluate the proposed SSRL framework. In the first experiment, we compare the proposed sound recognition algorithm with another classical algorithm based on short-time Fourier transform (STFT).^33,34 We perform a numerical analysis to show the accuracy and anti-interference ability of our system. Then, we test our framework in the indoor and outdoor environment to evaluate the performance of SSDE. We compare SSDE with GCC-PHAT-based algorithm and classical SRP-PHAT-based algorithm to illustrate the efficiency and robustness of our framework. Additionally, we apply the proposed system for one application in humanoid robots NAO and test it on RoboCup SPL Technical Challenge.

Sound source recognition test

We evaluate our sound recognition algorithm on humanoid robot NAO(V5) which is equipped with four microphones on its head. The configuration parameter of microphone arrays is presented in Table 1.

Table 1.

NAO(V5) robot body and head with microphone arrays configuration parameters.^a

^a The position is relative to end transform of head chain.

We put the robot in indoor environment and compare our algorithm with the STFT-based way. The whistle, speech, and clap sound are collected to test the whistle recognition accuracy and anti-interference ability. To test the anti-interference ability of our algorithm, we collect the same kind of whistle with noise. The noise is simulated by sound player which is located at 0.5 m away from the robot and test audio is set to be played at 1 m away from the robot. The level and frequency of audio is presented in Table 2. The frequency of clap sound and speech is overlapped with whistle, that is why STFT does not work in some cases.

Table 2.

The level and frequency of audio.

Audio type	Level (dB)	Frequency (kHz)
Whistle	120	2.5–3.5
Clap sound	60	2–3
Speech	40–60	0.3–3.4
Noise	60	0.02–20

The audio sampling frequency is set to be 48 kHz in this experiment. The sampled microphone signal including whistle, speech, and clap is normalized as shown in Figure 8. The signal segments are different in frequency as we can clearly see from the figure and the data are consistent with what we provided in Table 2. The frequency of whistle and clap sound is very close to each other.

Figure 8.

Three kinds of microphone signal segments wave including whistle, speech, and clap.

The STFT-based method detects the principal frequency components using STFT. For instance, the STFT frequency spectrum of a section of whistle is shown in Figure 9. The range of whistle identification is 2.5–3.5 kHz, so the signal can be convinced to be whistle if the principal frequency is within the range.

Figure 9.

The STFT frequency spectrum of a whistle signal. STFT: short-time Fourier transform.

The SSRL proposed in this article uses the cross-correlation function between current audio segment and reference whistle as the standard to identify the whistle. The ratio of cross-correlation function between two signals to reference signal auto-correlation function with a whistle blowing event is shown in Figure 10.

Figure 10.

$max (R (τ_{n})) / max (R^{'} (τ_{n}))$ waveform based on SSRL. (It contains two whistle blowing events during the test period, and the threshold is set to be 0.4.) SSRL: sound source recognition and localization.

Two hundred samples are collected to compare the two algorithms. The identification results are shown in Table 3 and Figure 11.

Table 3.

Identification results.

Method	TP	FN	TN	FP
STFT	52	11	89	48
SSRL (ours)	75	2	98	25

STFT: short-time Fourier transform; SSRL: sound source recognition and localization.

Figure 11.

The identification results comparison of STFT and SSRL on 200 audio data samples, and the number of TP, FN, TN, and FP is counted. (200 samples including 50 whistle samples, 50 noised whistle samples, 50 speech samples, and 50 clap samples.) STFT: short-time Fourier transform; SSRL: sound source recognition and localization.

The criterion used to judge the accuracy in this article is

r % = \frac{TP}{TP+FP}

And the criterion of anti-interference ability is

e % = \frac{TN}{FN+TN}

As we can see from the result, the two algorithms have different performance on the test set. The STFT-based recognition method gets an accuracy of 0.52 and anti-interference ability of 0.89 while SSRL proposed in this article can reach an accuracy of 0.75 and anti-interference ability of 0.98. Obviously, SSRL outperforms STFT in accuracy and anti-interference ability.

SSDE test

In this section, we will evaluate the efficiency and robustness of three source direction estimator including SSDE, GCC-PHAT, and SRP-PHAT in indoor and outdoor environment. The angle estimation accuracy and time cost are measured to distinguish which algorithm is better. To test the robustness of the algorithm, the corresponding audio is collected in the indoor and outdoor environment. The indoor environment is set in a totally enclosed room of length 13.0 m and width 8.0 m, while the outdoor environment is set in a very large stadium. There is reverberation caused by wall reflection in the indoor environment, while there are noisy voices in the outdoor environment.

Both SRP-PHAT and SSDE use the same kind of SRP feature. Classical SRP-PHAT algorithm estimates the source direction by searching in the 3D space using 3D coordinates while SSDE searches in angle space. The SRP-PHAT searching step is set to be 0.05 m in this experiment, and the searching domain is 9 × 6 m² with height of 1.3–1.9 m.

In indoor environment with reverberation, sound source in different azimuth-elevation position is acquired to test the angle estimation effect of three algorithms. Only some selected angle ranges are tested and results are shown in Tables 4 and 5. The error distribution and time cost are shown in Figure 12 and Table 6.

Table 4.

Azimuth angle test result in the indoor environment.

Ground truth (°)	GCC- PHAT (°)	SRP-PHAT (°)	SSDE (°)
0	0	−7.911202	−0.000032
30	2.245494	36.6884865	25.012652
45	12.829337	45.0643245	45.022792
60	19.130520	59.219863	59.8221575
90	21.521501	88.255417	90.045613
110	140.226336	70.382068	110.055755
120	146.77559	117.107849	120.060832
160	143.951159	161.842782	155.078572
180	180.0000	174.342641	184.911227
−135	−180.0000	−134.977121	−135.068488
−90	−30.432812	−86.142745	−90.045671
−80	−20.374262	−74.991428	−75.038069
−50	−43.275510	−50.3582175	−50.025391
−20	−23.397169	−21.204592	−25.012715

GCC-PHAT: generalized cross-correlation phase transformation; SRP-PHAT: steered response power-phase transform; SSDE: sound source direction estimator..

Table 5.

Elevation angle test result in the indoor environment.

Ground truth (°)	SRP-PHAT (°)	SSDE (°)
14	13.565676	15.007578
17	17.933968	17.300603
21	20.612855	20.010115
30	28.538149	30.015187
50	49.919726	50.025328
58	56.6964	60.030399
14	13.565676	15.007578

SRP-PHAT: steered response power-phase transform; SSDE: sound source direction estimator.

Figure 12.

Angle estimation error distribution of three algorithms in indoor environments with reverberation. (The figure only shows the test angle data. In the error distribution map, the red line represents SSDE, green represents GCC-PHAT, and blue represents SRP-PHAT. And the scale of error coordinates is degree. What’s more, GCC-PHAT cannot estimate the elevation because of the arrival assumption.)

Table 6.

The average time cost of three algorithms when processing the same audio data segments indoors.

Method	Average time cost (ms)
SSDE (ours)	36.9580
GCC-PHAT	32.5853
SRP-PHAT	59.3405

SSDE: sound source direction estimator; GCC-PHAT: generalized cross-correlation phase transformation; SRP-PHAT: steered response power-phase transform.

Similarly, we set up the same experiments in outdoor environments and test the three algorithms. The results are shown in Tables 7 to 9 and Figure 13.

Table 7.

Azimuth angle test result in an outdoor environment.

Ground truth (°)	GCC-PHAT (°)	SRP-PHAT (°)	SSDE (°)
0	−1.429738	−10.3108	2.292994
45	90	41.2240	50.0253
90	180	78.7283	85.043
135	136.724485	121.6938	135.068
180	155.896228	173.5138	165.083
−135	−173.0507	−134.977	−135.06848
−90	−17.809810	−89.3009	−90.0456
−45	−9.138095	−39.19307	−40.0203

GCC-PHAT: generalized cross-correlation phase transformation; SRP-PHAT: steered response power-phase transform; SSDE: sound source direction estimator.

Table 8.

Elevation angle test result in an outdoor environment.

Ground truth (°)	SRP-PHAT (°)	SSDE (°)
21	21.011967	22.303140
30	28.538149	30.015187
50	48.103226	50.025328

SRP-PHAT: steered response power-phase transform; SSDE: sound source direction estimator.

Table 9.

The average time cost of three algorithms when processing the same audio data segments outdoors.

Method	Average time cost (ms)
SSDE (ours)	38.7301
GCC-PHAT	30.2985
SRP-PHAT	53.2898

SSDE: sound source direction estimator; GCC-PHAT: generalized cross-correlation phase transformation; SRP-PHAT: steered response power-phase transform.

Figure 13.

Angle estimation error distribution of three algorithms in outdoor environments. (In error distribution map, the representation is as Figure 12.)

The different error rates for different angles may be caused by the random noise, but it is clear to see from the results that angle estimation error of SSDE is less than GCC-PHAT and average time cost of SSDE is less than SRP-PHAT. In summary, GCC-PHAT-based way is faster than other two methods but performs not well in angle estimation in complicated environment. The SRP-PHAT-based way performs well in angle estimation but it has low efficiency and costs too much time in searching. For our proposed SSDE, it shows higher accuracy and efficiency and it is robust to both indoor and outdoor environment.

Application

We apply SSRL for 3D SSL on humanoid robots NAO. To test the recognition and localization simultaneously, we design a whistle recognition and localization experiments in robot soccer competition field.

The experiment environment configuration is shown in Figure 14, robots are put in the predefined initial position and stay still. Robot 1 is placed at (2.3 m, 1.7 m) facing at −62.1°, robot 2 is placed at (−1.5 m, −1.3 m) facing at 40.9°, and robot 3 is placed at (−3.3 m, 0.6 m) facing at 0°. Once the referee blows the whistle in any place, the robot will detect the signal and recognize the whistle. Meanwhile, the audio data segments from all microphones will be used to locate the whistle. The robots will contact with each other via Wi-Fi and share their SSDE result. Finally, the robot will calculate the 3D sound source position using 3D SSL, and the console will present the result.

Figure 14.

SSRL test environment. The robot is standing on the robot soccer standard platform competition field and the whistle will be blown by the referee. SSRL: sound source recognition and localization.

Three indexes are used to measure the localization result in this article: The first one is the 3D absolute position error, the second one is the relative distance error to the robot with closest distance to whistle, and the third one is the relative azimuth rotation error.

Suppose the actual position of whistle is $q_{GT}$ , and the SSRL estimated position is q_S , then three indexes will be defined as equation (21)

\begin{array}{l} e_{1} = ‖q_{GT} - q_{S}‖ \\ e_{2} = |‖q_{GT} - L_{r_{min}}‖ - ‖q_{S} - L_{r_{min}}‖| \\ e_{3} = |α_{GT} - α_{S}| \end{array}

where $L_{r_{min}}$ is the position of robot with closest distance to whistle position, $α_{G T}$ is the angle to whistle position in closest robot’s coordinates, and $α_{S}$ is the angle to estimated position in closest robot’s coordinates. All errors are normalized to their absolute value.

We take robot soccer competition field as the test field and place the robot at set place in Figure 15. The whistle is blown by a standing adult at every 0.5 × 0.5 m² area. The yielding error distributed throughout the whole field is shown in Figures 15 to 17. In field with size of 9 × 6 m², the error is very small in most areas but there still exists some points with large error. The possible reason may be the misidentification caused by too close or too far distance and the accumulated angle estimation error of robots.

Figure 15.

Error distribution map of e ₁

Figure 16.

Error distribution map of e ₂

Figure 17.

Error distribution map of e ₃

The average error of three indexes is shown in Table 10.

Table 10.

Test result average error.

	Average error
e ₁	0.8636 m
e ₂	0.6760 m
e ₃	4.8325°

In addition, we applied the SSRL in humanoid robots NAO and took part in RoboCup2019 Standard Platform Directional Whistle Technical Challenge. Real competition field is set at a big exhibition hall and the whole area (18 × 30 m²) contains four SPL standard filed. There are walking spectators and noises in the hall so it brings a lot of challenges to our system. Some of the localization data are shown in Table 11.

Table 11.

Data excerpts of RoboCup2019 SPL Technical Challenge.

	Ground truth (m)	Estimated position (m)	e ₁ (m)	e ₂ (m)	e ₃ (°)	Score (points)
1	(0, 3.35)	(0.532, 3.973)	0.819	0.049	16.476	1.54128
2	(4.85, 1.1)	(4.801, −0.187)	1.288	0.513	23.806	1.66435
3	(7.3, 5.9)	(10.506, 5.181)	3.286	2.384	17.052	1.51826
4	(2.25, −2.5)	(0.158, −1.874)	2.184	2.183	1.352	2.000

As we can see from the result, using humanoid robots microphone arrays, the SSRL framework proposed in this article can locate the sound source in both indoor and outdoor environments. The 3D distance absolute error is approximately 0.8636 m, the relative distance error is 0.6760 m, and the relative azimuth angle error is 4.8325°. In directional whistle technical challenge, the minimum distance error can reach to 0.049 m and the minimum angle error can achieve 1.352°.

The SSRL algorithm proposed in this article is proved to be able to locate the sound source in indoor and outdoor environments on humanoid robots NAO with limited computation resources and critical real-time requirement after application testing. Compared with other methods, the SSRL achieves higher efficiency and robustness.

Conclusions

This article presents an SSRL framework approach implemented for 3D SSRL on humanoid robots. The humanoid robots microphone arrays are fixed and configured like human-ears, and limited computation resources make humanoid robots SSRL a challenging problem. This article addresses the problem by following ways.

The sound recognition based on cross-correlation feature classifies the sound type using sampled microphone signals. And the sound source direction is determined by a single robot SSDE which employs a new discrete angle space SRP-PHAT function. Aiming at getting the 3D position of the sound source, a 2D multi-robots SSL is firstly presented with distance weighting revision. The following 3D multi-robots SSL calculation works out the certain 3D sound source position.

Compared with other methods, SSRL is proved to be more efficient and robust after experiments test. However, extensive research is still necessary to further improve the system accuracy and robustness. Deep neural networks-based methods in the literature^20

–23 inspire us to extract deep feature of audio data and use it as a new cue to locate the sound source. We believe that more progress on SSRL will be made to transcend the field and improve robot audition in general.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported in part by the National Natural Science Foundation of China (grant nos 61673300, U1713211 and 61733013).

ORCID iDs

Hao Chen

Chengju Liu

References

Magassouba

Bertin

Chaumette

. Aural servo: sensor-based control from robot audition. IEEE Trans Robot 2018; 34(3): 572–585.

Basiri

Schill

Lima

, et al. Localization of emergency acoustic sources by micro aerial vehicles. J Field Robot 2018; 35(2): 187–201.

Vidal-Calleja

Miro

. Asynchronous microphone arrays calibration and sound source tracking. Auton Robots 2020; 44(2): 183–204.

Cowling

Sitte

. Comparison of techniques for environmental sound recognition. Pattern Recog Lett 2003; 24(15): 2895–2907.

Rascon

Meza

. Localization of sound sources in robotics: a review. Robot Auton Syst 2017; 96: 184–210.

Argentieri

Danes

Souères

. A survey on sound source localization in robotics: from binaural to array processing methods. Comput Speech Lang 2015; 34(1): 87–112.

Kang

Graczyk

Skoglund

. On pre-filtering strategies for the GCC-PHAT algorithm. In: 2016 IEEE international workshop on acoustic signal enhancement (IWAENC), Xi’an, China, 13–16 September 2016, pp. 1–5. IEEE.

Velasco

Taghizadeh

Asaei

, et al. Novel GCC-PHAT model in diffuse sound field for microphone array pairwise distance based calibration. In: 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), Brisbane, QLD, Australia, 19–24 April 2015, pp. 2669–2673. IEEE.

Van Den Broeck

Bertrand

Karsmakers

, et al. Time-domain generalized cross correlation phase transform sound source localization for small microphone arrays. In: 2012 5th European DSP education and research conference (EDERC), Amsterdam, Netherlands, 13–14 September 2012, pp. 76–80. IEEE.

10.

Silverman

. A real-time SRP-PHAT source location implementation using stochastic region contraction (SRC) on a large-aperture microphone array. In: 2007 IEEE international conference on acoustics, speech and signal processing-ICASSP’07, Hawaii, USA, 15 April 2007, vol. 1, pp. I–121. IEEE.

11.

Cobos

Marti

Lopez

. A modified SRP-PHAT functional for robust real-time sound source localization with scalable spatial sampling. IEEE Signal Process Lett 2010; 18(1): 71–74.

12.

Takeda

Komatani

. Performance comparison of MUSIC-based sound localization methods on small humanoid under low SNR conditions. In: 2015 IEEE-RAS 15th international conference on humanoid robots (humanoids), Seoul, Korea, 3 November 2015, pp. 859–865. IEEE.

13.

Birnie

Abhayapala

Samarasinghe

. Reflection assisted sound source localization through a harmonic domain MUSIC framework. IEEE/ACM Trans Audio Speech Lang Process 2019; 28: 279–293.

14.

Hoshiba

Nakadai

Kumon

, et al. Assessment of MUSIC-based noise-robust sound source localization with active frequency range filtering. J Robot Mechatron 2018; 30(3): 426–435.

15.

Valin

Michaud

Rouat

, et al. Robust sound source localization using a microphone array on a mobile robot. In: Proceedings 2003 IEEE/RSJ international conference on intelligent robots and systems (IROS 2003)(Cat. No. 03CH37453), Las Vegas, NV, 27–31 October 2003, vol. 2, pp. 1228–1233. IEEE.

16.

Lee

Park

. Three-dimensional sound source localization using inter-channel time difference trajectory. Int J Adv Robot Sys 2015; 12(12): 171.

17.

Grondin

Michaud

. Lightweight and optimized sound source localization and tracking methods for open and closed microphone array configurations. Robot Auton Syst 2019; 113: 63–80.

18.

Bustamante

Danès

Forgue

, et al. An information based feedback control for audio-motor binaural localization. Auton Robot 2018; 42(2): 477–490.

19.

Athanasopoulos

Brouckxon

Verhelst

. Sound source localization for real-world humanoid robots. In: Proceedings of the 11th international conference on signal processing (SIP 2012), Beijing, China, 2 April 2012, vol. 12, pp. 131–136.

20.

Motlicek

Odobez

. Deep neural networks for multiple speaker detection and localization. In: 2018 IEEE international conference on robotics and automation (ICRA), Brisbane, Australia, 21 May 2018, pp. 74–79. IEEE.

21.

Zhang

Wang

. Deep learning based binaural speech separation in reverberant environments. IEEE/ACM Trans Audio Speech Lang Process 2017; 25(5): 1075–1084.

22.

Yalta

Nakadai

Ogata

. Sound source localization using deep learning models. J Robot Mechatron 2017; 29(1): 37–48.

23.

Sun

Chen

Yuen

, et al. Indoor sound source localization with probabilistic neural network. IEEE Trans Ind Electron 2017; 65(8): 6403–6413.

24.

Brindza

Thomas

Lee

, et al. Active sound localization in a symmetric environment. Int J Adv Rob Sys 2013; 10(7): 301.

25.

Adavanne

Pertilä

Virtanen

. Sound event detection using spatial features and convolutional recurrent neural network. In: 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), New Orleans, LA, 5–9 March 2017, pp. 771–775. IEEE.

26.

Lei

Yang

. Passive localization in the deep ocean based on cross-correlation function matching. J Acoust Soc Am 2016; 139(6): EL196–EL201.

27.

Freeman

Buckingham

Freeman

, et al. Cross-correlation, triangulation, and curved-wavefront focusing of coral reef sound using a bi-linear hydrophone array. J Acoust Soc Am 2015; 137(1): 30–41.

28.

Xie

Tan

. Iterative constrained weighted least squares source localization using TDOA and FDOA measurements. IEEE Trans Signal Process 2017; 65(15): 3990–4003.

29.

Shen

Wang

, et al. Real-time sound source localization for a mobile robot based on the guided spectral-temporal position method. Int J Adv Robot Syst 2012; 9(3): 78.

30.

Imran

Hussain

Qazi

, et al. A methodology for sound source localization and tracking: development of 3D microphone array for near-field and far-field applications. In: 2016 13th International Bhurban conference on applied sciences and technology (IBCAST), Islamabad, Pakistan, 12–16 January 2016, pp. 586–591. IEEE.

31.

Velasco

Martín-Arguedas

Macias-Guarasa

, et al. Proposal and validation of an analytical generative model of SRP-PHAT power maps in reverberant scenarios. Signal Proc 2016; 119: 209–228.

32.

Hao

Zhao

, et al. A sensor selection method for TDOA and AOA localization in the presence of sensor errors. In: 2017 IEEE/CIC international conference on communications in China (ICCC), Qingdao Shi, Shandong Sheng, China, 22–24 October 2017, pp. 1–6. IEEE.

33.

Parchami

Zhu

Champagne

, et al. Recent developments in speech enhancement in the short-time Fourier transform domain. IEEE Circ Syst Mag 2016; 16(3): 45–77.

34.

Nakagawa

Wada

. Diagnosis system for abnormal respiratory sound using divided pulmonary sound waveform. J Signal Proc 2016; 20(4): 221–224.