Sage Journals: Discover world-class research

Abstract

We tested whether sensitivity to acoustic spectrotemporal modulations can be observed from reaction times for normal-hearing and impaired-hearing conditions. In a manual reaction-time task, normal-hearing listeners had to detect the onset of a ripple (with density between 0–8 cycles/octave and a fixed modulation depth of 50%), that moved up or down the log-frequency axis at constant velocity (between 0–64 Hz), in an otherwise-unmodulated broadband white-noise. Spectral and temporal modulations elicited band-pass filtered sensitivity characteristics, with fastest detection rates around 1 cycle/oct and 32 Hz for normal-hearing conditions. These results closely resemble data from other studies that typically used the modulation-depth threshold as a sensitivity criterion. To simulate hearing-impairment, stimuli were processed with a 6-channel cochlear-implant vocoder, and a hearing-aid simulation that introduced separate spectral smearing and low-pass filtering. Reaction times were always much slower compared to normal hearing, especially for the highest spectral densities. Binaural performance was predicted well by the benchmark race model of binaural independence, which models statistical facilitation of independent monaural channels. For the impaired-hearing simulations this implied a “best-of-both-worlds” principle in which the listeners relied on the hearing-aid ear to detect spectral modulations, and on the cochlear-implant ear for temporal-modulation detection. Although singular-value decomposition indicated that the joint spectrotemporal sensitivity matrix could be largely reconstructed from independent temporal and spectral sensitivity functions, in line with time-spectrum separability, a substantial inseparable spectral-temporal interaction was present in all hearing conditions. These results suggest that the reaction-time task yields a valid and effective objective measure of acoustic spectrotemporal-modulation sensitivity.

Keywords

psychoacoustics auditory threshold hearing modulation transfer function reaction time

Introduction

Human speech and other complex sounds in the natural environment are typically dynamic signals that rapidly change in amplitude over both time and frequency. Fluctuations in the temporal domain provide information about the rhythm of speech, such as syllable and word boundaries, whereas variations in the spectral domain are essential for formant and voice pitch perception (Liberman, 1996). Sensitivity to these joint spectral and temporal modulations is deemed crucial for the identification of complex sound features (McDermott & Simoncelli, 2011) and for speech comprehension (Elliott & Theunissen, 2009; Shannon et al., 1995).

Spectrotemporal dynamic ripples have been introduced in psychoacoustics to investigate the spectrotemporal modulation sensitivity of auditory perception. Ripples are broadband noise stimuli that are modulated sinusoidally in amplitude over time and/or frequency (Bernstein & Green, 1998; Supin et al., 1994). Ripples are ideal to assess hearing performance as they represent features of, but are not recognizable as, naturalistic sounds. Sensitivity of the healthy human auditory system has been studied thoroughly with ripples and generally shows a band- or low-pass response to spectral and temporal modulations, reflecting the limits of auditory sensitivity at higher modulation rates (Chi et al., 1999; Viemeister, 1998; Zheng et al., 2017). Speech understanding is thought to relate mostly to joint spectrotemporal sensitivity. Chi et al. (1999) reported that in normal-hearing listeners the modulation transfer function of combined spectrotemporal ripples is highly separable, as it can be well approximated by the product of a single temporal and spectral filter. Separability implies that the joint spectrotemporal sensitivity can be directly obtained from pure temporal and spectral sensitivity measurements.

In the present study we used manual reaction times to construct the spectrotemporal modulation transfer function (stMTF), rather than the conventionally used modulation detection or discrimination thresholds. Research in monkeys shows that reaction times systematically depend on acoustic modulation rates (Massoudi et al., 2014). Several models have been proposed to explain the underlying process of response latency in reaction-time tasks (Ratcliff & Van Dongen, 2011). It is commonly assumed that a decision signal rises with accumulating evidence of the stimulus, until a certain threshold is reached that triggers the response. As such, reaction times are directly related to the difficulty of a task and could thus provide more detailed information on the audibility of spectrotemporal ripples.

Furthermore, reaction times allow for testing the presence or absence of binaural integration based on monaural responses, by comparing binaural reaction times against the prediction of a so-called “race model”. In such a model, the signals from either ear compete independently to reach the detection threshold, so that the response latency is determined by the winner of an independent parallel race between the two ears (Raab, 1962). Due to statistical facilitation, this race to threshold leads to faster reaction times for binaural stimulation than for monaural stimuli, as the distribution of minimum monaural reaction times yields faster responses than those produced by either ear (Gielen et al., 1983; Hershenson, 1962). However, when this so-called redundant stimulus effect differs from the race-model prediction, it could imply true binaural integration in an underlying neural interaction process (Gielen et al., 1983; Miller, 1982; Schröter et al., 2007).

We tested whether reaction times are an objective measure having convergent validity of auditory sensitivity to moving ripples with various spectrotemporal modulations for normal-hearing listeners. We assessed the extent of separability of joint spectrotemporal sensitivity and investigated how binaural listening affected modulation sensitivity compared to monaural listening conditions by comparing the data with the race-model prediction. As a validation of our reaction-time paradigm, we also collected data under more challenging impaired-hearing simulations that are known to affect temporal and spectral sensitivity (Bacon & Viemeister, 1985; Golub et al., 2012; Henry et al., 2005; Moore & Glasberg, 2001) and binaural integration (Ausili et al., 2019; Sharma et al., 2019, 2021; Veugen et al., 2016a, 2016b).

Methods

Listeners

Six listeners participated in this study (3 male, ages 20–25 years), none of whom reported a history of auditory deficits. All listeners had normal hearing (< 20 dB HL) in both ears from 125 to 8000 Hz. Except for two of the authors, listeners were naïve to the purpose of the experiments. We included the data from the two authors as they were highly similar to the data from the naïve listeners (see Supplemental Materials) and excluding those data did not appreciably affect the results and conclusions. The study was approved by the Local Ethics Committee of the Radboud University Nijmegen (protocol number 40327.091.12).

Apparatus

Listeners were seated in an acoustically shielded sound chamber. Stimuli were presented through TDH 39 headphones (Telephonics Corporation, Farmingdale, NY, USA). For sound processing and data acquisition we used Tucker Davis Technologies System 3 (Alachua, FL, USA). Stored sounds were sent via the PC to a real-time processor (RP2.1) at a sampling rate of 48,828.125 Hz, and passed through a programmable attenuator (PA5). Stimuli were set at a comfortable, well-audible loudness of 65 dB(A) (calibrated using a KEMAR head calibration set, connected to a Brüel & Kjaer measuring amplifier type 2610 [Nærum, Denmark]).

Stimuli

Dynamic ripples were created in MATLAB [version R2012a; Mathworks Inc., Natick, MA, USA] as described by Depireux et al. (2001). The carrier of these stimuli consisted of a broadband spectrum of multiple harmonic tones, each described by:

c_{i} (t) = \sin (2 π f_{i} t + φ_{i})

(1)

where t is time (s), f_i frequency (Hz) of the i-th harmonic, and

φ_{i}

is its phase (rad). In this experiment, the broadband carrier consisted of 128 harmonic tones, equally spaced (20 tones/octave) over 6.4 octaves (250 Hz–20.4 kHz). All components had random phase except for the first (

φ_{0}

= 0). The i-th frequency was determined by

f_{i} = f_{0} 2^{i / 20}

with

f_{0}

= 250 Hz the lowest frequency, and i = 0–127. All harmonic tones had the same amplitude, effectively yielding the same spectrum as white noise.

The spectrotemporal envelope determined the ripple fluctuations in amplitude over time and/or frequency:

s_{i} (t, x_{i}) = 1 + Δ M \cdot \sin (2 π ω t + 2 π Ω x_{i})

(2)

with t is time (s),

x_{i}

is the position on the frequency axis (in octaves above the lowest frequency),

Δ M

is the modulation depth, ω is the ripple velocity (Hz) and

Ω

is ripple density (cycles/octave). Unpublished data on freefield, normal-hearing ripple detection from our lab suggested that the actual value of the modulation depth is not very crucial. Therefore, we set the modulation depth rather arbitrarily to 0.5 for all components. Testing only one modulation depth reduced the number of potential parameter combinations and trials. This decreased the duration of an experimental session, which was already substantial.

Together the carrier and the modulator formed the dynamic ripple in our experiments as follows:

R (t) = {\begin{matrix} t < t_{o n s e t}, \sum_{i} c_{i} (t) \\ t > t_{o n s e t}, \sum_{i} s_{i} (t - t_{o n s e t}, x_{i}) \cdot c_{i} (t) \end{matrix}

(3)

The modulated sounds were thus preceded by a non-modulated harmonic complex (

c_{i} (t)

) with a randomized duration (

t_{o n s e t}

) between 700 and 1200 ms with a step size of 100 ms. Moving ripples were presented for a duration of maximally 3 s, with velocities of 0 Hz and ± [0.5; 1; 2; 4; 8; 16; 32; 64] Hz and densities [0; 0.25; 0.5; 0.75; 1; 2; 4; 8] cycles/octave, yielding a total of (17 velocities × 8 densities = ) 136 different stimuli.

Cochlear Implant Simulation

Cochlear-implant vocoder simulations were created using a previously described method by Litvak et al. (Litvak et al., 2007) that models the Advanced Bionics Harmony cochlear-implant processor. Briefly, the vocoder algorithm works as follows. After resampling the input signal to 17.4 kHz, the vocoder applied a high-pass pre-emphasis filter (cut-off at 1.5 kHz). Then, the signal was band-pass filtered by a short-time Fourier transform with 256 bins and 75% temporal overlap (192 bins). Bins were grouped into 6 non-overlapping logarithmically spaced channels (Figure 1B; at center frequencies: 452, 715, 1132, 1792, 2836 and 4488 Hz). Random-phase noise bands with similar center frequencies were modulated with amplitudes equal to the square root of the total energy in the channel. The channels were summed, and inverse short-time Fourier transformed to reproduce a temporal waveform for presentation to the listeners.

Figure 1.

Moving-ripple spectrograms. Ripple with velocity 4 Hz and density 0.5 cycles/octave) for the normal-hearing condition (NH), cochlear implant simulation (CI) and hearing-aid simulation (HA). Examples of the temporal and spectral modulations of this ripple presented separately are shown at the top and right of each panel. The signals on the top row represent the temporal waveforms for a purely amplitude-modulated sound (4 Hz, 0 cycles/octave). The signal on the right of each panel visualizes a pure stationary spectral ripple modulation (0 Hz, 0.5 cycles/octave). For clarity, the sound is shown after t_onset (at t = 0).

We used 6 vocoder channels to simulate hearing via a cochlear implant, as cochlear-implant users are typically unable to effectively utilize information from all available cochlear-implant channels (Henry & Turner, 2003). Normal-hearing listeners have shown similar performance as cochlear-implant users for speech understanding scores in quiet with 4–6 channels (Loizou et al., 1999). This is in line with pilot experiments in our lab that revealed that five normal-hearing listeners achieved a performance level of ∼80% in a consonant-vowel-consonant recognition test, when the words were vocoded with only 6 channels.

Hearing aid Simulation

Hearing-aid simulations were generated by using a fourth-order Butterworth low-pass filter with a cut-off at 500 Hz, mimicking residual hearing in the lower frequencies present in the bimodal cochlear implant users of our previous studies (bimodal here refers to listeners using a cochlear implant in one ear, and a hearing aid in the other; Sharma et al., 2019; Veugen et al., 2016a, 2016b). Additionally, the loss of frequency selectivity (spectral smearing) was simulated as previously described by Baer and Moore (1994). Asymmetrically broadened auditory filters were used with broadening factors 6 and 3 for the lower and upper branch respectively, as these are representative for moderate-to-severe hearing impairment (Glasberg & Moore, 1986). The cochlear implant and hearing-aid simulated stimuli were normalized to the same root-mean-squared value as the original non-vocoded sounds (Figure 1 visualizes the effect of the cochlear implant and hearing-aid simulation on ripples).

Paradigm

Listeners were instructed to press a button as quickly as possible when they heard the sound change from static noise to modulated ripple. Modulated ripples lasted for 3000 ms, unless the button was pressed, in which case the sound was ended prematurely, and the next trial was initiated after a brief (0.5–1 s) period of silence between each trial. If the button was pressed before ripple onset, the trial was reiterated, but no more than 4 times. The outcome measure of the experiment was the listener's manual reaction time, which was defined as the time between the onset of the ripple and the moment the button was pressed.

We tested five different listening conditions; acoustic stimuli were presented 1) monaurally (monaural normal hearing), 2) monaurally via cochlear-implant vocoder (unimodal cochlear implant), 3) monaurally via hearing-aid simulation (unimodal hearing aid), 4) binaurally (binaural normal hearing), 5) binaurally via bimodal cochlear implant and hearing-aid simulation (bimodal). In monaural conditions, both ears were tested separately. In the bimodal condition, cochlear implant and hearing aid were tested in both the right and left ear in different sessions. We did not test the binaural unimodal listening conditions (cochlear implant-cochlear implant or hearing aid-hearing aid). Each stimulus was presented 5 times in each listening condition. A complete data set thus contained a total of 6120 stimuli, which were split in 12 sessions of 30–40 min, each containing 510 trials. Sessions were distributed over 6 days of two sessions each. Ripples and conditions were presented in pseudo-randomized order.

Because of time constraints, data collection was not fully completed in the four naïve listeners. Two naïve listeners completed 11 out of 12 sessions, with all ripples measured at least twice. The other two listeners completed 9 out of 12 sessions (20 and 23 ripples not measured, respectively; 83 and 141 ripples were presented only once in these listeners).

The four naïve listeners performed one training session under normal hearing conditions prior to the recording sessions, to become familiarized with the ripple stimuli and experimental procedures. We observed no systematic change in the average reaction times during the training session for these four listeners, or over the time course of the experimental sessions for all six listeners (e.g., for binaural trials, the mean reaction time decreased marginally by −6 ms [95% confidence interval lower, upper bound: −42, + 47 ms] in the 401^st to 500^th trial compared to the first 100 trials, yielding a P value of a two-sided permutation t-test of 0.815; changes in mean reaction times varied between −27 to + 21 ms across listening conditions, all of which did not reach significance [P > 0.323]). This observation indicates that procedural learning effects did not confound the reaction-time data.

Analysis

Data analysis was performed with custom-written MATLAB software. Reaction times generally show a skewed distribution with an extended tail towards longer reaction times. To obtain normally distributed data (Carpenter et al., 2009), the reaction-time data were transformed to their reciprocal (1/reaction time), referred to here as “promptness” (1/s). This also allows the measurements to be more readily interpreted as sensitivity measures to the different spectrotemporal modulations, as a higher/lower promptness (as opposed to a shorter/longer reaction time) indicates a higher/lower sensitivity. Responses were pooled across listeners and ears for grand average analyses. Reaction times below 150 ms (clear anticipatory responses) were removed from the analysis. If a response was not initiated within 3 s (considered a sign for inattentiveness, or of an inability to detect the ripple), we set the response time (promptness) to 3 s (1/3 s⁻¹). Non-responses were found in 10% of the trials under normal-hearing conditions and in 46% of the trials of the hearing-impaired conditions (especially at the high spectral modulations). We do not explicitly account for the percentage of non-responses but note that in our analyses a higher number of non-responses would yield a median promptness (reaction time) closer to 1/3 s⁻¹ (3 s). The non-modulated sound (velocity 0 Hz and density 0 cycles/octave) served as a catch stimulus, to determine the guess (or false-alarm) rate of the participant. The guess rate varied from 7% for binaural normal hearing, to 21% for monaural hearing-aid listening, with the average guess rate across conditions at 12%.

Spectrotemporal Transfer Function

For each of the five listening conditions (monaural and binaural normal-hearing, and monaural and bimodal cochlear implant and hearing-aid simulation), we calculated the mean promptness per ripple to construct a two-dimensional spectrotemporal modulation transfer function $s t M T F (ω, Ω)$ as a joint function of ripple density, Ω, and ripple velocity, ω. Similarly, we determined the temporal modulation transfer function tMTF, F( $ω$ ), and the spectral modulation transfer function sMTF, G( $Ω$ ), for the 0-density and 0-velocity stimuli, respectively.

Separability

To analyze the degree of separability of the stMTF, we applied singular value decomposition (SVD) for all listening conditions. SVD transforms the stMTF into two unitary matrices containing temporal and spectral singular vectors, respectively, and a rectangular diagonal matrix that contains the singular values: stMTF(ω, Ω) = F(ω)·Σ·G(Ω). In case of a fully separable stMTF, the spectral and temporal components are independent of each other and the total of all 136 spectrotemporal responses can be expressed by the vectorial outer product of a single temporal F₁(ω) (17 components) and spectral G₁(Ω) (8 components) modulation transfer function, as follows:

s t M T F_{1} (ω, Ω) = σ_{1} \cdot F_{1} (ω) \times G_{1} (Ω)

(4)

with σ₁ the largest singular value.

We calculated the separability index (cf. inseparability index, as used by, for example, Massoudi et al., 2015; Versnel et al., 2009), which ranges between zero (totally inseparable) to one (fully separable), and is based on the relative dominance of the first SVD component:

α_{1} = \frac{σ_{1}^{2}}{\sum_{i}^{n} σ_{i}^{2}}

(5)

The separable stMTF estimate was reconstructed according to Equation (4).

We also reconstructed the stMTF based on the first two singular values, according to

s t M T F_{12} (ω, Ω) = σ_{1} \cdot F_{1} (ω) G_{1} (Ω) + σ_{2} \cdot F_{2} (ω) G_{2} (Ω)

(6)

and determined the relative contribution of the first two SVD components:

α_{2} = \frac{σ_{1}^{2} + σ_{2}^{2}}{\sum_{i}^{n} σ_{i}^{2}}

(7)

Before applying the SVD, the stMTF data was centered by subtracting the mean promptness for each listening condition. This mean was added to the reconstructions.

Race Model

We compared the observed reaction times for binaural hearing with the quantitative predictions of performance based on the monaural reaction times, using the race model of statistical facilitation. This model assumes independence of the two monaural processes (Gielen et al., 1983; Raab, 1962). Any violation to the race model suggests neural interactions when processing the input from both ears:

P (τ_{r a c e} \leq t) = P (τ_{M 1} \leq t) + P (τ_{M 2} \leq t) - P (τ_{M 1} \leq t) \times P (τ_{M 2} \leq t)

(8)

with P(τ ≤ t) the cumulative probability function (CDF) of an observed reaction time τ at time t; M₁ and M₂ represent monaurally-presented stimuli (normal hearing, cochlear implant and hearing aid). We estimated the cumulative probability density functions (CDF) from the promptness values. The race model CDF was constructed from the two monaural CDFs according to Equation (8). For comparative purposes, we calculated the difference in the medians (at the 50% cumulative probability level) between actual performance and race-model predictions. Ripples, for which fewer than 10 responses were collected, were discarded from this analysis because no reliable CDF could be constructed. Non-responses (reaction times > 2500 ms) were also discarded from the race-model analysis.

Statistics

Data were always reported as mean values ± 1 standard deviation. We also calculated 95% confidence intervals of promptness for the pure temporal and spectral ripples. As a criterion of significance for a statistical difference we took p < 0.05.

Results

Reaction-Time Task

We will first illustrate the systematic dependence of the manual reaction times on the acoustic conditions with the data of one listener (Figure 2). Pure amplitude-modulated noises elicit cumulative distributions of reaction times in the binaural listening condition that shift systematically with the velocity of the stimulus (Figure 2A). The cumulative distributions are plotted on a probit scale as a function of the reciprocal of reaction times (Carpenter & Williams, 1995; Corneil et al., 2002). In this format, the data points for each velocity fall closely on a straight line indicating that the promptness responses form a normal distribution. One may note that the singly-most distinguishing feature of these lines is that they are shifted versions of each other, with the slopes being similar across the velocity modulations. This may suggest that the mean reaction time and its inverse, promptness (the promptness at 50% cumulative probability) is a good point estimate of the effect of a ripple's velocity on reaction speed. Indeed, the mean promptness of this listener systematically and monotonically increases when velocity is increased (Figure 2D). Also, for spectral modulations, reaction time distributions (Figure 2B) and mean promptness (Figure 2E) vary systematically with a ripple's density, albeit that reaction speed seems to decrease with increasing density. Listening condition also affected this listener's response speed, as exemplified for a [1 Hz, 0 cycles/octave] modulation (Figure 2C and F), with binaural hearing eliciting the fastest responses and the monaural hearing-aid simulation yielding the slowest responses. In the following sections, we will quantify this relationship with promptness and the ripple modulation parameters for all listeners.

Figure 2.

Example reaction times to temporally and spectrally modulated target sounds. (A–C) Reaction time distributions plotted in the form of cumulative percentage probability, on a probit scale, as a function of reciprocal reaction time (promptness), measured in one listener (L1) in various listening regimes. The temporal and spectral modulation of the target sound and the listening condition was varied, leading to changes in the listener's detection speed. Each curve corresponds to a different modulation or listening condition. (a) Reaction time distributions to amplitude modulations (density = 0 cycles/octave) in a binaural listening condition are shown. As each distribution seemed to lie close to a straight line, regression lines are drawn, for illustrative purposes. Reaction time distributions to (b) spectral modulations (velocity = 0 Hz) in a binaural listening condition and to (c) different listening conditions for a single amplitude-modulated target sound (velocity = 2 Hz, density = 0 cycles/octave) are shown. (D–F) Mean promptness (cf. reaction time, see left ordinate) is plotted as a function of (d) velocity, (e) density, and (f) listening condition (stimulus parameters are the same as in A–C). Cochlear implant and hearing aid indicate cochlear-implant and hearing-aid simulation, respectively. Colours in each plot (2A–F) indicate velocity, density and listening condition as indicated by the abscissa-values of the coloured, filled circles (in 2D–F). Coloured margins of error around point estimates indicate 95% confidence intervals of the mean.

Temporal-Only Modulations

We will first elaborate on how reaction times reflect the detection of temporal-only modulations (Figure 3A). For the normal-hearing listening conditions (binaural and monaural), the mean promptness as a function of velocity for the purely temporal amplitude modulations (density = 0 cycles/octave) resembled a high-pass characteristic (Figure 3A, dark blue circles and light blue triangles). Responses were fastest (higher promptness) for the highest absolute velocities and were slower for lower velocities. If the sounds were vocoded, simulating hearing device processing (cochlear implant, hearing aid, bimodal), the promptness dropped especially for the higher velocities, so that the curve exhibited bandpass properties. Responses were now fastest for intermediate absolute velocities and were slower for both higher and lower velocities. Overall, both the maximal promptness and at which velocity this was attained were affected by listening condition; the fastest responses, with an average promptness of 3.3 (monaural, light blue triangles), 3.5 (binaural, dark blue circles), 2.5 (bimodal, pink squares), 2.5 (cochlear implant, light green diamonds) and 2 s⁻¹ (hearing aid, dark green triangles) were observed at ± 32, ± 32, ± 16, ± 16, ± 8 Hz, respectively. The longer response times to pure amplitude modulations under impaired-hearing conditions clearly implicate increased difficulty in the detection of temporal modulations. Differences in response times between binaural and monaural listening will be considered in more detail below with race-model predictions. Responses to upward (<0 Hz) and downward (>0 Hz) moving ripples were very similar; correlation coefficients between the responses to up- and downward ripples were between 0.91–0.99 for all listening conditions.

Figure 3.

Pure temporal and spectral modulation transfer functions. (a) Mean promptness (line and markers) and 95% confidence interval of the mean across subjects (patch) as a function of velocity (density = 0 cycles/octave) and (b) as a function of density (velocity = 0 Hz), for each listening condition: normal-hearing monaural and binaural conditions, and the impaired-hearing cochlear-implant, hearing-aid and bimodal simulations. White dots and colored vertical lines around 0 indicate the mean promptness and its 95% confidence interval for the catch trials with unmodulated stimuli ([velocity, density] = [0 Hz, 0 cycles/octave]).

Spectral-Only Modulations

For the static ripples (purely spectral modulations at velocity = 0 Hz), the promptness as a function of density resembled a low-pass characteristic, at least for the binaural and monaural normal-hearing conditions (Figure 3B, dark blue circles and light blue triangles, respectively). For these conditions, detection is very poor for the highest density of 8 cycles/octave. This property presumably reflects the limits of resolvable power of the human auditory filters, leading to a poorer detection of spectral patterns with finer spectral detail.

Responses made for the cochlear-implant simulation (light green diamonds) resembled a band-pass filter characteristic with a cutoff around 0.75 cycles/octave and responses in the hearing-aid condition (dark green triangles) followed a band-pass characteristic with its highest promptness around 1–2 cycles/octave. Bimodal responses (pink squares) resembled the best values of the cochlear-implant and hearing-aid conditions. Overall, the impact of hearing-impairment simulation on reaction times was generally larger for the spectral modulations (Figure 3B) than for the temporal modulations (Figure 3A). This behavioral finding seems in line with the acoustic transformation effects of the vocoders on the sounds, that preserve temporal modulations to some extent (cf. Figure 1, top row), but heavily perturb spectral modulations (cf. Figure 1, side columns).

Joint Spectrotemporal Modulation

Figure 4A–E show the stMTFs for the two normal-hearing conditions, and for the three impaired-hearing simulations, for all joint spectrotemporal ripples, as mean promptness (averaged across listeners; see supplemental Figure S1-6 for individual stMTFs) per ripple density (abscissa) and velocity (ordinate). Deep red colors correspond to high spectral-temporal sensitivity, dark blue colors to low sensitivity (low promptness values). The results for pure amplitude-modulated stimuli (cf. Figure 3A) are at the bottom row of the stMTF matrix, at Ω = 0 cycles/octave; the results for pure spectral modulations (cf. Figure 3B) are found along the central column, at ω = 0 Hz.

Figure 4.

Spectrotemporal modulation transfer functions. Mean promptness as a function of velocity and density, representing spectrotemporal sensitivity in the normal hearing and simulated hearing-impaired conditions. F–J) Reconstructions of the stMTFs using only the first singular value after singular value decomposition (SVD; Equation 4). K–O) Reconstructions of the stMTFs using the first and second SVD components (Equation 6).

Ripple detection in the binaural normal-hearing condition (Figure 4A) was faster (mean promptness = 3.0 s⁻¹) than in the monaural condition (Figure 4B; mean promptness: 2.8 s⁻¹). Combined spectrotemporal modulation sensitivity for both listening conditions resembled a two-dimensional band-pass for both density and velocity, with fastest reaction times around velocities of ± 8–16 Hz and densities around 0.75–1 cycles/octave (binaural maximum promptness = 4.0 s⁻¹ at [ω, Ω] = [8 Hz, 0.75 cycles/octave]; monaural maximum promptness = 3.7 s⁻¹ at [ω, Ω] = [ − 16 Hz, 1 cycles/octave]). The ripples were well detectable up to and including 4 cycles/octave.

The stMTFs for the impaired-hearing simulations (Figure 4C–E) were distinctly slower when compared to normal hearing (mean promptness = 0.9, 0.9 and 1.1 s⁻¹ for cochlear implant, hearing aid, and bimodal vocoder, respectively). Temporal modulation sensitivity again showed a band-pass filter characteristic with fastest detection rates around ± 16 Hz. Cochlear-implant simulations (Figure 4C) mainly reduced the detection of high spectral modulations, which is consistent with the modus operandi of a cochlear implant (and vocoders), whereby its band-pass filtering mechanism reduces spectral modulation sensitivity. The fastest responses (maximum promptness = 2.8 s⁻¹) were elicited to −16 Hz amplitude-modulated sounds (Ω = 0 cycles/octave). Ripple detection with the cochlear-implant vocoder became impossible for densities exceeding 0.75 cycles/octave. Of all hearing conditions, listeners reacted slowest for the hearing-aid simulations (Figure 4D; maximum promptness = 2.0 s⁻¹). However, in contrast to the cochlear-implant condition, higher densities of up to 4 cycles/octave could still be detected if temporal modulation rates were not too fast (>16 Hz). A local dip in promptness exists for ripples with a density around 0.5 cycles/octave.

The bimodal simulation resembled a conjunction of the cochlear-implant and hearing-aid simulation results, seemingly exhibiting a “best of both worlds” principle (Corneil et al., 2002) with responses almost as fast as for the best unimodal condition (Figure 4E; maximum promptness = 2.6 s⁻¹ at [ω, Ω] = [ − 16 Hz, 0 cycles/octave]). For high spectral modulation frequencies, the bimodal condition was comparable to the hearing-aid condition; for low spectral modulations it followed the cochlear-implant condition.

Separability

We assessed the degree of separability of the stMTF into a pure temporal and spectral component through singular value decomposition (SVD) using the separability index α₁ (Equation (5)) and the α₂ index (Equation (7)) to determine the relative contribution of the first (Equation (4)) and of the first two components (Equation (6)). If the α₁ index is close to 1, the MTF is considered to be separable (for individual-level separability indices and confidence intervals, see supplemental Figure S7).

The central row of Figure 4 shows the reconstructed stMTF₁ for the various hearing conditions. The first-order reconstruction of the stMTF yielded purely orthogonal patterns in the matrix, resulting from the full-separability assumption. For both normal hearing conditions, the separability index α₁ was high (0.87 and 0.86 for binaural and monaural listening conditions), which suggests that the variability in the normal-hearing stMTFs can be captured quite well with a first-order approximation (Figure 4F and G) and that the matrix is highly separable (equation (4)). Notably missing in the reconstructions are the slow responses to the slow amplitude-modulated sounds (ω<1 Hz, Ω = 0 cycles/octave; cf. Figures 4A and B and Figure 4F and G at the bottom of the images near the center). By adding the second singular value with its spectral and temporal components (bottom row of Figure 4), the stMTF reconstructions improved considerably: α₂ = 0.96 (monaural) and 0.97 (binaural). Now, the responses to the amplitude-modulated sounds seemed to be accounted for as well.

The separability index was best for the cochlear-implant simulation (Figure 4H), which equaled or was better than for the normal-hearing conditions (α₁ = 0.89). The first-order reconstructions were worse for the hearing-aid and bimodal conditions (α₁ = 0.61 and 0.77, respectively), suggesting a considerable inseparable spectrotemporal component to the responses of these two listening conditions. Incorporating the first two SVD components improved reconstructions (Figure 4M–O): α₂ = 0.96, 0.87, 0.93.

Race Models

To investigate to what extent monaural reaction times can predict binaural performance, we used the race model of statistical facilitation, which postulates independence between ears. As an example, Figure 5A displays the cumulative reaction time probability of listener L6 for a stimulus with a 2 Hz, 0 cycles/octave modulation, for the monaural and binaural normal-hearing conditions, as well as for the promptness that would be reached based on the race model of statistical facilitation. For this ripple, binaural performance was faster than both the monaural condition and the race model. For the simulated listening conditions (Figure 5B), the bimodal responses to this ripple were faster than the hearing-aid data and resembled the cochlear-implant data and the race model.

Figure 5.

Binaural versus monaural detection of spectrotemporal modulations. (a) Reaction time distributions of listener L6 plotted in the same format as in Figure 2A–C for a stimulus with a 2 Hz, 0 cycles/octave modulation while listening monaurally (light blue) or binaurally (dark blue). The race model (determined from the monaural distribution according to Equation 8) is indicated by a light blue dashed line. Dots indicate individual reaction times; lines indicate the best linear fit through the data. The horizontal dotted line is drawn at a cumulative probability of 0.5 crossing the median reaction times. These medians for the bimodal data and the race model are indicated by an open square and circle, respectively. (b) The same as in A, but now for the cochlear-implant, hearing-aid and bimodal listening conditions. Note that the binaural data is slightly faster than the race model, while bimodal data resembles both the cochlear-implant data and the race model. (c) The median promptness as observed in the data is plotted as a function of the race-model prediction (closed circles and unbroken lines). Blueish and reddish colors indicate binaural and bimodal listening conditions, respectively. For each listener (indicated by different tint), the mean across ripple modulations is indicated by the closed circles, and standard deviation in the direction of the two axes with largest variability is indicated by lines. Note that the data fall closely on the unity-line (black dashed line), which holds both for the mean and the main axis with largest variability.

To quantify this for all ripples and listeners, we compared the median predictions from the benchmark race model to the median data (Figure 5C). Overall, listeners were as fast as the race model prediction, both for binaural (Figure 5C, blue circles on the diagonal) and bimodal hearing (Figure 5C, reddish colors). These results show that binaural and bimodal performance seemed to follow statistical facilitation (Equation (8)).

Discussion

Summary

This study used a speeded-response paradigm to determine the auditory stMTF in human listeners. The reaction times obtained appeared to be a valid and effective objective measure for ripple sensitivity, given its systematic relationship with the parameters that determine both temporal and spectral modulation rates. Sensitivity was highest for ripples with modulations around 16 Hz and 1 cycle/octave and decreased for higher and lower modulation rates. Using simulations of a cochlear-implant, hearing-aid and bimodal restorative hearing, spectrotemporal sensitivity worsened (reaction times increased) compared to normal hearing, as expected from the impaired signal processing of the simulations. Although the separability of the stMTF into a spectral and temporal component was high for both the normal-hearing and for the cochlear-implant simulated data, the inseparable second-order spectrotemporal component was still substantial, with a value between 7–10%. For the bimodal and especially the hearing-aid conditions, inseparability was larger by about 16 and 28%. For all ripples, binaural and bimodal reaction-time performance was comparable to the prediction of the benchmark race model of statistical facilitation, suggesting independent detection of two monaural signals, rather than true binaural integration.

Normal Hearing

Constructing the stMTF based on reaction times is a fairly new approach that has been introduced so far only in monkey research (Massoudi et al., 2013, 2014). Still, the observed stMTFs (Figures 3 and 4A–E) correspond well with the results of other studies in humans, which have measured modulation detection thresholds. Chi et al. (1999) and Zheng et al. (2017) measured the full stMTF for normal-hearing listeners using an adaptive modulation-detection threshold paradigm and found band-pass functions for both the spectral and temporal dimensions. They found best ripple detection-thresholds at spectral modulations below or at 1 cycle/octave, and temporal modulations around 4–16 Hz. Despite these small quantitative differences between studies, the general patterns were similar, and in line with our results (Figure 6): when comparing our promptness data (Figure 6C and D) with the modulation thresholds collected in the earlier studies (Chi et al., 1999; Zheng et al., 2017 in Figure 6A and B), the stMTFs resemble each other, at least qualitatively. These convergent findings suggest that reaction times are indeed a valid objective measure to determine the spectrotemporal sensitivity of (naïve) listeners.

Figure 6.

Comparison of spectrotemporal sensitivity. Images depict stMTFs obtained using (a, b) a threshold-searching paradigm from the studies of (a) Chi et al. (1999) and (b) Zheng et al. (2017) or (c, d) a reaction-time task from this study in the (c) binaural and (d) monaural listening conditions. Data from previous studies was obtained from Figure 3 of the respective papers. Image format is the same as in Figure 4. Data (modulation index) in (A) was log-transformed (20log₁₀) to match the data (modulation threshold in dB) in (B). Data from this study in C and D was replotted from Figure 4, matching the velocities and densities used in the previous studies (A vs C, B vs D). Note that the colour scales in (A, B) are reversed in order, as best responses correspond to high promptness values, but low modulation indices or thresholds.

Chi et al. (1999) proposed a computational model to explain their data, in which the spectrotemporal modulation sensitivity is based on cortical responses to the ripple's spectrogram. The modulation transfer functions generated by their model closely resembled their data, and thus will resemble our data as well. They concluded that “the upper limits of the spectral and temporal modulation rates are related through the effective bandwidths of the cochlear filters” (Chi et al., 1999).

Narne et al. (2016, 2018, 2020) have studied spectral resolution by means of a spectral-ripple or a moving-ripple test. They found thresholds around 5 to 6 cycles/octave for normal-hearing listeners in optimal conditions. Again, this seems to be in line with the strong drops in sensitivity observed in our data for densities at 8 cycles/oct in the non-processed conditions. This also suggests that we might have missed a more gradual decline of promptness, as we did not study any densities between 4 and 8 kHz.

Impaired-Hearing Simulations

To evaluate our reaction-time test under more challenging listening situations, we manipulated the ripple stimuli using hearing-aid and cochlear-implant simulations. Both simulations made it substantially harder to detect ripple modulations and even impossible for certain parameters, eliciting much longer reaction times compared to monaural normal hearing for all ripples. Bimodal hearing exhibited a “best-of-both-worlds” effect, following the fastest unimodal condition, which was the cochlear-implant condition for spectral modulations below 0.75 cycles/octave, and the hearing aid for higher spectral modulations (Figure 4). An improvement in spectral ripple discrimination for bimodal hearing over the cochlear-implant alone has also been found in users with combined electro-acoustic stimulation in the same ear (Golub et al., 2012).

Other studies have shown a 5–10 dB reduction in the temporal-modulation detection threshold for cochlear-implant users compared to normal hearing (Bacon & Viemeister, 1985; Golub et al., 2012; Won et al., 2011), whereas we found a decrease in promptness of 0.4 ± 0.2 s⁻¹ for the well-detectable rates below 16 Hz. In these studies, hearing-impaired listeners performed in between normal hearing and cochlear-implant users. Our hearing-aid simulation, however, showed longer reaction times for temporal modulations (at 0 cycles/octave) compared to the cochlear implant condition. It should be emphasized that our hearing-aid condition was based on a worst-case-scenario, simulating very little residual hearing, whereas hearing thresholds of the hearing-impaired listeners in the study of Bacon and Viemeister (1985) still reached up to 10–20 dB HL at 1 kHz. Their study also showed a link between degraded temporal sensitivity and reduced listening bandwidth.

Impaired spectral modulation sensitivity with a cochlear implant is a likely result of its band-pass filtering mechanism that limits the spectral information to a set number of spectral bands. Henry et al. (2005), Berenstein et al. (2008) and Narne et al. (2020) all found lower spectral ripple modulation thresholds for cochlear-implant users compared to normal-hearing listeners, roughly corresponding to the increased reaction times in our study. Spectral modulation thresholds in hearing-impaired listeners have been reported to be 5–10 dB worse than for normal hearing (Davies-Venn et al., 2015; Summers & Leek, 1994), which may agree with the longer reaction times of our hearing-aid simulation compared to normal hearing. A few studies investigated combined spectrotemporal modulation detection thresholds in hearing-impaired listeners, which were often worse compared to normal hearing listeners, especially for low temporal modulation rates (Bernstein et al., 2013; Mehraei et al., 2014; Zheng et al., 2017; Zhou et al., 2020).

Race Model

To get insights in the mechanism of combining input at both ears, we used race models to test whether monaural responses could predict binaural performance. For normal-hearing conditions, faster reaction times were elicited when stimuli were presented binaurally compared to monaural presentation (Figure 4A and B). Binaural responses seemed as fast as the race model of statistical facilitation (Figure 5C). This suggests that ripple detection was determined by a parallel race between the two ears, rather than from neural integration.

Bimodal responses were also as fast as the race model (Figure 5C), suggesting that there is also no benefit of integration for the poorest listening conditions. This is interesting if we compare this finding to audiovisual gaze-orienting experiments that aim to study neural integration of visual and auditory signals. Strongest benefits of multisensory interactions (i.e., increased speed, accuracy, and precision of responses) are obtained for stimuli that overlap both in space and time, and thus provide multisensory evidence for a single object. Moreover, these interactions are strongest when the uni-sensory evoked responses are variable and slow (i.e., away from ceiling performance). This phenomenon is known as the “principle of inverse effectiveness” (e.g., Corneil et al., 2002; Stein & Meredith, 1993; van de Rijt et al., 2019; Van Wanrooij et al., 2009). We here propose that beneficial effects of bimodal (cochlear-implant-hearing-aid) integration will depend on whether the auditory system has sufficient evidence that left vs. right acoustic inputs arose from the same auditory object, rather than from unrelated sounds. The strongest bimodal benefits (i.e., enhanced sensitivity) will thus be found: (i) when spectral ranges of cochlear implant and hearing-aid overlap sufficiently (for within-spectral comparisons), and (ii) when monaural reaction-time distributions have sufficient variability and overlap considerably. Since neither of these two requirements seem to be fulfilled, it may be unsurprising that bimodal listening does not exceed race-model performance and does not benefit from neural integration. Instead, the benefit of bimodal listening seems to consist purely of predominantly perceiving low-density temporal modulations with the cochlear-implant ear and low-velocity spectral modulations with the hearing-aid ear (Figures 4C–E).

In contrast to our results, several studies have shown reaction times to stimuli that exceeded the predictions based on statistical facilitation. However, these studies typically involved responses to multisensory stimuli, or to the dichotic presentation of two spectrally distinct sounds (Gielen et al., 1983; Miller, 1982; Schröter et al., 2007; Townsend & Nozawa, 1995). Like the findings of Schröter et al. (2007) for auditory stimuli that fused into a single percept, we did not obtain faster responses than expected from statistical facilitation in the bimodal conditions. The benefits of an integrative process must be found outside the parameters presented in this study, and likely include the ability to localize sounds and enhancement of speech perception in noisy environments.

Separability

Measuring the stMTF is typically a time-consuming process, for which it would be valuable to know whether the two-dimensional function is simply the product of a temporal and spectral component. A large degree of separability of spectrotemporal sensitivity has been found for normal-hearing listeners and cochlear-implant users (Chi et al., 1999; Zheng et al., 2017). Our data support this notion of large separability (but not “full” separability) for the normal-hearing and cochlear-implant listening conditions. Nevertheless, the contribution of the second singular value was still substantial, close to 10% in the normal-hearing condition. For the hearing-aid and bimodal listening conditions, the contribution could be as large as 20%, implying a large degree of inseparability. Studies by Bernstein et al. (2013) and Zheng et al. (2017) found that spectrotemporal cues may enhance speech intelligibility and modulation sensitivity in CI users over spectral or temporal modulations alone. These findings hint at some form of spectrotemporal integration, but interestingly this may not be captured by the relatively large separability of the stMTF (Zheng et al., 2017).

As can be appreciated from the observable differences between the single and two-component reconstructions (Figure 4), the second component adds a diagonal interaction component to the two-dimensional MTF. Most of this interaction yields changes to the MTF at 0- or low-density modulations: reactions become slower for low-velocity modulations, and faster for higher velocities. We conclude that a full assessment of spectrotemporal performance requires testing the complete or large spectrotemporal space of all ripples, by including the second SVD component (Figure 4).

A Novel Hearing Test?

We believe that a reaction-time paradigm could provide an alternative novel hearing test. This test potentially provides a description of acoustic sensitivity that is potentially closer to speech perception accuracy than pure-tone audiometry. Here, we demonstrated the convergent validity of the test by showing that the stMTF obtained from reaction times resemble the stMTFs obtained from standard detection thresholds (Figure 6). Earlier studies have shown that reaction times (chronometric function) and detection thresholds (psychometric function) are tightly coupled for certain experimental paradigms (Palmer et al., 2005a, 2005b). By using sequential-sampling or drift-diffusion models (Palmer et al., 2005a, 2005b; Rach et al., 2011), the two measures could be integrated into a single framework. A study on audiovisual integration shows that the two measures provide complementary, but not the same information (Rach et al., 2011) and a study on pure-tone detection also found the measures to be not interchangeable (Abel et al., 2009). How strong the coupling is for the modulation-onset detection test is of yet unknown.

The test is reliable across listeners, yielding highly similar stMTFs (except for an idiosyncratic reaction time offset, supplemental Figures S1-6) and separability indices with narrow confidence intervals (supplemental Fig. S7). An advantage over a modulation-detection threshold paradigm is that stimuli can be presented at supra-threshold levels, allowing for a relatively easy task, potentially suitable for clinical assessments, and for studies with children, or with experimental animals. This is supported by the fact that we did not observe procedural learning effects during the experiments.

Nevertheless, usability and construct validity of a reaction-time hearing test should be explored further. There is reason to believe that stimulus parameters other than velocity and density may influence the stMTF. Modulation depth, sound level, and spectral profile each may affect the stMTF by enhancing or suppressing modulation sensitivity. For example, a full modulation depth of 100% may produce a richer characterization of the modulation space especially in the impaired conditions and reduce the number of conditions in which the stimuli are undetectable. Similarly, by having a carrier with a speech-shaped spectrum or a pink spectrum, detection of modulation onset might correlate better with detection of natural sounds, such as speech (phonemes or words) or music. In fact, one can think of developing a framework that relates the sensitivity to the spectrotemporal modulations of the moving ripples to the recognition and pitch accuracy of speech materials such as phonemes and words (Elliott & Theunissen, 2009). While beyond the scope of this paper, note that theoretically these dynamic signals can be decomposed into their constituent ripples. In principle, there could be a direct relationship, although confounded by noise in the responses and by cognitive, non-acoustic aspects (such as predictability of a word in a sentence).

Numerous studies have shown that non-acoustic aspects, such as attention and reward expectation, could affect the spectrotemporal receptive fields of cells in primary auditory cortex (Atiani et al., 2009; Bakin et al. 1996; David et al., 2012; Fritz et al., 2003, 2005, 2007, 2010; Jaramillo & Zador, 2011; Ji et al., 2001; Kilgard et al., 2001, 2002; Kilgard & Merzenich, 2002; Lee & Middlebrooks, 2011; Ohl & Scheich, 1997, 2005; Suga et al., 2002; Suga & Ma, 2003). It would not be farfetched to believe that these changes would also be reflected in the overt behavior. In our experiments, we chose to randomize the order of listening conditions and stimulus parameters, making it impossible for the listener to predict what is coming up. This procedure allows for a direct, acute assessment of acoustic spectrotemporal sensitivity. The downside, of course, is that any adaptive change in the stMTF will not be captured by this test. Conducting the same task following a blocked design, might produce different filters. Similarly, the stMTFs observed for the cochlear-implant and hearing-aid simulations might differ from those that would be obtained from actual hearing-impaired listeners. Due to long-term experience, they could have acclimatized or adapted to their hearing impairment.

Conclusion

Reaction times are a valid objective measure for ripple sensitivity. The joint spectrotemporal transfer function closely resembled data from earlier studies that used modulation detection thresholds. Responses to spectrotemporal modulated ripples could be reconstructed by using the first two components of singular value decomposition, suggesting significant spectrotemporal inseparability, especially for the hearing-aid and bimodal listening conditions. We further found that binaural and simulated bimodal reaction times could be predicted from statistical facilitation induced by a race of independent monaural inputs.

Supplemental Material

sj-docx-1-tia-10.1177_23312165221127589 - Supplemental material for Reaction Time Sensitivity to Spectrotemporal Modulations of Sound

Supplemental material, sj-docx-1-tia-10.1177_23312165221127589 for Reaction Time Sensitivity to Spectrotemporal Modulations of Sound by Lidwien C. E. Veugen, A. John van Opstal and Marc M. van Wanrooij in Trends in Hearing

Footnotes

Acknowledgements

We thank Ruurd Lof, Stijn Martens, and Günter Windau for their technical support and Daisy Louvet for her assistance in data collection.

Data Availability Statement

Data are available from the Donders Institute for Brain, Cognition and Behaviour repository at https://data.donders.ru.nl and can be found via the following persistent identifier: .

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the H2020 European Research Council, (grant number 693400).

ORCID iD

Marc M. van Wanrooij

Supplemental Material

Supplemental material for this article is available online.

References

Abel

S. M.

Rajan

D. K.

Giguere

. (2009). Stimulus parameters in detection and reaction time. Scandinavian Audiology, 19(4), 229–235. https://doi.org/10.3109/01050399009070777

Atiani

Elhilali

David

S. V.

Fritz

J. B.

Shamma

S. A

. (2009). Task difficulty and performance induce diverse adaptive patterns in gain and shape of primary auditory cortical receptive fields. Neuron, 61, 467–480. https://doi.org/10.1016/j.neuron.2008.12.027

Ausili

S. A.

Backus

Agterberg

M. J. H.

van Opstal

A. J.

van Wanrooij

M. M

. (2019). Sound Localization in Real-Time Vocoded Cochlear-Implant Simulations With Normal-Hearing Listeners. Trends in Hearing, 23. https://doi.org/10.1177/2331216519847332

Bacon

S. P.

Viemeister

N. F

. (1985). Temporal modulation transfer functions in normal-hearing and hearing-impaired listeners. Audiology: Official Organ of the International Society of Audiology, 24(2), 117–134. https://doi.org/10.3109/00206098509081545

Baer

Moore

B. C. J

. (1994). Effects of spectral smearing on the intelligibility of sentences in the presence of interfering speech. The Journal of the Acoustical Society of America, 95(4), 2277. https://doi.org/10.1121/1.408640

Bakin

J. S.

South

D. A.

Weinberger

N. M

. (1996). Induction of receptive field plasticity in the auditory cortex of the guinea pig during instrumental avoidance conditioning. Behavioral Neuroscience, 110, 905–913. https://doi.org/10.1037/0735-7044.110.5.905

Berenstein

C. K.

Mens

L. H. M.

Mulder

J. J. S.

Vanpoucke

F. J

. (2008). Current steering and current focusing in cochlear implants: Comparison of monopolar, tripolar, and virtual channel electrode configurations. Ear and Hearing, 29(2), 250–260. https://doi.org/10.1097/AUD.0b013e3181645336

Bernstein

L. R.

Green

D. M

. (1998). Detection of simple and Complex changes of spectral shape. The Journal of the Acoustical Society of America, 82(5), 1587. https://doi.org/10.1121/1.395147

Bernstein

J. G. W.

Mehraei

Shamma

Gallun

F. J.

Theodoroff

S. M.

Leek

M. R

. (2013). Spectrotemporal modulation sensitivity as a predictor of speech intelligibility for hearing-impaired listeners. Journal of the American Academy of Audiology, 24(4), 293–306. https://doi.org/10.3766/jaaa.24.4.5

10.

Carpenter

R. H. S.

Reddi

B. A. J.

Anderson

A. J

. (2009). A simple two-stage model predicts response time distributions. The Journal of Physiology, 587(16), 4051–4062. https://doi.org/10.1113/jphysiol.2009.173955

11.

Carpenter

R. H.

Williams

M. L

. (1995). Neural computation of log likelihood in control of saccadic eye movements. Nature, 377(6544), 59–62. https://doi.org/10.1038/377059a0

12.

Chi

Gao

Guyton

M. C.

Shamma

. (1999). Spectro-Temporal modulation transfer functions and speech intelligibility. The Journal of the Acoustical Society of America, 106(5), 2719–2732. https://doi.org/10.1121/1.428100

13.

Corneil

B. D.

Van Wanrooij

Munoz

D. P.

Van Opstal

A. J

. (2002). Auditory-Visual interactions subserving goal-directed saccades in a complex scene. Journal of Neurophysiology, 88(1), 438–454. https://doi.org/10.1152/jn.2002.88.1.438

14.

David

S. V.

Fritz

J. B.

Shamma

S. A

. (2012). Task reward structure shapes rapid receptive field plasticity in auditory cortex. Proceedings of the National Academy of Sciences of the United States of America, 109, 2144–2149. https://doi.org/10.1073/pnas.1117717109

15.

Davies-Venn

Nelson

Souza

. (2015). Comparing auditory filter bandwidths, spectral ripple modulation detection, spectral ripple discrimination, and speech recognition: Normal and impaired hearinga). The Journal of the Acoustical Society of America, 138(1), 492. https://doi.org/10.1121/1.4922700

16.

Depireux

D. A.

Simon

J. Z.

Klein

D. J.

Shamma

S. A

. (2001). Spectro-Temporal response field characterization with dynamic ripples in ferret primary auditory cortex. Journal of Neurophysiology, 85(3), 1220–1234. https://doi.org/10.1152/jn.2001.85.3.1220

17.

Elliott

T. M.

Theunissen

F. E

. (2009). The modulation transfer function for speech intelligibility edited by K. J. Friston. PLoS Computational Biology, 5(3), e1000302. https://doi.org/10.1371/journal.pcbi.1000302

18.

Fritz

Shamma

Elhilali

Klein

. (2003). Rapid task-related plasticity of spectrotemporal receptive fields in primary auditory cortex. Nature Neuroscience, 6, 1216–1223. https://doi.org/10.1038/nn1141

19.

Fritz

J. B.

Elhilali

Shamma

S. A

. (2005). Differential dynamic plasticity of A1 receptive fields during multiple spectral tasks. Journal of Neuroscience, 25, 7623–7635. https://doi.org/10.1523/JNEUROSCI.1318-05.2005

20.

Fritz

J. B.

Elhilali

Shamma

S. A

. (2007). Adaptive changes in cortical receptive fields induced by attention to complex sounds. Journal of Neurophysiology, 98, 2337–2346. https://doi.org/10.1152/jn.00552.2007

21.

Fritz

J. B.

David

S. V.

Radtke-Schuller

Yin

Shamma

S. A

. (2010). Adaptive, behaviorally gated, persistent encoding of task-relevant auditory information in ferret frontal cortex. Nature Neuroscience, 13, 1011–1019. https://doi.org/10.1038/nn.2598

22.

Gielen

S. C. A. M.

Schmidt

R. A.

Van Den Heuvel

P. J. M

. (1983). On the nature of intersensory facilitation of reaction time. Perception & Psychophysics, 34(2), 161–168. https://doi.org/10.3758/BF03211343

23.

Glasberg

B. R.

Moore

B. C. J

. (1986). Auditory filter shapes in subjects with unilateral and bilateral cochlear impairments. The Journal of the Acoustical Society of America, 79(4), 1020. https://doi.org/10.1121/1.393374

24.

Golub

J. S.

Won

J. H.

Drennan

W. R.

Worman

T. D.

Rubinstein

J. T

. (2012). Spectral and temporal measures in hybrid cochlear implant users: On the mechanism of electroacoustic hearing benefits. Otology and Neurotology, 33(2), 147–153. https://doi.org/10.1097/MAO.0b013e318241b6d3

25.

Henry

B. A.

Turner

C. W

. (2003). The resolution of Complex spectral patterns by cochlear implant and normal-hearing listeners. The Journal of the Acoustical Society of America, 113(5), 2861. https://doi.org/10.1121/1.1561900

26.

Henry

B. A.

Turner

C. W.

Behrens

. (2005). Spectral peak resolution and speech recognition in quiet: Normal hearing, hearing impaired, and cochlear implant listeners. The Journal of the Acoustical Society of America, 118(2), 1111–1121. https://doi.org/10.1121/1.1944567

27.

Hershenson

. (1962). Reaction time as a measure of intersensory facilitation. Journal of Experimental Psychology, 63(3), 289–293. https://doi.org/10.1037/h0039516

28.

Jaramillo

Zador

A. M

. (2011). The auditory cortex mediates the perceptual effects of acoustic temporal expectation. Nature Neuroscience, 14, 246–251. https://doi.org/10.1038/nn.2688

29.

Gao

Suga

., (2001). Effects of acetylcholine and atropine on plasticity of central auditory neurons caused by conditioning in bats. Journal of Neurophysiology, 86, 211–225. https://doi.org/10.1152/jn.2001.86.1.211

30.

Kilgard

M. P.

Merzenich

M. M

. (2002). Order-Sensitive plasticity in adult primary auditory cortex. Proceedings of the National Academy of Sciences of the United States of America, 99, 3205–3209. https://doi.org/10.1073/pnas.261705198

31.

Kilgard

M. P.

Pandya

P. K.

Vazquez

Gehi

Schreiner

C. E.

Merzenich

M. M

. (2001). Sensory input directs spatial and temporal plasticity in primary auditory cortex. Journal of Neurophysiology, 86, 326–338. https://doi.org/10.1152/jn.2001.86.1.326

32.

Kilgard

M. P.

Pandya

P. K.

Engineer

N. D.

Moucha

. (2002). Cortical network reorganization guided by sensory input features. Biocybernetics, 87, 333–343. https://doi.org/10.1007/s00422-002-0352-z

33.

Lee

C. -C.

Middlebrooks

J. C

. (2011). Auditory cortex spatial sensitivity sharpens during task performance. Nature Neuroscience, 14, 108–114. https://doi.org/10.1038/nn.2713

34.

Liberman

A. M.

(1996). Speech : A Special Code. 458.

35.

Litvak

L. M.

Spahr

A. J.

Saoji

A. A.

Fridman

G. Y

. (2007). Relationship between perception of spectral ripple and speech recognition in cochlear implant and vocoder listeners. The Journal of the Acoustical Society of America, 122(2), 982. https://doi.org/10.1121/1.2749413

36.

Loizou

P. C.

Dorman

. (1999). On the number of channels needed to understand speech. The Journal of the Acoustical Society of America, 106(4), 2097. https://doi.org/10.1121/1.427954

37.

Massoudi

Van Wanrooij

M. M.

Van Wetter

S. M. C. I.

Versnel

John Van Opstal

(2013). Stable Bottom-up Processing during Dynamic Top-down Modulations in Monkey Auditory Cortex. European Journal of Neuroscience, 37(11), 1830–1842. https://doi.org/10.1111/ejn.12180.

38.

Massoudi

Van Wanrooij

M. M.

Van Wetter

S. M. C. I.

Versnel

John Van Opstal

(2014). Task-Related Preparatory Modulations Multiply with Acoustic Processing in Monkey Auditory Cortex. European Journal of Neuroscience, 39(9), 1538–1550. https://doi.org/10.1111/ejn.12532.

39.

Massoudi

Van Wanrooij

M. M.

Versnel

John Van Opstal

(2015). Spectrotemporal response properties of core auditory Cortex neurons in awake monkey. PLOS ONE, 10(2), e0116118. https://doi.org/10.1371/journal.pone.0116118

40.

McDermott

J. H.

Simoncelli

E. P

. (2011). Sound texture perception via statistics of the auditory periphery: Evidence from sound synthesis. Neuron, 71(5), 926–940. https://doi.org/10.1016/j.neuron.2011.06.032

41.

Mehraei

Gallun

F. J.

Leek

M. R.

Bernstein

J. G. W

. (2014). Spectrotemporal modulation sensitivity for hearing-impaired listeners: Dependence on carrier center frequency and the relationship to speech intelligibility. The Journal of the Acoustical Society of America, 136(1), 301. https://doi.org/10.1121/1.4881918

42.

Miller

. (1982). Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology, 14(2), 247–279. https://doi.org/10.1016/0010-0285(82)90010-X

43.

Moore

B. C.

Glasberg

B. R

. (2001). Temporal modulation transfer functions obtained using sinusoidal carriers with normally hearing and hearing-impaired listeners. The Journal of the Acoustical Society of America, 110(2), 1067–1073. https://doi.org/10.1121/1.1385177

44.

Narne

V. K.

Jain

Sharma

Baer

Moore

B. C. J

. (2020). Narrow-Band ripple glide direction discrimination and its relationship to frequency selectivity estimated using psychophysical tuning curves. Hearing Research, 389, 107910. https://doi.org/10.1016/j.heares.2020.107910

45.

Narne

V. K.

Prabhu

Van Dun

Moore

B. C. J

. (2018). Ripple glide direction discrimination and its relationship to frequency selectivity estimated using notched noise. Acta Acustica United with Acustica, 104(6), 1063–1074. https://doi.org/10.3813/AAA.919272

46.

Narne

V. K.

Sharma

Van Dun

Bansal

Prabhu

Moore

B. C. J

. (2016). Effects of spectral smearing on performance of the spectral ripple and spectro-temporal ripple tests. The Journal of the Acoustical Society of America, 140(6), 4298. https://doi.org/10.1121/1.4971419

47.

Ohl

F. W.

Scheich

. (1997). Learning-Induced dynamic receptive field changes in primary auditory cortex of the unanaesthetized Mongolian gerbil. Journal of Comparative Physiology A, 181, 685–696. https://doi.org/10.1007/s003590050150

48.

Ohl

F. W.

Scheich

. (2005). Learning-Induced plasticity in animal and human auditory cortex. Current Opinion in Neurobiology, 15, 470–477. https://doi.org/10.1016/j.conb.2005.07.002

49.

Palmer

Huk

A. C.

Shadlen

M. N.

(2005a). And accuracy of a perceptual decision. 376–404.

50.

Palmer

Huk

A. C.

Shadlen

M. N

. (2005b). The effect of stimulus strength on the speed and accuracy of a perceptual decision. Journal of Vision, 5(5), 1–1. https://doi.org/10.1167/5.5.1

51.

Raab

D. H

. (1962). Statistical facilitation of simple reaction times. Transactions of the New York Academy of Sciences, 24(5 Series II), 574–590. https://doi.org/10.1111/j.2164-0947.1962.tb01433.x

52.

Rach

Diederich

Colonius

. (2011). On quantifying multisensory interaction effects in reaction time and detection rate. Psychological Research, 75(2), 77–94. https://doi.org/10.1007/s00426-010-0289-0

53.

Ratcliff

Van Dongen

H. P. A

. (2011). Diffusion model for one-choice reaction-time tasks and the cognitive effects of sleep deprivation. Proceedings of the National Academy of Sciences of the United States of America, 108(27), 11285–11290. https://doi.org/10.1073/pnas.1100483108

54.

Schröter

Ulrich

Miller

. (2007). Effects of redundant auditory stimuli on reaction time. Psychonomic Bulletin & Review, 14(1), 39–44. https://doi.org/10.3758/BF03194025

55.

Shannon

R. V.

Zeng

F. G.

Kamath

Wygonski

Ekelid

. (1995). Speech recognition with primarily temporal cues. Science (New York, N.Y.), 270(5234), 303–304. https://doi.org/10.1126/science.270.5234.303

56.

Sharma

Mens

L. H. M.

Snik

A. F. M.

John van Opstal

van Wanrooij

M. M.

(2019). An individual with hearing preservation and bimodal hearing using a cochlear implant and hearing aids has perturbed sound localization but preserved speech perception. Frontiers in Neurology, 10(637). https://doi.org/10.3389/fneur.2019.00637

57.

Sharma

Nogueira

John van Opstal

Chalupper

Mens

L. H. M.

van Wanrooij

M. M

. (2021). Amount of frequency compression in bimodal cochlear implant users is a poor predictor for audibility and spatial hearing. Journal of Speech, Language, and Hearing Research 64(12), 5000–5013. https://doi.org/10.1044/2021_JSLHR-20-00653.

58.

Stein

B. E.

Meredith

M. A.

(1993). The Merging of the Sense. The MIT Press.

59.

Suga

. (2003). Multiparametric corticofugal modulation and plasticity in the auditory system. Nature Reviews Neuroscience, 4, 783–794. https://doi.org/10.1038/nrn1222

60.

Suga

Xiao

. (2002). Plasticity and corticofugal modulation for hearing in adult animals. Neuron, 36, 9–18. https://doi.org/10.1016/S0896-6273(02)00933-9

61.

Summers

Leek

M. R

. (1994). The internal representation of spectral contrast in hearing-impaired listeners. The Journal of the Acoustical Society of America, 95(6), 3518–3528. https://doi.org/10.1121/1.409969

62.

Supin

A. Y.

Popov

V. V.

Milekhina

O. N.

Tarakanov

M. B

. (1994). Frequency resolving power measured by rippled noise. Hearing Research, 78(1), 31–40. https://doi.org/10.1016/0378-5955(94)90041-8

63.

Townsend

J. T.

Nozawa

. (1995). Spatio-Temporal properties of elementary perception: An investigation of parallel, serial, and coactive theories. Journal of Mathematical Psychology, 39(4), 321–359. https://doi.org/10.1006/jmps.1995.1033

64.

van de Rijt

L. P. H.

Roye

Mylanus

E. A. M.

John van Opstal

van Wanrooij

M. M.

(2019). The principle of inverse effectiveness in audiovisual speech perception. Frontiers in Human Neuroscience, 13(335). https://doi.org/10.3389/fnhum.2019.00335

65.

Versnel

Zwiers

M. P.

John van Opstal

(2009). Spectrotemporal response properties of Inferior colliculus neurons in alert monkey. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 29(31), 9725–9739. https://doi.org/10.1523/JNEUROSCI.5459-08.2009

66.

Veugen

L. C. E.

Chalupper

Snik

A. F. M.

John van Opstal

Mens

L. H. M.

(2016a). Frequency-Dependent loudness balancing in bimodal cochlear implant users. Acta Oto-Laryngologica, 136(8), 775–781. https://doi.org/10.3109/00016489.2016.1155233

67.

Veugen

L. C. E.

Hendrikse

M. M. E.

van Wanrooij

M. M.

Agterberg

M. J. H.

Chalupper

Mens

L. H. M.

Snik

A. F. M.

John van Opstal

(2016b). Horizontal sound localization in cochlear implant users with a contralateral hearing aid. Hearing Research, 336, 72–82. https://doi.org/10.1016/j.heares.2016.04.008

68.

Viemeister

N. F

. (1998). Temporal modulation transfer functions based upon modulation thresholds. The Journal of the Acoustical Society of America, 66(5), 1364. https://doi.org/10.1121/1.383531

69.

Van Wanrooij

M. M.

Bell

A. H.

Munoz

D. P.

John Van Opstal

(2009). The effect of spatial-temporal audiovisual disparities on saccades in a Complex scene. Experimental Brain Research, 198(2–3), 425–437. https://doi.org/10.1007/s00221-009-1815-4

70.

Won

J. H.

Drennan

W. R.

Nie

Jameyson

E. M.

Rubinstein

J. T

. (2011). Acoustic temporal modulation detection and speech perception in cochlear implant listenersa). The Journal of the Acoustical Society of America, 130(1), 376. https://doi.org/10.1121/1.3592521

71.

Zheng

Escabí

Litovsky

R. Y

. (2017). Spectro-Temporal cues enhance modulation sensitivity in cochlear implant users. Hearing Research, 351, 45–54. https://doi.org/10.1016/j.heares.2017.05.009

72.

Zhou

Dixon

Zhu

Dong

Weiner

. (2020). Spectrotemporal modulation sensitivity in cochlear-implant and normal-hearing listeners: Is the performance driven by temporal or spectral modulation sensitivity? Trends in Hearing, 24. https://doi.org/10.1177/2331216520948385

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

2.61 MB