The speaker diarization task pertains to the automated differentiation of speakers within an audio recording, while lacking any prior information regarding the speakers. The introduction of the self-attention mechanism in End-to-End Neural Speaker Diarization (EEND) has elegantly resolved the issue of overlapping speakers. The Transformer model equipped with self-attention mechanism has shown great potential in collecting global information, yielding remarkable outcomes in various tasks. However, the individual speaker characteristics are predominantly reflected in the contextual information, which conventional self-attention would not adequately address. In this study, we propose a hierarchical encoders model to augment the encoders’ acquisition of speaker information in two distinct ways: (1) Constraining the perceptual field of the self-attentive mechanism with left-right windows or Gaussian weights to highlight contextual information; (2) Utilizing a pre-trained time-delay neural network based speaker embedding extractor to alleviate the shortcomings of speaker feature extraction ability. We evaluate the proposed methods on a simulated dataset of two speakers and a real conversation dataset. The model with the most favorable outcomes among the proposed enhancements achieves a diarization error rate of 7.74% on the simulated dataset and 21.92% on MagicData-RAMC after adaptation. These results compellingly demonstrate the efficacy of the proposed methods.
ParkT.J., et al., A review of speaker diarization: Recent advances with deep learning, Computer Speech & Language72 (2022), 101317.
2.
KandaN., et al., Acoustic modeling for distant multi-talker speech recognition with single-and multi-channel branches, in ICASSP 2019–2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, IEEE.
3.
ShafeyL.E., SoltauH. and ShafranI., Joint speech recognition and speaker diarization via sequence transduction[J], arXiv preprint arXiv:1907.05337, 2019.
4.
KandaN., HoriguchiS., FujitaY., et al., Simultaneous speech recognition and speaker diarization for monaural dialogue recordings with target-speaker acoustic models[C], 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), IEEE, 2019:31–38.
5.
LandiniF., et al., Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: Theory, implementation and analysis on standard tasks, Computer Speech & Language71 (2022), 101254.
6.
PrasadA. and Rajeswara RaoR., Metaheuristic adapted convolutional neural network for Telugu speaker diarization[J], Intelligent Decision Technologies15(4) (2021), 561–577.
7.
WangQ., et al., Speaker diarization with LSTM, in2018 IEEE International conference on acoustics, speech and signal processing (ICASSP), 2018. IEEE.
8.
FujitaY., et al., End-to-End Neural Speaker Diarization with Self-Attention, in2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2019.
9.
HoriguchiS., et al., End-to-end speaker diarization for an unknown number of speakers with encoder-decoder based attractors, arXiv preprint arXiv:2005.09921, 2020.
10.
LiuY.C., et al., End-to-end neural diarization: From transformer to conformer, arXiv preprint arXiv:2106.07167, 2021.
11.
ZhangX.-L. and XuM., AUC optimization for deep learning-based voice activity detection, EURASIP Journal on Audio, Speech, and Music Processing2022(1) (2022), 1–12.
12.
MedennikovI., et al., Target-speaker voice activity detection: a novel approach for multi-speaker diarization in a dinner party scenario, arXiv preprint arXiv:2005.07272, 2020.
13.
KimJ. and HahnM., Voice activity detection using an adaptive context attention model, IEEE Signal Processing Letters25(8) (2018), 1181–1185.
14.
WangW. and LiM., Incorporating End-to-End Framework Into Target-Speaker Voice Activity Detection, in2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
15.
DelacourtPerrine, KryzeDavid and WellekensChristian J., Speaker-based segmentation for audio data indexing, ESCA Tutorial and Research Workshop (ETRW) on Accessing Information in Spoken Audio, 1999.
16.
MalegaonkarA., et al., Unsupervised speaker change detection using probabilistic pattern matching, 13(8) (2006), 509–512.
17.
DehakN., et al., Front-end factor analysis for speaker verification, IEEE Transactions on Audio, Speech, and Language Processing19(4) (2010), 788–798.
18.
GałkaJ. and JaciówP., Speaker identification using fuzzy I-vector tree[J], Journal of Intelligent & Fuzzy Systems37(4) (2019), 4937–4949.
19.
SaonG., et al., Speaker adaptation of neural network acoustic models using i-vectors, in2013 IEEE Workshop on Automatic Speech Recognition and Understanding, 2013.
20.
Garcia-RomeroD., et al., Speaker diarization using deep neural network embeddings, in2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.
21.
SnyderD., et al., Deep neural network-based speaker embeddings for end-to-end speaker verification, in2016 IEEE Spoken Language Technology Workshop (SLT), 2016.
22.
SnyderD., et al., X-Vectors: Robust DNN Embeddings for Speaker Recognition, in2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018.
23.
ZhangY., SongX. and ZhangJ., Speaker diarization for multi-speaker conversations via x-vectors, in2019 IEEE International Conference on Signal, Information and Data Processing (ICSIDP), 2019.
24.
J. Luque and J. Hernando, On the use of agglomerative and spectral clustering in speaker diarization of meetings, 2012.
25.
Q. Lin, et al., LSTM based similarity measurement with spectral clustering for speaker diarization, 2019.
26.
K.J. Han and S.S. Narayanan, A robust stopping criterion for agglomerative hierarchical clustering in a speaker diarization system, 2007.
27.
J.E. Rougui, et al., Fast incremental clustering of gaussian mixture speaker models for scaling up retrieval in on-line broadcast, 2006. IEEE.
28.
S. Novoselov, et al., Speaker diarization with deep speaker embeddings for DIHARD challenge II, 2019.
29.
RyantN., et al., The first DIHARD speech diarization challenge, Proceedings of Interspeech 2018, 2018.
30.
RyantN., et al., The second dihard diarization challenge: Dataset, task, and baselines, arXiv preprint arXiv:1906.07839, 2019.
31.
WatanabeS., et al., CHiME-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings, 2020.
32.
VaswaniA., et al., Attention is All You Need, Nips’2017 (17), p. 6000–6010.
33.
J. Devlin, et al., Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.
34.
LiuZe, et al., Swin transformer: Hierarchical vision transformer using shifted windows, Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
35.
DongLinhao, XuShuang and XuBo, Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, A. Gulati, et al., Conformer: Convolution-augmented transformer for speech recognition, arXiv preprint arXiv:2005.08100, 2020.
36.
LinZ., et al., A structured self-attentive sentence embedding,, arXiv preprint arXiv:1703.03130, 2017.
37.
BeltagyI., PetersM.E. and CohanA., Longformer: The long-document transformer, arXiv preprint arXiv:2004.05150, 2020.
38.
GuoMaosheng, ZhangYu and LiuTing, Gaussian transformer: a lightweight approach for natural language inference, Proceedings of the AAAI Conference on Artificial Intelligence33(01) (2019).
39.
DawalatabadN., et al., ECAPA-TDNN embeddings for speaker diarization, arXiv preprint arXiv:2104.01466, 2021.
40.
DesplanquesB., ThienpondtJ. and DemuynckK., Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification, arXiv preprint arXiv:2005.07143, 2020.
41.
YangZ., et al., Open Source MagicData-RAMC: A Rich Annotated Mandarin Conversational (RAMC) Speech Dataset, arXiv preprint arXiv:2203.16844, 2022.
42.
SnyderD., ChenG. and PoveyD., Musan: A music, speech, and noise corpus, arXiv preprint arXiv:1510.08484, 2015.
43.
KoTom, et al., A study on data augmentation of reverberant speech for robust speech recognition, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2017.
44.
KingmaDiederik P. and BaJimmy, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).
45.
SeelandM. and MäderP., Multi-view classification with convolutional neural networks[J], Plos One16(1) (2021), e0245230.
46.
GuarinoA., LettieriN., MalandrinoD., et al., Adam or Eve? Automatic users’ gender classification via gestures analysis on touch devices[J], Neural Computing and Applications34(21) (2022), 18473–18495.
47.
ZaccagninoR., CapoC., GuarinoA., et al., Techno-regulation and intelligent safeguards: Analysis of touch gestures for online child protection[J], Multimedia Tools and Applications80 (2021), 15803–15824.
