cs.SD - 2023-07-10

  • paper_url: http://arxiv.org/abs/2307.05588
  • repo_url: None
  • paper_authors: Michèle Duguay, Kate Mancey, Johanna Devaney
  • for: 本研究使用的 CoSoD 数据集包含 331 首多个艺人合作的歌曲,来自 2010-2019 年 Billboard “Hot 100” 年度榜单。
  • methods: 本研究使用的 annotation 方法包括 formal sections, 声音生成方法(包括延迟、层次、扬声和performer 的性别)以及相关的 metadata。
  • results: 研究发现,男性艺人的 vocals 通常具有 menos 的延迟和更窄的位置在声音混合中,与女性艺人的 vocals 相比。
    Abstract The Collaborative Song Dataset (CoSoD) is a corpus of 331 multi-artist collaborations from the 2010-2019 Billboard "Hot 100" year-end charts. The corpus is annotated with formal sections, aspects of vocal production (including reverberation, layering, panning, and gender of the performers), and relevant metadata. CoSoD complements other popular music datasets by focusing exclusively on musical collaborations between independent acts. In addition to facilitating the study of song form and vocal production, CoSoD allows for the in-depth study of gender as it relates to various timbral, pitch, and formal parameters in musical collaborations. In this paper, we detail the contents of the dataset and outline the annotation process. We also present an experiment using CoSoD that examines how the use of reverberation, layering, and panning are related to the gender of the artist. In this experiment, we find that men's voices are on average treated with less reverberation and occupy a more narrow position in the stereo mix than women's voices.
    摘要 《合作歌曲数据集(CoSoD)》是2010-2019年度 Billboard "Hot 100" 年度榜单中331首多位艺人合作的数据集。该数据集已经被标注了正式部分、声乐生产方面的属性(包括延迟、层次、扬声和performer的性别)以及相关的元数据。CoSoD 与其他流行音乐数据集相比,专注于独立艺人之间的音乐合作。此外,CoSoD 还允许对歌曲结构和声乐生产进行深入研究,以及对 gender 与不同的时间、抽象和形式参数之间的关系进行深入研究。在这篇文章中,我们详细介绍了数据集的内容和标注过程,以及使用 CoSoD 进行了一个实验,发现男性艺人的声音在延迟方面比女性艺人的声音更为低,并且男性艺人的声音在立体混合中占据更窄的位置。

The NPU-MSXF Speech-to-Speech Translation System for IWSLT 2023 Speech-to-Speech Translation Task

  • paper_url: http://arxiv.org/abs/2307.04630
  • repo_url: None
  • paper_authors: Kun Song, Yi lei, Peikun Chen, Yiqing Cao, Kun Wei, Yongmao Zhang, Lei Xie, Ning Jiang, Guoqing Zhao
  • for: 这个论文旨在提出一个基于自动语音识别(ASR)、机器翻译(MT)和文本到语音(TTS)的语音到语音翻译(S2ST)系统,用于从英语语音多源到中文语音的翻译。
  • methods: 这个系统采用了多种数据增强策略和ROVER基于多个ASR模型输出的分数融合,以提高多源语音输入的Robustness。此外,为了更好地处理含杂ASR转文本的问题,我们提出了三个阶段精度调整策略。
  • results: 实验结果显示,我们的系统在翻译准确率、语音自然度、声音质量和 speaker相似性方面具有优秀的表现,同时也在多源数据下表现良好的Robustness。
    Abstract This paper describes the NPU-MSXF system for the IWSLT 2023 speech-to-speech translation (S2ST) task which aims to translate from English speech of multi-source to Chinese speech. The system is built in a cascaded manner consisting of automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS). We make tremendous efforts to handle the challenging multi-source input. Specifically, to improve the robustness to multi-source speech input, we adopt various data augmentation strategies and a ROVER-based score fusion on multiple ASR model outputs. To better handle the noisy ASR transcripts, we introduce a three-stage fine-tuning strategy to improve translation accuracy. Finally, we build a TTS model with high naturalness and sound quality, which leverages a two-stage framework, using network bottleneck features as a robust intermediate representation for speaker timbre and linguistic content disentanglement. Based on the two-stage framework, pre-trained speaker embedding is leveraged as a condition to transfer the speaker timbre in the source English speech to the translated Chinese speech. Experimental results show that our system has high translation accuracy, speech naturalness, sound quality, and speaker similarity. Moreover, it shows good robustness to multi-source data.
    摘要

Exploiting an External Microphone for Binaural RTF-Vector-Based Direction of Arrival Estimation for Multiple Speakers

  • paper_url: http://arxiv.org/abs/2307.04460
  • repo_url: None
  • paper_authors: Daniel Fejgin, Simon Doclo
  • for: 减少计算复杂性的听力器 DOA估计方法
  • methods: 使用外部麦克风和听力器麦克风的低相关性假设进行RTF向量估计,并使用covariance whitening方法获取RTF向量
  • results: 在听力器麦克风和外部麦克风的听音环境中,提出了一种低计算复杂性的DOA估计方法,并且在两个说话人和杂音环境下获得了相当于CW方法的DOA估计性能,而且具有较低的计算复杂性。
    Abstract In hearing aid applications, an important objective is to accurately estimate the direction of arrival (DOA) of multiple speakers in noisy and reverberant environments. Recently, we proposed a binaural DOA estimation method, where the DOAs of the speakers are estimated by selecting the directions for which the so-called Hermitian angle spectrum between the estimated relative transfer function (RTF) vector and a database of prototype anechoic RTF vectors is maximized. The RTF vector is estimated using the covariance whitening (CW) method, which requires a computationally complex generalized eigenvalue decomposition. The spatial spectrum is obtained by only considering frequencies where it is likely that one speaker dominates over the other speakers, noise and reverberation. In this contribution, we exploit the availability of an external microphone that is spatially separated from the hearing aid microphones and consider a low-complexity RTF vector estimation method that assumes a low spatial coherence between the undesired components in the external microphone and the hearing aid microphones. Using recordings of two speakers and diffuse-like babble noise in acoustic environments with mild reverberation and low signal-to-noise ratio, simulation results show that the proposed method yields a comparable DOA estimation performance as the CW method at a lower computational complexity.
    摘要 在听觉器应用中,一个重要的目标是准确估计多个说话人的方向来达(DOA)在噪音和反射环境中。最近,我们提出了一种双耳DOA估计方法,其中DOA的估计是通过选择RTFVector的 Hermitian角spectrum与一个数据库中的prototype静音RTFVector之间的最大匹配来实现。RTFVector是通过covariance whitening(CW)方法来估计,该方法需要计算复杂的普通值归一化分解。 spatial spectrum是通过只考虑可能有一个说话人在其他说话人、噪音和反射中占据主导地位的频率来获得。在这种贡献中,我们利用了一个外部麦克风,该麦克风与听觉器麦克风之间存在低空间协方差,并考虑了一种低复杂度RTFVector估计方法,该方法假设外部麦克风和听觉器麦克风之间的不desired ком分布具有低空间协方差。通过对两个说话人和杂音噪音的记录在带有柔和反射的Acoustic环境中进行模拟,结果表明,我们的方法可以与CW方法具有相同的DOA估计性能,但具有较低的计算复杂度。

HCLAS-X: Hierarchical and Cascaded Lyrics Alignment System Using Multimodal Cross-Correlation

  • paper_url: http://arxiv.org/abs/2307.04377
  • repo_url: None
  • paper_authors: Minsung Kang, Soochul Park, Keunwoo Choi
  • for: 本研究targets the challenge of lyrics alignment, which involves aligning the lyrics and vocal components of songs.
  • methods: 我们提出了一种基于supervised learning的模型,利用歌词和音频模态之间的cross-correlation矩阵来适应这个挑战。我们的模型采用了层次结构和顺序串行的设计,首先在句子级别预测同步时间,然后在词语级别预测同步时间。
  • results: 我们的提议模型在比较 experiment中显示了significant improvement in mean average error,并且在 deploying 在多个音乐流媒体服务上的实践中也得到了良好的结果。
    Abstract In this work, we address the challenge of lyrics alignment, which involves aligning the lyrics and vocal components of songs. This problem requires the alignment of two distinct modalities, namely text and audio. To overcome this challenge, we propose a model that is trained in a supervised manner, utilizing the cross-correlation matrix of latent representations between vocals and lyrics. Our system is designed in a hierarchical and cascaded manner. It predicts synced time first on a sentence-level and subsequently on a word-level. This design enables the system to process long sequences, as the cross-correlation uses quadratic memory with respect to sequence length. In our experiments, we demonstrate that our proposed system achieves a significant improvement in mean average error, showcasing its robustness in comparison to the previous state-of-the-art model. Additionally, we conduct a qualitative analysis of the system after successfully deploying it in several music streaming services.
    摘要 在这个工作中,我们面临着歌词对齐挑战,即将歌词和声音组件对齐。这个问题需要对两种不同的modalities进行对齐,即文本和音频。为了解决这个挑战,我们提议一种在有监督下训练的模型,使用歌词和声音的横排相关矩阵来训练。我们的系统设计为层次和分层的结构,首先在句子水平预测同步时间,然后在单词水平进行精度调整。这种设计使得系统能够处理长序列,因为交叉相关使用了序列长度的二次memory。在我们的实验中,我们示出了我们提议的系统在平均误差中具有显著的改善,这反映了它在比前一个状态艺术模型时更加稳定。此外,我们还进行了音乐流服务中成功部署后的系统Qualitative分析。