results: 在 official challenge 结果中，我们的系统在 cross-domain 任务中表现出色，得分第1和第2位，分别在自然性和相似性两个指标上。此外，我们还进行了一些缓解分析，以证明我们的系统设计的有效性。Abstract
This paper presents the T02 team's system for the Singing Voice Conversion Challenge 2023 (SVCC2023). Our system entails a VITS-based SVC model, incorporating three modules: a feature extractor, a voice converter, and a post-processor. Specifically, the feature extractor provides F0 contours and extracts speaker-independent linguistic content from the input singing voice by leveraging a HuBERT model. The voice converter is employed to recompose the speaker timbre, F0, and linguistic content to generate the waveform of the target speaker. Besides, to further improve the audio quality, a fine-tuned DSPGAN vocoder is introduced to re-synthesise the waveform. Given the limited target speaker data, we utilize a two-stage training strategy to adapt the base model to the target speaker. During model adaptation, several tricks, such as data augmentation and joint training with auxiliary singer data, are involved. Official challenge results show that our system achieves superior performance, especially in the cross-domain task, ranking 1st and 2nd in naturalness and similarity, respectively. Further ablation justifies the effectiveness of our system design.
摘要
Partial Rank Similarity Minimization Method for Quality MOS Prediction of Unseen Speech Synthesis Systems in Zero-Shot and Semi-supervised setting
results: 实验表明，PRS在零shot和半supervised设定下表现出色，与真实值更高度相关，而MSE和linear correlation coefficient metric可能不适用于评估MOS预测模型。Abstract
This paper introduces a novel objective function for quality mean opinion score (MOS) prediction of unseen speech synthesis systems. The proposed function measures the similarity of relative positions of predicted MOS values, in a mini-batch, rather than the actual MOS values. That is the partial rank similarity is measured (PRS) rather than the individual MOS values as with the L1 loss. Our experiments on out-of-domain speech synthesis systems demonstrate that the PRS outperforms L1 loss in zero-shot and semi-supervised settings, exhibiting stronger correlation with ground truth. These findings highlight the importance of considering rank order, as done by PRS, when training MOS prediction models. We also argue that mean squared error and linear correlation coefficient metrics may be unreliable for evaluating MOS prediction models. In conclusion, PRS-trained models provide a robust framework for evaluating speech quality and offer insights for developing high-quality speech synthesis systems. Code and models are available at github.com/nii-yamagishilab/partial_rank_similarity/
摘要
这份论文介绍了一种新的评价函数，用于预测未看过的语音合成系统的质量 mean opinion score（MOS）。提出的函数测量在一个小批次中预测的MOS值相对位置的相似性，而不是实际的MOS值。即使使用了partial rank similarity（PRS）而不是L1损失，我们的实验表明，PRS在零批次和半指导学习 Setting 中表现更好，与基准数据 exhibit stronger correlation。这些发现反映了考虑 rank order 的重要性，当训练 MOS 预测模型时。我们还认为 mean squared error 和 linear correlation coefficient metrics 可能不可靠地评价 MOS 预测模型。 conclusion，PRS 训练的模型提供了一种robust的 speech quality 评价框架，并且为开发高质量语音合成系统提供了意见。代码和模型可以在github.com/nii-yamagishilab/partial_rank_similarity/ 找到。
SALT: Distinguishable Speaker Anonymization Through Latent Space Transformation
results: 在 Voice Privacy Challenge 数据集上，我们的系统实现了最佳的匿名度指标，同时保持语音质量和可理解性。Abstract
Speaker anonymization aims to conceal a speaker's identity without degrading speech quality and intelligibility. Most speaker anonymization systems disentangle the speaker representation from the original speech and achieve anonymization by averaging or modifying the speaker representation. However, the anonymized speech is subject to reduction in pseudo speaker distinctiveness, speech quality and intelligibility for out-of-distribution speaker. To solve this issue, we propose SALT, a Speaker Anonymization system based on Latent space Transformation. Specifically, we extract latent features by a self-supervised feature extractor and randomly sample multiple speakers and their weights, and then interpolate the latent vectors to achieve speaker anonymization. Meanwhile, we explore the extrapolation method to further extend the diversity of pseudo speakers. Experiments on Voice Privacy Challenge dataset show our system achieves a state-of-the-art distinctiveness metric while preserving speech quality and intelligibility. Our code and demo is availible at https://github.com/BakerBunker/SALT .
摘要
PromptSpeaker: Speaker Generation Based on Text Descriptions
results: 研究证明PromptSpeaker可以生成与训练集外的新发音人，并且synthetic speaker voice具有相对合理的主观匹配质量。Abstract
Recently, text-guided content generation has received extensive attention. In this work, we explore the possibility of text description-based speaker generation, i.e., using text prompts to control the speaker generation process. Specifically, we propose PromptSpeaker, a text-guided speaker generation system. PromptSpeaker consists of a prompt encoder, a zero-shot VITS, and a Glow model, where the prompt encoder predicts a prior distribution based on the text description and samples from this distribution to obtain a semantic representation. The Glow model subsequently converts the semantic representation into a speaker representation, and the zero-shot VITS finally synthesizes the speaker's voice based on the speaker representation. We verify that PromptSpeaker can generate speakers new from the training set by objective metrics, and the synthetic speaker voice has reasonable subjective matching quality with the speaker prompt.
摘要
近些时间，文本指导内容生成已经受到了广泛关注。在这项工作中，我们探索了使用文本描述来控制发音生成过程的可能性。具体来说，我们提出了PromptSpeaker，一种文本指导的发音生成系统。PromptSpeaker包括一个描述符编码器、一个零拟合VITS和一个Glow模型，其中描述符编码器根据文本描述预测一个优先分布，并从这个分布中采样以获取一个semantic表示。Glow模型然后将semantic表示转化为发音表示，零拟合VITS最后将发音表示转化为声音。我们证明了PromptSpeaker可以新生成不同于训练集的发音，并且synthetic声音具有合理的主观匹配质量与发音描述。