Deep learning for low-resource speech recognition : a thesis presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Computer Science, School of Mathematical and Computational Sciences, Massey University, Albany, Auckland, New Zealand

Abstract

Low-resource Automatic Speech Recognition (ASR) focuses on transcribing speech signals into text for low-resource languages, which suffer from significantly limited linguistic resources (e.g., corpora, dictionaries). Many of these low-resource languages are also endangered languages. Developing deep learning-based low-resource ASR systems presents significant challenges due to the scarcity of transcribed speech data and the lack of diverse and representative speakers. These limitations hinder the generalization to out-of-domain speech, making advancements in this area critical. In this thesis, we address these challenges through four research directions: data augmentation, continual learning, text-to-speech (TTS) synthesis, and synthesized data selection.

Firstly, we propose CyclicAugment, a novel approach designed to enhance generalization in low-resource ASR systems. By dynamically transforming speech data using a cosine annealing scheduler, CyclicAugment assists escape local optima during training, improving model robustness and performance.

Secondly, we introduce Randomly Layer-wise Tuning (CLRL-Tuning), an innovative continual learning method for pre-trained ASR models. CLRL-Tuning incrementally fine-tunes randomly selected layers of the model when trained on sequentially collected datasets. This approach effectively mitigates catastrophic forgetting and enhances generalization in low-resource ASR systems.

Thirdly, we develop Zero-Voice, a zero-shot TTS model comprising an Acoustic Feature Encoder, an Acoustic Feature Refiner, and a Waveform Vocoder. Trained on 27 hours of Te Reo Māori speech data, an official and endangered language of New Zealand. Zero-Voice synthesizes high-fidelity audio from reference speech samples. This enables the generation of additional Te Reo Māori speech data, significantly enhancing ASR performance for the language.

Lastly, to address the domain mismatch between true and synthesized Te Reo Māori data generated by the Zero-Voice model, we propose a score-distribution-matching data selection approach. By aligning the score distribution of synthesized data with the prior distribution of true data, this method effectively filters synthesized data, optimizing ASR performance for Te Reo Māori and advancing the state-of-the-art for low-resource ASR.

Through these contributions, our research addresses the key challenges in low-resource ASR research. By integrating these innovations, we achieve state-of-the-art performance in Te Reo Māori ASR.