NeMo Conformer-CTC Predicts Same Word Repeatedly When Fine-Tuning

I'm using the NeMo Conformer-CTC small on the LibriSpeech dataset (the clean subset, around 29K inputs, using 90% for training and 10% for testing). I use Pytorch Lightning.

When I try to train, the model learns 1 or 2 sentences in 50 epochs and gets stuck at a loss of 60-something (I trained it for 200 epochs too and it didn't budge). But when I try to fine tune it using a pre-trained model from the toolkit, it predicts correctly on the Validation Sanity Check and then when it starts training it predicts the same word or couple words repeatedly and the loss keeps increasing until it hits 3e+07 and becomes nan.

I had the same result after changing the learning rate and using another dataset (VCTK). I tried to do the same with another model (quartznet) and it worked fine.

Anybody knows what could be going on?

Thank you!