Adding words to vocabulary on pre-trained ASR model

I have a pre-trained ASR model but want to add some missing words to the vocabulary. Can I do this or will it invalidate the entire training?

Lets say I use the pretrained model: wav2vec2-base-960h and want to use it on sports commentary but a lot of the players' names are missing in the vocabulary.

Is there any way I can add the names and maybe train on a few clips where the names appear or do I have to train everything from scratch?