Is it possible feed BERT to seq2seq encoder/decoder NMT (for low resource language)?

I'm working on NMT model which the input and the target sentences are from the same language (but the grammar differs). I'm planning to pre-train and use BERT since I'm working on small dataset and low/under resource language. so is it possible to feed BERT to the seq2Seq encoder/decoder?

Topic bert sequence-to-sequence deep-learning machine-translation machine-learning

Category Data Science

1
answered at

Sure, why not? An encoder/decoder is basically agnostic to the format of the token vectors, whether they be derived via Word2Vec, BERT, GPT2, etc.

The more challenging aspect of this, should you not have figured it out already, might be finding a pretrained embedding model for your low resource language. Given a small dataset, training your own from scratch seems infeasible. You could potentially find a BERT pretrained over a similar language (e.g. within the same family or grammatical structure) and fine-tune the last layer according to your dataset. Conventional BERT is trained over the English language, with some BERTs spinning off in other languages.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.