How does an RNN differ from a CBOW model

CBOW: We are trying to predict the next word based on the context (defined as a certain window of words around the target word)

RNN can also be used for predicting the next word in a sequence, where each time the input is the present input and the recent past (i.e. output of the previous step)

I am not able to understand how the RNN's approach is somehow better, because I could define a very large window for CBOW, and then it would also predict words based on previous information. Why do we say RNN will be better at predicting because it has memory? If we supply everything to CBOW at the same time, which is given to RNN in steps, why is one better than the other? Is it only because we save on computation in the latter case?

Topic prediction rnn nlp

Category Data Science

1
answered at

RNNs can use the hidden state to store relevant history in its hidden state and use it to make a prediction of what word can follow. This makes RNNs potentially very good language models, i.e., models of distributions over sentences.

CBOW only considers a window of few surrounding words regardless of their ordering, which is, however, crucial information for language modeling. CBOW is, therefore, a pretty bad language model, but language modeling is not what the model was designed for. Although it is a pretty poor language model, it provides a similarly good training signal for word embeddings. Because it is much faster than RNN, the embeddings can be trained on much larger data. Also, the model design allows further optimization for large vocabulary sized.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.