ngram and RNN prediction rate wrt word index

I tried to plot the rate of correct predictions (for the top 1 shortlist) with relation to the word's position in sentence :

I was expecting to see a plateau sooner on the ngram setup since it needless context. However, one thing I wasn't expecting was that the prediction rate drops. In my understanding since we already have a context of 3 words, the plateau should converge asymptotically to its highest value. But both the recurrent network and the Ngram models are experiencing this drop. I have no idea why it would be.

(Note RNNLM is the name of the framework used to build the recurrent neural net, it uses 500 neurons and 100M direct connections, RNN25 is the same setup but with a training base divided by for)

Here is the sentence size distribution :

Thanks in advance.

Topic ngrams rnn neural-network nlp statistics

Category Data Science

1
answered at

Recurrent Neural Network (RNN) create a single state vector over time. Thus that curve is to be expected. Initially, the state vector does have enough information to make a quality prediction. Then quickly reaches asymptotic performances. Overall, the predictions are between 15% and 22% correct.

The shape of the graph might be a function of sentence length in the training corpus. Possibly sentences could be between 3 and 7 words long. The drop could be because there is less training data for longer sentences.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.