How to stop a text-classification model from depending on only couple of the words from input text instead of entire sentence?

I have a text classification deep-learning model, which takes in a text and outputs a softmax probability. I am using glove embeddings to represent my input text in numerical form for the DL model.

the DL model is actually quite simple too. the embedding layer is trainable and no weights has been passed to it.

And after the training, while I was making predictions with unseen text, I could realise that only one of the key-words had huge importance in predictions. To make sure that that is the case, I have changed input texts to only include those key-words and softmax-probabilites were more or less same.

So, what changes do I make in my model, from embedding-technique to network, or in anyway to not have the model rely on some of the words and force it to consider entire sentence?

Topic explainable-ai text-classification deep-learning text-mining machine-learning

Category Data Science

1
answered at

Dense layer | Feature in-dependency

There are a couple of things that may be going on in your experiment. First of all, it may as well be that the particular keyword you are seeing to play the most important role is intrinsically correlated with a correct prediction.

That aside, by feeding the network words in the form of a dense layer, these words are introduced with no intrinsic inter-dependency and are treated as orthogonal in the forward pass.

RNN layer | Feature step-wise dependency

force it to consider entire sentence

I think what you may be saying is that, some there is some "sense" in the ordering or temporal structure of your text, and that you would like to impose that as an intrinsic characteristic of your data as it is being passed over to your network. The above is exactly what a recurrent layer is meant to be doing: impose a step-wise dependency between the words/features as they take place in time and only allow your data to be seen in this particular recurrence structure. In this way, a fixed-length representation of your target/input sequence is downprojected into the hidden state vector of the last timestep of your sequence.

Your model will get adjusted based on this particular last-hidden-state distillation of your input texts, and thus enforced to look at the entirety of sentences by proxy.

If you try this, beware of vanishing gradients ;)

1
answered at

Adding Dropout, randomly omitting units during training, encourages a neural network to learn a more robust representation.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.