Keras most important features for text classification

I am working on a problem where I need to classify phrases in one of the two categories (let's A B). I used the Keras SepCNN model (similar to this) for that and it is giving me some results.

Now, I want to analyse the predictions and more specifically I want to know why the model classified a certain phrase in category A or B, which set of features played an important role in labeling that phrase as category A or B?

I am not sure if this is even possible to do but need inputs on how to approach this.

#----Model Structure for your reference--
Vectorize text (tfidf)  
SelectKBest features (using scikit-learn feature_extraction)  
Fit the model  
Predict on test set*

I looked on the internet and found nothing helpful. It would be very helpful if someone can point me the right direction. I also explored eli5 library but that does not seem to work for Keras SepCNN model (at-least I could not get it to work, let me know if anyone has used successfully it?)