Naives Bayes Text Classifier Confidence Score

Question

Naives Bayes Text Classifier Confidence Score

Jim

2022年6月3日 20:06

I am experimenting with building a text classifier using Naive Bayes which has been pretty successful on my test data. One thing i am looking to incorporate is handling text that does not fit into any predefined category that I trained the model on.

Does anyone have some thoughts on how to do this? I was thinking of trying to calculate the confidence score for each document, and if 80 % confidence, for example, it should label the data as N/A

This is my code so far:

df_train = pd.read_csv(__________)

text_clf = Pipeline([('vect', CountVectorizer()),
                  ('tfidf', TfidfTransformer()),
                  ('clf', MultinomialNB()),])
text_clf = text_clf.fit(df_train.text, df_train.label)

df['predicted'] = predicted

Like I said, it works well for documents that do fit into one of the categories, but if I have something that clearly does not fit into anything, it will still try and assign it a label, my guess is based on some kind of confidence calculation but just not sure how that works

Topic naive-bayes-classifier nlp python

Category Data Science

Erwan · Accepted Answer · 2021年4月16日 22:38

First there is a simple theoretical reason why relying on the probability provided by a NB model is not a good idea: what NB predicts is the posterior probability, i.e. the conditional probability $p(C_i|d)$ for every class $C_i$ for a given document $d$. The class $C$ which is predicted is just the one which obtains the highest probability $p(C_i|d)$. The sum of all the posterior probabilities for a particular $d$ is 1 since it's a conditional, and this means that a class is always predicted relatively to the other classes: a high probability for $C$ doesn't mean that the document is very likely to be $C$ in general, it only means that it is much more likely to be $C$ than any of the other classes. If it actually belongs to another class unknown to the model, there's no way the model can represent this.

There is also a practical reason why this is not a great idea: experimentally it is known that NB tends to systematically assign extreme probabilities (and to overfit easily but that's not the point). It does this whether or not the class is correct, and this can easily be checked with your data: if you plot the predicted probability for every instance with a different colour for correct/incorrect prediction, you're likely to see that even the incorrect instances were predicted with a very high probability. It's tempting to interpret the predicted probability as a confidence measure, but it's usually a mistake.

But there is hope: the proper way to deal with potential unknown "classes" (these are not classes technically) is not with regular classification methods, which always rely on a "closed world" assumption (only the classes seen during training exist). There are also open-set classification methods: these are much less common but they are not restricted to the known classes. I think the most common method is probably one-class classification, in particular with one-class SVM.

Naives Bayes Text Classifier Confidence Score

About