Confidence score for all observations is between 0.50 - 0.55

Hello Data Science Stack Exchange Community,

This question will appear to be open-ended, however any answers or thought will be much appreciated. I am trying to go-through a pre-trained Random Model Classifier with minimum documentation like what was the confusion matrix, ROC-AUC curve of the classification problem when the model was developed. I only have the pickle file and the data set on which it needs to run. When I ran the model I observed for most of the cases the prediction score or confidence score is between 0.50 to 0.55

I would like to know is it right to say that model might have learn the pattern even tough all the confidence score is between 0.50 and 0.55

Because I would like to explain that since the all the predictions score are concentrated around a single value it is not right to say that the model learnt and probably if some one would have tried to plot AUC it would have been straight line.

Again any thoughts and answers will be much appreciated.

Topic confidence machine-learning-model random-forest scikit-learn

Category Data Science

1
answered at

Random Model Classifier

It feels like there's a word missing here, you probably mean "Random Forest [model] classifier" or "Random Fields classifier"?

the confusion matrix

This is an important piece of information, because if you have a new (annotated) sample this allows you to compare a few things:

  • compare the distribution of the classes in the original test data and the new dataset
  • calculate the expected performance by class, analyze the differences if you evaluate the model against the new sample
  • ...

I would like to know is it right to say that "model might have learn the pattern even tough all the confidence score is between 0.50 and 0.55"

It would be technically correct to say that indeed, but maybe not for the reasons that you mention (see below). While this predicted value is often called a confidence score or probability, it doesn't necessarily represents the confidence of the model in a normalized range [0,1] like one would expect, even though ideally it's supposed to. The details depend on the exact type of model, but in general in ML the model is only as good as the data it's trained with so most of the time the predicted score is just the "best guess" of the model. So it shouldn't be interpreted as "probability of class C", and anyway in the case of classification the only use of the value is to compare two instances x and y relatively to each other: if x has a higher predicted score than y, then the model considers x more likely than y to be in class C. The actual value doesn't matter and it's better not to rely on it.

Because I would like to explain that since the all the predictions score are concentrated around a single value it is not right to say that the model learnt and probably if some one would have tried to plot AUC it would have been straight line.

This conclusion is incorrect. To be clear, it might be true in some particular case, but one cannot conclude this from the information given in the question. Why? Because the ROC curve (and the AUC) don't depend at all on the actual values of the score, they depend only on the order of the instances according to the score. The ROC curve shows the performance for different theresholds separating the predicted positive/negative instances. Whether the scores range in [0.51,0.52], [0,1] or even [-1234,+5678] doesn't have any influence, it's the proportion of negative instances under the threshold and positive ones above it which determine the performance at any given point.

In case you're interested, I gave some more detailed explanations about ROC curves in this answer.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.