Interpreting evaluation metrics with threshold/cutoff

I was doing churn prediction for a company. I've got the following results by applying 3 classifier.

Model Accuracy AUC
Logistic Regression 0.671 0.736
Decision Tree (pruned) 0.681 0.665
Decision Tree unpruned 0.623 0.627

Now, I want to know two things:

  • which model has a better accuracy for a cutoff of 0.9?

As the logistic regression has highest AUC so, in my opinion, Logistic Regression is better

  • Which model is the best in terms of ranking the predictions according to their probability of leaving

Can anyone explain how I can interpret them?

Topic auc weka evaluation data-mining machine-learning

Category Data Science

1
answered at
  • The accuracy is likely to go down if you change the cutoff point to 0.9, since any model tries to separate the classes so that the probability of the correct class is higher than 0.5. But the only way to know would be to actually do the experiment (I assume that the results that you show are obtained with the default cutoff).
  • AUC is a complex measure for a soft classifier, i.e. it doesn't use any cutoff point but provides a performance value across cutoff points. Importantly, the AUC score considers one class as positive, like the precision, recall and F1-score measures. Btw I would suggest looking at these scores instead, which are more precise than accuracy and more easily interpretable than AUC.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.