Custom thresholds on categorical classification

Question

Custom thresholds on categorical classification

nprime496

2021年4月6日 20:21

When assessing a binary classification task, it is possible to search for particular threshold in order to have better score on some metrics (f1,recall,etc) through numerous methods. Unfortunately, it looks like the method cannot be applied on categorical classification (more than two classes) task. I've thought about training a Simple Classifier (SVC,Log,...,Tree) on top of a already trained model in order to find best thresholds to apply on outputs to maximize similarity of results.

My proposed workflow is to train a model on training set and evaluate it, then use outputs on training set of this model as entry for another model which may learn how to handle threshold to maximize similarity to ground truth on training set (all of those steps being validated using the same strategy).

I've never heard about it and I want to know whether it makes sense ?

Topic methodology optimization classification

Category Data Science

Yaroslaw Homenko · Accepted Answer · 2021年4月5日 18:28

What you are talking about sounds a lot like meta-learning (use outputs of several models and use another model to train on that predictions). You can read through more detailed example here (link to machinelearningmastery)

Should you do it for your binary classification task - no. (this in not a strict "no", but generally you should not do it given your task can really benefit from such approach, but this has little to do with classification threshold)

About threshold search, you can check this thread with quite good answers: https://stats.stackexchange.com/a/386433/232020

From your question, it sounds like you are dealing with the binary dataset, not sure what you mean by Unfortunately, the method seems not to be scaled on categorical classification.