binary classification pipeline to select threshold
There are quite a few questions regarding the optimisation of binary threshold in a classification problem. However, I haven't found a single end-to-end solution to this problem.
In an existing project, I have come up with the following pipeline to train a binary classifier:
- Outer-CV due to small to moderate data size.
- Inner-CV to tune hyperparameters
- Train model with tuned hyperparameters on outer-cv trainset
- Predict on the outer-cv test set
- Find optimal threshold using prediction probabilities
- Get score converting prediction probabilities to class with the optimal threshold
- Report avg/std scores along with thresholds
Since there's tiny to no deviation on the score across different folds. (However, the optimal threshold stddev is 3.2)
- Tune hyperparameter on entire data
- Train model with tuned hyperparameters on entire data
Now my questions are:
- Is this pipeline reasonable/correct? i.e., have I missed anything or parts are unnecessary?
- How to get the final optimal threshold for my model when predicting in production.
Topic hyperparameter-tuning cross-validation classification
Category Data Science