DecisionTreeRegressor under the hood of GradientBoostingClassifier

Question

DecisionTreeRegressor under the hood of GradientBoostingClassifier

Oliver Foster

2020年10月30日 01:39

I'm inspecting the weak estimators of my GradientBoostingClassifier model. This model was fit on a binary class dataset.

I noticed that all the weak estimators under this ensemble classifier are decision tree regressor objects. This seems strange to me intuitively.

I took the first decision tree in the ensemble and used it to predict independently on my entire dataset. The unique answers from the dataset were the following:

array([-2.74, -1.94, -1.69, ...])

My question is: why and how does the gradient boosting classifier turn the weak estimators into regressor tasks (instead of classification tasks) that are not bound by 0 and 1? Ultimately the GradientBoostingClassifier outputs a pseudo-probability between 0 and 1: why aren't the ensemble of weak estimators doing the same?

Topic natural-gradient-boosting ensemble-modeling regression scikit-learn classification

Category Data Science

Oliver Foster · Accepted Answer · 2020年10月30日 01:39

After reading through more of the documentation I found the section that covers the classification case. It can be found here. Additionally this statquest was very useful.

1.11.4.5.2. Classification

Gradient boosting for classification is very similar to the regression case. However, the sum of the trees is not homogeneous to a prediction: it cannot be a class, since the trees predict continuous values.

The mapping from the value to a class or a probability is loss-dependent. For the deviance (or log-loss), the probability that belongs to the positive class is modeled as where is the sigmoid function. For multiclass classification, K trees (for K classes) are built at each of the iterations. The probability that belongs to class k is modeled as a softmax of the values.

Note that even for a classification task, the sub-estimator is still a regressor, not a classifier. This is because the sub-estimators are trained to predict (negative) gradients, which are always continuous quantities.

DecisionTreeRegressor under the hood of GradientBoostingClassifier

About