Which other algorithms fit residuals like XGBoost?

Question

Which other algorithms fit residuals like XGBoost?

Bobby

2020年7月28日 17:22

XGBoost and standard gradient boosting train learners to fit the residuals rather than the observations themselves. I understand that this aspect of the algorithm matches the boosting mechanism which allows it to iteratively fit errors made by previous learners.

Which other algorithms or also train single or multiple learners to fit residuals?
Does this method only make sense for learners built in a sequence? Or also for any ensemble methods?
Is there a deep significance to fitting residuals or is this a mathematical convenience that allows for iterative improvement from learners built in succession?

Topic natural-gradient-boosting boosting xgboost

Category Data Science

nimar · Accepted Answer · 2020年7月28日 17:22

Training of XGBoost is based on a boosting model, which is a general ensemble method creating a strong model from a number of weak models. This process is performed by building a model from the training dataset, then, creating a second model that attempts to correct the errors from the first model. Models are added until the training set is predicted perfectly or a maximum number of models are added.

All machine learning models based on boosting almost follow the above procedure. For example, AdaBoost is one of the boosting algorithms developed for binary classification. Methods like LightGBM and Catboost use this algorithm.

Unlike the boosting algorithm, in the bagging algorithm, the models are independent and each model is directly fitted to a subset of the original training data.

Which other algorithms fit residuals like XGBoost?

About