Stacking neural nets with cross validation

I am trying to implement stacking model for a ML problem and having hard time figuring out the cross validation strategy. So far I have used 10-fold cross validation for all my models and would like continue using that stacking as well. Here's what I came up with but not sure if it makes sense,

  1. At each iteration of 10-fold CV, you will have 9 folds for training (training dataset) and 1 fold for testing (testing dataset).

  2. Divide the training dataset into 3 parts - F1, F2 and F3.

  3. Train base classifiers on F1, use F2 for early stopping and get out of fold predictions on F3 - F3' (F3' is the set of predictions made by base-classifiers on F3)

  4. Train base classifiers on F2, use F3 for early stopping and get out of fold predictions on F1 - F1'

  5. Train base classifiers on F3, use F1 for early stopping and get out of fold predictions on F2 - F2'

  6. Train meta-classifier on (F1' + F2' + F3'), train base-classifiers on any two folds and use remaining fold for early-stopping.

  7. Validate meta-classifier on test dataset

Repeat these steps for every fold in 10 fold CV.

Topic ensemble-modeling cross-validation machine-learning

Category Data Science

1
answered at

In the diagram below that you can find here, it is shown the general approach to train a stacking model:

Thes steps are:

Given $T$ a train set of shape $mxp$

  1. Divide $T$ into $k$ folds
  2. Fit classifiers $M_1$, $M_2$,...,$M_n$ on $k-1$ and predict on fold $k$
  3. Save the predictions of the $M$ classifiers on fold $k$ (predictions on step 2)
  4. Repeat steps 2 and 3 $k$ times

After those steps, you will end up with a dataset $D$ of shape $mxn$

  1. Train a model $\Phi$ (meta model) on $D$ and retrain the $M$ different classifiers on $T$

Then you can use $\Phi$ to make predictions on unseen data

enter image description here

Hope it helps!

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.