To freeze or not, batch normalisation in ResNet when transfer learning

Question

To freeze or not, batch normalisation in ResNet when transfer learning

amateurjustin

2022年5月26日 20:06

I'm using a ResNet50 model pretrained on ImageNet, to do transfer learning, fitting an image classification task.

The easy way of doing this is simply freezing the conv layers (or really all layers except the final fully connected layer), however I came across a paper where the authors mention that batch normalisation layers should be fine tuned when fitting the new model:

Few layers such as Batch Normalization (BN) layers shouldn’t be froze because, the mean and variance of the dataset will be hardly matching the mean or variance from pre-trained weights. So, auto-tuning [30] is adapted for the BN layers in ResNey50, i.e. few of the layers which are in the top of ResNet50 shouldn’t be frozen.

source here

I initially froze everything, but in the interest of time and not having to commit computation to testing the effect of not freezing the BN layers, how important is it to not freeze them?

Topic transfer-learning cnn batch-normalization

Category Data Science

ShinyDemon · Accepted Answer · 2022年4月19日 10:37

The batch normalization layer helps with effectively training the model. Since you are transfer learning, you may have frozen everything up to the fully connected classifier. If you want to keep the parameters of the frozen layers exactly the same as the original model, you can load the weights of only the retrained head during inference and/or evaluation.

Your best bet is to try everything and see what works best for you.

To freeze or not, batch normalisation in ResNet when transfer learning

About