What is the different between Fine-tuning and Transfer-learning?

Usually the neural network training has at least 2 steps:

  1. first trained on a large set of some standard data (ImageNet, ...)
  2. and then the resulting weights are trained on a small set of my data (in this step we can train all layers or only one last layer)

What is the same of 2-nd step, is it Fine-tuning or Transfer-learning? And what is the different between Fine-tuning and Transfer-learning?

Topic caffe computer-vision deep-learning neural-network machine-learning

Category Data Science


Fine-tuning is the process in which the parameters of a trained model must be adjusted very precisely while we are trying to validate that model taking into account a small data set that does not belong to the train set.

That small validation data set comes from the same distribution as the data set used for the training of the model. The split of the available data to train and validation set is random.


Transfer Learning or Domain Adaptation is related to the difference in the distribution of the train and test set.

So it is something broader than Fine tuning, which means that we know a priori that the train and test come from different distribution and we are trying to tackle this problem with several techniques depending on the kind of difference, instead of just trying to adjust some parameters (usually we are doing this for reasons as preventing overfitting etc.)


Generally, I would refer to this as transfer learning or network adaptation. That is, taking a network that has learned useful features from one domain and adapting that network and its developed features to another domain.

That said, there appear to be many sources that closely conflate fine tuning with transfer learning. Therefore, I would say the difference in terminology is primarily opinion-based and suggest closure of this question on those grounds.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.