How to use learning curve in reality
CONTEXT:
I have some simulated data by which I made and trained a model. during my training, I enjoyed having a large number of samples, and therefore my model is leveraging it by being decently complex. Yet since this model is trained on simulated data, in action, the model must be trained again once the real data is collected. Obtaining those real data is hard and we want to know how many data samples are needed before the model starts giving accurate prediction. So we started to look at the learning curve.
in summary,
In real-world problems, it's vital to check the learning curve of a model, but not because we'd like to know if excess data is being used, but to make sure enough data has been fed to the model and the model has actually learned something.
SOME EXPERIENCE:
we should be sure of the convergence of the model in order to make a decision about the adequacy of samples. Yet, it might be the case that the convergence rate of the model using a small number of samples is different from once a huge bulk of samples are used for training.
Experimentally I realized that for reaching an optimal solution, you might need to change the learning rate or some structural features of your estimator/optimizer. For instance, once learning a tree-based model, one might want to think again about the minimum number of samples she/he like to have in each leaf.
QUESTION
Is it acceptable to change the meta-parameters in the estimater just for the sake of convergence in small number of samples?
Is it really needed to have convergence at small sizes of training set? My intuition is that the model starts to behave differently (and depart from its transient behavior) as soon as enough number of samples are given.
How in practice learning curve is being used? I assume all the businesses out there are data-hungry except a hand full of companies which already acquired zillions of samples. So I suppose, no one really thinks of the learning curve from the first point of view I mentioned above.
Topic machine-learning-model learning
Category Data Science