Dataset and why use evaluate()?

Question

Dataset and why use evaluate()?

Murilo

2021年12月3日 18:24

I am starting in Machine Learning, and I have doubts about some concepts. I've read we need to split our dataset into training, validation and test sets. I'll ask four questions related to them.

1 - Training set: It is used in .fit() for our model to learn parameteres such as weights in a neural network?

2 - Validation set: Can also be used in .fit(). The validation set is used so we can validate our model at the end of each epoch (to tune some hyperparameteres, like the number of nodes in a hidden layer)?

3 - If 2 is correct (i.e, the validation set was already used in .fit()), do we still need to use .evalute()? And why?

4 - Test set: New inputs (x) never seen by the model, so i can make predictions on them? Used through the .predict() function?

Topic test validation training dataset

Category Data Science

Erwan · Accepted Answer · 2021年12月3日 18:24

First to be clear we're talking specifically about supervised learning here: there's a training stage during which the model is provided with labelled instances (features and class).

Simple analogy: the model is like a student.

Training: the student sees many exercises with their answer. The goal is for the student/model to learn how to find the answer based on the input in the exercise.
Evaluation: Equipped with their new knowledge, the student takes a test where the answers are not provided with the questions. The teacher knows the answers, and grades the student based on how correctly they answered. This step also requires annotated data in order to calculate how the model performs.
Production: Later the student may have a job where they apply the knowledge they acquired. In general nobody checks their answers, that's not the goal.

The most important distinction is between training set and test set, these two are always necessary in supervised learning. A validation set is optional: it's an intermediate test set which is used when there are different "levels" of training. For example at level 1 multiple models (e.g. with different features or hyper-parameters) are trained on the training set and evaluated on the validation set. At level 2 the best model is picked out of these multiple models: level 2 is a kind of "meta-training", so the final model still needs to be evaluated with a fresh test set (unseen data). Deep NNs also require a validation set, since they iteratively apply a big training/evaluation loop in order to train a model.

So the answer to question 3 is: because the validation set was used during training (level 2), it cannot be considered a good evaluation of the final model (since the model uses information from it).

Udaya Unnikrishnan · Accepted Answer · 2021年12月3日 17:22

Training dataset is used to train a model to learn information from data to solve a problem(Prediction).

Validation dataset is used for checking whether the trained model is good enough to solve the problem (prediction) on non seen data and for selecting best trained model for use. Validation data is not used for training , but it is used for selecting a best model for future use

Dataset and why use evaluate()?

About