Is my model underfitting?

Question

Is my model underfitting?

CJSSSSS

2021年1月28日 15:34

Model:

model = models.Sequential()
#add model layers
model.add(layers.Conv1D(64, kernel_size=3, activation='relu',padding='same'))
#model.add(layers.Conv1D(16, kernel_size=3, activation='sigmoid',padding='same'))    
model.add(layers.MaxPooling1D( pool_size=2, strides=None, padding='same', data_format=None))


#model1.add(tf.keras.layers.LSTM(100,return_sequences=True))
model.add(layers.Flatten())
model.add(layers.Dense(500, activation='relu'))
model.add(layers.Dropout(0.5))
model.add(layers.Dense(120, activation='relu'))
model.add(layers.Dropout(0.5))
model.add(layers.Dense(60, activation='relu'))
model.add(layers.Dropout(0.5))
model.add(layers.Dense(3, activation='softmax'))

I have initialized an early-stopping as well.

Is my model underfitting?

Epoch 213/400
360684/360684 [==============================] - 38s 106us/sample - loss: 0.3868 - acc: 0.8426 - val_loss: 0.2698 - val_acc: 0.9050
Epoch 214/400
360684/360684 [==============================] - 38s 106us/sample - loss: 0.3851 - acc: 0.8439 - val_loss: 0.2542 - val_acc: 0.9105
Epoch 215/400
360684/360684 [==============================] - 38s 105us/sample - loss: 0.3843 - acc: 0.8444 - val_loss: 0.2532 - val_acc: 0.9130
Epoch 216/400
360684/360684 [==============================] - 38s 105us/sample - loss: 0.3849 - acc: 0.8436 - val_loss: 0.2506 - val_acc: 0.9116
Epoch 217/400
360684/360684 [==============================] - 38s 105us/sample - loss: 0.3825 - acc: 0.8450 - val_loss: 0.2529 - val_acc: 0.9127
Epoch 218/400
360684/360684 [==============================] - 38s 106us/sample - loss: 0.3815 - acc: 0.8446 - val_loss: 0.2541 - val_acc: 0.9120
Epoch 219/400
360684/360684 [==============================] - 39s 108us/sample - loss: 0.3821 - acc: 0.8442 - val_loss: 0.2598 - val_acc: 0.9094
Epoch 220/400
360684/360684 [==============================] - 40s 110us/sample - loss: 0.3818 - acc: 0.8456 - val_loss: 0.2545 - val_acc: 0.9123
Epoch 221/400
360684/360684 [==============================] - 37s 104us/sample - loss: 0.3789 - acc: 0.8457 - val_loss: 0.2436 - val_acc: 0.9154
Epoch 222/400
360684/360684 [==============================] - 38s 105us/sample - loss: 0.3819 - acc: 0.8456 - val_loss: 0.2506 - val_acc: 0.9115
Epoch 223/400
360684/360684 [==============================] - 38s 105us/sample - loss: 0.3795 - acc: 0.8456 - val_loss: 0.2507 - val_acc: 0.9151
Epoch 224/400
360684/360684 [==============================] - 37s 104us/sample - loss: 0.3791 - acc: 0.8466 - val_loss: 0.2558 - val_acc: 0.9091
Epoch 225/400
360684/360684 [==============================] - 38s 106us/sample - loss: 0.3793 - acc: 0.8466 - val_loss: 0.2531 - val_acc: 0.9100

The difference between acc and val_acc is significantly small, like 5-6%. Should I be worried?

Topic generalization cnn tensorflow

Category Data Science

Kulikr · Accepted Answer · 2021年1月28日 15:34

I think the best practice is to overfit intentionally to make sure your model is not underfitted (pick a huge number of epochs). Then, you will know for sure if the learning process is over and you are just overfitting... In your case, it is not certain whether your model can still improve on the validation set.

I would be more suspicious about the better fit on the validation set. It might happen on some occasions where the validation set belongs to a certain area of the training distribution where your model better detects the classes. If it is not the case, it might be caused by leakage.

Mrityu · Accepted Answer · 2021年1月28日 14:37

NO. It's not Underfitting( Yay!!! ). But yes, there are some scopes of improvement such as:-

I can see too many dropout layers. Try to oust them with BatchNormalization. That way you can have a more generalized model along with leveraging
Try to reduce no. of Dense layers and increase no. of Conv1D layers. It will have two effects. First, your training time will decrease significantly and your model will most likely perform better.
Use Cross-Validation( Stratified K-Fold preferably ) as it will provide a better insight into your model's generalizability.
Use ReduceLROnPlateau callback as you can see your model isn't learning anything substantial from in the epochs you have attached in the question. It may be possible that your model is going through a plateau phase.
( NOT SURE ABOUT THIS ) Try to reduce your epoch.

Do point out if I have suggested something absurd. Thank You

David Masip · Accepted Answer · 2021年1月28日 09:52

I don't think you need to worry, instead I would ask myself if the accuracy I'm getting is good enough for the task that the NN is supposed to do.

Having higher training loss than validation loss can mean different things:

Your validation data is easier to assess than training data. If the train/validation split is done randomly and there is enough data in both subsets, this shouldn't be the case.
You're using dropout in training but not in validation. This is the default of some deep learning libraries, and it makes sense. If this is the case and you want to see less of a gap, try to reduce the amount of dropout rate and you'll see less of a gap.

To sum up, I don't think it's an issue but you might be able to improve your validation performance by reducing the amount of regularization or increasing the complexity of the NN. However, this is just a hypothesis and the only way to know is to re-train and check the new performance.

Edit

By default, keras doesn't do dropout in prediction so this is likely your case since you have high dropout rates.

Is my model underfitting?

Edit

About