Val Loss and manually calculated loss produce different values

I have a CNN classification model that uses loss: binary cross entropy:

optimizer_instance = Adam(learning_rate=learning_rate, decay=learning_rate / 200)
model.compile(optimizer=optimizer_instance, loss='binary_crossentropy')

We are saving the best model so the latest saved model is the one that achieved the best val_loss:

es = EarlyStopping(monitor='val_loss', mode='min', verbose=0, patience=Config.LearningParameters.Patience)
modelPath = modelFileFolder + Config.LearningParameters.ModelFileName
checkpoint = keras.callbacks.ModelCheckpoint(modelPath , monitor='val_loss',
                                                         save_best_only=True,
                                                         save_weights_only=False, verbose=1)
callbacks = [checkpoint,es]
history = model.fit(x=training_generator,
                    batch_size=Config.LearningParameters.Batch_size,
                    epochs=Config.LearningParameters.Epochs,
                    validation_data=validation_generator,                              
                    callbacks=callbacks,
                    verbose=1)

on the course of the training the logs show that the val_loss has decreased to 0.41. At the end of the train we load the best model that was saved during the training process and predicted the validation dataset. Then we calculated the BCE manually and received a totally different value of 2.335.

here is the manual loss calculation:

bce = tf.keras.losses.BinaryCrossentropy()
binaryCSELoss = bce(y_valid, preds)
print(Calculated Val Loss is:  + str(binaryCSELoss ))

here is the end of the training logs:

10/10 [==============================] - ETA: 0s - loss: 0.0778
Epoch 40: val_loss did not improve from 0.41081
10/10 [==============================] - 4s 399ms/step - loss: 0.0778 - val_loss: 0.5413
% of marked 1 in validation: [0.51580906 0.48419094]
% of marked 1 in Test: [0.51991504 0.480085  ]
---------------------------------
Calculated Val Loss is: 2.3350689765791395

We thought that it might have to do something with the face that we are using data generators and the loss is then calculated on the batches individually so we added another test where we do not use data generators:

history = model.fit(x=trainX,y = y_train,
                      epochs=Config.LearningParameters.Epochs,
                      validation_data=(validateion_x,y_valid),
                      callbacks=callbacks,
                      verbose=1)
predictions_cnn = model.predict(validateion_x)
bce = tf.keras.losses.BinaryCrossentropy(from_logits=False)
binaryCSELoss = bce(y_valid, predictions_cnn)
valloss = binaryCSELoss.numpy()
print(binaryCSELoss logits=false on all Val Loss is:  + str(valloss))
bce = tf.keras.losses.BinaryCrossentropy(from_logits=True)
binaryCSELoss = bce(y_valid, predictions_cnn)
valloss = binaryCSELoss.numpy()
print(binaryCSELoss logits=true on all Val Loss is:  + str(valloss))

Here is the end of the training log. Again the loss is no the same:

54/54 [==============================] - ETA: 0s - loss: 0.5015
Epoch 6: val_loss did not improve from 0.66096
54/54 [==============================] - 8s 144ms/step - loss: 0.5015 - val_loss: 1.9742
% of marked 1 in validation: [0.28723404 0.71276593]
% of marked 1 in Test: [0.52077866 0.47922137]
loading Model: E:\CnnModels\2022-06-03_11-53-53\model.h5
Backend TkAgg is interactive backend. Turning interactive mode on.
binaryCSELoss logits=false on all Val Loss is: 0.6353029
binaryCSELoss logits=true on all Val Loss is: 0.7070135

How can this be?