CNN Eliminate Wrong Results

I extracted images of human faces from the videos, but the model also recorded images without faces. I wrote CNN for emotion classification. In the obvious pictures, the probability is closer to a probability in the softmax function in the last layer, for example, in a photo that is certain to be happy, a probability of 0.95 for the happy class appears, but if there is no face in the picture, it disperses between classes such as 0.3 and 0.2. Do you think it would make sense if I only included the images with a high softmax distribution in the result report, both to eliminate the images without faces and to get more accurate results?