Fine-grained image classification

I have a dataset which has 4 classes (say A,B,C,D). The task requires fine-grained image classification. The problem I am facing is that for 2 of the classes (C,D), the model's performance is not so great. Out of around 200 test images belonging to class C, 46 are being predicted as belonging to class D. So, class C's recall is poor, as a result class D's precision is poor.

Now, I also looked at the validation set images belonging to class C, and all of them are being predicted correctly. Visual inspection shows that the labelling accuracy is quite good and the test images belonging to class C are not so different from those present in the validation set. The precision and recall for all the classes in the validation set is quite decent.

I am using a pretrained resnet50 and the fastai training loop (learning rate finder, 1-cycle lr scheduler,etc). Hyperparameter tuning is being done using optuna.

Can someone please give me ideas of how to do some root cause analysis here? Or, what else I could try?