Improve performance of my CNN model

I am working on an image classification problem. There are 876 images in the training and 600 in the test dataset. It is a multi class classification for plants.

Since this is my first CNN problem, I started working with tensorflow and keras to build my model and then started using transfer learning to improve my performance. The best model which I built so far using Keras is getting me a score of 0.68 ( since the test dataset has no labels, I have to upload my results over and they generate the score using log-loss).

Also, I tried using Fastai and Pytorch which significantly improved my single model score to 0.46.

I have used Resnet50 image dataset for both cases but i cannot experiment using Resnet101 or increasing the batch size since my gpu goes out of memory( also tried using kaggle and google colab).

Is their any way which i can try to further improve the performance? I am basically stuck now since i can't experiment by increasing either batch size or using a better model like Resnet101 etc.

I have uploaded both codes at : https://github.com/akshatshreemali/ImageClassification