Improve model accuracy in multi-classification problem

I use a MLP to classify three different classes A, B, C. The loss function I use is categorical cross entropy and the optimiser is adam. To estimate my models performance I use 10-fold Cross Validation. On average i get 60% accuracy score but I need it to be higher. The confusion matrix for the classes A,B,C, I get is the following:

The amount of data points I have for each class is sufficiently large and equal (the ratio is 1:1:1). I was trying to see how the model would fare, if instead of 3 classes it classified two classes. Each time I remove from the dataset the data points of the class I wont be using. So when I train it to classify data points between class B and C I get around 80% accuracy scoreand the following confusion matrix:

However when I train the model to classify data points between A and C i get around 69.5% and the following confusion matrix:

For the classes A and B I get around 72% accuracy score and the following confusion matrix:

In all cases the precision score, recall score and f1 score are more or less equal to the accuracy score.

• Could the reason why I get low accuracy when I classify the 3 classes be class A?
• Maybe some data points from class A are too similar with data points from class B and others are too similar with data points of class C?
  
  If so what could I do to improve the score provided that I can't improve the dataset.