Neural Network Stuck at Low Accuracy

I am new to deep learning so forgive me if this is an obvious mistake, I have tried to find similar questions online yet none seem relevant to my problem.

I am using pytorch for image classification and my accuracy is stuck at ~40% even though my framework seems fine. Am I missing something major? My data has 5 columns: Age, Gender, Ethnicity, Image name , and pixel values (which are already scaled). I want to target the ethnicity output, which has 5 classes.

A problem I spotted is that when I pass the training data into my model, the output tensor is 5 values (for 5 different classes), and the value at index 0 is always the lowest negative value (the argmax), making much of my models predictions the ethnicity at index 0.

I have also tried training on 100 epochs but the accuracy and loss remains relatively the same. Here is my code:

data = pd.read_csv(age_gender.csv)
data['pixels']=data['pixels'].apply(lambda x:  np.array(x.split(), dtype=float32))
data['pixels'] = data['pixels'].apply(lambda x: x/255)

X = np.array(data['pixels'].tolist())
X = X.reshape(X.shape[0],48,48,1)

X = torch.tensor(X)
y = torch.tensor(data['ethnicity'])

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=37, shuffle = True)

Model

class NeuralNetwork(nn.Module): 
  def __init__(self):
    super(NeuralNetwork, self).__init__()
    self.flatten = nn.Flatten()
    self.linear_relu_stack = nn.Sequential(
      nn.Linear(48*48,64),
      nn.ReLU(),
      nn.Linear(64, 64),
      nn.ReLU(),
      nn.Linear(64, 5))

 def forward(self, x): 
   x = self.flatten(x)
   logits = self.linear_relu_stack(x)
   return F.log_softmax(logits, dim=1)

model = NeuralNetwork()

BACKPROP, LOSS, OPTIMIZER

import torch.optim as optim
loss_fn = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

for epoch in range(3): 
  optimizer.zero_grad() 
  output = model(X_train.view(-1,48*48))  
  loss = loss_fn(output, y_train) 
  loss.backward()
  optimizer.step() 
  print(loss)  

  output
   tensor(1.4611, grad_fn=NllLossBackward0)
   tensor(1.4678, grad_fn=NllLossBackward0)
   tensor(1.4475, grad_fn=NllLossBackward0)


correct = 0
total = 0

with torch.no_grad(): # we dont want to calculate gradients
  output = model(X_test.view(-1, 2304))
  for index, i in enumerate(output):
    if torch.argmax(i) == y_test[index]:
      correct += 1
    total += 1

 print(acc:, round(correct/total, 3))

acc: 42.5