Accuracy drops when adding a fully connected layer for dimensionality reduction to a ResNet50

I'm training a ResNet50 for image classification and I'm interested in decreasing the dimensionality of the embedded layer, in order to apply some clustering techniques.

The suggested dimension is something in the range 64-256, so I thought I'd start from 128.

I'm using PyTorch. After loading the pretrained ResNet50 from the official release I would usually do this:

model = t.load(cfg.resnet_path)
model.fc = nn.Sequential(nn.Linear(in_features = 2048, out_features = num_classes, bias = True))

Everything worked and I reached an accuracy of around 85% for the 33-classes dataset.

To apply dimensionality reduction I modified the fc layer in this way:

model = t.load(cfg.resnet_path)
model.fc = nn.Sequential(nn.Linear(in_features = 2048, out_features = 128, bias = True),
                         nn.ReLU(),
                         nn.Linear(in_features = 128, out_features = num_classes, bias = True))

I re-trained it from scratch on the same dataset and suddenly my accuracy drops to near 0. The same happens if tanh is used instead of ReLU. Am I doing something wrong?

These are the last layers of the un-modified ResNet50:

layer4.2.conv1 Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
layer4.2.bn1 BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
layer4.2.conv2 Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
layer4.2.bn2 BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
layer4.2.conv3 Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
layer4.2.bn3 BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
layer4.2.relu ReLU(inplace=True)
avgpool AdaptiveAvgPool2d(output_size=(1, 1))
fc Linear(in_features=2048, out_features=1000, bias=True)