CycleGAN: Both losses from discriminator and generator drop fast, after 100 epochs outputs blurred original image

I'm trying to train a 3D Cycle-GAN on medical image synthesis, more specifically CT to MR.

Currently I'm using a 3-Layer Discriminator and a 6 layer UNetGenerator borrowed from the official CycleGAN codes. Same lambda A, B of 10 and .5 of identity.

The discriminator loss drops to around 0 in the first few epochs, and the total loss for generator drops to around 1 as well. The generator continues to ouput blurred original input image.

During my debugging I noticed that the discriminator can distinguish the fake and real image very well. Should I increase the complexity of the generator? Can someone help to explain the problem with my model?