Preserve colour in convolutional autoencoder

at the moment i work with convolutional autoencoder and now I'am looking for paper or methods that adresses a colour preversation. Most of the AE paper use grayscale images and loss functions such as SSIM that preserve the structure very well are also focused on grayscale images. My networks are good in preserving structure (with SSIM as loss) but have a hard time representing the right colour.

I use a all convolution architecture without any pooling. my downsampling is derived by strides > 1 and the upsampling is obtained by transpose convolution.