Image as input and output in keras

I am trying to make a model of this image. Here is the relevant code:

base_model = VGG16(weights='imagenet')
conv4_3, conv3_3, conv2_2, conv1_2 = base_model.get_layer('block4_conv3').output,
    base_model.get_layer('block3_conv3').output,
    base_model.get_layer('block2_conv2').output,
    base_model.get_layer('block1_conv2').output

# Use the output of the layers of VGG16 on x in the model
conv1 = Convolution2D(256, 1, 1, border_mode='same')(BatchNormalization()(conv4_3))
conv1_scaled = resize(conv1, 56)
.
.
.
conv5 = Convolution2D(3, 3, 3, border_mode='same')(merge([ip_img, conv4], mode='sum'))
op = Convolution2D(2, 3, 3, border_mode='same')(conv5)

for layer in base_model.layers:
    layer.trainable = False

model = Model(input=base_model.input, output=op)
model.compile(optimizer='sgd', loss=custom_loss_fn)

I have a bunch of colored images in a directory. The input image should be the grayscale of an image stacked thrice(224x224x3) and the op should be the UV planes of the image(224x224x2) which I can add to the grayscale(224x224x1) to get the YUV image. The custom loss function works on the UV of the original image and UV of the prediction.

How do I train it?