Autoencoder not learning walk forward image transformation

I have a series of 15 frames with (60 rows x 50 columns). Over the course of those 15 frames, the moon moves from the top left to the bottom right.

Data = https://github.com/aiqc/AIQC/tree/main/remote_datum/image/liberty_moon

I am attempting a walk forward autoencoder where:

  • The input data is a 60x50 image.
  • The evaluation label is a 60x50 image from 2 frames later.
  • All data is scaled between 0-1.
    model = keras.models.Sequential()
    model.add(layers.Conv1D(64*hp['multiplier'], 3, activation='relu', padding='same'))
    model.add(layers.MaxPool1D( 2, padding='same'))
    model.add(layers.Conv1D(32*hp['multiplier'], 3, activation='relu', padding='same'))
    model.add(layers.MaxPool1D( 2, padding='same'))
    model.add(layers.Conv1D(16*hp['multiplier'], 3, activation='relu', padding='same'))
    model.add(layers.MaxPool1D( 2, padding='same'))

    model.add(layers.Conv1D(16*hp['multiplier'], 3, activation='relu', padding='same'))
    model.add(layers.UpSampling1D(2))
    model.add(layers.Conv1D(32*hp['multiplier'], 3, activation='relu', padding='same'))
    model.add(layers.UpSampling1D(2))
    model.add(layers.Conv1D(64*hp['multiplier'], 3, activation='relu'))
    model.add(layers.UpSampling1D(2))

    model.add(layers.Conv1D(50, 3, activation='sigmoid', padding='same'))
    # last layer tried sigmoid with BCE loss.
    # last layer tried relu with MAE.

Tutorials say to use a final layer of sigmoid and BCE loss, but the values I'm producing must not be between 0-1 because the loss goes way negative.

If I use a final layer of relu with MAE loss it claims to learn something.

But the predicted image is notttt great:

Topic convolutional-neural-network autoencoder deep-learning neural-network

Category Data Science


Image auto-encoders are great for style change, but not so great for content change.

Anyways, as you keep ~95% untouched in your transformation, a ResNet would probably be very helpful! If you use an hourglass network, you shall see it quickly converging into a self-encoder (which is ~95% right) and than focus all the training just on the difference (the moon part).

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.