Custom layer for Simple Exponential Smoothing

Question

Custom layer for Simple Exponential Smoothing

Florian

2022年5月22日 08:05

I am writing a test custom layer which implements the Simple Exponential Smoothing algorithm. The problem: when I train it, the alpha (smoothing) coefficient always converges to value 1. This means that the one step forward forecast always takes the previous actual value. I most probably miss something obvious but couldn't figure it out, yet. Any idea? Thanks.

The code:

from tensorflow.keras.layers import Layer

class SES(Layer):
    def __init__(self, dtype=tf.float32):
        super(SES, self).__init__()
    def build(self, input_shape):
        alpha_init = tf.keras.initializers.random_uniform(minval=0., maxval=1.)
        self.alpha = tf.Variable(name=alpha, initial_value=alpha_init(shape=(1,),dtype='float32'),
                                 constraint=tf.keras.constraints.min_max_norm(0,1),trainable=True)
        
    def call(self, inputs): 
        '''SES formula: yhat (one step fwd) = alpha*y_previous (aka actual_previous) + (1-alpha)* yhat_previous'''
        def predict_one_step(y_previous, alpha, yhat_previous):
            yhat = (alpha*y_previous) + ((1-alpha)*yhat_previous)
            
            return yhat #prediction one step ahead
        
        predictions = []
        for timestep in range(inputs.shape[0]):
            if timestep == 0: 
                yhat_previous = inputs[timestep]
            
            yhat = predict_one_step(inputs[timestep], self.alpha, yhat_previous)
            yhat_previous = yhat
            
            predictions.append(yhat)
            
        return tf.concat(predictions, axis=-1)


--------------------- SES training loss --------------------
Loss at epoch 000: 0.439, alpha: 0.250
Loss at epoch 020: 0.226, alpha: 0.433
Loss at epoch 040: 0.129, alpha: 0.581
Loss at epoch 060: 0.069, alpha: 0.705
Loss at epoch 080: 0.031, alpha: 0.810
Loss at epoch 100: 0.011, alpha: 0.892
Loss at epoch 120: 0.003, alpha: 0.949
Loss at epoch 140: 0.000, alpha: 0.981
Loss at epoch 160: 0.000, alpha: 0.995
Loss at epoch 180: 0.000, alpha: 1.000
Loss at epoch 200: 0.000, alpha: 1.000
Loss at epoch 220: 0.000, alpha: 1.000
Loss at epoch 240: 0.000, alpha: 1.000
Loss at epoch 260: 0.000, alpha: 1.000
Loss at epoch 280: 0.000, alpha: 1.000
Loss at epoch 300: 0.000, alpha: 1.000
Loss at epoch 320: 0.000, alpha: 1.000
Loss at epoch 340: 0.000, alpha: 1.000
Loss at epoch 360: 0.000, alpha: 1.000
Loss at epoch 380: 0.000, alpha: 1.000
Loss at epoch 400: 0.000, alpha: 1.000
Loss at epoch 420: 0.000, alpha: 1.000
Loss at epoch 440: 0.000, alpha: 1.000
Loss at epoch 460: 0.000, alpha: 1.000
Loss at epoch 480: 0.000, alpha: 1.000
Final loss: 0.000
alpha = 1.000

Topic keras neural-network time-series

Category Data Science

Flavia Giammarino · Accepted Answer · 2021年8月9日 05:30

The issue is that you are using inputs[timestep] instead of inputs[timestep - 1] as y_previous in the call method, which is why the loss is exactly zero when alpha is equal to one. This has been fixed in the code below.

import tensorflow as tf
from tensorflow.keras.layers import Layer

class SES(Layer):

    def __init__(self, dtype=tf.float32):
        super(SES, self).__init__()

    def build(self, input_shape):

        alpha_init = tf.keras.initializers.random_uniform(minval=0., maxval=1.)

        self.alpha = tf.Variable(
            name="alpha",
            initial_value=alpha_init(shape=(1,), dtype='float32'),
            constraint=tf.keras.constraints.min_max_norm(0, 1),
            trainable=True
        )

    def call(self, inputs):

        def predict_one_step(y_previous, alpha, yhat_previous):
            return alpha * y_previous + (1 - alpha) * yhat_previous

        ### original
        #yhat_previous = inputs[0]
        #predictions = [yhat_previous]

        ### proposed modification ###
        yhat_previous = [inputs[0]]
        predictions = [tf.convert_to_tensor(yhat_previous, dtype='float32')]

        for timestep in range(1, inputs.shape[0]):

            yhat = predict_one_step(inputs[timestep - 1], self.alpha, yhat_previous)

            yhat_previous = yhat
            predictions.append(yhat)

        return tf.concat(predictions, axis=-1)

Custom layer for Simple Exponential Smoothing

About