LSTM Autoencoder set-up for multiple features using Pytorch

I am building an LSTM autoencoder to denoise signals and will take more than 1 feature as it's input.

I have setup the model Encoder part as follows which works for single feature inputs (i.e. sequences with just one feature):

class Encoder(nn.Module):
    def __init__(self, seq_len, n_features, num_layers=1, embedding_dim=64):
        super(Encoder, self).__init__()
        
        self.seq_len = seq_len
        self.n_features = n_features
        self.num_layers = num_layers
        self.embedding_dim = embedding_dim
        self.hidden_dim = 2 * embedding_dim
        
        # input: batch_size, seq_len, features
        self.lstm1 = nn.LSTM(
            input_size=self.n_features, 
            hidden_size=self.hidden_dim, 
            num_layers=self.num_layers,
            batch_first=True
        ) # output: batch size, seq_len, hidden_dim 
        
        # input: batch_size, seq_len, hidden_dim
        self.lstm2 = nn.LSTM(
            input_size=self.hidden_dim,
            hidden_size = self.embedding_dim,
            num_layers = self.num_layers,
            batch_first=True    
        ) # output: batch_size, seq_len, embedding_dim
        
    def forward(self, x):
        print(x)
        x = x.reshape((1, self.seq_len, self.n_features))
        print(x.shape)
        
        x, (_, _) = self.lstm1(x)
        print(x.shape)
        x, (hidden_n, _) = self.lstm2(x)
        print(x.shape, hidden_n.shape)
        print(hidden_n)
        
        return hidden_n.reshape((self.n_features, self.embedding_dim))

When I test this setup as follows:

model = Encoder(1024, 1)
model.forward(torch.randn(1024, 1))

with the 1 representing a single feature all is well. However, when I do the following (where 2 represents a sequence of 2 features):

model = Encoder(1024, 2)
model.forward(torch.randn(1024, 2))

I get the following error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [296], in cell line: 1()
---- 1 model.forward(torch.randn(1024, 2))

Input In [294], in Encoder.forward(self, x)
     36         print(hidden_n)
     37 #         print(hidden_n.reshape((self.n_features, self.embedding_dim)).shape)
--- 39         return hidden_n.reshape((self.n_features, self.embedding_dim))

RuntimeError: shape '[2, 64]' is invalid for input of size 64

The hidden_n shape comes out as torch.Size([1, 1, 64]). I would like to understand that if we have more than 1 feature, e.g. 2, do we want to get that shape into the format of 1, 2, 64 such that the hidden state has weights for both features?

Can someone please explain why reshape is not liking the way I'm trying to restructure the output of the Encoder and how I should do it such that the model is able to take any feature size into account.

What am I missing/ perhaps misunderstanding here?

Topic reshape pytorch lstm autoencoder python

Category Data Science

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.