Training neural network for regression with gaussian output layer

How does one train a neural network model that does regression over real values, using a gaussian output layer? ie estimating the mean and std parameters of the prediction. Since during training there will only be one real valued target. Would one sample from the output gaussian distribution and then perform loss with the target? Or would there somehow be two targets, one for the mean or std.

The Deep AR forecasting model by Amazon does something like this for timeseries forecasting, where the FC layer at the output of the RNN predicts the mean and std parameters of a gaussian distribution. However, the target label would only contain one real valued number. So how would the loss be computed? Sample from the gaussian distribution and then use that? This kind of sampling is done in RL to encourage exploration but i’m not sure how it’s done in training regression models

Topic rnn regression deep-learning

Category Data Science


You can optimize for the likelihood of the data under the predicted Gaussian distribution. In other words

  1. You have a predicted mean (u) and standard deviation (s) for each datapoint (x)
  2. Calculate the negative log likelihood of that datapoint under that distribution. Here is the pytorch implementation, but you can solve this analytically from the PDF of a Gaussian and calculate it with whatever automatic diff library you want.
  3. Treat this as your loss and optimize.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.