Physical modelling with neural networks - single output + stack ensemble vs multi-output

We are trying to replace an existing physical model (8 inputs/7 outputs) with artificial neural networks. The physics behind the existing model is mainly thermodynamics of humid air for air conditioning, with some turbomachinery involved, which yields most likely complex functions between inputs and outputs.

One approach was already done: single output neural networks (10 NN with same # hidden layers but different parameters like batch size, # epochs, optimizer, etc). Then some sort of stacking ensembled was used: every prediction was used as new inputs for a single NN to predict a final value.

The accuracy is pretty well, however, there are some test data points where the absolute errors are high enough to be worried about the predicted value (this could be used for example for air conditioning control strategies, so a bad prediction would result in an uncontrolled system).

In order to improve accuracy, some colleagues were suggesting to keep it simpler and perform just a multi-output regression with a single neural network.

From the mathematical point of view, I have the following questions:

  1. would a single output NNs stacking ensembled outperformed the multi-output single NN?
  2. is the way of stack ensembling using NN a good approach? I saw some different techniques like arithmetic averaging the inputs

Thanks for your time! Regards

Topic multi-output ensemble regression neural-network

Category Data Science

1
answered at

My guess: I agree with your colleagues. I see no reason to do anything other than a single neural network with multiple outputs. If necessary, increase the capacity of that single neural network until you see no further improvement.

An stacking ensemble where you have a few neural nets whose inputs are fed as input into another neural network is itself equivalent to one bigger neural net. I don't see any reason to expect the stacking approach to train more effectively or to be better for some other reason.

However, this is ultimately an empirical field. The only way to find out what will work best is to try different approaches and see. There's not a huge amount of theory that enables us to make predictions about what approaches will be most effective.

You might consider whether you can obtain more training data, particularly in the regimes of particular interest. Often better training data sets offer higher gains than modifying the neural network architecture.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.