About neural network ability to generalize

Question

About neural network ability to generalize

user131341

2022年1月22日 02:03

I had this question during an interview that I wasn't able to answer, even after researching on the internet.

Which of the following can affect an artificial neural network’s ability to generalize???

absence of biais
learning bias
size of output layer
no.of examples

And please can you explain a little why? Thank you

Topic generalization neural-network

Category Data Science

Raymond Kwok · Accepted Answer · 2022年1月22日 02:03

My two cents:

Simple redefinition of well "generalized": it does not overfit. it does not underfit.

absence of biais: I think you meant keeping/taking-away the bias terms in your NN layers. In the case that true data-generating process does have a constant bias term which is therefore independent of all of your features, your model won't be able to cover that, you may underfit that. In the case that such constant bias term does not exist, or effectively removed by normalization, although you may argue that there is a chance that the model will fit the bias term as ZERO, getting rid of the bias (equivalently hard-setting them to ZERO) helps your overfitting problem because your model has less freedom to fit itself to the data.
learning bias. if it is as defined in wiki (which I doubt), then it refers to model assumptions. If you read examples in the Type section of the wiki link, they are assumptions which simplify the problem, and so your model tends to underfit the real world.
no.of examples. Generally more good data helps overfitting problem. With more model parameters (such as more layers), you need more good data.

Finally, I don't see how the "size of output layer" can affect generalization ability. The size should adapt to the problem. If it is a regression which requires 1 output number, then the size is 1. If it is a multi-class classification of N classes then it requires N outputs.