NNs for fitting highly oscillatory functions

in a scientific computing application of neural networks, I have to maximize several neural networks with scalar output with respect to a target/loss function (coming from a weak form of a PDE).

It is known from theoretical considerations that typically the functions that would be optimal with respect to the target function (i.e. the maximizers) are extremly oscillatory functions. I suppose that this is the reason, why - according to my first numerical experiments - typical network architectures, initializations and training procedures from Computervision and related fields do not perform well in fitting those oscillatory functions.

Does anyone have any ideas, suggestions or even references for how to deal with this problem? Are there any standard network architectures, activation functions, initializations and so on that work well in such a situation, where the optimal result would be a highly oscillatory function? (The training points stem from certain quadrature rules and in principle I can choose arbitrary many, restricted of course by the computational cost).

Best regards

PM

Topic weight-initialization training neural-network

Category Data Science

1
answered at

One option is to translate extremely oscillatory functions into the scalar values that are expected by standard neural networks. Fourier transform (FT) does this by decomposing a frequency domain representation to a function of space or time.

1
answered at

The problem of approximating periodic functions is studied in the article Neural Networks Fail to Learn Periodic Functions and How to Fix It presented at NeurIPS 2020. From the abstract:

[...] we prove and demonstrate experimentally that the standard activations functions, such as ReLU, tanh, sigmoid, along with their variants, all fail to learn to extrapolate simple periodic functions. We hypothesize that this is due to their lack of a “periodic” inductive bias. As a fix of this problem, we propose a new activation, namely, $x + sin^2 (x)$, which achieves the desired periodic inductive bias to learn a periodic function while maintaining a favorable optimization property of the ReLU-based activations.

I suggest that you try their proposed activation functions to evaluate if their inductive bias is enough for your case.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.