Finding global optimum of unknown and expensive function

I would like to find optimal combination of parameters for the algorithm affecting the disk space used by some storage. Therefore, several algorithm parameters (x1, x2, x3, where 0 x1 1, 10 x2 100, 0.1 x3 0.5) are used as an input for the model, and the disk space occupied by storage S(x1, x2, x3) is the cost function I'd like to minimize.

The problem is that every function call S(x1, x2, x3) is very expensive, and it may take minutes or even hours to end up, so it's almost infeasible to prepare the training set. Instead, I would like ML library to suggest, which combination of (x1, x2, x3) values to use to estimate S on every next optimization step.

I believe this task is quite common for the major ML frameworks, but I wasn't able to find anything suitable yet. So what is the name for this class of problems, and which ML framework provide the routines to solve it?

Topic cost-function optimization

Category Data Science

1
answered at

What you have is a standard optimization problem, which has got nothing to do with ML. I don’t know what modelling language you use, but Scipy in Python offers many nonlinear optimization algorithms - global or local.

If I were you, for this problem I would stay away as much as possible from gradient descent: if the evaluation of your cost function does not also provide gradient information, then you have to calculate it numerically. Which involves many more functions evaluations. Whether your cost function provides gradient outputs or not, gradient descent is just about the worst optimization algorithm you can think of in the realm of nonlinear solvers.

Global solvers will attempt to give you a “global optimum” - but they are much more expensive to run and you will never know whether the minimum/maximum you found is the actual “global” one. Local algorithms are faster to converge but they only ever provide a locally optimum minimum/maximum.

https://docs.scipy.org/doc/scipy/reference/optimize.html

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.