Tuning model by my metric

You didn't mention which parameters you want to tune or which regression algorithm you're using. I'm going to assume that you just want to tune a few hyper-parameters.

The generic method for tuning hyper-parameters is very simple and can be implemented easily: iterate over all the combinations of values that should be tested, train and test a model using every such combination of parameters and then pick the one which obtains the best performance.

For instance if you have two parameters $A$ and $B$ with $A\in \{ a_1,a_2,a_3\}$ and $B\in \{ b_1,b_2\}$:

for A in { a_1, a_2, a_3 } do
  for B in { b_1, b_2 } do
    train model M with parameters A and B on the training set
    apply model M to the test set to obtain predictions
    calculate metric score on the predictions
    store score for parameters (A,B)
  done
done
pick the parameters corresponding to max performance

Note that after this step of parameter tuning the final model with the selected parameters should be tested on a fresh test set.

If some complex cases it's worth using a genetic algorithm instead of exploring the full space of all the parameters.