Custom regularisation for logistics regression
My understanding of l2 regularisation:
Weights of the model are assumed to have a prior guassian distribution centered around 0. Then MAP estimate over data adds an extra penalty in cost function.
My problem statement:
I am making a reasonable assumption(based on domain knowledge) that my features are independent which means I can use the weights of the features to infer the importance of features in influencing Y.
From domain knowledge, I want to assume priors about the ratio of weights. Eg:
W1/W2 is a guassian distribution with mean 0.7
W2/W5 is a guassian distribution with mean 2.4
And then MAP estimate over data will give me a cost function, with extra penalty added.
Is my thought process correct? My intent is weights should be close to priors computed from some heuristic. This will also help me handle sparse data.
Implementation Details:
Is there any library(in any language) where it is easy to do this? Or do I have to compute the cost function and it's gradient myself and implement gradient descent over it?
Help with Math:
Also I have started learning stats(and maths in general) more rigorously, but I have lots of ground yet to cover. Assuming all Wi/Wj are given, can someone please give me the new cost function. I will write a program to solve for weights based on that cost function.
EDIT: As I know Wi/Wj, can I assume sum(Wi) = 100 and fit a bayesian logistics regression? Also bayesian logistic regression fit method api requires a hessian matrix. http://bayes-logistic.readthedocs.io/en/latest/usage.html?highlight=fit_bayes_logistic calls hessian matrix covariance matrix of fitted MAP parameters. Assuming that features are independent and variance of a feature to be 1, can I assume hessian matrix to be an identity matrix?
Topic bayesian regularization logistic-regression
Category Data Science