Linear combination of features reverses importance of all features
I am trying with a logistic model with 2 features independently or with linear combination, but in the linear combination, combining these features would reverse importance through significance levels and regression coefficients and reduce prediction accuracy (AUC).
For example,
With a logistic regression where each of the predictor was scaled with MinMaxScaler
When using each features independently:
Feature A: coefficient = 0.5, P=0.005, AUC=0.59
Feature B: coefficient = 0.1, P=0.5, AUC=0.502
When using linear combination, the statistics are:
Feature A: coefficient: 0.5, P=0.6
Feature B: coefficient: 1.9, P=0.03
== AUC = 0.52
This shows that in the linear combination approach, the less important feature in the single-feature model (Feature B) increase its importance and vice versa for the more important one in the single-feature model (Feature A).
I am not sure how to explain this phenomina,
I would really appreciate if anyone has explanation and comments for this situation.
Thank you very much!
Topic auc logistic-regression
Category Data Science