Feature selection with information gain (KL divergence) and mutual information yields different results

I'm comparing different techniques for feature selection / feature ranking. Two of the techniques under scrutiny are the mutual information (MI) and the information gain (IG) as used in decision trees, i.e. the Kullback-Leibler divergence.

My data (class and features) is all binary.

All sources I could find state, that MI and IG are basically "two sides of the same coin", i.e. that one can be tranformed into the oher via mathematical manipulation. (For example [source 1, source 2])

Yet, when I rank my features using the two measures they do not result in the same ranking order. But if the two measures are equivalent, shouldn't the ranking be the same?

Can someone help me understand why the rankings are different?

Topic mutual-information information-theory ranking feature-selection

Category Data Science

1
answered at

Remind that $I[X;Y]$ is symmetrical, but $KL(P \Vert Q)$ is not. $I[X;Y] = KL(P(X,Y)\Vert P(X)P(Y))$. Take a look if you are not computing $KL(P(X)\Vert P(Y))$.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.