Visualizing mutual information of each convolution layer for image classification problem

Question

Visualizing mutual information of each convolution layer for image classification problem

Ashesh

2021年7月13日 00:01

I recently came across this paper where the author has proposed a compression based theory on understanding the layers of a DNN. In order to visualize what was going on the authors showed Figure 2 of this paper which is also shown as a video here. For my image classification problem I want to visualize the mutual information exactly in this format. Can someone kindly explain to me how to calculate this numerically for images passing through conv layers in a convolutional neural network. Do I need to fit a kernel density estimator for a vectorized feature maps at each layer and numerically calculate entropy or is there a simpler way to do this? Thanks in advance

Topic cnn mutual-information information-theory

Category Data Science

Fred Guth · Accepted Answer · 2021年6月5日 10:10

Measuring the real mutual information is impossible because you would need to know the true distribution $P(X,Y)$ that is what you want to learn in the first place. You can proxy the real mutual information with an estimation, the log determinant of the Fisher Information Matrix, for example.

I don't know how the video was made, but it seems related to the Information Bottleneck Theory of Tishby et al. Tishby showed videos like this using a synthetic dataset for which he knew the real distribution in advance. You will find in the papers of Achille and Soatto justification for using the Fisher Information instead.

Visualizing mutual information of each convolution layer for image classification problem

About