weight sharing among neurons at same depth

Question

weight sharing among neurons at same depth

Ilya.K.

2020年8月22日 02:58

I'm trying to understand some visual illustrations about the wight sharing in the Convolutional Neural Network as following:

In this picture we see that for different outputs different inputs share the same weights, as the weights are associated with the color (same color - same weight).

So in the Stanford example we can see it eather:

As for example, for the o(0,0,0)=5 and for o(0,1,0)=7 both we used the w0[0],w01,w2 weights, and it fits the picture above.

Now there is this picture:

And I'm not fully understand the 3 estimation. It says that the same color means the same weight, but as I can tell in the picture we see a filter, each neuron, let's mark it as $f$ refers to the multiplication of $f(x)=wx(+b?)$, and as much as I can tell from the Stanford exmaple, the weight can be all different and not the same color, and the weight sharing isn't representable on the filter, but only in any view that includes the input, as in the first image, so isn't it a bit misleading ?

Or maybe I just didn't understand it and anyone could explain it more briefly ?

Topic neural convolution neural-network

Category Data Science

cag51 · Accepted Answer · 2020年8月22日 02:58

I agree that these pictures are a bit confusing. I think they are all hinting at the following:

As you know, CNNs involve splitting up the image into many small "blocks" and multiplying each block pairwise by a filter (actually, by several different filters). Weight sharing means that all the blocks use the same set of filters -- blocks in the upper-left corner, middle, and bottom of the image all use the same set of filters.
There is no weight sharing across the channels -- we don't use the same filter banks for the red channel and the blue channel. This makes sense because different classes have different signatures in each channel. There are various ways to think about this. Personally, I find it easiest to visualize that both the "block" and the filter are three-dimensional, and the pairwise multiplication is across all three dimensions.

So, to your pictures:

The top one shows that $x_{i-1}$, $x_i$, and $x_{i+1}$ are three different channels, and so use three different weights (colors). But when switching from block $i$ to block $j$, the same weights (colors) are reused.
The middle one has four columns:
- The left column is the image itself -- three rows, one for each channel.
- The second column is one filter -- also three dimensional. This also shows that each channel of the image is multiplied pairwise against each filter.
- The third column I assume is a second filter bank, which works the same way as the original filter bank (CNNs usually apply many different filters at each stage of the network, so that multiple patterns can be learned).
- Then the fourth column is the output, i.e., the next layer in the network.
The bottom one is not representing the filter bank, but actually represents the subsequent layer of the network (i.e., the output from the convolutional operation). In this subsequent layer, each "channel" refers to the output from a different filter in the preceding layer. Given this, the picture makes sense: everything within the same channel was formed by applying the same weights to different parts of the image. And each channel is in a different color because they were formed using a different filter, and therefore different weights.

weight sharing among neurons at same depth

About