I'm not sure how to implement this architecture. I'm following this thesis (pages 17-19) or this paper but I'm not sure how to train it. I want to use this to extract features from raw audio.

I know I have to compute the positive and negative correlations, but I don't know how to do this exactly since I can not find any detailed documentation of this.

What I have done so far is:

Positive correlation

To compute it I do a matrix multiplication: $$\langle xh\rangle^+ = \frac{x_{data}^Th_{act}}{batchsize}$$

Where $x^T_{data}$ is be my batch data transposed and $h_{act}$ are the hidden units activations. $h_{act} = relu(x_{data}W + bias_{hidden})$. Not sure if I have to add noise in this point.

Negative Correlation

To do this I use contrastive divergence (no tag??) with $k=1$

Same thing: $$\langle xh\rangle^- = \frac{x_{free}^Th_{free_{act}}}{batchsize}$$

but I'm really not sure how I have to compute these terms. So for me $x^t_{free}$ would be the samples of $\mathcal{N}(ΣWh+b;Σ)$, where $h$ here is activation from previous step ($h_{act}$) without noise. Lastly $h_{free_{act}}$ is computed as the same as the previous $h_{act}$ but using $x^t_{free}$

Is this correct? Because I don't know when I have to use samples or activations or maybe meanfields