Derivative of MSE Cost Function

Question

Derivative of MSE Cost Function

Adilson Medronha

2022年5月18日 01:59

The gradient descent:

$\theta_{t+1}=\theta_t-a\frac{\partial}{\partial \theta_j}J(\theta)$

But specifically about $J$ cost function (Mean Squared Error) partial derivative:

Consider that: $h_\theta(x)=\theta_0+\theta_1x$

$\frac{\partial}{\partial\theta_j}J(\theta) = \frac{\partial}{\partial\theta_j}\frac{1}{2}(h_{\theta}(x)-y)^2$

$\ \ \ \ \ \ \ \ \ \ \ \ =2\frac{1}{2}(h_{\theta}(x)-y)*\frac{\partial}{\partial\theta_j}(h_{\theta}(x)-y)$

$\ \ \ \ \ \ \ \ \ \ \ \ = (h_{\theta}(x)-y)*\frac{\partial}{\partial\theta_j}(\sum_{i=0}^{n}\theta_ix_i-y_i)$

$\ \ \ \ \ \ \ \ \ \ \ \ = (h_{\theta}(x)-y)x_j$

It´s not clear to me how $x_j$ is calculated:

$\frac{\partial}{\partial\theta_j}(\sum_{i=0}^{n}\theta_ix_i-y) = x_j $

Can anyone help me to understand in detail this part of the partial derivative? Thanks in advance.

Topic mathematics deep-learning machine-learning

Category Data Science

Neil Slater · Accepted Answer · 2022年5月17日 07:49

Any term $f$ that is not a function of $\theta_j$ in any equation will have a partial derivative $\frac{\partial}{\partial\theta_j}(f) = 0$. Importantly, no $x_i$, $y$ or $\theta_{i \ne j}$ depend in any way upon $\theta_j$, so they are effectively constants when figuring out the partial derivative. This is also true for any function of them, provided that also does not depend on $\theta_j$.

So for example $\frac{\partial}{\partial\theta_j}(f(y)) = 0$, $\frac{\partial}{\partial\theta_j}(f(y)\theta_j) = f(y)$ and $\frac{\partial}{\partial\theta_j}(f(y)\theta_j^2) = 2f(y)\theta_j$

From this:

$\frac{\partial}{\partial\theta_j}(\sum_{i=0}^{n}\theta_ix_i-y) = x_j $

When $i \ne j$, then $\frac{\partial}{\partial\theta_j}(\theta_ix_i-y) = 0$, because no term inside the brackets depends on $\theta_j$.

When $i = j$, then $\frac{\partial}{\partial\theta_j}(\theta_jx_j-y) = x_j$. Only the term $\theta_jx_j$ depends on $\theta_j$, and it is a linear multiplication.

Derivative of MSE Cost Function

About