using logsumexp in softmax
I saw this equation in somebody's code which is an alternative approach to implementing the softmax in order to avoid underflow by division by large numbers.
softmax = e^(matrix - logaddexp(matrix)) = E^matrix / sumexp(matrix)
logsumexp = scipy.special.logsumexp(matrix, axis=-1, keepdims=True)
softmax = np.exp(matrix - logsumexp)
I understand that when you log equations that use division you would then subtract, i.e. log(1/2) = log(1) - log(2). However, in the implantation of the code above, shouldn't they also log the matrix
in order to subract the logsumexp
?
Topic softmax mathematics
Category Data Science