How to find the number of operation ( multiplication or addition etc) required given a Keras model?

I want to implement an FPGA code or hardware code of a Keras model. As a first step, I want to find the number of mathematical operations required to evaluate a predicted output given a model. The model below is a two-class classifier and a sample of input is a vector of size 232X1. The model is:

model.add(keras.layers.Dense(5, input_dim=232, activation='relu'))
model.add(keras.layers.Dense(1, activation='sigmoid'))

The question is given in the model above, how many mathematical operations (plus, minus, multiplication, division) are required to find the output value. In my understanding since there are 5 output neurons in the first layer we have 5232 weights so we need to calculate 5232 multiplication in the first stage and next as we have 5 relu activation calculation. As there are no other layers except the last layer, which is just the output we need only 5 multiplication and 5 sigmoid calculation, and 5 addition.

Is the above approach correct?

Topic hardware keras

Category Data Science

1
answered at

To compute the number of elementary operations, you need to understand what is happening under the hood.

Let $x$ be an input vector of size $n$.

Given such a vector, a dense layer of $m$ units with an activation function $f$ will execute the following operation: $$a = f(Wx + b)$$ $W$ is the weight matrix associated with the dense layer (its size is $m \times n$) and $b$ is the bias vector (of size $m$).

We can derive from this formulation the following:

  • The number of multiplications is $mn$ (this comes from the definition of the product $Wx$).
  • The number of additions is $m(n-1) + m = mn$, where $m(n-1)$ comes from the definition of $Wx$ again, and $m$ comes from adding the bias vector $b$.

Then, $f$ is applied $m$ times (once on each component of the resulting vector $Wx +b$).

Applying this to your example:

  • Layer 1 does $5 \times 232 = 1160$ multiplications and $1160$ additions, and applies $ReLU$ 5 times (because $m=5$, $n=232$ and $f=ReLU$),
  • Layer 2 does $1 \times 5 = 5$ multiplications and $5$ additions, and applies $\sigma$ the sigmoid function 1 time only (because $m=1$, $n=5$ and $f=\sigma$)

The total number of multiplications is: $1165$ and the total number of additions is: $1165$.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.