Neural Network not Deep

Alexnet was a revolution in image recognition in 2012 (https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf) and the paper was titled "Deep convolutional neural networks". Referring to neutral networks as "deep" was unusual at that time.

They used 8 layers, and hugely increased the state of the art (SOTA).

I conclude from that, that at least in 2012, 8 layers was considered deep.

Since then, I'd argue that 8 layers is no longer considered deep, but pretty standard for many image processing tasks.

I'd therefore argue that the definition of deep has changed and increased over time, and we've become better at training ever-deeper networks.

One of the reasons deep neural networks have taken off is because GPU hardware has improved. The computations needed for practical deep networks are often highly parallelizable. Thus, I think one indicator of "deep" in addition to the standard criteria of ">= 2 layers" is that the network is structured so that many computations can be done in parallel.

Conversely, practical networks which are not highly parallelizable are often not "deep".

The single layer neural network - the Perceptron - was first implemented by Frank Rosenblatt in 1957 using hardware (!!) - that is: resistors, electrical motor, and a potentiometer! The perceptron had one layer of neurons, (not including the output layer), and it could tell the difference between simple shapes and letters.

According to Google's N-Gram Viewer it was a long time before the term 'Deep Neural Network' was used in a book (~1980), and much longer (~ late 2000's) before it gained popular usage:

So to answer the question, I suspect although it's somewhat subjective, I think the Perceptron is the only true single-layer neural network in your diagram, and therefore the others would be considered 'deep'. Some may consider other 'shallow' neural networks (like 'feed forward netural nets') to not be deep, but it's worth noting that although shallow in structure, the term 'feed forward netual network' didn't come into popular parlance until slightly after the term 'deep neural network' (see here).

The term 'deep' in deep learning is not well defined, but is generally meant to apply to networks that have stacked multiple layers on top of each other to create a deep network. It is therefore up for debate when a network is considered deep or not, but you could use the definition that a deep network has two or more hidden layers. From your overview, examples of non-deep networks would therefore be the perceptron and simple feed forward network. The activations used in the network have no relation to whether a model is deep or not as it only relates to the number of layers in the network.