Is GNU Octave a perfect place to code neural networks

No.

The deep learning community is overwhelmingly using Python as a language. PyTorch (maintained by Facebook) and Tensorflow (maintained by Google) are the main two frameworks. Since both frameworks are ultimately bindings on top of C++ code, it is pretty difficult to beat the performance that they give you.

R has some neural network packages, and so does Java and Julia, and I am sure other languages do too. But there is simply no reasonable evidence why you shouldn't use Python for deep learning.

I can only see two valid reasons to not use Python for deep learning:

1. You don't have a choice and have to use another language
2. You are capable of writing a numerically stable, hardware accelerated, deep learning framework from scratch in the language of your choice.

Also, you don't really need to "code" neural networks anymore. Frameworks do the heavy lifting for you. They have optimized implementations of pretty much everything you'd want to create the neural network you want.

No.

The key to implementing neural networks easily and fast is having automatic differentiation. This is the key feature that allows programs using Python libraries like Pytorch or Tensorflow (and Keras), to implement only the forward pass of a neural network and have the backward pass computed automatically.

Octave does not have automatic differentiation. While Matlab's deep learning toolbox seems to include automatic differentiation, I think it is not included with Octave. And even if it were, the deep learning community is mainly focused on the Python ecosystem, and the rest of the environments like R or Matlab are paid almost no attention in comparison.

As far as I know GNU Octave is an open-source alternative for Matlab. Although you can use Octave for machine learning (nnet package), I doubt you could use it for "faster coding" of neural networks.

Here are my thoughts on why that is the case:
When you say "fast" there are two interpretations, writing code fast and writing fast code.

• Writing code fast: Beating any of the python libraries seems to be a difficult task, especially keras. I have to admit, I never used Octave (but I have experience with Matlab), but I doubt you can write a simple network model + training / evaluation loop faster than with keras or pytorch.
• Writing fast code: Most libraries fall back to fast C++ routines (BLAS / LAPACK) when it comes to heavy numerical computations or implement functionality in C++ straight away. I assume Octave does the same. You may know that GPUs are the way to go today for large problems since they are much faster than CPUs for deep learning and I'm not sure if you can use a GPU together with Octave. If not, this would be for sure a huge disadvantage.

In the end you have to decide what fast means for you and what do you like to achieve. Here is a quick overview of current deep learning libraries. Here is a recent paper which benchmarks TensorFlow, Caffe, Torch and Theano in terms of speed for CPU as well as GPU. Unfortunately I did not found any comparison bewteen python (TF,Torch, ...) and Octave (nnet) / Matlab (Deep Learning Toolbox).