FP16, FP32 - what is it all about? or is it just Bitsize for Float-Values (Python)

FP32 and FP16 mean 32-bit floating point and 16-bit floating point. GPUs originally focused on FP32 because these are the calculations needed for 3D games. Nowadays a lot of GPUs have native support of FP16 to speed up the calculation of neural networks. If you look at some benchmarks (https://blog.slavv.com/titan-rtx-quality-time-with-the-top-turing-gpu-fe110232a28e) you will see that GPUs that support FP16 are almost twice as fast calculating FP16 than FP32. Taking into account that newer cards that support FP16 (like NVidia 2080 series) are also about 20% faster for FP32 compared to their predecessor (1080) you get an increase of 140% to train FP16 neural networks compared to FP32 on previous cards.

But there is a caveat. Your neural network needs to be written using FP16 and it should also have the same accuracy. FP16 have lower accuracy by design, because they have much less bits to represent the same number. Some choose to use mixed precision models in order to be fast and accurate (https://hackernoon.com/rtx-2080ti-vs-gtx-1080ti-fastai-mixed-precision-training-comparisons-on-cifar-100-761d8f615d7f). But you see in the last link that the speed is faster for mixed precision, but is not 2 times faster as when you use pure FP16. You could also theoretically use FP32 weights and convert some of them to FP16 weights, but the accuracy could fall.

So in the end you need to understand whether you could rewrite your neural network to use FP16 fully or partially. If you cannot then you do not get any additional benefits from FP16 compatible cards.

The maximum value for FP16 is 65504 and the minimum is 5.96 × 10−8. The best precision for 5989.12345 will most likely be 5988.0 (played with bits on https://www.h-schmidt.net/FloatConverter/IEEE754.html) If this precision and magnitude is not enough for you then you could scale your data before training to fit FP16 and then train with double the speed. Or use mixed precision models where you have FP32 as input and then reduce precision in later layers.