When do I need Statistical Signifcance testing and when not?

Hi there I have a handful of questions regarding statistical significance testing. As a newcomer I have sometimes topics that I do not really understand entirely.

One of them is checking for statistical significance. For example, when I do A/B Testing I understand that I have to check whether my results are statistically significant (p value test) before looking for effect sizes.

1. Question: One question is if I only do Statistical Significance Tests in the context of Hypothesis Testing? This question comes up when I think about doing EDA before moving to train a model. Till now I haven’t done any statistical significance testing on datasets. I did some research and found out that this might be crucial for deep learning use cases. But often it was said that if you are using Machine Learning then there is no reasonable hypothesis about the underlying distribution, so it does often not apply. However one could you statistical significance testing for:

• After building a model to check on test set – “How well does the performance on the test set represent the performance in general?”
• check if performance metrics are statistically significant (but we do have other metric such as Cross Validation, Accuracy, Recall etc.?)

2. Question: I have seen another example where you also could use statistical significance test in a case where you assume that data has a normal distribution (hypothesis testing; e.g. Shapiro Wilk test). Would it not be to easier to check visually if the data is normally distributed or check for:

• Skeweness
• Kurtosis
• Mean=mode=media

Thanks in advance for your help!