Using Z-test score to evaluate model performance

I think I know the answer to this question but I am looking for a sanity check here: Is it appropriate to use z-test scores in order to evaluate the performance of my model?

I have a binary model that I have developed with a NN in Keras. I know the size of my (equally balanced) training set and it has a proportion of 0.5 (duh!). I know that with my business use case, false-positives are financially expensive so I'm focusing on Precision as my metric. So, in validation, can't I take that Precision metric as a proportion of my validation set (which I also know the size of) and then get the z-test score calculation? That should give me a threshold for validation Precision at which my model is doing more than just flipping a coin.

Can someone confirm my line of thinking or am I way off base here?