How to assess whether neural network performance is associated with a nuisance variable

Problem

I have a convolutional neural network model which intakes a video and outputs a continuous variable. I want to assess whether the performance of the model is associated with another continuous variable (age; not included in the model).

Solution attempt

If this were a linear regression model, I think I could do a Spearman rank correlation test: basically, plot the absolute value of the residuals (true value - predicted value) against the nuisance variable (age), then calculate the Spearman rank correlation between the absolute values of the residuals and age, and determine if they are significantly correlated.

Can I use the same approach here? Would another approach be more appropriate?